Tags
Business, Design, experiment, HCI, human factors, innovation, politics, science, Study, usability, UX
Study Slain by Swamp Monster!
I’m trying a new format for blog posts.
For those of you in a hurry, to get to the “bottom line” of this post, you can skip the story and go right to the bold-faced “lesson” at the end. I’d really you rather read the whole thing of course, but I know some readers are harried and hurried. So, if that describes you right now, feel free.
——————————————————
In the early 1980’s, researchers at the IBM Watson Research Center invented a new kind of system. Originally, this was called the “Speech Filing System.” It was initially designed to allow so-called “office principals” (sales people, managers, executives, engineers, etc.) to dictate letters and memos which could then be typed up by the pool of typists. Instead of requiring each “office principal” to have (or borrow) a dedicated piece of dictation equipment, they could accomplish this dictation from any touch tone phone. While this offered some savings in cost and convenience in the office, it was even more wonderful on the road. People did not have to take their dictation equipment with them on their travels. They could use any touch-tone phone.
Photo by Pixabay on Pexels.com
The system was invented largely by tech-savvy psychologists (including Stephen Boies, John Gould, John Richards, & Jim Schoonard). When they observed people actually using the system, they discovered that the trial users more often used the ancillary messaging facility than they did the “real” dictation features. So, the system was redesigned and repurposed and then renamed, “The Audio Distribution System.” In some ways, using the “Audio Distribution System” was much like leaving a message on an answering machine. However, there were some crucial differences. Typically, a person calling someone and encountering, instead of a human being, a message asking them to leave another message was somewhat taken aback. Many messages on answering machines went something like this: “Hi. Stephen? Oh, you’re not there. OK. This is John. I was hoping … well, I thought you’d be in. Uh. Let’s see. You know what? Call me back. We need to talk.” And, when Stephen discovered that he had a message, he might listen to it and call back John. “Hi, John. Stephen here… I … oh. OK. A message. Sorry. You just called me. Well, um. I’m not sure what you wanted to talk about so. Call me back when you get a chance.”
Photo by Public Domain Pictures on Pexels.com
By contrast, when someone called the “Audio Distribution System” they knew ahead of time they’d be interacting with a machine. So, they could compose a reasonable message before calling the system. Hence, the messages tended to be more coherent and useful; e.g., “Hi, Stephen. This is John. If it’s okay with you, I’m taking off this Friday for a long weekend. If you have any issues with that, let me know.” See? Easy and efficient.
A second critical difference was that you could listen to your message and edit it. People didn’t do this so often as you might think, but it was comforting to know that you could in case you really messed up. (For instance, a person might say, “You are fired!” when all along they meant to say, “You are NOT fired.”).
Introducing any new system will have consequences, both intended and unintended. I wanted to see what some of these consequences might be. Corporations, IBM included, like it when they sell lots of product and make lots of money. A related question then was – what is the value of this product to the customer? Why should they want to buy it?
One hypothesis I wanted to test out was that such a system would increase people’s perceived Peace of Mind. After you leave a meaningful message for someone, you can “cross off” that little item off your mental (or written) “to do” list. By using the Audio Distribution System, I thought one of the user benefits would be increased “Peace of Mind” because they would be able to leave a message any time and any place they had access to a touch tone phone. They could save their working memory capacity for “higher level” activities such as design, problem solving, and decision making. We were going to roll out a beta test of the Audio Distribution System at the divisional headquarters for the IBM Office Products Division (OPD), in Franklin Lakes, New Jersey. Not coincidentally, OPD would be the division selling the Audio Distribution System (just as they were now selling dictation equipment). Before the trial commenced, I developed a questionnaire designed to get at how much people felt harried, too busy, coping, etc. The hope was that I could compare the “Peace of Mind” scores of people who did and did not get the Audio Distribution System and perhaps show that those with the system felt more at peace than those without. I could also compare “before and after” for those internal beta customers who had the system.
Photo by Suraphat Nuea-on on Pexels.com
Before I was to roll-out and administer the “Peace of Mind” questionnaire to a sample of people at the OPD Franklin Lakes location, guess what happen just two days before the beta roll-out? OPD was re-organized out of existence! The people who worked there would now be looking for another job elsewhere in IBM (or, failing that, just elsewhere period). The beta trial was cancelled. In any case, even if it hadn’t been cancelled, the impact of the re-organization would have completely swamped (in my estimation) the impact of this new tool. Moreover, it struck me as insensitive and slightly even unethical to ask people to fill out a questionnaire about how hassled they were feeling just days after finding out their entire division had been blown up. How would you react if some psychologist from the Research Center showed up asking you to fill out a questionnaire two days after finding out you no longer had a job?
Photo by Anna-Louise on Pexels.com
————————————————-
What is the lesson learned here? You have to understand what is going on in the lives of your users over and above the functions and features directly related to your product or service. Of course, there is always a fairly good chance that some of your users will have overwhelming things going on in their lives that will impact their reactions to your product. Generally you won’t know about divorces, deaths in the family, toothaches, etc. But if something is impacting all your users, you’d best be aware of it and act accordingly.
————————————-
Audio Distribution System – NY Times
Longer explanation of Audio Distribution System
Video of Audio Distribution System’s cousin: “The Olympic Message System”
———————————————————————
petersironwood 
Thanks for this! I’m about to conduct longitudinal user research looking into educational computer games and their impact on mathematics anxiety. It’s becoming abundantly clear some kind of diary study element should be introduced to capture outside factors that could affect the results. However, it seems the dropout rate could get rather high given the level of participant commitment involved. Then there’s the issue of having an eye-watering amount of data to analyse.
You might consider some sort of incentive to get people to stick with the diary info; e.g., a lottery prize. Maybe chances are proportional to the number of pages? Or, feedback? IDK. It was long ago, but my high school English teacher used to have us make collages (mainly cut-outs from magazines) that were supposed to represent “us” – and then each collage was given to a different student who did not know the identity. It might be possible that asking kids to represent their thoughts and feelings with links and images — as well as words — might make it more interesting and motivating? Maybe there’s even a way to involve them in the data analysis so it isn’t so onerous for you. Maybe if they were analyzing data about themselves, some might be more interested?
Thought I’d let you know, I’m finalising my methodology at the moment. Essentially, I’ll be incentivizing students with daily competitions of Amazon vouchers. So hoping the gambling approach will keep them engaged for long enough.
Interesting analogy regarding visualisation of thoughts and feelings. For now, the plan is to instruct students to highlight how anxious they using a 1-10 Likert scale, and vocalising their thoughts while playing the game.
Have also considered getting participants to analyse data too. Perhaps by allowing them to watch footage of themselves playing the games and reflecting on their thoughts at the time?
Thanks for this! I’m about to conduct longitudinal user research looking into educational computer games and their impact on mathematics anxiety. It’s becoming abundantly clear some kind of diary study element should be introduced to capture outside factors that could affect the results. However, it seems the dropout rate could get rather high given the level of participant commitment involved. Then there’s the issue of having an eye-watering amount of data to analyse.
I hope you’re successful! My oldest daughter came back from school in the 7th grade one day and announced to me that she and her friends had decided that they *hated* their math teacher and *hated* math and that they were no good at it. I was astounded. She had always had (and still has) a brilliant mind and had always been good at math. I couldn’t seem to shake her of this new belief.
It’s very common actually, particularly if students have bad experience with teachers (usually humiliation in front of peers, or received inadequate help and fallen behind) in the classroom. It also carries across at university level too, mostly for students who have avoided maths since high school for the very same reasons.
It’s very common actually, particularly if students have bad experience with teachers (usually humiliation in front of peers, or received inadequate help and fallen behind) in the classroom. It also carries across at university level too, mostly for students who have avoided maths since high school for the very same reasons.
Pingback: I’d Like Sauerkraut on the Ice Cream | petersironwood