This is part of a series on experiences in my career in Human Computer Interaction and some lessons learned.
I joined IBM Research on the winter solstice of 1973. I had earned a Ph.D. in Experimental Psychology from the University of Michigan and for the previous few years, I had managed a research project at Harvard Medical School on the “Psychology of Aging.” At the time, I was married and had three small children. I mention this because I was funded by so-called “soft money” which basically meant that my salary depended on a research grant. I helped write a renewal of the grant but the decision was “deferred”; that is, it was neither funded nor unfunded. Then, it was deferred again. This meant that if the grant were not funded, I would only have a few weeks to find a new job. That seemed far too short so I began to look other places for a job.
Lessons Learned: #1 If you want continuity of personnel in your laboratory, make sure you have overlapping and multiple grants or other sources of income.
In this case, the grant actually was ultimately approved, but by that time, I had already agreed to join IBM Research. That turned out to be fine, by the way. It was a wonderful place to work.
One of the reasons that I got the job at IBM was that I already knew something about computers. I had taken several computer science courses in grad school along with the needed psych courses. More importantly, our “Psychology of Aging” study was run by a PDP-8 and I had programmed the computer to run our suite of experiments and to do data analyses on the results. I had taken a week-long course at DEC in Maynard, Massachusetts on the assembly language, another week-long course on the machine language, and another week-long course actually tracing the circuitry with a probe and oscilloscope. I felt I “understood” the PDP-8 at a fairly deep level.
At IBM, I did not have that familiar machine. Instead, I was connected to a mainframe via a dumb terminal. The first day at IBM, I got my userid and tried to log on to APL (A programming language I had not used before). I tried following the manual but I could not seem to get logged on. After hours of trying, I finally gave up and went down to the computer room and found someone willing to help. I showed him the logon instructions I was trying to follow and he immediately said, “Oh, yeah, that doesn’t work any more. We changed that months ago. Here’s how you need to do it now.” The manual I had may have looked new, but it was out of date.
Lessons Learned: #2 Manuals can be wrong. These days, most are online. But they can still be wrong.
Lessons Learned: #3 Someone who knows how to do something can save you hours with a few minutes of their time.
Of course, it’s more respectful, efficient, and a better learning experience if you can figure it out on your own. But sometimes you can’t. My stumbling block was not due to an error in logic, or a lack of in-depth knowledge. It was simply that the computer center administrators had changed something arbitrary so that the documentation I was given about how to log on for the first time was no longer accurate.
In order to teach myself APL, I wrote a very small program to “predict” how long I was going to live “based on” some behaviors that I was interested in controlling. My main goal was to learn APL. My secondary goal was to motivate myself, for instance, to exercise more, lose weight, and not drink too much alcohol. I had no intention or pretensions of making this prediction “accurate.” If I had been doing a consulting gig for an insurance company setting life insurance rates, for example, I would have given far more attention to see precisely what the real data were and incorporated many more variables into the regression model.
Here’s a link,
by the way, to a more accurate model than the one I used, but it’s still simple to use. Note that my goal was to motivate myself and so I intentionally exaggerated the impact of those behaviors I was trying to change. I had programmed it. I knew how “bogus” the calculation was — nonetheless — here’s the interesting thing though:
Lessons Learned #4: Even an over-simple model that the user knows is over-simple can still motivate change.
At last we come to the actual project I worked on — the usability and learnability of Query By Example. One of my colleagues, Moshe Zloof, invented the language for relational data bases. He had designed the language but not yet implemented it. I did not immediately test the design; first, I sought to understand it. In seeking to understand it in depth, prior to testing it, the two of us had some sense-making discussions. Moshe improved the design; in particular, our discussions uncovered some ambiguities and inconsistencies that were not at all obvious when he simply gave talks about the design. This brings me to the next lesson learned which has proven true in nearly every study of early stage designs that I’ve been involved with over the course of five decades.
Lessons Learned #5: Don’t just accept a surface description of something; understand it as deeply as you can before designing a study.
In this particular case, it was possible for me to understand it in some depth. Relational data bases and second order logic are things I was capable of understanding. If it had been an interface to running a nuclear reactor or using the artificial heart that Moshe had designed earlier in his career, that would have been a much more difficult task for me.
I wanted to understand, not just the “logic” of Query By Example, but also possible contexts of use. For instance, my manager & I visited Burlington, Vermont to talk with IBMer’s who actually used query languages to understand what was happening in chip production lines. At one point, a particular production line that had been producing nearly 100% perfect chips starting having a much higher error rate. Using their query facility, they were quickly able to diagnose the cause of the change which was a supplier of one of the raw materials using a different source. In turn, this meant a slightly different profile of trace impurities in the substrate. Of course, this is only one example, but to me, understanding something in depth means not only understanding its internal logic but also understanding real users, their real tasks, and their context of use.
I won’t go into all the details of the pencil & paper study or the results. High School students and then college students were taught the basics of the language and then given a simple relational data base and a set of questions stated in English which they had to translate into Query By Example. Briefly, the bottom line was that Query By Example was easy to learn and easy to use. However, there were still questions that people had difficulty with. In analyzing the data and doing some further experiments, the difficulties that people tended to have, stemmed not so much from Query By Example per se, but from what I much later came to call “labelism” — that is, confusing a label with the thing that label refers to.
Here’s a simple example of the type of confusion we saw. In Query By Example (and other query languages) there is usually an OR operator and an AND operator. (These operators can be important for doing advanced queries with search engines as well). If you are interested in getting a list of pets you might adopt and you’re willing to adopt dogs or cats, you might ask for “cat OR dog.” If you only want long-haired cats, you might ask for “cat” AND “long hair.”
English, however, can be tricky.
If you and I (as opposed to you and a query language) are having a conversation, you might say, “I hear there are many pets that need to be adopted.”
I say, “Yes, there are all kinds of pets. There are snakes, dogs, turtles, rabbits, cats…”
You say, “Let me stop you right there. I’m only interested in adopting cats and dogs. Those are the only animals I’d want to adopt.”
See what you said there? You exact words included: “…cats and dog.” If you put “Cats AND dogs” into a query against the data base of available pets, however, you will get the null set (that is, nothing) back. There are no animals who are both cats and dogs! (Though my part Main Coon cats do play fetch like dogs).
When people were presented with an English statement that included the English word “and” — regardless of the actual syntax and context, some of them had difficulty using the OR operator. If instead, the query in English had set up like this: “Oh, I don’t want reptiles. I’d be happy with adopting a cat or a dog, however” then, they’d have no problem translating it into the OR operator in the query language.
Lessons Learned: #6 Sometimes the difficulty people have in using a product, a service, or a prototype is not due to the interface details but with the structure of the task, their background, and their training.
By analogy, you will not allow me to beat Nadal or Djokovic at tennis by giving me a better tennis racquet! (Although if you gave one of them a toothpick for a tennis racquet, I might have a shot).
That sounds obvious and even absurd, but I promise you, some companies get so greedy that they want you to design a system that allows people who do not understand the task and have minimal background and training to nonetheless be able to perform that task.
One example you may have run into is having “help desk” personnel who have no understanding of a product go through a script to help you “solve your problem.” Sometimes, it works. But many times it doesn’t. When it does not work, you might not be able to “fix” the system by making the interface to the scripts easier to use for the help desk folks. The problem is much deeper (in some cases). Yes, a really bad interface can make it difficult even for a really knowledgeable and capable person to do the job. But often, even a really great interface cannot always substitute for actual expertise.
Other posts on problem formulation: