“Wizard of Oz”

18 Friday Jun 2021

Posted by petersironwood in Uncategorized

Tags

HCI, IBM, research, usability, UX, Wizard of Oz

(Some Lessons Learned from studies in Human-Computer Interaction/User Experience conducted at IBM Research in the mid-70’s.)

Wizard of Oz

One of the studies I conducted at IBM Research in the mid 1970’s was part of an effort to do “Automatic Programming” — a department under Pat Goldberg. The first level manager I worked with was Irving Wladawsky (later Irving Wladawsky-Berger). His group wanted to develop a system that would allow the owner/operator of a small business to type requirements into a computer in English (or something English-like) and have the system itself produce RPG code to run the business so described.

The underlying motivation from an IBM business perspective was that many small businesses could well afford a computer to do inventory, fulfill orders, etc. but they couldn’t afford to hire programmers to create such a system from scratch. The small business owner in the mid-1970’s did not program! Yet, for the most part, they understood how their business worked. The notion was that a natural language understanding and generation program could dialogue with the user/owner and through that process, understand their “business rules.” No costly programmers needed!

An interesting side note: at that time, we were told that IBM corporate forbade us to use the terms “Artificial Intelligence” or “Robotics” to describe our work because some PR firm had determined that these terms were too scary for the general public. So, IBM had research in “mechanical assembly” but not “robotics.” We had work in character recognition, speech recognition, handwriting recognition, automatic program generation, and compiler optimization. But no work in “Artificial Intelligence.” (Wink, wink, nod, nod).

Labelism: Confusing a thing with the label for that thing.

Another interesting side note: I worked at IBM Research for a dozen years; started an AI lab at NYNEX where I worked another 13 years; came back to IBM Research and several years later found myself working on the same problem! We were still trying to make a system to allow small businesses to generate their code automatically. In my second iteration, rather than using natural language, we were trying to make the specification of business rules in a graph language that was intuitive enough for business owners. This was a different approach, but trying to address the same underlying desire: to bring computing to small business without incurring the heavy costs of programming and maintenance.

Let’s return to iteration one — the natural language approach @ 1975. Well, one issue was that no-one had a natural language program that even approximated being able to do the job. So…how to study people’s interaction with a system that doesn’t exist?

We used an approach that my colleague Jeff Kelly called the “Wizard of Oz” technique; viz., use a human being (in this case, me) to simulate how the system might work and record people’s behavior. In this way, we could discover many of the issues that such a natural language programming system would have to deal with. I had already had plenty of experience interacting with a computer; and I had acting experience. I could “play the part” of a computer fairly well as I typed in my questions and answers.

(Description of “The Wizard of Oz” technique).

IBM Research in Yorktown had roughly a thousand people including not only scientists, programmers, and engineers but also a number of business people (who did not know how to program). I knew some of them from playing tennis and table tennis and we used those folks as initial subjects. What did I find? Good news and bad news.

Dealing with natural language is tricky for many reasons. One of those reasons is that English, including the English that people normally use to describe their business, is filled with words that have multiple meanings; e.g., “file”, “run”, “program”, “object”, “table”, etc. But here is the good news: although it’s true that many English words have many meanings, when these business people described business procedures, almost all of the lexical ambiguity vanished! The program to understand business English would not have to distinguish between a business file and a nail file; it wouldn’t have to worry about distinguishing a run in baseball or a run in stockings from a run of the payroll program; it wouldn’t have to distinguish between the table in a relational data base and the table in your dining room. The domain would mainly constrain! That’s the good news.

The bad news was dialogue management. How can the machine recognize a misunderstanding and how can it correct it? To make matters worse, while business people were fairly consistent in the way they described how their business ran, they were not consistent in how they talked about the communication. If a human being senses that another one is misunderstanding, then, depending on context they might: raise their eyebrows, say “Huh?”, “Come again?”, “What?” “I think I lost you.” “WTF?” “Are you kidding?”, “We’re on different wavelengths,” “I don’t get it.” “But…wait.”

Sometimes, these are referred to as “meta-comments.” Here’s a simple example that took place in the study.

One of the business people told me about various discounts. I had assumed (playing the part of the computer) that he was talking about discounts for items that were being discounted due to inventory management. I recorded all the various percentages and so on. Then, he said, “Now, we also give discounts for various items.”

At that time, most natural language systems of that era simply ignored words like “now” and “also” in this context. Stepping out of my role as a “computer system” and thinking about from the perspective of a human conversational partner though, these words are crucial! What it signals is a change in topic. In the larger context of our conversation, it shows that everything that had just been said, which I thought had been about item discounts, was not about item discounts!

This is just one example, but there were many more. In my more recent experience interacting with various computer dialogue systems, being able to recognize the signals of miscommunication and being able to repair misunderstandings is still not very well-handled more than four decades later.

I’d be interested in any pointers you have to a system that you think deals with meta-communication in a natural and robust manner. I do not think that it is beyond the pale of possibility. The general categories of the ways that people misunderstand each other is not infinite. John Anderson developed excellent tutoring systems for LISP and geometry and those systems worked something like human tutors in that, the tutor inferred the mental model of an individual student and focused instruction on correcting any misconceptions. My intuition is that a generic system built with equal complexity could deal with most of the issues as well as the average human being deals with them; i.e., imperfectly.

—————————————

Lessons Learned: #1 You can test aspects of a system even before it’s built or even completely defined. One method that has been used many times: “Wizard of Oz.”

Lessons Learned #2: Language used by professionals to talk about their domain is much more constrained in terms of lexical ambiguity than is language when considered by all native speakers.

Lessons Learned #3: People in “our culture” (i.e., US business culture) do not have an agreed upon and consistent vocabulary for talking about communication nor a consistent process for dealing with them.

Lessons Learned #4: Speaking of communication errors, I don’t recall why, but it was about this time, that I realized that my notion about how research results would be transferred to other parts of IBM was a complete and utter fantasy. I hadn’t articulated it, but it was basically that I would do research, write the results up for publication in scientific journals for an academic audience and publish Research Reports which would be eagerly consumed by anyone who needed to know. I’m not proud of this. LOL. But that’s really kind of how I viewed it. And, then, after a few years, I realized that it really mainly came about through relationships. That was something that people had been showing me all my life, but which I don’t think anyone ever stated it explicitly enough.

———————————-

Author Page on Amazon

The Myths of the Veritas (an exploration of leadership & ethics in free, no ads fiction)

Index to a Pattern Language for Collaboration and Teamwork

Experiences in Human-Computer Interaction

Post on “The Story of Story”

Query By Example

15 Tuesday Jun 2021

Posted by petersironwood in Uncategorized

≈ Leave a comment

Tags

expertise, HCI, IBM, QBE, research, usability, UX

This is part of a series on experiences in my career in Human Computer Interaction and some lessons learned.

I joined IBM Research on the winter solstice of 1973. I had earned a Ph.D. in Experimental Psychology from the University of Michigan and for the previous few years, I had managed a research project at Harvard Medical School on the “Psychology of Aging.” At the time, I was married and had three small children. I mention this because I was funded by so-called “soft money” which basically meant that my salary depended on a research grant. I helped write a renewal of the grant but the decision was “deferred”; that is, it was neither funded nor unfunded. Then, it was deferred again. This meant that if the grant were not funded, I would only have a few weeks to find a new job. That seemed far too short so I began to look other places for a job.

Lessons Learned: #1 If you want continuity of personnel in your laboratory, make sure you have overlapping and multiple grants or other sources of income.

In this case, the grant actually was ultimately approved, but by that time, I had already agreed to join IBM Research. That turned out to be fine, by the way. It was a wonderful place to work.

One of the reasons that I got the job at IBM was that I already knew something about computers. I had taken several computer science courses in grad school along with the needed psych courses. More importantly, our “Psychology of Aging” study was run by a PDP-8 and I had programmed the computer to run our suite of experiments and to do data analyses on the results. I had taken a week-long course at DEC in Maynard, Massachusetts on the assembly language, another week-long course on the machine language, and another week-long course actually tracing the circuitry with a probe and oscilloscope. I felt I “understood” the PDP-8 at a fairly deep level.

At IBM, I did not have that familiar machine. Instead, I was connected to a mainframe via a dumb terminal. The first day at IBM, I got my userid and tried to log on to APL (A programming language I had not used before). I tried following the manual but I could not seem to get logged on. After hours of trying, I finally gave up and went down to the computer room and found someone willing to help. I showed him the logon instructions I was trying to follow and he immediately said, “Oh, yeah, that doesn’t work any more. We changed that months ago. Here’s how you need to do it now.” The manual I had may have looked new, but it was out of date.

Lessons Learned: #2 Manuals can be wrong. These days, most are online. But they can still be wrong.

Lessons Learned: #3 Someone who knows how to do something can save you hours with a few minutes of their time.

Of course, it’s more respectful, efficient, and a better learning experience if you can figure it out on your own. But sometimes you can’t. My stumbling block was not due to an error in logic, or a lack of in-depth knowledge. It was simply that the computer center administrators had changed something arbitrary so that the documentation I was given about how to log on for the first time was no longer accurate.

In order to teach myself APL, I wrote a very small program to “predict” how long I was going to live “based on” some behaviors that I was interested in controlling. My main goal was to learn APL. My secondary goal was to motivate myself, for instance, to exercise more, lose weight, and not drink too much alcohol. I had no intention or pretensions of making this prediction “accurate.” If I had been doing a consulting gig for an insurance company setting life insurance rates, for example, I would have given far more attention to see precisely what the real data were and incorporated many more variables into the regression model.

Here’s a link,

https://www.death-clock.org

by the way, to a more accurate model than the one I used, but it’s still simple to use. Note that my goal was to motivate myself and so I intentionally exaggerated the impact of those behaviors I was trying to change. I had programmed it. I knew how “bogus” the calculation was — nonetheless — here’s the interesting thing though:

Lessons Learned #4: Even an over-simple model that the user knows is over-simple can still motivate change.

At last we come to the actual project I worked on — the usability and learnability of Query By Example. One of my colleagues, Moshe Zloof, invented the language for relational data bases. He had designed the language but not yet implemented it. I did not immediately test the design; first, I sought to understand it. In seeking to understand it in depth, prior to testing it, the two of us had some sense-making discussions. Moshe improved the design; in particular, our discussions uncovered some ambiguities and inconsistencies that were not at all obvious when he simply gave talks about the design. This brings me to the next lesson learned which has proven true in nearly every study of early stage designs that I’ve been involved with over the course of five decades.

Lessons Learned #5: Don’t just accept a surface description of something; understand it as deeply as you can before designing a study.

In this particular case, it was possible for me to understand it in some depth. Relational data bases and second order logic are things I was capable of understanding. If it had been an interface to running a nuclear reactor or using the artificial heart that Moshe had designed earlier in his career, that would have been a much more difficult task for me.

I wanted to understand, not just the “logic” of Query By Example, but also possible contexts of use. For instance, my manager & I visited Burlington, Vermont to talk with IBMer’s who actually used query languages to understand what was happening in chip production lines. At one point, a particular production line that had been producing nearly 100% perfect chips starting having a much higher error rate. Using their query facility, they were quickly able to diagnose the cause of the change which was a supplier of one of the raw materials using a different source. In turn, this meant a slightly different profile of trace impurities in the substrate. Of course, this is only one example, but to me, understanding something in depth means not only understanding its internal logic but also understanding real users, their real tasks, and their context of use.

Photo by Chokniti Khongchum on Pexels.com

I won’t go into all the details of the pencil & paper study or the results. High School students and then college students were taught the basics of the language and then given a simple relational data base and a set of questions stated in English which they had to translate into Query By Example. Briefly, the bottom line was that Query By Example was easy to learn and easy to use. However, there were still questions that people had difficulty with. In analyzing the data and doing some further experiments, the difficulties that people tended to have, stemmed not so much from Query By Example per se, but from what I much later came to call “labelism” — that is, confusing a label with the thing that label refers to.

Here’s a simple example of the type of confusion we saw. In Query By Example (and other query languages) there is usually an OR operator and an AND operator. (These operators can be important for doing advanced queries with search engines as well). If you are interested in getting a list of pets you might adopt and you’re willing to adopt dogs or cats, you might ask for “cat OR dog.” If you only want long-haired cats, you might ask for “cat” AND “long hair.”

English, however, can be tricky.

If you and I (as opposed to you and a query language) are having a conversation, you might say, “I hear there are many pets that need to be adopted.”

I say, “Yes, there are all kinds of pets. There are snakes, dogs, turtles, rabbits, cats…”

You say, “Let me stop you right there. I’m only interested in adopting cats and dogs. Those are the only animals I’d want to adopt.”

See what you said there? You exact words included: “…cats and dog.” If you put “Cats AND dogs” into a query against the data base of available pets, however, you will get the null set (that is, nothing) back. There are no animals who are both cats and dogs! (Though my part Main Coon cats do play fetch like dogs).

When people were presented with an English statement that included the English word “and” — regardless of the actual syntax and context, some of them had difficulty using the OR operator. If instead, the query in English had set up like this: “Oh, I don’t want reptiles. I’d be happy with adopting a cat or a dog, however” then, they’d have no problem translating it into the OR operator in the query language.

Lessons Learned: #6 Sometimes the difficulty people have in using a product, a service, or a prototype is not due to the interface details but with the structure of the task, their background, and their training.

By analogy, you will not allow me to beat Nadal or Djokovic at tennis by giving me a better tennis racquet! (Although if you gave one of them a toothpick for a tennis racquet, I might have a shot).

That sounds obvious and even absurd, but I promise you, some companies get so greedy that they want you to design a system that allows people who do not understand the task and have minimal background and training to nonetheless be able to perform that task.

One example you may have run into is having “help desk” personnel who have no understanding of a product go through a script to help you “solve your problem.” Sometimes, it works. But many times it doesn’t. When it does not work, you might not be able to “fix” the system by making the interface to the scripts easier to use for the help desk folks. The problem is much deeper (in some cases). Yes, a really bad interface can make it difficult even for a really knowledgeable and capable person to do the job. But often, even a really great interface cannot always substitute for actual expertise.

——————————————————-

Essays on America: Labelism

Career Advice from Polonius

20 Saturday Mar 2021

Posted by petersironwood in Uncategorized

≈ 1 Comment

Tags

career, HCI, human factors, usability, UX

Career Advice from Polonius: To Thine Own Self be True

“To thine own self be true.” This advice comes from Polonius who is giving advice to his son in Act I, scene 3 of Shakespeare’s Hamlet.

Polonius says: “This above all: to thine own self be true. And it must follow, as the night the day, Thou canst not then be false to any man.”

Let’s focus on the first part.

One of the dreams of education is to customize teaching to the specific learning style(s) of individual students. This was a hot topic when I was in graduate school.

Around 50 years ago.

Some day, your grandchildren or your great-grandchildren may be the beneficiaries of learning experiences that are individualized to their specific styles. I wouldn’t hold my breath, but it could happen. It isn’t only a question of research on what various styles are and how to present material that resonates with these various styles. There is also the question of priorities and dollars and personnel.

But meanwhile, here’s the good news. You don’t have to wait for another 50 years of research and a reshuffling of priorities so folx spend more money on education and less on, let’s say, cosmetics and professional sports. As I say, don’t hold your breath.

But let’s get back to the good news. The good news is that you can discover for yourself how to maximize your own learning as well as what your particular talents are.

One cautionary note: Don’t be a jerk about it. If you’re in a group dealing with grief, don’t say, “Well, I learn best if a subject is reduced to a few hundred polynomial formulae. So, let’s start right there. Let’s reduce grief to three dimensions. Later, of course, we can do a proper multidimensional scaling exercise to determine the optimal number of dimensions.”

Photo by Andrea Piacquadio on Pexels.com

No. Don’t say that. Of course, you’re free to suggest that approach, but chances are, in this situation, and in most realistic group situations, you will be treated to information in the same manner as many others who have different styles from yours.

However, in many situations, you are, far and away the main important stakeholder. You can use your knowledge of how things work for you in order to strategize and plan how you will learn about things. You can organize and arrange your work so you’ll be more productive.

Here’s a trivial example. I have learned that my eyes have a wisdom of their own. If, for instance, I’m going out for a walk around the garden to take some pictures of the sunset on the flowers, I grab my stuff and find myself turning and staring at the hat-rack on the way out the door. When I was younger, I would ignore this. But what I have learned is that my eyes are really good at knowing what to look at. So, even if I’m in a hurry, I take a moment to reflect on why my eyes are looking there. And, then, it comes to me. I’ll do better if I wear a brimmed hat to keep the sun out of my eyes while I look at my iPhone.

By paying attention to this little quirk, I’ve saved myself a lot of grief over the years; e.g., not left the house without my wallet, etc.

Here’s another example. I’m very good at seeing “patterns” emerge from a small number of examples or when there is considerable noise involved. This serves me well as my hearing diminishes because I can use top-down processing. Generally, but not always, I understand what people are saying. If I try to listen to a foreign language tape that is only isolated audio words, I have no hope of knowing what they are saying. “Key” “Tee”, “Pea”, sound exactly the same.

Seeing patterns easily is generally a nice capacity. However, I’m horrible at finding my own typos immediately after I write something. I actually “see” what I meant to type. A week later, I’m pretty good at catching the errors. If I had more patience, I would wait a week to proofread for every blog post, but being patient isn’t a strength either. I do go back over old posts occasionally and fix the typos (which I never saw at the time).

When I go to the movies — remember when we used to go to movies? — anyway, if I went to a comedy, I was very likely to laugh too soon. I “hear” the punchline two lines earlier than it actually occurs. There’s no benefit to my laughing early! But that’s when the punchline hits me. I do keep it soft so as not to disturb the others in the audience. On the other hand, I’m pretty good at “discovering” the playing patterns of my tennis opponents and anticipating what they are going to do. Naturally, I don’t always guess right, but I do way better than chance.

I bring up these examples to illustrate a generality; that most of these individual differences have an upside and a downside. Mainly, learning about my own styles and capacities is something I learned well after leaving high school. That makes sense. In school — or at least, the schools I went to — everybody got the same instruction in the same way almost all the time. But as an adult, you often have a lot of control over your own timing, flow of information, etc. I think it’s worth your while to look back at your experience and discover what you have difficulty with, what you’re OK at and what you are exceptionally good at. When you have a choice, use the approach you’re really good at.

—————————————-

More background on “knowing yourself”

https://en.wikipedia.org/wiki/Know_thyself

History of Know Thyself

Problem Formulation: Who Knows What?

18 Monday Jan 2021

Posted by petersironwood in Uncategorized

≈ 7 Comments

Tags

browser, HCI, problem formulation, problem framing, problem solving, query, search, thinking, usability, UX

This post focuses on the importance of discovering who knows what. It’s easy to think (without thinking!) that everyone knows what you know.

At IBM Research, around the turn of the century, I was asked to look at improving customer satisfaction about the search function on IBM’s website. Rather than using someone else’s search engine, IBM used one developed at IBM’s Haifa Research lab. It was a very good search engine. Yet, customers were not happy. By way of background, it’s worth noting that compared with many companies who have websites, IBM’s website was meant for a wide variety of users and contained many kinds of information. It was meant to support people buying their first Personal Computer and IT experts at large banks. It had information about a wide variety of hardware, software, and services. The site was designed to serve as an attractor for investors, business partners, and potential employees. In other words, the site was vast and diverse. This made having a good search function particularly important.

A little study of the existing data which had been collected showed that the mean number of search terms entered by customers was only 1.2. What?? How can that be? Here’s a website with thousands of products and services and designed for use by a huge diversity of users and they were only entering a mean of 1.2 search terms? What were they thinking?!

Of course, there were a handful of situations when one search term might work; e.g., if you wanted to find out everything about a specific product that had a unique one-word name (which was rare) or acronym. For most situations though, a more “reasonable” search might be something like: “Open positions IBM Research Austin” or “PC external hard drives” or “LOTUS NOTES training.”

We invited a sample of users of IBM products & services to come into the lab and do some tasks that we designed to illuminate this issue. In the task, they would need to find specified information on the IBM website while I observed them. One issue became immediately apparent. The search bar on the landing page was far too small. In actuality, users could enter as many search terms as they liked. Their terms would keep scrolling and scrolling until they hit “ENTER.” The developers knew this, but most of our users did not. They assumed they had to “fit” their query into the very small footprint that presented itself visually. Recommendation one was simply to make that space much larger. Once the search bar was expanded to about three times its original size, the number of search terms increased dramatically, as did user satisfaction.

In this case, the users framed their search problem in terms of: “How can I make the best query that fits into this tiny box.” (I’m not suggesting they said this to themselves consciously, but the visual affordance led them to that constraint). The developers thought the users would frame their search problem in terms of: “What’s the best sequence of terms I can put into this virtually infinite window to get the search results I want.” After all, the developers knew that any number of terms could be entered.

Although increasing the size of the search bar made a big difference, the supposedly good search engine still returned many amazingly bad results. Why? The people at the Haifa lab who had developed the search engine were world class. At some point, I looked at the HTML of some of the web pages. Many web pages had masses of irrelevant metadata! I found some of the people who developed these web pages and discussed things with them. Can you guess what was going on?

Many of the developers of web pages were the same people who had been developing print media for those same products and services. They had no training and no idea about metadata. So, to put up the webpage about product XYZ, they would go to a nice-looking web page about something else, say, training opportunities for ABC. They would copy that entire page, including the metadata, and then set about changing the text about ABC to text about product XYZ. In many cases, they assumed that the strange stuff in angle brackets was some bizarre coding stuff that was necessary for the page to operate properly. They left it untouched. Furthermore, when they “tested” the pages they had created about XYZ, they looked okay. The information about XYZ was there. Problem solved.

Only of course, the problem wasn’t solved. The search engine considered the metadata that described the contents to be even more important than the contents themselves. So, the user would issue a query about XYZ and receive links about ABC because the XYZ page still had the “invisible” metadata about ABC. In this case, many of the website developers thought their problem was to put in good data when what they really needed to do was put in good data and relevant metadata.

A third issue also revealed itself from watching users. In attempting to do their tasks, many of them suggested that IBM should provide a way for more than one webpage to appear side by side on the screen so that they could, for instance, compare features and functions of two different product models rather than having to copy the information from the web page about a particular model and then compare their notes to the second page.

Good suggestion.

Of course, IBM & Microsoft had provided this function. All one had to do was “Right Click” in order to bring up a new window. Remember, these were not naive users. These were people who actually used IBM products. They “knew” how to use the PC and the main applications. Yet, they were still unfamiliar with the use of Right Click. Indeed, allowing on-screen comparisons is one of the handiest uses of Right-Click for many people.

This issue is indicative of a very pervasive problem. Ironically, it is an outgrowth of good usability! When I began working with computers, almost nothing was intuitive. No-one would even attempt to start programming in FORTRAN or SNOBOL, let alone Assembly Language or Machine Code without looking at the manual. But LOTUS NOTES? A browser? A modern text editor? You can use these without even looking at the manual. That’s a great thing. But —

…there’s a downside. The downside is that you may have developed procedures that work, but they may be extremely inefficient. You “muddle through” without ever realizing that there’s a much more efficient way to do things. Generally speaking, many users formulate their problem, say, in terms like: “How do I create and edit a document in this editor?” They do not formulate it in terms of: “How do I efficiently create and edit a document in this editor?” The developers know all the splendid features and functions they’ve put into the hardware and software, but the user doesn’t.

It’s also worth noting that results in HCI/UX are dependent on the context. I would tend to assume that in 2021 (when I first published this post), most PC users knew about right-clicking in a browser even though in 2000, none of the ones I studied seemed to realize it. But —

I could be wrong.

————————————

The Invisibility Cloak of Habit

Essays on America: Wednesday

Index to a catalog of “best practices” in teamwork & collaboration.

Author Page on Amazon

Getting Into (the “right”) Shape

07 Monday Sep 2020

Posted by petersironwood in Uncategorized

≈ 1 Comment

Tags

cats, consumer products, Design, form, function, HCI, human factors, kittens, usability, UX

A truism we have all heard is that “form should follow function.” I tend to agree with this as a good general principle, but only if the designer has given more than 30 milliseconds of thought about what the actual function is. Even better is to observe function being used in practice. Below, I give examples of how form may look like function but not actually follow (actual) function.

The first comes from the complex and technical domain of nail clippers. My nails are tough and I actually need to use toenail clippers to cut my fingernails. But the same principle applies to both fingernail clippers and toenail clippers. I see many many examples where the designer has attempted to curve the surface of the nail clipper to “match” the curve of nails. This is a brilliant idea, but only if every nail on every human being on the planet has the same curvature. A priori, I would tent to think this is not the case, but being empirically oriented I decided to test it out by actually looking at real nails. I looked at my thumbnail and the fingernail on my little finger. Sure enough, my hypothesis was borne out. They are NOT the same. What this means is that a nail clipper that is curved so that it fits my pinkie will wreak havoc when applied to my thumbnail. I am probably going out on a limb here, but I suspect that if one were to include fingernails from other people in this sample, one might find an even wider variation in curvature. What are people thinking when they make curved nail clippers? I can only speculate that they have never looked at the fingernails of more than one person and that, indeed, they never looked at more than one fingernail on that one person.

If only there were a solution. Sigh. Oh, wait! There is a solution. Make the cutting surface of the nail clippers flat. This enables the person to clip nails of any curvature. It does, of course, require multiple cuts. It has the added advantage, that if you so wish, you can sharpen your nails so they resemble cat claws.

Cats bring me to my second example. When we moved to California amid a large garden, we wanted to let our cats to spend most of their time outdoors, partly so litter box cleaning would be at a minimum. Unfortunately, we soon discovered that while the outdoors here offers many opportunities for cats to be hunters of lizards and mice, it also offers even more opportunities for them to be prey for bobcats, cougars, eagles, and especially coyotes.

Now, here is a beautifully shaped litter box (a gift). It even has a place for the cats to clean their paws before they track litter back into the living room. Nice. Unfortunately, this is a beautiful shape by someone who has never cleaned a litter box, at least not by litter box shovel. Perhaps they clean litter boxes with their bare hands? Anyway, this curved shape does not jibe well with the typical litter box shovel. Of course, the cats could choose to do their business along the gently curving side of the litter box. And, of course, they never do. They choose instead the places along the edge of the litter box where there is maximum curvature.

The idea that there is a place for the cats to clean their feet before venturing back out into the living room or pouncing up on the kitchen counter is a sweet idea. It is an idea that would never occur to the owner of an actual cat, however. Here are two cats we obtained from a shelter (Tally on the left, Molly on the right).

They are cute, but defective in that they do not speak English, nor so far as I have been able to discover, any other language. So, despite my explanations that they are supposed to wipe their feet on the way out of the litter box, they do not. Instead, they do their business on the foot-wiping section of the litter box. So, apart from the annoying high curvature, if you are unlucky enough to get a cat who either does not understand complex sentences or just doesn’t care, this is probably not the litter box for them either. It might work for cats who: 1) speak your language fluently and 2) are cooperative. (Recent estimates indicate the total number of such cats is zero).

The third example comes from health care and is a bit more abstract. On my insurance ID card is a field which is labeled: “Identification number.” In order to use this, I have to go to their website and “register.” In order to register, the website says I need to enter my “identification number.” But in actuality, that does not work! No. Instead, I am supposed to leave off the first three characters in what is labeled my identification number. The website doesn’t say this. But the help desk is quite familiar with the issue and will happily explain it to you after you listen to musak for three or four hours. This is not so much shape not matching real function, but label not matching function.

The fourth example comes from some of our “bookcases.” Why, I hear you ask, are there scare quotes around bookcases. I will tell you why. I put scare quotes because although the shelves are flat and just the right size for books, and although this piece of “furniture” is sold as a bookcase, in fact, it is a nick-knack shelf. My wife and I foolishly tried filling it with books and it collapsed. So, in this case, the label and the shape lead one to believe it serves a particular function but the underlying functionality is insufficient to fulfill that dream of ours (that the “bookshelf” would actually hold books).

Photo by Pixabay on Pexels.com

The fifth example comes from my experience with companies who want to simplify things for their customers. That sounds worthwhile. So, they launch major efforts to make their products “consistent.” But they soon learn that making products behave consistently years after they were independently developed is way too expensive. So instead, they focus on making them look the same and using consistent terms across products, while leaving the underlying functionality behave quite differently. To me, this is quite akin to the bookshelf case. Making things look the same while continuing to have them act differently is actually worse for the user than having things that act differently also look different!

The moral of the story? It’s fine to have form follow (and signal) function, but you need to understand how users actually behave. They won’t necessarily behave as you imagine they are supposed to any more than a cat will read your mind in order to please you. Of course, if you see yourself, not as a partner of your users, but rather out to deceive them into thinking they are buying and using something different from what they really are…

—————————————

Introduction to Pattern Language for Collaboration and Cooperation.

Index for Collaboration Patterns

Author Page on Amazon

Using Stories and Storytelling

What do you do when the client insists you solve the “wrong” problem?

Regression to the Mean

10 Monday Dec 2018

Posted by petersironwood in America, management, psychology, sports

≈ 2 Comments

Tags

Business, experiment, family, Feedback, HCI, learning, life, politics, science, sports, testing, usability, UX

Regression to the Mean

selective focus photography of yellow leaves

Photo by Haugenzhays Zhang on Pexels.com

While working full-time at IBM Research, I was also a Fellow at the Institute for Rational-Emotive Therapy in Manhattan. I wrote an article in 1978 for their Journal, Rational Living. The title was: “Why Do I Self-Down? Because I’m an Idiot?” Indeed, many people put themselves down and it is not helpful. I hypothesized several different causes for this kind of self-slamming behavior. Most of these causes you could probably figure out on your own. But one in particular is subtle and non-intuitive. It is based on a statistical phenomenon which few people know about despite the fact that it is extremely pervasive. This phenomenon is called “Regression to the Mean.”

I want to define this term by explaining some examples. Imagine that you have a new soft drink which contains a combination of herbs that will purportedly make you smarter; e.g., gingko and bacopa. (There is some evidence these may actually work but let’s assume that they don’t or that your tea has too little to be effective). Here’s what you do to “prove” that it works anyway. You give an IQ test to 10,000 people and choose the 50 who score the lowest on the test and have them drink your tea for the next six months. At the end of that time, you give those 50 people an IQ test again and — Voila! The average (or mean) of the IQ scores has almost certainly gone up. Yay! It works!

Or does it? One of your competitors is not too happy about your study. In fact, they aren’t even happy you put your tea on the market. They decide to prove that your tea is not only ineffective but that it makes people less smart. So what do they do? They give an IQ test to 10,000 people and they pick the 50 who score the highest. They have them drink your tea for six months and at the end of that time, they have them take another IQ test. In this case, the mean (average) score is lower than the first time! Ouch! They say your tea causes brain damage!

Photo by meo on Pexels.com

How can the same tea make people smarter and make them dumber? In this case, it does neither. What is going on? Here’s what is going on. When you measure something, there is always some error. Whether you are measuring your weight, your height, your blood pressure, or your IQ, the measurement is never exactly perfect. Your weight may vary slightly because of atmospheric pressure and more so because of water retention. If you take an IQ test, your score will partly reflect how well you do on such tests in general, but it will partly depend on luck. You may have felt particularly good that day, or a few of the questions might have been on topics you just heard about on TV the day before, or you may have made some lucky guesses. Or, you may have been unlucky on a particular day. You might have had a cold or misread one of the questions or forgotten your morning coffee. On any given day, some people will be a little lucky and some people will be a little unlucky. These things tend to balance out in a large group and if you tested all 10,000 people after six months, then assuming the tea has no real effect, no effect will be shown in the data.

Photo by Pixabay on Pexels.com

However, if you select the very best scores, you are partly picking smart people, of course, but you are also picking the people who were lucky that day. When you test just those people six months later, they will generally be just as smart but there is no reason to suppose they will be lucky again. Some will be lucky both times, most will not be particularly lucky or unlucky and a few will be unlucky. The average score will be lower. Conversely, if you choose the lowest scoring people, you will partly be choosing people who don’t do well on such tests in general. But you will also be choosing people who were tired, sick, guessed wrong or were otherwise unlucky that day. When you retest, those people will still tend to be people who do poorly on such tests, but they won’t necessarily all be unlucky again. Some will. Some won’t. On average, the scores will be higher than they were the first time.

The phenomenon of “Regression to the Mean” was first noted by Francis Galton in the 1880’s. Tversky and Kahneman, so far as I know, were the first to note that this phenomenon could easily cause managers, coaches, and parents to end up being unnecessarily negative. Here’s how it works. Let’s say you are learning to hit tennis serve. Although you will likely improve in general, over time, there will also be a lot of variation in your performance. Sometimes, everything will work well together and you’ll hit an excellent serve, one that is above your average level. At first, the coach’s natural inclination will be to praise this by saying, “Wow! Great serve!” or something like that. Unfortunately, your next serve, due to regression to the mean is very likely not to be quite as good as that one was. Your coach’s praising behavior was thereby punished. On the other hand, if you hit a particularly poor serve for your level, your coach might say, “Oh, come on. You can do better than that!” If they choose to say such things only on your very worst performances, then, due to regression to the mean, your next serve is likely to be somewhat better. In other words, their slamming you will be rewarded by your doing better the next time. The same general tendencies will apply to managers and parents as well.

Photo by Pixabay on Pexels.com

The same applies to you! Whatever you are doing, your performance will vary somewhat over time. If you begin by praising yourself internally whenever you hit a particularly great shot, your next shot will most likely be not so great. On the other hand, if you put yourself down when you find your performance particularly bad, “You idiot! How could you miss that!?” Your next shot will tend to be somewhat better. Over time, your positive self-talk will tend to be punished and your negative self-talk will tend to be rewarded.

It’s no wonder then that many managers, coaches, and parents end up saying very negative things about their charges. It’s also no wonder that many people say (or more likely think) many more negative things about themselves than they say positive things.

Is there anything to be done? First, simply be aware of this phenomenon. That is step one. If you are running a study, you need to be careful in selecting. The study about your tea could be fixed by re-testing the entire population; by selecting a random group of 50 rather than the best or worst; or by using a control group who did not drink tea but was retested anyway. When praising or punishing someone’s performance, do not bother with trying to reward or punish outcomes based on one trial. That’s actually a pretty poor way to coach yourself or others in any case. See The Winning Weekend Warrior for more on this. Also watch out for this when you read about various conclusions of other studies. Did the investigators select either the “best” or the “worst” for their study? If they did such a selection, did they talk about the bias this introduces? Did they have a control group?

Meanwhile, treat your mistakes as opportunities to learn, not as opportunities to put yourself down. There’s really no point in self-downing. But if you do find yourself self-downing, remember that it’s common; relax; smile at this human foible; then quit doing it. At least give yourself a break for the holidays.

Photo by freestocks.org on Pexels.com

————————————-

Tversky, A., & Kahneman, D. Judgment under uncertainty: Heuristics and biases. Science, 1974, 185, 1124-1131.

Author Page on Amazon

Study Slain by Swamp Monster!

19 Thursday Jul 2018

Posted by petersironwood in America, management, psychology, Uncategorized

≈ 8 Comments

Tags

Business, Design, experiment, HCI, human factors, innovation, politics, science, Study, usability, UX

Study Slain by Swamp Monster!

I’m trying a new format for blog posts.

For those of you in a hurry, to get to the “bottom line” of this post, you can skip the story and go right to the bold-faced “lesson” at the end. I’d really you rather read the whole thing of course, but I know some readers are harried and hurried. So, if that describes you right now, feel free.

——————————————————

In the early 1980’s, researchers at the IBM Watson Research Center invented a new kind of system. Originally, this was called the “Speech Filing System.” It was initially designed to allow so-called “office principals” (sales people, managers, executives, engineers, etc.) to dictate letters and memos which could then be typed up by the pool of typists. Instead of requiring each “office principal” to have (or borrow) a dedicated piece of dictation equipment, they could accomplish this dictation from any touch tone phone. While this offered some savings in cost and convenience in the office, it was even more wonderful on the road. People did not have to take their dictation equipment with them on their travels. They could use any touch-tone phone.

Photo by Pixabay on Pexels.com

The system was invented largely by tech-savvy psychologists (including Stephen Boies, John Gould, John Richards, & Jim Schoonard). When they observed people actually using the system, they discovered that the trial users more often used the ancillary messaging facility than they did the “real” dictation features. So, the system was redesigned and repurposed and then renamed, “The Audio Distribution System.” In some ways, using the “Audio Distribution System” was much like leaving a message on an answering machine. However, there were some crucial differences. Typically, a person calling someone and encountering, instead of a human being, a message asking them to leave another message was somewhat taken aback. Many messages on answering machines went something like this: “Hi. Stephen? Oh, you’re not there. OK. This is John. I was hoping … well, I thought you’d be in. Uh. Let’s see. You know what? Call me back. We need to talk.” And, when Stephen discovered that he had a message, he might listen to it and call back John. “Hi, John. Stephen here… I … oh. OK. A message. Sorry. You just called me. Well, um. I’m not sure what you wanted to talk about so. Call me back when you get a chance.”

Photo by Public Domain Pictures on Pexels.com

By contrast, when someone called the “Audio Distribution System” they knew ahead of time they’d be interacting with a machine. So, they could compose a reasonable message before calling the system. Hence, the messages tended to be more coherent and useful; e.g., “Hi, Stephen. This is John. If it’s okay with you, I’m taking off this Friday for a long weekend. If you have any issues with that, let me know.” See? Easy and efficient.

A second critical difference was that you could listen to your message and edit it. People didn’t do this so often as you might think, but it was comforting to know that you could in case you really messed up. (For instance, a person might say, “You are fired!” when all along they meant to say, “You are NOT fired.”).

Introducing any new system will have consequences, both intended and unintended. I wanted to see what some of these consequences might be. Corporations, IBM included, like it when they sell lots of product and make lots of money. A related question then was – what is the value of this product to the customer? Why should they want to buy it?

One hypothesis I wanted to test out was that such a system would increase people’s perceived Peace of Mind. After you leave a meaningful message for someone, you can “cross off” that little item off your mental (or written) “to do” list. By using the Audio Distribution System, I thought one of the user benefits would be increased “Peace of Mind” because they would be able to leave a message any time and any place they had access to a touch tone phone. They could save their working memory capacity for “higher level” activities such as design, problem solving, and decision making. We were going to roll out a beta test of the Audio Distribution System at the divisional headquarters for the IBM Office Products Division (OPD), in Franklin Lakes, New Jersey. Not coincidentally, OPD would be the division selling the Audio Distribution System (just as they were now selling dictation equipment). Before the trial commenced, I developed a questionnaire designed to get at how much people felt harried, too busy, coping, etc. The hope was that I could compare the “Peace of Mind” scores of people who did and did not get the Audio Distribution System and perhaps show that those with the system felt more at peace than those without. I could also compare “before and after” for those internal beta customers who had the system.

Photo by Suraphat Nuea-on on Pexels.com

Before I was to roll-out and administer the “Peace of Mind” questionnaire to a sample of people at the OPD Franklin Lakes location, guess what happen just two days before the beta roll-out? OPD was re-organized out of existence! The people who worked there would now be looking for another job elsewhere in IBM (or, failing that, just elsewhere period). The beta trial was cancelled. In any case, even if it hadn’t been cancelled, the impact of the re-organization would have completely swamped (in my estimation) the impact of this new tool. Moreover, it struck me as insensitive and slightly even unethical to ask people to fill out a questionnaire about how hassled they were feeling just days after finding out their entire division had been blown up. How would you react if some psychologist from the Research Center showed up asking you to fill out a questionnaire two days after finding out you no longer had a job?

photography of green and red fire works display

Photo by Anna-Louise on Pexels.com

————————————————-

What is the lesson learned here? You have to understand what is going on in the lives of your users over and above the functions and features directly related to your product or service. Of course, there is always a fairly good chance that some of your users will have overwhelming things going on in their lives that will impact their reactions to your product. Generally you won’t know about divorces, deaths in the family, toothaches, etc. But if something is impacting all your users, you’d best be aware of it and act accordingly.

————————————-

Speech Filing System

Audio Distribution System – NY Times

Longer explanation of Audio Distribution System

Video of Audio Distribution System’s cousin: “The Olympic Message System”

———————————————————————

Author Page on Amazon

Support Both Flow & Breakdown

21 Monday May 2018

Posted by petersironwood in America, management, psychology, Uncategorized

≈ 3 Comments

Tags

collaboration, contextual design, Design, environment, error messages, HCI, human factors, learning, pattern language, pliant systems, politics, usability

Support Both Flow & Breakdown

Prolog/Acknowledgement/History:

Only a few days after moving into our San Diego home (with a beautiful drip-irrigated garden), I glanced outside to see a geyser sprouting about ten feet into the air. San Diego can only survive long term if people conserve water! Yet, here we were — wasting water. I rushed outside to turn off the sprinkler system. As I ran to the controller, I noted in passing that the nearby yard lay soaked with pools of water. I turned off the sprinklers — except for the geyser which continued its impersonation of “Old Faithful.” I tried turning the valve on that particular sprinkler and did manage in that way to completely soak myself but the water waste continued unabated. We called the gardener who knew and explained the location of the shutoff valve for the entire house and garden. Later, he came and replaced the valve with a newer type. The old type, which had failed, failed by being stuck in the fully ON position!

Often in the course of my life, I have been frustrated by interacting with systems — whether human or computer — that were clearly designed with a different set of circumstances than the one I found myself in at the time. In a sense, the Pattern here is a specific instance of a broader design Pattern: Design for Broad Range of Contexts. The specific example that I want to focus on in this Pattern is that design should support the “normal” flow of things when they are working well, but also be designed to support likely modes of breakdown.

During the late 1970’s, I worked with Ashok Malhotra and John Carroll at IBM Research on a project we called “The Psychology of Design.” We used a variety of methods, but one was observing and talking with a variety of designers in various domains. One of the things we discovered about good designers was a common process that at first seemed puzzling. Roughly speaking, designers would abstract from a concrete situation, a set of requirements. They would then create a design that logically met all the requirements. Since we were only studying design and not the entire development process (which might include design, implementation, debugging, etc.) it might seem that the design process would end at that point. After all, the designer had just come up with a design that fulfilled the requirements.

What good designers actually did however, at least on many occasions, was to take their abstract design and imagine it operating back in the original concrete situation. When they imagined their design working in this concrete reality they often “discovered” additional requirements or interactions among design elements or requirements that were overlooked in the initial design. While unanticipated effects can occur in purely physical systems, (e.g., bridges flying apart from the bridge surface acting like a wing; O-rings cracking at sufficiently cold temperatures), it seems that human social systems are particularly prone to disastrous designs that “fulfill” the requirements as given.

woman in white wedding gown near orange car

Photo by Slobodan Jošić on Pexels.com

The Pattern here specifically focuses on one very common oversight. Systems are often designed under the assumption that everything in the environment of the system is working as it “should” or as intended. This particular type of breakdown was featured in an important theoretical paper authored by Harris and Henderson and presented at CHI 99. That paper claimed systems should be “pliant” rather than rigid. A common example most readers have had with a non-pliant system is to call an organization and be put into an automated call-answering system that does not have the appropriate category anywhere for the current situation but still does not have a way to get through to a human operator.

A telling example from their CHI Proceedings article is that of a paper-based form that was replaced with a computerized system with fixed fields. So, for example, there were only so many characters for various address fields. When someone needed to make an exception to the address syntax with a paper form, it was easy. They could write: “When it’s time to ship the package, please call this number to find out which port the Captain will be in next and ship it there: 606-555-1212.” In the computerized form, this was impossible. In fact, there were so many such glitches that the workers who actually needed to get their work done used the “required” “productivity-enhancing” computer system and also duplicated everything in the old paper system so that they could actually accomplish their tasks.

As part of the effort (described in the last blog post) to get IBM to pay more attention to the usability of its products, we pushed to make sure every development lab had a usability lab that was adequately equipped and staffed. This was certainly a vital component. However, usability in the lab did not necessarily ensure usability in the field. There are many reasons for that and I collaborated with Wendy Kellogg in the late 1980’s to catalog some of those. This effort was partly inspired by a conversation with John Whiteside, who headed the usability lab for Digital Equipment Corporation. They brought people who used a word processor into their usability lab and made numerous improvements in the interface. One day he took some of the usability group out to observe people using the text editor in situ in a manuscript center. They discovered that the typists spent 7 hours every day typing and 1 hour every day counting up, by hand, the number of lines that they had typed that day (which determined their pay). Of course, it was now immediately obvious how to improve productivity by 14%. The work of this group seems to have been inspirational for Beyer & Holtzblatt’s Contextual Design as well as the Carroll & Kellogg (1989) paper on “Artifact as Theory Nexus.”

Photo by Pixabay on Pexels.com

Author, reviewer and revision dates:

Created by John C. Thomas in May, 2018

fullsizeoutput_17

Related Patterns:

Reality Check, Who Speaks for Wolf?

Abstract:

When designing a new system, it is easy to imagine a context in which all the existing systems that might interact with the new system will operate “normally” or “properly.” In order to avoid catastrophe, it is important to understand what reasonably likely failure modes might be and to design for those as well.

Context:

For people to design systems, it is necessary to make some assumptions that separate the context of the design from what is being designed. There is a delicate balance. If you define the problem too broadly, you run the risk of addressing a problem that is too intractable, intellectually, logistically or financially. On the other hand, if you define the problem too narrowly, you run the risk of solving a problem that is too special, temporary, or fragile to do anyone much good.

In the honest pursuit of trying to separate out the problem from the context, it happens that one particular form of simplification is particularly popular. People assume that all the systems that will touch the one they are designing will not fail. That often includes human beings who will interact with the system. Such a design process may also presume that electrical power will never be interrupted or that internet access will be continuous.

Systems so designed may have a secondary and more insidious effect. By virtue of having been designed with no consideration to breakdowns, the system will tend to subtly influence the people and organizations that it touches not to prepare for such breakdowns either.

Problem:

When the systems that touch a given system do fail, which can always happen, if no consideration has been given to failure modes, the impact can be disastrous. Most typically, when the system has not been designed to deal with breakdowns, the personnel selection, training, and documentation also fail to deal with breakdowns. As a result, not only are the mechanisms of the systems unsuited to breakdowns; the human organization surrounding the breakdown is also unprepared. Not only is there a possibility of immediate catastrophe; the organization is unprepared to learn. As a result, mutual trust within and of the organizations around the system are also severely damaged.

architecture building fire exit ladders ladder

Photo by Photo Collections on Pexels.com

Forces:

Design is a difficult and complex activity and the more contingencies and factors that are taken into account, the more difficult and complex the design activity becomes.

Not every single possibility can be designed for.

People working on a design have a natural tendency to “look on the bright side” and think about the upside benefits of the system.

People who try to “sell” a new system stress its benefits and tend to avoid talking about its possible failures.

It is uncomfortable to think about possible breakdowns.

When anticipated breakdowns occur, the people in relevant organizations tend to think about how to fix the situation and reduce the probability or impact of breakdowns for the future.

When unanticipated breakdowns occur, the people in relevant organizations tend to try to find the individual or individuals responsible and blame them. This action leaves the probability and impact of future breakdowns unimproved.

When people within an organization are blamed for unanticipated system failure, it decreases trust of the entire organization as well as mutual trust within the organization.

* Even when consideration of support for breakdown modes is planned for, it is often planned for late in an ambitious schedule. The slightest slippage will often result in breakdowns being ignored.

Solution:

When designing a system, make sure the design process deals adequately with breakdown conditions as well as the “normal” flows of events. The organizations and systems that depend on a system also need to be designed to deal with breakdowns. For example, people should be trained to recognize and deal with breakdowns. Organizations should have a process in place (such as the After Action Review) to learn from breakdowns. Having a highly diverse design team may well improve the chances of designing for likely breakdowns.

Resulting Context:

Generally speaking, a system designed with attention to supporting both the “normal” flow of events and likely breakdown modes will result in a more robust and resilient system. Because the system design takes these possibilities into account, it also makes it likely that documentation and training will also help people prepare for breakdowns. Furthermore, if breakdowns are anticipated, it also makes it easier for the organization to learn about how to help prevent breakdowns and to learn, over time, to improve responses to breakdowns. There is a further benefit; viz., that mutual trust and cooperation will be less damaged in a breakdown. The premise that breakdowns will happen, puts everyone more in the frame of mind to learn and improve rather than simply blame and point fingers.

fullsizeoutput_12e0

Examples:

1. Social Networking sites were originally designed to support friends sharing news, information, pictures, and so on. “Flow” is when this is what is actually going on. Unfortunately, as we now know, social media sites can also not work as intended, not because there are “errors” in the code or UX of the social media systems but because the social and political systems that form the context for these systems have broken down. The intentional misappropriation of an application or system is just one of many types of breakdowns that can occur.

2. When I ran the AI lab at NYNEX in the 1990’s, one of the manufacturers of telephone equipment developed a system for telephone operators that was based on much more modern displays and keyboards. In order to optimize performance of the system, the manufacturer brought in representative users; in this case, telephone operators. They redesigned the workflow to reduce the number of keystrokes required to perform various common tasks. At that time, operators were measured in terms of their “Average Work Time” to handle calls.

In this particular case, the manufacturer had separated the domain into what they were designing for (namely, the human-machine interface between the telephone operator and their terminal) from the context (which included what the customer did). While this seemed seemed like a reasonable approach, it turned out when the HCI group at NYNEX studied the problem with the help of Bonnie John, the customer’s behavior was actually a primary determiner of the overall efficiency of the call. While it was true that the new process required fewer keystrokes on the part of the telephone operator, these “saved” keystrokes occurred when the customer, not the telephone operator, was on the critical path. In other words, the operator had to wait for the customer any way, so one or two fewer keystrokes did not impact the overall average work time. However, the suggested workflow involved an extra keystroke that occurred when the operator’s behavior was on the critical path. As it turned out, the “system” that needed to be redesigned was not actually the machine-user system but the machine-user-customer system. In fact, the biggest improvement in average work time came from changing the operator’s greeting from “New York Telephone. How can I help you?” to “What City Please?” The latter greeting tended to produce much more focused conversation on the part of the customer.

Just to be clear, this is an example of the broader point that some of the most crucial design decisions are not about your solution to the problem you are trying to solve but your decision about what the problem is versus what part of the situation you decide is off-limits; something to ignore rather than plan for. A very common oversight is to ignore breakdowns, but it’s not the only one.

black rotary telephone beside beige manekin

Photo by Reynaldo Brigantty on Pexels.com

3. In a retrospective analysis of the Three-Mile Island Nuclear Meltdown, many issues in bad human factors came to light. Many of them had to do with an insufficient preparation for dealing with breakdowns. I recall three instances. First, the proper functioning of many components was shown by a red indicator light being on. When one of the components failed, it was indicated by one of a whole bank of indicator lights not being on. This is not the most salient of signals! To me, it clearly indicates a design mentality steering away from thinking seriously about failure modes. This is not surprising because of the fear and controversy surrounding nuclear power. Those who operate and run such plants do not want the public, at least, to think about failure modes.

Second, there was some conceptual training for the operators about how the overall system worked. But that training was not sufficient for real time problem solving about what to do. In addition, there were manuals describing what to do. But the manuals were also not sufficiently detailed to describe precisely what to do.

Third, at one critical juncture, one of the plant operators closed a valve and “knew” that he had closed it because of the indicator light next to the valve closure switch. He then based further actions on the knowledge that the valve had been closed. Guess what? The indicator light showing “value closure” was not based on feedback from a sensor at the site of the valve. No. The indicator light next to the switch was lit by a collateral current from the switch itself. All it really showed was that the operator had changed the switch position! Under “normal” circumstances, there is a perfect correlation between the position of the switch and the position of the valve. However, under failure mode, this was no longer true.

Photo by Pixabay on Pexels.com

4. The US Constitution is a flexible document that takes into account a variety of failure modes. It specifies what to do, e.g., if the President dies in office and has been amended to specify what to do if the President is incapacitated. (This contingency was not really specified in the original document). The Constitution presumes a balance of power and specifies that a President may be impeached by Congress for treasonous activity. It seems the US Constitution, at least as amended, has anticipated various breakdowns and what to do about them.

There is one kind of breakdown, however, that the U.S. Constitution does not seem to have anticipated. What if society becomes so divided, and the majority of members in Congress so beholden to special interests, that they refuse to impeach a clearly treasonous President or a President clearly incapacitated or even under the obvious influence of one or more foreign powers? Unethical behavior on the part of individuals in power is a breakdown mode clearly anticipated in the Constitution. But it was not anticipated that a large number of individuals would simultaneously be unethical enough to put party over the general welfare of the nation. Whether this is a recoverable oversight remains to be seen. If democracy survives the current crisis, the Constitution might be further amended to deal with this new breakdown mode.

5. In IT systems, the error messages that are shown to end users are most often messages that were originally designed to help developers debug the system. Despite the development of guidelines about error messages that were developed over a half century ago, these guidelines are typically not followed. From the user’s perspective, it appears as though the developers know that something “nasty” has just happened and they want to run away from it as quickly as possible before anyone can get blamed. They remind me of a puppy who just chewed up their master’s slippers and knows damned well they are in trouble. Instead of “owning up” to their misbehavior, they hide under the couch.

Despite the many decades of pointing out how useless it is to get an error message such as “Tweet not sent” or “Invalid Syntax” or “IOPS44” such messages still abound in today’s applications. Fifty years ago, when most computers had extremely limited storage, there may have been an excuse to print out succinct error messages that could be looked up in a paper manual. But today? Error messages should minimally make it clear that there is an error and how to recover from it. In most cases, something should be said as well as to why the error state occurred. For instance, instead of “Tweet not sent” a message might indicate, “Tweet not sent because an included image is no longer linkable; retry with new image or link” or “Tweet not sent because it contains a potentially dangerous link; change to allow preview” or “Tweet not sent because the system timed out; try again. If the problem persists, see FAQs on tweet time-out failures.” I haven’t tested these so I am not claiming they are the “right” messages, but they have some information.

Today’s approach to error messages also has an unintended side-effect. Most computer system providers now presume that most errors will be debugged and explained on the web by someone else. This saves money for the vendor, of course. It also gives a huge advantage to very large companies. You are likely to find what an error message means and how to fix the underlying issue on the web, but only if it is a system that already has a huge number of users. Leaving error message clarification to the general public advantages the very companies who have the resources to provide good error messages themselves and keeps entrenched vendors entrenched.

Photo by Pixabay on Pexels.com

References:

Alexander, C., Ishikawa, S., Silverstein, M., Jacobsen, M., Fiksdahl-King, I. and Angel, S. (1977), A Pattern Language: Towns, Buildings, Construction. New York: Oxford University Press.

Beyer, Hugh and Holtzblatt, Karen (1998): Contextual design: defining customer-centered systems. San Francisco: Elsevier.

Carroll, J., Thomas, J.C. and Malhotra, A. (1980). Presentation and representation in design problem solving. British Journal of Psychology/,71 (1), pp. 143-155.

Carroll, J., Thomas, J.C. and Malhotra, A. (1979). A clinical-experimental analysis of design problem solving. Design Studies, 1 (2), pp. 84-92.

Carroll, J. and Kellogg, W. (1989), Artifact as Theory-Nexus: Hermeneutics Meets System Design. Proceedings of the ACM Conference on Human Factors in Computing Systems. New York: ACM, 1989.

Casey, S.M. (1998), Set Phasers on Stun: And Other True Tales of Design, Technology, and Human Error. Santa Barbara, CA: Aegean Publishing.

Gray, W. D., John, B. E., & Atwood, M. E. (1993). Project Ernestine: Validating GOMS for predicting and explaining real-world task performance. Human Computer Interaction, 8(3), 237-309.

Harris, J. & Henderson, A. (1999), A Better Mythology for System Design. Proceedings of ACM’s Conference on Human Factors in Computing Systems. New York: ACM.

Malhotra, A., Thomas, J.C. and Miller, L. (1980). Cognitive processes in design. International Journal of Man-Machine Studies, 12, pp. 119-140.

Thomas, J. (2016). Turing’s Nightmares: Scenarios and Speculations about “The Singularity.” CreateSpace/Amazon.

Thomas, J.C. (1978). A design-interpretation analysis of natural English. International Journal of Man-Machine Studies, 10, pp. 651-668.

Thomas, J.C. and Carroll, J. (1978). The psychological study of design. Design Studies, 1 (1), pp. 5-11.

Thomas, J.C. and Kellogg, W.A. (1989). Minimizing ecological gaps in interface design, IEEE Software, January 1989.

Thomas, J. (2015). Chaos, Culture, Conflict and Creativity: Toward a Maturity Model for HCI4D. Invited keynote @ASEAN Symposium, Seoul, South Korea, April 19, 2015.

Author Page on Amazon

Find and Cultivate Allies

14 Monday May 2018

Posted by petersironwood in America, management, psychology, Uncategorized

≈ 4 Comments

Tags

allies, Business, collaboration, cooperation, HCI, IBM, organizational change, pattern language, politics, teamwork, usability

Find and Cultivate Allies

Prolog/Acknowledgement:

The idea for this Pattern comes from personal experience although I am sure there must be many other writers who make a similar point.

Author, reviewer and revision dates:

Created by John C. Thomas in May, 2018.

Abstract:

Human beings are highly social beings by nature. We work more effectively in groups (for many tasks) and it’s also more pleasurable. In a group of any size and complexity, people will have a large variety of goals and values. To achieve a goal, including but not limited to change within the group itself, it is useful to make common cause with others within the larger group. Whenever it becomes useful to promote social change of any kind, it is important to seek out and then cultivate allies. You will achieve greater success, enjoy the process, and learn much.

Context:

Complex problems and large problems can often only be solved by groups. Within a large group, there will be many sub-groups and individuals whose motivations, expertise, and values are partially different from those in other sub-groups or from those of other individuals. In order to achieve any kind of goal including but not limited to changes within the group itself, a great deal of knowledge must be brought to bear and a large number of actions will be required. Generally, an individual or a small group will not have the knowledge, power, or resources to take all of these actions.

The variety of goals, values, experiences, and scope of power of various individuals and subgroups within a larger group can be viewed as a resource. The interactions among such individuals be a source of creativity. In addition, in order to accomplish some goal, you may seek and find among these individuals and groups those whose goals are compatible with yours and whose power and resources allows them to do things you cannot do yourself.

Individuals are subject to a variety of perceptual and cognitive illusions and these may be exaggerated by being in a large group. Changing a group, team, organization, corporation, NGO may be even more difficult than changing an individual even if the change would benefit the group, team, organization, corporation or NGO. Within any organization, there come to be entrenched interests that are orthogonal to, or even antithetical to, the espoused purposes of the group.

iPhoneDownloadJan152013 527

Problem:

Over time, organizations eventually begin to behave in ways that are ineffective, inefficient, or even antithetical to their purpose. Whatever the cause, an individual who recognizes these infelicities in the organization will typically not, by acting alone, have the power to change them. Force of habit, custom, the culture, and the entrenched power of others will tend to make change by an individual extremely difficult or impossible despite their pointing out that the current way of doing things is counter-productive.

Forces:

People who wield local power in an organization are often afraid that any change will weaken their power.

Changing one part of the organization generally means that other parts must also change, at least slightly.

What works best for an organization must necessarily change over time because of changes in personnel, society, technology, competition, the environment, and so on.

Organizations typically codify the way they currently work by documenting procedures, providing training, incorporating current processes into software systems, floor layouts, and so on.

Each person in an organization is typically rewarded according to the performance of a small area of the organization that centers on or near them.

* People within an organization of any size will exhibit large variations in knowledge, skill, values, goals, and the resources available to them.

* In many organizations, a valid reason for continuing to do X is simply to say, “That’s the way we’ve always done it.”

* It is not considered a valid reason for change from doing X to doing Y to simply say, “We’ve never done it this way before.”

* Organizations are therefore prone to continuing along a path long after it is a fruitful, ethical, or lawful path.

Solution:

If a person wishes to change how a large organization does things, they need to find and cultivate potential allies within the organization. Allies may be people who can be convinced that the change is best for something that is best for that individual, their department, the organization as a whole, for society or for life on earth. These allies will have crucial information, power, friends, or resources to help make the change possible.

Example:

For two years, in the early 1980’s, I worked in the IBM Office of the Chief Scientist. My main mission was to get IBM as a whole to pay more attention to the usability of its products. No-one worked for me. I had no budget. I did, however, have the backing of the Chief Scientist, Lewis Branscomb. Among his powers, at the time, was the ability to “Non-Concur” with the proposed plans of other parts of IBM. This meant that if other IBM divisions did not have usability labs or adequate staff, the Chief Scientist could block the approval of those plans. Lewis himself was a great ally because he had a lot of personal credibility due to his brilliance. Having the power to block the plans of other divisions was also critical.

IBM at this time already had some Human Factors Labs who had done excellent work for years. However, there were large areas such as software that were mainly untested. In addition, most of IBM’s users were technical people and many of the usability tests had been done on other technical people. This had been appropriate but with the extension of computing into other areas of life, many of IBM’s “end users” were now people with little technical computer background. This included administrative assistants and clerks; even chemists, physicists, MD’s, lawyers and other people with advanced education found IBM products hard to use. None of these fairly new groups of users had typically used computers much or had been taught their use in their schooling.

I needed to find allies because the changes that were necessary to IBM were widespread. One important ally was already provided: Tom Wheeler had a similar position to mine within another corporate staff organization called “Engineering, Products, and Technology.” Tom could also get his boss to non-concur with the plans of divisions who were unwilling to “get on board” with the changes. But I needed more allies.

One obvious source of allies were the existing Human Factors Groups. Where they existed, they were typically staffed and managed by excellent people; however, they were often understaffed and often brought in near the end of the development cycle. In many cases, only their advice on “surface features” or documentation could be incorporated into the product. This was frustrating to them. They knew they could be more effective if they were brought in earlier. Often, this did happen, but typically because they had developed personal reputations and friendships (allies) within their organization. It was not mandated by the development process.

Who else would benefit from more usable IBM products? There’s a long list! A lot of “power” within IBM came from Sales and Marketing. The founder, Thomas J. Watson was himself primarily a sales and marketer. Most of the CEO’s had been from this function of the organization. Many in Sales and Marketing were beginning to see for themselves that IBM products were frustrating customers. Finding people within such organizations who were willing to stand up and “be counted” was critical. It was especially useful to find some allies in Europe who were on board with suggested changes. In many countries in Europe, there were various social and legal constraints that gave even more weight to having products that did not cause mental stress, repetitive motion injuries, eyestrain, hearing loss and so on.

In many parts of IBM, there were also “Product Assurance” organizations that required products to be tested before final release. In this case, two simple but crucial and fundamental changes needed to be made. Again, people who worked in Product Assurance wanted these changes to be made. First, we needed to convince development to work with Product Assurance earlier rather than later so that any problems would not be the cause of product announcement slippages (or ignored). Second, we needed to convince Product Assurance to test the procedures and documentation with people outside the development teams. Current practice was often for the Product Assurance people to watch people on the development team “follow” the documented process to ensure that it actually worked. The problem with this process is that language is ambiguous. The people on the development team already knew how to make the product work, so they would interpret every ambiguity in instructions in the “proper” way. IBM customers and users, however, would have no way of knowing how to resolve these ambiguities. Instead of making sure that the documentation was consistent with a successful set-up, the process was changed to see whether documentation actually resulted in a successful set-up when attempted by someone technically appropriate but outside the development team.

fullsizeoutput_1164

People within IBM product divisions did care about budgets. Adding human factors professionals to existing labs or, in some cases, actually setting up new labs, would obviously cost money. We needed to show that they would save money, net. Some of the human factors labs had collected convincing data indicating that many service calls done at IBM’s expense were not due to anything actually being wrong with the product but instead were because the usability of the product was so bad that customers assumed it must not be working correctly. In most cases, fixing the usability of products would save far more money than the additional cost of improving the products.

In some cases, developing allies was a fairly simple business. For example, IBM had a process for awarding faculty grants for academic research relevant to its technologies and products. These were awarded in various categories. Adding a category to deal with human-computer interaction required a single conversation with the person in charge of that program. Similarly, IBM awarded fellowships to promising graduate students in various categories of research. Again, adding the category of human-computer interaction resulted from a single conversation. It should be noted that the ease of doing that resulted much more from the fact that it was known throughout the company that usability was now deemed important and the fact that I worked for the well-respected Chief Scientist than from any particular cleverness on my part.

In at least one case, an ally “fell in my lap.” Part of how I operated was to visit IBM locations around the world and give a talk about the importance of usability for IBM’s success. Generally, these talks were well-received although that did not guarantee any success in getting people to change their behavior. When I gave the talk to the part of IBM that made displays, however, I got a completely hostile reaction. It was clear that the head of the division had somehow made up his mind before I started that it was complete nonsense. I had no success whatever. Only a few months later, the head of this division got an IBM display of his own. He couldn’t get it to work! He did a complete 180 and became an important supporter, through no fault of my own. (Of course, there may have been additional arm-twisting beyond my ken).

iPhoneDownloadJan152013 499

There were also two important, instructive, and inter-related failures in lining up my allies. First, it was very difficult to line up development managers. An IBM developer’s career depended on getting their product “out the door.” Not every product development effort that began resulted in a product being shipped. Once the product was shipped, the development manager was promoted and often went to another division. So, from the development manager’s perspective, the important thing was to get their product shipped. If it “bombed” after shipment, it wasn’t their problem. In order for the product to be shipped, it had to be forecast to make significant net revenue for IBM. No big surprise there! However, these predictions did not take into account actual sales, or the actual cost of sales, or the actual service costs, or even the actual production costs. The only thing that was really known were the development costs. So, for every additional dollar the development manager spent during development, there was one dollar added to the development costs, but also an additional dollar added to predicted service costs and predicted manufacturing costs. Moreover, there were an additional five dollars added to the predicted sales and marketing costs. If they spent an extra dollar doing usability tests, for example, it added not just one but eight dollars to estimated overall costs. Moreover, since IBM was in business to make a profit, an increase of 8 dollars in costs, meant an increase of nearly 20 dollars in projected price. This meant fewer predicted products sold.

In actuality, spending an additional dollar to improve usability of products should reduce service costs and sales and marketing costs. But that is not the formula that was used. The logic of the formula, corroborated by correlational data, was that bigger, more complex products had higher development costs and also had higher service, manufacturing and sales costs. When one compared a mainframe and a PC, this formula made sense. But when used as a decision tool by the development manager, it did not make sense. (By analogy, there is a strong correlation between the size of various species of mammals and their longevity. This, however, does not mean that you will live twice as long if you double your own body weight!).

Recall however, that the development manager’s career did not much depend on how successful the product was after release; it mainly depended on showing that they could get their product shipped. Development managers proved to be difficult to get “on board.” In some cases, despite the organizational pressures, some development managers did care about how the product did; were interested in making their products usable; did spent additional money to improve their product. Making such allies, however, relied on appealing to their personal pride of ownership or convincing them it was best for the company.

Some development managers suggested that perhaps I could get the Forecasters to change their formula so that they would be given credit for higher sales to balance the projected increase in price (and attendant reduction in sales volume forecasts). It would have been an excellent leverage point to have gotten the Forecasting function as an ally. I was not, however, sufficiently wise to accomplish this.

The organizational payoff matrix for the forecaster was quite skewed toward being conservative. If they used the existing formula and ended up thereby “killing” a product by reducing the sales forecast because of the money spent improving usability, no-one would ever find out that the forecaster might have erred. On the other hand, if I had convinced them by giving them evidence (which would necessarily be quite indirect) that the product, by virtue of its being more usable, would therefore sell many more units, there were at least two logical possibilities. First, I might be right and the product would be a success. The forecaster would have done the right thing and would keep their job (but not be likely to receive any special recognition, promotion, or raise). Second, I might be wrong (for a variety of reasons having nothing to do with usability such as unexpected competition or unexpected costs) and the product might tank. In that case, the company would lose a lot of money and the forecaster might well lose their job. While I occasionally found development managers I could convince to be allies because I could get them to value making the most excellent product over their own career, I never was able to gain any allies in the Forecasting function. In retrospect, I think I didn’t take sufficient time to discover the common ground that it would have taken to get them on board.

Resulting Context:

Finding allies will often enable the organization to change in ways that will benefit the organization as a whole and most of the individuals and sub-groups within it. If done with the best interests of the organization in mind, it should also increase internal mutual trust.

There is a related Anti-Pattern which is finding allies, not to change the organization in a positive way, but to subvert the organization. If, instead of trying to make IBM be more effective by making its products more usable, I had tried to ruin it by finding allies who, in the process of ruining IBM would also profit personally, that would have been highly unethical. Such a process, even if it ultimately failed, would decrease internal mutual trust and decrease the effectiveness of the organization. Of course, one could imagine that some competitor of IBM (or of a government or team) might try to destroy it from the inside out by favoring the promotion of those who would put their own interests ahead of the company or its customers. Finding allies is likely to be ethical when it is for the best interests of the overall organization and all its stakeholders and if is a known initiative (as was the case for improving the usability of IBM products).

References:

https://gps.ucsd.edu/faculty-directory/lewis-branscomb.html

Branscomb, L. and Thomas, J. (1984). Ease of use: A system design challenge. IBM Systems Journal, 23 (3), pp. 224-235.

Thomas, J. (1984) Organizing for human factors. In Y. Vassilou (Ed.) Human factors and interactive computing systems. Norwood, NJ: Ablex.

Thomas, J.C. (1985). Human factors in IBM. Proceedings of the Human Factors Society 29th Annual Meeting. Santa Monica, CA: Human Factors Society. 611-615.

——————————————————

Author Page on Amazon: https://www.amazon.com/author/truthtable

petersironwood

~ Finding, formulating and solving life's frustrations.

Tag Archives: usability

“Wizard of Oz”

Query By Example

Career Advice from Polonius

Problem Formulation: Who Knows What?

Getting Into (the “right”) Shape

Regression to the Mean

Study Slain by Swamp Monster!

Support Both Flow & Breakdown

Find and Cultivate Allies