Wizard of Oz | petersironwood

Tags

HCI, IBM, research, usability, UX, Wizard of Oz

(Some Lessons Learned from studies in Human-Computer Interaction/User Experience conducted at IBM Research in the mid-70’s.)

Wizard of Oz

One of the studies I conducted at IBM Research in the mid 1970’s was part of an effort to do “Automatic Programming” — a department under Pat Goldberg. The first level manager I worked with was Irving Wladawsky (later Irving Wladawsky-Berger). His group wanted to develop a system that would allow the owner/operator of a small business to type requirements into a computer in English (or something English-like) and have the system itself produce RPG code to run the business so described.

The underlying motivation from an IBM business perspective was that many small businesses could well afford a computer to do inventory, fulfill orders, etc. but they couldn’t afford to hire programmers to create such a system from scratch. The small business owner in the mid-1970’s did not program! Yet, for the most part, they understood how their business worked. The notion was that a natural language understanding and generation program could dialogue with the user/owner and through that process, understand their “business rules.” No costly programmers needed!

An interesting side note: at that time, we were told that IBM corporate forbade us to use the terms “Artificial Intelligence” or “Robotics” to describe our work because some PR firm had determined that these terms were too scary for the general public. So, IBM had research in “mechanical assembly” but not “robotics.” We had work in character recognition, speech recognition, handwriting recognition, automatic program generation, and compiler optimization. But no work in “Artificial Intelligence.” (Wink, wink, nod, nod).

Labelism: Confusing a thing with the label for that thing.

Another interesting side note: I worked at IBM Research for a dozen years; started an AI lab at NYNEX where I worked another 13 years; came back to IBM Research and several years later found myself working on the same problem! We were still trying to make a system to allow small businesses to generate their code automatically. In my second iteration, rather than using natural language, we were trying to make the specification of business rules in a graph language that was intuitive enough for business owners. This was a different approach, but trying to address the same underlying desire: to bring computing to small business without incurring the heavy costs of programming and maintenance.

Let’s return to iteration one — the natural language approach @ 1975. Well, one issue was that no-one had a natural language program that even approximated being able to do the job. So…how to study people’s interaction with a system that doesn’t exist?

We used an approach that my colleague Jeff Kelly called the “Wizard of Oz” technique; viz., use a human being (in this case, me) to simulate how the system might work and record people’s behavior. In this way, we could discover many of the issues that such a natural language programming system would have to deal with. I had already had plenty of experience interacting with a computer; and I had acting experience. I could “play the part” of a computer fairly well as I typed in my questions and answers.

(Description of “The Wizard of Oz” technique).

IBM Research in Yorktown had roughly a thousand people including not only scientists, programmers, and engineers but also a number of business people (who did not know how to program). I knew some of them from playing tennis and table tennis and we used those folks as initial subjects. What did I find? Good news and bad news.

Dealing with natural language is tricky for many reasons. One of those reasons is that English, including the English that people normally use to describe their business, is filled with words that have multiple meanings; e.g., “file”, “run”, “program”, “object”, “table”, etc. But here is the good news: although it’s true that many English words have many meanings, when these business people described business procedures, almost all of the lexical ambiguity vanished! The program to understand business English would not have to distinguish between a business file and a nail file; it wouldn’t have to worry about distinguishing a run in baseball or a run in stockings from a run of the payroll program; it wouldn’t have to distinguish between the table in a relational data base and the table in your dining room. The domain would mainly constrain! That’s the good news.

The bad news was dialogue management. How can the machine recognize a misunderstanding and how can it correct it? To make matters worse, while business people were fairly consistent in the way they described how their business ran, they were not consistent in how they talked about the communication. If a human being senses that another one is misunderstanding, then, depending on context they might: raise their eyebrows, say “Huh?”, “Come again?”, “What?” “I think I lost you.” “WTF?” “Are you kidding?”, “We’re on different wavelengths,” “I don’t get it.” “But…wait.”

Sometimes, these are referred to as “meta-comments.” Here’s a simple example that took place in the study.

One of the business people told me about various discounts. I had assumed (playing the part of the computer) that he was talking about discounts for items that were being discounted due to inventory management. I recorded all the various percentages and so on. Then, he said, “Now, we also give discounts for various items.”

At that time, most natural language systems of that era simply ignored words like “now” and “also” in this context. Stepping out of my role as a “computer system” and thinking about from the perspective of a human conversational partner though, these words are crucial! What it signals is a change in topic. In the larger context of our conversation, it shows that everything that had just been said, which I thought had been about item discounts, was not about item discounts!

This is just one example, but there were many more. In my more recent experience interacting with various computer dialogue systems, being able to recognize the signals of miscommunication and being able to repair misunderstandings is still not very well-handled more than four decades later.

I’d be interested in any pointers you have to a system that you think deals with meta-communication in a natural and robust manner. I do not think that it is beyond the pale of possibility. The general categories of the ways that people misunderstand each other is not infinite. John Anderson developed excellent tutoring systems for LISP and geometry and those systems worked something like human tutors in that, the tutor inferred the mental model of an individual student and focused instruction on correcting any misconceptions. My intuition is that a generic system built with equal complexity could deal with most of the issues as well as the average human being deals with them; i.e., imperfectly.

—————————————

Lessons Learned: #1 You can test aspects of a system even before it’s built or even completely defined. One method that has been used many times: “Wizard of Oz.”

Lessons Learned #2: Language used by professionals to talk about their domain is much more constrained in terms of lexical ambiguity than is language when considered by all native speakers.

Lessons Learned #3: People in “our culture” (i.e., US business culture) do not have an agreed upon and consistent vocabulary for talking about communication nor a consistent process for dealing with them.

Lessons Learned #4: Speaking of communication errors, I don’t recall why, but it was about this time, that I realized that my notion about how research results would be transferred to other parts of IBM was a complete and utter fantasy. I hadn’t articulated it, but it was basically that I would do research, write the results up for publication in scientific journals for an academic audience and publish Research Reports which would be eagerly consumed by anyone who needed to know. I’m not proud of this. LOL. But that’s really kind of how I viewed it. And, then, after a few years, I realized that it really mainly came about through relationships. That was something that people had been showing me all my life, but which I don’t think anyone ever stated it explicitly enough.