This post focuses on the importance of discovering who knows what. It’s easy to assume (without thinking!) that everyone knows what you know. 

At IBM Research, around the turn of the century, I was asked to look at improving customer satisfaction about the search function on IBM’s website. Rather than using someone else’s search engine, IBM used one developed at IBM’s Haifa Research lab. It was a very good search engine. Yet, customers were not happy. By way of background, it’s worth noting that compared with many companies who have websites, IBM’s website was meant for a wide variety of users and contained many kinds of information. It was meant to support people buying their first Personal Computer and IT experts at large banks. It had information about a wide variety of hardware, software, and services. The site was designed to serve as an attractor for investors, business partners, and potential employees. In other words, the site was vast and diverse. This made having a good search function particularly important.  

A little study of the existing data which had been collected showed that the mean number of search terms entered by customers was only 1.2. What?? How can that be? Here’s a website with thousands of products and services and designed for use by a huge diversity of users and they were only entering a mean of 1.2 search terms? What were they thinking?!

Of course, there were a handful of situations when one search term might work; e.g., if you wanted to find out everything about a specific product that had a unique one-word name or acronym (which was rare). For most situations though, a more “reasonable” search might be something like: “Open positions IBM Research Austin” or “PC external hard drives” or “LOTUS NOTES training.” 

We invited a sample of users of IBM products & services to come into the lab and do some tasks that we designed to illuminate this issue. In the task, they would need to find specified information on the IBM website while I observed them. One issue became immediately apparent. The search bar on the landing page was far too small. In actuality, users could enter as many search terms as they liked. Their terms would keep scrolling and scrolling until they hit “ENTER.” The developers knew this, but most of our users did not. They assumed they had to “fit” their query into the very small footprint that presented itself visually. Recommendation one was simply to make that space much larger. Once the search bar was expanded to about three times its original size, the number of search terms increased dramatically, as did user satisfaction. 

In this case, the users framed their search problem in terms of: “How can I make the best query that fits into this tiny box.” (I’m not suggesting they said this to themselves consciously, but the visual affordance led them to that self-imposed constraint). The developers thought the users would frame their search problem in terms of: “What’s the best sequence of terms I can put into this virtually infinite window to get the search results I want.” After all, the developers knew that any number of terms could be entered. 

Although increasing the size of the search bar made a big difference, the supposedly good search engine still returned many amazingly bad results. Why? The people at the Haifa lab who had developed the search engine were world class. At some point, I looked at the HTML of some of the web pages. Many web pages had masses of irrelevant metadata. I found some of the people who developed these web pages and discussed things with them. Can you guess what was going on?

Many of the developers of web pages were the same people who had been developing print media for those same products and services. They had no training and no idea about metadata. So, to put up the webpage about product XYZ, they would go to a nice-looking web page about something else, say, training opportunities for ABC. They would copy that entire page, including the metadata, and then set about changing the text about ABC to text about product XYZ. In many cases, they assumed that the strange stuff in angle brackets was some bizarre coding stuff that was necessary for the page to operate properly. They left it untouched. Furthermore, when they “tested” the pages they had created about XYZ, they looked okay. The information about XYZ was there. Problem solved.

Only of course, the problem wasn’t solved. The search engine considered the metadata that described the contents to be even more important than the contents themselves. So, the user would issue a query about XYZ and receive links about ABC because the XYZ page still had the “invisible” metadata about ABC. In this case, many of the website developers thought their problem was to put in good data when what they really needed to do was put in good data and relevant metadata. 

A third issue also revealed itself from watching users. In attempting to do their tasks, many of them suggested that IBM should provide a way for more than one webpage to appear side by side on the screen so that they could, for instance, compare features and functions of two different product models rather than having to copy the information from the web page about a particular model and then compare their notes to the second page. 

Good suggestion. 

Of course, IBM & Microsoft had provided this function. All one had to do was “Right Click” in order to bring up a new window. Remember, these were not naive users. These were people who actually used IBM products. They “knew” how to use the PC and the main applications. Yet, they were still unfamiliar with the use of Right Click. Indeed, allowing on-screen comparisons is one of the handiest uses of Right-Click for many people. 

This issue is indicative of a very pervasive problem. Ironically, it is an outgrowth of good usability! When I began working with computers, almost nothing was intuitive. No-one would even attempt to start programming in FORTRAN or SNOBOL, let alone Assembly Language or Machine Code without looking at the manual. But LOTUS NOTES? A browser? A modern text editor? You can use these without even looking at the manual. That’s a great thing. But — 

…there’s a downside. The downside is that you may have developed procedures that work, but they may be extremely inefficient. You “muddle through” without ever realizing that there’s a much more efficient way to do things. Generally speaking, many users formulate their problem, say, in terms like: “How do I create and edit a document in this editor?” They do not formulate it in terms of: “How do I efficiently create and edit a document in this editor?” The developers know all the splendid features and functions they’ve put into the hardware and software, but the user doesn’t. 

It’s also worth noting that results in HCI/UX are dependent on the context. I would tend to assume that in 2021 (when I first published this post), most PC users knew about right-clicking in a browser even though in 2000, none of the ones I studied seemed to realize it. But —

I could be wrong. 

————————————

The Invisibility Cloak of Habit

Essays on America: Wednesday

Index to a catalog of “best practices” in teamwork & collaboration. 

Author Page on Amazon

What about the butter dish?

Labelism

The Stopping Rule

The Update Problem