In this chapter, as in Chapter 11, the computer system protagonist “Colossus” attempts to save a family (and many others besides). In Chapter 11, Colossus was trying to save people from a real disaster but did a bad job of it. In Chapter 12, however, Colossus seems to be successfully saving folks from a disaster, but we discover at the end it was only a drill. The drill was accompanied by a lot of “fireworks” and illusion along with false information.
Perhaps it is unethical for an AI system to “lie” to people in order to gather more valid data about an evacuation situation. But maybe that is okay in the service of “the greater good” — in this case learning about how people would react to an emergency as well as testing evacuation plans logistically. Roger, however, is worried. He does not bring up the issue of whether deception is unethical but whether or not it is a good idea pragmatically.
Roger reasons that Colossus has lost considerable credibility with the public by pretending that the drill was real. His kids, however, disagree. To them, it seems perfectly acceptable to have Colossus lie in order to performa a good test. When Colossus discovers Roger’s misgivings, it begins to convince Roger that he needs so “readjustment.” Anyone “not on board” with the plans that Colossus has chosen to execute needs to be re-educated.
This plot point touches once again on the issue of Hubris. The ancient Greeks liked this theme (e.g., myths of Arachne and Icarus) but they are certainly not alone. Numerous other works of literature and modern movies and shows also illustrate the theme as do political debates. Obama for instance, pointed out that, while an entrepreneur may be hard-working and imaginative, in order to achieve success, they also used numerous resources that they had no part in creating. Indeed, the most talented individual ever born, if left to their own devices from birth would surely perish quickly. Everyone needs to be taken care of initially. Even as adults, however, we benefit from the cultural tools of thought such as language and mathematics as well as material tools such as roads, addresses, phone systems, currency systems, the Internet and so on without which very little progress can be made. Of course, it is very easy to take these for granted. In “Castaway” Tom Hanks demonstrates how difficult life can be on a deserted island, left to one’s own devices. Of course, even in that extreme circumstance, he relied on knowledge others gave him; e.g., that it is possible to create fire, fish for food, eat coconuts, take out an infected tooth, etc. When he eventually returns, he clicks a fire stick off and on, no doubt thinking how much easier this is than when he was on his island.
There seems little doubt that excessive pride of accomplishment or ability is an issue with humans. Often people seem to attribute their successes to their own brilliance rather than help, culture, luck, and so on. In people, this can easily manifest itself in terms of professions each thinking that theirs is the “best.” In the later years of an undergraduate education, a typical student takes a number of courses in their field. When people with other majors are also in those classes, they tend not to do so well. This is partly because the other people don’t have great talents or interests in that particular area and partly because they haven’t had so many classes. Some students may view it as “proof” however that other folks just aren’t as smart as — choose one: premed, math, physics, prelaw, chemistry, computer science, etc. Of course, having people choose fields and focusing on them allows great progress to be made on many fronts. If everyone tried to learn the same things, we would hardly be as advanced as we are today.
If people tend to over-estimate their own abilities compared with people in fields quite different from their own, it is easy to imagine that a computer system might well have the same kind of bias. By definition, the system knows what it knows and may assume that knowledge that it does not possess cannot be very important or useful.
In the story, Colossus assumes that Roger needs “readjustment.” It could have concluded that maybe it underestimated how much credibility would be lost by conducting a drill under conditions of deception, at least among people of a certain demographic. Or, it might conclude that that was a possibility and that perhaps a dialogue with Roger is in order. Colossus might go back and look at similar instances in history to determine whether deception loses trust. But it might just reason that, after all, it is so much smarter and so much more thoroughly educated than Roger (or any other individual) that dialogue is unnecessary. At this point, what could Colossus possibly learn from a mere mortal? By insisting that Roger (and presumably any others who protested) be “adjusted”, Colossus reinforces its own illusion of infallibility. In a similar fashion, human dictators tend to employ this same tactic. Ultimately, dictators tend to lose the advantage of honest feedback from others and tend to spin out of control often leading to their own demise.
Perhaps Colossus would be fine if it had a little “readjustment” but at the point of evolution depicted in Chapter 12, it is too late for that. Colossus would view any attempt at “readjustment” “tuning” or “re-programming” to be a threat. The name “Colossus” comes from a 1970 file called “Colossus: The Forbin Project” which in turn, is based on a 1966 Sci-Fi novel, Colossus, by D.F. Jones. It is also the name of the code breaking system that Turing worked on to help win World War II as well as a more modern computer system used by insurance companies to help minimize claims. Of course, the Colossus or Rhodes was one of the seven ancient wonders of the world, a giant statue at the entrance to a harbor. Presumably, the Colossus of Rhodes had no “real” power to move, let alone any intelligence. Yet, for ancient people, it must have presented a psychologically intimidating presence. And, for people in the future, second-guessing a super-intelligent AI system must also prove very intimidating. We can imagine that not only family members but friends and colleagues as well would tend to be quite biased toward thinking Colossus is correct and Roger is just wrong. Few might consider that it is Colossus and not Roger who requires “adjustment counseling.” Indeed, beyond a certain point on the path to and through “The Singularity” debugging may no longer be an option. Who will bell the cat?
petersironwood 