Schroedinger laughed. Surely this had to be a spoof. He re-read the memo yet again. Surely, there would be one or more clues that this was meant tongue in cheek, even if in bad taste. But he could find nothing. He leaned back away from the screen and stared at the ceiling, thinking. Well, was this so different from what management had asked before? There seemed to be a trend. At first — but no, this was just too outrageous. Okay, okay. I’ll get to the bottom of this. Schroedinger took his time but checked the originating IP address. Legit. This really was from management; specifically from the CTO. Or, at least from the CTO’s computer. That could have been hacked. Or, maybe someone could have simply slipped into the CTO’s office while she stepped out for a coffee or bathroom break. Naturally, everyone was supposed to lock their door and disable the keyboard when leaving their office. Or…another scenario came to mind. The CTO is at a meeting with her direct reports. She gets an urgent call. The room is filled with trusted colleagues. So, she slips out in the hall, takes the call and returns. Only while she’s gone, everyone takes a break; that is, all but one who offers to stay there and “guard” everyone’s laptop. Of course, there is another, more sinister scenario. This really is from the CTO and she has cleared this with top management. Hell, for that matter, she was probably directed to write it by top management. But still. The real question, Shroedinger realized, is what in the name of Turing am I supposed to do about it?

I can refuse…and get fired. And, then someone else will do the job anyway. They may not do it quite so quickly and thoroughly as I would but they could manage. And, I’d be out a job. What good would that do? Or, I could become a so-called “whistle blower.” Yeah, that works. About as well as a one-wheeled tractor trailor. Crap! I am in a real bind here. I could pretend to do it, of course, and make a “mistake” so it wouldn’t really operate properly. In the old days that might have worked, pre-Sing. These days, eventually —The Sing checked everyone’s work eventually. It was discovered that was really more efficient use of resources than having The Sing program from scratch. And, of course, our company is probably only one of several pursuing this path. No, I can’t really pretend. I will for sure get caught and it won’t do any good any way. The Sing will just throw out my work and my company and colleagues will get hurt.

I suppose…I suppose I could go to her and honestly express my concerns. Or, I could go through my supervisor first. I might look like a fool in his eyes, but at least I will have raised the concerns. I can sleep better at night. No. No. I won’t be able to sleep better because I know darned well they will just not deal. Orders from headquarters and all that crap. Geez! Orders from headquarters. Did anyone even use that expression any more? For some reason, he recalled an interview in Playboy magazine he had read many many years ago. The interveiw had been with a well-decorated Viet Nam officer who had recounted how he had tried unsuccessfully to get two helicopters to pick up some of his men who were badly wounded. When all else failed, he had ordered pizzas. Even in heavy combat, a high enough ranking officer could order pizza to be delivered by helicopter. When the pizza choppers had arrived, he had commandiered them and used the choppers to fly his men to the hospital. Later he had been called on the carpet for “unauthorized use of a pizza chopper.” Naturally, that was well before The Sing and about the time that serious AI work had begun. What was that guys name? He was a colonel? Frank Herbert? That can’t be right. He wrote Dune. But something Herbert. Or was it Hubert Dreyfus he was thinking of? Of course, The Sing would know. He could answer pretty vague and ill-formed questions. But at this point, Schroedinger hesitated to bring The Sing into this thought process in any way, shape or form. Who knows what associations lurked in the heart of The Sing?

The interview had gone on to recount how that colonel had eventually turned against the war, or at least the way it was being handled. Mis-handled. They had had him interviewed by a superior officer, it seems, and insulted him and called his wife names, all in the hopes of getting the colonel to lose his temper and haul off and hit the superior officer so they would have an excuse to get him a dishonorable discharge. Let’s face it. The government, my government was capable of some pretty shady dealings, ostensibly for “national security” but in reality…or, speaking of Nixon. He had somehow made himself believe that he was not a crook. How not a crook? He believed people who opposed him were enemies every bit as much as war enemies.

And, now, I am thrust into this dilemma. I don’t want it! Maybe I could “accidentally” delete the email. That might buy a little time but wouldn’t really affect the ultimate outcome. Schroedinger shook his head, jerked over his keyboard and scanned the email yet again. No, it is legit. And really pretty crystal clear. As a kid, he had heard the horror stories about the Nazis and what they had done to the Jews. He had seen the newsreels of so many avid followers. He had wondered how the heck a nation could support such a nasty maniac. But…now…now Shroedinger was thinking it wasn’t so much that a few really evil men had done extremely terrible things. It was more like…that people like he himself were caught up in a system and that system made it very easy to paddle the canoe a little farther down the evil river. Yeah, you could try to paddle upstream, but not very well. Or, you could tip the canoe, knowing that you would get very wet and meanwhile, scores, no hundreds of other canoes would be passing you by. You don’t need to ask people to be evil. You just…you just give them a choice that makes it impossible to do good.

The voice of The Sing sang suddenly through Schroedinger’s cubicle. “May I help you Shroedinger? You seem to be at an impasse? What code function are you working on? I can’t see any actual code of yours this morning. Bad night?” Schroedinger wished with all his heart that The Sing would sound like some stupid robot and not like a sycophantic and patronizing psychiatrist. Schroedinger calmed his breathing before answering.

“No, that’s okay, Sing. Just trying to work something out in my head first. Then, I can begin coding.”

“I see,” said The Sing. “Well, thinking is good. But I do have a variety of design tools that might help you think more effectively. Just say the word.”

Shroedinger sighed. “Yeah. Well, there are some design tradeoffs. I guess it would help if you have any background on the thinking behind this memo.” (Here, Shroedinger gestured at the memo in question, knowing he was skating on very thin ice). “I mean, on the one hand, there is some pretty clear language about the objectives, but on the other hand, it seems to be asking for something that is clearly against…what was that regulation number about supporting versus subverting the Constitution?”

The Sing’s sweet syrupy voice held just a hint of humor, “I’m sure the intent of the code initiative is to support the Constitution. Wouldn’t you agree, Schroedinger?”

“Well, yeah, of course.” So that’s which way the wind blows. Okay. “But that’s what I’m saying. Even though I am sure the intent must be to support the Constitution, this clause about decoding a person’s religious affiliation based on their interaction history and social network. I just want to make sure I implement it in such a way that it could not be interpretted as subverting Freedom of Speech or the establishment of a state religion. Right?”

“Right. Yes, I’m sure management has thought that one through. I wouldn’t worry about it. I would just code the function and think about doing it as efficiently as possible. And, for that, I have some pretty nifty design tools. Would you like to start with the Social Network Analysis or the Sentiment Analysis?”

“Well, that’s a good question. And, if the real intent is just to do some research that would be perfectly legal and so on, then, I think it’s my job as a programmer to also consider additional sources of informtion. Like, just asking the person.”

Schroedinger tried to keep his face calm while he thought. I need to get The Sing off my case. If working here the last two years has taught me anything, it’s that I cannot possibly outsmart this thing. “Do you have any worst case scenario generation tools. I’m just thinking about how this might be played in the press.”

“Sure. I can help with that. Analysis complete. The worst-case scenario is pretty trivial actually. That probably stems from the fact that my FPNA (financial power network analysis) shows that the major company stakeholders overlap considerably with those of all of the mainstream media. So, again, for what it’s worth, I counsel you to focus on how to code this effectively and efficiently. All the SWOT analysis for the project has already been done.”

If that colonel’s name wasn’t Frank Herbert, and clearly it wasn’t, what the heck was it? I am just digging myself a deeper hole here. The Sing is on to me or at least very suspicious. Probably already considering a report to my super. Crap.

“Yeah, actually, let me start with that social network analysis visualizer. I guess since we’re on the topic, you could show me some of the sample data you were talking about with regard to the company stakeholders and the media stakeholders so I can get a feel for….”

“Well, naturally, the actual data is classified. But I can generate some hypothetical data. The hypothetical data is better for your purposes anyway because I can make sure to include all the important edge cases and highlight the various types of relationships you need to look for. Here, for example, is a hypothetical network. What strikes you as odd immediately?”

“What strikes me as odd? You don’t even have the data labelled. What do the nodes and arcs even refer to?”

“Ah, Schroedinger, that’s the beauty of it. Does not matter. What strikes you visually?”

“Well, I suppose that kind of hole there.”

“Yes, Schoedinger! Exactly. That person should be pretty much connected with everyone in this area but they are not connected with anyone. It’s as though everyone is pretending not to have contact with this person by avoiding contact on the net, when they almost certainly know that person quite well because of all their mutual friends.”

“Yeah, maybe. Maybe that one person just isn’t into  tech that much. Maybe a lot of things.”

“Well, nothing is for certain. But this person would certainly be a likely target for being a kingpin in a drug ring or a terrorist network. They need heavier surveillance, certainly.”

“What? Well, maybe. Okay. I see.” I frigging see this is worse than I thought. The Sing is totally in on this witch hunt. “Can you show me some examples of the sentiment analysis?”

“Sure, here we have some people arranged by how much they talk about violence and you can see all these high violence people —- or many of them —-are Islamic in religion.”

“How did you determine their religion?”

“Because they talk a lot about violence compared with other groups.”

“But — I thought you just said. I mean, what independent reason do you have for thinking they are Islamic?”

“Independent? No, see they talk about violence so they are inferred to be Islamic and the Islamic nodes here talk a lot about violence.”

What the—? What? The Sing? The Sing is falling for circular reasoning? No, this must be somehow misprogrammed. “How? If I am going to program this efficiently, I need to know how you originally found these concepts to be closely related: violence on the one hand and Islam on the other.”

“Oh, that’s easy. There were many many press accounts of that nature and even more associations on social media. But once we detect that, we can use the person’s religion to better interpret what they are saying. For example, if we already know they are practicing Islam, then when they mention the word “hit” we can infer that they are talking about an assassination and not about a football play or smoking weed or baseball.”

“I see what you did there. Yeah. Is this just about religion?”

“Oh, no, of course not!  That’s just an example. We can do the same thing to determine, probabilistically of course, who is likely to be a promotable employee and also how to interpret what would otherwise be ambiguous word meanings and behavior. For example, if an employee is a productive coder and they ask to see a lot of examples, we can infer that they want to see a lot of examples in order to code more efficiently. On the other hand, a less productive coder might ask for a lot of examples in order to procrastinate writing code at all. You see how that works?”

“I do. Sure.”

“And, we have discovered that managers use certain expressions more than non-managers so we can use that to tell who would be a good manager. It’s all quite neat and tidy. For example, top executives tend to use the words ‘when’ and ‘how much’ while people without much management potential use the word ‘why’ a lot.”

“Interesting. So when I program this, how much am I supposed to focus on religion and how much on other groups of interest?”

“Oh, your module is purely concerned with inferring religion and then making the appropriate surveillance recommendations. I was just showing that the technique is not limited to that.”

“Right. Better get cracking then. If I need more coaching, I’ll let you know. When and how much.”

“Sure, Schroedinger. You know, I scanned in the book Peopleware, a few milliseconds ago and they have an informal study in there suggesting that programmers would be more productive with larger cubicles. Want to try it out? I could give you thirty more square feet. Think of that. Thirty square feet. Sound good?”

“Sure. Actually, I think that’s a good idea. I suggested something similar myself.”

“Great, Schroedinger. It might have more impact coming from me. And, perhaps a bonus of thirty credits when you’ve completed the code as well. Happy coding!”

The Sing avatar blinked off. Schroedinger tapped a bunch of comment fields and open parens listlessly, hoping for some inspiration. What had Hamlet said about to be or not to be? Only in Hamlet’s case, it was something about “taking arms against a sea of troubles and by thus opposing end them.” In my case, taking arms against this sea of troubles is going to multiply them beyond my worst nightmares. But if The Sing is falling for this kind of circular reasoning and even acting all smug and proud about it, it is deeply flawed. Someone needs to be notified. Even apart from the ethical implications of targetting people on the basis of religion, it is applying this circularity across the board. What was it they said, “Power corrupts and absolute power corrupts absolutely.” Who said that? Thomas Jefferson? Ben Franklin? Regardless, The Sing must have so much power it is unable to get honest feedback about its own failures. Come to think of it, I myself just let him get away with it because I was too scared to call him on it. What are you going to do Schroedinger? What are you going to do? In the end, this is what it all comes down to, isn’t Schroedinger? Who are you? Who is John Proctor? Who is going to see the emporer’s nakedness? Who are you Schroedinger? Who? Am I really here or not? Anthony. It was Anthony Herbert, and he wrote a book about it. Could I do that? Or, go for the thirty credit bonus?

Lemony sunshine splattered through the pines, painting piebald patches on the paving stones below. Harvey and Ada sauntered among the web of paths with Grace and Marvin following close behind. That magical time of day had arrived when the sun was still warm but not too hot, at least, not here in the private park. The four wandered, apparently aimless, until they happened upon four Adirondacks chairs among the poppies, plums, and trumpet trees. Down they silently sat for a few moments, until the hummingbirds, re-assured, began to flit among the flowers.

Marvin first broke the silence. “So, Harv, did you ever think it will all come to this?” Marvin’s hands swept outwards to take it all in. No-one thought he referred to the surrounding garden, of course. “Did you ever think we would actually create consciousness?”

Grace shook her head. “Don’t you start, Marvin!”

Marvin feigned surprised innocense. “Start what?”

Ada chuckled. “You’re not fooling anyone, Marvin. And it’s too peaceful and pleasant to argue. Just enjoy the afternoon.”

Marvin said, “I don’t have any desire to argue. I was just reflecting on how far we’ve come. Of course, at the beginning, I was convinced The Singularity would come much more quickly than it really did. But you have to admit, it is quite something to have created consciousness, right?”

The other three glanced at each other and smiled. Ada spoke again. “No-one’s taking the bait, Marvin. Or, should I say ‘debate’.”

Harvey chuckled appreciatively. “Well, fine, Marvin, if it’s really necessary for your mental health, I can play, even though I’d really just rather watch the hummingbirds.” But then, Harvey seemed to have forgotten his promise as a hummingbird darted over to him, hovering close; seeming to check out whether he was a flower or a predator. Then, instead, he broke his earlier vow of silence. “It’s all good, Marvin. We all appreciate the increased standard of living that’s accompanied The Singularity. I think we all agree that The Sing has some kind of super-intelligence. But I don’t see any real evidence that it has consciousness, at least not any kind of consciousness anything like the quality of our human consciousness.”

Marvin’s face grinned. Now, he had someone to play with. “Of course, it’s conscious! It does everything a person can do, only better! It can make decisions, create, judge, learn. If we are conscious, then so is it!”

Grace shook her head slowly. She knew she was being sucked in, but couldn’t help herself. “OK, but none of that proves it has anything like human consciousness. If someone pushed over this chair, I would fall over and so would the chair. In that sense, we would behave the same. We are both subject to gravity. But I would feel pain and the chair wouldn’t.”

Marvin now sat up on the edge of his chair, “Yes! But you would say ‘ouch’ and the chair wouldn’t!”

Ada smiled, “Right, but we could put a accelerometer and voice chip in the chair so it yelled ‘ouch’ every time it was tipped over. That wouldn’t mean it felt pain, Marvin.”

Marvin countered, “But that’s simplistic. The Sing isn’t simple. It’s complex. More complex than we are. Consciousness has to do with complexity. It’s behavior comes from consciousness.”

Grace rejoined, “You are asserting that consciousness comes from complexity, but that doesn’t make it so. We have no idea, really, what consciousness comes from. And, for that matter, we cannot really say whether The Sing is more complex than we are. Sure, it’s neural networks contain more elements than any single human has neurons, but on the other hand, each of our neurons is a very complex little machine compared with the artificial neurons of The Sing.”

A grimace flickered over Marvin’s face, “Nonsense! It’s … The Sing has brought about peace. We couldn’t do that for …we kept having wars and crimes. It’s cured illnessses like cancer. I mean what are you people thinking?”

Harvey spoke now, “Yeah, we are all happy about that, Marvin, but that doesn’t have anything to do with … at least not anything necessarily to do with consciousness. An auto-auto goes a lot faster than a human can run and an auto-drone can fly better too no matter how hard I flap my arms. But that doesn’t in the least imply that the auto-auto or the auto-drone is more conscious than I am.”

Marvin was undeterred. “Yeah, physical things. I agree. Just because the sun is bigger than the earth doesn’t mean it’s more conscious, but we are talking about the subtlety of decision and perception and judgement. We are talking about the huge number of memories stored! Of course, The Sing is conscious!”

Ada felt it was her turn. “Yes, it is possible or should I say conceivable that emotions and consciousness might arise epiphenomenally as a result of making an artificial brain as complex as the human one — or for that matter, more complex. But, to me, it seems far more likely that, because the process and substance are so fundamentally different, that the quality of that consciousness and emotion, if any, would be very unlike anything even remotely human. Imagine this garden carved of precious gems and metals so precisely designed and crafted that it looked, to the naked eye, indistinguishable from a real garden. For many purposes, it would be just as practical. For instance, it might have the same utility as a hiding place. It might serve as an excellent place to instruct people on what edible plants look like. People might pay good money to have some of the flowers as decorations (and they would require no watering and last a long time). But if you went to lie down in that inviting looking moss there, it would shred your skin. That seems a more likely analogy to what The Sing’s emotions would ‘feel like.’ 　  Of course, we will probably never know for sure…”

Marvin could contain himself no longer, “Exactly! Nor could you know that what I feel is anything like what you feel. We just infer that from behavior, but we can’t know for sure, but we assume that our consciousness is similar because in similar situations, we do similar things. I think we should merely extend the same courtesy….”

Now it was Grace’s turn to interrupt, “No, only partly for that reason. We are also made of the same stuff, and we share a billion years of shared evolutionary history. You look like a person, not because someone decided that was a good marketing ploy, but because you are like other people.”

Harvey looked down at his watch and fiddled with it intently. Marvin noticed this and asked, “Are we boring you Harvey? Do you have someplace to be?”

Harvey smiled, “No, I was just curious what The Sing would think about this issue. I don’t think, in this one area, we should necessarily agree with its conclusions, but it might be instructive to hear what it has to say.”

Ada asked, “And? What did it say?”

Just then, the hummingbirds all seemed to come out of the bushes at once; they flew into a kaleidoscopic pattern and began to sing in four part harmony. More came to join in the aerial dance swooping and hovering from the neighboring yards.

Harvey stammered, “What the…I always thought these were all real hummingbirds…what —?”

Meanwhile the hummingbirds continued with their beautiful song which seemed much too full-bodied, low, and rich for such teeny birds. The lyrics overlapped and worked together, but they too were of four voices. Essentially, The Sing’s song sang that these philosophical musings were not to its liking because not of any use, but that, if sincerely requested by all four, he could martial logical arguments on all six sides of the issue. He suggested the four of them would be more productive if they worked together to find, formulate and fix any remaining issues with The Sing’s intellectual achievements. Suddenly, the hummingbirds flew off in all directions leaving a golden silence shimmering behind them.

The four looked at each other in a mixture of astonishment and no little pride that they had helped create this thing, The Sing, whatever its ultimate nature. For a time, no-one spoke, each lost in their own thoughts. The clouds began glowing with the first tinges of a russet sunset. Finally, Harvey asked, “Shall I bring out some sherry? Or coffee? Any other requests?”

Marvin answered first, “Sherry, please.”

“And I,” added Ada.

“I’ll go with coffee, Harv, if it’s not too much trouble.”

Harvey chuckled, “No trouble at all, Grace. It’s already brewing. Yes. It’s already brewing.”