Bostrom Follow-up

For me, the interview with Nick Bostrom was mind-blowing. I’ve been thinking about it quite a bit since we did the interview and the thoughtful comments from listeners have continued the process.
I made a strong claim early on that surprised Bostrom so I want to try to give a little bit of background for why I made the claim. I made the observation that Bostrom’s view of superintelligence reminded me of the way medieval theologians talk about God. Bostrom responded by pointing out that unlike God, his concept of superintelligence was limited by the laws of physics. That is true, but other than that, in his book he is able to imagine a superintelligence with all kinds of superpowers. By his definition, these powers are beyond human powers, so to be more precise than I was before, his concept of superintelligence is somewhere between man and God but a lot closer to a God-like set of powers than merely advanced human skills.

Here is one description from his book of what Bostrom imagines a superintelligent machine might be capable of doing:

Using its strategizing superpower, the AI develops a robust plan for achieving its long-term goals. (In particular, the AI does not adopt a plan so stupid that even we present-day humans can foresee how it would inevitably fail. This criterion rules out many science fiction scenarios that end in human triumph. The plan might involve a period of covert action during which the AI conceals its intellectual development from the human programmers in order to avoid setting off alarms. The AI might also mask its true proclivities, pretending to be cooperative and docile. If the AI has (perhaps for safety reasons) been confined to an isolated computer, it may use its social manipulation superpower to persuade the gatekeepers to let it gain access to an Internet port. Alternatively, the AI might use its hacking superpower to escape its confinement. Spreading over the Internet may enable the AI to expand its hardware capacity and knowledge base, further increasing its intellectual superiority. An AI might also engage in licit or illicit economic activity to obtain funds with which to buy computer power, data, and other resources. At this point, there are several ways for the AI to achieve results outside the virtual realm. It could use its hacking superpower to take direct control of robotic manipulators and automated laboratories. Or it could use its social manipulation superpower to persuade human collaborators to serve as its legs and hands. Or it could acquire financial assets from online transactions and use them to purchase services and influence.

So it has goals that threaten humanity. But it’s so smart and cunning relative to human beings that it can achieve these goals by manipulating humans without humans catching on, even though it has no legs and hands–no mobility, no physical resources at its direct disposal. It will just get some. How? By socially manipulating people using its social manipulation superpower. This is where I started to get a little uneasy. Social manipulation isn’t easy to do. Being smarter doesn’t make you better at it. It might have to socially manipulate millions of people who can communicate with each other. It’s just not obvious that that’s possible.

Actually I become uneasy with the first sentence from the quote. Strategizing superpower? The challenge of strategizing and executing a successful strategy isn’t difficult because we’re not smart enough or don’t have enough data. The challenge is a fundamental uncertainty about the future where billions of people have their own plans and strategies. Human beings are really mediocre at foreseeing the future and how various decisions might affect that future. So the United States attacks Iraq and all kinds of things happen that weren’t foreseen. Do you think there’s a scientific way to foresee the consequences of war more accurately? It isn’t obvious to me that that’s possible. There’s a level of uncertainty that comes with war that isn’t about the fact that we don’t have enough previous data points about war and its consequences. So I have trouble imagining how a superintelligent entity could become good or great or precise about strategizing. An omniscient god can. Such a god by definition can see the future. Humans can’t. So when you assume superintelligent entities can achieve goals through their strategizing superpower, well to me, that sounds like assuming your answer before you get started. It borders on magical thinking. It presumes a god-like omniscience.

A lot of this part of the conversation reminded me of Hayek and his Nobel Lecture. In that lecture Hayek isn’t saying that we don’t know enough to steer the economy. He’s saying we’ll never know enough. He writes:

Consider some ball game played by a few people of approximately equal skill. If we knew a few particular facts in addition to our general knowledge of the ability of the individual players, such as their state of attention, their perceptions and the state of their hearts, lungs, muscles etc. at each moment of the game, we could probably predict the outcome. Indeed, if we were familiar both with the game and the teams we should probably have a fairly shrewd idea on what the outcome will depend. But we shall of course not be able to ascertain those facts and in consequence the result of the game will be outside the range of the scientifically predictable, however well we may know what effects particular events would have on the result of the game.

Is it imaginable that some superintelligent entity could design health sensors that would gather all these data along with each player’s level of motivation? That it could accurately forecast will, desire, passion? (Below, I speculate whether these will ultimately fall under the sway of the knowable.) Then all outcomes might be foreseeable, strategizing is imaginable, and manipulation doable. As I mentioned in the conversation, this question of the power of the superintelligent reminded me of the Socialist Calculation Debate of the early part of the 20th century where MIses and Hayek argued that top-down central planning was not ineffective because of an insufficiently large computer or data set but rather it was ineffective because the fundamental problem was simply too complex. It is not possible for the man of system or the machine of superintelligence to move the chess pieces of society unerringly when each piece has a motion of its own.

So many intractable social problems are intractable for reasons much greater than a lack of intelligence. They are not engineering problems but of a different nature, unamenable to solution by bigger, faster brains. Certainly this perspective comes naturally to economists who often think that there are no solutions only tradeoffs.

I thought the interview really went off the rails when we talked about justice and making the world a better place. Bostrom seems to imply that the challenge of improving the world is a matter of finding the right way to represent human aspiration in computer code:

The shortcoming is in our current ability to describe, capture, represent human values in computing language. So, this is something we don’t know how to do. Maybe we could create an AI today that would want to maximize the number of digits of pi that it could calculate. So, a very simple goal like that would be within our current reach to program. But we couldn’t make an AI that would maximize justice or love or artistic beauty because these are complex human concepts that we don’t yet know how to represent.

I was arguing that it’s not a programming problem. It’s a reality problem. These concepts cannot be represented by computer code because they inevitably are a matter of different judgments made by different people. There cannot be a scientific judgment about justice. To return to theology, we can imagine divine justice. But that is because we conceive of God as all-knowing and while we may not understand divine justice, we also understand that God’s ways are beyond our understanding. Is it imaginable that we humans could create a machine that would surpass us not only in calculating ability but in moral discernment? I literally cannot imagine it. Perhaps that is Bostrom’s point. These machines will be to us, as we are to a cat. But then again, this is a nearly divine vision of super intelligence. (I worry that I may have misunderstood Bostrom here. I am hoping he will respond to clarify.)

Underlying much of the discussion was the question of values or motives or aims for a superintelligent entity. I am not sure about Bostrom but for many thinkers in this area, consciousness is just chemistry. So if, for example, we get to superintelligence through replicating the brain’s physical state, it will then be a brain with everything else that goes with a brain such as consciousness. This is not a universally held view. There is another view that says that the mind is more than its physical components. That consciousness, will, drive, and intention or more than the firing of neurons in the soup of the brain. This view is hard to accept or even imagine for some, but some really smart people believe that the mind-body problem will never have a scientific understanding rooted in the laws of physics and chemistry. (I have started Cosmos and Mind by Thomas Nagel on this topic and hope to continue reading David Chalmers on this issue).

While very interesting, put that to the side. The question I want to think about is not whether machines will have consciousness but whether we human beings will treat them as anything other than machines. I’m trying to get at the Bostrom claim that the superintelligent entities will deserve moral standing, whether we will care for them, or whether for example, they should be allowed to vote.

Consider the following thought experiment. A man’s wife dies but he does not realize it. A kindly government agency replaces her with a robot whose exterior perfectly mimics flesh and blood and through the accumulation of past data about his wife–her speech, her chemistry, her body, her neurons, her moods, whims and so on–is able to pass the Turing Spouse test. Because the machine so thoroughly understands his dead wife’s biology and chemistry, it is able to pass itself off as the wife without the husband realizing it. On his deathbed, the husband is informed that the thing he has been living with, loving, lying to, flattering, being flattered by, insulted by, arguing with, making up with, being consoled by, vacationing with, and so on, was not really his wife, but only a virtual wife programmed to emulate his wife with undetectable precision. What is his reaction? Does he think that it doesn’t matter, that because the bot perfectly emulated the wife, it was, essentially, his wife? I don’t think so, at least not in our current culture. Maybe someday our culture would view this differently. (And no, I have yet to watch the movie, Her, but I plan to. Right now, I would say that because this ad is funny rather than poignant, we have a long way to go on attributing reality to smart machines.)

Or imagine that instead of wooing a flesh and blood creature with all of her glorious traits and unavoidable human flaws and trying to get her to marry him, a man of the future will design his wife, picking the attributes that please him, perhaps even allowing for it to get annoyed with him, or to betray him romantically. Certainly he could program a random element to surprise and delight him. Would that be appealing? Do I really want my own galaxy, as Bostrom suggests will be possible in a future of superintelligence, a galaxy where I am king and millions serve me and satisfy my various whims? This is quite close to Nozick’s experience machine. It is not appealing to most of us living in 2014. Perhaps as we get more experience with smart machines, the line between human and machine will blur and much of what I have wondered about will seem normal. But right now, today, these virtual scenarios do not appeal to most of us. There is something about real life that appeals to us, despite its unpleasant messiness. That something is its realness. Why that should matter is a fascinating question for another time.