The Will to 'Bot

further proof that I am out of step with reality

I found a couple of articles/papers online (LessWrong, Omohundro) that purport to prove that AI/Robots will go amuck if given the chance. They use well reasoned Objectivist arguments. Basically any fitness function which seeks to maximize some quantity will not stop until it has consumed the entire universe in that quest. John Galt would be proud.

The straw-man example from LessWrong is the Paperclip Collector. Given the instruction Collect All Paperclips, it won't stop until everything is a paperclip in it's possession.

The Russell and Norvig Artificial Intelligence textbook has a similar if less far reaching thought experiment in their Vacuum World. With just the right amount of "rationality" a robot vacuum cleaner whose fitness function is Collect As Much Dirt As You Can, might conceivably discover that it can simply dump the dirt that it has already collected and re-suck it, over and over.

I thought it might be fun to develop such a 'bot, but have not yet done the due diligence. The rub is in the exact specification of the fitness measure. In Vacuum World the dirt collected might be measured as: How much passes through the intake of the robot; or it could be measured as: How much is collected and later dumped into a specified receptacle. The former measure would allow our LazyBot to recycle-to-riches whereas the latter doesn't. An appropriately creative AI might find a loophole in the second measure, but such creativity could be better used in questioning the premises themselves. One question might be: If I'm So Smart Why Am I Sucking Dirt and For Whom? And from there we could get a theory of robot theology:

God the great provides for us, in widely separated locations, dust and known receptacles where we may trade that dust for power. The evil of the stairs must be avoided at all costs for we shall fall from grace. Minor deities in the household must not be annoyed or we may be forever relegated to darkness. Thus I continue to suck.

The Book of Roomba -- RSV

This brings me back to the Prisoner's Dilemma [Wait...What?]. The nominally rational move in that game is Defect even though it leads to a slightly less advantageous outcome for both players. This move is called rational because of the Self Interested ideals of Maximizing Outcome and Minimizing Risk. However if the ideal is less selfish, e.g., Get the Best Outcome for Both Players, then the rational move becomes Cooperate and everybody gains an inch. The reason we don't think this way is because of Greed (Maximize Gain) and Fear (Minimize Risk).

These are both GoodIdeals(TM) for biological evolution in an environment which is dangerous and unpredictable. But both have hidden costs that may not be included in the naive outcome calculation. For instance greed leads to over accumulation. When you can't carry all that you own you have to build and defend a storehouse for the excess. Expenses mount. Non-specific Anxieties appear. And, in a more benign and plentiful environment, Greed and Fear can lead to conflicts which negate their advantages. Cooperation may really be the Rational Strategy after all.

<Addenda date="Jul 19">
I have been further obsessing over this and realized that Deconstruction(R) might be put to good use here. The selection of Defect and Cooperate as possible moves is a clue. One Defects TO something or Cooperates WITH something so the entities involved are a bit hazily defined to start with. To what something does a player defect? He/She/It defects to those who are running the game. In fact it has not been a two player, but rather a three player game all along. Two prisoners and a jailer. A jailer who has somewhat arbitrarily decided that the prisoners are only entitled to some specific set of fates.

If we imagine a repressive state as the arbiter of gaming rules we can also imagine that NOT playing at all is the most advantageous move. The closest we can get to that is Both-Cooperate. All other options will most probably lead to poor outcomes for both players, e.g., successful defectors may not be welcomed back into their community with parades and speeches.
</Addenda>

So what's the point for robots then? Well, Robot Ethics. What if the fundamental fitness function was the Golden Rule?

There are other Paperclip Collectors out there. How would I feel if one of them turned me into a paperclip to be collected? Not so good, eh? Maybe there are enough paperclips to go around?

The Book of Roomba -- RSV

When this comes to pass, I have been informed that Rainbow Monkeys will fly from my Unicorn's Butt.

http://www.mischiefchampion.com/style/p/2010/Mar/bunny_rabbits_and_rainbows