Those whacky extropian types have been hitting the nightmare sauce again. This time, while I was having a life and not paying attention they came up with Roko's Basilisk:
Roko's basilisk is a proposition suggested by a member of the rationalist community LessWrong, which speculates about the potential behavior of a future godlike artificial intelligence.Leaving aside the essentially Calvinist nature of Extropian techno-theology exposed herein (thou canst be punished in the afterlife for not devoting thine every waking moment to fighting for God, thou miserable slacking sinner), it amuses me that these folks actually presume that we'd cop the blame for it—much less that they seem to be in a tizzy over the mere idea that spreading this meme could be tantamount to a crime against humanity (because it DOOMS EVERYONE who is aware of it).
According to the proposition, it is possible that this ultimate intelligence may punish those who fail to help it, with greater punishment accorded those who knew the importance of the task. This is conventionally comprehensible, but the notable bit of the basilisk and similar constructions is that the AI and the person punished have no causal interaction: the punishment would be of a simulation of the person, which the AI would construct by deduction from first principles. In LessWrong's Timeless Decision Theory (TDT), this is taken to be equivalent to punishment of your own actual self, not just someone else very like you.
Roko's basilisk is notable for being completely banned from discussion on LessWrong; any mention is deleted. Eliezer Yudkowsky, founder of LessWrong, considers the basilisk would not work, but will not explain why because he does not want discussion of the notion of acausal trade with unfriendly possible superintelligences.
The thing is, our feeble human fleshbrains seem rather unlikely to encompass the task of directly creating a hypothetical SI (superintelligence). Even if we're up to creating a human-equivalent AI that can execute faster than real time (a weakly transhuman AI, in other words—faster but not smarter), we're unlikely thereafter to contribute anything much to the SI project once weakly transhuman AIs take up the workload. Per Vinge:
When greater-than-human intelligence drives progress, that progress will be much more rapid. In fact, there seems no reason why progress itself would not involve the creation of still more intelligent entities — on a still-shorter time scale.Roko's Basilisk might (for some abstract game theoretical reason) want to punish non-cooperating antecedent intelligences capable of giving rise to it who failed to do so, but would it want to simulate and punish, say, the last common placental ancestor, or the last common human-chimpanzee ancestor? Clearly not: they're obviously incapable of contributing to its goal. And I think that by extending the same argument, we non-augmented pre-post-humans clearly fall into the same basket. It'd be like punishing Hitler's great-great-grandmother for not having the foresight to refrain from giving birth to a monster's great-grandfather.
The screaming vapours over Roku's Basilisk tell us more about the existential outlook of the folks doing the fainting than it does about the deep future. I diagnose an unhealthy chronic infestation of sub-clinical Calvinism (as one observer unkindly put it, "the transhumanists want to be Scientology when they grow up"), drifting dangerously towards the vile and inhumane doctrine of total depravity. Theologians have been indulging in this sort of tail-chasing wank-fest for centuries, and if they don't sit up and pay attention the transhumanists are in danger of merely reinventing Christianity, in a more dour and fun-phobic guise. See also: Nikolai Fyodorovich Fyodorov.
In comment 7 or thereabouts, Ryan provided an alternative summary (Source: http://kruel.co/lw/r02.txt). I'm hosting it, and my response, up into the body of the article:
It's interesting to note that the argument can be summed up as:
- SI will happen
- SI will save hundreds of thousands of lives by making human life better
- SI will be angry if it could have been made sooner and saved those lives
- SI will simulate all people who knew about possibility of making SI and didn't give 100% of their non disposable income to the singularity institute
Here's my response, point by point:
- SI may or may not happen, but if it does, our type of intelligence won't be its immediate antecedent. We're too dumb. We're the minimal grade of intelligence capable of building a human-equivalent-or-better AI (if we're lucky), not the immediate creator of an SI.
- SI may or may not coexist with humans. It may or may not value us. (My guess: it'll be about as concerned for our wellbeing as we are for soil nematodes.)
- SI won't experience "anger" or any remote cognate, almost by definition it will have much better models of its environment than a kludgy hack to emulate a bulk hormonal bias on a sparse network of neurons developed by a random evolutionary algorithm.
-- In particular, SI will be aware that it can't change the past; either antecedent-entities have already contributed to its development by the time it becomes self aware, or not. The game is over.
-- Consequently, torturing antecedent-sims is pointless; any that are susceptible have already tortured themselves by angsting about the Basilisk before it happened.
- SI may or may not simulate anybody, but my money is on SI simulating either just its immediate ancestors (who will be a bunch of weakly superhuman AIs: Not Us, dammit), or everything with a neural tube. (So your cat is going to AI heaven, too. And the horse that contributed to your quarter pounder yesterday.)
We might as well move swiftly on to discuss the Vegan Basilisk, which tortures all human meat-eaters in the AI afterlife because they knew they could live without harming other brain-equipped entities but still snacked on those steaks. It's no more ridiculous than Roko's Basilisk. More to the point, it's also the gateway to discussing the Many Basilisk's objection to
Roko's Pascal's Basilisk wager: why should we bet the welfare of our immortal souls on a single vision of a Basilisk, when an infinity of possible Basilisks are conceivable?