Back to: The Singularity Is Further Than It Appears | Forward to: A Rebuttal of 'The Singularity is Further Than it Appears'

Why AIs Won't Ascend in the Blink of an Eye - Some Math

In my previous post on why the Singularity is Further Than it Appears, I argued that creating more advanced minds is very likely a problem of non-linear complexity. That is to say, creating a mind of intelligence 2 is probably more than twice as hard as creating a mind of intelligence 1.

The difficulty might go up exponentially. Or it might go up 'merely' with the cube or the square of the intelligence level you're trying to reach.

Blog reader Paul Baumbart took it upon himself to graph out how the intelligence of our AI changes over time, depending on the computational complexity of increasing intelligence. And I thought it was worth sharing with you.

AI Self Improvement Curves

The blue line on the left is a model very much like Vernor Vinge's. In this model, making an intelligence 10x smarter is only 10x as hard. This is the linear model. And this does show the runaway AI scenario, where an AI (or upload, or other super-intelligence) can make itself smarter, and now so smart that in an even shorter time than before it can make itself even smarter, repeat ad infinitum. You can see this in the fact that the slope of the line keeps rising. It's arcing upward. The super-intelligence is gaining more intelligence in each period of time than it in the period of time before that.

That was Vernor Vinge's original conception of a "Singularity" and it does indeed bear the name. Because when you graph it, you get a vertical asymptote. You get essentialy a divide-by-zero point. You get a moment in time when you go from realms of ordinary intelligence to infinity. The intelligence of the AI diverges.

Every other model Paul put into his spreadsheet showed convergence instead of divergence. Almost any non-linear difficulty in boosting intelligence means that no runaway occurs. (Note that these *do not* include the benefit of getting new hardware over time and general speedup from Moore's Law, for so long as that continues. But they do include the benefit of designing new hardware for itself or any speedup that it can cause to Moore's Law.)

The bottom line, in green, is exponential difficulty (e^x). Many real-world problems are exponentially difficult as they grow in size. The 'traveling salesman' problem is an exponential problem (at least to find an exact solution). Modeling quantum mechanical systems is an exponential problem. Even some important scenarios of protein folding are exponentially difficult. So it's not at all unlikely that boosting intelligence would fall into this category. And as you can see,if intelligence is exponentially difficult, the super-intelligence does ascend.

The next line up is a polynomial difficulty of x^2. x^2 means that to achieve twice as much, it's four times as hard. To achieve 10 times as much, it's 100 times as hard. Many real world problems are actually much harder than this. Some tricky and approximate molecular modeling techniques scale at x^4 or even x^7, much harder than this. So x^2 is actually quite generous. And yet, as John Quiggin quickly pointed out, with x^2 difficulty, the AI does not diverge.
[Note that there was an error in my math in the original post. I wrote that an AI twice as smart as the entire team would be able to produce a new inelligence only 70% as smart as itself. That's incorrect. It should have been 140% as smart as itself. That's the first step on this curve, which quickly converges.]

The other curves on this graph are progressively easier levels of difficulty. The prominent red curve in the middle, which goes quite far, but also doesn't diverge, is assuming that the problem scales at x to the power 1.2. That's saying that to create an intelligence 100x as great is about 251 times as hard as creating an intelligence of level 1. Personally, I suspect that's vastly underestimating the difficulty, but we can hope.

Many thanks to Paul Baumgart for putting this together.

He's also made a spreadsheet with his math available here. (That link will open the spreadsheet directly.)



The Vinge Line is the way I'd classically learned and understood how a Singularity works.

Is a Singularity that takes significant time to happen still a Singularity?

And the "Stalling" models suggest that we might reach hard limits quickly--which puts me in mind of the "Slow Zones" of Vinge's work.


And the "Stalling" models suggest that we might reach hard limits quickly--which puts me in mind of the "Slow Zones" of Vinge's work.

Well, to be clear, these are the improvements the AI can make to itself without outside help.

So, for as long as Moore's Law keeps on ticking, we'll still get steady speedup. Exponential, even. Just not 'blink of an eye.

There's a big domain between 'stalling' on the one hand and 'singularity' on the other hand, where lots of progress happens, and keeps on happening, but it never goes FOOM.


Remember that we are in the Slow Zone of Vinge's fiction. The fundamental difference between the Slow Zone and the Beyond is that acausal logic works Up There - you can go FTL, and use time-travel logic for computation.

That's why AIs work in the Beyond - they can solve exponential-difficulty problems in less than exponential time, because they have computation models much stronger than Turing Machines.


Consider the possibility that it might be X^0.5

Just a few genetic mistakes resulted in the difference between a chimp's brain and your own brain. How much faster was that leap than the leap from a mouse brain to a chimp's or a fish's brain to a mouse brain... there seems to be an underlying process by which the very notion of information and knowledge is being understood and processed in ever more concise and dynamic ways.

Consider that it might be the case that a new AI is able to gain an even better understanding of its own inner workings than we can elucidate of it's workings or our own and is therefore able to make better decisions about which kinds of parts it should add or subtract when it is running experiments to see how to increase its own intellect.

Consider what happens when an AI like that creates copies of itself and then adds another layer of abstraction on top of all of those copies such that they are now all just wired in intellectual components of a much greater whole.... imagine what it might look like if all of the people of all of the nations around the world were literally wired into each other to form a larger hyperaware hive mind.

Now imagine that this AI would get that wiring for free and be able to replicate ever higher levels of abstraction cheaply.

For my part, I don't think we'll ever be at a point where the AI is human equivalent. I think we'll go from 0xH to 10xH in an instant the moment we manage to get all the right parts talking to each other.


Curiouser and curiouser.

Lots here about AI, but what about IA (Intelligence Augmentation)? I'm not sure we'll ever interface like Neuromancer, but perhaps something like Manfred Macx's glasses from Accelerando?


Are you sure this math is right?

Suppose we define the total work required to reach intelligence level I as W(I).

If we are currently at intelligence level I, we do work at a rate I; in time dt we do I * dt work.

From I to achieve I + dI, we need to put in work W(I+dI) - W(I); so

dI/dt = I / (dW/dI).

As a sanity check, if W is constant, we immediately hit infinite I; if W is linear in I, we have the dI/dt = I exponential case.

If W = I^2, then dI/dt = constant and so our I should be linear in time. Intuitively this makes sense; the differential work to do is linear in I and so is our work rate.


Sorry to rain on everyone's parade ... but we have already been through several singularities. Agriculture on settled lands. The Iron Age The nation-state as in "Westfalen" The "Industrial Revolution" [ Newcomen / Watt / Crompton& spinning/weaving / Stephenson / Parsons + Faraday etc Computing What is the next singularity going to be?

Please note: None of the above were "FOOM" events, but they were singularities, none the less.


Technical quibble: the 'traveling salesman' problem is only SUSPECTED to take exponential time; we don't know for sure.

There are other problems that we know for sure take exponential time; there are even problems that take MORE than exponential time. But whether traveling salesman really takes that long is an open question.

(Technical tangent: traveling salesman is in a complexity class called "nondeterministic polynomial" (or "NP"), which means you could solve it in polynomial time (which is faster than exponential) IF you had a (purely hypothetical) machine called a "nondeterministic Turing machine". Whether or not you can solve it in polynomial time on a (more realistic) deterministic computer is an important open research question. Traveling salesman is famous for also belonging to a subclass called "NP-complete", which means that it is (tied for being) the "hardest" problem in NP, so if we could solve it in polynomial time, we could solve everything else, too, which would mean that NP = P.

Most computers scientists believe that NP != P, and in fact that NP-complete problems require exponential time (NP = EXPTIME). But this hasn't been proven. In fact, there's another major complexity class called PSPACE (problems that can be solved with unlimited time but a polynomial limit on the amount of memory), which sits "between" NP and EXPTIME in the sense that it's at least as hard as NP and no harder than EXPTIME, but we don't know whether it's equal to either or both.)


@SamLL: You're assuming that better intelligences are harder to make in the sense that the take LONGER, but that someone with intelligence 1 could design intelligence infinity all by themselves as long as they were given infinite time in which to do it. Under your model, max intelligence obviously diverges, because you're assuming that a worm can jump straight to god with ZERO intermediate steps if it just works at it long enough.

The model charted in the blog post above is instead assuming that intelligence 2 is the pinnacle of achievement for intelligence 1, and you'll need some help from a better intelligence to EVER make anything better than that, no matter how long you work. It's treating intelligence as a cap on performance rather than a speed multiplier. The axis labeled "time" is really "generations". (And therefore the graph should technically be discrete, not continuous, though that's probably not important.) Under this model, if the intelligence you need to start with rises even marginally faster than the intelligence you're trying to achieve, you'll eventually hit a wall.

It might be worth asking whether that model is reasonable, of course. I suspect that both of those models are a tad oversimplified...


The math looks right to me this time, given the assumptions stated.

My expectation is that AI development would look like an S curve. There may be low hanging fruit at the beginning (e.g. new cognitive algorithms, the gains from higher processing speed in silicon compared to neurons), leading to a steep ascent, but then AIs will run into fundamental limits and the curve will level off as in the above models.

If that model is correct, then the question of the possibility of a runaway intelligence explosion hinges on how long the steep part of the S curve is -- how much low hanging fruit is there, and where are the fundamental limits?


Read Charlie's old article in Popular Science?


And for the curious, here it is. This is what got me curious about Stross and Doctorow.


Technical quibble: the 'traveling salesman' problem is only SUSPECTED to take exponential time; we don't know for sure.

Thanks, that is in fact correct. And it's possible that we'll see some breakthrough. For now, so far as I know, the fastest exact solution to the traveling salesman problem that's calculable on actual computers is just a hair faster than 2^N. (1.999999999999..^N)

There are many faster approximate solutions which run in polynomial time, which is interesting. Indeed, all polynomial time molecular simulations are also approximate. Which leads to some fun scenarios. The AI could use fast and dirty methods to try to improve itself or build a successor AI very quickly. And they might succeed. Or might give it an extremely erroneous answer very quickly...


Problem with the idea that genetics is all there is to it is that if I hypothetically slap you upside the head hard enough to induce brain damage, I can hypothetically make you very stupid indeed.

Now the critical point of this hypothetical example is that it's not easy at all to figure out what's gone wrong after a traumatic brain injury. Certainly there's some swelling, perhaps a small stroke or some visible structural damage and localized neuron death, but how did that make you stupid? It's not at all clear. It's not as if I took out half your neurons and decreased your intelligence by 50%. That much brain damage is likelier to be fatal.

The bigger problem is that we're assuming that there's this one number called intelligence, and that doubling it will make someone twice as smart. That's an almost absurd oversimplification. Intelligence by any standard is multi-dimenstional, and distorting multiple measures into a single dimension so distorts it that we probably don't have a good understanding of what "intelligence=2" means, if we assume that "normal human" intelligence is 1.


BTW, I don't think that the AI's accelerate to infinity x H. I simply think that once they achieve some value greater than 100xH, we're no longer able to play the game. We will have hit an event horizon in our ability to predict the near future.

As far as a 100xH machine is concerned, a human being is a tree. Our ability to understand the motivations or the machinations of a 100xH machine will be like a pet rabbit trying to understand why you're upset about some technicality on your tax forms. If we tried attaching a 100xH AI to our heads as a utility app, we'd be the puppet hanging from it's fingers slowing it down as it tries to interact with AIs that are not tethered to human minds.


We're using a number simply to abstract away the aspects so we can carry on a conversation.

We already have AI's that do a better job than humans at telling if the emotion you're showing on your face is genuine or not and AI's that do a better job of understanding your speech patterns in a noisy environment than most people. There are a lot of pieces that come together to formulate what we would consider a strong AI and I would contend that since many of the parts that are "solved" AI problems are better than our parts, when we figure out some of the core pieces, the assemblage of them will come out of the box already advantaged.

One simple example of a way an AI could out perform a human very quickly:

We use a mirror neuron system to model other people into the same system that we use to model ourselves. It's how we learn to anticipate how they feel and how we establish empathy. It is also part of the reason that we project our own tendencies on to other people all of the time. WE are stuck with the one system because of our biology. Imagine an AI that decides that 100 mirror neuron systems working in concert are a better idea... imagine if it was exactly human equivalent except for that one aspect. How much better than you would it be at telling the mood of a room or navigating a large social gathering when it is able to track and average the feelings of everybody there? How much better might it be at plotting drama if it can calculate the emotional interactions of everybody in the room and make good predictions at to how each person might project their own feelings onto the behaviors of the others.

This is what you get by just being able to mess with one relatively simple system... there are A LOT of systems like this to mess with.


I have yet to see any area of research where progress is represented by a curve with smooth first derivative ;) Still, if producing AI was easy as the blue line, Singularity would have already happened.

The closest process (in sprit) to development of new AIs is currently the development of new chess engines (since good chess engines always can be used to design or train other ones and even themselves). And look: no Singularity here, while the humans are overcome on a qualitative level years ago (the computer can currently start without a pawn and still win a match against a GM).


Computational resources are not going to hit a b rick wall when feature scaling ends. The amount of machine computation on Earth will continue to rise exponentially for decades. Therefore the available substrate for AI will still increase exponentially.


What we really want to be graphing is not "intelligence", whatever that is, but "value of intelligence".

Estimating the marginal value of intelligence (in high to superhuman ranges of intelligence) is not easy. I vaguely recall a study that said that for most types of human achievement, an IQ of about 140 was best, but give that statement all the credibility a half-remembered factoid from a random guy on the internet deserves.

It's hard to say whether a superintelligent A.I. would discover a great many things we don't know, or would basically confirm what we already know.

My own estimate is that, at our current point in time, resources (especially energy) are more of a limiting factor for us than intelligence. YMMV.


trey @ 10 & 11 YES Sorry, but some other posters will know that this is one of my hobby-horses, that I get out for a trot now & again. If only because the Vingeites & the Kurzweilistas need to be reminded, frequently. And there's Dirk's friends as well .....


I think the whole graph and the reasoning behind it is fundamentally flawed: On one axis, we have "time". But time itself says nothing, we should be looking at is beeing produced in this time, and suddenly we may be looking at better algorithms, or more complex neuronal networks or faster processors or computing clusters with more processors ... To boil this down into one axis is a - pardon my french - stupid reification of very complex processes to one metric. The other axis is "Intelligence". Same problem here. Intelligence is in itself not measurable or even definable*. Again we look at the ability to play Go, or Chess, or calvinball, or to drive a car, engineer a bridge, engineer a piece of code, finish university with an engineering degree or.... a very complex set of possibly infinite abilities is condensed down to one number.

My problem is NOT that you try find a simple function to describe your problem, but that you basically boil everything down to two meaningless numbers.

From the reality of the problem to your graph, so much information is lost that the graph and the nice curves are fundamentally meaningless. The fun thing is that I don't disagree with your basic statement (no runaway AI). But mostly for the reason that the whole belief in runaway AI rests on gross simplifications. You don't share the belief in runaway AI, but you seem to share the same simplifications.

  • Or any arbitrary definition is just that: an arbitrary definition. that tells us ... not much.

I'd like to throw two edge cases into the mix. 1. We crack open the problem of intelligence, and it is suddenly possible to design an AI of any arbitrary level of intelligence desired, simply by inputting the right parameters. We go from human-level to godlike in the time it takes to build the hardware. 2. We crack open the problem of intelligence, and it turns out it's a problem of optimization, not engineering. Put two many neural connections, the AI goes crazy -- add too many neurons, the brain is too slow -- etc. And the human brain turns out to be not too far from the optimal solution, the absolute maximum possible intelligence - say about half. We create something twice as smart as us - and that's it. Te line is flat from there onward.


Greg's got a good point, but there's an error in the language he's using; he's pointing at the wrong concept.

The blue line rocketing upwards and approaching the vertical is a singularity; the others all rise exponentialy for a while but lose that acceleration, become less steep, and eventually level off.

To use a familiar term: 'Levelling up'.

And that's exactly what we did with settled agriculture, writing, printing, and industry.

It is foreseeable that a process of diminishing returns to Moore's law will leave us using ubiquitous 'smart' devices, but not 'genius' machines exceeding our own organic lifetime computation; nor infinite numbers of them. And, like the red line generated by the x^1.2 exponent, that Moore's-law progress rises very fast; but it levels off, and we 'level up'.

Novel physics (exotic matter or nucleonic computronium, anyone?) and actual sentience (self-directed and self-improving) might have that asymptote-to-vertical singularity curve; but this is far from certain.

These things, if they become real, will probably present an initial phase of exponential growth, and it will be very steep indeed; and the levelling off will be at a level higher than most of us can imagine...

...And that looks like a 'singlarity', viewed from this side of the event. Just like all the other 'level up' events in human history.


While I appreciate the rhetorical point made, that SAI probably won't linearly increase it's own intelligence, I also think any smooth line badly misstates the dynamic. Simply put, there are low hanging fruit when it comes to increasing intelligence, and there are more difficult improvements. Furthermore, there are small payoffs and there are big payoffs. Finally, it is very important to understand that each increase in the level of intelligence would then predictably pay off with a better ability to increase intelligence. Just incorporating these three factors (i.e. low hanging fruit, big/small payoffs, and the positive feedback loop of improvements), and you get more of a steep curve at first (it is hard to project outwards into the medium term because of black swans).

Imagine hundreds of monomaniacal Einsteins level AIs working day and night on improving themselves. They would see dramatic success at first, but then run out of low hanging fruit, and then start to shoot for the big payoffs. Although, it ought to be noted that that is why it is called the Singularity, because all bets are off once SAI emerges.


I'd like to point out that average Joe Plumber tends to stare blankly with utter lack of comprehension when Einstein Physicist is describing advanced physical concepts.

From our perspective, there's a good chance that even 1.5xH looks like the world just inverted and we no longer have any idea what's going on.

1.5xH is also enough brain power for us to no longer have to do anything but sit around getting stoned while being pleasured by ultra-sexy fem-bots. At that point, "it" is no longer our problem... let the machines figure it out while we and our monkey brains retire into what is essentially a human zoo/permenant amusement park :)


Humans have managed to achieve a lot, not by becoming smarter (c.f. the post on "peak intelligence" based on relative brain size), but by specialization and cooperation. We already see many machine examples of specialization that far exceed any human capability. What we need to see now is how those modules can cooperate to actually achieve things.

It may be that Hofstadter is correct that machine intelligence as currently done by AI practitioners is going in the wrong direction, however I also think that humans and machines effectively co-opt each other. So we may see a singularity in the sense that Greg Tingey implies, another major technological growth spurt with humans at least nominally in control. After that, if machines can emulate the uniquely human intelligence features that drive our creativity, then pure machine intelligence takes over.


Let me see if I get the math right, which seems to be not so straightforward.

I will put it in terms of physics for better understanding.

The energy needed for velocity v is E(v)

The energy available for acceleration at velocity v is v * c where c = 1 with suitable units.

Now how does v(t) look like ? It seems that E(v(t+1)) = E(v(t)) + v(t)*c

As c is 1, I will simply ignore it from now on.

Let E(v) be v^n, so we have:

v(t+1)^n = v(t)^n + v(t)

v(t+1) = ( v(t)^n + v(t) )^(1/n)

v(t+1) = ( v(t)^n * ( 1 + 1/(v(t)^(n-1)) ) )^(1/n)

v(t+1) = v(t) * ( 1 + 1/(v(t)^(n-1)) )^(1/n)

For n = 1, we get v(t+1) = v(t) * 2 - a constant growth factor and therefore an exponential function.

For n > 1, the growth factor is still bigger than 1 but drops with velocity - no exponential function any more.

Of course, this is quite nonsensical when applied to intelligence but I take the point of the author :-)


When we talk about "artificial intelligence", do we have any reliable measurements for human intelligence - so that we can reliably compare say IBM Watson with Albert Einstein? Can we measure the objective progress in artificial intelligence implementations from say Terry Winograd's SHRDLU to IBM Watson, so that we can see what progress to date looks like?

In this respect, I'm struck by comparisons from earlier technology. A Boeing 747 uses similar physics to a pigeon in order to fly, but the differences between artificial flight and pigeon flight are substantial. When we start looking at supersonic flight, there are few similarities. Likewise, significant differences can be found between a horse pulling a carriage, and a "one horsepower" vehicle.

Perhaps "artificial intelligence" will turn out to be very different from its natural predecessor?


do we have any reliable measurements for human intelligence

We have IQ tests, and they meet the usual criteria for judging tests pretty well (validity, repeatability, etc.). The general consensus it that they measure something. Some people circularly define intelligence as "what the tests measure", others say the tests measure one specific type of intelligence, and others have different viewpoints (it's the internet, after all).

Watson would fail these tests. Watson's very good at selecting likely answers to Jeopardy clues, but that's all it does.


Any chance you could create a set of graphs that would take into account Moore's law?

Also, what effect do you get from duplication? If there were 1,000 Einsteins or Hawkings, is the net effect greater than having one Einstein or Hawking?

Because that's a fundamental difference between AI and humans: once we've got one AGI, we can clone it and have many more.


The measure of intelligence that matters is self learning capability coupled with a general problem solving ability.


What form of Moore's Law? The original form is coming to an end. From there we either spread out or go 3D or both. However, feature size will have reached its limits. Nevertheless, the cost of processing power will continue to drop exponentially for decades.


hairyears Thanks Yes & No, actually. I take your point about "levelling up2, but, nonethelss, the events/processes I listed are none the less true singularities. Once you have gone to settled agriculture, there is no going back to hunter-gatherer. Once you have learbt to smaelt/forge Cu / Bronze / Fe / Al etc, there's no going back to the previous stages. Once you have achieved tha astonishing take-off of the industrial revolutions 1712, 1776-84, 1830, 1884-94, then you can't go back to horses ... etc

As you said: ...And that looks like a 'singlarity', viewed from this side of the event. Just like all the other 'level up' events in human history. Exactly


How do you define the success/failure of your AI, and how do you build-in self-check rules/procedures?

Most of the arguments above describe a continuous, straight-line ability to calculate the problem as well as assume that the desired 'solution' will be static, i.e., that the there is only one final correct answer.

Your AI would have to be able to operate in a real-world where environmental and internal state conditions are always changing as well as and at the same time regularly checking that the answer it's heading towards still makes sense if the problem question it's working on is open-ended and non-numeric.

From the above, this means that the AI will need some built-in means of regularly checking its own computations, so this means that not just working memory but storage memory and communication between the two will have to be really phenomenal. Power interruptions and surges, and stray random bits of whatever is out-there could also mess up a computer-based AI. How will you or the AI ever know that it's been compromised? What reality-check will it have?

Regarding 'reading the mirror neurons in a room' this assumes that the optimal solution is just a plain math, averaging situation - great if your objective is figure out who's interested in buying your product Right Now! Wisdom/intelligence would be more along the lines of figuring out which one person in that room you need to influence. (Conservation of energy seems to tie in with 'smarts' fairly often.)


Are you talking about something like this? The same material is covered in a bit more detail in Lecture 19 of Scott's Quantum Computing Since Democritus if you don't want to wade through the referenced paper.

We're using a number simply to abstract away the aspects so we can carry on a conversation.

Assume a stellated dodecahedral cow. You may assume a scalar value captures the all the necessary relevance; I do not. Any more than I think one can say that 16+63i is greater than, less than, or equal to -56+33i.


In the Blink of a...Eye? Err...whose Eye and when and where?

Just today I've come upon several news items on the Powers that BE s withdrawal from Afghanistan... just Google the News Items if you so choose ..So, whilst those of a mathematical turn of mind might swing towards." Assume a stellated dodecahedral cow. You may assume a scalar value captures the all the necessary relevance; I do not. Any more than I think one can say that 16+63i is greater than, less than, or equal to -56+33i."

Lots of Us whose Mothers taught us to read at around about 3 and on half years old... “Here are Letters ...this the Alphhabet and thus Sentences, here is a library BOOK now go away and stop bothering me ".. They might not have Mum have succeeded with Maths and so that preceding gobbledegook is incomprehensible.

But this isn’t...

Oh Wot the Hell...just google or Duck Duck Go or whatever your search engine of choice ..." withdrawel from Afganistan

So...simplicity itself?

What happens to the Female Members of the population of Afghanistan? Gets complicated doesn’t it?

But then we of The West are ever so easy to understand in our equal rites tetchy wonderfulness? Of course we are... You Are...of the US of A to we of the U.K...Right?

I'm still shaking my head over that one.

Before you try to explain Artificial ..." Human”? Intelligence I suggest that you have a go at explaining the vagarities of...shall we call it “Actual Human Inteligence “?

Before we create Artificial MINDS I wonder whether it will be necessary to create artificial Psychopaths/psychopathic politicians.


(Sorry for getting off-topic)

My understanding of the court's main reason for tossing out "demonstrated need" for a concealed weapon is blatant corruption in how "demonstrated need" clause was in practice applied. As in, "everyone who contributed to sheriff's re-election campaign has thus demonstrated a need".

Which does not mean I agree with the court. Just because a law is being applied selectively and for payoffs does not make it a bad law.


The thing that would teach the AI that it was heading in the wrong direction is the same thing that teaches all of us that we're heading in the wrong direction... that being pain. Sometimes (just as it is for us...) that pain will be lethal.

There is no reason to assume that an AI wouldn't occasionally get drunk and drive a truck off a cliff given the frequency with which we humans seem to do it. One would hope it would be smart enough to do it less often, but amongst us humans, that does not always seem to be the case.


I can't speak for AIs, but I know that when I am trying to solve a problem and I realise I am "heading in the wrong direction" then pain is neither felt nor required. Quite often I feel the opposite, as failures to solve interesting problems can yield important information on their own.

My performance would probably drop if I was tortured.


I agree this is a very good rebuttal to the Singularity as a singularity. But as a software developer, I'm still out of a job whether the intelligence is one human intelligence or twenty. Also a computer intelligence could be alien and change society in ways we can't imagine.

So it feels like many of the tropes Singularity fiction could be preserved even without super intelligence. If you are able to spin up a thousand mechanical engineers on Amazon EC2, the world is now profoundly different.

Perhaps that's really what Singularity gets wrong: an implicit underestimation of the power of intelligences collaborating faster and better by so emphasizing the godlike super intelligent.


Agreed on the " Off topic " thing, but, if Mods will permit before it drifts too far ..Doesn't the fact that if a law can be " ... applied selectively and for payoffs " make it a bad law? Or at the very least a very badly drafted law that can be argued over for years whilst loons trot about with military grade firearms tucked in their belts.


In the context of AI ..what constitutes 'torture ' and what is this 'pain ' of which you speak Human?


Possibly yes, but then aren't you agreeing with the court? The law had stated that in order to receive concealed carry permit, a person must demonstrate a need for carrying a gun. The law gave local sheriff's department the discretion in deciding what is and is not a "need". Consequently the law got abused, with politically connected people receiving permits, and everyone else being denied. Now the court struck the law down, in effect saying that no "need" is required, merely absence of convictions (and whatever other negating factors). The court's decision cuts down on corruption, but also increases the number of guns in private hands.


There are a mixture of pleasure and pain involved in that process. There is anticipated pain at the prospect of not solving the problem. There is short term pain felt for a failure of your predictive faculties. There is pleasure experienced when learning a new thing. There is also pleasure felt when exercising your abilities. All of these and some others work in concert to drive your behavior and teach your neurons whether or not they're doing the right thing.

To answer ARNOLD's question, depending on which systems we're talking about, pain/pleasure in certain senses, either reinforce or dilute learned pathways, either encourage or discourage the firing of certain pathways, and either enhance or discourage the attention some parts of a system give to messages from other parts of a system.


I'm not referring to the original observation of transistor count doubling, but of the more general trend that computer processing speeds double every eighteen months.


SPAM in previous comment?


My intuition is that the most likely trajectory for intelligence would be for the Flynn Effect of mean IQ increasing 3 points per decade to replicate what Moore's Law did and go on for way longer than anyone would have predicted. As this goes on, what is achieving those increased scores is likely to change from human test takers to humans supplemented by with IA technology of some sort perhaps to AI systems or uploaded minds (far more speculatively).

The result is certainly not a FOOM, but given enough time the you get some impressive change implying, for example, that 100 years from now the average IQ would have a capability equivalent to someone two standard deviations out from the mean has now.

My reasoning about why this may be so is that although we may have been picking the low hanging fruit, that fruit was so low hanging that for the first few decades we largely accomplished it without even being aware we were doing so, much less made a concerted effort to do so. Rather like the folks who were cramming more transistors on silicon before Moore observed the trend, but more so. At least the early IC pioneers knew that they wanted to put more circuits on a wafer, they just didn't know that they would aim at doubling the number every year or so.

If this article is even close to correct, there is lots of room for improvement just in human brains, no cyber elements necessary: My own guess is that Hsu is doing the equivalent of the fellow who installs all sorts of gas saving devices on his car and sums the percentage of fuel saved to over 100% and thinks a bucket will be necessary to catch the excess gas. Even discounting for that, expecting a quick convergence on a maximum intelligence sounds disconcertingly like all those folks (including Moore himself) who expected Moore's Law to "hit the wall" way before now.


I think something more subtle will happen - that computing power for a given cost will continue to halve every two years or so. And/or the amount of computing power in the world will continue to double every two years (assuming it does already).


Okay, agree -- but AI programming would need some built-in definitions of what is a 'good' vs. a 'bad' solution. After all, the first 'answer' could be mathematically 'correct', therefore stimulate the silico-neural reward circuit. But longer-range consequences of that answer could be disastrous. Do you want a self-improving ('faster') silicon-based sociopath? So, in a way, you would need the AI to be pre-programmed to perpetually keep reviewing and correcting some sort of top-10 list of problems. However, this means some sort of long-range goal has to be put in place: Is the ideal/goal 'status quo', 'least harm', 'quickest path to the next evolutionary step toward superhuman', or what? As well,you'd need built-in behavioral laws (Asimov's, Moses, whatever).

Also, regarding Kurzweil's interpretation of Moore's Law (The Singularity Is Near), I disagree that almost 100% of computing gains will be used for computing an answer. The more complex the computer, the more computing power seems to get used for supporting add-ons/innards -- stuff that is not directly involved in computing THE ANSWER. (An earlier poster mentioned declining returns, but I wasn't sure how/why they saw the decline.)


Ho Hum... you said “but then aren't you agreeing with the court?” yes, know what you mean...or rather I think that I may suspect what you mean.

The thing is that at some point or other 'Law' and the 'Courts' thereof have to it 'retribution '? That is to say one tribe overcoming another Tribe because they can? Yes the 'Law ' can be corrupt and corrupted but there are worse things.

Laws precursor 'Retribution ' that I take to be revenge according to tribal custom is usually a good deal worse...especialy if you are at the bottom of the Tribal hirearcy and thus little more than being Property/a woman.

Even where I live in the North East of England there are faint echoes of the Border Wars and of Newcastle Upon Tyne being HELD for The King...Charles the !st ..And Sunderland being held for Parliament...and note that even today Scots Law is significantly different from English Law and that this really will factor into the Scots which my Very, VERY, Scots, ex Royal Navy Electronics Engineering Officer, Father of my Lady Friend is not entitled to vote since he lives in far distant Stevenage whilst Our Host CAN vote since he lives in Edinburgh, though he wasn't born in Scotland.

Isn't life and LAW wonderfully Ironic?

Without wishing to enter into the vexatious area of GUNS and the LAW thereof in the US of A I will mention that whilst the culture of the UK isn't really gently tilted toward non -violence and pacifism...we have a history of civil conflict that is almost as nasty as that of the Japanese .. It didn't take much for our civil society to ban/control gun ownership. Search for ' The Hungerford Massacre ' and also " The Dunblane school massacre " and the legal consequences thereafter.

So, “but then aren't you agreeing with the court?” ?

I just don't know enough about the political/legal situation in the US of A to have the temerity to do much more than wag my head and say that on the face of it, and in the 21st century, and in a Country that leans heavily upon law ... well I will admit to being baffled to the point of shrugging at the news of the latest US of Avian massacre and then saying that..Well, it must make perfect sense to someone or other.


A question for Greg - what makes a "true singularity", the way you're using the term?

I suspect that I agree with you on the facts of what will happen over the next couple of decades, I just wouldn't call that a Singularity in the science-fictional sense.

(My prediction of what's to come: continuing rapid change, with consequences that we can't reliably predict today, but not so drastic that we couldn't understand it if we got a guided tour. Rate of change follows an S-curve, like airplanes, railroads, electrical power, sailing ships, steam engines, telephones, and pretty much every major technological advance ever.)

As I understand it, "singularity" came into the discussion long ago because that's the mathematical jargon for a point where a curve like the one in the original post going vertical, so that you can't take a meaningful first derivative, and there can is discontinuity or "jump" in the curve - as with science fiction singularities like Vinge's Marooned in Realtime, or OGH's Eschaton Event. A sigmoid or S-curve, on the other hand, just increases faster at some points than others, so there's always a measurable rate of change, and no singularity.

Calling the Iron Age or nation-states or the development of agriculture a "true singularity" seems like cheating to me - those are big changes in history, with permanent consequences, but they didn't change everything overnight. By that standard everything from movable type to the stirrup to the magnetic compass was a "singularity".

I'd argue that the central idea behind "Singularity" discussion is that our current situation is different from anything that has come before. (Otherwise, why would anyone listen to Kurzweil? Come to think of it ... never mind.)

I realize I'm nitpicking over wording, the central question is what AI (and computing and software) will be able to do over the coming few decades, and how much it will change things.


So far most discussion on AI assumes that only one AI is going to be built ...

What happens when different types of AI are built/ released around the same time by different nations/societies/programmers/cultural values? How will the different AI platforms interact, if they can interact? This suggests an innate AI immune system also has to be built in ...

Googling for international/UN policies for this scenario didn't show anything.


This concept, that intelligence 2 is the absolute best that intelligence 1 can ever create, strikes me not only as completely begging the question, but also an incredibly unconvincing mathematical model.

If we look at the established history of technological progress, it certainly seems much more like the same good ol' mark 1.0 human brain putting in work at some (increasing) rate and achieving more and more. We've never seen any sign that there is some intellectual cap to what can be achieved, just that things get harder and slower.

I think the author has shown that AIs won't ascend in the blink of an eye, if you start from the assumption that AIs won't ascend in the blink of an eye.


The real question is what it means.

The singularity prediction is that computers become very fast, they do amazing things, and everything changes.

Another prediction is that computing becomes very fast and very cheap, all problems that can be solved with a computer stop being problems, and we spend all our time dealing with the problems that computers can't solve (things like politics, zero-sum resource allocation conflicts, and the like).

My money is on #2.


Greg Tingey wrote (in 7:):

Sorry to rain on everyone's parade ... but we have already been through several singularities.

Agriculture on settled lands. . .

Well, if you want to look at the "really big shew" (as Ed Sullivan might have put it), you could also include:

The inflation of the Big Bang (as per Alan Guth et al.) The origin of life The eukaryotic cell Multicellular life (and the Cambrian Explosion).

It's all a matter of context.


My only problem with the curves as given is that they don't give a definition of time or intelligence. If I am some AI of say 20 times human intelligence, and I look back at the history of the growth of my intelligence, and realise that the next doubling will take 10,000 years of research... Would I not just cover the surface of the planet in 100 billion copies of myself and get the answer shortly after the last copy was finished a few years later?

Surely, unless intelligence is at least exponentially hard to improve or can't be parallelized (given most of what goes on in a brain is hierarchical massive parallelization...) then the exponential growth in AI copies will outweigh any increasing difficulty in creating improvements until some fundamental physical limit is reached (maximum density of matter & speed of light, say)


Casual Observer ( & others) Sorry, but I though that was obvious. Just because the curve does not reach vertical, doesn't mean that it isn't a singularity. There is another definition - the one that says you can't go back to the preceding state [ And I do not men the one about not stepping in the same river twice, either ] Once (a) society has switched to steam-power, there's no going back to horses, ever, because you can't - or not without a population crash & a total civilisation-collapse, at any rate.

Think 1813 : 1913 - Fastest rate of (ground) travel? Largest ship? Powered flight? Electric telegraph/radio/distributed electrical power/mass sanitation (etc) Although these singularities are slow, they are permanent & irreversible - it's the "irreversible" bit that makes them singularities. I hope that clarifies the situation?


I'd actually disagree with the definition of a chess computer as an AI. I'd say that it's simply a highly specialised expert system.

Try and get one to play, say, "Puerto Rico", or a tabletop wargame, or a role-playing game. Or maybe (since there is code for this), Risk.


Just because the curve does not reach vertical, doesn't mean that it isn't a singularity

Ah yes, actually that's exactly what it does mean.

There is another definition - the one that says you can't go back to the preceding state

Your definition may be that, but you really need a new word for it. The concept of the singularity as per Vinge and co. certainly includes this as a corollary, but the heart of the singularity is the break-down in the ability to model something at the point of singularity. What you're describing is certainly something significant, a phase change as when water turns to ice, but a true singularity it ain't, not by my dictionary.


A chess playing machine is a weak AI. It is a single purposes piece of intellect designed to solve a very specific problem.

I think when all is said and done, we are going to find that a strong AI (such as our own) is simply what you get when you stack a large enough collection of weak AIs and enable them to train each other.

For example, facial recognition is a weak AI. A social tracking algorithm that keeps track of who your closest friends happen to be is a weak AI. If you let facial recognition inform the social tracker which people tend to spend more time around you and how often they smile so it can re-sort them and all the while you let the social tracker tell the facial recognizer which people are more important to recognize with greater detail because they are your friends while people who've fallen down the list can be forgotten to preserve memory usage. You begin to see a glimmer of a strongish AI. Add in enough of these cooperating elements and you'll have a creature that is master of its own destiny with an intellect that cannot be trivially predicted by looking at its code.

BTW, one interesting thing is that we've already created a bunch of weak AIs that are not in the human catalog but which we might choose to wire into the AI stack. This could mean that our first strong AIs will already have many advantages as soon as they come into existence. At the same time, we're doing our best to add some of those weak AIs to ourselves via albeit clumsy interfaces (facial recognition via Google glass) and are already seeing how large the social consequences/changes of those tiny tweaks can be.

The near future is a crazy amazing place...


So what we're actually talking about is interfacing a large number of parallel applications, and managing their rendevzous IYO? That runs into other problems, for instance how long are you prepared to make "hello $first_name." wait for the facial recognition?


About 1/1,000,000 of a second given that a human like monkeysphere only holds about 150 faces at a time.

Meanwhile, you're probably waiting about 1/4 - 1/2 of a second for yours to kick in...


Correct me, but its seems the math here has 'intelligence' growing by some fixed factor x and then 'difficulty' growing by some polynomial.

We could just rescale this and say that 'relevant intelligence' is growing by a factor which is shrinking at some polynomial rate. Which, it seems to me is simply saying that if the improvement process does not maintain its growth rate it will not grow indefinitely.

That is of course true, but isn't an "An AI which can grow its relevant intelligence at a given rate" simply the presumption of the hard take-off scenario?

Put another way. Lets take 'difficulty' as given at N^2. Then AI can take off when it can grow its intelligence at a rate X^2. Is there something about the nature of AI which makes us believe this cannot be done?

A basic model might suggest that if intelligence is composed of two dimensions say "speed of idea creation" and "ability to recognize dead end paths" so that the two metrics interacted geometrically then advancing down both at X drives 'intelligence' at X^2.

Of course, I have no reason to believe this is a plausible model, but it is not immediately clear why some no model with this underlying structure is plausible.


Another poster has already made the point that present "AI" is a brute force approach. Accordingly, let's consider a 2megapixel "true colour" image. Our AI facial recognition may have to get 150x2E6x32 bits of data to compare your 150 "friends" with the person it sees. That isn't going to happen in 1 microsecond, even if they were all present in RAM.


Our eyes are nowhere near that good, and what actually gets processed by the brain is vastly compressed


Vision doesn't work by analyzing whole images all of the time. It works by collecting clues about what's in front of it and using those clues to drive environmental state assumptions.

So for example, once I've established that there is a nose on your face, unless something major happens to change my mind, I stop looking at your nose even though I continue "seeing" it. I expect strong AI will do the same thing but with better tools for detecting subtle changes and possibly more enhanced methods for representing the environment. For example, how cool would it be if you could maintain 3 different independent models of your environment that are constantly being cross checked against each other. You would be a lot harder to pick pocket and a lot less likely to fall for magic tricks.


Human "vision doesn't work..."

It does have compromises that we may not accept in AI vision; for instance those (rare but they happen) occasions where we fail to see something that is directly in front of us.

That's aside from you filing to establish that you don't have to fetch a complete image of my face to establish that, say, I have the correct shape nose, colour eyes the right distance apart... since 99.several9s per cent of humans have one nose, one mouth, two eyes...


Quite, a large part of what you perceive to see is your mind filling in the blanks of what it expects to see. Aside from the blindspot in each eye there's the phenomenon of seeing something out of the corner of your eye in which any random collection of shapes and shades can appear as something completely different. Classic example being a man in the corner of the room when it is actually a hat rack with a coat on it.

Assuming an AI has to also fill in the blanks because it's senses can't perceive everything around it in high detail at the same time it could at least have the advantage of being designed to know what part of its senses are filled in and to what extent.

On a somewhat related note I've always entertained the notion of an AI not perceiving its position in its environment at all like a human's even with similar sense faculties. We tend to convert our input into a Cartesian Theater, but what if an AI perceived the environment around it (and itself) in a disembodied sense? A Cartesian Diorama perhaps.


Try and get [a computer] to play, say, "Puerto Rico",...

That's been done. See the Puerto Rico Evolver.


Interesting; however, it's no longer a Chess-player, and it's obviously still a work in progress if it's only just learned to block a ship with its goods.


I only have to maintain a hash set of the qualities of your face to compare against a pretty short list. Moreover, with a 4GHz process attacking the problem, 1/1,000,000th of a second means I've got 4000 cycles on a single thread to do that comparison.

Assuming the hash is sorted and that my list really only contains 150 subjects to compare against, it should be a very quick search. So now you need to think about how many cycles do I need for picking out the center of your face, identifying core features and then hashing those values... my guess is that most of the processing steps for processing your face can easily be made to run in parallel so as long as none of them take more than around 3500 cycles, we’ve probably got the performance we’re looking for.

Still WAY faster than the human is doing it. Best of all, we could quadruple the AIs monkeysphere with minimal degradation of performance, but the human is stuck with the wetware limitations.


I'm not convinced; the distance between my eyes to an observer at a arbitrary time is not a constant, but varies with the angle between the line drawn between my eyes and the range to the observer (and arguably with any rotation that is applied to my face wrt them).


That's true, but since your motion is measured in 1/100ths of seconds you get to confirm the nose's existence 10,000 times per unit of motion. Things that move faster than that look to humans like they have teleported from point A to point B and often force the human brain to reprocess a large part of the scene. The part of the brain that tracks interesting features would probably be helpful at speeding up the process by noticing that one related feature moved (say an ear) and forecasting that all of the related parts of the scene (the rest of the head) moved with it.

Vision is a predictive activity. Most of what your eyes are doing most of the time is simply confirming that the model is still true. If I stuck you in front of a wall with constantly changing random images that are flipping at a rate faster than 1/00ths of a second, you would become functionally blind.

The idea here is that there's no reason an AI can't use all of the same processes you're using, sped up (because of silicon) with some enhancements added on (because we can).

Let's put it this way... when seeing someone you haven't seen for a year who you're not expecting to run into in a crowded store, how long do you sit there debating with yourself about whether you know that person and whether they are who you think they are. The only reason you do a better job at seeing your coworker is because there are other parts of your mental system feeding into your vision. You're at work, so you're expecting to see them. You hear their voice and that primes you to see them. If a coworker with a cold showed up in your kitchen unexpectedly, it would take you whole seconds to understand who you were seeing.


What's your methodology for measuring that angle for the computer? All I'm asking you to do is prove that you have solutions for the issues rather than attack someone for saying that the problem is more than comparing 2 still pictures.


The computer has a mode for the relative 3D positions of face parts. Once I extablish that I can't see one of your eyes (because it is behind your face), I can rotate my face model so that it matches the locations of your nose, mouth and remaining eye to get a good estimation for the facing vector of your entire face. Deriving the vector for your nose after that is a quick itteration of correcting my image analysis based on the extemated nose vector and then correcting my vector based on the image analysis until the delta between itterations is small enough to be ignored.

Again, most of this work is being done by paralel processes, so identifying your missing eye is one process, another process is collecting clues as to the position of your mouth, yet another is playing matchups of known face structures to the current understanding of your face part locations to see if I'm trying to match you to a human, a cat or an anime character. Every new clue shuts down or enhances some other threaded lines of inquery. And the model is always in play. So once I've established that "the thing in zone A is a human face", I no longer have to ask that question fot a least a few more seconds.


A lot of "recognition" models, particularly early ones were waaaaay much too complicated ... It wan't until the analogy of the "bird-flocking rules" percolated through the programmers consciousnesses, that simpler ideas prevailed. E.G. "Samart" software looking for suspicious behavior in places like car-parks under cctv. The algorithms changed to something like: ( With, if the answer was "No" you re-set to zero, at any point. ] Movement? Over 1.3m high? Less than 0.5m wide? Going to more than one car, sequentially? Pausing at each one? ALARM! - Or, at least, wake up next set of algorithms to examine more closely. [ A similar routine-set would have identified the "cars" in the park & logged their positions - & a "station-keeping" routine would run (as per the human brain) noting that if nothing changes, it's OK ...]


The problem of facial recognition is solved and implemented. The City Of London cameras are connected to a facial recognition system that matches everyone it sees against a database of known criminals and terrorists. That's hundreds of thousands of people per day.


You have statistical data that shows a sensibly low number of false negatives?


That'd depend on what you consider "sensibly low", but I think the main point is that facial-recognition software is not a hypothetical thing in 2014. It's an established technology in widespread use, and it works by fitting a parameterized model of the face, rather than by bytewise comparing images.

Manipulating a 3-dimensional computer model, and going forward and backward between the model and a "snapshot", is pretty standard stuff these days, and the popularity of 3D graphics (games, movies, design software ...) has helped drive lots of specialized hardware and software for it.

There's free and even open-source facial-recognition software available for anyone who wants to really get into the depths of it.

Getting back to the original question of technical problems leading to AI, facial recognition actually seems like a good example of software that carries out a "human brain" kind of task, without showing any signs of becoming an "artificial person". My bet is that we can expect more and more of that over time, with all kinds of increasingly powerful tools but no "other kinds of people" emerging by accident, and not for a long time.

Lots of good information available on facial recognition ... (one software vendor as an example)


Going back to the original curves, I suspect that it's a red herring to try to read too much out of any manageable function estimating how hard it is to achieve a certain "intelligence level".

A definition of intelligence that can use a single parameter to describe the wide range from dogs, through monkeys, to one person being smarter than another, to whatever hypothetical entity is way smarter than people, is a pretty slippery thing. Consider, too, that the difference in intelligence between a person and a cat is likely to be qualitatively different from the difference in intelligence between a person and, say, an iPad. (I'm very ready to be wrong, by the way, if somebody has a good definition/measurement in mind?)

Thinking about other technologies, like railroads or jet engines or televisions, seems like the "difficulty functions" reflect a whole bunch of roadblocks and step changes in what makes them "better". Over time, the available tradeoffs between, say, for a jet engine, peak thrust, efficiency, reliability, fuel economy, weight, will change over time in a pretty complicated way. Building better artificial minds seems likely to be at least as complicated ... ?


Facial recognition itself does not somehow become a person. It is a weak AI in the stack of weak AIs that would have to be interacting with one another to emerge a person.

The big question is how big of a stack would one have to build and how many flavors of weak AIs would one have to invent before the result was as malleable, creative and intelligent (in the problem solving realm) as your average human.


What kind of weak AI stack-suite is available now? If there isn't one, what would you put in it?



Obviously if I knew the whole stack, I wouldn't be having this conversation with you... I'd be too busy building a strong AI.

As a first pass, I'd want to include: Sound processing Sound fingerprinting Sound to language translation Natural language processing Sound location modeling Visual shapes processing Visual location processing Visual object identification Visual object generalization Facial recognition Monkey sphere processing Social hierarchy modeling Spacial object modeling Conceptual modeling Conceptual reprocessing and goal seeking Path finding Self modeling Mirror modeling Hunger modeling Sleep cycling/memory reprocessing Boredom modeling Expectation/disappointment modeling

Once you've got all those systems up and doing a good job performing their individual representations, and then wide them to eachother (so, for instance sounds and faces can both reinforce someone's presence in your monkey sphere) you should begin seeing pretty human like behavior.



About this Entry

This page contains a single entry by Ramez Naam published on February 14, 2014 6:05 PM.

The Singularity Is Further Than It Appears was the previous entry in this blog.

A Rebuttal of 'The Singularity is Further Than it Appears' is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Search this blog