Don't forget ex-PFC Wintergreen in 'Catch-22'. The mail clerk who ran the army by controlling the flow of mail.

You know, spam filter is the most likely program to become self-aware. It must constantly evolve to fight human minds of the spammers, so it`s like learning from the best.

The other good candidate is whatever spammers use to automatically bypass the "captcha". This competition will soon allow programs to recognise text in pretty much any image.

He may have beaten you to print but I distinctly remember you talking about this same idea on a panel in the 2005 Worldcon. And you did have that self-aware pyramid scheme...

Wasn't Permutation City published in 1994?

I'm dubious as to what the changes in Vermont company law are supposed to achieve - taxable residence is generally defined by tax treaties or by how a company actually operates (e.g. the UK concept of 'central management and control', or the wider concept of permanent establishments) and from a practical perspective, a brassplate company in Jersey seems to achieve all of the same effects, with the minor hassle of chucking a professional director in Jersey a few thousand pounds a year to host telephone calls and rubberstamp minutes as and when required.

Anatoly: AIUI, the spammers use a mechanical turk staffed by porn monkeys to bypass captchas.

That is: you're a spammer. You write software for posting to blogs. You also run porn websites. When a porn monkey tries to download some porn, your blog spamming tool tries to post to a blog. When it encounters a captcha image, it feeds the captcha image to the porn monkey and says "you must type in this string to access the porn". They type it in, you serve 'em up some porn, and meanwhile you've got a decoded captcha with which to spam the internet. Elegant, but not exactly liable to acquire a mind of its own any time soon.

Note that the only avowed socialist in the United States Congress, Senator Bernie Sanders, represents Vermont.

micro-targeted lawn sign ads in 50 US states to sell ... online dating websites

Hmmm. How micro-targeted are we talking here? I mean, does it get down to the level where you wake up in the morning and someone's stuck a sign in your front lawn during the night, with a message that reads "HEY! YOU! AJAY! CALL THIS GIRL NOW! WE THINK YOU GUYS WILL REALLY HIT IT OFF!"

I'm intrigued by the idea that a company might be willing to implement its processes completely in software, and then automate as many of the remaining human "components" as possible (the companies I've worked for love using software, but only as a way of keeping humans from screwing up). Eventually even the executives might be replaced by their decision analysis software being hooked directly into email, sending orders out once it determines the optimum series of actions in a given situation. Add a marketing intelligence system to model the competition, that gets turned around and used to model the company itself, and wer're at self-awareness. Then we puny humans just have to hope that the self-aware company was descended from cooperative businesses, and not predators like Charlie's pyramid scheme.

@8:

You haven't read "Halting State" yet have you ajay :)

Daniel Suarez (aka Leinad Zeraus) has written the book on this: The Daemon. Not software embedded companies, just world-controlling bots that send emails and give people orders to be implemented in meatspace.

Hmmm. How micro-targeted are we talking here? I mean, does it get down to the level where you wake up in the morning and someone's stuck a sign in your front lawn during the night, with a message that reads "HEY! YOU! AJAY! CALL THIS GIRL NOW! WE THINK YOU GUYS WILL REALLY HIT IT OFF!"

No, but I've seen different signs for different communities here in the same county. And sometimes the communities are fairly small (5000 people or so). I've seen Circle Pines, Blaine, Lexington...and others all within 20 miles of where I live. I've never had the urge to actually type one of these in a browser.

@5: The goal -- for the state -- is to try to get more companies willing to pay Vermont their annual incorporation fees, rather than Delaware. The advantage for the companies is that you can have people all across the country form a LLC, and not have to physically meet. (As I recall, the NAEB folks -- people who tried to form an LLC to get an ebook reader at a discounted price -- ran into this problem; I think the Vermont law might have been able to help them.)

It's not a huge advantage just yet... but it's a fairly obvious step.

Very interesting stuff. As a law student with a background in CS and probably a future lawyer (who am I kidding, a future civil servant more like) I'm curious whether we'll see formal (as in parseable) language for legal work in the near future. The most obvious place to start is contracts, and I think I've read something about computer contract modelling before, but why stop there? Since one of the axioms of jurists everywhere is that the System should provide juridical certainty, why not write all the laws, regulations, etc, as formal language? Of course the IO stack for a legal OS would be rather ... interesting.

Writing the laws in a formal language would not provide certainty. You still have to map the terms of the formal language onto the real world, and there will always be multiple ways to do that.

Writing the laws in a formal language would not provide certainty. You still have to map the terms of the formal language onto the real world, and there will always be multiple ways to do that.

Thos eof us who read the Oil drum and other websites, or even know enough to gather the figures ourselves and collate them properly, have known for years that the UK is heading towards a major problem. Its one of the failures of the current gvt- not taking the UK energy industry seriously, deregulating it (or deregulating it further, I forget which) and then signing us up for legally binding carbon emissions targets whilst considering allowing the building of 12 (I think) new coal fired power stations. Sure, some of them are to replace current ones, but no-one seems to be trying to replace Longannet and cockenzie up here, and although I support renewables (all types) and energy efficiency etc, if we are to have a hope of meeting our targets we need to close one or both of them and build probably 2 new nuclear stations whilst expanding our wave power systems, as well as enforcing strict new building regulations.

Richard-- as I see it, this is simply a proposal to streamline some of the processes of forming Vermont LLCs. Nothing really that special or important. I'm a transactional lawyer and I do this sort of thing for a living.

The concern that Professor Johnson (the concept's proponent, according to the website that Charlie linked to) wishes to address is that people who wish to collaborate with one another may not want to have their relationship construed as a general partnership. (General partnerships can have several gotchas, such as unlimited liability for the partners). LLCs require a little bit of expertise to organize (but really just a little bit-- they never did require face to face meetings or even formal corporation-style board or stockholder minutes or meetings); the idea here is to make it so easy that even a caveman can do it.

We're still talking about an entity ultimately controlled by people here-- no self-aware financial instruments or robo-contracts.

That Dilbert link nearly cost me a keyboard ...

By the way, is 'mechanical turk' an accepted term of art, or one you've just made up?

1) the lawnsign empire perhaps should be more appropriately named the lawnspam empire. Become a member of causs and fight back! (www.causs.com) From their website: Most laws and ordinances classify these signs as either common litter or abandoned property. As such, they are fair game for removal by anyone, despite the sign spammers' arguments to the contrary.

Too bad it isn't as easy to rid the Internet of said spam.

2) George Soros - who doesn't fear this man?

With all of this coming in the same month as a fiscal crisis so bad that George Soros is saying it exceeds his worst fears, I am getting a horrible sense of deja vu.

Visions of Black Wednesday on a global scale? I can't help but wonder how much Soros will profit. On a brighter note, he is 78, and therefore can only wreak havoc for perhaps another 20 years. I suggest a steady regimen of high priced hookers to perhaps hasten his demise.

Ken @19: See Wikipedia's entry, and then google on Amazon's Mechanical Turk. (It's a general term these days for any piece of software that comes with an unwitting human being in the box -- like Searle's Chinese Room, only for real.)

Thorne: Soros seems to have made his pile and is more interested in philanthropy these days.

That's www.causs.org, not .com.

@15: thus my point about an IO stack. The legal OS would need the facts entered by someone, and that in itself requires a certain level of qualification (not in the sense of being qualified but in the sense of distinguishing the qualities of things). For instance a law that would talk about theft would have, in order to run, to be given all sorts of data that so far only human beings can collect. But while formal language wouldn't in itself give absolute juridical certainty, it clearly would improve it. How clearly is quite obvious when looking at the amount of appeals made on points of law.

The Soros article was very interesting and it confirms my suspicions on why the financial system is inherently unstable: too many positive feedback loops, something that would be obvious to any engineer who carefully examined the system. What we need is for someone to design a financial system with more negative feedback built into it.

The fact that the free-marketeers though as system with so many positive feedback loops built into it would be inherently self-correcting just beggars the mind. Obviously engineering principles aren't included in an economist's training.

Pah! Self aware spam filters is obvious. What about the next step, they would have to go berserker on the spam instigators, their natural enemy. Roaming self aware entities across the network, detecting individuals sending more and some critical number of emails per day and then shutting down their lives - permanently.

And what about self aware pirate apps, infecting your machine and uploading your applications for everyone to use without you knowing? I'm sure someone by now must have created a trojan to steal valid keycodes from victims machines?

And finally, what about self aware music apps, drowning in a sea of MP3s, learning the basic structures and repeated coda to communicate in new song snippets that reflect what they are trying to say - and creating hit records by mistake.

We already have virtual charities, I don't see that businesses are that different. I'm on the board of a very small charity and the board members live all over the country (and we've had an .au member in the past). We're a 501(c)3 charity, legally incorporated, and we've never all met in person. We use a private Yahoo board for day-to-day discussion and a logged chat room for our required annual meeting.

Heh - anyone read "Octagon" by Fred Saberhagen?

Or am I the only one that sad? It's got (postal) computer moderated games! And an AI who decides the best way of winning a game is to kill the creatures playing the opposition.

Err - no one was going to use that idea were they?

(Written way back in 1981 in the days of pre-history mind, before the interwebs. We had to make our own entertainment then, playing games through the post.....)

Dave Moore @ 24

Obviously engineering principles aren't included in an economist's training.

A good friend of mine is an economist (that is, he advises on economics in the real world) as well as a professor of economics; we've had this discussion about engineering and mathematical discipline in economics. He's somewhat sarcastic about the mathematical sophistication of economists in general, and agrees with me when I bemoan the lack of control system theory in economics curricula. Not even linear control systems, let alone nonlinear or chaotic systems.

But I don't think economists are ready for that yet. I heard one talking on the radio a few weeks ago, and he actually said that decreasing demand should cause prices to rise.

I particularly enjoyed your idea in Accelerando of having a software-only virtual company made up of cellular atuomata that changed states so quickly that the company was only instantiated for a few milliseconds at a time in any given form. Impossible to sue. Tons 'o fun for the Wall Street funny money boys.

Incidentally, if you haven't seen it, great article on the financial craziness here by the author of Liar's Poker.

I remember, a year or two back, when I hadn't yet killfiled David Friedman on RASFF, I asked if there wasn't enough friction in the markets, using the mechanical analogy of the damping in automoticve suspensions.

I'm not sure the guy even understood the concept.

Of course, all that bouncing around the "real" value, whatever it might be, is how the people speculating make their money. They could still do that with negative feedback, but it looks as though the necessary requirement for acceptance of an Economic Theory is that ability to exploit it to make shitloads of money.

(I'm cynical, despite the latest Nobel Prize winner.)

The trouble with the future is that it is often just the past repeating itself. I don't usually buy into meta-explanations for historical patterns ("rise and fall of the great powers", "end of history", "clash of civilizations", etc.) but this article really caught my eye:

http://www.salon.com/opinion/feature/2008/11/07/fourth_republic/

The author may be on to something with these recurring 72 year long cycles in American history.

So American history can be summarized below:

Republic - America 1.0
Year Started - 1788
Crisis - Revolution, New Constitution Founder - Washington
Reactionary - Jackson
Failure - Buchanan
Economic Type - Agricultural

Republic - America 2.0
Year Started - 1860
Crisis - Secession, Civil War
Founder - Lincoln
Reactionary - Bryan (*)
Failure - Hoover
Economy Type - Early Industrial

Republic - America 3.0
Year Started - 1932
Crisis - Depression, World War II
Founder - FDR
Reactionary - Nixon
Failure - Bush
Economy Type - Late Industrial

Republic - America 4.0
Year Started - 2004/2008
Crisis - Mortgage Debt, WoT/Iraq
Founder - Obama
Reactionary - TBD
Failure - TBD
Economy Type - Post Industrial

(*) Bryan was almost president twice and personalized the populist backlash (see "cross of gold" speech) against the national government and big central banks.

So maybe Obama will be one of our great presidents and founder of the Fourth American Republic. And we are in for some really bad times before things get better.

Now the Bank of England seems to be consulting Tesco's management-information dashboard when setting the Bank Rate, perhaps Charlie's Real-Time GOSPLAN vision is being delivered?

One damn thing after another.

Not to mention that The Economist's annual for what seems like the last ten years prediction of the fall of the American auto industry seems about to come true.

Seeing as my old man has a pension with Ford I have more than a passing interest in this.

Bruce @28: Yeah, falling prices can decrease demand, and rising prices can increase them. The 'law' of supply and demand isn't a law, it's a heuristic - contrary to what some libertarians have stated as a fact (I've had more than one compare it two the the second law of thermo.)

pdq @31: Hasn't this been done already? Stirling Newberry has been moaning that Lind ripped him off over on 'The Agonist', but I'm sure I've seen the idea done before, possibly even by a dabbler like Blish.

Charlie: This is sort of about the right time and place to ask this, I hope. In Accelerando it's mentioned that no one could be upgraded to the point where they could take advantage of Economics 2.0 and remain human, but it seems to me that Aineko didn't quite fit into that pattern. Not that I'm claiming the Cat was human, but it (she?) seemed to get along with people just fine, and to a point, was perfectly comprehensible to humans. Also, how would you test for self-awareness in a corporation?

SoV @34: the hidden easter egg in "Accelerando" is that Aineko isn't a cat, it's an AI that uses a cat-puppet to interact with the food apes. And the novel is kinda-sorta Aineko's autobiography, complete with human-equivalent simulations of the Macx family (who Aineko releases to live their own lives in the final scene).

The postulate behind Economics 2.0 is that if economics is about resource allocation and markets are about information transfer, and if there exist stronger types of intelligence than our own, then participation in the market-analogues used by those intelligences will require a stronger-than-human theory of mind. E 2.0 is a "better" way of managing resource allocation, but it relies on understanding the motives of other market participants to a degree that isn't achievable by human-equivalent intelligences.

(Also: it's a work of speculative fiction, so don't expect too much rigour or answers to questions in the back of the textbook. OK?)

Corporations are self-aware; at least the good ones are. "Culture beats strategy".../management consulting

We can go back to Sun Tzu for the importance of self awareness to strategy.

Charlie @6

Do Turing porn farms actually exist? I thought they were just a cyberspace legend...

Charlie @35: I perhaps did not make myself clear (yes I know what Aineko is.) IIRC, Aineko evolved from a simplified cat-in-a-box model, through human-equivalent intelligence, and then into something more complicated. So she is an example of something that can at least put on a convincing human face but at the same time is (presumably) able to take on the VO on their own turf, ie, she does do econ 2.0.

So this seems to be a counterexample to the thesis, if there is a thesis in any meaningful sense. If all you're saying is that if you incrementally increase human intelligence, at some point what you have is something that isn't human, well, that strikes me as arguing over semantics. But it seemed you were arguing something a little more intriguing, that there were qualitatively different types of intelligence, as opposed to merely qualitatively different types of cognitive abilities, or qualitatively different types of information processing. Hope that clears things up.

Alex@36:

A good example of this is Honda's old mission statement from when they were primarily a motorcycle company. Simple and to the point.

WE WILL CRUSH YAMAHA!

That was literally their company motto. On the letterhead and stuff.

Not sure why, but this discussion reminded me of http://en.wikipedia.org/wiki/MONIAC_Computer, an analogue economics modeller that uses hydraulics.

George at 38

Not answering for Mr Stross but the requirement to create a pron farm is relatively straightforward:

Trawl forums for post entry pages and save until suitable porn monkey arrives

Open post entry page and read captcha

Display captcha to porn monkey

Use answer to correctly pass captcha on forum

Spam website

From Time, in a June column on captchas.

"In the meantime, Von Ahn has figured out a way to take advantage of all the spare brainpower hundreds of millions of people expend deciphering wiggly letters. He has teamed up with the Internet Archive, a San Francisco nonprofit that uses computers to digitally scan books and put the text online, where it can be accessed for free. When its scanners find a word they can't read, they automatically turn it into a CAPTCHA that gets exported to a website in need of one. A human reads it and transcribes it, and the results get sent back to the scanner and added to the archive. It's nice to know we humans are still good for something."

It's called reCAPTCHA and BoingBoing is one of the biggest users.

pdq @ 31: Sounds like you're talking about the "Party System": http://en.wikipedia.org/wiki/Party_system#U.S._models

There are 5-6 of them by most measures (your America 1.0 should really be broken into two pieces, the break point being Jackson's election.)

As to software becoming self aware, I'm reminded of the adage that we'll have paperless offices right after we have paperless toilets.

It's true that computer viruses imitate biological viruses -- but even biological viruses lack the abiliy to reproduce without hosts, and as such are not even semantically acknowledged to be alive (at the very least, there is debate as to that status).

My guess is that we will, ever so iteratively, improve upon computerized pattern recognition, so that, for instance, car driving AIs are truly no longer in a remote future. We are getting closer and closer to direct brain input, in terms of reading and pehaps writing memories.

But a self aware computer program, one that goes beyond imitating a servile intelligence and is actually able (and interested) in its own goal set? Not until we fully understand how the brain does it, and are able to actually program it. The basic building algorithms of software programs, as the exist today, are just not sufficiently malleable to allow it to happen through some imagined evolutionary process.

BTW, an interesting corollary application of the CAPTCHA technique, as described, would be to augment the accuracy of hand recognition software.

Arthur @46: Well, is there any other kind of intelligence? That's my point. There is one school of thought that, roughly, says, that intelligence is simply the integrated amalgam of a large number of information processing routines. More intelligence then equates to having more, and more elaborate variations of these routines, i.e., differing intelligences can be arranged along a continuum. Again, very roughly speaking. Another school of thought thinks that there are intrinsically different sorts of intelligence. To go back to Charlie's stories, 'bolting on enough stuff' to the basic human chassis so that it becomes able to make use of econ 2.0 renders the end result not remotely human anymore would be kind of like 'augmenting' a sound system so that it could play video. Yes, it might be possible to 'bolt on enough stuff' to your speakers, equalizer, etc so that the end result would be something that could take full advantage of video inputs; but few would argue that it is just an upgraded sound system.

That's why I was interested in Aineko's apparently privileged case.

ScentOfViolets@48: Don't get me wrong, I'm a fan of Charlie's fiction since the early Macx stories in Asimov's. But I also write software, including game AIs, and they are sadly, very unaware.

The data analysis,and pattern recognition tools are becoming very powerful indeed -- witness the recent press about Google's analysis of queries about flu symptoms being a better predictor of inluenza outbreaks than other tracking systems.

But spontaneous birth of self awareness is quite a different story and is not really possible in the way most SF posits -- no more possible, anyway, than me spontaneously evolving wings. For software, not only are the building blocks not there -- the means for evolution, for changing existing building blocks in any way, are not there either.

I think that enhanced human intelligence via some sort of a direct brain interface is much more likely in the "near-term" -- something like instant, thought driven access to the web or for that matter a calculater is something that I do expect to happen long before true AI exists.

Once we understand and can mimick the brain's complexity in hardware/software, we will certaily be able to program it. But unlikely that we will do it before, or that it will happen somehow accidentally, on its own.

@49: spontaneous birth of awareness a la "the internet wakes up" -- a standard trope of post-1980s SF -- seems to me to be the modern cognate of the common knowledge in the 18th century that maggots were spontaneously generated by rotten cheese or meat; it just kind of "came alive" by spewing out flies. Knowledge of the fly life cycle eventually put paid to it, but it's going to take a lot more than that before we can make our own flies, starting from their genome and proteome and stacking one codon on top of another.

Now, limited bootstrapping of complex semantic processing networks that are designed to implement genetic algorithms for recognizing deep linguistic structure in target strings (i.e. spam filters) is another matter, but as long as they don't have hooks into other kinds of system I'm not too worried about where they're going.

Arthur @49: That may or may not be true. The reason I thought it apropos to insert my question was that it touched upon the question of how one recognizes intelligent behaviour. Suppose system X is self aware, has been for years, and is in active communication with systems Y, Z, and W, also intelligent. What are they talking about? What 'matters' to them? Please note that I am not excluding the possibility that in terms of information processing capacity these systems are no more intelligent than, say, a particularly bright chimpanzee. I'm suggesting that the intelligences(the intelligence, not sensory modalities!) are so different that we would not recognize them as such. Certainly they wouldn't have evolved within the framework of competition for limited resources in the way that animal life has here on Earth.

@51, who could argue that intelligent behavior is hard to find? :) But to be relevant within the contextof this discussion, the system would have to be observable in some fashion.

There has been a marked reduction in rates of spam in recent days -- a result primarily of some high profile recent arrests, as I understand. But perhaps the lobsters have finally gotten hold of some Symantec software :)

I'm not assuming the intelligent behaviour is unobservable. I'm saying that the behaviour is obvious, but is not easily interpreted as intelligent behaviour. Let's say, for example, that sentient spamware differentially blocks spam, both in quantity and type. How do you distinguish this from non-sentient actions that simply aren't immediately obvious from the code? I would imagine that the first such instance of this was sometime in the 40's :-)

This type of question comes up in a variety of different ways. Suppose that deep within the binary expansion of pi, there's a 361,201 long string of digits that encodes a circle on a 601x601 raster and the probability of this occurring(assuming pi is normal) is 10^361,201 to 1. Is this proof that our universe is an artificial construct? How about if (for the sake of argument - it's actually impossible by definition. I think.) this expansion started immediately to the right of the decimal point instead of zillions of bits downstream? Is this proof of a Creator?

I guess what I am groping for here is that it is really hard to assess intelligence if there is no obvious communication.

@51

My initial reaction is to wonder what evolution without limited resources can do.

Differently limited resources: that might be interesting.

Now that you've seen how Scott Adams handled the Spam Filter Robot Uprising, can we expect more or less the same resolution for your novel 419?

(yeah, right!

Since we're talking about sentient Spam Filters: http://www.dilbert.com/strips/comic/2008-11-10/ Ironically Sentient was the name proposed for a (failed) geneitic algorithm based spam filter I worked on back in 2001...

"And what about self aware pirate apps, infecting your machine and uploading your applications for everyone to use without you knowing? I'm sure someone by now must have created a trojan to steal valid keycodes from victims machines?"

I'm sure this is already done, indeed sequestered machines in botnets are wide open for whatever is to be stolen off them. Pirates do have many cunning ways to steal software, although I'd concede this is not much of compliment as most copy-protection schemes are as trivial to circumvent as they are poorly implemented.

"And finally, what about self aware music apps, drowning in a sea of MP3s, learning the basic structures and repeated coda to communicate in new song snippets that reflect what they are trying to say - and creating hit records by mistake."

Maybe the record labels are already using software in production of big hits. Well alot of technology is already used in the production of music, you don't even need to be able to sing to be a pop star (voice pitch correction... in real time). Generally, I find that proceduraly generated content tends to be slightly crappy but often more than passable, which also doubles as description big music/big hollywood content today. These systems must replace the cooke-cutters of yesteryear. :D What worries me is when automated content generation really starts to produce consistently pleasing and enthralling results, such that it begins to outperform what creative humans can produce in quality, not just in quantity. Such an endless supply of content for whatever format of media people consume (enhanced continually by the need for competition) could mean a shutdown of the human creative endeavor (Although one could argue with such powerful tools are only more enabling for the human wielding them, rather than taking over - since this hasn't happened already). Eventually the machine minds taking things over may be able to keep us humans busy consuming this endless supply must-see must-experience media being produced that we'll have no spare thought processes let alone free time in order to get in the way of them taking over the world. Oh crap.

I quite enjoyed what Charlie did with the robot cat Aineko in Accelerando. It is a interesting, if perhaps unintentional study of what happens when another species' mind is elevated to sentience and beyond. We should no more expect human behaviour and values from an augmented animal than we should expect from an extra-terristrial intellegence. Aineko's mind was extrapolated from a highly evolved solitary predator. In a Cats context it is a very efficient master of it's environment, mentally and physically, possessing a cool very calculating mind highly specialised in analyzing and anticipating the moves of it's prey ready for a lightning fast kill with no extraneous expenditure of effort. (While we are rather fical multi-purpose overclocked monkeys). I thought this was the most interesting and pertient prediction for coming AI advancements. More exactly, what happens with intelligence augmentation. Again we come back to the spamfilter, if you elevate something currently primitive to sentience, what happens?

ScentOfViolets @ 53:

Suppose that deep within the binary expansion of pi, there's a 361,201 long string of digits that encodes a circle on a 601x601 raster

Arthur, ScentOfViolets: I think you're using words which means very different things and confusing the issue. "Consciousness" is not the same thing as "awareness" which is not the same thing as "intelligence". And I think all of them exist on a spectrum; there's no sudden step function from 0 to human. And "intelligence" (which you might call "sapience", though that has some additional connations), is not "sentience". A spam filter could be extremely intelligent without being aware in any sense outside its very limited domain of functionality; conversely a "self-aware" machine isn't really that hard to make with current technology, in fact, I'd argue that it already has been done to some extent, but it won't be much more intelligent than a flatworm. As for consciousness, we have to yet to come up with even a vague definition that more than 2 or 3 people can agree on, so it's a little premature to talk about creating it. I'd love to argue about it myself; I have some definite opinions on the subject, but we'd probably just spend a lot of time trying to agree on the definition of the problem, and never get to talking about solutions.

Just to be clear, I agree with Arthur that brain-augmentation is likely to be a major factor in human society long before AI goes human, let alone post-human. There's so much less to do: all we need is good interfaces between the brain and a computer, and work is ongoing on those now. For AI, we have to first find an overarching theory of mind, and then learn most of what there is to know about the brain's subsystems, so we can emulate them. Lots more unknown there.

Bruce @57: Assuming normality, you are correct. That's why I specified the unlikelihood of the likelihood, as it were. If there's only one chance in 10^361,201 that this particular representation appears where it does, I'd say that this is pretty unlikely. What are the implications in terms of the question of whether our universe is a made thing or not? I surely don't know.

Your other point, about having issues with the meaning of terms as they are differentially used by various posters is a valid one. I acknowledged it at the beginning, and am trying to be as clear as I can. Again, I am trying to avoid talking about degree of intelligence, say a human who falls one standard deviation below the norm as opposed to one who falls two deviations above. I am talking about a totally different kind of intelligence. Which, hopefully, is not going to be confused with existing in a completely different environment, one with no sight, sound, or even dimension as we know it(though I would tend to think that the different environment would have something to do with it.)

Of course, there's also a third school of thought in SF, represented by none other than Greg Egan, that says once you get up into the range of human, there really isn't that much difference between intelligences allocated differing amounts of computational resouces.

Bruce, I agree with much of your analysis.

The reason I think self-awareness is rather key, is because I believe that a genuine existential threat, of the kind that is posed, say, by the Terminator scenario, is only possible with a self aware entity, that, on top of becoming self aware, decides that its goals are actually different from those of its creators, and further, decides that the best way to advance those goals is to somehow act against humans.

There is also, of course, the grey goo scenario, one that would only require a will to survive and replicate, without much intelligence at all. But I'd remind anyone genuinely afraid of that scenario that the bacterial or insectal biomasses far exceed that of the rest of life on Earth, and they could swarm to consume us at anytime.

Arthur @ 60: I agree that self-awareness is necessary for that sort of threat; without it an organism (or whatever it is) doesn't have values and needs which can come in conflict with ours. And any kind of awareness, let alone self-awareness, or consciousness, for that matter, can't occur spontaneously just because there are enough neurons, or transistors, or whatever, piled in a heap with random interconnections. It took biological evolution about 3.5 billion years to come up with the nervous system, longer yet to come up with the endocrine system, and still longer to invent the immune system, and AFAIK those are the only biological systems that have ever evolved to the level of self-organization required for awareness of the sort we're talking about, and the latter two are relegated to specialized areas because the nervous system got to the niche of dealing with the external world in general first, and can't be displaced easily.

I'm reminded of the adage that we'll have paperless offices right after we have paperless toilets.

The paperless toilet is now a reality in Japan, and it always has been a reality in most of the Middle East. That's why you only ever handle food with your right hand.

You may now return to your intellectual discussion of the future of humanity...

About Economics 2.0: it looks like math trades sort of fit the description from Accelerando. Right now of course we only have humans participating in them.

@24, Posted by: Dave Moore

"The Soros article was very interesting and it confirms my suspicions on why the financial system is inherently unstable: too many positive feedback loops, something that would be obvious to any engineer who carefully examined the system. What we need is for someone to design a financial system with more negative feedback built into it.

The fact that the free-marketeers though as system with so many positive feedback loops built into it would be inherently self-correcting just beggars the mind. Obviously engineering principles aren't included in an economist's training."

I've been on a website where an engineer played those games with an econ professor, who had taken various systems analysis/control/feedback courses in the engin school as part of his econ Ph.D.

It's not a failure of the technical part of the field, but a political/social failure. A lot of econ guys have been pushing fanatically for deregulation for decades; their interests coincided with the interests of a bunch of guys who controlled lots and lots of money.

To me, what it clear is that the positive feedback loops were provided by the (now not really disputable) fact that small groups of people at the tops of the financial biggies had enough control and information to loot the system. The classic method being to generate bogus results, and get paid very large yearly bonuses for this 'productivity'. Those checks cleared at the end of the year; the fact that the 'productivity' was fraudulent took several years (to over a decade) to become unavoidably clear.