Charlie's Diary

I'm looking forward to reading it. Keep up the good work!

Posted by: James Reynolds | July 10, 2007 12:16 PM

Well, I was going to report a silly editing error (if you're going to build your data storage out of carbon atoms, it helps if one of them isn't radioactive) but such is the power of the intertubes that it was fixed by the time that I finished typing this comment.

Mind you, you do say `And some time after our demise, this information will be available to historians.' which makes me wonder how on earth we will prevent it being available to the police before that, and how we'll prevent society's implosion into a total-surveillance police state.

Searching that mass of almost completely unstructured data will be really hard, too. `Play back that conversation I had in a pub two months ago in which someone used the word `fishhooks' in a strong Glaswegian accent with intense background noise'. A lot of *humans* have trouble discriminating sound from the background in that situation!

... and the visual problem is vastly harder, and collecting the data and analyzing it when AI improves might not help, because cameras are fundamentally much crappier than the human eye. (Perhaps you think there'll be cameras with the same field of vision and range of intensity response as the human eye anytime soon, but I doubt it because you're not that starry-eyed. The human eye is rather crappy by biological eye standards and is not a very good precision optical instrument but it's *far* better than anything we can fit on a phone.)

Posted by: Nix | July 10, 2007 12:32 PM

And it's currently the #2 most emailed article on their site :) Only baby mammoths beat you out!

Posted by: Canis | July 10, 2007 12:33 PM

I, for one, welcome our new baby mammoth overlords!

Posted by: Charlie Stross | July 10, 2007 12:36 PM

Saw a piece about your ideas on the BBC website today. Imagine pattern (see URL) to be complete, like a DVD game, imagine "the NOW" to be a shockwave on the pattern that allows you and me and any thinking being to move to preferred parts of pattern easily and you have imagined reality. Without the shockwave we would be "stuck", fall from the shockwave and you are dead but not gone just frozen again in pattern. Create a small set of shockwaves from here in the NOW and you can pick people up and put them on a new surfboard here in the NOW again. Now that isn't novel science fiction but it is certainly novel science fact. And it rather destroys the idea that we don't have access to perfect memories, we all do, we just didn't realise it until now. We will be able to devise things for simply looking back in pattern, liars will no longer have the advantage, we can call these wrist watches because they will be worn on your wrist and you can watch things from the past, anyone's past, with them. Rubbish? Of course it is. Go to bodgeitandscarper.org for the basics. You will need some imagination to free yourself from a box in a few years so perhaps it is worth a read now to begin to see how it might be done. The clues are all there but do you have the imagination? Best Regards John. PS If nothing else you may find a few ideas you can nick. All I want is for the ideas to take on a life of their own so I can retire to the beach with a beer and a dolly bird.

Posted by: John | July 10, 2007 12:47 PM

Interesting article (though is it similar to a blog you posted not long back?).

Are BBC news articles going to be a new staple of your publishing Mr S? Or was this a one off?

Posted by: Serraphin | July 10, 2007 12:50 PM

John @5: any chance you could maybe repost that in paragraphs? I'm finding it somewhat difficult to understand ...

Posted by: Charlie Stross | July 10, 2007 12:56 PM

We all know how (in)accurate Amazon can be but I'd like to point out to UK-based fans that the £7.99 mass market paperback is not due until March 2009 & not simultaneously with the £10.99 trade PB in January next year. Given the current exchange rate we might as well buy the US edition anyway - snip at $24.99 in October.

Posted by: gmilton | July 10, 2007 1:19 PM

I, for one, welcome our new baby mammoth overlords!

Mammoth baby overlords would be a far better story..

Posted by: Alex | July 10, 2007 1:37 PM

gmilton: you'll find the US hardcover somewhat hard to order through amazon.co.uk, and by the time you add air mail on top of the price of getting it from amazon.com you'll be saving precious little money.

(Also, I don't want to encourage Brits to be nasty to by UK publisher by buying gray imports. It doesn't do my book advances any good ...)

Posted by: Charlie Stross | July 10, 2007 1:40 PM

Alex: Mammoth babies? I can smell the toxic waste from here! And you wouldn't want to be in earshot when they start crying because they've just tried teething on the Walter Scott Monument ...

Posted by: Charlie Stross | July 10, 2007 1:49 PM

Dr. Stross is of the opinion that advances in electronic memory will produce a happy state where 'And if you're a student, it means you can concentrate on understanding your lecturer, and worry about making notes later.'

If only it were that simple.

Many students of my experience are far more likely to zone out in the lecture theatre (snog their bloke/bird, play videogames etc) while recording the lecture - a recording which will never subsequently be looked at or examined ever again.

Here's another point. If all human life becomes recordable, and if future generations are able to access all human, historical events in their entirety, and accurately, what will human communities do for myth? Malinowski defined myth as the charter a human society relies upon for its foundation, and myth is often the product of misremembered history. . .

Posted by: D.O'Kane | July 10, 2007 2:10 PM

D. O'Kane: (a) I'm not a doctor, and (b) I didn't have room to examine second-order consequences, or even legal and ethical considerations, while simply working over the issue of unlimited data storage.

Yes, some students doze off during lectures. But many don't. And some who doze off in some, don't doze off in others -- and vice versa.

The question of myth is an interesting one. It reminds me of the question Karl Schroeder posed recently: what if there is an end to the scientific enterprise? What if everything that can be discovered will be discovered, and any remaining questions are obviously beyond our ability to probe? At that point, the mechanics of doing science become .... not useless, exactly, but no longer relevant. And knowledge is a dead thing, the domain of scholars and librarians, not researchers.

The future: a place without mythology, science, or secrets. Very strange.

Posted by: Charlie Stross | July 10, 2007 2:16 PM

Hi. Long time reader, first time commenter.

D. O'Kane @ 12: What's the difference between students in the future who never examine their electronic memory and students today who never read their notes?

Charlie Stross @ 13: Well, if (and yes, it's a big if) we ever finish Science, we'll spend the rest of the time making deliberately subversive readings of the data, like those people who arge that Sauron's actually the good guy in "The Lord of the Rings"...

Posted by: A. J. Saunders | July 10, 2007 2:41 PM

Wherever there are humans there will be mythology--we make this stuff up. I might find the lack of secrets to be the curse of an All-Knowing being. No surprises ever? Sounds very boring.

Jeff

Posted by: Jeff Minor | July 10, 2007 2:42 PM

Fair enough Charlie - I won't be buying through Amazon anyway: I have my own, perfectly legal, way of getting it. Have you no way of getting simultaneous publication in the UK & US?

Posted by: gmilton | July 10, 2007 2:46 PM

New book sounds interesting. Scot's vernacular? Bank's book (Feersum Endjinn)has passages in heavily accented Anglish. I look forward to being entertained.

Jeff

Posted by: Jeff Minor | July 10, 2007 2:47 PM

Re: Grey imports - isn't it swings and roundabouts, or is the hit on your UK advances much bigger than what you make as your rabid fans buy the US editions? Speaking as someone who bought the US hardcover of Glasshouse as soon as I saw it in Forbidden Planet, telling me to wait n-months to a year to get my grubby hands on the crunchy plot goodness seems a little mean...

Posted by: Jakob | July 10, 2007 3:04 PM

"a recording which will never subsequently be looked at or examined ever again."
That's because it's not searchable, which for lengthy, heavily linear media means its effectively dead weight as far as research goes. There's no way to retrieve a crucial nugget of information without sitting through the whole thing all over again -- complete with lecturer's rambling about his summer holiday -- in real-time, or maybe 2x if your player does Chipmunk Voice Fast-Forward(tm). On the other hand, if, come revision time, certain key words could be plugged into my.life.google.com ...

Posted by: Canis | July 10, 2007 3:16 PM

Re: Scots vernacular. This is notoriously difficult to do well & done badly it's just plain irritating. Irvine Welsh is the best exponent I've read & I don't think even he got it quite right. Nivirthiless ah goat ivry coanfidence in ye Charlie (see?)

Posted by: gmilton | July 10, 2007 3:17 PM

An end to the scientific enterprise? Vinge did this in _A Deepness in the Sky_, didn't he? (Or, at least, he explored the far future of a metaculture in which scientific *discovery*, as opposed to rediscovery, ground very nearly to a halt a very long time ago.)

I can't see any way in which scientific discovery would ever *stop*, but it might become a process of filling in, very occasionally, increasingly small gaps in a nearly-complete world model.

I suspect some sciences, notably mathematics, are intrinsically endless.

Posted by: Nix | July 10, 2007 3:20 PM

This is an interesting contrast to the fictional future in Glasshouse, where our descndants know less than the past than we do. I hope the future described in the BBC article is closer to the truth.

As for Halting State, a police procedural with characters talking in Scottish vernacular sounds a bit like an Ian Rankin novel. Well, if Ian Rankin were to start writing in the second person present tense, anyway.

Posted by: Brian R | July 10, 2007 3:42 PM

gmilton @16: simultaneous publication in the US and UK relies on one of two things happening: (a) a multinational publisher buys world English language rights to a book and publishes it simultaneously on both sides of the Great Undrinkable, or (b) two different publishers buy local rights at the same time and synchronize their publication schedule. I have two different publishers in each of the US and the UK -- long story, boring business-related stuff -- so (a) is ruled out. (b) has happened in the past -- it worked for ACCELERANDO -- and might happen again in the future, but basically it ain't happening for HALTING STATE because one publisher was a bit doubtful about the original book proposal and didn't make a satisfactory offer for the rights until they had the final thing in front of them, which left them running behind the other publisher's schedule.

Jakob: it's not so much individual punters, but if, say, a big name book chain begins importing the US hardcover in large quantities, they will correspondingly decrease the size of their order for the UK hardcover or trade paperback when it finally emerges, blinking, looking at its pocket watch, and shouting "I'm late!" If that happens too often, the British publisher is screwed, and shortly thereafter (as of the next book) I'm screwed too. (The comments re rip-off Britain cut both ways, and I make more money per UK book sold than per US book sold.)

On the topic of Scots dialect, you need to bear in mind that the reviewer is American, and just throwing in the odd "crivens!" is enough to give it that quaint ethnic flavour. Alternatively, I am no Irvine Welsh (or Iain Banks) and I'm not pretending to be.

(On the other hand, I'd love to aspire to the status of a messy collision between Neal Stephenson and Christopher Brookmyre. There ya go ...)

Nix: mathematics almost certainly is endless, but whether endless mathematics remains interesting in and of itself is another question. (Is the end of "Diaspora" by Greg Egan heaven or hell for the protagonist? My guess is that it's meant to be heaven ... but the protagonist is very, very unlike most of us Version 1.0 Humans.)

Posted by: Charlie Stross | July 10, 2007 3:46 PM

Ah, I see. Thanks for the info - I will feel no guilt in asking one of my mates to mule me a copy when they're next over the pond.

The thing that amused me in that review was the description of DI Dalziel in the telly version of Dalziel and Pascoe speaking 'thick Scots English'... somewhere a white rose is weeping.

Posted by: Jakob | July 10, 2007 3:55 PM

Could you please post a mirror of your article on your own site. The BBC is blocked pretty impossibly by the Great Firewall of China, none of the proxies I've tried work.

Posted by: albert | July 10, 2007 5:10 PM

Albert @25: I'll go ask, but as it was a paid commission, they've got first serial rights for sure -- I may be able to repost it later. (In the meantime, it's basically a much shorter, low-brow version of this.)

Posted by: Charlie Stross | July 10, 2007 5:20 PM

Good article, my main problem is a pedantic one and that's the prefixes and conversions of bytes are all wrong. The IEC international standard requires binary prefixes for binary numbers not the SI decimal prefixes which are defined incorrectly anyway in the article.

Posted by: Mobius | July 10, 2007 5:37 PM

21: There are many generative sciences -- ie, sciences with unbounded objects of study. Right off the top of my head there is chemistry, biology, and computer science. These sciences are by their very nature inexhaustible.

Concretely, take chemistry. You can always synthesize new chemicals, and they will have new properties. And no, we will never become able to predict the properties of any possible chemical in advance -- computational complexity considerations rule that out. Even the transcendent AIs on the other side of the Singularity will have to get their robotic manipulators dirty with labwork.

Posted by: Neel Krishnaswami | July 10, 2007 5:43 PM

You might want to talk to your reviewer about this:

"Detective Chief Inspector Andy Dalziel speaks rather thick Scots English"

Andy Dalziel speaks (On TV at least) pure Yorkshire. (Or "pure bloody Yorkshire" as he would put it.)

Posted by: Anthony Cunningham | July 10, 2007 5:48 PM

Look forward to reading the new book. As a US based reader, I must say that your books are not always available in the major Silicon Valley bookstores - Barnes and Noble and Borders. I saw Marc Andresen put you at near the head of his favorite authors on his blog, which I think should give you a small boost here, but for whatever reason, they don't stock many, leaving Amazon as the best route to buy (not that I am complaining of this, but showcasing and spot purchases are important to gain readership). I know you are not unique in this regards, and British authors do seem to get poor exposure here. Maybe that is the inevitable price we have paid for the big box bookstores.

Posted by: Alex Tolley | July 10, 2007 6:07 PM

Alex @30, I'm published in the US first -- my agent's in New York and sells to Ace and Tor before the UK get my stuff. In fact, the British publishing biz basically treats me as a US import.

This suggests a problem with your local B&N and Borders, not my US publishers.

Posted by: Charlie Stross | July 10, 2007 7:06 PM

Congratulations, Charlie. Being compared to John Brunner is high praise indeed (and well-deserved to my mind, after Glasshouse). I'm getting a pre-order for Halting State in later today.

Nix: sorry, it's not true that we can't build a phone camera that's as good as the human eye. We just can't do it the same way we build normal cameras. and we can't match all the capabilities in one camera cheaply (now). Our cameras are uniform geometry sensors; they have the same resolution over the entire image frame. The human eye has a high resolution (but definitely not higher than a very expensive professional digital camera back, some of which run up to 40 megapixels) at the center of vision and is almost blind at the edges (motion-detection only). As for sensitivity, the human eye is capable of very fine discrimination of shades in the green, but very poor discrimination in the red and violet (and image resolution varies in the same way by color). So it's possible to do as well, but matching the color response is difficult with solid state sensors. It's simpler, though very expensive, to just have the same discrimination across the board, and modify the output image with a filter to match the eye. And so on, for light response and dynamic range; I don't want to write a textbook here and bore you all to tears.

As for the end of science, there's another way it can go: we don't know everything, but everything yet to be discovered is too complex for human brains and only the AIs can do science. Not a new idea, I know, but I think it still bears thinking about. Would that engender a racial inferiority complex, as some have suggested? Maybe not; not that many people really care about science.

Posted by: Bruce Cohen, SpeakerToManagers | July 10, 2007 7:09 PM

Charlie: I'll look into it and find out how the local purchasing decisions are made. We should be in SciFi heaven here, so I would expect a good selection of your works on the shelves, especially when they hit pb.

I note that Amazon has the publish dates for "Halting State" in the US and UK as the same, Oct 2 2007 date, but it looks like they are for the same US edition.

Paradoxically, Greg Bear's "Quantico" was published in the UK nearly 6 months before the US release. Go figure.

Posted by: Alex Tolley | July 10, 2007 7:38 PM

@32:

> Would that engender a racial inferiority complex, as some have suggested?

Thre was very good book by Soviet writers Arkady and Boris Strugatsky on the subject (
http://en.wikipedia.org/wiki/Arkady_and_Boris_Strugatsky), published in US as "The Time Wanderers" ( http://www.amazon.com/Time-Wanderers-Arkady-Strugatsky/dp/0312910207/ref=sr_1_1/102-1901546-7993702?ie=UTF8&s=books&qid=1184093498&sr=1-1 )

(pls disregard "Editorial review". They have no clue)

I have no idea how good translation is.

I am waiting impatiently for "Halting State".

Posted by: Alexey Goldin | July 10, 2007 7:57 PM

Alex: Amazon.co.uk shouldn't be listing the US edition for import into the UK. And the UK edition is a trade paperback due out in March 2008.

Posted by: Charlie Stross | July 10, 2007 7:58 PM

Charlie @11:
Alex: Mammoth babies? I can smell the toxic waste from here! And you wouldn't want to be in earshot when they start crying because they've just tried teething on the Walter Scott Monument ...

Isn't that the mammoth from "Trunk and Disorderly"? I bet it would make a great sequel if you also included NINJAS IN SPAAAACCEE!!!

Posted by: Soon Lee | July 10, 2007 8:12 PM

No, no, no, it's mammoth as an adjective, not a noun!

Posted by: Charlie Stross | July 10, 2007 8:21 PM

Kitchen sink approach. How about: Mammoth babies AND baby mammoths AND NINJAS IN SPAAAACCEE!!!

Posted by: Soon Lee | July 10, 2007 8:49 PM

Regarding the storage mechanism. On the way to your hypothetical c12/c13 carbon lattice, we could store the data at the molecular level very soon, using DNA. The A/T =0, C/G=1 base pairs could be used. What is interesting is that MIT is already working on DNA sequencers that detect the size the nucleotide as it is forced through a pore. Controlling the molecular machinery to construct the DNA sequence is quite possible, providing nice linear "tapes".

Putting that into perspective. A human genome contains 3*10^9 bases ~= 3*10E8 bytes. A human has ~ 50*10E12 cells. Using your 2003 data storage values, I calculate that this could be stored in 1/100 of a gram of cellular material. Nowhere near a grain of sand, but pretty small and it might even be stored as part of yourself.

Posted by: Alex Tolley | July 10, 2007 9:15 PM

Alex T: yes, DNA is a feasible storage medium. On the minus side: (a) it's held together with hydrogen bonds (read: it's not terribly stable if you heat it up), (b) nucleotide bases mass about 325 Daltons (compared to 12.5 Daltons for a C12/C13 mix) so it's about 25 times as massive per unit data storage, and (c) it's prone to transcription errors (can't give you an exact error frequency, but as it's one of the mechanisms by which mutation occurs ...).

The flip side is, we might be able to improve on nature's transcription and error checking mechanisms, and we might be able to improve on the standard base pairs in DNA by using some other nucleotide-like system.

Posted by: Charlie Stross | July 10, 2007 10:20 PM

charlie: I was trained in biology and worked for many years in biotech. All your points are absolutely correct. I just wanted to offer a solution that might be quicker in coming that trying to read isotopes in a lattice, plus a possible read mechanism as well - although I well understood that yours was a concretized thought experiment to anchor the possibilities of storage at this dense level. I had read your "Shaping the Future" address and recognized the basis of the Accelerando computronium theme.

Unlike starships, or even human planetary exploration, this increasing density of storage and computation is coming up on us extremely fast. (I'm so old, I can hardly believe that 1GB flashRAM SDcards are old hat - I started on punch cards on an ICL 1905 at the University of London). How does one even start to plan how to use this stuff? As a programmer, I'm already starting to shift my sights to functional programming in order to use the capabilities of hugely more processing power and storage. Won't be long before I need "future shock" therapy.

Posted by: Alex Tolley | July 10, 2007 11:24 PM

Charlie, I'd say you can scrap the "might", at least as long as you consider the mechanisms employed in humans. Fault-free transcription would be quickly eradicated by selection for flexibility. However, what we are looking for might be found in old organisms that haven't changed much in the last hundred million years or so. (Their innate flexibility and resistence might have provided them with enough time to bend the rules of selection a bit.) Cockroaches are likely to be a good starting point.

One thing to worry about might be genetic modifications of human beings in order to prevent mutations and cancer in that way. Those might be a boon for all individuals, but might be a curse for humanity as a whole, should the lack of genetic flexibility backfire. But then again, once human genetic modification is out of the box, the lost flexibility would probably be made up by human ingenuity, rather than natural trail and error. The genetic make up of humanity might however change the way that names change today, following trends and fads, booms and bursts.

Posted by: tp1024 | July 10, 2007 11:32 PM

OK, vertebrate biologist in the house :)

Bruce @32: what on earth are you smoking? And why, like most geeks, are you taking a digital camera as some sort of standard for image quality? Even a 40 megapixel camera is only on a par with ordinary film for resolution, and its dynamic range is much worse than film (and likely to remain so in the foreseeable future), which is turn has far less dynamic range than a retina. Almost blind at the edge of our visual field? Rubbish. The edge of our visual field is primarily formed by rod cells, which only see grayscale, but are way more sensitive to low light and movement than the cone cells at the centre of our visual field. The lack of rod cells at the centre of our retina is part of the reason we have crap night vision (and I hasten to point out that our low-light vision is still better than most cameras, even though we've got nothing on a seal).

What's more, the nerves in our retina perform all sorts of on-the-spot tricks to enhance contrast and accentuate edges (and it took the visual psychologists forever to realise that most of that stuff happens in the eye, rather than in the brain). If you want to see really sensitive to motion, check out what insect eyes can do. I'm sure insect eyes would be a better model for machine vision, actually, but that's a digression :)

Charile @40: The error rate for DNA transcription is very low, and there's a whole lot of error-correction stuff hanging around chromosomes. Environmental things are a far larger source of mutations (free radicals, UV light, even cosmic rays). Although, as someone else pointed out, we're not descended from the organisms that did the best job of keeping their genome stable. People have calculated the lifespan of DNA, based purely on entropy and such; it's in the hundreds of thousands of years, in ideal conditions. I suspect that DNA information storage implies a rather different future to the one you're thinking of, though. Have you read Starhammer, by the way?

Posted by: Chris | July 11, 2007 12:17 AM

Using nanoscale diamond as data storage, six hundred grams (about one and a quarter pounds, if you're my generation) can store a lifelog, a video and audio channel, with running transcript and search index, for six billion human beings for one year.

I saw this movie. It was called Zardoz.

If we end up with a big flying head that pukes guns and James Bond running around in a yellow diaper I'm blaming it all on you, Stross.

Posted by: Dan Flanery | July 11, 2007 12:20 AM

Chris @ 43

I used to work in a physiology graduate research department at a university, so I'm not just a silicon geek. And I was talking about digital cameras, not because they're the best thing around (I used a 35 mm film SLR for 37 years before I bought a digital) but because we were talking about surveillance cameras feeding shape and motion recognition systems.

Almost blind at the edge of our visual field?

Yes, the edges of the eyes have very little spatial resolution. In fact they have about none because they do detect motion, not shape. They add almost nothing to the image you perceive of the visual field. If you detect motion at the edges, your brain will move your eyes and head to put the source of the motion nearer to the center of the field so you can get an image.

Ok, if you want to include the entire visual cortex as part of the eye, then sure, it's much better than any camera. But then I get to add as much visual processing hardware and software as I want. As I said, we can do as well, we just can't do it cheaply.

Even a 40 megapixel camera is only on a par with ordinary film for resolution, and its dynamic range is much worse than film (and likely to remain so in the foreseeable future),

And the eye is not as good at resolution as film. There are what, about 100 million receptors in the retina counting both rods and cones? And as I said the receptors at the edge don't add much to the image. I've seen 100 megapixel sensors; they're not common because they're expensive at the moment, but that won't last. Remember Moore's Law before you talk about the forseeable future.

As for dynamic range, one of the things that makes the eye so good is the iris; put an iris on a camera and you've got a similar mechanism. And I know at least 3 ways to increase the dynamic range, again by pumping money into the sensor:

1. Make the image array with several different sizes (and therefore different sensitivities) of pickup cell. That can multiply the dynamic range by the ratio of the areas of the largest and smallest cells. My digital camera allows me to do something similar by shooting 2 frames at different exposures and melding them together with software.

2. Don't use CMOS sensors. They're standard in large image sensors because they're relatively cheap to make. But they have a high black noise current, which puts a floor on the lowest light level you can reliably detect. CCDs are less noisy, and it's possible to tune them for dynamic range by adjusting the geometry of the cell and of the holding cells that store the image.

3. Cool the sensor. This also reduces noise. Stick a Peltier-effect cooler on the back of the sensor chip and cool it down to 150 Kelvin or so. Noise current is exponential with temparature, so you get a lot of help there.

(and I hasten to point out that our low-light vision is still better than most cameras, even though we've got nothing on a seal).

Right, "most". There are cameras that can detect single photons. You can't do better than that.

and it took the visual psychologists forever to realise that most of that stuff happens in the eye, rather than in the brain

And it doesn't need to be done in the camera if we've got sufficient cpu cycles or bespoke parallel processing to massage the images once we've got 'em (in real time if we need to). It's better to do it in the eye than the brain because of the trade-offs resulting from the design of the neurons in the central nervous system and the hard realtime requirements of the sensory system. Silicon has different trade-offs.

Posted by: Bruce Cohen (SpeakerToManagers) | July 11, 2007 7:44 AM

If you're interested in the end result of that comment of Charlie's about students and note-taking, watch the movie "Real Genius". There's a sequence in the movie depicting an arms race between students and teachers: the students carry audio recorders so they won't have to take written notes, then the teachers record their lectures and play them back to the students during class, and then the students just drop their recorders off in class to get the lecture, and go off somewhere else. The total automation of education?

Posted by: Bruce Cohen (SpeakerToManagers) | July 11, 2007 7:53 AM

Pwned :)

I'm not sure that the deal with edges of our visual field is as clear as you're making it out, though. There's nothing stopping rod cells from forming an image, but the eye just isn't focussed that way (for good evolutionary reasons, probably; 'tis better to just move if you see something coming towards you from the side, rather than try to figure out exactly what it is). Cameras do have an iris, too.

I can't remember how few photons a retina cell needs to fire, but it's down to fairly small numbers of discrete quanta. Things have been hunting each other in the dark on this planet for a very long time, after all.

No bite on the compound eye? Some insects have as many visual elements as a cheap digital camera has pixels.

Posted by: Chris | July 11, 2007 7:59 AM

Alex@33: Quantico was released first in the UK because Bear's US publishers refused to publish - too controversial a subject matter at the time in the US. It was good for the company I work for because we sold loads of copies of it back to the US.

Posted by: gmilton | July 11, 2007 8:05 AM

Wait wait now...sorry to go all thick (Wed' mornings affect me like Mondays).

So it there a UK Hardback? I'm all luddite with books and like to have nice leathery looking dead tree in my bookcase. I'll post you a cheque for £5 if it helps, just to make up the shortfall for buying a US edition (or I'll buy a friend the paperback ;))

Posted by: Serraphin | July 11, 2007 9:13 AM

Serraphin: there's no UK hardback of HALTING STATE. Just a trade paperback. (There is going to be a leatherbound, gilt-trimmed Easton Press special hardback collector's edition in the US, just to annoy everyone -- I don't enjoy signing a stack of frontspieces as high as my knee -- but that's not a retail edition. So if you specifically want a hardcover, you're stuck with the Ace import via amazon.com.)

Posted by: Charlie Stross | July 11, 2007 9:41 AM

One of each it is then (Support your author and all that).

I expect the sames treatment when I 'make it big'©

Thanks Charlie

Posted by: Serraphin | July 11, 2007 10:03 AM

Eukaryotic DNA has an approx 1/1x10^8 base probability of replication errors, of which circa 99% are fixed by DNA repair enzymes, leading to a `final' error rate of about 1/1x10^10 per replication.

This seems small until you realise that it means that each of us contains a few hundred unique mutations (mostly thanks to our dads, and varying depending on our father's age at our conception: spermatogonia replicate about once every 14 days...)

It's amazingly reliable for a solute-chemistry system running at the temperatures it does, but it's not exactly RAM-chip reliable: if DNA wasn't as tolerant of coding errors as it is we'd have no chance. (Of course it can be made more reliable, viz D. radiodurans, but that has enormous metabolic costs: poor old radiodurans gets outcompeted by just about everything except in the nastiest of dessicating or radioactive environments because it spends so *much* effort on DNA repair and RAID-array-like error checking.)

Larry Moran just wrote a good article on this, from which I snarfed most of the figures above:

Posted by: Nix | July 11, 2007 10:51 AM

Oh, and what makes you think cockroaches haven't changed much? Arthropods may not change much *morphologically* (not true for all arthropods), but they experience far higher rates of genetic change than, say, vertebrates. The molecular data are pretty unambiguous here, especially now that we have some non-vertebrate, non-arthropod metazoan outgroups to compare to.

We don't have explicit data on the cockroach, but we do have explicit data on several other arthropods, and they've all discarded and changed a lot more ancestral genes than have vertebrates. If you want a `genetic living fossil', you'd do better looking at *us*, but better yet looking at extremophile bacteria (like, well, D. radiodurans).

Posted by: Nix | July 11, 2007 11:54 AM

C.S. Regarding Greg Bear's Quantico: Would his US publisher really be that sensitive? I heard he didn't get a hard-back US deal until the numbers were in from a British publisher. Seems Bear's last few hard-backs didn't do so well. I like Bear, but didn't read Quantico, a place near and not-so-dear to my heart.

Jeff

Posted by: Jeff Minor | July 11, 2007 2:04 PM

Well I don't have any links to it but I did research this when the UK edition came out and got the impression the decision was political.

Posted by: gmilton | July 11, 2007 2:10 PM

It won't be the vast amount of data that will be useful, but the ability to organize it ans sift through it and make connections. The librarian's mind and the patternmaster mind will be hugely in demand. However, the patternmaster's mind gone wild will also find or make patterns in the data where there are none, and conspiracy theories will spring up like tumbleweeds. Welcome to another sort of Brave New World - make it into an all-time best-selling novel?

Posted by: Patricia Mathews | July 11, 2007 2:50 PM

Chris @ 47

There's nothing stopping rod cells from forming an image, but the eye just isn't focussed that way (for good evolutionary reasons,

Agreed, but I think some of the difference between the edge and the center of the visual field is just a good tradeoff to get high resolution at the center from a somewhat challenged lens and a sub-optimal retina design*. And the tradeoff is especially good because of the feedback mechanisms in the CNS that control focus of attention: highly-stereotyped and efficient motion-detection causes automatic shift of attention, bringing the high-resolution, high-processing power parts of the visual system to bear, and engaging the higher-level systems beyond visual processing to examine the results. That's the kind of tradeoff you can make with electronics too: correct for flaws and limitations in the sensor by adding lots of clever post-processing.

I can't remember how few photons a retina cell needs to fire, but it's down to fairly small numbers of discrete quanta

This paper claims a quantum efficiency for the human eye of about 10%, which certainly sounds reasonable.

* What was the design committee thinking when they put the retina on back-end to the light?

Posted by: Bruce Cohen (SpeakerToManagers) | July 11, 2007 3:14 PM

ghamiltion, Jeff Minor: Thanks for the info on Quantico. I met Greg Bear last summer and had a chance to ask him about Quantico's delay. He was somewhat vague about the explanation, and subsequent events showed that even his hoped for US release date was optimistic.

Clearly his fan base is not wildly happy about his shifting direction, but I gather the SciFi market is not good and almost necessitates moving to a wider audience. He and Bob Sawyer talked about the awful "War Porn" sub-genre that is taking up SciFi shelf space. I can certainly detect the shrinking shelf-space B&N and Borders devote to SciFi (and science too) in favor of other categories.

Posted by: Alex Tolley | July 11, 2007 3:27 PM

The librarian's mind and the patternmaster mind will be hugely in demand. However, the patternmaster's mind gone wild will also find or make patterns in the data where there are none, and conspiracy theories will spring up like tumbleweeds.

There's this thing called the Internet..

Posted by: Alex | July 11, 2007 3:34 PM

Yes and even on the sci-fi shelves it's mostly TV spin offs & huge amounts of crappy derivative swords & sorcery fantasy.

Posted by: gmilton | July 11, 2007 3:34 PM

DNA, and other, error prone systems for storage: Compensation for random errors can be done using redundancy and polling for consensus answers. The total storage requirements just increases.

Of course, DNA is highly redundant in both organisms and species, so random errors in the germ cells of individuals is needed for evolution, a requirement that is not part of the posited stable, information storage device. Of course simple minerals have a much longer shelf life.

In practice, data storage is not permanent. I was reading that Google captures the last updated web page and caches the previous version, the previous cached version being destroyed. Thus modifications to any data can be deliberately or accidentally introduced with each page update (analogous to each biological cell generation).

Posted by: Alex Tolley | July 11, 2007 3:51 PM

@45

Very nice summary, I'll save it for future perusal :-)

Here is another argument: how often looking at digital picture you notice something you have not seen when you took a picture? Obviously, camera picks something that is not part of your experience of the scene. Even lousy old 2Mpixel consumer camera.

Obviously, staring at picture is not the same as being there. But I do not think "storing lifelong experience" is the same as "ability to recreate any experience with 100% fidelity".

Hey, we are talking about using one atom per bit for data stoarge and arguing about silly megapixels :-). I expect digital cameras to progress a bit. And compression as well.

This might be a relevant link (dealing with text rather then images):

http://prize.hutter1.net/

Posted by: Alexey Goldin | July 11, 2007 3:52 PM

Re: US rights. I've just read an article in the trade rag 'The Bookseller': "Amazon.co.uk gives publishers leave to remove US editions".

Apparently they have introduced a new system enabling publishers to remove infringing US editions from the website, following pressure from the PA (Publishers' Association I presume). This seems to be following a recent legal ruling about similar shenanigans with CDs.

Seems like we'll have to wait until January after all.

Posted by: gmilton | July 11, 2007 4:03 PM

gmilton@60 yes there is a lot of dreck on the shelves. However I have to assume the big box bookstores are rational and maximizing $/floor area. just like any retailer. If I was a buyer, I would use Amazon's ranks as a primary guide to buying and then regionalize/localize to taste. So I assume the SciFi dreck sells well.

Having said that, a couple of weeks after Andresen blogged his favorite SciFi authors ( http://blog.pmarca.com/books/index.html ) I wanted to buy a selection for my vacation. Amazingly my local B&N did not have even 50% of the key titles, and in one case, not even the author (Ken MacLeod) on the shelf. It's possible this was due to a run on the books, but I have seen this before when searching for a particular author or title.

I think Stross is making headway - Glasshouse was displayed in B&N's "new releases" section on publication (although I don't recall seeing the The Jennifer Morgue), but the shelves are very sparse regarding his other titles (and TJM was not among them as of 3 weeks ago).

The problem for authors is that bookstores are showcases to quickly find interesting books, even while Amazon may account for a significant fraction of sales if you can wait a week or two. Thus their inventory is like advertising, reaching out to the casual customer. I value that, so I do continue to buy some retail (for now). Amazon is improving the experience for finding interesting books, but it will never be able to supply the demand for "get it now", and it is spotty (not their fault) for viewing the content and samples.

Hint to Charlie: "The Jennifer Morgue" and "Toast" do not have "search inside". Neither do the upcoming "Missile Gap" and "Merchant's War: 4". That is the publisher's responsibility, no? I cannot stress how important this feature is to me when buying w/o recommendations on Amazon (although more so with expensive technical books).

Posted by: Alex Tolley | July 11, 2007 5:39 PM

nix @53:

What makes me think that cockroaches haven't changed much genetically?

Well, my ignorance of molecular biology and the factoid that cockroaches have been around in pretty similar phenotypes for quite some time.

As I said, this was just a guess in complex organisms. Microbes would have been my next one, but since I won't be involved in any such thing it would not have mattered much.

Posted by: tp1024 | July 11, 2007 5:47 PM

tp1024@53 I'm afraid you'd be wrong on the microbes too. They change very quickly. What you have to realize is that some genes are highly conserved, especially in the DNA sequences that are critical to their functionality. Other genes less so. What is worse is that microbes can acquire genes from other microbes directly, bypassing the hereditary requirement for higher organisms. A good example is the acquired drug resistance of some bacteria, like Staphylococcus Aureus, even though morphologically they don't look any different than when we first observed them.

Posted by: Alex Tolley | July 11, 2007 6:08 PM

Alex T @64: "The Jennifer Morgue" is currently only available in hardcover from a small publisher, Golden Gryphon, who do not have high street distribution. A trade paperback edition will be forthcoming in the US from Ace around the end of the year, and a paperback is coming out in the UK from Orbit around September 1st -- both of which should be widely available in their respective countries.

In the USA, "Glasshouse" came out in hardcover last July, and a mass market paperback is due imminently (within about four weeks), so B&N have probably pulled any remaining HC stock from their shelves and will have the paperback on order.

"Missile Gap" was published by a smaller publisher (again: no chain bookstore distribution) and has sold out after two reprintings. You can now read it on the web.

"The Merchants War" isn't due to be published until November 1st, so it'd be kind of surprising if there was an Amazon "search inside" at this point in time.

Finally, about Marc Andreesen's list: it covers what are in his opinion the best authors of the new century. That's going back seven years at this point. Publishing is seasonal, and bookstores don't like to carry lots of old stock; some of the titles Marc recommended date to 2000 (or earlier -- some of Ken's date to the late 90s) and are simply unavailable right now in high street stores.

Amazon, AbeBooks, or your local SF specialist bookstore are where to go if you have a specific shopping list.

Posted by: Charlie Stross | July 11, 2007 6:09 PM

Alex Tolley @ 66

What you have to realize is that some genes are highly conserved, especially in the DNA sequences that are critical to their functionality

Which is why arthropod phenotypes seem so stable to the casual observer. Mostly people see morphological features, and those are stable (IIRC) due to the body plan strategy arthropods use: segments generated by patterned invocation of the homeobox genes, which are very strongly conserved.

Vertebrate morphology is much less stable; the cetaceans had legs only 40 or 50 million years ago. That's why morphology isn't a good way to estimate either genetic relationships or genetic drift.

Posted by: Bruce Cohen, SpeakerToManagers | July 11, 2007 6:38 PM

Charlie: Thanks for the info. You'll be pleased to know that I have you on my "hardcover" list - authors whose works are worth buying that way for the library at home.

I'm not quite sure what you mean by "seasonal". Asimov is always well represented, Heinlein is going through a mini-revival (although we may be talking about reprints) and Clarke, whilst declining very quickly now, still has some of his classics on the shelves. There are no end of Star Wars and Star Trek titles, some very old now. Surely what sells is important. My point about Andreessen (we both spelled it incorrectly) is that he is a geek and his list was well circulated, so I would have guessed that any titles on his list would have enjoyed a minor blip. Sure, Amazon and ABEbooks are great for new and used books respectively, but they are no help when shopping for titles a day or so before heading off for vacation. And amazingly, there isn't a good SF specialty store on the Peninsula AFAIK south of SF or Berkeley - certainly nothing like "Forbidden Planet" or the defunct, but better, "Dark They Were...".

Posted by: Alex Tolley | July 11, 2007 6:54 PM

How many Asimovs, Heinleins, and Clarkes are there?

Most authors in the genre field -- the vast majority, being about 98% of us -- are here today, gone tomorrow. The aforementioned three are superstars; citing them as evidence that book-buyers in the big chains don't churn their stock seasonally is like pointing to David Bowie and The Rolling Stones as evidence that rock musicians are all rich.

In reality, 70% of the sales of a new hardcover or mass market paperback in the US market take place within 90 days of its launch; if it's still in print a year later, the author's doing well. And the bookshops gear their product turnover to this cycle.

Posted by: Charlie Stross | July 11, 2007 7:01 PM

bruce@66 - you are speaking to the choir. Conversely, the convergence of shark morphology led many to believe sharks haven't evolved, whilst in fact they have widely speciated and gone through the usual extinction patterns since their emergence.

Posted by: Alex Tolley | July 11, 2007 7:02 PM

Charlie@70 --

I recently had a lot of trouble finding stories by Henry Kuttner for my kid to read. I liked them a lot when I was younger but would have no luck buying a book (or even finding it in Chicago Public Library) if not for the recent movie losely based on one of his stories. He is definetely a very underappreciated author.

I wonder if this phenomena is specifically American. I talked to many guys in US who love SF but do not know about Kuttner or Robert Sheckley. Reaction from Canadians: "of course I know his books!". The same responce from Mexicans or French. His books are still easy to buy in Russian translation...

Posted by: Alexey Goldin | July 11, 2007 7:17 PM

Charlie - Poor choice of authors on my part - although I suspect Clarke may almost completely disappear from the shelves soon over here.

I was aware of the sales loading, and I have read that it is getting even more concentrated, much like movies. However, I do peruse titles frequently enough, and am sufficiently aware of title age, to note that older titles do seem to be on the shelves - which I assume is due to longevity of sales demand. We're not saying anything different. Most titles are ephemeral, a few have longevity and stores plan their inventory based on their expectations and actual sales.

What I wasn't aware of, based on what I think you said, is how little flexibility chain stores have in adjusting to high frequency events. I guess that is Amazon's good fortune.

Posted by: Alex Tolley | July 11, 2007 7:24 PM

72: The Science Fiction Book Club has a reasonably priced edition of Two-Handed Engine.

Posted by: James Nicoll | July 11, 2007 7:27 PM

Alex@58: Compare Michael Marshall Smith moving to "paranoia-porn" via his Michael Marshall titles.

Posted by: Mike | July 11, 2007 10:17 PM

Bruce @57: For retinas the right was around (and a whole bunch of other obvious anatomical improvements), let me offer you the cephalopods.

Posted by: Chris | July 11, 2007 11:24 PM

James@74:

Thanks, I already bought another book, but sfbc look interesting. I did not know about it.

Posted by: Alexey Goldin | July 12, 2007 12:18 PM

Charlie

I've seen Glasshouse (mass-market edition) on the shelf within the last two weeks at my friendly big-box book store in Los Angeles.

Posted by: P J Evans | July 12, 2007 4:29 PM

PJE: it takes a while for them to filter out into the wild.

Posted by: Charlie Stross | July 12, 2007 4:37 PM

Is there a link to the Radio Wales audio of the interview? I have looked for it, quite unsuccessfully, on the BBC web site.

Posted by: Alex Tolley | July 12, 2007 5:58 PM

Bruce @46:

It's already happening. In a number of tertiary institutions, lectures (mainly Power Points) are put online (intraweb) for students to access, so during lectures, they can concentrate on learning rather than scribbling notes.

Some may forgo turning up to lectures and depend on e-lectures alone, but the sensible ones still turn up to lectures, and get a greater learning experience. Nothing (yet) replaces face-time & RL interaction.

Posted by: Soon Lee | July 12, 2007 10:27 PM

My entire university course is online with the OU. I probably won't have to attend a lecture until the last stages.
All my course work is posted or available online, there are audio transcripts and MP3 files sent to me along with a CD full of study aids for each course.

I can communicate with a lecturer by e-mail or call him/her should I so need further assistance.

So in reality - we're there.

Posted by: Serraphin | July 13, 2007 8:38 AM

82 - Yes, but only a very small minority of OU programmes are entirely available online, and there's a lot of research to indicate that we (I work for the OU) get a lot of benefit out of face-to-face tuition. I'm still writing courses in boxes.

Charlie, have you read Tim Burke's response to yr article? It's online here:
http://hnn.us/blogs/entries/40740.html
It saved me the bother of writing a comment saying pretty much the same thing. To sum up: if I wanted to read every surviving record from England for any one year after about (waves hands) 1640 (and probably much earlier), I would die of old age before I could finish the job. Searchability is good, but to find out what was going on, you can't trust the index.

Posted by: Chris Williams | July 13, 2007 10:36 AM

Chris: that's where algorithms come in. Once this level of data is online you can do really funky stuff. For example: model social networks over time and see how the overlapping sparse networks we move in change. If you want to zoom in on a Man On A White Horse, you can backtrack through his timeline and see how his circle of friends changes and try to pinpoint just who it was who first turned him on to a specific ideology. Or if you're trying to monitor larger social-historical trends, you can track how much time people spend in different domestic activities over the years, and start looking for influences.

Of course you can't do the data reduction by hand. But once all the data is online, there's some fascinating stuff you can do with it that should reveal stuff about the past that's simply inaccessible today because we don't, for example, have exhaustive records indicating when, where, and who with everybody in England sat down for lunch during the 18th century so we can track trends in food preparation as they feed into issues of domestic service. And so on.

Posted by: Charlie Stross | July 13, 2007 11:10 AM

I agree about the data mining for social history. In fact, here's a nice (if I say so myself) example that I prepared earlier:
http://headheeb.blogmosis.com/archives/031384.html

It's the Man on the White Horse stuff that I'm rather more skeptical about. Traffic analysis can only get you so far: eventually you have to immerse yourself into his reality feed and see what he did. Then you need to repeat the process with his key associates, in order to find out who their key associates were, then...

People who suspect this might happen to them might go to lengths to counter it. Think about the 'Randy codes in prison' sequence from _Cryptonomicon_.

Ever tried to analyse oral history tapes, or cine/TV footage? I have and it takes bloody ages. Information overload.

BTW, I'm not trying to argue that this will change nothing, merely that I think that you're overestimating the impact of the change. AI agents would change everything, of course. But they'd want a share of citation index credit...

Posted by: Chris Williams | July 13, 2007 11:21 AM

I suggest you check out Google Video's techtalks/engedu stuff; Google puts on regular lecture events for their staff, where people from all over the place present some really interesting stuff (and they're filmed and put online for the public to watch too).
One I saw recently addresses the specific problem of searching lengthy video for 'interesting' events. The short form is that it tracks seperately moving elements in a scene over time, and packs them all down, overlaid, into one short sequence.
It's a bit like time-lapse footage, except multiple events happen simultaneously. The human brain can do quite a bit when it comes to prioritising multiple simultaneous inputs, one you've found something worthwhile, you can drill down to the original source video.
The time-overlay stuff is a bit tricky to explain, but one of the early examples takes footage of a water-cooler over a period of time. Various people come and go, getting drinks. The compression version shows everyone arriving and leaving at once. You can quickly pick out if one of those people is a Person Of Interest(tm), and bring up the original, linear-time footage to see who else was actually there at the same time.
Audio is always more difficult for humans to 'scan' because you need to listen to more of it to figure out if you've got to the right/interesting material; you can't "glimpse" it like a still from video. But audio is also more amenable to voice-recognition, automatic transcription, and thus machine-searchability. Sure, the transcriptions are kinda flawed now, but that's just a factor of time.
Even tricky issues like homonyms become easier in the face of vast amounts of data. Consider Google's translation software -- it learns languages automatically, based on statistical analysis of large bodies of work that are published in multiple languages (eg UN declarations). I think similar techniques will deal with homonyms over time.

Posted by: Canis | July 13, 2007 12:37 PM

It's the Man on the White Horse stuff that I'm rather more skeptical about. Traffic analysis can only get you so far: eventually you have to immerse yourself into his reality feed and see what he did. Then you need to repeat the process with his key associates, in order to find out who their key associates were, then...

Hang onto that thought, there's an entire novel plot in it. Subtype: SF, historical. (Protagonist is a historian, trying to understand a great upheaval from centuries before. Immerses self in reality stream of the pivotal character. Disturbing things begin to happen in the present: is history in danger repeating itself ...?) Damn, I could write that up as a pitch and sell it tomorrow.

Posted by: Charlie Stross | July 13, 2007 12:38 PM

Ta for the heads-up, Canis - I'll check it out. Remember though that face recognition (manual or auto) from video images is a difficult task, say my forensic psychology mates.

Posted by: Chris Williams | July 13, 2007 12:42 PM

"Hang onto that thought, there's an entire novel plot in it. Subtype: SF, historical. (Protagonist is a historian, trying to understand a great upheaval from centuries before. Immerses self in reality stream of the pivotal character. Disturbing things begin to happen in the present: is history in danger repeating itself ...?)"

Part of this plot was used in Walter Jon William's "Green Leopoard Plague", which is of course rather good.

Posted by: Stephen Shevlin | July 13, 2007 12:45 PM

Here's another: Cross wikipedia with Second life to make an immersive VR virtual history of everything we know. What a useful way to communicate the corpus of academic historical knowledge (a topic close to my heart: http://www.open2.net/historyandthearts/history/bridging.html)and to update it. Of course, it's going to be vulnerable to all sorts of serdarargic. Ghosts in the machine...

Posted by: Chris Williams | July 13, 2007 12:48 PM

Which us back rather neatly (I think) to the history/myth problem. In the early days of the Northern Ireland conflict a press photographer is reputed to have been hit with an umbrella by an old woman who told him, 'you're photographing things that aren't happening'.

Or as Richard Pryor put it, 'who are you going to trust? Me, or your own lying eyes?'

Posted by: D. O'Kane | July 13, 2007 2:00 PM

A historian friend of mine would be seriously delighted to have such a detailed access to the past, because so much that is relevant is hidden, lost or simply not recorded. It's Brin's "Transparent Society" in spades.
One can get a hint of what this will be like when watching very old movies - the details of life and the environment are almost alien to the modern viewer, details that are not generally recorded in novels or other "historical" documents.

As for navigating in this ocean of data - we are already seeing rudimentary signs of some aspects of this - geo and time mashups and it is only going to get much better as we get more data to play with. I don't doubt the ingenuity of people to develop the tools and make them accessible. Sterling has written about SPIMES which is conceptually similar - every artifact will create a wealth of data about itself over a lifetime. He uses the term "data wrangling" to handle it.

The technology is almost less interesting than the implications - who will record, how and where will records be kept, tampering, deliberate obfuscation, privacy and of course stability of storage and format changes. Some of these Charlie has written about, others have written about them too, even if the technology is different - eg "The Light of Other Days" by Clarke and Baxter. Will handling the data become a general skill like Googling, or will it be specialized? Will our lives require us to learn how to navigate it, or not?

My sense is that the eyeballs/attention problem will likely overwhelm humans pretty quickly and that tools and AIs will handle much of the grunt work. Which means that the storage volume will need CPU cycles to keep pace with it. So expect processing power everywhere too. Can those same high density substrates - like the carbon latices - also be used as processors or will we need something very different, but also at similar scales? I think we've been here too - "computronium". :-)

Posted by: Alex Tolley | July 13, 2007 2:51 PM

Charlie @87:

Sort of like here : http://qntm.org/responsibility ?

Alex @92: "who will record, how and where will records be kept, tampering, deliberate obfuscation, privacy and of course stability of storage and format changes"

And at some point someone will decide enough is enough and release Curious Yellow :-)

Posted by: Alexey Goldin | July 13, 2007 3:51 PM

I think the "Curious Yellow" topic is very interesting. How does it actually find the location of the information that it edits? I can see how a person's body could be digitized/mapped, but how does the CY find that info in the brain? Would those memories all have the same chemical signatures, or can the virus "play" all the memories in that digitized brain and selectively edit? It's a great idea, seeing how it gets into gate software. Would those gates just have to use microscopic mapping? Actual quantum states would have to be treated statistically, wouldn't they? I mean, you can't map a quantum state (of all the particles compossing the matter) unless it's frozen at absolute zero. Maybe.

Jeff

Posted by: Jeff Minor | July 13, 2007 4:43 PM

I totally agree that there will be *new* things we can do with the data that's flowing in now, and network analysis is prominently at the top of the list. But that data won't resolve any of the old epistemological problems with historical knowledge, and may aggravate some of them.

Posted by: Timothy Burke | July 13, 2007 5:24 PM

Alexey@92 : yes, "Curious Yellow" would be an issue. In a similar vein, Vernor Vinge has asked, are silicon chip processors a potential single point failure (eg EMP vulnerability)? Now in the world of store-everything-in-crystals, maybe the redundancy and resiliance of the substrate might protect us from data loss at least. But perhaps not to deliberate attack and tampering.

One of the biggest pivotal data losses in history is the deliberate destruction of the library at Alexandria, rather like Google's server farms going down today. Historians think more than was realized escaped, but even so, what a loss, and setback it was.

To mix this thread with the star flight thread: what if this technology could summarize much of human existence and used to seed the universe with our knowledge and exoperience? A very different play on the SETI approach.
Rather than assume contemporary civilizations' existence and halting communication (time and bandwidth), just send out tiny spores containing all data and a reader to all points in the galaxy and beyond. The receiver could view the lives of humans, and easily pore over all the records stored on the net today - stored on crystals the size of grains of sand. Most SF has used the cliche of a few, large, enigmatic ruins, much like those of our history. But what if the knowledge of a million civilizations was scattered about us, ubiquitously?

So here is my SF story themethought for the day. All around us, there are patterned crystals with massive amounts of data from civilizations, some of it in geologic layers. The "readers" only exist when we have sophisticated enough computers that can be hijacked by the S/W that comes with the data via RFID-like technology. In the near future, RFID scanners start detecting at their sensitivity threshold, some examples of this data, which soon turns into a flood...

Posted by: Alex Tolley | July 13, 2007 5:25 PM

Congrats on "Halting State" buzz. Can't wait!

Scotland: my son and I consider it our adopted country, via his Scotland-native mom (my wife). I see Charles Stross as Sir William Wallace in Cyberspace.

When I described, in a refereed paper of 1979, storage of all human data in diamonds, I did not yet have the data on pure Carbon-12 and pure Carbon-13 diamonds, which is now available. I did coin the term "Shannon" for a mole of bits.

"I suspect some sciences, notably mathematics, are intrinsically endless."

Hermann Weyl said that "Mathematics is the science of infinity."

G.H. Hardy's "A Mathematician's Apology", says that 2 + 2 = 4 is absolutely true, but anything in the real world is not as definite, so he thinks that the world of mathematics is more real than our world. This is a view that goes back to Plato.

Gregory Chaitin writes: " I have information-theoretic results on the limits of reasoning, and that leads me to think that to prove more you have to assume more, and this is a little bit more like the way that physicists work. Mathematicians think that you can start with a few self-evident principles and get to all of mathematical truth. Physicists don't think that. Physicists know that when you go to a new kind of phenomenon you need new physical laws to understand it. My own work says that mathematical truth has an infinite amount of information and any finite set of axioms only has a finite amount of information, therefore you have to add new axioms. Well, where are you going to get them? You have to work intuitively in a quasi-empirical way, it seems to me. In a pragmatic, empirical way, like a scientist does. At least that's my feeling.... I think that computers are changing the way we do science completely, and mathematics too. The computer can provide an enormous amplification of our own mental abilities, and it's really changing the way everything is done. George Johnson just had an essay on simulation in the New York Times in the Week in Review where he points out that now it doesn't matter what field of science you work in, the computer is fundamental in the work you do. [NYT 3/25/01, "In Silica Fertilization; All Science Is Computer Science"]

Posted by: Jonathan Vos Post | July 13, 2007 5:26 PM

Chris @76; I, for one, welcome our new cephalopodic overlords.

Posted by: Genghis 2.0 | July 13, 2007 6:18 PM

Charlie, are you aware of HT Buckle?

Social network analysis is exciting - I thought this over 15 years ago (blody hell I'm getting old) and checked it out. The conclusion that I quickly reached was that I'd have to write my MA using something else. The problem is that it's impossible to sample a network: 10% of the nodes yields only 1% of the connections. This makes it very hard to use to through historical sources. I don't see Strossian 'ubiquitous' data capture producing the kind of total information that you need to do it properly, merely a rather better assemblage of historical source material.

Posted by: Chris Williams | July 13, 2007 7:09 PM

PS - Ken M talks about Buckle here:
http://kenmacleod.blogspot.com/2004_04_01_kenmacleod_archive.html

PPS - do you remember the days when we thought that 100 was a lot of comments on an antipope thread? I recall them like, well, last week.

Posted by: