Re: Metacrap

A few objections to that article.

1) He specifically is referring to structured metadata, which is of artificially limited utility. His points are valid for that, but the scope of his thesis is as limited as structured metadata is. 2) I never thought the objective was "ubiquitous, complete metadata". I just thought the goal was to have things relatively self-describing. 3) Similarly, I didn't think the expectation was that it was ever meant to be completely accurate. "Accurate enough" seems good enough. Easily corrected seems more useful than always accurate. 4) Ideally, any metadata system wouldn't rely on humans to supply metadata. While humans could contribute, relevant metadata should be generated by the system, not the people.

In short, his article was intelligent and well written, but arguing against a vision that only the most extreme Semantic Wonks actually have.

Of course, I'm just a different flavor of extreme- I'm the one running around saying that data (and metadata) shouldn't be stored in a structured way, but in self describing heaps.

The MetaCrap discussion makes me think of one of themes of the last David Zindell "Neverness" novel: that even a universe made of computronium could not model itself -- not enough resources.

I also have a recurring thought, that we can have an infinitely expanding economy as long as its information-based. You can always derive more information aboht the existing information (or metadata about metadata).

I have recently loaded on my iPhone a pedometer app from trailguru.com. It records my walks and and bike rides. You upload it to their website ("meet single hikers in your area"), and then can get: complete stats including calories burned; a map where you can replay the walk; an area chart of your elevation profile; miles walked in each minute; minutes for each mile. Plus generate reports by activity, time period, etc. So, gigabytes every time I walk the dog ???

t3knomanser: you missed the Jacqui Smith reference, perhaps, or the following anectode about HMRC? (There's a specific British political context here ...)

@Charlie Stross: Smith is less a metadata issue and more a Fallacy of Omniscience issue. Just because you can log vast amounts of data doesn't mean you can synthesize it. I can see how it's applicable to the discussion of metadata, since metadata might help you organize the vast quantity of information. Realistically though, that's not the problem metadata was meant to solve.

As for the HMRC- yes, data is unclean. Ergo, metadata will also be unclean. That's to be expected.

@Chris Heinz: I refer to the modeling problem as the "Rucker Objection". He wrote an anti-singularity essay with that premise: why build a computer to model the Universe when we already have a model of the Universe at a 1:1 scale- the Universe itself.

There are a few things to keep in mind about that though: this assumes that the Universe isn't compressible; that is entropy is already maximized and that all models would be lossy or equal size. This may very well be true, but we don't know that.

Also, it supposes that a lossy model is somehow objectionable. While the Universe itself is full of important little details, if we were creating a habitable model of the Universe for human-type minds, we only have to present a model with the resolution of human senses. On demand, the resolution could be increased so that humans could use tools to observe low level phenomena, but for the vast scope of human experience, a lossy model is perfectly acceptable.

Could you put up a link, or some pictures of a visit to, the fabbing shop? Because that sounds mighty interesting.

I have some problems with the Cheeseburger numbers (which I hadn't seen before). Firstly it's comparing the total cost of the Cheeseburger to the amount of gas emitted from an SUV without taking into account the sunk costs of creating the SUV in the first place (which need to be amortized over the lifetime of the SUV).

Secondly, the numbers look bad for burgers simply because of the scale factor; 300 million people (his number) worth of burgers is equivalent to (approx) 10 million SUVs; ie per person the cheeseburger is only 3% of the cost of an SUV's emissions. Each and every person in the US would need to eat over 2000 burgers a year to match one SUV.

Or, since there are approx 16 million SUVs in Amercica (his numbers), the footprint caused by approx 5% of the population driving SUVs exceeds the total footprint caused by the whole of America eating burgers.

Yes, eating burgers looks bad but anything involving 300 million people looks bad!

About metadata...

Metadata are often used when computers are not (yet) able to process data that metadata describe.

Take for instance the problem of images. Computers handle very easily numbers and array of numbers (aka text strings), but cannot (yet) "recognize" pictures as humans do in the same manner they don't (yet) recognize the "meaning" of words they process.

Since a few years ago, the only way to handle a large database of picture was to tag them, to use metadata. If you wanted to find a picture, you searched for specific tag. Now there are more sophisticated piece of software that enable everyone to search pictures in the same manner they google words; I am referring to TinEye website.

IMHO, the goal should be to write better "semantic" software that processes data, not to optimize meatadata, even the latter is easier.

@Luigi Rosa: but the fact that picture X is a picture of a cat is metadata. Whether we've tagged it, or the computer uses image recognition to detect it, the fact that it's a picture of a cat is still metadata.

@t3knomanser: data and metadata can switch they role or be both in the same time (take for instance the title of a newspaper article). In a picture of a cat, the "cat" could be the datum just like a single word can be a datum in a newspaper article.

We risk to cross the border between information technology and philosophy :)

@Luigi Rosa: If the picture is data, the fact that the picture contains a cat is metadata. If the article is data, the headline is metadata. If, on the other hand, I was building a database of headlines, the headlines themselves would be data.

It's "data" that's tricky to define. What constitutes data is domain specific, but once we've identified what our "atom" of data is, everything we track about that atom is metadata.

Metadata has a delightfully straightforward definition: data about data. It's data itself that's a slippery concept.

Hence my Anti-Link proposal. People may not know what they know, or admit to what they know, but they can surely express what they hate.

It's been suggested before; I claim parallel evolution. But note that mine works the other way around; it's directly equivalent to an ordinary hyperlink.

Metadata is Not The Point.

Data Corruption is The Point.

The Point is the Future.

(Are you receiving? Over.)

Charlie, that may be the best poem you have ever written!

@Charlie Stross: Data corruption is a given. It's unfixable, and as omnipresent as the weather.

t3knomanser: Would you be implying that "absolute data corrupts absolutely?"

Or are we returning to a particle/wave argument?

For the heard of hearing- the UK gvt seems intent on herding us into a bureacratic nightmare of a surveillance society with capabilities which historical dictatorships have killed for. Not to mention wasting tens of billions of pounds on consultants and IT whilst doing so.

@Josh Robbins: I think that's a pretty good assesment. The more data you attempt to gather, the more corruption you will have. And I think the quantity/quality problem is one that scales geometrically: the quality of your data is inversely proportional to the quantity of your data raised to some exponent.

One of the problems I have with counting the greenhouse "emissions" of farm animals as opposed to just those of the industrial support system that gets them to our table is that they don't seem to take into account what would replace them in a natural setting.

There are roughly 100 million cattle in the US. As it happens, that's roughly the population of American Bison in the 19th century, before we tried to make them extinct for fun. Add to that the other wildlife we've cleared off in order to set up farms, etc. Even feed-lot raised animals are replacing the animals that were living on the land used to raise the food shipped to the feed lot.

I wouldn't be surprised if the emissions of domesticated animals was roughly equivalent to all of the wild animals we've replaced with our cities and farms.

Billions of pounds wasted on IT consultants? Damn, how can I get my fingers in that pie? It's a struggle to make more then the odd millions off the governemts oover here . . .

Billions of pounds wasted on IT consultants? Damn, how can I get my fingers in that pie? As an occasional IT Consultant myself I have to say it's a real struggle to make more then the odd millions from governments over here...

Oh well - there go my IT credentials up in smoke - couldn't even press "Preview" instead of "Post"... Just as well I don't have to type for a living.

Andrew G Looking at it from the other end, we are going to use the sunlight directly for photosynthesis and indirectly for irrigation (by way of evapotransmission, evaporation, clouds, rain, rainbows, etc) to grow cellulose, and the cellulose will be burnt, digested, or otherwise oxidised (except that portion thankfully stored away in anoxic landfills) and something is going to be emitted. Termites, cows, bison, rabbits, fungi, bacteria, wildfires (very promiscuous emitters of greenhouse organic compounds because they don't completely oxidise to carbon dioxide), and the very occasional person eating a cheeseburger. The numbers on cheeseburgers are not just bogus, they are irrelevant.

Re: #12: "The Point is the Future."

and #16: "... herding us into a bureacratic nightmare of a surveillance society with capabilities which historical dictatorships have killed for."

Given the recent breakthroughs in what software can deduce a person is thinking or intending or seeing (while their head is in a Functional MRI) raises the ante on what a surveillance society will be able to do, technologically.

Traditionally, what was in your head was relatively safe. Torture and coercion weren't very effective at winkling it out.

This is NOT necessarily true in the forseeable future. So, how much of a head start do we want to allow the governments of your evil empire or mine to have, regarding what historical dictatorships have killed for?

They [recent historical dictatorships] have tried labor-intensive approaches, where up to 10% of the adult population are informants or secret police. Now they are trying automated data-intensive approaches. However noisy or corrupt the data is, is a second-order consideration.

In the USA (what another Scotland-based SF author calls FU2, the USSR being Former Union1) about half of adults surveyed believe that (never mind those pesky Magna Carta or quaint Geneva Conventions) it's okay for cops to enter your automobile or home or snailmail or phonecalls or email without a warrant, so long as the government says that it's to stop drug dealers or terrorists.

JVP @ 23

so long as the government says that it's to stop drug dealers or terrorists.

You forgot pedophiles.

Apropos cheeseburgers, wkwillis: if you keep dismissing the energy inputs issue, sooner or later it's going to bite you. (Hint: most of the energy that goes into the cheeseburger comes out of the ground, not from sunlight landing on grass.)

Andrew G: there are alternatives to large ruminants -- even if you're in favour of farming animals for meat, there are edible animals that emit a far smaller amount of methane than the ones we traditionally use. (And last time I looked, the problem with kangaroos outside Australia was stopping the bloody things going native, not keeping them alive; there are wild wallabies in the UK.)

The real worry apropos @16 and @17 is that social standards change over time but there's potentially no end to data retention, and the data is innaccurate and sometimes just plain wrong. Put it another way: imagine there's a cultural shift over the next 30 years that makes a currently-tolerated practice intolerable and illegal (say, drinking real ale or driving while listening to the radio). Firstly, we're vulnerable to retroactive fishing expeditions. Secondly, the fishing expeditions will find dirt on a lot of people who were not, in fact, practitioners of the now-illegal practice (because the data corpus is inaccurate or corrupt).

Most of us aged over 25 haven't yet internalized the implications of everything we do permanently being On The Record; and those of us under 25 don't have any sense of what permanent means.

Nobody seems to be getting Charlie's point, or Cory Doctorow's point, for that matter.

Metadata won't work not because of accidental data entry errors or croggled bytes caused by some MySQL fubar...metadata won't work because humans will game the system when inputting the system.

The classic example of this kind of insanity is the TSA entering a 3-year-old's name in a database as a terrorist. That metadata is there forever. It can never be deleted. That 3-year old will have a "terrorist" data shadow following him around for the rest of his life, and as he keeps getting arrested and stopped and questioned, that crazy contrafactual data shadow will grow and grow and grow. Pretty soon he'll be stopped and questioned, or arrested, or detained, so many times, that a SWAT team will shotgun the hinges off his doors and burst into his bedroom and drag him away to Gitmo. Why? Someone labeled as a "terrorist" in the TSA database, and who has been detained for questioning so many times, must have something to hide. Right?

Right.

Metadata is garbage because people scam the database by making false entries. They do it to rickroll people into visiting their porn site, they do it to make their arrest count looks good, they do it to inflate the importance of their trivial scientific publication, they do it to crank up sales of their crappy products, they do it to weasel their way into a dating site even though they're still married.

Want to see the future of Jacqui Smith's insane database scheme?

Here it is.

Maryland political activists in groups like the Maryland Campaign to End the Death Penalty got entered into the Maryland state criminal database as "terrorists."

Expand that to the 60 million people living in Britain, and you begin to get an idea of what this insanity will mean.

The first law of a police state is: do not question the police's methods, they have your (the abstract "you") best interests at heart.

The second law of a police state is: everyone is guilty of something, if you look hard enough.

I'm wondering where all the metadata pollution is going to lead, a generation down the line, long after the tech-illiterate politicians who hired the contractors to implement it have retired to their gated communities.

@mclaren: That's Brasil.

One of the most fascinating books I know of on the effects of technological change on the exercise of control is, slightly unexpectedly, a work of naval history. Andrew Gordon's The Rules of the Game, ostensibly about the Battle of Jutland, turns into an analysis of the deleterious effects of the steam engine on naval thinking. The hidden hazard was that the speed of a sailing ship was essentially uncontrollable to anything beyond 'slow' or 'slightly less slow' and dependent on so many factors (wind, state of the sails, fouling of the hull, competence of the crew, etc.) that effective formation keeping was uncontrollable beyond 'stay in a straightish line' - this threw responsibility and authority right down to the captain of each individual ship - thereby encouraging the competence and independence of thought that any truly successful organisation requires.

The effect of steam power, however, was that it allowed the admiral of a fleet to precisely order the position and speed of ships under his command. This led to a destruction of initiative, obsession around the making pretty geometric patterns (rather than, say, accurate shooting [1]), and the illusion of control to the detriment of actual effectiveness. The application of this to any office should be obvious.

[1] Not helped by the RN only paying for one fresh coat of paint a year. One of the more subtle ways that captainship was kept to the independently wealthy was that warships of the time needed at least three or four coats an annum (more if firing exercises were undertaken) if they were to look what was considered appropriate for the time...

Slightly skewing but adding to the conversation, I actually had a conversation with someone who works as a custody assistant at the local police stations.

I couldn't seem to get through to them why the retaining of DNA and fingerprint records of people who were proved innocent was bad.

They stipulated that they were retained because it made life easier if you should ever do something in the future - and that (this was an exact opinion) had this been done properly in the past they would have caught Ian Huntly quicker.

So wait now - you're guilty till found innocent? Because this suggests that if I'm falsly arrested for something, then proven wholly completely innocent - the first time something similar happens locally I will be on the list of first people they drag in under suspicion.

They really believe that we're all guilty until proven otherwise (and actually honestly believed that 'in the US' this was how things worked).

I despair.

THIS INSANITY has been going on for some time - and the mainstream press have JUST NOTICED .....

What is also, just about, now, at last, being said is the way in which we are regulated and controlled.

Britain has all the apparat of a Police state. It just isn't being used ( much )

When you put together the provisions of the following: Serious & Organised Crime Act Police & Criminal Evidence Act Regulation of Investigatory Powers Act The Civil Contingencies Act - plus the current proposals for not only ID cards, but the "proof of ID if you have entered Britain"

We have a completely Stalinist or Nazi apparatus, presently lying idle.

How long before it is activated? Answer - they're Beta-testing it. RIGHT NOW ....

They do it to rickroll people into visiting their porn site...

I've just flashed on a very clear vision of Rick Astley porn. Time to quit the internet for the day...

Strictly speaking, information reduces the probability of uncertainty.

This is one of those precise mathematical definitions which is almost impossible to use; probability is hard to think about, and uncertainty -- especially the uncertainty of someone else in the future -- is hard to measure.

The useful heuristic is "information causes change"; you are going to do something different than you would if you did not have this particular bit of information.

Note that the concerns Charlie is raising are about information that is false to fact being indistinguishable from information that is factual in how the state apparatus acts on it. (Historically, this is what a jury trial is for -- is the state apparatus proposing to act on the basis of information that is factual or not?) Throw in that it takes a long time before anyone checks primary sources; factual status is usual treated as "this other database or record says the same thing", not "we have a material basis" and the situation is essentially one of arbitrary exercise of power.

Which would not be completely horrible as a situation -- there are, after all, legal and cultural mechanisms to say "wait just a minute" about that -- if it were not for the actual definition of information part.

Reduces the probability of uncertainty among whom, and the answer is "people who belong to a doomed power-exercising class".

Ubiquitous computing means that the traditional -- socially mediated, class based -- mechanisms for making decisions, assigning priorities, and distributing power are obviously replaceable by something that works better. In commercial enterprises, this is already happening (look at how extremely flat the hierarchy has become around shipping anything anywhere, for example; all the complexity management can and has been automated out of the human activity part of it), and there are all these opinion aggregator applications out there. Anyone even slightly self-aware in the political class recognizes that this is what they do, and not as well.

So their insecurity is enormous; they're reaching for the tools that require their replacement in an attempt to guarantee their continued hold on power.

This is sort of thing, well, the last clear example is the 30 Years War. "Ends well" is not all that probable.

Ah, yes: "we outsourced everyone else -- how come we haven't outsourced the politicians yet?"

At which point, see also: the final argument of kings.

Sorry Charlie - missed that last reference - please explain?

G. Tingey: French (pre-revolution) cannon used to bear the inscription, ultima ratio regum -- "the final argument of kings" (more or less mangled in translation).

(Meaning that anyone who picks an argument with Le Roi is eventually going to find themselves facing his final word -- the guns.)

And even a cursory examination of the history of Italian city states in the Renaissance suggests that outsourcing one's military never pays off well...

Come to think of it, the US and USSR's experiment in proxy wars in the Cold War didn't exactly work too well either.

As for the argument of Kings (Thank you) - that's one of the things that bothers me. Our wonderful guvmint is spending all this money on security and control, and NOTHING (really) on real defence .... Reminds me of the SovUnion 1936-39, before the whirlwhind of '41 ....

And, if you really want to blow a gasket LOOK HERE .....

G. Tingey: that doesn't surprise me at all. Mind you, I think you underestimate the capabilities of modern weapons systems compared to those of previous decades -- national defense is fairly well funded, despite the usual procurement scandals (much as there were during the 1980s or earlier; today we've got the FRES mess, but do you remember the GEC AEW Nimrod fiasco?).

What's really annoying is that the prison population has grown from around 50,000 under Thatcher to over 80,000 today -- and it correlates with the growth in imprisonable offenses, not an actual increase in the underlying traditional crime rate (that is, robbery, burglary, assault and murder). We're in danger of acquiring a US-style prison-industrial complex, and that's a very bad thing indeed in a system so prone to being gamed by newspaper crusaders.

MikeB #19- I could tell you, but I charge 65 pounds an hour or 500 pounds a day.

More seriously, purchase a copy of the book "Plundering the public sector" by David Craig and Richard Brooks. Then get employment with or cosy up to EDS, or ICl, or consultancies such as Price waterhouse coopers.
If you have your own consultancy, hire some marketing types who are good at making shit up, and pretend you have expertise in databases, healthcare or suchlike. It is irrelevant if you actually have expertise or not, as long as it sounds like you have.
Next step is to recruit some civil servants and ideally a gvt minister or MP as a director or special advisor. They will then get you entry into the necessary areas of gvt for your marketers to go to work.

Thus far, a quick glance at some of my old Private eyes suggests that in 2001-2, the MOD gave IT consultants 171 million pounds. The NHS national IT project thingy will probably cost around 20 billion pounds over ten years, and its been going since 2002. (It was originally supposed to cost a couple of billion)

There are other examples, but those are what I find. ID cards will cost probably over 10 billion (although the gvt claims only 4 or 5). I'm sure you can find a trough to suit you.

Charlie @ 40 (it's "Greg" for "g" btw) ....

I'm not sure this is the right place to air this particular hobbyhorse, but here goes:

I agree that modern wepons-systems are hot but ... We got the Falklands War because the idiot (in many ways) from Grantham acceded to defence cuts. After we won that war, the defence cuts went ahead, anyway ... Successive guvmints have continued with this folly, to the point that a fundie-muslim control of either/both Pakistan or Saudi would completely screw our seaborne supplies ...

That makes one of several areas where IT DOESN'T BLOODY MATTER WHICH GUVMINT IS IN POWER - they STILL screw up. The other areas are Education and Transport. Now, it looks as if we hav another - the IT/Surveillance fiasco, because if anyone thinks the tories or even the Lemocrats are REALLY any better, I think we are in for a nasty shock .... The control-freaks in the Civil Service, and the Quangoes will re-sell them the SAME pup.

We need another Roy Jenkins, and actually as PM, this time - the best PM we never had, the opposite, in every way of the aforementioned Eden.

Because this suggests that if I'm falsly arrested for something, then proven wholly completely innocent - the first time something similar happens locally I will be on the list of first people they drag in under suspicion.

I suspect their argument will be that as they already have your DNA, they can tell if you're involved without having to find you first. Not sayin' as I agree with it, but I've heard it made.

(The phrase "false positive" doesn't seem to work with these folks. They're often the same ones that have trouble with the idea of, say, contaminated samples…)

Till @28: Sorry, but no, no, no, no, no. Terry Gilliam's movie Brazil is the exact opposite of the real problem here.

In the movie Brazil, the problem starts with a fly landing in the info-processing system and causing a glitch. That's a technical problem. Most people here seem to be arguing that if we only use enough technology to make sure there aren't any glitches in the databases, everything will be fine. No Brazil-type scenario, no database glitch that turns "Buttle" into "Tuttle" and everything's fine, right?

Wrong!

Technical glitchs are not the basic problem with Jacqui Smith's proposal.

The basic problem is that everyone lies to the databases, so the databases wind up filled with false information. A prosepctive employee asked on a job questionnaire "Have you ever stolen from a previous employer?" doesn't mention that pen she took from her previous employer, and that cop who frisks a skateboarder and finds a joint in his pocket lies and says he observed the kid trying to sell it to make his arrest stick and create probable cause when there was none. And people don't just make mistakes -- they deliberately lie to the databases and deliberately put false information into the databases. That's 100% utterly entirely totally different from the scenario in the movie Brazil, where the false info gets generated innocently, by accident, courtesy of a technical glitch no one could really have foreseen.

Let's a take a concrete example. Why did the Maryland state police enter lies into the criminal database? Why did they knowingly and falsely identify non-violent anti-death-penalty political activists as "terrorsts" when they obviously weren't terrorists?

[1] To get promotions and bump up funding. A police officer who arrests 10 activitists for loitering when they chain themselves to the American flag outside the courthouse isn't going to get promoted to detective becaues of that. That's a trivial arrest, that cop may even get reprimanded by his superior for wasting resources. "Why are you arresting harmless little old ladies in a political protest instead of out catching real criminals?" But a police officer who arrests 10 terrorists, why, that cop will get a promotion. He's on the fast track. This is somebody to watch! Look at this guy, he ferreted out dangerous terrorists, let's promote this guy.

And the whole police department benefits from this kind of lie entered into the database. When the Maryland state police department comes up for funding in FY 2009, the chief of the state police say, "Listen, guys, we caught 10 terrorists last year. We're doing a bang-up job here, we deserve all that new funding we've asked for." So the state legislature gets wowed and gives 'em plenty of funding.

[2] Mission creep. Every bureaucracy starts out with a simple narrowly defined mission, which gets more vague and constantly expands over time. This happens to every bureaucracy. In the case of law enforcement, it starts out with keeping order and enforcing laws, then you get mission creep into "anti-terror" efforts, and pretty soon you're doing pro-active pre-emptive anti-terror work, and bit by bit you wind up expanding the alleged "mission" of the police to include just about everything, from spying on people to hacking into their home computers to search without a warrant for any suspicious activity to chasing people who litter, to obtrusively stopping and questioning citizens on the street and forcing them to produce I.D. for no reason, with no probable cause of wrongdoing at all. These kinds of comprehensive databases will infinitely enlarge the mission creep of British police. Cops will be able to troll vast databases looking for minor infractions, which then get inflated to alleged major felonies, because of mission creep. As the U.S. Army War College says, "Capabilities create intentions." The essential problem with enlarging the databases is that it will vastly enlarge the definition of crime, creating vast new ranges of infractions, due to mission creep.

[3] Vast databases like these make it incredibly easy to "juke the stats." If you've seen the TV show The Wire, you know one of the basic themes of that show was the way politicians and cops play games with the arrest statistics, "juking the stats" to create phony drops in the crime rate before election and bogus increases in arrests to create the illusion that the police force is doing great work, when in fact nothing has changed.

Cops "juke the stats" in innumerable ways, by reclassifying, say, a woman beaten to death by her husband as "domestic violence" instead of "murder" to make the crime rate seem lower than it is, or, contrariwise, by falsely classifying innocent protestors as "terrorists" to make the police seem more vigilant and more important than they are. In reality, by 2003 U.S. violent crime had plummeted to a 30-year low....yet America is awash in police, with DHS and DEA and cops and every kind of prison guard and security guard you can imagine, cop after cop after cop increasing their number exponentially. How is this possible? How can American police organizations justify adding more more police and more internal gestapo-like security officers (DHS, TSA, ad infinitum) when crime has plummeted?

Juking the stats. And these kinds of giant databases encourage that kind of statistical game-playing. America doesn't need anywhere near the number of DHS/TSA/police we have now, yet we keep adding more cops, and we've layered the Department of Homeland Security plus the DEA plus all kinds of other police organizations on top of that. All these organizations are largely useless and worthless and they all have to "juke the stats" to make their activity seem important and worthwhile, rather than worthless and useless. So the DEA falsely labels a guy who gets caught driving under the influence of cough medicie as a "narcotics abuser" -- wow, looka here, all these druggies running around, man, we need more DEA personnel! It's gotten so insane that now in America, a school girl handing out cough drops was accused of selling drugs. These kinds of comprehensive databases will only create more incentive for this kind of crazy reclassifications of entirely innocent behaviors to "juke the stats" and create the illusion of vast crime waves where they are none, or contrariwse create the impression that crime has plummeted when it really hasn't. We're already seeing it all over the place in America, first heroin and cocaine were made schedule 1 drugs, then marijuana was made a schedule 1 drug, then ecstasy was made a schedule 1 drug, now the DEA is fighting tooth and nail to make legal herbs like salvia divinorum and "herbal ecstasy" into schedule 1 drugs...pretty soon, Mountain Dew will be a schedule 1 drug (too much caffeine!), green tea will be a schedule 1 drug (psychoactive!)... There's just no end to it. The mission creep goes on and on, without limit. Comrehsnsive databases like this are a happy hunting ground for this kind of statistical game-playing, where is looks like a tidal wave of dangerous drugs is sweeping the country... When what has actually happened is that previously innocent herbs have been reclassified as "schedule 1 narcotics" in order to justify the funding and manpower of the DEA, or the DHS, or whatever agency you're talking about.

[4] Empire-building. Every bureaucracy needs to grow. Any government department is only as important as its size and funding. The more nonexistent "terrorists" the DHS identifies, the bigger its funding, the more important the DHS becomes. Ditto the DEA, the border patrol, you name it. This creates enormous incentives to game the database and insert all kinds of deliberate lies and arrest innocent people for no reason. We're seeing this right now with checkpoints being set up hundreds of miles from the border to stop cars for no reason, with no probable cause, allegedly for "anti-terror" reasons, but actually, just for fishing expeditions. They'll inevitably turn up some doofus with weed in his glove compartment -- wow! Look what a great job the DHS! Looka here, all these arrests, boy, they're just going a great job, let's increase funding to the DHS.

Giant databases like this will destroy society. Everyone will wind up being guilty of something, essentially every activity will become criminalized, taxes won't have to rise because everyone will get fined automatically so often for so many bogus trivial offenses that the money will come pouring into government like Niagara Falls, and best of all, every government bureaucracy will explode in size, sucking up vast amounts of new funding and adding pahalanzes of new officers as the number of phony nonexistent new "crimes" skyrockets.

It's a nightmare. Such a society will become unlivable. Indeed, we're very close to that point right now. A society in which a student gets expelled from school under the zero-tolerance anti-drug policy because he brought an asthma inhaler on campus, and in which 15-year-old girls get arrested and charged with felony child pornography distribution because they sent nude photo of themselves to their boyfriend via cellphone, is a society that's already so insane it's unlivable. This isn't a technical problem we can fix with some policy, like "delete the data after 90 days" (the data will never actually get deleted -- once entered in one criminal database, any data will always migrate and percolate to every other linked database, you can no more actually delete data once entered into one of these interconnected criminal/social/financial databases than you can delete something from usenet) or some technological fix, like "use Reed-Solomon CRC to make sure of the database's data integrity" (the problem is that the original data entry is garbage, a crazy lie, completely bogus B.S., so making sure that the lie is correctly preserved in the database is altogether beside the point).

We can only stop this nightmare by simply not collecting the information in the first place.

"But that will allow criminals to run free!" some will whimper.

Good. That's the price you pay for a livable society.

@6 I too have some problems with the tale of the cheeseburger...

"A typical beef cow produces approximately 500 lbs of meat for boneless steaks and ground beef."

A short non-fiction tale follows:

The author needs to actually go through the process of purchase and slaughter. For any vegans in the crowd, avert your eyes, and more importantly your tastebuds.

At the end of August 2007, I became a full-fledged virtual worker out of my home. In an effort to avoid the 90dB continual hum of the freeways in and around Silicon Valley, the wife and I sold the palatial 910 sq. ft. 2 bedroom, 1 bath (did I mention that square footage includes the garage) home in Hayward, CA. and moved to Missouri. It was the first county fair I had attended in years, and being flush with cash from the sale of the home we decided to attend the cattle auction. We bought a 1,300lb beast that yielded after butchering 850lbs of the most delicious beef my wife and I have ever tasted. (Had to buy a 2nd freezer just to store it all!)

BTW, since we purchased this beef at a 4-H auction, the cost was fully tax-deductible. (If you redonate the steer, you can write off double the amount in taxes, otherwise known as double-dipping which is completely legal) So in the end we took a $3125 tax deduction and butchering costs were $325 and kept the beef. We had so much beef we actually gave it away.

Moral here is buy in bulk, and make your own burgers. Get your friends together and combine your income and take the tax-deduction. The money you spend goes to the kid's college funds that raised the steer. Beats the store bought stuff both in price and especially in taste. Then invite your friends over for a barbecue. They'll walk away saying (insert your name here)'s beef is the best they've ever tasted.

Robert @ 43 - Oh I agree that they've got more than enough dumb ass reasons for doing it. But all the excuses still don't dissipate the fact they're treating you as in guilty already.

"We can check you're not invloved first..." Still suggests that you're considered guilty of something even when you're proven innocent. i.e. You're a suspect of thecrime you never commited!

@44 ~It doesn't work that way in the UK, Buy in bulk fine but Health and Safety/ food standards agency etc get their fingers into the mix. The animal has to be inspected killed, hung and butchered at a registered and licensed abattoir at your expense (there are differences if you are a retail butcher); also you won't get certain portions back (from your numbers I'd say you got all the offal back as well - in the UK specified offals have to be disposed of, at your expense). Doesn't matter where you buy it, it ain't tax deductible for a private individual. If a farmer keeps one animal back for personal consumption the Tax authorities add a notional amount based upon average retail value of said animal (less killing fees) and add the tax due on that 'benefit' to the farmers taxable income. It's still done and is worth it but because of the hassle factor not many do.

Agree on taste and quality, but don't assume uk and US situations similar.

Before anyone asks, the BCMS (British Catle Movement Service) tracks every head of cattle from less than a week after birth until slaughter by means of a database and a flurry of paperwork including cattle passports, individual farm movement books and the fun that occurs when a registered animal dies but but not in an abattoir is painful to behold (the database doesn't have a section for death due to illness/accident!). We get about one query per forty animals - usually data corruption on their side.

I apologise to our host for encouraging thread drift!

Charlie We eat cheeseburgers. We also eat tacos. Lots of tacos. If so large a percentage of our pollution comes from cheeseburgers, how much pollution comes from our milkshakes, fries, tacos, fried chicken, coffee, ice cream bars, burritos, ice cream cones, cups of soda, donuts, egg McMuffins, corn dogs, tamales, fajitas, pizzas, chicken tikka masala takeout, sushi, cheese nachos, bottled water, etc. Then there is our sitdown restaurant food consumption, our home food consumption, our snack machine food consumption, etc. Then there is our nonfood consumption such as housing and commuting. Cheeseburgers are a small part of our fast food consumption, and fast food consumption is a small part of our total energy consumption. The total pollution production of America cannot be lower than the individual pollution sources.

hmm...

I wonder what the calculation would look like if the cheeseburgers would not be made with paddies produced in quantities of thousands, but one at a time, plus the cost of getting the necessary meat to the individual households piecemeal, instead of people going to a central place to consume it. Economy of scale is hard to beat.

Same goes btw for the cow mentioned above. Instead of getting the meat to the consumers as soon as possible, you keep the meat of a whole cow refridgerated refrigerated for a pretty long time, wasting energy. (Inviting lots of people to a barbecue at least helps to limit the waste.)

@46 No offal, yes same as Britain, registered butcher. I remember that taxation stuff, pretty much the reason for that revolution over 200 hundred years ago. But please keep any discussion of British taxation to a minimum, we really don't want any American politicians picking up more of your methods. P.S. Can i get a job with the BCMS?, it sounds like one of those jobs you hold for life as well as getting a nice pension when you decide to retire.

@48

you keep the meat of a whole cow refrigerated for a pretty long time, wasting energy.

tp1024, imho, you have a horrible sense of scale.

1st, frozen would be a better term. 2nd, getting the meat to the consumers as soon as possible means excessive transportation costs. Why do you think freezers and refrigerators were invented in the first place? Not to mention spoilage. 3rd, I know personally I waste more per month on heating than I do on cooling over the course of a year, just because of the locale in which I live.

If we were to look at a far grander scale than the contents of my two freezers containing beef which has basically fed us (as well as others) going on for a year and a half, one should actually consider moving entire populations out of less than temperate zones just to save on heating/cooling costs.

After having been more of a reader than a poster here, far too many of you have bought into the carbon footprint/global warming crisis hoof, line, and sinker.

@metadata - personally I already suffer from information overload, but haven't yet figured out how to draw disability because of it.

mclaren @44: yup.

See also: the panopticon singularity (intro) and my circa-2002 essay on the subject.

@48 P.S. Can i get a job with the BCMS?, it sounds like one of those jobs you hold for life as well as getting a nice pension when you decide to retire.

Well to be honest their online access system Really really sucks (and terrifyingly has been rated as one of the better Government sites!) see http://www.bcms.gov.uk/ they have only recently realised that people aren't using IE 6 or Netscape 7.X and still haven't tweaked the site I still have a copy of Netscape solely to access the site! adjusting Firefox 2 (or heavens 3!) bothers me on a visceral level. Oh and their so called encryption refers only to the log on process and not to the uploading of Data!

@48 one butchered animal will fit a large chest freezer, running costs of a new A rated freezer is approximately 200kWh per annum not sure what this equates to.

Ah, Charlie, reading that link you posted explains what happened to the "Whole Earth" people .... I still have a "millenium" Whole Earth Catalog(ue) - but the Web killed it, the thing it helped to create, in a way ..... [ hiatus ]

I just read your Panopticon Singularity Essay (again)

How far down that list are we now .... P2P excange of info - here yet - or delayed because guvmint computing is so crap? Gait Recognition - here, but not being used much / yet? Labs-on-Chips $Bignum-problems? Only being made (at present in small numbers for specific apps? THz Radar - being rolled-out at SOME airports - "morality"/"privacy" concerns being voiced - bu they're doing it ..... RFID dust - not yet, not soon? another 5-10 years? "Trusted" computers and DRM - Legal probs, I understand -as in "Bought it legally in USA - won't play in rest of World, and vice versa ..... Cognitive Radio ????? Data Mining - here, now, in use, probs with false data, as described, not that THAT stops them ....

Couple that with the list of UK legislation now in use (I'm not posting it again right now) and it's REALLY scary.

How, short of violent and bloody revolution, which is inherently wasteful, how can one stop it?

NO political party has the guts to even try to stop it, not even the Lemocrats.

@52: I dug out the specs on my two 13.8 Cubic Feet Freezers. 539 kWh/yr (We specifically looked for a model that used the least amount of energy) It took two freezers to completely house the butchered steer. Yearly operating cost $49 based on 9 cents per kWh. So $98 per year, but seeing as we have emptied one them, it's now unplugged.

Oddly enough, Missouri State Law requires I maintain an outdoor very bright night light since I live out in the country in case of emergencies at night on your rural property. This runs me $144/yr. Personally I view this as highly wasteful, as I can't power it off with a switch.

I agree with you about government always being a few steps behind in terms of tech, but I've noticed that merely moving out of the Silicon Valley. The average person doesn't care about Web 2.0, cloud computing, or any of the other buzzwords that flow from California like so much raw sewage.

@53: On the deluge of legislation coming from our mutual governments, I don't think it's going to ever stop. I can only wonder if the Roman Empire would have fallen faster if they had computers.

Ok, frozen is correct. Then again, as a non-native speaker just looking up the correct spelling of refrigerator and using it a couple of times helps even if the usage is wrong. ;)

getting the meat to the consumers as soon as possible means excessive transportation costs.

As soon as possible is meant to be much less than a year. The time from the butcher to the frying pan, with a stop at Wal Mart or whatever, is perhaps 3-4 weeks (probably less). If the frying pan is at McDonalds (I know they don't have any), this is likely to be even less. Nothing about the need for timely transportation is excessive in the scope of the usual logistics that you need anyway.

Now, say you have 1000 cows slaughtered and brought to homes where it is put in 2 individual freezers each stored for consumption for, say, 12 months. I'll assume that you will only need one freezer after 6 months (hoping you shut it off when it's empty and don't fill it with other stuff, but most people will keep using it ...), so you'll use 300 kWh for each cow (thanks for the numbers). Your cows will use 300,000 kWh for cooling. (Costs about $60,000)

If you have the 1000 cows in one store room for 2 weeks this will cost maybe on the order of 5,000 kWh for cooling, because it is only 1/25th of the time and the larger your freezer the more efficient it can be insulated. (Much higher ratio of volume/surface area.) Energy for transport will be roughly the same (bulk transport is vastly more efficient because you have to move much less metal per pound of meat and I kind of hope you don't drive a couple of miles to eat out in a restaurant each day. When you buy meat in a grocery store you'll be there anyway and only a part of cost of transport can be accounted to the meat.)

And finally you have to take into account the cost (in terms of energy and human labor - also see labor division) of preparing the food piecemeal instead of in bulk, which is also more efficient, essentially because of the same reason that refrigeration is more efficient in bulk. (Less heat is lost in the cooking process because more meat is being cooked at one time with much less surface area to lose heat.)

In terms of energy usage there is hardly anything more efficient than precooked meat (ok, vegetarian is more efficient, but lets stay with meat), cooked in quantities of hundreds of pounds and prepared for individual consumption in a microwave oven (because a microwave only heats the meat and not too much of its environment.)

3rd, I know personally I waste more per month on heating than I do on cooling over the course of a year, just because of the locale in which I live.

You waste much less if the house you live in is a) well insulated and b) has a high ratio of inhabitants/surface area. (Renting/owning a flat in a well insulated house with 50-100 people is far more energy efficient than owning a so-so insulated house for 4-5 people in the suburbs. And because of the smaller surface area of the house, insulation is much cheaper on a per person basis, even though it's better.) If you want to save energy, low population density is a killer. The subprime crisis will help a bit to overcome that problem though, hopefully.

Charlie #40: "We're in danger of acquiring a US-style prison-industrial complex..."

I had the dubious pleasure of attending a talk given by Ed Gibson, Microsoft UK's head security honcho. Prior to that gig, he was the FBI wonk who allegedly threatened Gary McKinnon with "being fried" for the heinous crime of hacking into ultra-super-secret computers that were attached to the Internet and protected with passwords such as "password".

Mr. Gibson was distraught and horrified that the UK couldn't treat computer criminals as they deserved because "Her Majesty's Hiltons are all full". While there are some fairly horrible criminals who use computers to further their deeds, I don't think it'll be hitmen or human traffickers that'll be seeing the inside of a prison if Mr. Gibson gets his way, but "illegal downloaders" and the like.

Thorne @45, my condo is 863sqft including the balcony and storeroom. When I was a kid, we bought a calf and unfortunately kept going to visit it. So in the end, another family ate Meatball and we ate Sausage, because they'd taken their kids to see Sausage. Buying it at a fair and having it slaughtered before you make friends is much better.

@57: 910 sqft including the garage. Sold for $432,000, before the housing market fell apart. Although I still have a mortgage, I bought a home in Missouri, which by all standards is palatial and sits on 17 wooded acres. The storage shed alone (40ftx60ft) that came with the property would easily accomodate both your condo and my previous home.

On your second point, I agree wholeheartedly, never befriend your food.

Thorne, by London standards that would have been cheap. Before sterling tanked (over the past 6 months) London was rather more expensive to live in than Tokyo. (Which is still pricey.)

Mind you, even through a housing bubble and over-valued currency it's been possible to buy a genuine castle (with a couple of hundred acres of land around it) for around the £250K-500K mark. The only problem is the maintenance costs (£2-10K/month, minimum) and the location (a two hour drive from everywhere).

I remember an SF short story from the sixties/seventies in which a man who started out being chased by a book club winds up on death row due to cumulative errors in passing data around. Seemed far fetched at the time, but now more a harbinger of things to come - innocence is no protection.

Ah, revisiting the Panopticon Singularity again ... Scary stuff. No suprise, though.

I can only wonder if the Roman Empire would have fallen faster if they had computers.

From the Illuminatus trilogy (probably, people who put this kind of stuff on the internet happily lie and change it):

In Beamtenherrschaft [Bureaucratic] ages there is ceaseless activity, all planned in advance, begun at the scheduled second, carefully supervised, scrupulously recorded— but inevitably finished late and poorly done. The burden of omniscience on the ruling class becomes virtually intolerable, and most flee into some form of schizophrenia or fantasy. Great towers, pyramids, moon shots, and similar marvels are accomplished at enormous cost while the underpinnings of social solidarity crumble entirely... Theoretically, an Age of Bureaucracy can last until a paper shortage develops, but, in practice, it never lasts longer than 73 permutations.

Thanks to computers, the paper shortage has been delayed.

Charlie @ 25:

I agree that it's good to consider a number of different sources of meat, for food security reasons as well as environmental. Kangaroo hasn't caught on here (I've had it, it's good but people expect it to be weird). Ostrich has to a degree, there are places in the West where they are ranched.

Though in the case of North America, the natural ecology would seem to include large numbers of ruminants -- bison, deer, elk, goats, antelope. Replacing them with domesticated versions wouldn't seem to make a big change to emissions. Now, it's probably different for other continents. Australia, for example. South America outside of the llama habitats.

Charlie #59- I can't think of any inhabitable castle for sale for less than half a million, especially with 200 acres of land as well. Which one were you thinking of?

Not necessarily any inhabitable castle, if you follow my drift; and half a million is near the floor right now, following years of inflation -- but picturesque hulks are somewhat commoner here than they might be in the vicinity of Palo Alto.

Thats cheating, you dind't say it could be uninhabitable- even with the passing of the good old days when farmers would sell you the land and throw the castle in for free, you can still get a ruined one for well under 200,000. Of course you then have to get planning permission and anything from 200000 to 1.5 million to do it up.

Thorne @58, no garage, but I do have my own handicapped spot right outside my door. I'm currently upside-down on the condo, but I expect things will come back up in a few years, and it's not as if I'm planning to move. I bought this condo with the idea of staying until I die. Your new property sounds very spiffy, but since I can't walk on uneven surfaces like grass, gravel, mud, it wouldn't be very practical for me! (Canceled PT on Tuesday because of the sleet.)

I confess that I know nothing about metadata as described, but I distrust anything information-related with the prefix meta due to the astonishing misuse of using metastudies in medical and related fields and pretending that they are in any way scientifically valid.