Charlie's Diary

I keep meaning to write something (quite long) on information warfare - but realised this would be "information useful to terrorists" (or people not wanting to have the government monitoring every aspect of thier lives).

Then I realised that my primary data source is information I have in my head.

I keep having kafkaesque nightmares of being banged up for being potentially useful to terrorists. Large doses of ECT would probably be the only recourse to make me "safe".

Seems to me the only way to preserve my brains will be to disseminate that sort of information as widely as possible........

I wouldn't worry too much. After all it's not like Nulabor have a track record of seeing complex computer system projects through to successful conclusions, have they?

They do have a track record of being a bunch of obnoxious fakes who took a once-great party and turned it into a system of outdoor relief for a gang of upper-middle class simps, feebs and three-time losers.

This panopticon singularity will be rather a monetary black hole. Which is bad enough, but doesn't mean we'll find ourselves in room 101 in five years time.

Or am I wrong?

You remember the comment about US politics, about the Current Administration using "1984" as a guide? It sounds like the UK is trying to do the same thing with "V for Vendetta."

You missed a bit; Ed Balls is introducing a nominated teacher in each school to whom pupils can inform on their fellow students for "extremist" tendencies.

Speaking as someone who, at 15, was reading a heavy rotation of Kropotkin, Bakhunin, Proudhon, etc... yeah, that would have been me.

I think that news probably nudged zombie Honnecker to the Billy Mill Roundabout...

been reading that the way to beat inflation/recession is to increase public spending by a large amount - anyone think this may be one of the ways they do it? I find it hard to believe they would actually want to spend lots of money on making things better for the people, but better for the corporations seems to be the modus operandi of the day....

The trouble is that the economic basis for spending out of recession (Keynes?) depends on the assumption that the money doesn't leach out of the local economy.

There's a lot of the hardware which we cannot build in this country, even if "security" means that it won't involve outsourcing of the IT work.

Ugh. On a smidgen more positive note, it seems as if there's at least a hint of movement to repeal the FRA wire interception law in Sweden.

And someone on making light comments this week was saying New labour had done some good things, like pass the human rights act...
Of course I pointed out they take away what they give, with the surveillance society and detention without trial.

Also as Dave Bell says the money from the stimulus of wasting lots of cash on the surveillance would leak abroad, furthermore it would commit us to a vast unproductive infrastructure and the people necessary to run it. This infrastructure would always consume resources, never increase them the way investment in new technology or building roads when you didn't have any before, would do so.
(Although one wonders if the demand would improve computer and surveillance stuff? What if everyone could be hooked up to the net with their own surveillance stuff and neighbourhood watch schemes started tracking strangers, and lynch mobs found local criminals using their own informal CCTV networks using the same software as the police do?)

I thought the idea behind Keynsian spending was that government spending on infrastructure stimulated the economy by both circulating money and, well, creating useful infrastructure. This doesn't sound like useful infrastructure.

Unless, of course, you're a corporation looking for decent market data. Any bets on how long it would take for the data to be sold off if they ever managed to get this working?

And of course the problem with mass civil disobedience and camera smashing is that most of the meeja would run banner headlines for weeks about how evil civil libertarians smashed the cameras that could have saved some childs life.

And the servers etc would be in guarded bunkers, although I suspect their datapipes might be a bit more accesible.

Robert- we already know they sell things like the electoral register, and allegedly the current departments and suchlike leak badly anyway. I expect people to be prosecuted on a regular basis.
Perhaps those familiar with the UK databases explain how often they are compromised and how the criminals love them?

Look on the bright side, if present trends continue, then when China takes over the world, c. 2040, at least they'll have better human rights than ours.

@4: Ed Balls is introducing a nominated teacher in each school to whom pupils can inform on their fellow students for "extremist" tendencies.

This government is beyond parody.

The UK does seem to be turning into quite the surveillance state, doesn't it? I wonder why.

I suspect the issue is not so much any particular technology (east germany did it mostly manually), but a return in the UK to a form of government with an elected legislature and executive, instead of one where the nominal government are just supplicants at the court of the tabloids.

If you look at the author of any of those initiatives, it will not be someone who thought it a good idea, but someone literally in fear of being sacked, within a week, from their job if they don't do it. And the person doing the sacking, the one with the actual yes/no decision to make, would be a journalist, not a mere politician.

The theoretical advantage of democracy is supposed to be that you have all those people finding out facts, and so making a collective decision more in touch with those facts than any central administrator, however enlightened (see earlier discussion on famines).

Too much reliance on centralised mass media flips this advantage to a liability. That type of media don't print things people already know, can experience themselves. Man bites dog is news, dog bites man isn't. Train or plane crash is news, car crash isn't: it happens too often. Stabbing is news, domestic violence isn't (and stabbings, especially amongst the poor, only became news recently when other forms of crime, including shootings, dropped). Nuclear power gets a reputation as dangerous precisely because it is safer than things, like coal mining, that kill as a matter of routine.

This leads to a form of government that systematically and inevitably tends to doing precisely the stupidest possible thing on any matter of public concern. No-one wants to live in a new East Germany, but any politician who directly tried to stand up to the forces taking us there would be crushed like a bug.

If what the current economy needs is a job creation program, then fixing this situation should be a priority. Treble the size of the BBC, found a similar corporation for daily local print media, multiply by a thousand the number of paid political staffers, think tankees and political academics, treat broadband access as a civic right...

I remember a couple of years ago talking to a guy who was working on a early release of the database that both Charlie and Ken have relied on so heavily in recent publications. He seemed to have come to the conclusion that the only way forward was to trash it completely, before it could be implemented, so that it wouldn't wreck lives for years to come. Is this a sensible attitude for public servants to have?
Not being a local girl I haven't heard any more gossip ( about such a sensitive system! ) but I don't expect much as changed.
Re the children's database ( gone awfully quiet since they lost those Family allowance details… ) some one working on them said the biggest problem was getting the soft ware to understand that 49 The High Street was the address same as 49 /High St…
As you know Bob… So when's the day_they_hung_the_lawyers coming?
As for me I'm sure the PTB think that I was gurning when the wind changed, as I pull faces for the cameras as often as I can.

ps whats the difference between leaving secure information on tube seats/ lost in the post and selling it on?

Generally, in this sort of case, I tend to fall back on "fear makes you stupid; OK, what are these guys afraid of?"

So far as I can tell -- our conservative party, too, or the American neocons; this is pretty ubiquitous at present -- the core fear is some mix of being thought weak and having existing weakness noticed.

Since the futility of warfare between industrial states got publicly internalized, you can't look strong by making foreigners afraid. So instead of the externalized, fear-driven aggression version, you get the internalized, fear-driving oppression version.

(And since current government have effectively no reliable sources of information, and know it, the weakness is very, very real.)

There's a reason ruling classes, historically, have strongly selected for great physical courage. It has any number of drawbacks, as a practice, but it does tend to produce the kind of self knowledge that avoids this particular failure mode.

Great leaders find a way to trust the people they lead. Good leaders convince those whom they lead that they are trusted. Weak leaders or panicked leaders start asking "who can I trust?" and wind up in Stalinesque realms of paranoia. The Ground Ape social brain really can't do anything else.

One from the archives: A report on the state of the National Identity Register, May 2016.

(I made a significant error in my 2006 prediction: I pegged 2009 as the year when a panicking Home Secretary would order the implementers to incorporate data from existing government databases, thus polluting the shiny new built-from-scratch ID register. In reality, it happened in late 2007 ...)

The old Tory slogan "New Labour, New Danger" was quite prescient, just not about any of the things it was originally targeted towards.

There is so much to hate about this abortion , I don't know where to start. One of the main problems , is if it is introduced to reduce terrorism , then it probably won't work. I point to a recent report from the NRC

"The most extensive government report to date on whether terrorists can be identified through data mining has yielded an important conclusion: It doesn't really work.

A National Research Council report, years in the making and scheduled to be released Tuesday, concludes that automated identification of terrorists through data mining or any other mechanism "is neither feasible as an objective nor desirable as a goal of technology development efforts." Inevitable false positives will result in "ordinary, law-abiding citizens and businesses" being incorrectly flagged as suspects. "

http://news.cnet.com/8301-13578_3-10059987-38.html?part=rss&subj=news&tag=2547-1_3-0-20

Unfortunately, I don't think this will put them off implementing it.

However it is a big government I.T. project , and we all know how well those usually work out.

Canada ahoy? New Zealand?

B. Dewhirst: in the first instance, I've got my eye in Ireland. (Closer and easier to get to, long history of not being entirely enthused by HMGUK's attempts to clamp down on civil rights.) Might be too close, though.

Perhaps the government sees it as a way to achieve full employment without resorting to conscription? After all, someone has to read all those e-mails they'll be sucking in.

I'm glad you're ready to foot-vote, though... I'm sure your writing will suffer terribly if they decide to bring out the rat cage.

Or we fight back, and flee only if we lose.

The small problem being that the internet probably isn't the right place to discuss this.

Suggested non-violent protest - start cc'ing in the Home Office minister responsible (Vernon Coaker, minister of state for counterterrorism) on all your emails. All of them. Let him know politely that you are entirely behind the IMP and keen to get started with it, so if he could just have a quick look through your traffic and forward anything relevant to the competent authorities...

Cameras are objects.

Objects can be burned, blown up, sprayed with paint, etc.

I've seen what Brits can do to speed cams. :)

One guy wearing a ninja suit and a Guy Fawkes mask

can do a LOT of damage to the Panopticon.

November 5th: destroy cams day?

I'm surprised nobody's mentioned the 'Echelon' system, part of which resides on your side of the Atlantic, Charlie. When rumors began to surface a decade and a half or so ago, people said "there is no way that could exist. Imagine the kind of bandwidth and storage that would take to actually implement."

Yet, today the existence of the system is not exactly classified, and its purpose isn't really, either. There were even a couple articles in Discover and other pop-science magazines about multi-terabyte SSD arrays (which was pretty Hot Stuff in 2002).

Maybe if you cut your power estimates in half, or more, and consider that they're not using technology you really know about or expect to be deployed at that sort of scale (one idea tossed around at Microsoft years ago was a single-system-image computer running on 20,000 ARM cpu's, with three cores per each – very low power, very low heat, incredibly high compute power, and all this on top of Windows Server...) you could extrapolate a little. Probably not, though, as you don't know where to inflate or defalte the importance of one technology vs another. Or, "This isn't your grandfather's Transmeta."

There are a few ideas that have been thrown out to the scientific community that were never fully realized; SDI is an example of one. However, one could argue that any project that is sufficiently well-funded is capable of attaining its goals. The nuclear program in the US during WWII is a typical example of hysterical, unlimited spending resulting in developed, deployable, and effective, uh, results. In contrast, SDI is an example of a program that was in concept, doable, but suffered continual budget shortages.

There are also a few of those programs out there today, and specifically in the fields you mention – storing, transmitting/intercepting, and analysis of data – which are flourishing in the intelligence and defense communities. I think it's kind of careless to assume that you can divine the implementation of classified programs with nearly unlimited budgets from what is public knowledge. The common question posed to these communities is "We want to do X. Assuming X is possible, how would you do it?" This may be as simple as saying "We'd like to be able to sense movement with a six-inch resolution in Afghanistan from a sensor in Florida." Ordinarily, you or I would say this is impossible. But if you assume it's possible, and the difficulties in getting from "here" to that wacky idea you'd need to actually do that would be covered by an unlimited checkbook, then technically anything is possible.

Except FTL, I suppose. The point is, there's a lot of stuff out there that's not being divulged and might not be for twenty or even fifty years. It's rather like the concept of a "singularity." You're trying to make predictions on what the world will look like (or what the spooks can do) without the information you need to do so.

Charlie2 22

Ireland is the UK's Sudetenland.

Canada is the USA's.

New Zealand and Oz are a good bet. At least, in the event of a Japanese takeover, you'll get to see Mecha. [1]

[1] Mecha, as used for elderly care, may not be as cool as you thought they were.

Also:

"UnFascist Britian" has already been done, dammit! Find your own AH!

Charlie : So where will you flee if the UK becomes a police state? Ireland? Canada? US? Australia? Belize?

Bunch of defeatists.

Thanks for the suggestions. I am already familiar with how, when and where to break things, haing devoted a minute or two to thinking about it whilst on long boring journeys. Not to mention the books I read.
I like the idea of flooding certain home office ministers boxes with stuff, that'll get some attention.

Theres the No 2 Id people, anyone else doing some campaigning? THis is an issue that gets together everyone who cares about freedom, from left or right.

(Of course I'd rather get a more democratically accountable gvt into power but that will take a while)

Andrew @31:

See above (22 for instance.)

Ajay@26:

Especially the spam. I would be really willing to send him all my spam - after all, it is from dodgy criminal types, so obviously relevant.

Dewhirst @ 33: Thanks, I missed that one... Though my vote would be for Belize, since Charlie can basically practice his profession from anywhere. Great weather, low cost of living, awesome beaches, interesting ruins...

martin @ 29: so are you saying that Roujin Z was not an accurate depiction? Aw, man..

Andrew @31:

The US, despite having suffered, um, a few civil liberties setbacks in recent years, still has plenty of empty geography to get lost in. Unfortunately, some of the best territory for going on the lam tends to be in places like Alaska. I guess we'll have to see how the election goes; if McCain and Caribou Barbie somehow end up in the White House, I'm considering bugging out myself.

You missed a bit; Ed Balls is introducing a nominated teacher in each school to whom pupils can inform on their fellow students for "extremist" tendencies.

Generally, in this sort of case, I tend to fall back on "fear makes you stupid; OK, what are these guys afraid of?"

...

I really wish I was better at applied memetics. There's some really simple antidotes to this kinda regulation if you can get The People to understand the (actually quite simple, if initially counterintuitive) reasons why this database will never, ever, ever work for tracking down terrorists.

Simply put, this is architected for fishing expeditions. It cannot be used for crime prevention. Therefore, there is no legitimate justification for creating it, and every reason to fear its misuse. Because misuse is the only thing it's good for.

If you want to shut up an inconveniently noisy protest group, you can plug their names into the database -- Google their private life, basically -- and pull out whatever you want. You have a single, named target or target cluster, with unique identifiers and all the data linked together -- it's brilliant for that. Expect scandals against opposition figures to shoot through the roof (Bruce Sterling calls these "centipedes", for no reason I can discern.)

But if you're the security services, trying to carry out a legitimate mission to protect the public, it's no bloody use at all. If you're hoping to discover terrorists you didn't know about, this will not help. Read up on Bruce Schneier to find out why, but the short of it is that even with truly awesomely spookily accurate filtering, false positives would outweight legitimate results by such insanely vast numbers that you'd never, ever be able to find the real terrorists.

(Here, I've dug out a suitable article for you: http://www.schneier.com/blog/archives/2006/03/data_mining_for.html runs the (USA) numbers and finds that the FBI would have to manually investigate 27 million plots a day after filtering.)

And if you already know who you legitimately suspect of terrar!, you don't need a vast trawl database of the entire country: We already have procedures in place for wiretapping suspected wrongdoers.

No good will come of this.

Canis @38:

"No good will come of this."

Amen, sir.

Martin 29, aren't NZ and Oz the personal domain of the guy behind the current US and UK leaders?

You know what's really scary? Similar things are happening here in Austria - but without anyone in the public knowing. Our police and ministry of interior are implementing pretty similar systems and only a few reporters found out about it by accident. No one has heard of it before that, and even when they found out, there was no public outrage - people simply ignored it.

People are so focused on their everyday lives, their TV shows, their job and their hobbies, they simply have no time left to think about these matters. If you ask an average citizen, they are in favor of such systems - it's all for the good of the children after all. Can't be against that now, can we?

One can only hope the incompetency usually displayed in such public projects stops them because of huge cost overruns or makes the resulting system unusable.

Um. I suppose we'll all just have to use cash, eschew the Internet, bin our mobiles and walk or cycle everywhere.

It might not work, but it'll probably make life more fun!

Martin @29: Oz a good bet? Google John Howard children overboard. Never has a country been more willing to bend over backwards for the sake of the merest hint of the possibility that something bad might be have a decent chance of not happenning to them. The creeping self-centred weirdness of Australian politics was one of the reasons I wasn't comfortable living there...

Robert @ 37: I think the key to living in the US is to find a city and state that you like, with a culture that you like. There's a big difference between NYC, Houston, Salt Lake City, Orlando, and Portland.

Not to mention good free speech protection and friendly libel laws which are good for authors...

32
May I suggest a national privacy day in which everyone goes about their business normally but wears a mask or a paper bag over their head. It would also be a good education by the participants on just how much their I.D. is required.

Wow, being from the states of course I assumed that we had it bad as far as intrusion of privacy went, but we've got nothing on this.

Okay. Suppose they do find the money for this thing. And suppose they do actually get it to work. Somebody'll only leave it in a pub or on a train.

Charlie, I know it's your soapbox, but...

I was at the WTC in 93. And in 2001. And just today, I had to walk out of my Wall St. office having a block cordoned off because a suspicious package was found.

Being born in the Soviet Union, I have a healthy disgust for Orwellian society models. But I gotta tell you, I'm scared of the alternative too -- because post that event, we'd have no freedom at all.

Bring back the 12th century, panoptican society on the cheap; built into your brain from birth… Oh sorry, I forgot that brainwashing was still practiced here.

Perhaps I was overegging the pudding with the 'bigest' problem with the children's database (aww they mean well!) but trying to bang several databases together, that must hurt.

Some one I know who builds databases, for those who's only parameter is that it works perfectly, but they don't know what they want to know yet*; finds it amusing that the post office can't admit that his house exists. All his internet toys have to be delivered to a neighbour.
How's that going to work? Your homeless, not because you don't not have a house but because you don't have a post code . What's more real? Real stuff or ideal stuff.

*but then isn't that all clients?

This isn't exactly new. Back in the 1970's when the IRA was bombing England, there was rumored to be a government program to listen in on all telephone calls and use computers to try to detect key words like "bomb". Obviously that either never existed or it didn't work.

Now I can believe the government could try monitoring everything, but in reality, they would drown in an ocean of data trying to do security. The best that they could do would be to use the system to track people of interest after they were identified. As with every other technique, those who want to do damage will be easily able to evade or fool the cameras and systems. It would just become a huge white elephant that could be used to keeps tabs on teh citizens, but useless for averting real threats.

Can't your Queen step in and do something here? Wasn't she busting caps at similar fascists 65 years ago?

I was at the WTC in 93. And in 2001. And just today, I had to walk out of my Wall St. office having a block cordoned off because a suspicious package was found.

Being born in the Soviet Union, I have a healthy disgust for Orwellian society models. But I gotta tell you, I'm scared of the alternative too -- because post that event, we'd have no freedom at all.

Again, read Schneier, particularly "Beyond Fear". You're scared, yes, because human beings have evolved to be much more scared of big, flashy, unexpected things, than humdrum ordinary things (that still happen to kill thousands of people) like car accidents, smoking, or chronic alcoholism. And people are exploiting your fear for their own ends. It's important to take stock, step back and figure out what to be frightened of, and what to accept as an inevitable risk of being alive in the modern world.

Either way, it's irrelevant. The point in my previous post is: It doesn't matter if you believe the risk of terror justifies such extreme measures as Total Orwellian Surveillance or not -- because it's fundamentally useless for that task.

The proof is in that cordoned-off suspect package. Just some lost shopping or a traveller's luggage who'll never be reunited with his socks, but look at all the hassle it created. Now imagine 27 million of those a day.

Actually the biggest driver for this will be coming from the Treasury since it's an absolute gem for cracking the grey and black economies (cross check spending with income and you know who to audit).

And by a strange co-incidence the PM lived at No. 11 for 10 years.

Of course, once it's in place you can use it for all sorts of other neeto stuff - like, for example, checking "voting trends"...........

Betting on the government screwing this project up as much as they have other IT projects is a bit iffy - this one is least likely to be outsourced and the auditing will keep the middlemen under control.

As has aleady been mentioned - such a database is sod all use for combating terrorists. But it's tailor made for oppressing the bulk of the population.

I do think we in the US have been using this type of system for years. "Total information awareness." The problem with all that data, though, is not designing the algorithms to pop up suspicious activity, it's finding the resources to deal with all the various types of suspicious activity once you've seen it. The political will may be there to punish this or that, but getting Joe Prosecutor or Joe FBIAgent to move on it is another thing... it's beyond an admissibility issue when you have 1000000's of potential cases and 10's of potential case officers, who are already overbooked.

Alex J. Avriette,

The issue isn't grand-scale hardware; there are some really powerful systems out there right now, and potentially some much more powerful ones that could be built for specific purposes (reduce a problem to dataflow that can be munged in a systolic pipeline and watch the petaflops happen). Also, hardware design teams are usually relatively small, and easy to manage.

But software to do the actual data manipulation and handle the integration of the existing databases, ah that's another kettle of god-like squid. Software management is an as yet unattained art, and the integration of existing software is blackest art of all. And every single time a large organization, including governments, has tried to implement a system of that level of complexity they have failed utterly even to finish the project, let alone produce a usable system. Sure it's possible this one will be different, but what are the odds of that?

Bruce @54, you're forgetting small organizations like Lincoln Labs, Southwest Research Institute, and others, that do implement systems of that level of complexity. A simple example of this is missile identification from launch plumes.

The odds are, in my opinion, pretty good. When one limits the scope of an application, and hires appropriate staff (from one of the abovementioned or similar research institutes), the amount of "cooks in the kitchen" is reduced, and time-to-deployment is a lot shorter, and the "product" (whether it be an idea or an algorithm) is better, less complicated.

They're not, I don't think, handing this sort of work out to Lockheed or Northrop or The Usual Suspects.

The Panopticon Singularity was first described by Charles Stross in an essay originally written for the Whole Earth Review's 111th issue, where he describes it as "a police state characterised by omniscient surveillance and mechanical law enforcement."

You only have yourself to blame, Charles.

Up next: thoughtcrime...

Chris L @43. Fortunately John Howard is no longer PM. Now we have a Labor government. Hey, why are you all running away screaming?

Dorian @57:

I'd heard that John Howard was no longer PM and that he's now helping his cousin Bob with some investigations. We'll see if he shows up in one of Charlie's future books....

"Your emotional state may also be monitored at this time for crude signs of aggression or depression that affect your posture or movement"

Oh dear. I cycle almost everywhere. Which, in a city like Manchester, requires a permanent state of paranoia about- and generates a fair amount of anger at- the muppets allowed to drive. The Police would be pulling me over every day.

47: I was at the WTC in 93. And in 2001. And just today, I had to walk out of my Wall St. office having a block cordoned off because a suspicious package was found.
Being born in the Soviet Union, I have a healthy disgust for Orwellian society models. But I gotta tell you, I'm scared of the alternative too -- because post that event, we'd have no freedom at all.

I could tell a very similar story to you - 9/11, and uncomfortably close to various other bombs both before and after. Which, I suppose, allows me to say "For heaven's sake, man up". Sixty years ago my late ancestors decided to volunteer to risk death rather than risk becoming slaves, and I do not intend, as they say, to feel ashamed when I enter their company.

Arthur @48: back in the mists of time, I was living in London (okay, within the suburbs and hopping on the tube whenever I wanted to socialize) during the last Provisional IRA bombing campaign on the mainland. I, too, have been in buildings that were evacuated due to bomb threats or suspicious devices.

(The surveillance trend in the UK started in the early 1990s with the "ring of steel" CCTV net around the City of London, after the NatWest Tower bombing IIRC.)

The damage a minority insurgency can do to an open society is tiny compared to the damage we can do to it ourselves by over-reacting.

Andy W @53: I think you called it right; this is going to murder the grey cash-in-hand economy. Especially when they convince everyone to stop using the folding stuff and switch to nice contactless swipe cards a la Oyster (aka MiFare, but that's another hacking story entirely).

Me @ #7:
Seems I was over-positive, the proposal to cut the FRA law off at its knees is cleverly formulated to be powerful but is in fact neutered itself (it proposes to curtail the changes to the "intelligence gathering laws", but the relevant data gathering was in fact regulated in other laws).

Canis @ #52
I wrote this mid-July 2005. On average, I was inconvenienced by bombs/bomb-threats more frequently in Stockholm than I have been in London.

So we're agreed that an open society (Whatever one of them is, I havn't read the book) is a good thing, and the proposals will close it.

I like calling the snit-teacher the political officer.

Data mining won't help. For the very, very simple reason that WE ALREADY FIND ALL THE TERRORISTS.

Firstly there's all the ones they arrest before anything happens; the ricin plotters, and the drinks-on-planes group. Obviously they get found.

Then there's the 7/7 bombers, the 21/7 bombers, the shoebomber, the Glasgow flaming muppets.

Now they were ALL on the suspects list. We already knew about them -- we knew they went to dodgy mosques, had training in Pakistan, hung around with the wrong people. They were, in short, already on the list of the 200 most likely terrorists.

Our problem is that we can't watch 200 people. We can only watch the top 50 on that list. Our problem is therefore either that we cannot pick the 50 out of the 200 properly or we that we cannot just watch all 200.

Data mining the rest of the population's emails is NOT going to help with that even one slightly little bit.

Here's an idea: let's take over Iceland. Surely, with a well-supported effort, we could outbid the Russians? Look at the advantages - wilderness! cheap geothermal power, and cooling, for our mighty data centres! fish! the world's highest sales of books per capita! good bandwidth (Hibernia Atlantic and the Faroes-Scotland cables)!

It's a nation founded on people who couldn't stand authority in Norway 800 years ago, as Bjork so wisely put it; I'm sure they'll get the point.

Charlie @ 62

I agree, when we're thinking 70, 80s style terrorism. But 9/11 can even claim some credit for today's economic crisis --the Fed's reaction fueled the housing bubble.
So while it seems absurd to take my shoes off each time I fly, I'd rather do that and dump my soda then go through that news cycle again. And a WMD attack on a major Western city would do irreparable damage to our way of life.

I'm very concious (and afraid) of us getting to the Orwellian vision as boiling frogs -- as liberty erodes. I would be hard pressed to say that it hasn't.

But while parallels can be drawn between 1984 and "War On Fill In The Blank", they only go so far. There is a genuine threat, and a drive towards a society far less tolerant of dissent than any nightmare Rovian vision you might have.

Certainly in the US the pendulum swings on these kind of events (think Alien & Sedition Act), and the Constitution provides a measure of protection. It's not meant to ensure that no single person is ever wrongly accused or treated. It's simply meant to ensure that there's a mechanism for the pendulum to swing.

#51 Can't your Queen step in and do something here? Wasn't she busting caps at similar fascists 65 years ago?

The late Queen Mum trained with a revolver; the Queen drove a lorry in the war. Also, she's 82. Oh, you mean have a quiet word with the Prime Minister rather than shoot people? If she did, we'd never hear about it.

#62 back in the mists of time, I was living in London... during the last Provisional IRA bombing campaign on the mainland.

Me too. In fact they blew up our bottle bank (and rattled the windows of our flat). Similarily my brother sometimes used the tubes blown up on 7/7 to get to work.

This worried my Mum more than either of us; she's perfectly happy to go into London herself. But terrorists trying to blow up her sons concerns her a lot. If I were forced to guess, it's people like her that the tabloids are egging on into "monitor everyone as they might be a terrorist".

You know, if datamining could predict criminality, the insurance company I used to work for would have been more profitable (also less interesting as we'd have weeded out all the stupid criminals before anything happened).

BTW, does anyone here see any parallels between alarmism on all ends? That one man's alarmism is another's absolute priority?

For instance, we're wringing our hand about surveillance, but routinely make people separate garbage. What's a bigger actual intrusion into someone's life?

Arthur #69- what is your value system? What do you value? How do you decide what you value, how open to evidence are you regarding your views on what is valuable and why?

guthrie -- I would venture as open as any here :)

I don't mean to argue global warming. The point of my comment was simply to remind people that for the things we are passionate about, we're quite willing to not only call for sacrifices, but work for the enforcement of those sacrifices on others, willing or not.

As to which threat is more real, which is more imminent, I would venture to say most here are probably not genuinely qualified to judge, or no more qualified to judge than me.

Arthur @71: I'd disagree - who better to judge how intrusive something is on my life than me?

Separating garbage is no real biggy - It's just recycling. It's one of the punitive steps towards being 'green'. Nobody actually comes into my house and watches me separate the rubbish though.

Or then files exactly what each piece of rubbish is.

Unessecary intrusion into our personal and private lives is what's at stake here. For little gain and at great expense.

The point is that it's a simple step in a long slide toward totalianarism. And whilst you might say "Yes but that would require x,y,z still" the point is that by allowing this 'they' have already completed step w...which was originally a part of the list.

Serraphin -- but you might get a ticket if you dont separate. Whereas you have no idea that Carnivore is scanning your email, until the Swat Team is at your house, and that, I would argue, happens much more rarely then the tone of Charlie's email implies.

Look, I really do care about this stuff. I am trying to point out that each "interest group" has its own set of issues that it cares about just as much, and for whom the world ends if things don't go the right way.

The first step to compromise and live and let live is acknowledging that each religion is equally wrong, and that most, if not all, people in government, are just trying to do the best they could, often incompetently, to be sure, but rarely in the caricature that some attempt to paint.

Arthur, I'll separate out my garbage if you let me listen in to your phone calls, read your emails, and look in through your windows. Being told to recycle isn't intrusive any more than being told to drive on the correct side of the road is intrusive. It's apples and oranges.

ajay -- I'm sorry, I dissagree. There are innumerable ways in which the state intrudes, always with the "best of intentions", always, at some level, as in your road example, to make order possible.

It is apples and oranges, but you might prefer apples, and I oranges -- it is not your call. In a democracy or a republic, it's concensus. And that concensus is not on your side at this point. It may get there, and of course you are free to work for it, and in fact, that's probably your social responsibility. But it doesn't make yours or Charlie's a uniquely valid point of view.

Hmmm - the problem doesn't so much arise from one special interest groups intentions so much as their actions being used by another groups for their own purposes.

For instance - racial data is gathered these days for quite sensible and benign purposes. In the NHS it's used to determine requirements for blood stocks and the like.

Now imagine the BNP got a bill through parliament ordering the repatriation of people based on racial group - they have a ready made data resource to aid them in that aim.

Now you may say that the BNP don't have that kind of power - and you'd be right. At the moment.

What about in 5 years time? Or 10 years? Or 25 years? After particularly bad terrorist incident? Etc, etc

The problem is that once those instruments are in place they will never ever be taken away again, even if the reason they were started has disappeared. (Trivial example: there are still no bins on UK stations, although the IRA stopped using bin-bombs years ago).

Putting the tools of oppression in place is a really bad idea, even if at the time it is purely for benign purposes. Someone will always find a way to use things in an unanticipated manner.

IT is not about keeping this sort of capability out of the hands of this government, but out of the hands of ANY government (and I remember the Thatcherite era - with this capability we would have had a police state).

Andy, there are unintended consequences to all laws and instruments, to be sure, and I totally get the slippery slope argument.

The argument that I picked is that all groups use the same "marketing" methods. You may see hysteria in global warming, I may see hysteria in terrorism, etc. We may both be right, or we may both be wrong. :)

Arthur: Rereading your posts, I see where you are coming from now.

But back to one of your earlier comments - Unlike most I am in a position to see the full potential impact of the introduction of these sorts of systems. I'm a database specialist, I know what I can do with those systems.

I find it scary what I can do with the current systems I can access via the interwebs, let alone this concoction!

If you want to see what the police do already to peaceful protesters outside Parliament, take a look at this this news item. I assume that people who don't object to continuous electronic surveillance, similarly find nothing wrong with this kind of physical search by the police.

I don't know what they do by Parliament, I live in the States. But this is exactly why I don't want the state to have a monopoly on gun ownership.

Arthur @80:

I have to agree. I occasionally get some odd looks here in the US, based on the facts that I self-identify as a liberal, but that I also interpret the Second Ammendment to include an individual right to own firearms.
To me, this isn't contradictory; I see it as simply one more personal freedom that should championed. I live in West Virginia, and my neighbors tend to be a socially conservative bunch, with whom I often disagree. Despite that, almost all of them own both handguns and longarms, and almost none of them have ever fired on another human being in a non-military context. As they've long demonstrated that they're responsible gun owners, why would I attempt to curtail their freedom?

Scott
They just investigate the tiny minority that annoy the boss or his boss or their friends, associates, and relatives.

This is something of a tangent, but is the any research into how much surveillance it takes to make people self-censor themselves? It strikes me that a government might convince people of panoptic-ness before they actually reach it.

Then again, social networking sites suggest acceptance of observation without deterring antisocial behavior...

Arthur @67: You say "There is a genuine threat".

If I say: Rubbish, support your statement with hard facts, perhaps by comparison with historical events (e.g. IRA bombing campaign in England), how would you respond?

Because I don't see that there is any more of a threat that there's ever been. Probably less, now. Certainly not enough to justify universal surveillance.

At risk of dragging the thread too far off topic, it is one thing to support peoples right to own guns (assuming some minimum responsibility is demonstrated, or heck, at all without any kind of check or balance) and another to suggest that individual gun ownership will in some way help curtail the gvt and its minions.

Blue09 @83: See also happy slapping for some insight into what it takes to deter violence (not).

Walter and Scott: I'd like to keep the subject away from guns, if you don't mind; it's a hot-button issue for too many people. (My opinion, for what it's worth, is that I believe in control of access to firearms but I think the UK has gone much too far in that direction, and I think the post-Hungerford but pre-Dunblane status quo, if properly enforced and administered -- they weren't, in Dunblane, with tragic consequences -- would be sufficient.)

As for the right to own arms being a check on government, get back to me when your right to own arms extends to owning your own carrier battle group or armoured brigade. Civilians with pistols do not an army make, in this day and age.

Chris @84, I would respond by saying you didn't feel the builing shake when the airplane hit it. I am -- admittedly -- not pursuadable on this subject.

For the record, I am not for universal surveillance -- unless I'm mistaken, my DVR does not yet record me in the room and sends reports to the government, and if it monitors what I watch, that's more likely to be used by Google than by the government.

However, if there's a face scanning algorithm that's monitoring crowds in airports, it certainly has my support.

As to data mining -- I'm skeptical as to its efficacy, but after 9/11, we underwent extensive hand wringing over NOT noticing patterns, and so now we are attempting to notice them. That is how I view it. Is it going to be abused? No doubt. Legal bounds do exist. But it cannot really be stopped, given advances in technology.

I think we better get used to living in a post-privacy society. Imagine the circa 2008 Facebook profile of a conservative candidate running for office in 20 years? No privacy includes total transparency at some level, and I believe it will compensate.

@83: Good question Blue and I'd be intersted in how much self-censorship is involved as well. Social networking though is dominated by the equation "Person" + "Total Anonymity" = "Fuckwad" (apologies for the intemperate language!)

@Arthur, Robert and other US friends: Think we have a case of "2 cultures seperated by a common language" here (thanks Oscar) and I'll self censor myself and bow out lest I overstay our hosts hospitality!

I tried writing stuff to describe it, but lets just say the UK is nothing like the US either socially or politically and leave it at that (Gun ownership? WTF?).

Peace.

But Arthur, you have some control over what you put on your facebook, and certainly the blowback 20 years later assuming things are archived that long, will hopefully help foster something closer to my favoured ideal of acceptance of our common humanity (i.e. we all fuck up).

Whereas having the gvt or their appointed minions in private companies watching you in a panopticon singularity is not voluntary, and you have no control over it. Possibly you could move abroad, but thats quite extreme. Better to fight it first. Here in the UK we have faaarrr too many cases of gvt supression and stupidity to make us blase about it all.
Plus this is not a Van Vogt novel, the technology is not omnipotent.

I'm going to throw out a couple of book suggestion and a link here.

The first book suggestion is Edwin Black's IBM and the Holocaust. Information gathered for innocuous purposes can demonstrably be used for nefarious purposes later. And, rather oddly, I'm not convinced that the large corporations that run many of our countries can be trusted either—see further details in Naomi Klein's Shock Doctrine.

The website reference is Peter Watts' blog posting summarizing the current knowledge of the effects of biology on political decisions:

http://rifters.com/real/2008/10/understanding-sarah-palin-or-god-is-in.html

(I'd actually be rather curious as to what Charlie could come up with riffing off those ideas, combined with the government having access to massive databases including the NHS, which presumably contains data on people's neurochemistry and medications.)

Arthur @80, you never know who's being watched. The former governor of Maryland not only had police infiltrate non-violent activist groups, but put 53 of the activists on the federal terrorist list.

This is something of a tangent, but is the any research into how much surveillance it takes to make people self-censor themselves?

I can't provide a reference to research, but I'd say none at all. You can get people to change their behaviour simply by suggesting that they might come under observation at some point. You don't actually have to do it.

A worrying number of people enjoy co-operating with the authorities and informing on their neighbours. A documentary on the Nazis shown on the BBC some years ago (The Nazis: a warning from history, I think), investigated how the Gestapo maintained control over the population when they only had about 2 men observing a medium sized German city.

They relied on ordinary people writing to them and providing details on the conversations and daily movements of the people who lived next door to them. The allies captured a whole Gestapo building full of these letters near the end of the war and historians used it to reconstruct how the system worked (naturally, the Gestapo destroyed most of their paper files as the Allied armies advanced toward them - in this city, they got liberated too quickly for that standard procedure).

The documentary makers talked to a German woman who had informed on her neighbour and whose letters appeared in this surviving archive. No evidence existed of any subversive views or activity on this other woman's part. She kept to herself, didn't talk much to anyone locally, had one friend who lived outside the city and visited her once a month. This neighbour wrote multiple times to the local Gestapo explaining that her neighbour, while she didn't say anything against Hitler, didn't express her support either. She spread innuendo about her, raising suspicions without any evidence whatsoever. Eventually, the Gestapo arrested this poor Fraulein and shipped her to a camp where she died.

For nothing at all.

It's not enough not to be neutral; you have to openly profess your support at all times for the government before your neighbours in situations like this. Not praising the authorities makes you look suspicious and very little at all can lead to action against you.

East Germany re-created the same system with the Stasi - friends, family members, work colleagues and neighbours all collaborated in spying on each other. It's frightening easy to get people doing this.

Arthur @87: If you really were in the WTC when the plane hit, I can understand your point of view (and you're incredibly lucky to be alive). If you weren't, then you're playing rhetoric rather than answering the question, and that's a game with no end in sight. Not to mention quite disrespectful to the memory of the people who did die.

Correct me if I'm wrong, but the FBI knew about the atypical flight-school students before the event: word failed to travel up the hierachy. How is increasing the amount of information available going to prevent that from happening again?

An interesting point for everyone to ponder, made by a security expert on a doco I watched: England is basically the only European country that's never had a secret police. So you don't have the memory of, and reaction to, the idea of surveillance that people in Germany (just to pick and example) would.

guthrie @89 -- My father spent time in the Gulag and I lived in the Soviet Union, and I'm a computer scientist who well knows what computers can and cannot do.

I have in fact been exposed to that system, and to terror, and to living in the States.

Warts and all, I do think citizens have to be vigilant -- of course. But I am not meaningfully worried about govt. spying on me. And I am meaningfully worried about being killed while I'm in NY -- from either crime or a terrorist act, and I have experienced such life threatening situations directly. So I don't think it's a language thing -- I think it's a life experience thing

@88:
"(Gun ownership? WTF?)"

http://en.wikipedia.org/wiki/Gun_politics_in_the_United_Kingdom
UK:
1903 - Pistols Act
1920 - Firearms Act
1937 - Firearms Act
1968 - Criminal Justice and Firearms Act

As far as I can tell, you Brits can still own guns, it's just your legislation is far, far stricter. The question I have is do you Brits have places where you can actually discharge firearms such as a target range or a place for skeet-shooting?

Now the state I currently live in has no restrictions at all:
http://crime.about.com/od/gunlawsbystate/a/gunlaws_mo.htm

I don't personally own any guns, but know many people who do.

92; Look at the "Denunciatrix of Kiev", Polina Nikolayenko. All tyrannies rest on these people. Up to a point.

The Stasi was much more comprehensive than the Nazi state; it really did have people directly on the payroll, with a direct relationship to a case officer, everywhere. This is almost certainly because the DDR had to work at it; they couldn't rely on the legitimacy of the German provinces and the Army and the civil service and the landowners, they were obviously, blatantly, a tyrannical imposition that rode into town on an enemy tank.

Similarly, the Stalinist terror relied on the fact that it had some legitimacy; you just can't keep on keeping on with those scales of numbers...

92; Look at the "Denunciatrix of Kiev", Polina Nikolayenko. All tyrannies rest on these people. Up to a point.

The Stasi was much more comprehensive than the Nazi state; it really did have people directly on the payroll, with a direct relationship to a case officer, everywhere. This is almost certainly because the DDR had to work at it; they couldn't rely on the legitimacy of the German provinces and the Army and the civil service and the landowners, they were obviously, blatantly, a tyrannical imposition that rode into town on an enemy tank.

Similarly, the Stalinist terror relied on the fact that it had some legitimacy; you just can't keep on keeping on with those scales of numbers...

Fellow @ 92: A large chain bookstore I worked at years ago had fake cameras all over the store and employee areas. They were big and obvious -- the mere idea that they were being watched was enough reduce theft. Actually catching and prosecuting theft wasn't a priority, the goal was to get people to worry they were being watched and not do it.

Sorry for the muppost.

Some of the stupid about the gun legislation has been unwound; to begin with, the Olympic marksmen had to train in Belgium, but this has been sorted out.

@95: That was more of a "What has that to do with the price of fish?" comment, I am familiar with UK gun legislation! If you want to shoot, join the military or it's air weapons, shotguns (clay pidgeons/vermin) or antiques only!

In the UK the issue of gun control is irrelevant to the issue of government surveillence, whereas in the US it seems to get linked to everything.

Like I said, cultural differences hidden by a common language (actually a quote form Oscar Wilde). As Arthur said in 94 - "It's a life experience thing" and in the UK we have had direct experience of terrorism for nearly 40 years.

Thome: To move off topic further (Apologies Charlie..), There are gun clubs, target ranges, and clay pigeon shooting sites, and I would assume that gun ownership in rural areas is relatively high. However, in urban areas, there is no need to own a gun, and gun ownership/gun clubs/shooting ranges are discouraged. Remember that the UK is generally much more densely populated that the United States, with correspondingly more strict gun laws.

This is something of a tangent, but is the any research into how much surveillance it takes to make people self-censor themselves? It strikes me that a government might convince people of panoptic-ness before they actually reach it.

Then again, social networking sites suggest acceptance of observation without deterring antisocial behavior...

According to research quoted by Peter Watts, it seems to take just a pair of eyes painted on the wall to change people's behaviour.

Paper in Science: http://www.rifters.com/real/articles/Science_TheOriginandEvolutionofReligiousProsociality.pdf

Interesting summary by Watts: Which brings us to Norenzayan and Shariff's review paper in last week's Science on "The Origin and Evolution of Religious Prosociality". To get us in the mood they remind us of several previous studies, a couple of which I may have mentioned here before (at least, I mentioned them somewhere — if they're on the 'crawl, I evidently failed to attach the appropriate "ass-hamsters" tag). For example, it turns out that people are less likely to cheat on an assigned task if the lab tech lets slip that the ghost of a girl who was murdered in this very building was sighted down the hall the other day.

That's right. Plant the thought that some ghost might be watching you, and you become more trustworthy. Even sticking a picture of a pair of eyes on the wall reduces the incidence of cheating, even though no one would consciously mistake a drawing of eyes for the real thing. Merely planting the idea of surveillance seems to be enough to improve one's behavior. (I would also remind you of an earlier crawl entry reporting that so-called "altruistic" acts in our society tend to occur mainly when someone else is watching, although N&S don't cite that study in their review.)

I sometimes wonder if all the talk of surveillance is as much to discourage cheating as it is to catch criminals. Then I cynically wonder if maybe the real target isn't criminals but opponents. Say, for example, anti-war protesters.

http://www.washingtonpost.com/wp-dyn/content/article/2008/10/07/AR2008100703245.html?nav=rss_print

England is basically the only European country that's never had a secret police.

Either that or it's really really secret.

Neil: like, nobody expects it-secret?

Has England/Britain/U.K. never really had a secret police? In the nineteenth century Irish physical-force nationalism inspired the creation of the Special Irish Branch at Scotland Yard, later to become the Special Branch. . . would that count as a secret police force?

Also, I remember reading in the New Statesman back in the '80s that in the event of a third world war the UK govt. had lists of thousands of potential subversives to be rounded and interned when the balloon went up. I wonder where they got the names?

Irish physical-force nationalism

Best. Euphemism. Ever.

"Well, I wouldn't call it rape as such, it's just physical-force matrimony..."

Arthur back @73 sez "but you might get a ticket if you dont separate. Whereas you have no idea that Carnivore is scanning your email, until the Swat Team is at your house, and that, I would argue, happens much more rarely then the tone of Charlie's email implies."

I really, really don't see how you can even use the two to compare!? AS pointed out it's like complaining you have to drive on the left (here).

The rubbish example is a simple example of how your local govt' has decided community services will work. Apparently they give you a kicking if you take a shite on the pavement too - does that bother you?

Lets put it this way - you don't mind if I read all your mail? I'll put it back, promise I won't do anything with it - I'll just read it...well that and photocopy it. And archive that copy.

Promise no-one will look at it and use it though.

I think you might be giving over-reliance on what the proposal would mean. If you've actually experienced terror then I can udnerstand how you want "Them" to protect "us".

But here's the perfect question.

You've got a copy of every e-mail, URI request, all net traffic, etc.

How do you use this to PREVENT an attack? (of which there are actually happily few).

Best euphemism ever? I'd say that title belongs to 'friendly fire' for 'killing your own troops'. And we need some way of distinguishing between the physical-force people and our constitutional nationalists. . .

Depending on how you define secret police, the British gvt was using paid informants and moles in subversive organisations back in the 1780's onwards. THey probably had them in Jacobite socities before that, but obviously in the 1780's they were concerned about the French revolution etc.

Arthur #94- I think you're right- its a life experience thing. But also people choose what views to take- after the July tube bombings one paper used a photo of an injured person to press for tighter anti-terrorism laws, the ones we don't like. The person whose photo they used objected to this, because they didn't like the proposed laws any more than people who hadn't been injured in the attacks.
You might not think a panopticon singularity is possible- Charlies point is that it doesn't even matter if it works properly or not, because they'll have wasted billions and changed our culture for the worse, as well as made it easier for the gvt to persecute those it doesn't like, and meanwhile the actual terrorists will end up falling through the gaps.

Didn't the first Queen Elizabeth have a secret police? Run by Walsingham?

Robert @110: the first Queen Elizabeth was also an absolute monarch, ruled more-or-less directly over a population of 2-3 million subjects, issued letters of marque (state-sponsored piracy, in effect), and the standard judicial penalty for most anything more serious than a misdemeanor was hanging.

Comparisons are, I feel, not terribly useful.

Going by my (faulty) memory, I think the UK gvt has had spies and suchlike in anything it considered subversie (i.e. dangerous to the current regime) for over 200 years.
However they require an actual police officer/ acredited representative of the law to carry out arrests and so on. They can't just arrest people themselves. Or has that all changed?

If 'they' ever get the ID card system to lumber off the ground, I suppose we'll have to carry them. As to being asked to produce them on demand, that's fine too, as long as I can be absolutely sure that the request is genuine. Unfortunately, I don't know what a police ID card looks like (let alone central or local government ID), so in the absence of certainty about the identity of the requester, I should be inclined not to produce it. Yes, I know that the police learn a sort of surly agressiveness when dealing with the public that is unmistakeable, but that does not constitute evidence of identity.

111: as mentioned on Making Light a week or so ago, the most recent letter of marque issued was in 1942, by the US government to (of all things) a civilian-crewed airship being used for anti-U-boat patrols.

108: "friendly fire" isn't a euphemism, it's a straightforward description. It's fire, and it's from friendly forces. The opposite is "enemy fire".

A euphemism would be something like "value-negative intra-force munition delivery".

112: AFAIK it is still the case that neither MI5 nor MI6 have powers of arrest.

Serraphin 107

Let's take a different example than the garbage example. Am I correct in assuming that you may get a visit from the authorities if your child's teacher noticed some unusual bruises? Doubtless every year children get abused, or even killed, because a teacher was not sufficiently vigilant. And yet doubtless every year there are also cases of parents getting dropped into a nightmare of having their kids potentially taken away from them because Johnny fell in the playground or is unusually clumsy. So a very intrusive law -- literally, the state is monitoring your fitness to be a parent -- but one that we put up with, whether we like it or not, to minimize risk to children.

When you call it "the surveillance of every email" you make it seem that a salivating beaurocrat is opening Serraphin's email box, impatiently skipping over the solicitations from Russian women that want to meet you, and looking for the meet up of the armed revolutionaries.

The reality is that it's scanning trillions of bits for keywords and patterns and you are in total anonimity, and no one cares about how you answer those Russian women. Yes, if it sees what it thinks is a bruise, it might call in the authorities. Likely it'll mostly miss, and occasionally it'll fire on a false positive, and boy, when that happens, that's really unpleasant. But no more, and no less.

I agree with most points made with relation to surveillance. But the ID card thing is blown out of proportion. I've had an ID for 12 years now, and I am compelled by law to take it with me to go anywhere. Somehow I haven't gone to the Gulag yet. I understand people who have no collective experience of ID cards can get really paranoid about it, but I don't think it is as big a deal as many make of it.

115: problem is, beating children is a lot more common than planning terrorist attacks. So the same rate for false positives would give you very very different results, and social policy has to take that into account. If a bruise-detection programme points at 50 people 10 out of which are guilty that might be justifiable. If the same programme points at 50000 people 1 of which might be guilty, you have to start thinking carefully about it.

@69' For instance, we're wringing our hand about surveillance, but routinely make people separate garbage. What's a bigger actual intrusion into someone's life?

Where I live, you can have both - there are designated neighbourhood representatives who will often keep an eye out for missorted stuff, and some of them are quite dedicated about tracking down the perpetrators. I've heard similar tales from another former Axis power (though it sounded like they were spying on each other more for a sense of nostalgia there).

David @116: the problem is not the ID card, but the National Identity Register the government is building (using it as cover). See No2ID for more information.

Further to Andy@76 and Robert@90, for an up to the minute example of how authoritarian overreach operates check out Brown's decision to freeze Icelandic assets as part of this week's OMG-Icebank-Is-Imploding! drama.

It was done using statutory instruments introduced by the Anti-terrorism, Crime and Security Act (2001). As you might expect, being tarred with the terrorist brush hasn't gone down too well with the govt of Iceland although I imagine it's mostly the loss of their UK assets that stings.

HMG now have 28 days to get the freeze approved by both houses of parliament or... they'll just have to issue another freezing order.

This one could run and run.

Regards
Luke

Luke@120:

And what's slightly disturbing is that the PM is threatening to use these powers to confiscate further Icelandic assets in the UK if the Icelanders don't cough up the money invested by UK councils. This is really extraordinary: countries' leaders don't usually make those sort of threats unless they have names like Gadaffi or Mugabe.

Bollocks. If Landsbanki wants to buy half the retail sector, they can damn well meet their obligations to depositors, or else. Everyone else has to pay their debts or have their assets liquidated to pay them, why not them?

Wait a minute. How much does it cost to do proper old-fashioned surevillance on someone? Pulling some numbers out the air, I guess you'd want three people watching and one supervising 24 hours a day. Give them 5 shifts to cover illness, leave, training etc. means you need 20 people. Double the number of people to give you backup, support, administration, management etc. With salaries, civil service pensions, transport, equipment, office space etc. I'd say somewhere in the region of £100,000 a year per person. So that's what, £4,000,000 a year to watch someone properly.

Up @65, katie suggests we can only watch the top 50 suspects. For the £12 Billion mentioned up the top, you could watch the next 50 for 60 years.

Hmm, thinking about it, proper secret agent work is expensive. I can almost see why you might be convinced that £12 billion on an IT project is better than employing 40,000 agents to watch the 1,000 most dangerous extremists (if someone put those as the two options - after all we don't want a terrorist attack on your watch do we minister?)

One of the interesting things about ID cards, and I think it carries over into a lot of other crap, is that, where they are used in Europe, both Police and People are used to them, and have an historical awareness of how the system can be abused.

Here in the UK, we might not be all Brazilian Electricians, but we have the feeling that we can't quite trust the Police with something new. We don't have any feel for how they'll use a new power. And, ID cards in particular, we have the lurking memeage of WW2 Europe complete with "Round up the usual suspects."

It doesn't help that the heads of British Police Forces are getting a general reputation as authoritarian thugs who always want more power, with some of their immediate subordinates, at least, inclined to fiddle expenses.

There's a lot wrong, I think. At least some of the paperwork claimed to be emasculating the Police is in response to past abuses. Do I have to mention Life On Mars? And a lot of the problems come from bit-by-bit political fixes. Nobody has looked at the system as a whole.

Done right, something such as the iPlod could make a huge difference.

But too often, I'm left with a feeling that the people at the top don't care about honesty and integrity. And they want to force us to carry ID cards?

The UK has a secret police force: the Serious Organised Crime Agency. They operate out of a PO Box number in London; their membership, chains of command and physical locations are state secrets.

Oh, and they're exempt from the standard Police loyalty of oath to the Crown, so they're secret state police... how does that go in German again?

Dave@124:

Good analysis of the complexity of the issue.

I work in biometrics and identity management for the US government, supporting projects for the Departments of Justice and Defense. I'm just a contractor, but I've seen firsthand the struggle that the career Federal employees make, the tightrope they walk to pursue legitimate security needs while not infringing on privacy rights. The better, more honest, more thoughtful ones acknowledge that their agencies have often had astonishingly bad civil rights records, and they try very hard to serve as moderating influences and to rein in the authoritarian instincts of some of their peers. In the past seven years, the moderates have felt pretty lonely, with the various Federal agencies trending toward paranoia, but the pendulum seems to be swinging back again. Only time will tell.

I see even the pretendy-wee rush job version of ID cards is already over budget and slidin' right.

I know this if off-topic, but when is the UK going to start the invasion of Iceland? After all, I just read the following which has me laughing for the first time in quite a while...

http://www.thisislondon.co.uk/standard/article-23571164-details/%E2%80%98Cold+War%E2%80%99+talks+as+Iceland+crisis+engulfs+charities/article.do

A spokeswoman for the National Cat Protection League — now known just as Cats Protection — said: “We believe there is a case to be made to the Treasury that its particular deposits are public money that has been donated for us to help cats.”

Charlie @111, I was responding to Chris L's comment @93: England is basically the only European country that's never had a secret police.

Given how much attention the continuity of England is given in history books (unless they've been rewritten since I left), the statement as made didn't seem to be true. No secret police since Victoria, sure.

WRT IDs, if you get a modern ID card you can do quite cool things with it. For instance, the new Spanish ID cards (new since 2005 or so, although they were in the works since the late 90s maybe) have a smartcard in them. In this smartcard, inter alia, there are digital certificates with a public/private keypair signed by the State's private key, so you can assure your identity with a physical token (needless to say you need to enter pin and so on). This has made it possible to do a lot of traditional bureaucratic paperwork over the net, just plugging your smartcard reader and using your ID card. This is IMO one of the core functions of the state: ensuring people's identities, so I can't say I mind. You can use these certs for e-mail and so on too, of course, and the privkey is generated in-chip so the state cannot decrypt things sent to you.

I thought I could find some discussion of the "anti terrorism" laws and Iceland's banks here. What is going on there? I thought the US financial crisis was bad-crazy enough already.

The latest story seems to say that Kaupthing's assets were frozen at will by Darling, before any kind of intent to default was announced, and that that in turn forced them to declare bankruptcy. (Because you can't very well repay your depositors or let them withdraw money if some government has just seized most of your assets, can you?)

The big problem here is that the entire world banking and monetary system is run on faith, and has been ever since a bunch of Italian goldsmiths invented fractional-reserve banking during the Renaissance. Virtually by definition, banks don't have enough assets to pay back all the depositors if they all demand their money at once; the trust is that in aggregate across all banks as a whole, the money is there somehow, somewhere, loaned out to others who will eventually repay it. At any given point in time, the fact is it's not there in any given bank or banks you look at, though. Only blind faith makes the system work.

Iceland's not unique except in being a small country. I expect if other countries were to panic and start doing the same with British banks, the UK would abruptly find that every UK bank was bankrupt too. Surely Darling understands that?

@123 only trouble is when you skimp on those 20 people and end up killing someone because the Ident guy was taking a slash

If we can't trust the 'government' to bang some databases together, or look after the data they have at the moment, if I share biometric data with 19 other people in the country; what's stopping the percentage of false ID's rising from its current 0.3% (60M/0.2M)?

http://www.guardian.co.uk/money/2008/oct/08/identityfraud.immigration

I was going to say something pithy about the young and social networking sites. Lets leave it at, the human mind is marvelously self deluding, especial if you have little experience of the adult world.

When I first heard about this, my reaction was "they can't do that". A quick back of the envelope estimate of the volume of data they'd be sucking in suggested they'd need to build a new 400Mw power station just to keep the drive arrays spinning. But with a price tag of £12Bn being bandied about ($20Bn, to you Johnny Foreigner types) it's time for second thoughts.

The population of the UK is 60.9 million. Let's say each of these 60.9 million loyal subjects sends 40 emails per day, which I think is on the high side. And let's say the government doesn't store duplicates of the mails (so that they only have to keep one copy of spams, and one copy of messages to mailing lists) and only keeps the first 10kB of text from each message, compressing it with bzip2 to get a factor of, say, 3. (The FM-index is a variant of bzip2 that permits full-text search as well as retrieval.) That's 8 terabytes of compressed, indexed mail per day. Disk currently costs US$0.13 or so per gigabyte, so that's about US$1100 per day, or US$770k for two years' worth of storage. (Maybe multiply that by two for the sake of redundancy, indexing, and scratch disk space for ad-hoc query evaluation, but it's still a cost that certain science fiction authors could pay out-of-pocket.)

I'm going to ignore the cost of storing URLs visited, SMSes sent, phone calls placed, and IM, because only IM could plausibly approach the 400kB per day number I'm citing here for the cost of the email.

That adds up to 6 petabytes of disk space, which could be 6000 disks in 1500 servers. Say each server uses 1.25 amps, and that the data center as a whole wastes another 1.25 amps per server for cooling, lighting, etc. (Google has shown that you can do a lot better, but I hear that a factor of 2 is typical.) That's 3750 amps, or 900 kilowatts, which would not require a new power plant. It would cost about US$90 per hour (US$0.8M per year) to run if you located it somewhere random, but could cost much less if you located it somewhere with cheap power and water.

It sounds like my figures are lower than what you came up with, by a factor of 450, and they are well within the budget of many private individuals, not to mention being around one ten thousandth of the £12bn budget you mention. Why is that?

Anyway, escaping the dystopian panopticon is one reason I moved from the US to Argentina in 2006. You should come try it out. We have a guest bedroom. (Ask Danny O'Brien for a character reference for me. ☺)

Graet Kragen, you just made it even more likely they'll do it...

Potential solution to the manpower problem:

Tell each person from childhood on it is their civic duty to monitor five other individuals, assigned by them to the government, who also provides the monitoring equipment. Each day, each person rates them on a scale of 1 to 10 of how much of threat their behavior seems. If the overall rating gets above, say, 35-40, the higher-ups get involved. Since it involves consensus, it is more resistant to personal feelings of hatred, etc. And people love to spy on others; it taps into the natural voyeuristic tendencies of being human.

Just to clarify, I do NOT support this system. I was just looking at it as a solution to the logical problem of how a government might go about monitoring everyone.

Of course, a problem is that this will tend to drive out eccentricities and outliers (including geniuses), with an ultimately detrimental effect on the culture.

I wasn't meaning, way back@93, to generalise right across European history! Can we insert "within living memory" or "since James Cook started making maps" where I've written "basically"? :)

Is there an international pattern here? I mean, here in Sweden we have the new law about internet surveilance, which supposedly uses a super computer and some foolproof filter that is said to be better than any known Bayesian filter to capture terrorists and other threats to national security while letting the everyday innocent guy through without as much as a glance from human eyes.

I am probably wrong, but it feels like all this kind of got started by the entertainment industry. I mean, they have been lobbying for this kind of surveilance since Napster, right?

Kragen @133: let's posit 20 million households in the UK with broadband.

Going by the ISP figures, only about 5% use more than 5Gb per month of bandwidth. Great, let's approximate that to our third normal distribution, and (being lazy) posit a bell curve for the rest. On that basis we're probably talking on the order of 1Gb/month per household for 20 million households. And that's just a starting point -- it works out at a little under 250 million gigabytes per year.

Bear in mind that 90% of email traffic is spam. Keeping only one copy of each spam sounds great, but the spammers are already aware that Google compares emails to multiple recipients and weeds out the massive identical mailings, so they've taken to adding random noise to them. Trying to keep spam out of the national email database is therefore going to be as hard for the authorities as keeping spam out of your own mailbox.

Bear in mind the government isn't talking about just checking email. There is a risk that any new TCP/IP protocol will be used as an organizing/communications channel by Bad Guys. (See for example the recent alarm stories about terrorists using a Second Life guild as a chat room for organizing.) Therefore we need to retain all packets delivered by unrecognized, new protocols until we can confirm that they're safe. Say you've got 5 million gamers regularly using 100Mb of bandwidth per day -- at that point the games alone are soaking up 500Tb of storage per day. (And their protocols are likely to be encrypted and highly proprietary; doubtless the games companies can be induced to play ball, but the users are always one step ahead of the admins' expectations. How long is it going to be before we see scare stories about terrorists with exotic input rigs holding meetings in Second Life by semaphore?)

Then there's peer to peer traffic (and spam again). You can't ignore them, because the p2p or spam zombies may be sending messages steganographically encoded in poor quality rips of "Life on Mars". Or the timing of UDP packets on the network may be being modulated, and a covert communication channel layered on top of the timing modulation -- very wasteful (without compression you're sending at most one bit of data per entire UDP packet), but still useful if you're talking about Al Qaida and the signal the bad guys are waiting for is ATTACK AT DAWN TOMORROW. So you need to monitor that rubbish -- and it mostly is rubbish, from an intel point of view -- and be very careful what you discard.

I'm assuming that they're not just logging a call register of all mobile phone calls, either -- they want to store location data (whenever your phone handshakes with a base station, they want to know which one) and, ultimately, to log the voice data stream (if they're storing email, why should voice calls be exempt?).

I think the 400Kb/day figure for individuals is massively on the low side, by about two (maybe three) orders of magnitude. Even more, once they join up the dots and start storing the entire take from the millions of public CCTV cameras. Oh, that's something else I forgot: business broadband is increasingly used to transmit the take from CCTV security cams to offsite storage (so that thieves can't nick the slow scan VHS recorder when they break in, to conceal their identity). This is only going to grow and grow, and we're probably going to see a situation where every one of the half-million shops in the UK need to be recording multiple high resolution video streams offsite on a 24x7 basis as a condition of their insurance. (See also: MMO gamers for an idea of the bandwidth cost.)

Finally: what about backup? Online backup at that? (Double the disk space: yummy!)

Yeah, you really can't discard the spam. I've long thought that spam is the ideal way for a sufficiently-motivated organisation to communicate covertly. It's like the Internet equivalent of a Numbers Station: It's easy enough to figure out roughly where it's coming from*, but it's being blasted out to the entire planet, all the time, so you have no idea who it's intended for, rendering all the traffic-analysis peer group identification systems useless. As Charlie says, they have random chaff inserted anyway, to defeat Bayesian filters, and are frequently horribly misspelt, so you don't even need to get fancy with the steganography, you can just dump code groups right into the spam.

Incidentally, there's another cost/source of data to be considered, that I haven't seen mentioned so far: the state itself. Once a surveillance state gets sufficiently baroque (and they inevitably do; total surveillance leads to total paranoia leads to total weirdness), it starts monitoring itself -- or rather, different branches start monitoring each other.

Departments never trust any of the others not to be infiltrated or inept, so they start poisoning the centralised database with fake data to see if it leaks, and withholding accurate data. Someone else has their suspicions about it. Trust in the central DB starts to go down. Secret copies of the data are captured further back up the food-chain by competing departments as a verification exercise (ie does the data that $department submits to the central database match the data they originally collected? "If we can show it doesn't, we'll nail those bastards in $department and secure funding for $project!").

And that's just the emergent effect of a series of well-intentioned (if flawed) behaviours. If you throw in intentional outright corruption -- someone, somewhere fiddles a piece of data for personal gain, or spying for blackmail purposes, or trying to protect someone they have a personal connection to, or whatever -- more and more of the surveillance infrastructure is dedicated to surveilling the surveillance infrastructure...

* assuming Total Surveillance, so you can trace back from the zombified XP box in Ohio back to the controller in $badplace. Doesn't help you much though, since you have no idea how it's bouncing around once it disappears inside non-cooperative territory

Not only could you run a terrorist network via spam, you could make a profit sending spam whilst doing so...

Oh wait, I guess the Russian Business Network has already beaten me to that business model.

And then the crime moves out of the cities, and you have to extend the surveillance network

"I'm assuming that they're not just logging a call register of all mobile phone calls, either -- they want to store location data (whenever your phone handshakes with a base station, they want to know which one) and, ultimately, to log the voice data stream (if they're storing email, why should voice calls be exempt?)."

The first part (calls and location) are logged already. All SMS messages are stored as well. I worked briefly at a telecomms company last year. I know this data is stored becuase I was in charge of the database storing it!

Bear in mind the majority of the data described is stored in some form already, and, in the case of e-mails and texts, on several intermediate servers as well.

The other thing to remember is that in many cases the only infomation that needs to be stored is that a call was made or message passed - who from, who to, when and where - and that can be encoded into a very compact form.

You build a "watchlist" and only keep the stuff for targeted individuals (potentially tens of thousands). Otherwise it's keyword scannng etc and only keep if interesting.

Ultimately what they are aiming for is the legislation to give the security and revenue services carte blanche to build whatever systems they can feasibly build (and it will comprise multiple systems) without having to come back to parliament in 2 years time when Gordon wants to tax the number of underpants people own.

It will only work against stupid criminals and terrorists since the smart ones will be able to evade it (with carzy commuications methods like sending hand-written letters through the post).

It will have blindspots, as well as be bloody intrusive should the government decide you are worth keeping an eye on.........

Charlie@138: I wasn't trying to estimate the storage you'd need to build a system that would actually prevent private communications altogether; I was only trying to estimate the storage you'd need to build the system they said they were going to build in the text you quoted, viz.:

retain details of all calls, texts, emails, instant messenger conversations and websites accessed in the UK for up to two years.

Now, it's all very well to point out that actual organized crime has the wherewithal to communicate by posting encrypted blobs of text as YouTube comments on popular videos and using the seconds field of message-board postings, but they're not promising to wiretap that. They're just promising to save details of all calls, texts, emails, IM conversations, and websites accessed, and that seems eminently feasible to me. You know and I know that that isn't going to stop the IRA from distributing bomb-making manuals over SATA at LAN parties, but I'm not trying to claim that the government's scheme is well-reasoned, just frighteningly feasible.

After all, when that MP who doesn't much like the MI6 is promising his secret lover he'll pay for the abortion as long as his wife doesn't find out, he probably isn't going to use private information retrieval through a dining-cryptographers network, or steganographically encode his message in the time-varying network load on an Amazon EC2 cluster gateway (with loads of ECC, natch) where his sweetie also runs an instance. He's going to use Facebook or send her a text, which will be duly logged and analyzed. And that's the kind of scenario I'm concerned about. I'm sure the police will perform their legitimate duties adequately even without access to such a database, and they might actually have an easier time of it. But it should be a goldmine for the intelligence services' ability to blackmail the citizenry.

The spam problem is an interesting one. But a number of companies seem to have come up with more-or-less working solutions to it, at a budget of under US$10M each, so I don't think it's a showstopper. You'd just want to tune them for a lower false-negative rate, perhaps. If you wiretap POP and IMAP rather than SMTP and QMTP, you can even use the user's own spam filter and save the cost of developing your own.

@134: believe me, the British intelligence services are capable of writing a capex budget proposal without my help.

@ 61 - you actually made tears come to my eyes with that post ....

@105 - Special Branch are selected ordinary police.
They have the same rtanks, pay and conditions as other police. Think of them as a select form of CID.
MI5 & MI6 are INTELLIGENCE departments - civil service....
BUT
@125 - true and SCARY!

@ 120/121 NOTHING NEW HERE - the Icelanders owe us money - if they don't pay up, we seize their assets. It's analagous to sending the Brokers' men in ...

Kragen- it sounds like we have similar concerns, but have there been many examples of the British intelligence services writing a truthful and publicly discussed capex proposal?

Almost completely off topic: I've just discovered I live about 10 miles from the town of Wankers Corners. Was Honecker buried there?

@54:

" The problem with all that data, though, is not designing the algorithms to pop up suspicious activity, it's finding the resources to deal with all the various types of suspicious activity once you've seen it. "

- Easy, just outsource it to private corporations. The media industry is just drooling over the possibility of using wiretapping to hunt down copyright violations, and I'm sure even tax evasion detection and other such things could be privatized, if the corporations get a small cut of each tax evasion that they detect... Harness the private sector to do the work of the public sector, or fascism, in other words.

@ 147: The history of "tax farming" in pre-Revolutionary France and elsewhere offers some cautionary notes to prospective private contractors.

Also, the excesses of the ancien regieme may have earned a lot of perjorative descriptions, but "fascism" is seldom used as one of them.

Charlie remarked: the problem is not the ID card, but the National Identity Register the government is building (using it as cover).

Permit me to demur.

Tens of thousands of Americans are currently unpersons, and the number grows daily. If you're not an unperson now, you will be eventually. As the requirements put on people for obtaining valid ID grow increasingly more stringent, fewer and fewer people will be able to comply. It's easy. It can happen to anyone. One day in the near future, it will happen to you.

Wilkommen zu die Reichsecuritaetsystem, Mein Herr, gib mir Ihre Papiere bitte, schnell!

Totally off-topic for this or any other thread currently up, but a portion of the Laundry headquarters is up for auction here: http://news.bbc.co.uk/2/hi/in_pictures/7672341.stm

@ 54

That's precisely what the NSA is doing right now in the US. All internet intercepts are done by private companies under contract to the telecom they're using for intercept, and the software that does the filtering apparently was written by the contractors as well. And most of the voice tapping has been outsourced to private contractors as well. The interesting thing is that not all of the contractors are US companies employing US citizens. That strikes me as downright weird from a security point of view.

Andy @ 142 wrote:
> All SMS messages are stored as well. I worked briefly
> at a telecomms company last year. I know this data is
> stored becuase I was in charge of the database storing
> it!

I wonder what they make of my routine pager test messages
then? Usually one word, "Aardvark", "Wombat", etc., but
I did get asked by the operator once what it meant, and
told them it was a test message. The following test
consisted entirely of five letter groups, just to be awkward.

3:O)>

Cadbury.

unperson @149, if you're Keith, you usually do a much better job of verifying info. You can get a PA birth certificate online. I recently got a copy of mine from WA that way. You need to know your SSN, but surely that man had something with the number -- like maybe his taxes?

Here's what's required to get a VA driver's license. (Warning, pdf)

76: Now imagine the BNP got a bill through parliament ordering the repatriation of people based on racial group [...]

Speaking of tracking ethnic groups, our current PM is aware that his party does not have best image with certain minority groups (well, most minority groups) so the PMO tried some outreach:

http://www.canada.com/ottawacitizen/news/story.html?id=198690d9-d9b8-4bbc-983f-d7236a2dfc8e&p=1

115: re how the system works. See the US, where recordings of phone calls which had no intel value were passed around, because they had salacious value.

123: Cost of surveillance. Is the tail open? If so, you need to have 6 people x 4 shifts (and 6 hours is a long chunk of time). They need to have vehicles, and a knowledge of the area (you need two per car, in case some subterfuge is entertained in the interest of losing the tail. Someone who jumps from a car at a tube station entrance has to be followed, and you can't just abandon the car in traffic).

For a secret tail... you have to have more people (and ways to communicate, as the subject moves, so that handoffs can be made and the subject kept in sight.) A moving tail needs not less than four people, and six is better.

Working as a single tail, on three people moving together, I've never managed to keep the subjects in sight for more than about three miles (in SF) when they were trying to keep an eye out for a tail. A single subject, who knows that s/he might be subject to a tail can "go dark" for at least a few moments, even when skilled agents are following him.

Can we keep an eye on someone well enough to stop them from doing something? Maybe. Can we keep them from dead-dropping info to their fellows? No.

As for false positives... de Menezes.

Marilee: yeah, I got a certified copy of my PA birth certificate via an online form, but I had to send them a jpg of my driver's license.