You have a markup issue in the third-to-last paragraph which has swallowed up some of the text into the link tag.

Ta. Fixed.

...Scotland has already implemented a national health statistics database, it just hasn't thrown it open to the world...

http://www.isdscotland.org/About-ISD/confidentiality/

Yes. Everybody seems to be implementing health statistics databases. As I said, they're generally a good thing for medical research.

Putting the raw, non-anonymized data on Google and letting anyone who wants access have it is not a good idea. And that's what HSCIC has inadvertently done.

This site appears to have some good information and references about the care.info programme : http://www.care-data.info/

"[...] they're generally a good thing for medical research."

It's actually much more than that: They are about the only reason why we can attack non-infectious illnesses effectively.

The Nordic countries have been trailblazers in this:

They introduced personal ID numbers back in the 1960ies to make income tax possible to automate and made them available as index in all governmental record keeping, including the free healthcare systems.

That has made a very large number of post-facto statistical surveys possible, which have uncovered a very large fraction of our sum of knowledge about cancerous activities in the workplace and general environment.

The most recent such study is only a few days old and showed a link between Tylenol/Paracetamol during pregnancy and offspring with increased incidence of ADHD.

However, there is a very large difference between the access rules for researchers in nordic countries, and just selling the data to anybody with a wad of money like UK did.

But I guess that if your government promises you more service for the same tax-revenue (or same service for less tax-revenue) it merely means that you are no longer a "subject" protected by your government, you are an income opportunity to it.

May I niggle off-topic? "Identity theft" used to be "impersonation", which I prefer. (Banks are on the hook if someone impersonates you, but will try to argue "you had your ID stolen, it's your fault" now that identity theft is a popular phrase. Anderson, Ross :"Security Engineering" 2nd Ed, chapter 10) It may be too late: even I find myself occasionally saying "identity thief" (:-()

The term "subject" is a bit of a red-flag when it's used by foreigners discussing the UK, because it relies on a bit of false mythology -- that we aren't citizens, but subjects. Usually it's symptomatic of right-wing nut cases or libertarians. Please mind your terminology, okay?

This doesn't help, but companies are pretty good at figuring out if you're pregnant even without your medical records.

There used to be an old joke about how to become a successful epideminologist in the US: you befriend someone in Scandinavia.

That said, there have been quite bad problems here too on how to handle sensitive data, and something similar could definitely happen here as well. Most recently the Lexbase scandal. Swedish court doucments are, by default, public, though you have to contact the relevant court and pay a small administrative fee to get copies the documents as a third party.

The people behind the site had systematically requested documents from the lower courts, then scanned, OCRed and cross-indexed (including geographical info) the documents. Including every type of court case - criminal acts, traffic violations, parenting in divorce cases, and so on. They also failed to properly secure their server so immediately was the target of data theft themselves.

It got so bad that Bahnhof (the select ISP for Wikileaks) shut them down.

Charles, No mythology intended, and no libertarien sympathies whatsoever from this dane. Apologies if my danglish terminology confuses the message.

Returning to the topic at hand, it makes some sense to have the data available under stringent controls to small, trustworthy groups of researchers. It's hard to anonymize large data-sets, and de-anonymizing them is frighteningly easy.

Using Google BigQuery would be good too, if Google offered a high-confidentiality service with "not to be exported from the UK" guarantees.

I wonder if this embarassment and some strong hints from the UK and EU might motivate Google to offer high-confidentality services in particular countries. I'd love to have had such a guarantee for a party-political effort in Canada last year!

--dave

"might motivate Google to offer high-confidentality services in particular countries."

Could and would you trust them if they did ?

Not a problem -- I know you're not a native speaker. Just thought I should mention it, because it's one of those dog-whistle terms that isn't obvious to outsiders.

Uh, "subjects" is a term that I've never used as best I can recall. But it is common usage on the various media over here in the US. Left, right, center, whatever. First time I've heard that people in the UK might take offense.

Here's a few interesting links...

The original blog post that the Tory MP quoted.

YouTube video from PA Consulting on how, "with the permission of the NHS, PA was able to upload anonymous patient activity approaching a billion units of data."

And here is PA winning an award for their work with the data last year.

To quote from the report where PA admitted this is what they had done:

" ...upload it to the cloud using tools such as Google Storage and use BigQuery to extract data from it. As PA has an existing relationship with Google, we pursued this route (with appropriate approval). This showed that it is possible to get even sensitive data in the cloud and apply proper safeguards."

So they are claiming that they (a) had permission to do this and (b) steps were taken to secure it. It may also be that PA via partnership with Google have access to a more secure version of BigData.

Ok, I know that Scots are citizens, because the Treaty of Arbroath (1320) is still in force. I'm less convinced that the English aren't subjects, and maybe this isn't the place, but this sort of thing is kind of important right about now.

The English are citizens, not subjects. Fixed in the British Nationality Act (1981).

Is there, at this late date, the horse having bolted, any way of getting it back into the stable & all the associated droppings swept up? I mean; all that data, including, presumably mine - I have hospital records, as well as my GP ones. It is stored, somehwere. Can it now be retrieved to secure servers, where only rally authorised persons have access, & the open records wiped?

Or is it already far, far too late? Whichever way it is, the scandal is going to be immense amongst those in the know & who fully appreciate the scale of the problem(s) involved. I assume, of course that guvmint/civil service will make empty reassuring noises about "Nothing to see / pass alaon now / etc" & similar lying bullshit, naturally.

It's probably much too late.

The servers it was uploaded to are probably backed up somewhere -- and cleaning the data off those backup tapes will be next to impossible; we can't even tell which continent they're on. The consultancy that uploaded the data had it on DVD, and almost certainly on multiple hard drives on different machines which will also have been backed up by their IT services, which may well have been outsourced. The agency that gave them the data obviously had multiple copies, too. If the uploads went out via an ISP service with some sort of transparent proxy/cache, there may be copies there, too. If the data was exposed on the public web it will have been spidered multiple times.

I give it a month or two before the first micro-targeted medical insurance spam starts appearing. It's the obvious market.

Well, I remember sitting there, listening to the debate about it and all the comments "It's really important we get this data" (yes, agreed) and "it's only through criminal activity that it will be deanonymised" so lets see how keen they are to prosecute.

I mean, I'm prepared to believe it's a cock-up when you leave your laptop on the tube, but uploading data over a period of weeks: surely somebody, somewhere in the process must have thought is this all completely legal and known better?

And while there's nothing that's going to get me particularly targeted, I am one of those affected - I've been a far too regular visitor to hospital over the last few months and years in a part of the UK covered by this.

GP data is presumably still NOT covered - that was due to be integrated soon but hasn't started yet.

I think it depends on the person. It wouldn't bother me but it's like a lot of these terms - they're a trigger to the right, or wrong, subset of people.

And without intending any offence to you, I don't know you but I'd quite possibly take offence if you (I'm guessing you're male since you're called David) called my partner or I dykes however nice you are. But I tease her and call her a butch dyke when she's acting that way or I just want to wind her up a bit.

If we became friends though, it can become OK as an occasional thing when we're acting particularly stereotypically lesbian which we do, and we have a male friend that calls us on it without offending either of us.

Ah, the glories of language and trigger terms.

Wow. A few years back while working for a web development shop, I wrote a training system for nurses for the DCU school of nursing. Nothing spectacular, and it was a small project, but one of the very first things discussed at the first meeting was putting together an anonymised dataset for development and subsequently building on it for training and their overriding concerns over privacy and anonymity. This wasn't some newfangled idea, this was a well-known fundamental issue and that was almost a decade ago and they were doing research even then into the automatic generation of anonymised datasets either from scratch or from heavily redacted records (and they weren't in favour of the latter from what I recall).

WTH were these people playing at? And how gross does it have to be before it stops being classed as "gross incompetence" and starts being classed as "now you're just taking the mick"?

"The servers it was uploaded to are probably backed up somewhere [..."

Not to mention the fact that the upload happened while NSA was tapping into Googles inter-datacenter fibres...

There is no way to prove that no (more) copies exists of data, so any attempted "success-criteria" for "repatriating" the data is balderdash.

"The term "subject" is a bit of a red-flag when it's used by foreigners discussing the UK,"

Thank you!

I'd seen it in occasional science fiction novels or spy stories, but didn't take it seriously. Whatever the terminology, it looks like british civil rights are a lot like those in the USA with differences in detail. The main differences I'd heard about involved free speech issues, the claim was that in Britain you could lose slander cases for saying true things, and they were more strict than the USA used to be about government secrets. Advertisers are supposed to be limited to saying true things, which leads to US claims that various US ads are illegal there.

Also the british police used to be famous in the USA for not usually carrying visible firearms. It used to be considered an endearing oddity, but I haven't seen mention of that in the media for a long time now. I just looked it up and it's still happening. I guess the US media just doesn't want Americans to imagine it could be a plausible choice. I don't think it's ultimately important whether police carry firearms all the time, but I approve of Britain showing that they can get by without them.

I had believed about the "subjects" thing, didn't consider it import and never mentioned it, and didn't know it could be a hot-button. Thank you for the warning. For once I learn about an issue before I step into it myself!

Also the british police used to be famous in the USA for not usually carrying visible firearms ... and it's still happening.

Like these guys?

(All police on patrol in the UK have personal radios, and armed response units are never far behind the first responders, if they're needed. But only trained specialists carry guns. Guns aren't generally carried by first responders unless responding to an armed incident or terrorism alert ... but when they need guns? Got guns.)

Or unless they're walking around Stansted Airport putting the wind up students from other unarmed-police states.

(There's something about seeing a pair of cops - in almost the same uniform as the police you're used to - carrying submachine guns while wandering around a busy airport that makes one feel deeply unsafe...)

Hi Mark,

In my own field of conservation, I've seen a proliferation of obsolete science trotted out as the Next Big Thing because someone wants to get rich off of it. This includes planting tree farms in parks as a carbon sequestration scheme (only geometrically simple trees and plantations are allowed, so that, you know, they can easily calculate the volume of wood and stuff), vegetation mapping has thrown out 20 years of progress in favor of schemes that were discarded in the 1980s as inadequate, and so on.

Because of this, I'm tempted to say that non-specialists pretty routinely makes these kind of bad decisions, especially when money is involved. Unfortunately, too often it seems that the only way to get them to stop is to spank the errant practitioners so hard that the shockwave traveling up their bodies leaves a handprint on the inside of their skulls (in a metaphorical sense, of course).

It being the truth is still a defence in a defamation case. You cannot reduce someone's reputation by telling the truth, because it is true.

Government secrets is interesting... there are various ways that the information can be put out but there are also things called D-notices that prevent publication. I don't know what the situation is in the the US but I guess there's something roughly equivalent.

The advertising standards require adverts to be "legal, decent, honest and truthful" so yes, if there are adverts in the US that tell outright lies, you can't get away with it. There are, however, ways and means - ads from Apple for iPhones and iPads are the same for example. They just say something like "the process for downloading apps has been shortened." The standards also require, in the other direction, when a statistic is quoted (79% of respondents approved of this product!) to quote the raw data as well.

And yes, the vast majority of UK police don't carry firearms. North Yorkshire, the county where I live, has 12 firearms officers total. Between them they do a 24/7 coverage pattern and cover about 3,500 square miles and a population of about 1.2M. Describing them as unarmed is incorrect though - they all carry some form of baton and some form of pepper spray or similar on day-to-day deployment. They wear body armour. They're trained with riot armour and the like too and it's easy to upgrade. My local football team doesn't deserve the full riot armour turnout, but when I lived in Liverpool you'd see it regularly. You also have to bear in mind that gun ownership over here is much lower of course. And gun crime. It's still true many farmers own a shogun, but they don't generally shot each other or the police. But it's still major news if there's a gun crime, or the police shoot someone. Google the Duggan Shooting for example.

We're very different cultures in many ways. Despite the Hunt-effect on the NHS, most of us still like a Nationalised tax-funded health service over here for example. We look at the US response to Obamacare and wonder wtf is going on.

For what it's worth, US HIPPA data in the Cloud has been in some cases made legal and compliant. The details published here are sorely lacking to try and figure out if this case would be.

If not... HIPPA does not specify US based persons or healthcare records. So this data DOES fall under HIPPA as I understand it. Which, will hurt.

So, 125 GB of data. Uncompressed and indexed, with no particular attention to storage requirements, this could blow up to maybe a terabyte. That used to be really big, but these days a stock relational DB (say, Postgres) running on decent server hardware will handle the load just fine. Spring for a server that holds the full terabyte in RAM, and you're still around $10,000. The sysadmin who runs the thing probably costs you more, and that may have been the issue --- but even so, that's not much of an excuse.

There are things for which Google BigQuery really does offer capabilities that are hard to get locally: mostly, good query performance on multi-petabyte datastores (which is what Google built the thing for internally, before commercializing it).

I'm dimly reminded of the CS professor I saw grousing on twitter about B-school students blindly following "big data" trends into ludicrous overkill: "To build our classifier we used a Hadoop cluster" [...] "we used 1043 tweets as training data". This isn't quite that bad, but it's very, very close.

I agree this whole situation is bad from a transparency point of view. But I do also think you're being disingenuous here. I think you've swapped out some realities for hyperbole.

125Gb isn't a lot of data, no matter whether it's on 27 DVDs or not*. It definitely isn't complete patient records of everyone in England. And how did it take them 2 weeks to upload it? PA Consulting are paying Google for the computing power required. It's safe to assume they're not going to be on a dial-up connection.

And Google don't have access to this data, no matter how many scare quotes Twitter users apply to the news. PA Consulting are using a storage service and a data query engine that Google provide. Google don't have legal access to that data. The only way it is likely to get out is if PA give it away.

I think it's dangerous to ascribe hyperbolic motives (and apocalyptic outcomes) to this. What it looks like is a test, using a representative sample, of whether patient data can be usefully queried. It looks like an attempt to win future tenders by demonstrating they can do this. It's reasonable to use Google's computing power**, and to do that, they need to use data centres that are not UK-based.

I think the whole situation has been badly handled. I'm not happy about data going to PA Consulting, or the mishandling of the whole IT project. But I also think that the story plays into an ignorance of what the processing of patient records will involve. And it's also an excuse to trot out the 'Google is the Devil Incarnate' lines.

Sorry if I went on a bit.

• That amount of data would fit on a single memory card the size of your thumbnail, which doesn't quite sound as much, but anyway. ** What alternatives are there? Amazon would be the obvious option, but the nearest major data centre is in Ireland, so it's still offshore.

The term "subject" is a bit of a red-flag when it's used by foreigners discussing the UK, because it relies on a bit of false mythology -- that we aren't citizens, but subjects.

This was one of the surprises for me when I spent a few months doing the whole cross-country road trip thang in the US back in the mid-90's. A hefty minority of the folk I encountered seemed to honestly believe that the UK wasn't a democracy.

It was... odd...

(That and being told I was "brave" for telling folk I was an atheist when asked about my religion ;-)

And Google don't have access to this data, no matter how many scare quotes Twitter users apply to the news. PA Consulting are using a storage service and a data query engine that Google provide. Google don't have legal access to that data. The only way it is likely to get out is if PA give it away.

Google may not have the legal right to access this data, but they certainly have the technical capability. And the legal questions are murkier than one might expect, starting with the nasty question of whose law applies to which Google personnel. (And which branch of it. The American secret FISA court has issued all sorts of orders for turning over bulk data which U.S. government agencies would not have access to by more ordinary means. And while that court pays token attention to the privacy rights of U.S. citizens, it treats foreigners and their data as, effectively, fair game.)

Ben Goldacre is saying that there's something "infinitely worse" about to break. https://twitter.com/bengoldacre/statuses/440466277673750528.

Comment seen on BoingBoing:

Every trans* person who uses the NHS in England and Wales has now effectively been outed.

"It's reasonable to use Google's computing power"

No it's not. I work in the medical research field (Admin side), and it is not at all reasonable to transfer confidential patient data to servers controlled by an uninvolved third party. It's criminally unreasonable.

Ben Goldacre's "infinitely worse" fizzled. Thankfully. See the update at the end of the original blog entry for details.

Links to statements from NHS and PA Consulting here: https://news.ycombinator.com/item?id=7335319

Notably, they claim that "the data set does not contain information linked to specific individuals". I read that as saying that names (and, hopefully, addresses, phone numbers, and other such identifiers) were scrubbed out before PA Consulting even saw the data.

If that's accurate, it changes the picture somewhat. There's still cause for concern --- databases like this can often be "de-anonymized" by correlating what's in them with other data sets that do have identifiers in them. (As I once heard asked about a different data scrubbed data set: "If you know you've got a 34-year-old male Japanese living in Lincoln, Nebraska, how much more do you need?")

The NHS statement is ... weird. It at once acknowledges that the NHS had prior notice that the data was going to Google, and claims that PA Consulting had given assurances that "no Google staff would be able to access the data". It might well be the case that no Google staff would be authorized to access the data ... or at least, not by PA Consulting. But there are certainly people at Google who would be able to.

And, to reiterate my comments from above, this is not a data set that is so large that "big data" tools like BigQuery are needed to handle it. A perfectly ordinary database on perfectly ordinary server hardware can handle a terabyte of data just fine, and by all reports, this data set wasn't even that large.

Out of curiosity, where do you get the computing power to rapidly query enormous data sets at a reasonable price? I'm assuming you're completely against 3rd party involvement. So lets say you run Hadoop, and you build the computing infrastructure necessary to pump data in and parse quickly it in a useful manner. How do you secure this? More pertinently, how do you secure this better than Google can? I'm not being facetious here, it's a reasonable assumption that a self-engineered solution is likely to be both much less useful and much less secure.

To defend the US Americans somewhat, I was also under the appreciation that British law on this issue was somewhat, err, strange, though given the influence the quackery guys have in German legislation, I'm not that certain if e.g. Goldacre et al. would have been less at risk in Germany and the main difference for the UK is at least somebody speaking up against the likes of Rath et al.:

http://www.badscience.net/2009/12/libel-reform/

Note aside, Rath was head of a party in Germany some time ago, I guess there'd be some discussion who bought the cake regarding surrealism in ads, those guys or some trotzkists.

We have a server room, and we don't run Hadoop, because we don't need it. Why would we need a system for managing a distributed computing system when we can buy our own servers? Cdodgson's comments about the misuse of Big Data tools applies here. They're not really needed for this sort of research, regardless of what PA Consulting seems to think.

Security is easy: We don't put confidential data on our email server or on any server accessible by anyone other than ourselves and our client (The government). If anyone screws up on that, they're fired, they've destroyed their professional reputation in the field, and they face possible fines and prison time.

At the PA side, they probably could destroy all the copies of the data they have if they set their minds to it. They do the sort of commercially confidential work that means they ought to have internal access controls and backups directly under their control, and 125 GB isn't the sort of thing that gets left on someone's desktop by mistake.

Of course that would require a fairly serious audit to find all copies of the data and wiping (or destroying) them, meaning a willingness to spend time and money on the problem. To get that to happen probably requires the prospect of criminal sanctions.

One person who was definitely opting out of the GP record uploads lives out in the country somewhere. He was interviewed on Radio 4. He and his wife are the only people that live in their post code. Post code and gender are going to be preserved. You can 100% identify him, or his wife of course.

According to him this affects some 10's of thousands of people which at first glance seemed high, but even if it's 50,000 people that's less than 0.1% of the population. Although it's no longer preserved that way, originally it was loosely "the postman parks his (it's old enough it was his) bike/van and walks to do the deliveries from here, then goes back to the bike/van" is a single postcode. That's why it's often one shortish street or part of a longer one in a city. So 10,000 or so living on a farm or similar too far from the next one that the postie can't walk there but must drive/walk? I can believe that.

My details get you to about 70 houses, many of which are divided into flats (or apartments) and so about 150-200 people. You can probably get to a few people that match me because quite a few of the 150-200 are students or young graduates. But I'm not paranoid to opt out. I could be targeted, yes, but I'll take the chance.

whilst not infinitely worse, it is getting worse. the ESRC appear to be sponsoring a mash up of all official data. via Martyn Thomas and fipr -

Vacancy Description

The Administrative Data Research Network (ADRN) is an exciting new national data initiative. Its purpose is to stimulate opportunities for innovative research and policy making, by facilitating the use and linkage of administrative data (i.e. data routinely collected by government departments, agencies and other statutory bodies). The ADRN, which is funded by the Economic and Social Research Council (ESRC) comprises four Research Centres (one in each country of the UK) and an Administrative Data Service liaising between the Centres, data owners and researchers. The UK Statistics Authority will be the reporting body to the UK Parliament for the ADRN. The ADRN Board will provide assurance to Parliament and the public about the achievement of the Network's core purpose. The ESRC is accountable for the infrastructure and management of the Network. We are seeking to appoint a minimum of three non-executive members to the ADRN Board. Candidates for non-executive roles should have one or more of the following specialist skills: . Relevant social science and/or health research expertise. . Relevant data security expertise. . Experience of managing or linking data. . Research ethics, governance and legal expertise. . Management of major scientific infrastructure or resources. . International expertise in data infrastructure.

We are also seeking to appoint at least one lay member to the Board. The role of a lay member is to ensure that the interests of the public, who provide data to government, are properly considered by the Board. Lay members are not expected to have particular specialist knowledge of the use of administrative data, but will need to be able to pick up issues, follow a technical discussion, and ensure that the public interest is at the heart of the discussions and decisions of the ADRN Board.

See candidate pack at http://www.statisticsauthority.gov.uk/appointments/

happy days...

Re: "And what they uploaded was the entire shooting match—full personal medical records indexed by NHS patient number—with enough additional data (post code, address, date of birth, gender) to make de-anonymizing the records trivial." -

A question from a non-techie:

At what point does encryption occur during data file upload onto servers/cloud? That is: are the data files that were uploaded to Google directly readable/usable, or would a user need some sort of magic wand or phrase to obtain access?

There's something about seeing a pair of cops - in almost the same uniform as the police you're used to - carrying submachine guns while wandering around a busy airport that makes one feel deeply unsafe...

I suppose it's about what you're used to.

I spent some of my childhood as an Army brat in Northern Ireland (where all police were routinely armed); seeing soldiers on the streets was not unusual. I spent more of it in Germany; seeing armed policemen (and armed customs officers, and passport control) was not unusual.

:) Winner of the stupid question of the decade went to the 19-year-old member of the Scottish shooting team who, when standing chatting to the armed cops at Heathrow (having just checked his own rifle onto the flight) unthinkingly asked of their MP5 "so - is that thing real?" :)

"Guns aren't generally carried by first responders unless responding to an armed incident or terrorism alert ... but when they need guns? Got guns."

Of course. They are backed up by the full power of the government. If it came down to a big enough necessity they could get tanks, flamethrowers, airstrikes, whatever.

To me it says something that the first guy who knocks on your door is not armed with any more than a truncheon, retch gas and taser. If it comes to a physical fight between you and the government you cannot win. You might do a little damage but if you choose to fight them then you are fey, and your life is basicly over.

I like to see a government that's willing to assume I'm rational and that I will submit to the legal system. I think that behavior creates more compliance. I doubt more police officers actually get killed because of it, and if things turn pear-shaped it won't make much difference in the ending.

It's only a matter of style and I approve of the British style in this.

I like to see a government that's willing to assume I'm rational and that I will submit to the legal system.

My brother in law is a retired cop from small town Oregon.

The basic issue as he saw it is that while most "normal" citizens are rational, the ones they deal with most of the time (he called them frequent flyers) tend to be no where near rational. Guns or not they tend to be way more stupid and of a incredibly inflated ego compared to most of the population. Since that's who they deal with most of the time they tend to assume the worst when dealing with situations.

Two comments that stick out that he's made over the years.

Domestic disturbance calls are the worst. Because you never know if the aggressor has a gun nearby and also the person getting the crap beat out of them will many times turn on the cops when they interfere to stop them from being beat to a pulp.

The other point was that most criminals are just plain stupid. He said 90% of the time when they go looking for someone in particular who knows the cops are looking for them they find them either at their spouse/girlfriends or at their mama's.

I'm afraid that, re. the police, on the police blogs before they were shut up/ leant on/ retired/ gave up in disgust, it was made clear that the only chance you have of an armed response squad being nearby is if you live in parts of London or near other potential targets/ police HQ. Otherwise, just go and shoot someone in a small town or the countryside and see how long it takes before armed police arrive - it'll probably be half an hour or more, long enough to do plenty of damage.

And the difference in the UK and USA is probably that more people in the USA see the cops as enemies, than in the UK. Of course they are for cetian minority groups and whoever else is the target of the time (e.g. miners, climate protesters), but the lack of guns and a less confrontational style means you are much less likely to get shot by a policeman in the UK. I'd like to know the continental statistics.

Right, onto the fuck up.
Who are these people and how the fuck did they get all your data?

Our recent work includes:

There's a link to their board as well. Basically it's a bunch of rich fuckers who do consulting/ ran banks/ venture capital stuff, with connections right through the money markets and internationally. I'm sure they are making someone a lot of money, but like many such businesses, it won't make its way down to the people whose lives they are meddling in.

Oh look, PA consulting have form:

http://news.bbc.co.uk/1/hi/7608155.stm

A company which lost the details of thousands of criminals held on a computer memory stick has had its £1.5m contract terminated after an inquiry.

But as we know, there's so few companies willing to do what the government wants that they always get another bite at the cherry.

At what point does encryption occur during data file upload onto servers/cloud?

Depends on the service. All of the ones I'm familiar with will encrypt over the wire (i.e., with ssh or SSL/TLS). Some of them then encrypt the data on the server side -- but whether they encrypt is so they can't see it is what you're really asking. Some of them claim to do so; I know that Dropbox doesn't.

If you can get at it without having to enter a password (e.g., sharing data), then it's not going to be encrypted on the server end. Unfortunately, requiring an account password doesn't tell you what it's like on the remote end.

I am playing devils advocate, because I think it unlikely that any consulting company on the planet has patient interests front and centre, and I do agree with you to an extent.

It's just that your answer seems to assume that PA Consulting aren't acting under the same strictures that you are. Yet they have specified to the NHS what they're going to do (and not going to do) with the data. And I don't see why those assurances shouldn't be true: PA Consulting either want to win a tender from the NHS, or the NHS has hired them (as they did in 2006) specifically to run tests using BigQuery to see if there is an advantage to using it.

There's also this idea running through the coverage that because it's Google, the data is unsafe. And again, to an extent, it is: someone could accidentally release it (which would likely be someone from PA, not Google). Or it could sold, or stolen. Just like you could sell or steal the data you deal with - if it's sufficiently valuable, and you can extract that value very quickly, why not? Obviously you wouldn't, and neither, in all likelihood, would anyone else.

Yes, but surely the aim is to find methods of dealing with multi-petabyte data?

El#30 - No, the US doesn't have the equivalent of a D-notice, though the government sure wishes they did. We still maintain a relatively strong pretense of a free press over here, and restrictions on revealing classified information apply to the people who have access to that information, not to the press. So when Vice President Cheney's buddy Scooter Libby leaked the fact that "diplomat" Valerie Plame Wilson was actually a CIA spy (in revenge for her husband revealing that the government knew that the "yellow cake uranium" pretext for the Iraq War was bogus), it wasn't illegal for the newspaper he leaked it to to publish it, just for Libby to have leaked it.

No, the obvious market isn't micro-targeted medical insurance spam: it's blackmail. The profit margins are higher.

And the data is tagged for identification by condition. Abortion - anything gynaecological at all - can be used to end a woman's political career. Even minor activism, advocacy, civic and local constituency work will do for a campaign of shaming and ridicule that will drive her back indoors.

Don't bother saying that they have more courage: some are, some aren't, and most are not in a sufficiently secure position to risk it.

Do it to enough women, and every female worker who has ever had an embarrassing condition treated in a hospital- or a friend, or a daughter who did - is a blackmail target who can be used against her coworkers and acquaintances.

It worked for the Stasi.

It'll work for the nice young men who man the phones for the Republicans at fundraising time, and do their bit to downvote everything on Reddit that suggests a criticism of neoliberal America.

Some of these nice young men have worked as subcontractors for the NSA. Hundreds, perhaps thousands of them. And, just like Edward Snowden, they know that what they've done with the data is right.

Unlike Snowden, they haven't told the media where they took your data. It was laughably easy for him, it was and still is just as easy for them.

And if you're bridling at my use of 'right', try this: "God's work it is, that those who have committed the sin of abortion shall be known". If that's the kind of god you serve, and the kind of nice young man you are, that's righteous and self-evidently true and virtuous. Worth going to prison for, although I doubt that whoever has done it already will ever be identified and prosecuted.

Next: mental illness and addictions, and now we're into things that can discredit men. It's not as hard on men - we triumph in our courageous struggle with depression, women are 'emotionally unstable' and the press will never let them shake off the label and the sigma of 'mentally illness' - but it's damaging enough to men to be worth doing.

Worth paying quite a bit to keep quiet, if you have a high-achieving career in a bank or a consultancy, let alone a public-facing kind of job.

Anything else? Something you wouldn'it like the world to know about your children?

The data is out there, waiting to be used. Or rather: 'in there', in the hands of organisations known to be evil, willing to engage in covert campaigns to discredit dissidents, and appallingly slipshod with their security.

The only good thing with the nationalisation of mass data theft - and its private-sector redistribution among the commercial partners of our governments - is that the profit margins for entrepreneurial medical and sexual blackmail will be driven down so far that micro-targeted medical insurance spam might just end up as the most profitable use of stolen NHS data.

Eventually.

The only good thing with the nationalisation of mass data theft - and its private-sector redistribution among the commercial partners of our governments - is that the profit margins for entrepreneurial medical and sexual blackmail will be driven down so far that micro-targeted medical insurance spam might just end up as the most profitable use of stolen NHS data.

Supply and demand works differently for toothpaste vs blackmail. For toothpaste, if I buy any one tube I have a tube of toothpaste. For blackmail, I have to pay off all the blackmailers to keep my secret, not just one. So the natural outcome is that many blackmailers demand a fee and then I have no choice but to let the secret leak because I can't afford to pay them all.

For each individual blackmailer, it's a tragedy-of-the-commons situation. The blackmailers don't have a relationship with each other so they can't collude to get the total blackmail sum down to something I can afford.

It was mock data after all?

I suspect it wasn't either "gross incompetence", nor "taking the mick". There is a huge amount of money to be made from this, both leitimately, semi-legitmately & as outright crime (Blackmail, in fact). Now... The real question is, who was paid off, & will they ever be prosecuted?

Do not hold your breath whilst waiting, is my advice.

the claim was that in Britain you could lose slander cases for saying true things, This used sometimes to be true... However it got so bad, & also the so-called "Libel Tourism" that the law has changed - this January. Truth is now an absolute defence - all you have to do is prove it. Public Interest is also now a blanket defence, again subject to burden of proof.

Pathetic, isn't it? The idea of a "Constitutional Monarchy" ( Belgium, Netherlands, Denmark, Sweden, Norway as well as us, & that's just in Europe ....) doesn't seem to be able to percolate their brains. Maybe it's the bnullshit about the "Glorious Americam Revolution" agaist EVUL GeorgeIII, & never mentioning the incompetent actually in charge, Lord North, rather than Geo Washington & pals defeding their monied (slave-owning) interests against the coming of emancipation .....

It's mildly scarier in Glasgow Airport I think; remember that this is the place where the last terrorist attackers had to be taken into protective custody, before the citizenry got a hold of them!

There are other aspects of this, running in parallel to / crossing over with the "Rule34/Kafka"thread.

Hopefully, I'm not breaching protocol by cross-posting this little piece of breaking news from there:

^^^^^^^^^^^^^^^^^^^

In fact, you are not safe, even if you are one of the "Right People" ( And, in this case "right" has two meanings, just for fun) as the arrest of Patrick Rock shows. Now this guy is an aide to No 10, & he was publicly known to be working on the problem of child abuse imagery on the web. But he's been arrested - though not charged, as yet, & he may never be - it may turn out to be "no case to answer". But, the condemneation is in the public outing & process, isn't it?

Personal view: I SUSPECT that, although he was legitimately working on this horrible stuff, he probably hadn't necessarily ticked every last possible box, to allow him to work with such imagery & someone has decide to "obey the law". Which just shows how dangerous this sort of Strict Liabilty legislation can be.

Oops - Second thoughts. This guy, (PR) has been around since the days of the madwoman. It's also distinctly possible that he treated the cops with whom he must have been working as , ahem, "plebs", and therefore that this is an internal stabbing by the police, to get their own back.

I've spoken with USians who seem unable to understand that "republic" and "representative democracy" are terms which describe different aspect of a national government.

Anonemouse, what you are feeling is actually mild unease. Once you've seen how cavalier and recklessly unsafe police officers can be with firearms on a shooting range, then you'll be feeling properly frightened...

No, the US doesn't have the equivalent of a D-notice, though the government sure wishes they did

Yes you do. A D-Notice is an advisory; legally unenforceable. It's "please don't," exactly what the NYT got, just more formal.

This is not to say the press in the US isn't freer - it undoubtedly is - but denying government influence over reportage isn't helpful.

Lots of conservative Americans are very clear that there is a difference between a republic and a representative democracy. They firmly maintain that the USA is NOT a democracy but is a republic instead. They say the difference is that democracy results in mob rule where the majority can eliminate everybody's rights, but a republic protects the rights of minorities by preventing the majority from getting away with that.

In some US states they learn this in the public schools. I'm not sure but I think the meme got started because the two significant US political parties are named Republican and Democratic.

My microbiology class just toured a local diagnostic lab and blood bank. We were told that a patient's survival may depend upon which type blood (out of dozens, beyond ABO) they received two decades before. The only answer is a universal database.

So we'll see how many people choose privacy over fear of biomolecular terrorists (that is, molecules gone wrong). You can make a strong case that privacy is more important than a chance of a bad blood reaction--until you're stuck in the hospital needing ten pints. Today's NYT column argues that privacy is already a luxury good.

Thanks for the explanation, Sean - much appreciated.

"If you can get at it without having to enter a password (e.g., sharing data), then it's not going to be encrypted on the server end." Hope this data is at least password protected.

More questions:

Is there a limit on the number of electronic identities any individual can assume/obtain at any given time? (It seems that part of the problem with tracking the bad guys is that they can exist for only milliseconds at a time.)

How long would someone need to be connected to this data to steal/download it off the system, that is, actually grab enough data to be a threat? My understanding is that 'data' behaves like a fluid in that you need to keep the 'fixed-size tap' open until you've downloaded the total volume of the data file; otherwise, the data's not useable.

I have to say I'm a bit pissed off at the implication that they put it on Google as in the public search engine or Blogger or something, which pretty much everyone who has commented on this has adopted. It is a fairly shameless way to hype what is a serious issue, and will cause a lot of people to stop caring as soon as they read far enough into the story to work out that, no, you can't literally google their medical records despite what the headline promised.

M'Lud, at this time the prosecution would like to introduce into evidence the Union of Soviet Socialist Republics.

Still think that a republic safeguards individual rights?

Lots of conservative Americans are very clear that there is a difference between a republic and a representative democracy. They firmly maintain that the USA is NOT a democracy but is a republic instead. They say the difference is that democracy results in mob rule where the majority can eliminate everybody's rights, but a republic protects the rights of minorities by preventing the majority from getting away with that.

Exactly.

And the system worked as designed until 1860.

(Because the minority whose rights it was designed to protect were the rich white slave-owners.)

What Sean meant was a password is necessary but not sufficient for encryption - if there's no password it's definitely not encrypted, but it having a password tells you nothing about its encryption status.

Sean mentioned SSL encrypting the connection over which the data was uploaded, which should prevent eavesdropping in transit. Unfortunately there's almost no way the data is encrypted "at rest" on Google's machines, if it's to be queryable - and if it's not to be queryable, why're they putting it into BigQuery?

"Fed into Google" is accurate; if it's been put in a Google service we have no way of ever removing it, and Google behaves as if all information is theirs by right. You could almost segment information into three domains; Google don't have it yet, Google have it but haven't publicly indexed it, and publicly indexed data you can Google. PA have moved these people's medical records from Domain 1 to Domain 2. What this means is unclear, but we have little to no reason to trust Google, especially as they believe in minimizing privacy rights at every opportunity.

Winner of the stupid question of the decade went to the 19-year-old member of the Scottish shooting team who, when standing chatting to the armed cops at Heathrow (having just checked his own rifle onto the flight) unthinkingly asked of their MP5 "so - is that thing real?" :)

I know it's off-topic, but I think the Irish shooting team can beat that. We showed up with twenty kids and their rifles in tow at Heathrow in the middle of a bomb scare allegedly called in by some IRA splinter group. Coach walked up to the security guard at the door. "Sorry sir, bomb scare from the Irish, you may have a delay". In as thick an Oirish, Darby-o-gill accent as he could manage (NOT an NI accent, there's funny and there's plain suicidal), and keeping a straight face, more impressively - "Ye think ye're having a bad day so far? D'y'see the bus behind me? We all need to check our guns through security for the flight home".

The poor security guard couldn't decide whether to blanch or cringe, so he managed to do both. We wound up having to escort our kids through security (child protection laws from Ireland require it) while five or six armed guards with MP5s escorted them (antiterrorism laws apparently requiring that 12-15 year olds be considered viable threats) with two more on the walkways overhead, all for an air rifle team of teenagers. And that's before the paperwork confusion (because the UK doesn't require olympic air rifles to have licences but Ireland does, and the lads in the airport just hadn't seen this before). I don't think we made friends. In fact, if the team hadn't been the junior team, I think there would have been more than a few cavity searches just to drive the point home, so to speak.

I swear, you think the daftest stories would come from the illegal use of firearms by nasty people, but the stories you find when good people trying to take part in one of the safest sports in the world while following all the really, really arcane laws meet untrained people charged with implementing those laws but not being given the training or resources needed to do so -- those are some of the funniest stories I know.

At least, they are now, with the benefit of a few years of distance from the events...

I have my own experience of that, at the county agricultural show, and no way am I going to get into an argument with cop about firearms safety.

I got my lessons from various elderly relatives and acquaintances whose speech mode when seeing safety idiocy was essentially paint-blistering drill sergeant.

I quietly went somewhere else.

"hosted on servers outside the UK (and possibly outside the EU, in breach of the EU Data Protection Directive)."

That's not actually true, Principle 8 of the DPA states "Personal data shall not be transferred to a country or territory outside the European Economic Area unless that country or territory ensures an adequate level of protection for the rights and freedoms of data subjects in relation to the processing of personal data."

This then means the following rules apply:

If you decide you need to transfer personal data outside the EEA, and the recipient is not in a country subject to a Commission ‘positive finding of adequacy’ nor signed up to the Safe Harbor Scheme, you will need to assess whether the proposed transfer will provide an adequate level of protection for the rights of the data subjects in connection with the transfer/processing of their personal data.

So there are still lots of questions to be answered.

"And the system worked as designed until 1860.

(Because the minority whose rights it was designed to protect were the rich white slave-owners.)"

Yes, and the system is still working to protect the rights of Rhode Island and Wyoming relative to California and New York.

We no longer protect slave states, but we do protect low-population states.

...has the US recipient of the data signed up to the US Department of Commerce Safe Harbor Scheme?

Apparently, Google complies with the US-EU Safe Harbor Framework, which might be the same thing.

I think the blood group argument may be hyperbole, or orthogonal. Events may well depend on what blood transfusion someone had some time ago, however they do not depend on the record of that, they depend on the measurable effect of it - development of antibodies principally.

And the lab measures that effect.

Blood type incompatibility is not, repeat NOT an argument for a universal database of anything at all. When a patient presents with a need for blood, you do not go looking on a universal database to see what blood type they are, and you do not trust the blood type bracelet or whatever they have with them. There is a simple reason for this.

People, basically, are stupid. People are stupid enough to wear a blood type bracelet which does not represent anything like their blood type; a medic trusting a bracelet has already led to anaphylaxis and near-death because of this.

Similarly, you do not rely on records of what blood type a patient is since if you do so you are relying on the first medic who encountered them performing the tests correctly, recording them correctly, and identifying the patient correctly. You are also relying on your own identification of the patient in front of you being the same one as is mentioned in the records, and you are relying on these records not having been falsified, altered or in some other way faked.

Are you feeling lucky? If so, go ahead and trust the ID records, after all it'll only cost you your career and possibly bankrupt you if you're wrong.

Everyone else will continue with the standard medical response to a patient in need of blood: blood type them immunologically immediately before the blood is given. If you do this, then you can be certain that the patient before you is this blood type, and you eliminate all the aforementioned uncertainties. You also eliminate the need for a universal register, but then you knew that already.

Greg@62: How dare you slander the Founding Fathers?!

It wasn't slavery/emancipation that motivated them; it was not paying the tab for redcoat services in the French & Indian War.

TL:DR version:- Never believe a statement that a patient is $blood-type. Always observe one that they are alergic to $substance, regardless of how odd it sounds.

"It wasn't slavery/emancipation that motivated them; it was not paying the tab for redcoat services in the French & Indian War."

It's usually hard to be certain about other people's motivation. I wondered about the slavery issue in 1776, it seems like the idea of forbidding slavery throughout the empire was just getting started then. It seems prescient.

Americans tend not to connect the French and Indian war, I've heard that only from Brits and Canadians. It makes some sense that the colonies should pay for that war 10 to 20 years later. After all, Kuwait should still be paying the USA for the Gulf war; if the USA hadn't fought they'd still be ruled by Saddam. But proto-Americans thought that they should have had the chance to argue it in Parliament and should have had some votes.

But for whatever reason, they did build in protections into their constitution for slave states and for small states. Also they built in protection for newspaper owners which unfortunately doesn't work as well for broadcast media, and they built in protection for state militias which later helped states revolt against the federal government. Some people argue that is protection for individuals who might want to revolt against the federal government, though the Iraq war shows us that they'd at least need IEDs and RPGs which are unfortunately fully illegal. Etc.

Greg@62: How dare you slander the Founding Fathers?!

It wasn't slavery/emancipation that motivated them; it was not paying the tab for redcoat services in the French & Indian War.

It wasn't slavery/emancipation per se that motivated them, because the British empire itself wasn't clearly anti-slavery until a few decades later. But taxation wasn't the only issue; there were also British constraints on westward expansion and local industrialization --- and the over-the-top enforcement measures once the conflict was underway. The Declaration of Independence has a pretty long bill of particulars which no one ever reads.

Known over here, of course as "The Seven Years War"

And, as c.dodgson says, there were other issues - the incompetence & greed of Lord North's "government" being large amongst them.

I'd love to see an analysis of which provides more personal usable data to bad guys: Facebook or an NHS-type data base? Seriously.

Shift of topics .... this just in ...

http://www.dailymail.co.uk/news/article-2573863/Bitcoin-exchange-CEO-dead-home-suspected-suicide-age-28.html

Please don't link to the Daily Heil if other sources are available, m'kay?

As to the content: oh dear ...

Apologies ...

Intuition sez it depends on the badness; while Facebook happily hands over way too much data to anyone, the NHS data is very bad news for already vulnerable people. (Are you Trans? Or a functioning opiates addict? Or...)

I also can't think of a reason to murder someone discoverable uniquely through Facebook data like I can NHS data. ("Find me all card carrying organ donors who are zero mismatch for my at-death's-door baby.")

It might of course be coincidental, due to factors other than her work. Huge numbers of people commit suicide all the time, and for that never to include anybody in the finance sector would be pretty damned unlikely.

(http://www.cnbc.com/id/101468694 for a link less likely to trigger the kitten filter, but there are plenty of others.)

Having said which, certain professions seem more prone to suicide than others. Farmers and doctors, IIRC, are particularly prone to it, though even chocolatiers can do it.

Sure, but (to a very crude first approximation) any data that FB can leak is data that I've chosen to put on FB. My elderly relatives' medical records, not so much.

I work in the NHS handling large amounts of patient data. Nowhere near as high up the food chain as HSCIC however. The systems I work on are those that make sure that the right patient gets the right drug, those drugs are paid for, and we get the drugs at the best possible price.

I'm in favour of care.data in general, but not the way they are rolling it out, specifically how they are handling the communication to the public and how they are dealing with anonymity.

While there are occasional accidental losses of patient data, those are generally just that, accidental. In fact, most of the losses report are only losses, no-one ever reports finding the data (well, they wouldn't would they!). Information governance and data protection is drummed into us here at every level, we are tested on it annually. All writeable CD/DVD drives are locked down, non-encrypted USB devices cannot be written to. Any email that contains identifiable patient data should only be sent within the encrypted NHS network etc... If anyone here had done what it appears HSCIC has done or plans to do with our data, they would be fined hundreds of thousands of pounds and probably imprisoned. To compound that, the (junior) Health Minister then stood up in parliament and "misled" parliament over what has been done (although she has subsequently admitted she was wrong, who noticed that?)

However, IT-wise we have to keep up with the times, and that means looking to the Cloud, which is why I was aware of all the provisos to Principle 8 of the DPA. We are looking to put data on the cloud, so there has been a lot of debate as to exactly what that means and whether it meets the requirements of the DPA.

So yes, Google is covered as a "Safe Harbour". Am I happy about NHS data being stored on Google servers? No.

I have virtually no usable data on Facebook (I still have an account but I don't use it, I think you can get my city from it but I don't want to log in to check) and you can get a random cross section of people I've met RL, people I've met online, and people I've met playing online annoying games from it. GL telling them apart.

My hospital records only? Well, my address, my pattern of migraines, my migraine medication, my blood type, my allergies, my major accident record, my lifetime history of operations (I've had a few), a couple of interesting things. You can see the long and sorry history of my knee problems. With some good reading between the lines you could piece together some hypotheses about my employment or social history (exactly WHY was I tested for HBV antibody status several times over 2 years then it stopped?). You can actually tell my weight and height, but if you don't read that far just by looking at the tests ordered you can guess I'm overweight. And I'm relatively lucky - I don't have anything in there that is blackmail material: no history of mental illness for example which shouldn't be but all too often is still regarded as shameful.

You want a horribly sick scenario? Suitable for a Criminal Minds type show? A serial rapist who hasn't been caught and likes revisiting his victims has lost touch with a few. They've been treated in hospital though. Get into the data, search for people treated for the right sort of injury in the right places on the right date and bingo... their personal hell restarts. FB might let the rapist do that, the hospital data if it gets you to a post code almost certainly will.

My medical records are now theoretically available to anyone who can pay enough to search them out. But I can't see them myself. Theoretically anyone can apply to see their own records but the record holders can censor them first. So the government has made money by selling my records but still doesn't allow me unrestricted access.

I was just being snide. Of course there were multiple causes -- and yes, Lord North's ministry distinguished itself for for inept crisis management.

My grandmother came from a New Brunswick line of Tory loyalists who'd left Massachusetts in 1783, which may predispose me to some skepticism about the US grade-school version. That includes seeing our theater of the Seven Years' War in a long pattern of

1) colonists' relentless westward expansion, often violating treaties, followed by

2) Unprovoked Attacks by Merciless Savages, prompting piteous appeals for more redcoats, followed by

3) resentment over the presence of redcoats, over taxation to cover their costs, and over more London meddling with our God-given divine right to further westward expansion.

Lather, rinse, repeat.

(And while in Canada, those Tories intermarried with Scots and Irish immigrants to Prince Edward Island -- so it's Bad Attitude all the way round that helix of the genome.)

Putting on my marketing hat, I see some opportunities for the general population alongside major risks for big pharma ...

Hospital and especially GP-level records would include all drugs (by brand name), surgical interventions, diagnostics, etc. Anyone with the right analytics tools/smarts could then mine these data for real honest-to-god evidence of just how effective current treatments actually are. (Malpractice law firms should start hiring math/computational analytics geeks now!) Sidebar ... Not too long ago, the NIH in conjunction with a few huge pharmacos announced that they were releasing complete clinical trials data for a few selected disease groups so that academic and other researchers could study these data. This is being done in the hopes that crowd-sourcing might come up with new insights. Comparing real-life GP-level versus 'clean' clinical trials data would also identify weaknesses/strengths in current clinical trials practices.

As for the current public bias re: mental illness/conditions: Perhaps finding out what the actual prevalence/incidence of said conditions is might actually reduce efforts at black mail. I believe that the over-a-lifetime-incidence of any 'psychiatric condition' (this includes depression, ADHD, age-related cognitive decline/dementias, etc.) is at least 1-in-6, and is likely to grow. On a per household/family basis, this is probably close to 1-in-2 which translates into pretty well everybody you know has some direct personal experience or "something to hide". This type of information could actually be a public boon: If that many people are affected, then isn't it time to do some serious research/funding?

Then again, I'm somewhat of an optimist when it comes to human beings.

Speaking as a 'murican, I think the big problem over here is that principles are often a post-hoc justification for doing what you want rather than as a guide for deciding what is just. Conservatives would argue state's rights for keeping segregation in effect but then deny the very same argument for allowing legalized pot. How is it bad for the Feds to use force to put black kids in white schools but fine for the Feds to use force to close down medical dispensaries?

Understanding the politics in this country makes a lot more sense if you just ignore the rationalizations and just substitute "Because I want to" or "because I said so."

As someone who aspires to have principles, I'm torn about the majority/minority argument. If we went with straight majority rights, if 51% says it's fine to kill x minority, that's straight democracy. Ok, so there's the safeguard that civil rights are not up for debate and so you aren't allowed to kill people, full stop. Well, that can be put up to a vote now as well, right? So we'll limit the right of the majority to be absolute. But then when does this eventually come around to the point where the minority can ignore the interests of the majority? Minority in America has become synonymous with brown people but oligarchs represent a fairly diminutive minority with a whole lot of power.

It's deeply depressing, especially when you consider that the side that abandons the rules first has a first mover advantage. It may not let them win but it lets them ruin it for everyone else. And if you stick to the rules when they are not, you'll get to die for your beliefs. Noble, if futile.

"Hospital and especially GP-level records would include all drugs (by brand name), surgical interventions, diagnostics, etc. Anyone with the right analytics tools/smarts could then mine these data for real honest-to-god evidence of just how effective current treatments actually are."

Well, that is the whole point of care.data. By studying the data for the entire population it should be possible to see which drugs and treatments are effective and which aren't, which hospitals/GPs are effective and which aren't etc... The data is very valuable and is one of the NHS's biggest assets, but the government are happy to give it away.

skepticism about the US grade-school version.

The grade school version did include such things as not allowing factory goods to be made on this side of the pond. Plus rather stringent banking restrictions. And some rather heavy handed operations by the troops stations here.

The "colonies" wanted to grow up. The parents didn't want them to. A fight ensued and the colonies got to grow up without parental supervision. Results were mixed. Lots of good. Lots of so so. Lots of not so good.

The current debate about healthcare records means that we get lots of people phoning us up asking us not to share their records (I sit across the office from the person who has to deal with their calls). What they don't seem to understand is that we have shared their records for years, within the NHS. Discharge summaries are sent to GPs describing treatment, and in order for a hospital to operate we have to send patient treatment data to Clinical Commissioning Groups and prior to that Primary Care Trusts in order to be paid for treating their patients.

When I read the title of this post on the main page, I reckoned it was probably about the fact that (in the US at least) large retailers can reliably infer pregnancy from historical purchasing data linked to a "credit card, name, or email address": How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did

Refusing to sign up for loyalty cards may not protect you from de-anonymization during checkout. See for instance this information about FICO's (possibly retired?) "Contact Builder" product:

Addressing the number one retail challenge - the anonymous customer.

Contact Builder provides you with contact information when you capture either the in-store customer name or telephone number and a customer-provided ZIP code or trade area information. The customer name can be keyed or captured from the encrypted name off the payment transaction...

(This is a US-directed product it seems likely that similar products are made for the UK.)

"I think the big problem over here is that principles are often a post-hoc justification for doing what you want rather than as a guide for deciding what is just."

Yes, but Americans tend not to listen to the details about principles much.

"As someone who aspires to have principles, I'm torn about the majority/minority argument. If we went with straight majority rights, if 51% says it's fine to kill x minority, that's straight democracy."

The USA has usually been blessed with a solid core of apathy. When somebody wants to do something extreme, usually the majority doesn't care about the arguments but just wants to say no. With important exceptions, when there's some big dispute they tend to ignore the details but go with the side that looks generally more reasonable.

I think they went along with desegregation because it looked like a lot of blacks who just wanted equal rights, and the racists were blowing up churches and using water cannon and attack dogs on peaceful demonstrations and killing white boys who came down to witness. If we'd seen more black violence and less MLK it would have been different. Watts and the DC riots didn't help.

"It's deeply depressing, especially when you consider that the side that abandons the rules first has a first mover advantage."

Usually they get a big disadvantage from that. It bothers me that it seems like the apathy is getting less. 9/11 drove pretty much everybody crazy. On lots of polls it seems like more people are aware of the issues. The Don't Knows seem to be fewer than before. More people can point out tibet or argentina on a map. I get a sense that more people are interested in politics. I hope it's just my own biased sampling which makes it seem that way.

Oh, please don't get me wrong - I've seen the publicity about it all, balanced the risks and decided to let them share my data from my GP and I knew they already shared my hospital data.

When asked specifically what could be worse about what's shared on Facebook or public access to my hospital data, public access to one's hospital data could be a lot worse in a number of ways for a number of people.

But even without direct commercial interests, the epidemiological pros from having the data out there are large. The risks to most of us are small, and those like my nightmare Criminal Minds story that opt out quite sensibly won't distort the population that badly - there aren't that many victims of uncaught serial rapists out there after all. Mental health data might be a more interesting case though. There's a fair number of cases and I can imagine more patients opting out of sharing 'just in case' still. And I think you'll find the lifetime incidence is higher than that. I was in New Zealand about a decade ago, a bit longer, and they were launching a big public awareness of depression campaign. Their statistic was 25% of the population have treatment for clinical depression in their lifetime. I doubt the Kiwi's are a particularly depressive bunch (the Finns maybe, but not the Kiwis). There was a rather similar stat a few years later here in Wales for something about depression in professional rugby players. So I'd guess 30-35% of the population probably have some form of mental health condition over their lifetime, and that pretty much guarantees it in your family... my sister and my mother, my partner, three of my close friends. No one believes me when I say I'm normal, it's the rest of the world that's mad, but the evidence is on my side!

While everyone should make an informed choice, my personal risk assessment is my personal risk of anything nasty in the rest of my lifetime is low, the benefit to others and possibly to me is higher than that, so I'm ok with it. However, I am equally ok with anyone else making a different assessment for whatever reason. I'd hope it's rational rather than purely based on fear of change but I'm not going to go out and try and persuade others to change their minds.

@J Thomas - you wrote "The USA has usually been blessed with a solid core of apathy." Not always. Prohibition comes to mind as a counterexample.

"Prohibition comes to mind as a counterexample."

Yes. And the Civil War. WWII. 9/11. Etc.

I kind of wish it was more consistent with fewer exceptions. But it's been pretty consistent.

"As someone who aspires to have principles, I'm torn about the majority/minority argument. If we went with straight majority rights, if 51% says it's fine to kill x minority, that's straight democracy."

Remember that governments and laws are just subsets of human behavior. Avoiding a government of the genocidal majority isn't a legal problem, it's a people problem- what makes people want to kill off a minority group? Prevent that condition and the laws will take care of themselves. Have that condition and no legal structure is going to help.

The thing to watch out for politically is people who want to create a situation where they move from a minority power to a majority power with their target group of choice under their control. Seceding from an increasingly anti-slavery US to keep slaves may be the most obvious example.

Actually, we're a rather depressed nation.

I have a certain expectation of privacy when it comes to my personal information and that includes medical details. I can see the benefits of mining a large dataset with the aim of improving public health but it needs to come with appropriate checks & balances. What I would object to is if my data was sold off to commercial interests.

Perhaps part of the on-going social stigma is because there's so little talk about any 'cures'. Progress in treating/curing Alzheimer's might help shift this attitude, as this disease has the right mix of attributes: combination of genetic/environmental causes, most AD sufferers are nice/ordinary people (i.e., AD is not a punishment for a sinful youth/lifestyle), all ethnic groups are affected, etc. Most importantly, the numbers of AD patients are going to grow even more as Boomers reach 70.

Civilization is an on-going process of updating attitudes, so there's still hope.

Yes well, "first mover adavantage" Polish-Russian border 1812 Belgium 1914 Manchuoko 1933(?) Poland 1939 Pacific 1941 Georgia 2008 Crimea 2014 & the one to worry about Königsberg ( oops, Kaliningrad ) 2017 (?)

Probably illegal under Data Protection Act rules ....

Afterwords .... What happened, in the end to all those (past) advantageous "first movers" I listed? Yes - but how many people got killed in the process & how long did it take? Ans still there are "leaders" who think it's a cool stategy ( Because "This time it will be different" I suppose ) Start worrying now?

It's a variation on the "Short, Victorious War" mistake, isn't it?

"What happened, in the end to all those (past) advantageous "first movers" I listed?"

You can add well over a hundred examples where the USA sent in the Marines to get a first mover advantage. Though usually there were incidents to prompt it, like incidents of civil disorder in which US citizens were hurt or could have been hurt.

Usually things worked out quickly enough that Americans hardly noticed. Sometimes it was harder, and for a few like Vietnam no amount of effort got a result they considered adequate.

First-mover advantage: you forgot: Sinai, 1967, and the rematch in 1973.

And in personal situations, there's the use of guns.

The first person to pull out a gun and point it at the other guy has a big advantage. When the other guy's gun is pointing at you, if your own gun isn't already in your hand it might as well be on the moon for all the good it does you.

And of course the first to shoot has a giant advantage.

They might later need to explain it, but the people who carry guns figure they'd rather deal with a trial than get shot.

I haven't had guns pointed at me very often, but in my limited experience if they haven't already shot you then your bargaining position is better than you'd think.

If there are impartial witnesses who can identify the gunman, you might possibly do the following:

It's psychologically harder to shoot somebody in the back while they're walking away. And it's mostly legally indefensible, for what that's worth.

I've never seen anybody anywhere recommend doing this.

Not quite. 1967 the Egyptians & Syrians were gearing up & making very threatening noises. "Mossad" (or whoever) had determined when they were going to strike & jumped he gun by (IIRC) about 36 hours [Note] ...Definitely not the same thing at all. Not so sure about '73 - didn't the Egyptians actually strike first, that time - can't remember.

Note: The equivalent of catching your enemy whilst he was deploying, which was one of Fredrick the Great's favourite tricks, wasn't it?

"1967 the Egyptians & Syrians were gearing up & making very threatening noises."

This sort of thing depends heavily on whose propaganda you listen to. Egypt was making a big splash for their own internal propaganda, saying what their people wanted to hear but not saying anything like that to the world. Israel publicized Egyptian domestic news. Going by what they did as opposed to what they said, Nasser lied to his own people all the time and it wasn't much indication that he intended war. He did make some troop movements into Sinai but not nearly enough.

Israeli propaganda also points out that Nasser had closed the Suez canal and the strait of Tiran to Israeli shipping which they said was an act of war, so Egupt had no reason not to expect a surprise attack. Neither nation had declared war, but because the canal was closed they were at war.

""Mossad" (or whoever) had determined when they were going to strike & jumped he gun by (IIRC) about 36 hours"

That's the Israeli propaganda. Hard to be sure who to believe. Neither side tells the truth much.

"Not so sure about '73 - didn't the Egyptians actually strike first, that time - can't remember."

Yes. What happened was Egypt and Syria wanted to have peace negotiations. But Israeli officials told the media that arabs couldn't fight so there was nothing to gain by talking about peace with them. Israel did not need peace.

So when Israel refused to discuss peace, Sadat announced there would be war. He said that 1972 would be the year of decision. Only he never attacked in 1972. The Israelis laughed at him.

A couple of times he mobilized his army and the Israelis mobilized theirs. Then he didn't attack. Israel had nothing to gain by crossing the canal and fighting the egyptians west of sinai, so when Sadat demobilized they did too.

The third time he did that they didn't bother to mobilize because they wanted to save on expense. Then he did a surprise attack that they were utterly unprepared for, one that he had announced more than a year ahead of time.

" The first person to pull out a gun and point it at the other guy has a big advantage. When the other guy's gun is pointing at you, if your own gun isn't already in your hand it might as well be on the moon for all the good it does you. "

Not necessarily. There was an instance that is recorded in one of my Dead Tree Books in which a police officer empties his revolver at an attacking knife wielder and misses every time whilst the knife man doesn't ..miss that is.

Damn can’t find that book! Not to worry, I'll just feed " pistol against knife " into Google, and, oh wot the hell this will do ..

http://www.youtube.com/watch?v=cGzeyO3pGzw

Gunmen/Gunpersons routinely train by killing the ever so deadly Paper Target that is imprinted with an attacking Bad Guy. Real combat on the street or in a confined space is rather different. Note that the gunman in the video clip is not wearing his pistol underneath a jacket that he has to sweep aside before he can draw his weapon.

Damn ..where did I put that book?

Not to worry I think that the point is that It all depends on situation and training ...

http://www.theppsc.org/Staff_Views/Tueller/How.Close.htm

I don't know how true it is, but the "knife to a gun fight" crowd say that if you're less than 20 feet away (which makes them American probably) if the first shot doesn't kill you, you kill them with the knife.

I'm sure there's lots of assumptions and so on (like you actually hit them and kill them first time with the knife, that you're aware of the shooter and able to move freely before the first shoot and so on) and I wonder just how much evidence they have to base it on.

OTOH armies around the world still teach bayonet drill. I assume it's not just for aggression training but that at various points troops around the world still fit bayonets and stick them in people. Maybe not an everyday occurrence any more, but it's still used enough to be taught.

"I don't know how true it is, but the "knife to a gun fight" crowd say that if you're less than 20 feet away (which makes them American probably) if the first shot doesn't kill you, you kill them with the knife."

If they are actually ready to shoot you, and they are pointing a gun at you when you make your first quick movement, you will probably get shot. The first one to pull out a gun and aim it has a great big advantage.

Since we're fantasizing about this stuff, I figure if you have a knife and the other guy has a gun aimed at you and he's inevitably going to kill you if you do nothing, then you have little to lose by rushing him and trying to kill him first. There's a chance he'll be so startled he won't shoot at all, though that's less likely if he's inevitably going to kill you.

My problem with this fantasy is that if he's inevitably going to kill you, why hasn't he already shot you? The evidence is that he doesn't want to shoot you right now, because he hasn't. If he's a mafia hitman who wants to intimidate you into going somewhere lonely where he'll shoot you, that's a problem. But usually when people point guns at you you can listen your way out of getting shot. Do you really want to kill him? Enough to risk getting killed?

US statistics claim that somewhere close to 3 million times a year gun owners believe they prevent violence by pulling a gun on somebody. But the total number of gun deaths per year is only around 30,000. That tells me that better than 98% of the time when somebody points a gun at you, you will not be killed.

It's a tremendous rush to point a gun at somebody. Imagine you're in a confrontation, they're angry, they raise their voice, it looks like they might get violent. You point a gun at them and all of a sudden they freeze. Then they start talking placatingly, calmly, like they think you're a dangerous lunatic or something. Talk about immediate reward! You'll probably be telling your friends about all the details for a week. You used your gun to prevent violence. You'll probably look for opportunities to do it again. There are something like 10 million Americans with concealed carry permits and by their own estimate on average they do it only once every few years. Admirable restraint.

But I digress. We were talking about first-mover advantage. The first person to pull out a gun and point it, has a big advantage. The first person to shoot also has a big advantage. It doesn't necessarily mean they will be the survivor, but they have a big first-mover advantage.

I know it's off-topic...

Hi Sparks :)

Agreed with most of that, although my experience of airports has been almost universally positive (barring the time BA managed to lose my ammunition and bolt for almost 24 hours).

Martin (PS it was Jon who asked :) )

Happened at Goose Green - last-minute charge by Brit commandos - quite a few Argentine conscripts were indirectky killed by their facsist guvmint - but the actual means of death was a bayonet. Euw.

That would be the Parachute Regiment you are thinking of. Not the Royal Marines. The units concerned take the distinction a tad seriously as I understand it.

Although other units did deliver bayonet charges in that conflict, the marines included. Just not at Goose Green as I understand it.

Ah, yes - my oops - red/maroon berets, not green/blue(ish) ones ... The guys wearing sand-brown berets, of course do not participate in bayonet charges ...