Freeman Dyson recounts an anecdote in his memoirs from the second world war, during which he worked as an analyst for RAF Bomber Command. Part of his job was to work out what parts of RAF heavy bombers were critically vulnerable to damage and needed extra armour. To do this, he and his colleagues examined aircraft that had returned to base after being badly damaged. They'd map out the areas that were most frequently holed ... and apply armour everywhere else. (The point being that, for a given chunk of airframe, if no bombers made it home with damage to that area, then damage to that area was very probably fatal.)
To identify the most lethal risks you must identify the situations where nobody has survived to deliver a warning ...
Here's another similar puzzle: myocardial infarctions (heart attacks) in Scotland. The last figures I saw suggested that a first heart attack is fatal to about 48% of men who experience one. (Subsequent heart attacks are less likely to kill, although they're still serious.) However, among men who presented at a hospital within three hours of onset of symptoms the survival rate for a first heart attack was around 90%. Which leaves us with a dilemma. Are first heart attacks in Scotland instantly fatal to nearly half their victims? Or could it be that around half the patients mistake their symptoms for bad indigestion, or an asthma attack, or just don't want to complain about the chest pains?
My money is on the latter — that most of the lethality of first heart attacks is down to the victims not being familiar with the symptoms and understanding their urgency — but of course we can't be sure: the people who could tell us are all dead. (We could run a health education program targeting the most vulnerable demographic, then see what happens to the survival rate, but that doesn't tell us about the people who have already been affected, or the bombers that never came back.)
It seems to me that space colonization and especially interstellar colonization missions using a generation ship may pose an extreme version of this paradox, which is a thinly veiled variant on the old anecdote about the scientist who has dropped his front door keys at night. (A neighbour finds him on hands and knees, hunting the sidewalk, and joins in. But there' s no sign of any keys. "Well, do you know where you dropped them?" asks the neighbour. "Oh, over there!" The scientist waves at the darkness down the street, between lamps. "Well, why are you searching here then?" "Because I can't see in the dark ...")
In previous discussions, I've looked at the minimum complexity of a biosphere necessary to support human life indefinitely (hint: blue-green algae and tilapia and soy is not sufficient). And asked what minimum population size is needed to support the skills and knowledge needed to maintain a modern technological civilization — as a proxy for the skills and knowledge needed to maintain a generation ship or space colony. I'm not even going to start asking what tools and techniques we'd need to colonize a terrestrial planet lacking an Earthlike biosphere — let's just say that it's obviously non-trivial and move swiftly on.
It seems to me that we don't currently know the extent of our own ignorance of the obstacles to interstellar flight. But by the same token there are some things we could do to start probing the blind spots. Assuming we ever get cheap-enough access to Earth orbit that asking these questions makes any kind of sense, then it follows that we should try and probe the extent of our ignorance before we get too far from home to evacuate.
So, proposal: before actually sending out a generation starship, a necessary first step is to assemble the habitat section, populate and provision it, and boost it out to the Earth-Sun L2 point (a stable solar orbit in Earth's shadow, about 1.5 million Km further out from the sun). L2 is a good proxy for interstellar space — it's in perpetual shadow and very cold — and if we can run a self-contained and unreprovisioned habitat there for a century, then we can probably strap a propulsion system to it and run it for a century-long interstellar cruise. On the other hand, it's close enough to home that if the biosphere crashes due to some obscure micronutrient cycle going nuts, it (and its inhabitants) can be brought back to Earth orbit.
In the absence of such an experiment, sending out a generation ship would be wasteful: not only would we be sending it out in ignorance of all the risk factors, but if it failed it would not return any useful information about the cause of failure. Just as the pilots of a stricken airliner seldom know exactly what's gone wrong (much less take the time to explain it to the flight voice recorder), we can't rely on the crew of a generation ship to diagnose a problem that they've allowed to arise. If they knew what caused the crisis, then almost by definition it wouldn't have arisen in the first place. They're having a first heart attack and they don't know what the symptoms are.
Those who're sending out the starships need to know where to bulletproof the design. And I doubt that starships will ever be cheap enough (or go missing frequently enough) to apply Freeman Dyson's world war two statistical methods successfully.