My first two published SF novels, "Singularity Sky" and "Iron Sunrise", have a long and tangled history. And I figure it's probably worth (a) explaining why there won't be a third one in that particular series, and (b) spoilering the plot thread I had kicking around that would have been in the third Eschaton novel if I was going to write it.
Let us rewind the clock to 1993, when a much younger version of Charlie was plotting his glorious career (or something like that). One thing I was clear on was that I wanted to do space opera (I said it was a much younger version of me, right?) and to make that work, I wanted an ACME general purpose space opera universe.
I had a number of bees in my bonnet, though.
For starters: how do you get a whole bunch of roughly-competitive human civilizations seeded across a wide volume of space, with time to grow and diverge?
For seconds: faster than light travel would appear to be a necessary precondition to writing wide-screen space opera. But if you permit violations of special relativity, you're also implicitly permitting global causality violation — time travel. (Go read a physics textbook if you're not sure why.) Permitting violations in the first place suggests that there'll be more than one way of doing FTL travel (just as there's more than one way of doing heavier than air flight — compare a helicopter to a jet airliner and a bee). And then you've got to ask, what are the implications of time travel?
And for thirds, as Vernor Vinge pointed out, there's the Singularity. Writing a space opera with FTL means accepting causality violation. And accepting causality violation means computing with closed timelike curves or, in simpler terms, really strong deterministic solutions to P=NP, and then some. Procedural AI hops out of the FTL hatlike a demented magician's rabbit and the singularity takes a shit all over your neatly designed Napoleonics-in-Spaaaaaace boardgame table.
For fourths, once you have causality violation you have to deal with time paradoxes. How do you avoid them? Larry Niven came up with one possible answer in the mid-seventies: a universe with laws of physics which permit global causality violation is only stable when its history contains no instances of GCV — and he invoked the weak anthropic principle as a solution: if you try to switch on your time machine, a freak accident will ensure that something breaks before it works. I found that kind of unsatisfying. On the other hand ...
What I came up with was this:
Circa-2060, some scientists working with New Physics™ set up the first computing device to make use of a causality violation shortcut. (It's easier to send a couple of electrons back in time than it is to build a working FTL starship.) It doesn't do what they expect and merely Do Sums Fast; rather, what they've done is hooked themselves up to one corner of a vast, four-dimensional entity (the Eschaton).
Which has it's own ideas about survival. Whackiness ensues as the Eschaton inflicts a very rapid hard-takeoff singularity on humanity, deporting most of them through wormholes to semi-prepared colony worlds across a radius of some 3000 light years. And not long thereafter, the survivors (on Earth) begin to detect signals from the nearest of the colonies; not only are they scattered across space, but across time — roughly back in time 12 months for every light year out in spatial distance.
What's going on behind the curtain is that the Eschaton is a local enforcer for the Strong Anthropic Principle; it can only ensure its survival if nobody else commits GCV within its light cone. So it intervenes to minimize the risk of that. Humans are meddlers and have just discovered GCV as a tool initially of computing and then, later, for FTL travel. But human civilizations that can extend complex new technologies (such as FTL starships) require a huge population base to fill all the necessary specialities. If split up and scattered, the resulting colonies' development will be retarded by their reduced population density (and hence smaller R&D base): it'll be a long time before they can get up to anything very alarming, and in the meantime the Eschaton has time to convince them that doing things it don't want them to do is liable to be very bad for their health.
The colonies are also life insurance for the Eschaton. If something nasty from outside it's light cone comes in and snuffs it out (presumably by projecting itself back in time to before the E's emergence and destroying the Earth) there will be isolated pockets of humanity seeded across a huge volume. Who are meddlers, and who will eventually come up with a GCV computing device and switch it on, thus hopefully re-bootstrapping the Eschaton ...
(Is your head hurting yet?)
There are problems with this set-up. In particular, there are problems with the evolution of the ACME General-Purpose Space Opera Universe™ from "Singularity Sky" into "Iron Sunrise". SS was quirky but not brilliantly plotted — a lot of crap ended up on the cutting room floor (equal to 140% of the final word count of the finished novel). IS was a more conventionally-plotted novel ... but I made a huge blunder in introducing the ReMastered and their Unborn God. By implication the UB is another Eschaton-level GCV-based AI. It's locked in conflict with the Eschaton, attempting to edit history into a shape in which it, rather than the Eschaton, is the dominant entity. The events of IS are set in the foreground of such a war between time-travellers, and I think I got some of the details badly wrong — at the level of how the events would appear from within.
Want to see that kind of conflict done right? Read Palimpsest (in my collection "Wireless") — in which I revisit time travel with fifteen years' more thought.
Having broken the Eschaton universe, there's no way to go forward (which is why any future space opera you see from me will feature a new setting — I have plans). However, here's a neat idea I was going to shoe-horn into the third Eschaton novel before I realized the universe was broken ...
The Eschaton-verse has multiple solutions to FTL. There are starships; big lumps of moving matter that shuffle from planetary orbit out into deep space, push a magic button, and re-appear in deep space a very long way away from where they started (and hopefully a little bit closer to their destination planet). It's your classic 1950s space operatic jump drive, chosen simply because it makes for good fiction. But there are also "causal channels" — limited bandwidth instantaneous communicators. The snag with causal channels is that they are created as a quantum-entangled one time pad: you create a limited number of bits that, once used up, can't be replenished. You then have to send them to their destination without violating causality (which scrambles them), i.e. on a slower-than-light freighter that takes decades or centuries to arrive.
Finally, starships don't land on planetary surfaces. For getting goods and passengers on board and off again, they dock with space elevators (the one component of this transport set-up that is theoretically plausible).
Have you noticed something? This set-up allows for narrative structures that map onto intercontinental travel circa 1880-1914; we have
railroads space elevators that link national planetary populations to ports space stations where steam starships dock, to transport passengers and cargo slowly between stops; and we have trans-oceanic telegraph cables causal channels to allow instantaneous (but expensive and limited-bandwidth) information transfer.
So here's the technical mechanism I was going to deploy in the never-to-be-written novel on which I pinned the working title "Space Pirates of KPMG" (or, for Brits, "The Crimson Permanent Assurance in Space") — note that a working title is just a temporary one that will be changed before publication; the working title for "Iron Sunrise" was "Space Nazis Must Die":
Starships are expensive, intricate pieces of machinery. They are difficult to build and maintain, and have to be continuously in motion, transporting cargo and passengers, in order to cover their running costs.
There are space pirates. They, too, have to pay huge amounts of money to keep their starships running, and they can't afford to be stupid about it.
The space pirates' business model is this: they identify a likely target merchant ship, match courses with it, and board.
They do not, however, rape, pillage, and murder the passengers and crew. That would leave them having to transport a lot of bulk merchandise and find somewhere to fence it, taking an inevitable hit in the commodity's resale value. It would also set everyone's hand against them. Not good for life expectancy ...
Instead, they audit the cargo. Then they search out for any secret items the ship is transporting, stuff that is of high value but not publicly announced. Many times they don't find any. But sometimes they stumble across a passenger liner with a safe full of quantum computing chips, or a bulk liquid carrier with much less freight volume in its cargo holds than expected and something extremely massive tucked away — a lump of stabilized neutronium, for example.
They do not steal the secret cargo. Instead, they notify their accomplices by means of their private causal channel to buy commodity options based on their insider knowledge of the secret cargo's impending arrival. Then they give the hijacked ship an armed escort (under communications silence) all the way to its destination, to ensure it arrives on time.
Thus: your typical space pirate in the Eschaton universe metaphorically wears a grey pin-striped suit, swarms aboard a merchant vessel with a spreadsheet between his clenched teeth, and has retirement plans involving a senior partnership in a firm of accountants. (Captain Jack Sparrow he ain't.)
Such pirates are tolerated by the majority of sane merchant captains (although they drive commodity traders up the wall) because space is big, space is dark, and space contains a small but worrying number of idiot barbarians who will, if they see a foreign merchant vessel, board it with rape, pillage and murder in mind. Idiot barbarians are bad for commerce. Professional space pirates strongly disapprove of this and will take drastic preventative measures when they run into them.
Now, fast-forward a decade after the events of "Singularity Sky". The New Republic of that book is, clearly, in dire straits. In fact, it's disintegrating under the stress of its own cultural singularity (inflicted by first contact with the Festival). The economy is in tatters; navy crews are not being paid. Is it any wonder if the crews of the surviving vessels of the New Republican Navy mutiny and light out for parts unknown, there to try and make their fortune flying the Jolly Roger (despite not knowing a put option from a hole in a bowling green)?
The pin-striped space pirates aren't going to approve ...