Charlie's Diary

(Note: In the following rant, I'm sticking to American currency and prices because (a) they're relatively familiar to non-Americans, and (b) they're where I've got the hardest data. Not to mention (c) being where the market I'm talking about is — or isn't.)

I've been ruminating for a whole long time now about the dog that didn't bark in the nighttime world of publishing — the coming ebook revolution, which has been coming now for something like 20 years and counting without much sign of actually arriving.

In point of fact, ebook sales figures are dismal. At best, they tend towards 20% of hardcover sales by volume — and that's for ebooks that are available in open formats that are not tied to a particular hardware platform, and that are not crippled by DRM (digital rights management) encryption schemes that prevent users from reading them on more than one machine. DRM-infested ebooks sell an order of magnitude fewer copies, in many cases not even covering the cost of taking the existing typeset masters and saving them in an ebook format.

The performance of the ebook market is in fact piss-poor. It can be explained in part by readers' natural aversion to DRM (if you change mobile phone or laptop, why should your entire library evaporate?), but also in part by publishers' idiotic aversion to the idea of trusting readers.

When you look at the "pirate" ebook field, things are a lot livelier. There are any number of locations on the internet where you can grab hundreds or thousands of novels, for free! — albeit in violation of the authors' copyright. These books are either produced by scanning a paper copy and feeding it through OCR (optical character recognition) software, or by cracking the DRM on an encrypted ebook. Lots of people download books off the net, but one thing even the proponents of ebook DRM agree is that it doesn't seem to have had any economic impact on the sales of dead tree editions. In fact, there's a lot of evidence from research into music file sharing that people who use "pirate" ebooks actually buy more of the real thing. (Eric Flint volunteered this source: The Effect of File-Sharing on Record Sales: An Empirical Analysis, by Felix Oberholzer-Gee and Koleman Strump, published in the February 2007 issue of the Journal of Political Economy.)

So what's going on?

As Cory Doctorow has observed, the common complaint that readers don't like staring at a screen for hours on end doesn't hold water; we actually spend a lot of our time staring at computer screens, PDAs, and tiny little displays on mobile phones.

Let me stick my neck out here, with an opinion that goes against the conventional wisdom in publishing circles.

In the pre-internet dark age, there was a subculture of folks who would get their hands on books and pass them around and encourage people to read them for free, rather than buying their own copies. Much like today's ebook pirates, in terms of the what they did (with one or two minor differences). There was a closely-related subculture who would actually sell copies of books without paying the authors a penny in royalties, too.

We have a technical term for such people: we call them "librarians" and "second-hand bookstore owners".

Library lending was tolerated by authors and publishers because it was widely accepted that in the long run, people who borrowed our books from libraries were more likely to read them than people who had no access whatsoever. And having read, they were more likely to become regular readers and to eventually buy — if not the books they'd already read, then the next one on. Library users were often poor, or casual readers, or young. I remember latching on to the local public library when I was five or six years old. I read my way through most of Andre Norton's childrens range before I was eight, and I certainly didn't pay for them. I couldn't pay for them; I didn't have enough pocket money to make a habit of buying books at anything approaching the rate I could read them until I was in my teens, and even then, I was mostly limited to second-hand paperbacks.

So, in the dark pre-digital age of the 1970s, I was an avid supporter of that period's equivalent of your demonic ebook pirates. And y'know what? I defy anyone to tell me I was wrong to do so. Or even to assert that it hasn't, overall, been a good thing for the SF field, because it got me into the habit of reading, and these days, with a disposable income, my biggest problem is finding bookcases to stick all the new hardbacks I've bought over the years since my teens.

Now, let's talk about ebooks.

I'm not going to flog the already-dead DRM horse; I've been there before and both sides of the debate are fairly well covered.

What I'd like to point out is that the economics of the commercial ebook market are sick.

Right now, many of the largest publishers charge a cover price for ebooks that is 80% to 100% of the hardcover price. Virtually nobody except Baen (and now a couple of other publishers who've dipped a toe in the Webscription market, and some self-publishers) is even thinking about trying to establish what an ebook is really worth in the market.

We know roughly what it costs to produce a book, and we can point to the areas where ebooks are cheaper than paper editions (no dead trees and ink, for one thing; no warehousing or distribution for another) and more expensive (downloads, website maintenance). But we don't really know what an ebook is worth to the readers, because the market that could give us meaningful feedback on pricing has been strangled in the crib.

My take on ebooks is that they are — and should be seen as — the cheapest form of disposable literature. They're not cultural artefacts (pace Cory Doctorow); you don't buy them in signed, slipcased, limited editions. They're like stripped mass market paperbacks without even the value-added of doubling as wood pulp wall insulation once you've read them.

Now, there exists within writing and publishing circles a neurotic fear that sooner or later (probably In Five Years' Time — that seems to be the normal window) a cheap digital paper based ebook reader will come along, that makes the experience of reading text on a screen no different from the experience of reading a lump of dead tree stitched inside a piece of pigskin. And, as the horror story has it, we will be In Big Trouble, because the pre-existing availability of pirate ebooks will lead to enormous proliferation and a total crash in the value of books. Some pretty smart people believe this story, and the result has been to give it more credibility than it actually deserves. And it leads them to draw what I believe to be faulty conclusions; if you want an example, look no further than this column by Jerry Pournelle. (NB: I don't want to single Jerry out for specific opprobium, and I think it's only fair to note that what he's really talking about is the DMCA. However, I think this article typifies the received wisdom among many writers on the subject of ebooks, piracy, and DRM, which is why I dragged it in here.)

It's the received, prevalent wisdom — and it's a load of rubbish.

First of all, if overlooks the point that publishers don't manufacture ebook readers; the consumer electronics industry does. And the consumer electronics industry will not cut off its own nose to spite its face by producing an ebook reader for $20, if it can produce one with extra bells and whistles that sells for $350. We've had the tech for a $20 (or $50, anyway) ebook reader for a decade; it would resemble a grey-scale palm pilot, albeit without even the PDA functionality. But the parts are dirt cheap these days! If a manufacturer thought they could sell the beast, they'd be churning them out by the bucketload — and it's perfectly possible to read ebooks on a 160x160 green screen. I used to do it all the time in the mid to late 1990s. The reason nobody makes such a beast is because it's simply not profitable to do so. Explaining why this is so ought to lead into a long essay on the cost structure of consumer electronics, but basically, unless the Chinese government decides to subsidize its indigenous manufacturers in order to deliberately destroy the western publishing industry, it ain't gonna happen.

Secondly, and more devastatingly for the sky-is-falling promoters of the "pirate ebooks will doom the publishing industry" theory, until ebook readers cost no more than a hardback, 90% of readers will ignore them. And that's regular readers, not the folks who own four books (and one of them is a Bible). Expecting people to cough up $200 for a reader so that they can then pay $25 for new novels to read on it — as opposed to buying the novels for $25 (less discount) in hardcover and having the cultural artefact — is, well, it's just bogus.

We might see such a device (at $200) take off in the book club market. Imagine you join the e-book club. Your first sign-up gets you an ebook reader loaded with five titles for $20. Then you have to buy a book a month for the next year before you can leave, and you're paying $20 a pop. After a year you've got 17 novels and an ebook reader, and you're out $240 for a $200 reader. Most abook-clubbable people will stay in (they're set up for the club and they've already got a small bookshelf on their reader) and over the next year the club can make the profits to pay for that first year's loss-leader.

But 80% of readers don't do book clubs. I've seen my book club sales, and they're piss-poor (except in France, which is different).

Basically, the universal ebook reader is a non-starter — at least for this generation — for the same reason that it's near-as-dammit impossible to sell hardcover midlist novels for more than US $24; consumers don't like being milked.

Now, having demolished the myth of the $5 ebook reader being just around the corner, the second problem the publishing industry has with ebooks is their misapprehension of exactly what the "pirate" ebook field is costing them. Some otherwise fairly intelligent folks in the SFWAs anti-piracy committee think they're potentially costing up to 30% of their revenue stream. I'd like to call bullshit on that.

There's a figure I've heard quoted (unfortunately I don't know the source so I can't cite you chapter and verse on it) to the effect that the typical dead-tree book has, over its life cycle, an average of four readers. Moreover, sell-through in paper is around 50-60%; that is, for every book sold to a customer, 0.8 to 1.0 other books end up being returned or pulped. So the real figure is more like ten readers per book actually printed by the publisher.

Think about that. Today, publishers try like crazy to tie ebooks to a single reader via DRM, in their misplaced zeal to reduce profit leakage; but for the economic hit from piracy to equal the economic hit from libraries and second-hand bookstores and friends lending friends books, the unlicensed distribution channels would have to be shifting nine ebooks for every one that is sold commercially.

And you know what? I don't think most of the ebook sharing subculture is even about reading the books in the first place — it's about collecting, and participating in a gift sub-culture where your kudos is governed by how much stuff you can give away. Yes, this probably sounds alien to a lot of you. All I can say is, you haven't spent enough time monitoring alt.binaries.e-books.flood and the other pirate ebook distribution channels. There are folks there who, of a weekend, post more books than I could read in a lifetimes. Random, eclectic, nonsensical collections of books, some of which are hopelessly corrupted and most of which are poorly proof-read. These folks are not reading what they put out. They're not putting it out with helping other people read the stuff as a primary goal, either. There's another dynamic at work, and no scheme to stop or reduce ebook piracy stands a chance of working until we understand why it's happening.

Interestingly, Baen's webscription titles are under-represented on the ebook warez newsgroups. I don't think this is an accident. Books that come up most often are either scanned and OCRd paper copies, or cracks of DRM-locked ebooks. If you look at the posters' activities in terms of proving status within a gift economy this makes sense; OCRing a book or cracking DRM takes time and effort, and is a demonstration of putting effort into something — it's a high value activity. Whereas posting something you grabbed off Baen's library of for-free books, or paid $5 for is just stupid — it's like turning up to a a wine and cheese evening your friends are running on a "bring a bottle" basis with a bottle of Buckfast or Mad Dog 20/20. It's cheesy, tasteless, and looks cheap, and that's how the ebook pirate elite will view you.

So, it's time for me to advance some tentative conclusions about why the commercial ebook market is broken:

Which leads to the next question:what is to be done?

(To be continued, when I get around to it ...)

Posted by Charlie Stross at 5:29 PM | Permalink | Comments (270)

From time to time, readers ask me questions: and one of the commonest questions is, "when is [X] coming out?" for values of [X] that are usually a novel I've been muttering about for the past couple of years and have finished some time ago. And my usual answer is along the lines of "some time next year (or even the year after)". This typically produces the response, "why does it take so long?" So here's my stab at explaining what happens when you hand a novel in, from the author's point of view (which is horribly skewed and subjective and nothing like the editor's point of view).

The first thing to understand is that a book publisher is typically running a production line. They have a monthly schedule with [n] slots in it, where [n] can be anything between 0 and 20 titles. Typically slots are allocated to authors up to 2 years in advance. A slot might be assigned to J. Random Specialist's learned treatise on the care and feeding of Swamp Guppies (hardcover, est. 2000 sales) or the latest paperback edition of Harry Potter and the Preposition of Noun (slay more forests, lay on more convoys of 18-wheelers). But it's still a production line, which means it runs at a constant speed in terms of the rate at which it pumps out finished products.

(I'm going to ignore the minutiae of marketing and sales, but suffice to say that without them, the book ain't going nowhere. If you're interested in the grisly details, there are plenty of publishing folk who deal with marketing and sales on a day to day basis: for example, Anna Louise — or, from an agent's point of view, Miss Snark. But for now, I'm just going to talk about the production process in genre fiction publishing, as seen from the writer's end of the business.)

Now, your agent is your reader of last resort. After all, they're on commission: they get a 15% cut of your income, in return for doing their damndest to maximize that income. (Which they indeed do. Tobias Buckell ran an anonymized survey of SF/F writers' advances and discovered that agented books on average get about a 60% higher advance than un-agented ones, and better royalty terms all round.) That's because your agent knows all the tricks the publisher's contracts people will try to pull. And they're also a professional salesperson. And they're on your side.

In my case, I rely on my agent not just for the small print negotiations, but as a sanity check. She's a former editor, and if she raises a red flag over some aspect of a book I'm handing in, I'd better take it seriously. She has a strong interest in not letting me shoot myself in my foot.

However, I'm not her only client, so it will probably take her a week or so to read the book and get her thoughts together once I send it to her.

Once my agent agrees it's ready, she then sends it to my editor, who reads it. Which takes another week or two, because in addition to publishing Charlie Stross, said editor also publishes folks whose sales I can barely dream of, and who are therefore ahead of me in the queue. What's more, they also work for a big corporation in what is largely a managerial role and therefore get to spend lots of time in meetings.

When my editor has read the book and approved it, they notify Finance that the book has been delivered, which causes them (in principle) to send me a cheque for the D&A component of the advance (which can be something like 25% or 33% of the total, depending on how the contract splits up the payments).

Now, the publishing industry habours a dirty little secret: even in this age of high-throughput low-overhead efficiency cultism, some editors like to edit. They will kick the tyres and piss on the fender and get back to the author and say "change this around, and get rid of the happy singing dinosaur in chapter 14". And the author will therefore have to do a whole bunch more work on the book before it's acceptable. For the sake of this essay, the editor I'm dealing withis tired and cynical and knows my agent is on the ball, and if my agent is giving feedback before the book is even delivered, why bother? Which means everything goes smoothly.

(But at another publisher, it's quite possible that the editor will say "change this", and you politely argue the case (30%) or obey (70%), until a satisfactory manuscript is achieved and your editor signs off on it. Which may take months because editors are busy folk and even if you make the changes immediately and email them a revised manuscript, it may be weeks before they have time to read it ... and say "hey, you didn't do what I told you to do! Bad author! No advance!".)

Anyway, once the editor is happy with the manuscript the process of turning it into a book begins. And it is printed out on 11" long strips of dead tree and mailed to an external copy editor.

You know what copy editors do, right?

It takes the copy editor about 2-3 days of wall-clock time to work over a novel. They then send it back to the editor (or rather, to your editor's editorial assistant), covered in red crayon chicken tracks. Bear in mind that postage time eats up 4-10 days either side of this 2-3 day process. $EDITOR then has their intern mail it to $AUTHOR, who is given three weeks to (a) vet the chicken tracks and approve or countermand them, and (b) get them back to $EDITOR. Three weeks may sound generous to you, but bear in mind that the postal time between $EDITOR and myself is at least five working days — there's an ocean between us! And if I'm either travelling on another continent, or down with the flu, that three week turnaround schedule is going to be missed. (In mitigation, any sensible author tries to keep their editors in the loop when they're expecting to be unavailable for more than a week, and tries to find out in advance when a bunch of copy edits is due to land on their desk.)

The Copy-Edited Manuscript (CEM) then goes to Production, along with an electronic copy of the raw manuscript as a Microsoft Word document. (Speculating as to why the publishing industry demands Microsoft Word is futile; it's like death and taxes.) At a pinch they can handle plain text from Joe Stick-in-the-Mud's manual typewriter, but keying it in costs money. Production slurp the Word document into Quark Publishing System or InDesign or another publishing program, and then a typesetter goes through it by hand, transferring the hand-made changes from the CEM, until they have a typeset book block that looks like the real thing. This process probably takes about 3-5 days to do properly (someone has to check it), but it's scheduled in a queue (remember what I said about that production line earlier?) and is therefore at the mercy of all the other jobs in the queue. For example, if it's an election year and H. Beam Piper wins the White House on a write-in vote, suddenly everyone will be rushing biographies of H. Beam Piper through the presses. (This kind of job is like meeting a Challenger tank on the motorway — you don't argue right of way with it, you just get out of its path.)

While this is going on, the book designer takes a look at the MS and the art director reads the marketing synopsis of it and commissions a cover painting and they put their heads together to design a cover. Which the author might be invited to comment on ... or not. (As most authors are not graphic design/marketing folks, they do not necessarily have anything more useful to contribute to this process than, as one editor put it, "squawk! $PROTAGONIST's hair colour is all wrong! I must immediately bring western civilization to a screeching standstill until this is corrected!")

Anyway.

In due course, someone prints out a PDF of the book and mails it to the author. This is still called a galley proof, although it's the end result of a rather different process these days. The author is expected to proof- read it again, and mark up all the little easter eggs and typos that Production introduced into the (always-perfect) CEM. Because, as you know, nobody ever scrawls something illegible on a CEM, or fails to correctly interpret said scrawl. It's another three week turnaround job.

When the corrected galley gets back to the publisher, it gets sent back to production, who update their typeset copy again. Oh, and with any luck the editor also proof reads a copy, and then there's a second round of checking and an external proof reader goes over the corrected galley (if you're lucky and they're not cutting corners this year). One of the laws of publishing is: the worst, gouge-your-eyes-out typo will be discovered by the first reader to open the first shipped copy of the book. It doesn't matter how many typos you hunt down and kill in the production process, there will be more, lurking in dark corners. But that's no excuse for not doing a thorough job ...

Finally the book is approved for manufacturing. Production send it to their printers (who are almost certainly a separate company — publishers don't own printing presses these days, unless they're newspapers, and not always then) and in due course a shipping pallet of hardbacks materializes in the warehouse and is distributed to boostores by the Sales Elves and the Distribution Fairies.

Believe it or not, this is not the end of the story. What's just gone out is a trade paperback or hardcover first edition. There will be errors, and your readers will email you to gloat about their genius in spotting them. So you save these errors up, and after a couple of months you email your little list to your editor, who will sigh and pass them on to production for the second printing or the mass market paperback. When it's paperback time, they'll go through the whole galley proof checking stage all over again, because as likely as not they've reflowed the typeset text for a new page size, and added corrections. But proofing a paperback isn't particularly onerous: most of the mistakes have already been extirpated.

Now, if you've been keeping track, you'll recognize that there are no less than eight mail shots involved in this process, where the manuscript or CEM or galleys have to go walkabout, and for each of which you need to allow a week (as many of them are trans-continental or trans-oceanic in scope. Because it's a whole book, you also need to allow a working week for each interim stage. We're up to 16 weeks, now. You're an idiot if you don't allow 25% for contingency time due to the usual vagaries of business (author has flu, editor is on vacation, typesetter is on maternity leave) so we're up to, say, 22 weeks. There's also disaster time. If the CEM goes missing in the mail on the way back to the publisher and the office photocopy goes missing, then it's possible that you'll have to re-do two steps (i.e. the initial copy-edit and then the author's check), taking another couple of months. Think it couldn't happen? I know an author it happened to just last year. (Me, I have a scanner and I scan my corrected CEMs to PDF before I mail them. But I'm paranoid, I hate checking CEMs, and I take pains to avoid ever having to do it twice.)

A publisher can rush a book through in just 10 weeks, if it's particularly time critical (see "Shock Election Outcome: Dead SF Author in the White House"), but the stolen time has to come out of various other books' schedules, and they also have to throw money at the process to make it work (i.e. expensive courier services instead of relying on the mail).

What you're left with is, a job that should be do-able in 30 weeks, even in the face of disasters, illness, and unscheduled excursions. And guess what? Once you cross out December (because like many businesses, publishing doesn't get a hell of a lot of stuff done in December) there will be approximately 30 weeks between my novel arriving on my editor's desk to the PDF being sent to the printing press.

Now. Can we do this any faster?

In theory, you might imagine all the above jobs could be done using email, Microsoft Word's change tracking facility, and some technical nous. Saving all that postal time would in principle result in a much faster-running production line. But the catch is, not everyone uses Microsoft Word. Not everyone is technically ept, or able to use Word's change tracking facility effectively. A publishing company has to be flexible enough to deal with uber-competent geeks and eighty year olds who still write everything in longhand and don't have email at all, or even a fax machine ... but who have an enormous and loyal readership. The production line can only crank out a finished product at the speed at which the slowest raw material supplier produces input. Also, if you try to run a production line faster, the effect of any delay is amplified. Delivering something a week late into a 30-week process isn't critical. Deliver it a week late in a 4-week process, and you can screw everything up. From the publisher's business point of view, consistency is more important than speed; thus, there is a positive incentive for larger publishers not to hurry things along.

Small presses are lithe, nimble, and lean. And they usually have laser printers and understand email. My Laundry novels were originally bought by Golden Gryphon, who're small enough that the editor handles copy editing issues himself. So we worked on the copy edits by bouncing annotated Word files back and forth in email and using change tracking. The trouble is, because it took about one minute (instead of five days) to deliver the CEM, we kept going back and forth across it, picking up more issues. The book got a very thorough copy-edit indeed, but the process ended up taking about as long as a normal pencil-and-paper cycle. On the other hand, there was no risk of the MS getting lost by the Post Office, we saved a bunch of shipping costs, and we probably did a better job on it.

Typesetting at a small press follows the standard pattern, except that rather than mailing me a dead tree they often email me a PDF, and I email back lists of changes: this ends up saving maybe two weeks in postage time. However, due to the repeated copy-edits using change tracking, the galleys were very clean compared to the normal pencil-and-paper routine: because they were typesetting from an already-corrected electronic CEM, rather than importing a rough manuscript file and then hand-inputting scribbled changes from a paper CEM.

The moral of this story is, if your editor, your typesetter and your author are technologically literate and have the right tools at each end, you can do the whole job a whole lot faster. But this was a small press, only publishing about four books a year. Companies who deal with hundreds of authors have to be able to handle everything between Neal Stephenson (writes his own Emacs LISP macros to format the output he wants to deliver) and Joe Schmoe who still uses a fountain pen and pays a copy-typist because he never learned to type. You can feed the technically literate types into the pencil-and-paper chain and they'll just grumble a bit and get on with it, but if you expect Joe Schmoe to grapple with Word's change highlighting facilities you're going to get a nasty shock to the production schedule.

And this is why, in a nutshell, my novels take somewhere between 6 and 18 months to appear after I deliver the finished manuscript.

Posted by Charlie Stross at 10:55 AM | Permalink | Comments (36)

I'm very pleased to announce that my latest SF novel, "Glasshouse", is out now in hardcover from Ace. (Orbit aren't publishing a UK hardcover — if you want a British edition you'll need to wait until next March when it comes out in paperback.) You can order it from Amazon.com or find it in bookstores — Amazon won't ship it until Friday, but most shops will already have it on the shelves.
Publishers Weekly said:

The censorship wars -- during which the Curious Yellow virus devastated the network of wormhole gates connecting humanity across the cosmos -- are finally over at the start of Hugo-winner Stross's brilliant new novel, set in the same far-future universe as 2005's Accelerando. Robin is one of millions who have had a mind wipe, to forget wartime memories that are too painful -- or too dangerously inconvenient for someone else. To evade the enemies who don't think his mind wipe was enough, Robin volunteers to live in the experimental Glasshouse, a former prison for deranged war criminals that will recreate Earth's "dark ages" (c. 1950-2040). Entering the community as a female, Robin is initially appalled by life as a suburban housewife, then he realizes the other participants are all either retired spies or soldiers. Worse yet, fragments of old memories return -- extremely dangerous in the Glasshouse, where the experimenters' intentions are as murky as Robin's grasp of his own identity. With nods to Kafka, James Tiptree and others, Stross's wry SF thriller satisfies on all levels, with memorable characters and enough brain-twisting extrapolation for five novels.

A perfectly tuned combination of gravitas and glee (the literary/cultural references are a blast). Stross's enthralling blend of action, extrapolation and analysis delivers surprise after surprise.

This Glasshouse isn't just glass. It's a prism that Charles Stross uses to split his storytelling into all of its component narrative colors — suspense, action, satire. It may be his best book yet. It's his most consistently suspenseful, and his funniest. It's got the trenchant humor of The Family Trade gene-spliced to the thrillaminute pacing of Iron Sunrise. It's set far into the same future as his wildly praised (except by yours truly) Accelerando. But whereas Accelerando seemed to strip-mine its future of humanity, and came across to me as cold and uninviting, Glasshouse presents its posthuman "network civilizations" as a never-ending Willy Wonka factory of phantasmagorical technowonders, as frightening as it is exhilarating.

Posted by Charlie Stross at 1:43 PM | Permalink | Comments (21)