Publishing is a whole bunch of different businesses flying in loose formation; which is by way of saying that this particular topic is specific to commercial fiction publishing and has nothing to do with text books, technical reference manuals, autobiographies, or cookbooks.
Why are novels (the prevailing form of fictional entertainment on retail sale today) generally the length that they are?
Back in the mid to late Victorian period, when books were frequently printed and sold as weekly serials, in chapter-sized magazines that could be bound together, the length of a book was really dictated by the author's (and printer's) stamina. In contrast, as I mentioned in my last blog entry, I've got a book coming out this month which is actually not a stand-alone novel, although that's what it's listed as in the publisher's catalog — it's the sixth (and final) installment in a multi-book story, six volumes long. Why isn't that story coming out in a single binding?
It looks obvious at first — novels are the length they are because, well, they're novels — but in truth, the length of a novel varies depending on the prevailing publishing industry distribution model when it's written.
Let's take SF and fantasy novels published in the USA as a case in point. Prior to the early 1920s the genre didn't really exist in its current form. From roughly 1923 and Hugo Gernsback's publication of Amazing Stories magazine, the pulps reigned supreme: monthly newsstand magazines publishing short stories and serializing novels. Newsstand magazine readers are fickle. Serial novels need to be short enough not to dominate a magazine completely, lest a reader who doesn't like this particular novel stops finding other reasons to buy the mag; and they need to be of finite length. It shouldn't be any surprise to discover that SF novels from the period 1923 to roughly 1952, when the newsstand fiction magazine industry more or less disintegrated (leaving only a tiny handful of survivors) are typically very short — 45,000 to 60,000 words.
The death of the pulps didn't take the SF novel with them; far from it. In fact, book-format novels were already being published (notably Asimov's Foundation series dates to this time — originally serialized in magazine form, they then saw success as individual books), and the mass market novel took over as the main outlet. The length of novels then began to creep up, and continued to creep up steadily through the 1980s.
Many earlier novels are still deceptively short by modern standards. A typical SF novel of the 1960s was 70,000 words long. By the 1980s, 80,000 words was the norm; by the 1990s it had bloated to 100-120,000 words. Why?
One account I've heard (from an editor who was active throughout this period) is that it was the distributors. The mass market for paperbacks prior to 1991 was dominated by wholesalers who supplied retail stores — not bookshops, but local supermarkets with wire-mesh book racks. The wholesalers knew their markets intimately, and would match mass-market titles to the supermarket customers on the basis of their clientelle — SF/F was popular near technical schools, for example. When the inflation of the 1970s and 1980s forced publishers to raise their cover prices, the distributors pushed back and demanded that if the product cost more, it had to be bigger — not taller or wider, else it wouldn't fit the racks, but fatter. (They were, after all, primarily in the grocery business rather than the book trade. You want to charge more for that lettuce? It better be bigger!)
Once a trend like that becomes established, it's hard to stop. Put yourself in the position of a bored browser in front of a supermarket wire-rack, contemplating novels by two authors you've never read. They both cost the same, and you have enough pocket money to buy one. The year is 1980; LibraryThing or other internet resources aren't available. How do you make your mind up? Well, you remember what you've heard about the authors, and you look at the cover painting, and you read the back flap blurb. Assuming all of these are equal ... you probably buy on weight, because you subconsciously anticipate a longer reading experience and, all things considered, good experiences that last longer are better than short ones. Remember that the actual cost of the paper and ink is only a small component of the retail price of a book — around 10-15%. Increasing a book block's size from 150 pages to 180 pages is cheap. And so, from the 1960s to the 1990s, publishers unconsciously trained readers to expect longer novels.
At the end of the 1980s and early 1990s, a wave of consolidation swept through the wholesale sector in the US; from over 500 wholesalers nationwide, the total was reduced to a couple of dozen. Local retail lore was lost due to centralization, and the mass market wire-rack sales more or less dried up; mass market paperback print runs crashed 50%, and this was in many places decried as the death of the midlist. (A midlist author is one who, like me, has a handful or double-handful of books in print and is earning a living but doesn't get their nose in the bestseller lists.) However, the death of the wire-racks coincided with the ascent of the specialist bookstore chains, led by Borders and Barnes & Noble; these chains made up for the drop in mass market paperback sales by providing a greatly enlarged outlet for hardback sales. Losing 20,000 off the top of your paperback print run is painful, but if you can sell 5,000 hardcovers you can make up for the shortfall: and that is why folks like me are still in business.
Now, there is one big problem with making hardcovers longer: binding technology.
In the UK, all retail fiction books (paperback and hardcover alike — aside from some special high-quality editions) are perfect-bound. Pages are printed and collated, guillotined to form book blocks, then bound into the cover using thermosetting glue. There's no obvious limit to the number of pages you can bind this way other than the reader's wrists and the flexibility of the glue — some 1970s paperbacks were notorious for disintegrating on first reading, but these days perfect-bound books up to 1400 pages aren't really unusual ... although midlist authors are not encouraged to go there: production costs scale with the size of the book, and you don't get to charge twice as much money for an 800 page novel as you would for a 400 page book.
In the USA, paperbacks are perfect-bound, but hardcovers are still frequently bound as groups of signatures (blocks of 16, 24, or 32 pages), which are stitched into a cloth binding. It's a higher quality technique, but it seems to be a bit less forgiving of large bundles of pages. In particular, I am told by my editors at more than one publisher that if the page count in a US hardcover goes over roughly 424 pages, this causes no end of problems: they have to outsource the binding to a bindery that uses a more expensive technique, disproportionately raising the production cost of the book. You can work around this to some extent by typesetting with smaller margins, less leading, and a smaller typeface ... but that'll only take you so far. My personal end-run on this was "Accelerando", a 145,000 word doorstep that was typeset in only 406 pages in the US hardcover, versus nearly 470 pages in the UK. (By way of comparison, the "Merchant Princes" books mostly run to 100,000 words and fit in 300 pages.)
The rules differ somewhat for A-list titles (if you can order a big print run, economies of scale ensue) and Epic Fantasy, where bloat has been de rigeur ever since "The Lord of the Rings". But in general there's a harsh brake on the length of hardback SF, and it's imposed by the step-up in binding costs at one end, and the booksellers at the other. One of the large chains did a study in the early 2000s and determined that for every $1 increment above a cover price of $24, a book's sales volume fell by roughly 25%; price it at $26 and it would sell only around 60% as many copies as at the $24 price point. (The price elasticity of demand for hardback fiction falls off a cliff above the $24 point; alas, it doesn't work the other way!) For this reason, they issued a diktat: no hardcover novels would be bought at an SRP over $24 unless they were from a really big-name author. And so the publishers were caught between readers who for three decades had been trained to expect ever-longer books, and a bookseller-imposed guillotine on prices.
The astute reader will have noticed that these constraints don't apply to two kinds of publishing operation: small presses doing specialist editions, and ebooks. The specialists can target a very specific market with a must-have product, albeit on a small scale; and ebooks can be any length the author's willing to write and the reader's willing to pay for.
In practice, however ...
Writing is work. I can produce around 250,000 words of paying fiction per year (300,000 in a good year; 100,000 in a really bad year). That can be packaged as either a single 700 page doorstep, or as two 300 page regular novels (and maybe a novella on the side). However, due to the price elasticity of demand my publishers can't make as much money from the 700-page doorstep as from the two 300-page regular-length novels. In fact, they probably can't make more than half as much money — books are sold as units, not by volume.
Now here's a confession. I originally planned my Merchant Princes books as a four book series. Book #1, written in 2002, ran to 196,000 words — a fine 600 page doorstep. Books 2 and 3 were going to be 800 pages each, in my imagination; and book 4 would be 500-600 pages. What can I say? I was inexperienced and naive in the ways of publishing, back in early 2002 when I wrote up the proposal.
One thing that you do when you're writing a doorstep-sized work of genre fiction is: you aim to keep it moving by delivering a partial climax every 250-400 pages that's about the size of the climax you'd put at the end of a 250-400 page novel. (Otherwise you risk boring your readers.) You then deliver a series-sized climax at the end of the book, but that's another matter.
By putting in these mid-book sub-climaxes, you keep the reader following your trail of breadcrumbs ... but you present your editor with a dilemma. Should they publish the novel as submitted, or take a hacksaw to it? $24 for 600 pages, or $48 for 600 pages — what would you do?
And so, the first book of the Merchant Princes, "A Family Trade", sprouted some hasty patchwork and a sequel, "The Hidden Family", which was originally the second half of the same book. And my editor's P&E calculations worked out alright and he bought a bunch more books — but laid down the law: "you've got 300 pages to work with per volume". Imagine my joy: I was 200 pages into the 800-page sequel when I heard this. And fans have been bending my ear about the lack of action followed by an annoying cliff-hanger ending in "The Clan Corporate" ever since, not realizing that it was actually the opening sequence and setup of a much longer book.
To add to the fun, when you take an 800 page book and split it into 300 page chunks, you do not get two 300 pages bits, or even three 300 page bits; each book has around 100 pages of scene-setting, recaps, and interweaving to make it work as a self-contained module. And stuff proliferates and gets out of hand, and you have to come up with sub-climaxes to make each book work satisfyingly as a book, and, and ... At the end of the day, the 800 page sequel turned into four books averaging 310 pages each; a 50% expansion! (Okay, so I found a few unexpected extras to stuff in there. But I wasn't planning on bloating it like that — it's a side-effect of trying to refactor a story to fit a form factor it wasn't designed for.)
If we were living in the brave new world of 100% electronic book sales, or selling to a British publisher, the book coming out this month would be book two of the Merchant Princes, and it would be about 900-1000 pages long. But we're not, and so it got squeezed.
Going forward, I speculate that if we make a successful transition to ebooks — that is: if ebooks become a major sales channel and authors are still writing professional quality work for money, and readers are finding some way to pay them — we may see a revival of other formats: novellas for one (they're undergoing a renaissance in SF publishing among the smaller publishers), the Dickensian serial for another, and the gigantic shoebox-sized monster for a third. The corsetting of the modern novel to fit between the tight constraints of binding costs and price elasticity of demand will be unstrung, or replaced by bras, or some other over-stressed metaphorical construct.