Back to: Spammers ... | Forward to: The Blitz spirit

Writing a novel in Scrivener: lessons learned

Some of you probably know about Scrivener, the writer's tool from Literature and Latte. (If you don't, the short explanation is that it isn't a word processor, it's an integrated development environment for books. It's cross-platform (although initially developed for Mac OS X —versions for Windows and Linux are available, and it's being ported to iOS and Android), modestly priced, and has more features than you can wave a bundle of sticks at, mostly oriented around managing, tagging, editing, and reorganizing collections of information including rich text files.) I've used it before on several novels, notably ones where the plot got so gnarly and tangled up that I badly needed a tool for refactoring plot strands, but the novel I've finished, "Neptune's Brood", is the first one that was written from start to finish in Scrivener, because I have a long-standing prejudice against entrusting all my data to a proprietary application, however good it might be. That Scrivener was good enough to drag me reluctantly in is probably newsworthy in and of itself.

First of all, I should note what Scrivener can't do for an author.

Many publishers these days have moved to electronic document workflow during production. Manuscripts are submitted in a standard format (they've settled on the hideous, proprietary, obsolete binary format of the Microsoft Word 97-2003 .doc file, simply because that's what most people use). Copy edits are applied to the .doc file using Word's change tracking feature with annotations in place of post-it notes. If you want to process copy edits in this brave new world, you need a word processor, because Scrivener's view of a book is so radically different from Microsoft Word's single monolithic file that there's no way to reconcile the two and add Word-style change tracking to Scrivener. Luckily LibreOffice, a free fork of OpenOffice, is (a) free, (b) under active development again, and (c) can chow down on basic Word documents with change tracking and notes without throwing up most of the time. (The copy-edited manuscript of a novel does not contain Word BASIC macros, complex tables, or illustrations: it's just a stream of text with paragraph styles.)

So Scrivener stops supporting publisher workflow once you have submitted the manuscript. And arguably it stops an hour before then, because figuring out how to modify the output format generated by the Scrivener "Compile" menu option is a black art ... I found it easier to slurp the resulting Word document into LibreOffice for final tidying up and reformatting before I submitted it. Scrivener doesn't support Word's paragraph style mechanism as far as I can tell; it simply emits styled text. So it's output isn't a direct product you can feed into an unattended turnkey pre-press package: you'll still have to pay someone to drive InDesign for you.

Other weaknesses: Scrivener 2.3 on OSX is a big program. There's an introductory tutorial project, and a video. And then there's a 300+ page manual in PDF. Why PDF, when Scrivener emits some of the cleanest epub files I've ever seen? And why doesn't it work with the OSX built-in help system? Who knows. Let's just say that learning Scrivener's ins and outs is an ongoing task. (For example, I was most of the way through this novel, the (counts) sixth that I've used Scrivener for to some extent, when I discovered that the Edit->Writing Tools submenu now contains a character name generator, as well as the obvious stuff like the spelling and grammar checker controls.)

General usage:

In Scrivener, if you're writing a book you start by creating a new project, just as you would if you were starting to write a program using an IDE like XCode. The project is a hierarchical outline-based container for your research notes (including PDFs and images and web pages, which you can slurp in as files or direct from the web by entering URLs) and the small files, or "scrivenings", that constitute the work in progress. Scrivenings are basically RTF files (more accurately, Apple's RTFD—a derivative format that allows the inclusion of additional sub-elements like images), or folders containing scrivenings. A chapter is basically a folder, and the scenes in the chapter are scrivenings, and you get a collapsible, hierarchical view. You also get the ability to edit scrivenings, either individually, or by multi-selecting a bunch of them and seeing them as a continuous scroll of text: most convenient if you want to edit scenes 1, 2, 4, 6, and 8 in a chapter but not 3, 5, and 7, for example.

That's treating it as a scene-based word processor. Scrivener provides other tools for looking at your data. There's a cork-board, in which you see each scrivening as an index card, and in which metadata (notes, defined keywords, all sorts of stuff) is transparently visible. Or you can display it as an outline in a classical outline processor mode. The general effect is to make it easy to search, organize, and see views of your data, and trivially easy to restructure a hierarchical document as long as you've broken it down properly into chapters containing sub-documents.

Other tools: in some ways the most useful feature it provides for a jobbing author is the Project->Show Project Targets option. You get a floating window with progress bars (updated in real time) containing (a) your progress towards the target word count for the entire document, and (b) your progress towards your target word count for the day. As motivational goads go, this one is invaluable when you're slogging through the difficult middle of a book, and the ending seems as far away as the beginning. (Seriously, measuring your progress is one of the under-stated but vital tasks associated with any job: good luck getting Microsoft Word to help you with that.) Again: Scrivener projects can get quite large, and are structured internally as a folder hierarchy. Scrivener has an option to package them up as a zip archive (which can be emailed around, or re-imported later), and also to back them up to a private folder. Mine is linked to my (private) Dropbox account, for obvious reasons: it gives me version-controlled offsite backups. It's not quite git or subversion, but if you want those, there's a "sync with external folder" option which looks like, yes, you could use it to sync with a heavyweight configuration management system. (Note: in my opinion, novels don't need heavyweight version control—they virtually never fork and you seldom have as many as two authors. Straight linear versioning is fine for 95% of cases.)

Stuff I don't use: there's a full-screen mode for folks who like to write without distractions. They are not me, and I just don't use it. The keyword tagging ... I can see types of work it would be useful for, but it's less obviously useful for fiction. Being able to define the status of a scrivening as planned, first-draft, or final is obviously useful to some people: but that's not how I work.

Finally, there's the question of how you get your data out of the application. You can do it piecemeal: Scrivener is happy to export individual scrivenings or files. Or you can do it wholesale, via the File->Compile menu. Which takes the assembled scrivenings, filters them in accordance with whatever crazy criteria you set ("exclude odd-numbered scrivenings in even-numbered chapters" looks like it ought to be possible), applies transformations to them (Scrivener understands MultiMarkDown, so if the idea of proprietary RTF brings you out in cooties you can write in MMD text files), and generates a finished document in one of the target output formats—Word .doc is one, but it can also produce RTF, PDF, ODT, Final Draft, and ebook formats—epub or Mobi for Kindle. What's more, if you used MultiMarkDown it can emit LaTeX; given its footnote and endnote support, it may be a very useful tool for preparing academic papers that need a final production pass in LaTeX (a horrible format to work with by hand, in my opinion).

This isn't a formal review: it's just a comment to the effect that Scrivener works pretty much from the moment of conception to the hour before final submission of a finished manuscript. It doesn't completely replace the word processor in my workflow, but it relegates it to a markup and proofing tool rather than being a central element of the process of creating a book. And that's about as major a change as the author's job has undergone since WYSIWYG word processing came along in the late 80s (actually the late 70s if you were a researcher at Xerox PARC, but the rest of us had to wait). My suspicion is that if this sort of tool spreads, the long-term result may be better structured novels with fewer dangling plot threads and internal inconsistencies. But time will tell.

PS: Comments are still switched off due to spammers. If you want to discuss it, the Google Groups Antipope storm refuge is open for new members and I'll start a topic thread there.

26 Comments

1:

"Why PDF, when Scrivener emits some of the cleanest epub files I've ever seen?"

If I were to hazard a guess, I'd say it was because everyone and their dog already has a PDF reader installed on their desktop, whereas most people seem not to read ebooks at their desktop.

2:

What's more, if you used MultiMarkDown it can emit LaTeX; given its footnote and endnote support, it may be a very useful tool for preparing academic papers that need a final production pass in LaTeX (a horrible format to work with by hand, in my opinion).

I'm not quite sure what you mean here - is it that LaTeX is a horrible format?

Once you get past the learning curve, it's pretty much like Word Perfect, or html markup code. And while actually inserting the codes for integrals, fractions and whatnot can look pretty ugly at first, after a while it just sorta becomes, um, transparent, I guess. When I see something like $\sum\limits_{j=n}^{2n-1}(2j+1)=3n^2$, I just "see" it as it appears in a pdf output file.

I'm only harping on this because I had to learn LaTeX in grad school at the ripe old age of 45; at the time it was a nightmare and I was wishing mightily for a text editor that was a little more friendly to noobs.

3:

Many thanks Charlie.

Seems like Literature and Latte could do with putting some focus on the end of the workflow - editing, revision, etc., particularly if javascript enabled eBooks find favour. Overloading the comments/changes elements such paid reviewers could still use Word, and that a user could pipe back errors they spot in an epub, would simplify some of what you are currently doing.

4:

Looking at it cynically, the answer is "no": far more people write books than succeed in convincing a publisher to buy them and subject them to the post-acquisition editing/processing steps. So support for editing/revision features in Scrivener would only be attractive to a minority of customers.

(I've also discussed it in email with Keith, who has looked at the problem and concluded that it's very hard. The trouble is, Scrivener's internal model of a document -- as a folder hierarchy containing lots of little files -- is very different from the monolithic Word file that editors work on, and pulling a Word file back into Scrivener as a basis for diff/merge ops would require some extremely fancy section-level detection and matching, at a minimum. Integrating Word change tracking into Scrivener is therefore not practical.)

5:

It's not just novel authors though. The real money is in business reports - and they certainly need the review and revision stage to work well.

And the reason for mentioning about overloading the track changes functionality is to tag the change with its location in the original document - bypassing the MS mess (eg delete the next five characters from position character no. 48261).

Although, we could always make the editing/review stage a web/cloud based activity ....

6:

Scrivener is pretty much useless for writing business reports. It's just not designed for that kind of project. It's a novel-writing tool, first and foremost. Why would the author of a highly successful niche product want to tune their product to do something entirely different?

7:

I'm (pleasantly) surprised to hear that LibreOffice can mostly support track changes and comments; the last time I looked, consensus seemed to be that those were the lock-in feature to Word for many markets.

In theory, Scrivener (and everything else) ought to be able to decode the Word 2007 XML formats (i.e. .docx), but for some reason the various industries are all very reluctant to move past Word 97. Making Scrivener work with Word 97 .doc files is basically the same problem as VCS magic-merge, with the added fun of dealing with Microsoft standards/code. It should be possible in theory, but I sure wouldn't want to do it.

I do a bunch of short-story-sized writing of interconnected projects, generally 10k-20k words, with in-lined images and tables. I frequently look at Scrivener as a way to escape the morass that is Word, but track changes and word comments are nearly impossible to remove from the process, so I've always shied away. Looking at the current version, I see that they've added SimpleNote integration, which is very tempting - the ability to get useful work done with the ipad instead of the mbp is pretty tempting. Have you tried this yet?

8:

It's a damn good comic writing tool also.

9:

A good overview of Scrivener but I think you blew it with your comment that "Scrivener is pretty much useless for writing business reports".

I use Scrivener for business writing all the time. The ability to treat ideas as individual chunks of text to be moved and recombined at will is just as useful for business reports as it is for fiction writing. Being able to write "inside out" by starting with a small detail and expanding from there is a great way to get text flowing. The Research folders are incredibly useful for capturing and organizing source materials, related corporate documents, etc.. There are even features that make it easy to transcribe recorded interview notes.

Scrivener's shortcomings in the business context show up at the same place as you describe in your publishing workflow -- when its time to get the draft into the hands of collaborators for review and additional input.

10:

I personally get sick of the track comments. Back in the day, I actually saw a bug that filled up the entire hard drive with an infinite loop, due to a mess with the comments. I'm happier when people change font colors or do other simple annotations, especially if people are working across platforms.

The simplest way I've seen to deal with comments is to enable the line numbering function, and then to simply comment on lines in a separate file.

I'm still dipping my toe in Scrivener, and I do want to see how it deals with something other than a novel, too.

The counter-argument to using Scrivener on business reports is that it's not particularly designed for them. That's why you're supposed to buy the uber-expensive Windows Office, instead of a $40 novel-writing application. You'll save money on the increased functionality, right?

11:
You'll save money on the increased functionality, right?
You will if you make your saving roll for sanity, but the odds are against you. Paragraph formatting was so buggy and difficult to build a mental model of in early versions of Word that after 2000 I just gave up on it completely.
12:

The windows version of Scrivener is much less powerful, but even so beats the pants of MS Word etc.

The dark art of sorting out the output is indeed dark, but once done need not be done again.

13:

My wife just finished her first novel in Scrivener. She loves it. But her editor insisted on edits in MS Word. It nearly drove her crazy. She'll try the edits in Libre Office next time, but would happily pay more for Scrivener if it could deal with comments and change tracking better.

14:

"And why doesn't it work with the OSX built-in help system?"

Perhaps because the developer realizes Apple's help system is one of the worst ever deployed on a computer platform. Search is dependent upon how the author has indexed the help, so you cannot depend upon finding something you're searching for - even if it is *in* the help.

PDF is much more useful and versatile.

-ccs

15:

@13:
> But her editor insisted on edits in MS Word.

The first book I sold, the publisher insisted that I send it to them printed on paper. They then paid someone to key it all back in to their compositor by hand. I guess I should have been grateful they didn't require I write it on parchment with a quill pen. For extra points, I had to airfreight the manuscript to their London group, which cost about US$100.

By the time I sold the second book, the publisher (a different one) would accept a computer file. And not only that, they were savvy enough that they'd take it online instead of mailed on a diskette! Fortunately I had a Compuserve account so I could talk to them, those being the days of the big time-share systems. Like AOLers later, they thought CIS was "online." They insisted I send it to them all preformatted with WordPerfect, and were astonished and confused when I told them I didn't own a copy, nor was I willing to go out and pay several hundred dollars for one. After several days of negotiation, they agreed to take the ordinary 80-column ASCII files my text editor put out.

For all I know, they printed it out and hired someone to key it all in to WordPerfect...

I ran into something similar selling to a couple of magazines, including PC Tech Journal, which had an absolute policy of only accepting submissions in the current WordPerfect format, no exceptions. I eventually sold most of that stuff to Computer Shopper (not the same one Charlie wrote for) during the Stan Veit era. Computer Shopper's editors were apparently computer geeks instead of journalism grads, and they'd probably have cheerfully accepted files on hard-sector floppies in EBCDIC in some arcane word processor format...

Word, though... blech!

16:

Do any publishers make available a sample file of an edited document?

17:

Not to the best of my knowledge. To see what one looks like you need to look for specialist sources that discuss the business of writing.

18:

Thanks, Charlie. Maybe not a problem, but it could make it hard for anybody wanting to make tools for writers to use.

I don't know what Open/Libre Office can cope with. Without reliable samples, how can any usefully test?

19:

My brother wrote his dissertation in Scrivener, and is the one who recommended it to me. So it's clearly useful for more than just novels.

One other thing worth mentioning is that Scrivener has a very reasonable trial mechanism. You get to try it (at full functionality) for a total of 30 non-consecutive days. So if you don't use it for a month in the middle of your trial, you still have trial time remaining when you come back to it. I think that's rather nice of them.

20:

Actually, WYSIWYG word processing was available, at the very latest, by 1984 with the arrival of the Macintosh, if not earlier with some other computer(s) of which I’m not aware. It certainly was not “the late ’80’s.”

21:

The word processing was available, but it wasn't until a bit later that laser printers came out and made desktop publishing affordable for small businesses, and shortly after that for everyone else.

So, yes, late 80s.

22:

Nice feedback. I don't use Scrivener, but many of the features listed here are also the ones Emacs Org-mode (http://orgmode.org) is focusing on. I wonder if there are Org-mode users outside of the geek/developers community. Thanks!

23:

They also do a special extended trial for NaNoWriMo in November. And I think I ought to get my "winner's" cut-price deal. You get 20% off for just trying, 50% for hitting the magic 50,000 words.

Charlie's style of working doesn't fit well with NaNoWriMo, but what I've picked up from trying amounts to two main things.

1: I know I can manage to organise things to have the time to write a lot of text.

2: Despite the efforts of my schoolteachers to constrain and denigrate my abilities—the essay for exams tends to kill talent (what can you write in a half-hour?)—I know I can produce a coherent 50,000 word story.

Of course, it doesn't have to be good writing, but knowing you can handle something that size is a considerable boost. It makes a hundred thousand words a less intimidating target.

Oh, 20% VAT... Ouch.

24:

Sounds a bit like Final Draft, the defacto standard for doing screenplays. While you can do screenplays in Word or other word processors, Final Draft has all the fiddly "rules" for screenplays built in, like what things need to be in all caps, properly annotating dialog broken across page boundaries, etc. And it has a really nice Courier font built in. (Screenplays have to be in Courier. Gotta emulate a Remington typewriters.)

One advantage for screenwriters is that while initial submissions are done on paper, you can turn in the electronic files after purchase, because everyone in the industry can handle them.

25:

Curious: how big of a factor is word count in writing your novel? I mean, do you start writing and keep writing until it's done? Or do you track your word count as you go along and trim/inflate pieces accordingly?

26:

@21: The word processing was available [by 1984], but it wasn't until a bit later that laser printers came out and made desktop publishing affordable for small businesses, and shortly after that for everyone else.

So, yes, late 80s.

I first met a WYSIWYG word processor around 1980 IIRC. Dedicated hardware that only did one thing, 8" floppies for storage, green screen, daisy wheel printer. Cost an arm, leg, kidney and first born. We didn't buy it, thank ${deity_of_choice}.

Laser printers weren't generally available until the late 80s but existed before that. <smug>I've got a draft copy of the blue Smalltalk 80 book straight off Adele Goldberg's laser printer from 1982, WYSIWYG formatted on her D machine.</smug>

Specials

About this Entry

This page contains a single entry by Charlie Stross published on July 11, 2012 1:13 PM.

Spammers ... was the previous entry in this blog.

The Blitz spirit is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Search this blog

Propaganda