Writing Tool

I speak occasionally on the Cambridge NaNo Discord about my writing tools, since they are increasingly unconventional.

Scrivener or Not

I was a huge Scrivener fan for quite some time, and as recently as March I was giving tips on how I like to configure it. It’s excellent for organising your work into multiple files and managing the structure, keeping notes alongside the manuscript, and has a powerful compiler for output.

But as I came closer to needing production-quality output for onward processing, I found myself increasingly untrusting of that compiler. It’s got so many options that I wish it had a much more robust notion of loading, saving and editing configurations.

I decided that I needed something I had more control over, a tool that covered the ‘manuscript to ebook’ pipeline in as few steps as possible (because otherwise I’m going to get something wrong somewhere), and while I was at it, something that suited the way I’d like to write as much as possible.

Philosophy

The goals of the tool have evolved a little as I’ve worked on it, but mostly represent the way I want to work:

Style

At secondary school I won a prize (for achievement in mathematics, or maybe for achievements at the local non-prestigious FE college, or something), and they gave me a book token. I bought a guide to HTML4, and it already had <font> deprecated. That was in about 1998; MS Word apparently already worked in styles before then, but even now it still promotes making individual changes to the formatting of individual things.

Don’t do it. In Word, give everything a style. When you’re tempted to apply some individual formatting you’re probably actually proposing a new style. (There are a few issues with spacing around lists and tables (among others) but making workarounds for bad tech should be the exception rather than the rule.) In HTML, use CSS to give your entities sensible defaults based on their role in the document, and put classes on the ones that need to deviate from it. Avoid putting style on individual elements: any item that needs style probably deserves a class.

Although there are exceptions (and I’m not writing those), a novel typically doesn’t need much style (and in traditional printing doesn’t support much). You need paragraphs, a distinct style for first paragraphs, something to break scenes, and maybe chapter/part headings. Inline you should probably only be using italics (although I think bold is becoming more common).

Scrivener’s RTF is necessary for some applications, but is overkill and a potential additional point of failure for mine. You can tell the compiler to ignore formatting in individual documents (and I think it leaves in your emphasis) but I don’t want to have to worry about it.

Mark-up and Configuration

I love mark-up languages. If I did more work with ‘proper’ programming languages I would love makefiles. The idea is that everything lives in the text files, where I can easily fix any problems. Scrivener’s compiler settings are pretty good, but a bit too black-boxy for me to really trust them with production work (especially since, as above, they do much more than I need).

Keeping the manuscript (including metadata) in mark-up gives me excellent control over what information I store and where, and since I’m also the tool developer I also have that control over what happens to it. Since it’s also plain text I can edit it anywhere and use any version control. Scrivener projects go OK in git, until you want to diff something.

Inline Notes

These weren’t originally an objective, but have emerged as a significant strength as I’ve worked with it. In Scrivener you can put comments into a file, or in the document notes, but you might have a hard time finding them again (they show up in searches, provided you remember what to search for). You can put them in their own documents, but then you need to leave the manuscript (or at least switch one of your panes to the notes file). Even then I can’t always remember where I put everything.

I had to overcome a number of challenges to make inline notes powerful enough, but I’ve now reached the point where all notes are inline (if only because it doesn’t matter where you put them.

Implementation

Since fundamentally what I’ve got is a mark-up language and processor, I wrote it in Python. I feed my project file to a command line script, which spits out a folder full of HTML files that are almost ebooks, and a report of analysis and notes.

In future I could make an editor that updates the report automatically and displays it beside the manuscript, but meh. I can put a browser next to the text editor.

The only thing I miss is that in Scrivener it’s very easy to reorder scenes. In my manuscript it’s easy to select a whole scene, and hence to move it manually, but it’s no longer a GUI task. I’m also toying with the idea of adding an ‘include’ command to spread a project across files, but without a GUI for working with multiple files (and ‘Scrivenings’ are very good) it’s not worth it.

I solved ‘jumping to a place in the file’ in a different way, and I’m still really pleased with myself for it.

Capabilities

Here are a few of the things it does.

Outliner

My first priority was to get parity with the Scrivener features I like, especially the outliner (as I showed last time).

I’ve blurred some volume, chapter and scene names (which are spoilers). The part names are from Film Crit Hulk’s description of Shakespeare’s use of traditional dramatic structure.

It does everything I wanted from Scrivener’s outline and a couple more. The ‘PoV’ field shows the point of view, which is what I’d use the label for in Scrivener; rather than use label colour on the row as Scrivener could, I’ve left a class on the cell and declared a style for it.

Bold names are for scenes (or ancestors of scenes) marked ‘key’, for the sake of Stuart Horwitz’s Blueprint your Bestseller. I still have synopses available on any block, but haven’t bothered using them since I put sensible names on each one.

The mark-up for opening blocks looks like this:

:part Act 4: The Spiral
:chapter Overlooking the lab

:syn <snip>

:scene Overlooking the lab
:pov Gladwyn
:status good

The colon formatting is partly a throwback to a previous, similar project, but suits me more than trying to make it (e.g.) valid markdown and remember which heading level corresponds to what. In that project a blank line was a scene break, but it’s much safer to ignore blank lines (and in any case I’m now trying to name all scenes).

I have a config parameter for saying which columns I want in the outliner, but all my current projects use the default. Funny that.

Line Links

In the outliner (and everywhere else in the report), each hyperlink points to the correct line of the manuscript. In this way I can jump not only to the start of any volume/part/chapter/scene, but also to any note, any occurrence of a series, or anything else. All without having to influence where those notes appear in the report.

It’s one of the most powerful features and the most technically challenging. It’s also the main reason I probably won’t bothered trying to redistribute the tool, because it requires:

• A text editor that accepts line numbers as command line argument (such as Notepad++).
• A URI handler. In Windows these are written straight into the registry(!), and the syntax for them is very odd.
• A script to parse the output from the URI handler, since it has a stupid format.

The second could be added by an installer, I suppose, and the third is just a text file, but even a person interested in the tool might not want to be using Notepad++ for it.

Output

I say the outliner was the main feature parity from Scrivener; that’s if you don’t count compiling and exporting the manuscript. Most of it is simple: lines of text in the manuscript become paragraphs in the HTML (the first one after a break gets a particular class); HTML in the manuscript is passed through (which I intended for <em>, but in principle lets me do all kind of stuff, in defiance of my style philosophy above).

Headings are more complicated, and so far I’ve coded more cases than I currently use. This is in the preamble for my current project:

:set part_mute True
:set chapter_override Chapter $n
:set chapter_tag h2

It says “don’t print anything for the start of a part”, “override chapter names with generic autonumbered headings” (the default would be to use the names) and “chapters are second-level headings”.

The *_mute, *_override, *_tag properties exist for all block levels. For example scene_override defaults to <hr/>, so scene names are never shown by default (it turns out that doesn’t work well with Draft2Digital’s ebook formatter, but it’s a one-line fix once I know what to change it to).

It also breaks volumes properly, starting a new output file for each.

Replacements

I put in some text replacements. This test passage appears at the end of my current project, and tries to describe the input for each substitution:

This bit was fun to code, sort of. I borrowed some smart-quote code from the internet, and had to fix various problems with it. It works by identifying and substituting all the right-hand quotes, then treating all the others as opening quotes, which is fine until you’re nesting, at which point you need to look more carefully at the nearest non-quote characters to work out which side you’re at.

The bug in the first line (note first-paragraph style without the indent, although it looks stupid among single-line paragraphs) is quite interesting; notice on the middle quote nesting test how it’s only a problem with single quotes inside double. This is because I did eventually fix the ‘other quote before makes this a closing quote’ bug for both, but I’m since using HTML entities that single opening quote on the first line doesn’t have a left-double-quote to its left, it actually has ” to its left, which is valid inside a quote.

I will try to fix it eventually, probably by replacing double quotes with a completely unused Unicode character so that the single quote logic can definitely spot them, then replacing the unused Unicode character with the HTML entity at the end.

It also can’t handle ’twas, and I don’t think it ever will. I never thought would never be an issue until I had to manually put in the correct quote entity for ’shroom.

Series

I mentioned Stuart Horwitz’s Blueprint your Bestseller above. I recommend it, if only because his notion of ‘series’ is helpful for tracking repetition and variation in motifs and ideas beyond just character subplots. I’ve adopted his ‘series grid’ – a table showing the iterations of each series by scene of occurrence – and while there’s plenty of room for improvement, it currently gives a helpful (if a little dense) overview:

I’ve blurred the scene and series names again. I’m tracking far more series (columns) than Horwitz would recommend (and below the table is a list of more that I identified but didn’t track, because I’ve marked them as hidden rather than deleting their metadata). The full table is massive: there are scene iterations in many or most of the ~150 scenes.

The mark-up for a series iteration looks like this;

:series assassins 0 "Mildgyth, are we assassins?"

After the command are two arguments. ‘assassins’ is the label for the series. The 0 (or other integer) is the change made to the ‘value’ of the series, so that I can keep track of its intensity (and maybe plot ‘series arc’ graphs later). I can also set some series metadata such as a display name, description etc.

The value at each iteration is the number in the grid. The little green marks on cells show that there’s a tooltip; in the case of a series it’s the metadata, for iterations it’s the actual text note (“Mildgyth, are we assassins?”).

The table shows at a glance how many iterations a series has, how many series are iterated in a scene (having lots is essentially the definition of ‘key scene’). I can see how many series there are, jump to any scene or any iteration, and so on.

There’s lots more I could do (and it desperately needs a ‘volume’ filter before I start identifying series in the next volume), but I’m pleased so far.

Ordering

Series (and characters; in principle everything) are left in the order they are first mentioned in the manuscript, including metadata. I’ve found this more convenient than giving them an extra parameter to order by: the series for the table above are named and described in priority order in a big preamble rather than being in the table in appearance order.

Characters

I make a character note like this:

"Ozgur, it's been a while. Since you last stopped to talk to me, that is."
"Yes, Hjalmar, a long time."
:char Hjalmar A slave trader.
Hjalmar remained in shadow, a hearty voice ringing from a large silhouette. "Need a certain someone?"

As you can see, I didn’t have to leave the document to register Hjalmar as a character (and hence record the spelling of his name, etc.) and make a quick note. Quite often I will have some description in a paragraph, then paste some of it into a character note on the line before (I toyed with telling the note to capture the next line, but so far it’s been easier and safer to have lines be independent, and I rarely want the whole of the next line).

In the report, the first note for each character is gathered into a master character list, then all the notes are grouped by characters.

Other Features & Next Steps

The report has a table of contents with all the bookmarks linked, because you may as well. In-keeping with the simplicity principle it’s generated by JQuery so I don’t need to clutter up the HTML of the report (which is already pretty complicated).

Other kinds of notes (locations, history, timeline) go into bullet lists grouped by the subheading (although I’ve made some aliases, e.g. :todo = :note todo). Blocks (especially scenes, maybe chapters) could have timestamps; for future works I might make a report view that lists scenes in chronological order, but in the current project they always are.

I’m considering a kind of ‘style’ notes (perhaps with subcategories) that output to a separate file that I can give to an editor. Currently I have some ad-hoc ‘note caps’-type notes, and it might help if my draft for edit comes with those in a stylesheet to explain why I made the choices I did. (Or not; perhaps making their own stylesheet is a necessary part of the process. I’ll ask.)

More filtering on the scene grid would be nice. It currently filters out (but can show) scenes with no series iterations. I need to be able to filter by volume, which shouldn’t be too hard. What I’d like is to be able to hide some series (but I need to leave a way of putting them back, ideally without having to undo all the hides at once) then update the ‘iterations per scene’ and the empty hiding of scenes to be current with only shown series.

Conclusion

So there you go, that’s what it does, with some hints about how. I hope it makes sense; you probably want to keep using a WYSIWYG tool, but I’m happy with the feeling that I can increasingly just write (or edit) in one place without jumping around (except that when I need to, I’m never far from a link to the right line), and that was worth the (considerable) work I’ve put into the tool (and will continue to).

I have no plans to try and distribute the tool for money (I can’t imagine there’s a market), but if you are also a technically-minded author and the philosophy appeals to you, let me know and I can show you more.