Back to: Brooklyn, Beer 2.0, this Thursday | Forward to: Typo hunt: Neptune's Brood

How I Stopped Worrying And Learned To Love Focus Groups

One of the Hot Topics in the big media / tech crossover world recently has been data-driven storytelling. Wired breathlessly reported that Hollywood gurus have reverse-engineered a "formula for success" from audience data. Netflix has revealed its ability to data-mine genres that they already know their audience will like.

The most common reaction to all this Big Data Story stuff is horror. Even less spontaneity. Even more focus group driven cookie-cutter movies. Artists straitjacketed by polling data.

Cats and dogs living together. It's the end of the world as we know it.

And yet, I feel fine.

Truth is, as a Web-based storyteller, I've been making use of data analysis, A/B testing, focus groups, and lots of other terribly un-artisty things for a long while now. I love 'em.

Data is good. Information is useful.

In fact, data saved my most recent film, Death Knight Love Story.

Key fact before I begin this tale: I suck at openings. I'm pretty darn good at entertaining endings, and once a story's rolling I can certainly keep it moving along, twist the plot, build up subtext and develop my characters. But opening a movie is something I always struggle with.

In the case of Death Knight Love Story, a dark fantasy love story loosely set in the world of the computer game World of Warcraft, we had a few choices. We could go with the classic fantasy Deep-Voiced Narrator opening. We could go for something fast and scrappy, Lock Stock meets Warcraft. Or we could go with a fast-paced in media res opening with our adventurer heroes battling evil right from the outset.

My strong belief was that we needed to go for the In Media Res option. Death Knight Love Story was targeted at a Web audience, and there's plenty of evidence that Web based viewers turn off after thirty seconds max if they're not hooked. We were targeting a weird mix of people who were and people who weren't familiar with the universe in which the film is set, and it seemed pretty reasonable to assume that WoW players would be bored to death if we started by explaining the world of the film.

So we animated and shot a fantastic high-energy opener. And then, thank goodness, Big Scary Data stepped in.

The first signs that not everything was well came when we screened the film to an informal focus group - something that a lot of movie-makers would argue was already polluting my artistic vision. The focus group sorta liked the film, but we noticed that just about everyone was confused by the plot on some level. And a few people said that it took them a couple of minutes into the movie before they really started to get into it.

We had an inkling that our introduction might not be ideal. Under normal filmmaking circumstances, we'd have debated it, I'd probably have made some minor changes but stuck with my "vision" of the opening, and we'd have released.

Instead, I channeled my inner Tim Feriss, and decided on a large-scale test using that most familiar of filmmaker's tools - Google Adwords.

We put a version of the first five minutes of the film online, engineered with analytics to show us exactly how far through viewers watched it. Then, we ran an advertising campaign targeted at phrases we figured would represent our core audience, pointed the ads to the film, and waited.

Some $200 or so of testing budget later (statistical significance being kind of important in this test), we had our result: the In Media Res intro suuuuuucked.

Almost no viewers were getting through the first minute and a half, let alone the whole of the first five minutes. For whatever reason, my intuition about my audience - founded on 15 years of filmmaking experience - was completely wrong.

One simple Adwords test had prevented me from totally tanking my film.

We subsequently tested some more openings, and finally, after some more Adwords, some anonymous user testing (via UserTesting.com, who are rather awesome), more focus groups, and various other techniques that would make any auteur filmmaker explode from spontaneous Muse-based rage, settled on the opening we have now - a classic narrator, albeit a female one. Turns out, cliches are cliches for a reason.

Thanks to that change, more people finish watching the entire 20 minutes of Death Knight Love Story part 1 than made it through the first minute and a half of my In Media Res version.

But didn't I end up ceding control of my story? Didn't I end up cheapening it and producing something that might have been what the audience wanted, but didn't reflect my inner views on the world?

Nope.

There are a lot of elements of Death Knight Love Story that I feel very strongly about. If Adwords had dictated that our heroine be thinner, less muscled, and possessed of revealing armour and gravity-defying boobs, for example, I would have gleefully told Adwords to fuck right off. (Actually, we didn't even do this test - because I'm 95% sure that a more fantasy-game conventional Miria would have tested better, and 100% sure that I don't give a shit.)

But the opening of the movie's just a story delivery system. We needed to have one, and it needed to get my audience into the interesting bits of the movie, but I had no particularly strong feelings about it artistically. So the best opening was the one that got the largest number of my viewers to watch the movie as a whole - the bits I did care about.

And that's what data is useful for. It'll tell you if your audience thinks your character's chin looks weird. (Fixed. Improved.). It'll tell you if you're better shunting the act break along one scene. (Fixed. Improved.)

Hell, it'll tell you if the front page of your movie's website would be better with or without a trailer on it.

We tested that: twice as many people finish watching Death Knight Love Story pt 1 if they're presented with a front page just containing the movie itself, not a trailer. You know what? If changing the look of my website doubles my audience, I'm 100% OK going with "soulless data" over auteur theory.

There are lots of ways in which blindly following test data could have ruined Death Knight Love Story. It could have told me to drop the slightly twisted romance / sexuality elements of the movie. It could have told me to make my characters more two-dimensional. It could have told me to use crappy Hollywood fight choreography or reverse the genders of our protagonists so that the our hero was male and our damsel in distress was female.

But those choices wouldn't have been the fault of the data. They would have been the fault of the person who made the choice to go with "popular" over "good".

Every successful author I know makes use of sophisticated audience analysis, based on a carefully selected, semi-representative focus group. They're called "beta readers", "editors", "writers' groups", and more colloquially, "any of my mates who owe me a favour large enough I can persuade them to read my unfinished novel". I've spent many hours participating in Charlie's natural language-based data-mining projects, otherwise known as "getting drunk and talking about his next novel".

And that approach seems to work.

So why should we fear a different way of collecting data, just because it involves less hangovers and more graphs?

18 Comments

1:

Try 'evidence' instead of 'data' - sounds more authoritative yet less intimidating in conversation.

2:

I've seen some critiques of focus groups, in that they don't always help create a good show or advertising scheme. In general, Big Data's impact lie in the increased ability of those with an agenda/narrative to plausibly boost a desired theory, as what Felix Salmon outlines here: http://blogs.reuters.com/felix-salmon/2014/02/02/viral-math/

When it comes down to it, I think that gossip between friends and those in the trade tends to provide the insights that usefully critiques the media product. Think of fashion designers talking about who's doing what in the latest show, etc, etc. It's very hard to know what people will put up with because it's a very zeitgeist sort of thing. People in a focus group may well not put up with In Media Res opening for a project, however, it may well be accepted among the broader public.

Of course, the other issue could be simply that the producers didn't have the needed storytelling chops to make that particular opening work. It's good to let that bad focus group score indicate your skill and direct that effort in another style. It's bad when when you think that it's the style that's at fault.

--shah8

3:

Nice post. Peer review works well in science. But it too is dynamic with information everywhere and every when. Sometimes sexy ideas are easiest to sell. A lot of science comes from negotiation, alignment, and a bit of luck. Art is really no different in this age of anyone's cat is more famous than an entire life's worth.

4:

Working in the "data" industry, so to speak, I have come to this conclusion (and I think it applies here too): Data is just data, it can be used wisely or it can be used foolishly, but having the data is nearly always better than not having it at all.

Or as my father often, and far more pithily, puts it when gathering tools for his latest project: You're better looking at it than looking for it.

5:

'If changing the look of my website doubles my audience, I'm 100% OK going with "soulless data" over auteur theory.'

Auteur theory isn't about independendently minded artists with a personal vision.

It's about professionals working in a business delivering a product.

Forget about directors a second and think about a makeup artist. A professional working on films as assigned. Now, if you can look at a film and identify the makeup artist without looking at the credits, what you have is an auteur.

6:

Hugh, did the "soulless data" make you think differently about how people experience media. That is, do you think there might be something misleading about people only looking at something on the web for thirty seconds?

7:

So tailoring your work to 90% instead of 10% of the audience is more successful. Who would have guessed? Of course, your point is (I assume) that your guess as to what the 90% desire or tolerate did not fully match reality. Probably because they were dumber than you assumed.

8:

Interesting! The Marx Brothers were doing this sort of thing years ago with their films.

http://www.sltrib.com/sltrib/news/56148615-78/marx-groucho-opera-brothers.html.csp

"A Night at the Opera," written by James Kevin McGuinness, was adapted for film by George S. Kaufman and Morrie Ryskind. To ensure the movie’s "laugh-worthiness" before committing any schtick to celluloid, Thalberg sent the script for a trial run in Seattle, Portland, San Francisco and Salt Lake City.

"We are kicking ourselves we didn’t think of this before," Groucho said in the April 13 Salt Lake Tribune. "A successful comedy depends almost entirely upon audience reaction, and if anyone tells you he can sit in Hollywood and judge in advance how much Salt Lake or any other city is going to laugh at any given ‘gag’ — don’t hesitate, put in a hurry call for the psychopathic ward. We expect our greatest help from Salt Lake, for it is our first stop … and we will get a definite idea of the script’s value."

Ryskind attended the weeklong performances at the Orpheum’s 1,160-seat vaudeville house. The Tribune ran ads: "The Marx Bros., on the Stage, in Person." Tickets for the matinee sold for 40 cents, 55 cents for evenings and kids got in for a dime. The theater was packed.

Ryskind recorded audiences’ reactions, timed laughs, analyzed groans and reworked the script for the next day’s show.

That April 17, The Tribune reported failed gags were "‘blue-penciled’ so that by the time the organization has been around the four-city circuit, the writers and producers will know pretty well just what and what not to include in the final script for a bang-up picture."

The film was a $3 million hit — and Groucho’s favorite.

There's a big difference between testing your material and selling out. I wish more movies scripts were tested artistically instead of as marketing exercises.

9:

"Probably because they were dumber than you assumed."

Why is that the only conclusion you would leap to?

10:

Why? Because we tend to mix with people like ourselves and Hugh seems like a smart guy. Surely if our friends like it everyone will? Except that Hugh's friends are probably not representative of the population.

11:

As long as the human race keeps reproducing and growing up in the same way, that is, infancy, childhood, adolescence, etc. -- which stories sell will change somewhat because each generation will look for the story that best defines them. The most difficult part will be guessing what teens will be rebelling against 30-50 years down the road.

BTW - I keep mentioning teens because this has been and remains a critical age for establishing brand/personal preferences.

12:

Some really interesting comments here - thanks!

I have just returned from a business trip of the awake-at-4:15am-three-countries-in-two-days variety, but will respond in more detail tomorrow!

@dirk.bruere - Indeed - my guess about the 90% was dead wrong, and arguably in a way that shows they're smarter than I expected. Turns out that reports of the "YouTube generation" only having an attention span of 20 seconds are dead wrong, at least for drama - given a modicum of storytelling people are happy to let a plot develop for at least a few minutes before turning off.

@gmuir77 - wow, I did not know that at all! There is nothing new under the sun...

13:

"But the opening of the movie's just a story delivery system", yeah. I like that. Kind of like, do you really care how the tail of the bacteriophage is constructed, when the whole point is to inject the DNA? It's taking over the cellular machinery afterwards where everything happens.

14:

The Marx Brothers road-trips produced a guaranteed script down to the intervals -- you can watch them doing bits of business, Groucho waggling his eyebrows with a sideways look, Harpo pulling a long face, allowing enough time for the laughter to die down before going into the setup for the next gag.

They took this script back to the studio, ran through it again on set to allow the film crew to stop laughing so hard they shook the cameras... and then on the next take they ad-libbed AND IT STILL WORKED. That's genius.

15:

@shah8 - I'm definitely not suggesting that an In Media Res opening can never work - that would be somewhat foolish. However, the data I gained from the general public, for values of general public meaning "people likely to watch this movie" (or at least a few thousand of them selected on the basis of "Who's Googling right now?") implied that they didn't get on with the In Media Res opening we were testing. Whether that's a general issue with the opening style or just an issue with my skill in executing it was somewhat beside the point, as I wasn't about to replace myself as writer. :)

@justin.boden - It made me question a lot of the conventional wisdom around making movies for the Web, certainly. Currently the craft of making streaming entertainment is in its infancy, and the craft of making streaming drama for YouTube (as opposed to curated platforms like Netflix) is even more so. I think a lot of people, including myself, are prone to extrapolating from data taken from video as a whole, and failing to remember that audience reactions will be vastly different between an amusing video of a cat and a 20-minute dramatic fiction.

16:

Thanks for your response! I know a friend who was quite indignant at the statistic that people only spend three seconds (or whatever it is) looking at a painting before moving on. I liked her so of course I told her about the times I spent maybe as much as seventy seconds, even entire minutes, looking at a piece of artwork. But that hardly tells it, right? How many pieces did I barely glance at? Even at a Saatchi exhibit I'm going to walk a few paintings. Likewise, how many books have I only read twenty pages into? The devil is not the intellect of the audience, perhaps, when we consider that even us highly intellectual individuals (as we no doubt appraise ourselves; as dirk certainly does, finding himself and all his peers so easily in the upper tenth percentile – yes, that was a snark; no, I'm not a little bit sorry) don't always stare at paintings for whole minutes, or read novels to their end, or stay awake during every movie, or indulge each second of a YouTube video. Perhaps there is something more in our brain's filtering abilities to go, "You might like this," or, "No, not even on a rainy day." And, knowing that, we can actually expect a bit of forgiveness from the audience if we're upfront about the content of the video. (If I already know I'm in for a long ride before I watch the video then I'm more prepared to invest my attention span.) Yes?

17:

Some really great studio albums came from not only rehearsing, but testing against reactions of others. If memory serves right, Dark Side of the Moon incorporated this technique.

18:

Interesting.

I am, of course, guilty of this (for values of "guilty of this" anyway). Near my Mother's house, there is a civic gallery which contains, amongst other things, a collection of portraits of Victorian gentlemen, some "Arcadian landscapes", and Dali's "Christ of St John of the Cross". Not being interested in the brushwork of Victorian portrait painters working in oils I tend to virtually ignore the portraits, stop and physically look at the landscapes, and spend several minutes on the Dali.

What this actually proves is that, for a portrait to interest me, it probably needs to be of someone I recognise rather than some rich businessman or landowner who died maybe 70 years before I was born, but that similar contraints don't actually apply to other forms of paintings or, to the same extent, to newer works.

Perhaps you could put this argument about taking the subjects of paintings into account to your friend?

Specials

About this Entry

This page contains a single entry by Hugh Hancock published on February 4, 2014 7:05 PM.

Brooklyn, Beer 2.0, this Thursday was the previous entry in this blog.

Typo hunt: Neptune's Brood is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Search this blog

Propaganda