Back to: Typo hunt: Neptune's Brood | Forward to: What's The Future For Virtual Reality Movies?

Minecraft, The Sims, And The Future Of Filmmaking

What does the future hold for movies?

That's a topic that has eaten more column inches than Scottish Independence, the Kardashians and Bitcoin combined. But whenever I hear it, I can't help but think that the writer is asking the wrong question.

A better, more illuminating question is "What does the future hold for movie-making?"

Our Gracious Host has written a number of articles explaining how the mechanics of the publishing industry shape the kinds of books that come out of it, from the length of novel that is published to their covers. Film bends in the wind of its production process even more than written fiction. CGI technology opened up entirely new genres to the industry, "video nasties" appeared in the wake of a censorship gap and various distribution and strategy changes led to the rise of the blockbuster.

If you want to predict the shape of the sausage that is film, you've got to look at the mincing machine. And right now, there's something very interesting happening.

People pontificate a lot about computer games rendering film obselete. That's clearly rubbish. Movies aren't going to spontaneously turn into games. They're different experiences.

But movie-making might become a game.

I've been studying - and to a certain extent, catalysing - the intersection of computer games and movies for fifteen years now, since I accidentally tripped and fell into the birth of a new type of movie-making. Over the years, I've used the sets and characters of popular computer games as a virtual backlot to make a well-received feature film and an epic love story starring Brian Blessed and Joanna Lumley amongst other things. Over that time, it's been obvious that games have drawn nearer to movies, with story-based, mini-movie heavy games like Mass Effect and the new Tomb Raider hitting the bigtime.

But at the same time, the techniques used in games have been invading the film world.

CGI has become a huge part of the movie-making process, and the craft of making a CGI character or world for a movie is very similar to the same process for a computer game. Indeed, major studios are now considering using the same 3D assets for both.

One problem that both industries face is the sheer amount of time that it takes to make 3D objects, particularly things like complex landscapes with foliage. In 2009, Minecraft famously solved the problem by procedurally generating surprisingly complex, beautiful worlds - but it's actually not the biggest-grossing procedural-based media product of 2009. That would be Avatar, James Cameron's Pocahontas-with-mocap megahit - which used Minecraft-like procedural content creation:

"Once Bluff's team knew SpeedTree was the tool they would use, they set to work, quickly churning out the trees they needed by the dozens. "Starting in the morning with five models from your library, one of our artists had 40 trees done by lunchtime," Bluff said. "Those 40 trees comprised about 80 percent of the trees we needed for the entire film."

Soon after, Bluff brought his work to Mr. Cameron, presenting a 23-second long flyover of the planet Pandora. "A hush fell over the screening room," Bluff recalled. "The first thing Mr. Cameron wanted to know was 'how are you doing your trees?' He was shocked at the match to his original vision.""

Now here's the thing about procedural content creation: it's fun. I've made movies using Minecraft, and one of the most entertaining parts of the entire process is location-scouting the world, wandering around looking for the perfect cliff and the perfect sunset. In fact, it's so fun that the procedural content frequently becomes a focus rather than a backdrop, whether it's exploring an infinite universe or letting the game evolve a race of aliens for you. And using it for filmmaking is just as much fun, to the point that I have been known to fire up a procedural landscape generator that I'm using for experimental movies right now just to play with it.

Making 3D content for movies is slowly evolving from a painstaking process of wrangling polygons to, well, Populous. And games are emerging - from major publishers - where the entire point is to create near-movie-quality environments and characters.

At the same time, of course, motion capture and CGI characters are becoming a big deal. I use motion capture to create my movies these days - and while it's wonderful, it's not without its flaws.

Creating a short motion-captured film with a single character is spectacularly fast. But if you're creating a scene with 300 characters visible, each character will take just as long to animate as the first, no matter whether they're the hero of the entire piece or Spearman #14. That time mounts up fast, and can make crowd scenes using pure mocap impractical.

The Lord of the Rings series famously solved this using the technology they called "Massive" , which creates a bunch of autonomous agents, gives them simple AI, and some instructions to follow. Yes, it's basically Medieval Total War - The Filmmaker's Edition. (In fact, some gamers have taken it full circle and used a total conversion of Medieval Total War to make their own epic Lord of the Rings battle scenes).

When I moved from using game engines to create my films to a more motion-capture based approach, I found I missed the ability to drop in game-style NPCs - so much, in fact, that my latest movie has been criticised for not having enough background characters. That's definitely not a mistake I'll be making again - very high up my priority list for future films is finding a good game-like tool to add NPCs to my worlds. And I'm far from the only movie-maker thinking like that - in fact, of all the major 3D tools used for movies, both 3D Studio Max and Softimage have recently added game-like "crowd simulation" tools

And these NPC characters are just getting more human-like. We've already got virtual stunt doubles who will generate their own realistic reactions to, for example, being dropped down an elevator shaft or blown sideways by an explosion. (That tool is so fun to use that there's a community treating it as, essentially, a computer game in its own right). The convergence of convincing muscle-based locomotion and research on how emotions affect the body at a musculo-skeletal level offers the possibility that we'll be able to use CGI-based characters who don't need any low-level animation input - instead, they can be directed just like human actors, given cues of mood, marks and intention.

There's a tool which already offers directable CGI actors, albeit using quite simple motion technology. It has an entire movie-making community based around it. Its name? The Sims. Indeed, EA's bestselling computer game series is so appropriate for movie-making that at least one pro movie creation tool took most of its interface cues from The Sims.

And it goes on. The Oculus Rift and other VR tools are amazing games experiences - which will also allow actors to view and react to the CGI worlds their characters inhabit. That's a pretty major step forward from yelling at sticks, and might have prevented Ian Mckellen from bursting into tears at the artificiality of it all whilst filming The Hobbit. Motion capture suffers from problems of having to perfectly adapt a virtual set to a physical set, which is particularly a pain if you're shooting on uneven landscapes - fortunately, games developers have solved that problem some time ago. And even motion capture itself is becoming a part of gameplay as much as of film.

It's a weird time to be in film. It's weird because distribution is changing, it's weird because of changing formats, it's weird for all sorts of reasons.

But beyond all that, it's weird because with every passing year, movie-making is becoming, well, a game.

20 Comments

1:

There was a controversy a few years ago when the late, great Roger Ebert said that videogames could not be art. I'm wondering what your reaction to that comment, briefly stated, would be, Hugh. I think he was wrong just in that he had yet to be convinced, and I think that you are one of the few people who could have convinced him this early in the game.

2:

I can fully understand the idea of "Looking around in Minecraft for the perfect landscape". The sadface is, it is much harder to find them in the current vanilla game.

If you want to find bigger variety, more interesting natural backdrops, there are two good tools.

One is to get a beta 173 world generator ported to a newer version of minecraft -- better world generation has one.

The other is a tool called "Mystcraft". It lets you make new worlds while playing. "Tiny biomes" is a world type that places the different landscape features much closer than normal -- and in doing so, the noise functions have bigger overlap (essentially, the entire biome is within the noise of the neighbors). Not only have I seen "oceans" with no water (just large areas of grass with nothing growing), but I've seen extreme-hill like overhangs in terrains that normally never see those overhangs.

3:

Roger Ebert actually reviewed one of my films many years ago - Ozymandias - which he definitely felt was art, and in fact compared to Grave of the Fireflies, which was a hell of a compliment.

Yeah, in what I suspect will not be a shocking turn of events, I don't agree with him on the "games as art" thing. I've never even seen a semi-decent rationale for why someone might think that.

Although I'm never terribly interested by definitions of what is or isn't art - I'm more interested by whether something provokes emotion, provides meaning or offers an experience.

4:

That's extremely interesting - thanks. I shall have a play with those tools.

Procedural generation has a long way to go, too, particularly for filmmaking. I can see procedural generators being developed in the future which are aware of the lines of action of a scene, of the shots being used, and of the rules of composition, and which generate interesting and beautiful landscapes based on those parameters...

5:

It does seem strange to claim that a game cannot be art, but the interaction between artist and audience is very different. And the "art" label can affect what people think.

But get a bunch of musicians together in a jam session and you have something that points in the same direction. There's the nature of SF fandom, readers and writers all in it together. Games as storytelling go back further than we think. Dungeons & Dragons is the obvious marker. Tony Bath was running his Hyboria wargaming campaign long before that, and there hints of the same sense of story in Little Wars from H G Wells. I first heard of Fletcher Pratt in connection with naval wargames, not fantasy novels, and he is supposed to have run a game which predicted the Battle of the River Plate.

Heck, I've been in a couple of story jams, braiding together different people's views of a time and place. (It involved the Graf Spee, and somebody using black magic against the Nazis, and a hint of the Matter of Britain.)

6:

That's an interesting question. I'm not sure how someone would code "interesting" into a landscape generator, though there's probably somebody out there trying already. Charlie has made the point that writing a novel is a hard problem for artificial intelligence and that creating good fiction is likely to require a full blown Turing capable mind, no matter how much number crunching one throws at the problem. I expect that making interesting films is substantially the same problem, in the same sense that many problems are mathematically equivalent to the Traveling Salesman Problem, but I'm curious what new tools could be developed in the near future to help filmmakers.

Some parts of this might be fairly straightforward. For example, if you hit the 'chase scene' toggle the generator could avoid dead-end paths, provide both long straight paths with good sight lines and twisty sections just narrow enough to challenge the characters, and call up the menu of Cliched Obstructions [chicken coop, fruit stand, two guys carrying big thing, woman with baby, etc]. Likewise, hit the 'sneaking around' option and the landscape generator could know to provide lots of cover, shadows, short obstructed sightlines, and a menu for easily surprised cats.

I'll predict that once such a tool exists a lot of new filmmakers will use it to produce a lot of really cliched crap as they let the computer do too much of the work. We'll see things that look like somebody dumped TVTropes into a video machine. But Sturgeon's Law has been around as long as storytelling has, and we'll survive the automated schlock producer.

7:

Possibly apocryphal story about Lord of the Rings: When they were testing the AI, they were setting up the battle at Helms Deep. They'd set up all the defenders, and just to test they dropped in one orc. The orc took one look at the elves lined up on the wall and promptly legged it over the horizon. Not sure it it's true, but AI bug can be amusing.

8:

I'm actually going to go into this question in a bit more detail on Monday (probably). Whilst I agree with Charlie that the creation of original story content is hard, my suspicion is that there is a surprising amount of the creative process, particularly on more grunt-work intensive artforms like film, that could in theory be automated.

On the Turing-Complete issue, it's also worth noting the work of Emily Howell, the AI composer developed by David Cope "whose" music has actually garnered "her" a record deal.

I've listened to some of her pieces and they aren't bad. It's a very weird moment realising they're generated by a computer.

http://www.youtube.com/watch?v=QEjdiE0AoCU

9:

I'd agree with Ebert-sama that computer games are not art, any more than a piano is art. Whether art can be performed in a computer game I don't know, I don't indulge deeply enough in the timesuck that is a modern graphics-intensive game. Art can be made within a game using the underlying core code as an engine of creation and it's possible an AI might one day use a game engine to create such art.

Art is in the eye of the beholder and sometimes it takes odd forms. Hugh and others on this blog are swordfighters and I'm sure they know of practitioners who do it so well their movements are art to observers, same with many of the other martial "arts". To the ones creating such art the sweat and effort and the knowledge of the "dropped notes" and minor mistakes usually prevents them from thinking of what they are doing as art. Sometimes though the thought "that wasn't too bad this time" suffices.

10:

The 'No Man's Sky' link was very interesting ... I'm not a gamer therefore not familiar with current game offerings, but this sounds as though it's modelling reality more closely than some of the games I've seen reviewed on TV. Especially like the birds underground and cows in pits ... stray bits of the unplanned.

One question: Do the story characters also have random/emergent issues/problems. It seems that in all the games I've seen reviewed, the main character/POV is a 'superman' in physique, and a yogi in terms of body control, i.e., infinitely large bladder, no need for sleep, no itches, etc.

11:

Interesting question!

There's an emergent genre of gaming, the "survival" genre, that's all about human weaknesses and limitations. The two main torch-holders for the genre are:

  • DayZ, the ultra-realistic survival horror game, which models blood loss, hunger, thirst, sleep requirement, infection and all sorts of other things.
  • Minecraft, the somewhat-friendly voxel crafting game, which nonetheless models hunger, and in which you start out extremely fragile.

Neither of them are story-based games per se, but their mechanics result in them generating a lot of emergent stories. In particular, the "DayZ Tumblr" is practically a separate genre of blog these days - here's one example.

Other games in the same genre include "Rust" (brutal PvP survival) and "Don't Starve" (does exactly what it says on the tin).

For more narrative-based survival, The Walking Dead game series incorporates a lot of survival-based elements, and there are dozens of mods for the Nordic fantasy game Skyrim that transform it into a very realistic experience indeed.

In general, the games world seems to be moving away from the "ubermensch" model of player character, particularly in the independent games development scene, and we're seeing more and more developers embracing human frailty as a way to create emergent stories. It's a very cool time.

12:

Another cool article from you Hugh :)

Re. the "are games art" thing - when I think of this, I think of particular peak moments in great CRPGs (BioWare, Bethesda), where some event in the game has moved me (to elation, to tears, etc.) as a virtual participant in an unfolding story, in a way roughly analogous to the way it might move me as a passive observer of a story in a film.

A "moment" like this in a relatively recent game that comes to mind was in Dragon Age: Origins, when I as a protagonist, managed to convince an "evil" person to drop their thirst for revenge, and make amends for what they'd done. The combination of the trouble I'd gone through to get to that point, plus the VO, the digital puppetry and the music, resulted in me having floods of tears as the guy did the right thing and the music soared. THAT was a form of art, to me. And to me, it has something to do with immersion, with being totally "there" in the story, in the virtual place, with virtual characters who, for a moment at least, seem real.

Re. the future of film, I agree with a lot of what you say, Hugh. Spot on. Particularly with the thing of actors being able to act in the virtual space. It must surely be incredibly hard for actors who don't have strong visual imagination to react to things that they can't see.

Also re. mixing real actors with digital "spear carriers", procedurally-generated stuff, etc., I agree with that too, that's the way forward, and an example of the kind of convergence you're talking about.

Great stuff! :)

Of course, our genial host was way ahead of the curve with Accelerando. In his Wired/Dotcom Bubble trance, he foresaw how it is all going to proceed ... :)

13:

ObNitpick: Turing-complete does not mean that s.t. passes the Turing test. Instead it's a minimal property of programming languages. Passing the Turing test, i.e. passing as a human to a human observer is much more difficult.

14:

Aspects such as "no toilet breaks" seem to be routine in movies. They're not necessary to the story. That doesn't mean they can't be used. Dekker is eating when he he dragged back into service, in Blade Runner. I vaguely recall a scene in one of the Crocodile Dundee movies set in a urinal. But they are there because they have a role in telling the story.

Movie makers have known for over a century that you don't have to show everything. And there's an 1899 film clip using film editing, by Bamforth & Co. Hollywood didn't even exist back then.

15:

Don't forget the, ahem, realistic alternative to procedural content creation: Scanning the entire planet.

http://www.polygon.com/2014/2/3/5374012/reroll-drone-sourcing-planet-earth-pixyul-pc

16:

Or indeed the Ordnance Survey with Minecraft

(However, that's at rather low resolution, due to limitations in Minecraft.)

17:

Ok, I'm about to land someplace in the middle (and probably need to duck and cover since I expect to be shot at from all sides at once).

Videogames are art, for values of art anyway. Not for the scenery or character design, but at least sometime for the story-telling. I've played Japanese style RPGs where you get so invested in the story and characters as to get emotional about their deaths, laugh with them etc... If that's not art, then neither film-making nor literature are art either.

Nojay, are claiming that "cabinet making isn't art" in #9? That's the only basis on which I can see any ground for a claim that "a piano isn't art".

14 - That makes me think of the scene in the orignal Crocodile Dundee where the eponymous character encounters a bidet for the first time ever. Carrying on, whilst not ubiquitous, I'd suggest that toilet scenes are actually fairly common in some genres of film-making, either as an excuse for isolating characters of one sex in comedy or romance films, or because that "break interestingly" in "action fims".
18:

Re your comment on #14, there's also the "bomb in the toilet" subplot in Lethal Weapon 2.

19:

Toilet breaks seem to be a regular feature in American road movies. It's also useful for the story: when you stop at a truck stop, the Villain can catch up with the Good Ones, the two main characters get temporarily separated so Interesting Things can happen etc.

20:

Understand what you mean by 'no toilet breaks' in movies. However, movies last only 2 to 3 hours and typically are a collection of high/key notes of their stories. Games often last much longer especially to play and my understanding is that games are intended to be more immersive than films. Which is why I wonder why the few I've seen haven't been more in-tune with human physical limitations. Also noticed that there aren't a lot of laughs to be had in most games.

Specials

Merchandise

About this Entry

This page contains a single entry by Hugh Hancock published on February 8, 2014 1:56 PM.

Typo hunt: Neptune's Brood was the previous entry in this blog.

What's The Future For Virtual Reality Movies? is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Search this blog

Propaganda