Back to: Burn The Programmer! | Forward to: Are VR esports going to become a thing?

From Here To The Holodeck

People sometimes ask me why I'm so keen on VR - keen enough to drop a 20-year career to move into it - and I always give the same response.

"I get to make worlds".

That's... quite the sales pitch. And I don't mean "making worlds" as a novelist or even a filmmaker (my former career) does it. I mean creating worlds you can walk into, explore, interact with, and get murdered by hideous creatures brought back to life by the blasphemous rules of the magical place you now inhabit.

(My creative approach - latest output of which is the VR horror/rpg Left-Hand Path - definitely tends in a certain direction, and that direction is deep, complex magic systems and disturbing consequences therof. Plus I was really inspired by Dark Souls this time around.)

And if I was speaking to someone whom I suspected might have watched Star Trek - you know, about 80% of the population - I might follow that up with "basically, I have a holodeck".

Horror On The Holodeck

If you don't know - the "holodeck" was the invention of Star Trek: The Next Generation. Ostensibly a recreational tool, it could conjure in perfect detail any environment its user could dream of. For plot reasons, the utility's obvious - as Wikipedia says,

From a storytelling point of view, it permits the introduction of a greater variety of locations and characters that might not otherwise be possible, such as events and persons in the Earth's past, and is often used as a way to pose philosophical questions.

(Wikipedia: Holodeck)

(Also, for storytelling reasons, it bugged out more often than Internet Explorer 6. I'm pleased to say current VR tech, including Left-Hand Path, doesn't have quite that problem.)

Ever since its introduction, a lot of people have regarded the Holodeck as the ultimate goal of games or virtual reality. A tool that can create a completely convincing world in which you can be anything you want to be.

The new wave of VR is a huge step in that direction. By "The New Wave" here I mean the Vive, the Oculus Rift, and - if reports are to be believed - Microsoft's Mixed Reality. Phone VR with no positional tracking or motion-tracked controllers is not the same thing at all, and should really not be taken as a representation of current VR. If the VR system doesn't allow you to get up (using your IRL body, not a controller), walk around a bit, and pick things up with your hands (mediated by controllers like Oculus Touch or the Vive wand) I would argue it's not "real" VR, and it's certainly not what I'm talking about here.

But there are still plenty of limitations to it, and you may well be listing them in your head as you read this.

"What? Lol. It's not anything like a holodeck. You can only walk about six feet! You can't feel objects, there's no wind, there's no smells, you've got a damn great cable attached to your head, and the other characters are just computer game NPCs!"

All true.

But for how much longer?

Losing The Wire

Wireless VR is pretty much a solved problem at this point.

The Vive has the TPCast wireless module, which gets some rather mixed reviews from users, but does work. More wireless solutions are on their way - notably one from Intel.

Meanwhile, Oculus have announced a completely untethered wireless headset and controller - the Oculus Go. It doesn't even need a computer - everything's contained in the headset. It's coming next year, and would be my current pick for the thing most likely to spark the next wave of VR adoption.

Walkin' On (Virtual) Sunshine

Locomotion. It's the bête noir of VR developers right now.

Put simply: most people don't have an infinite plane in their apartments. Even people living in Texas.

So if you jump into the endless vistas of Skyrim, say, or even the dank corridors of Nehemoth in Left-Hand Path, and you wander them freely, sooner or later you'll be interrupted by face meeting wall.

The default solutions to this problem at the moment are either an in-game "teleport" mechanic - which works well but breaks immersion for some people, and obviously isn't as good as walking - or a game-like "sliding" mechanic - which again, breaks immersion, and also causes some people to throw up.

In Left-Hand Path, you play a wizard, and so I wrote it into the background that one of the features of your powers was the ability to "blink" from place to place. (You can also use sliding locomotion, because wizards have feet too.) But that's a rather game-specific solution.

Some games and experiences have experimented with redirected walking or environmental redirection. Basically, you can fool the human brain into thinking that you're walking in a straight line or exploring a large environment, when you're really, really not. Notably, the VR game Unseen Diplomacy does this to great effect, fooling you into thinking you're in a much larger environment:

But this approach only works under some circumstances, and it requires you to have a rather large room.

You can enhance the effect with Galvanic Vestibular Stimulation, but then you're using magnets to alter your brain, with more or less unknown medical consequences. For some reason users seem not to go for that so much.

More promising approaches are mostly variants on the "treadmill" theme. The Virtuix Omni is the only one that's made it to market so far, and reviewers are not too enthusiastic:

Our best description of the experience is to imagine that you're moonwalking on the sides of a human-sized serving plate, while wearing slippery bowling shoes.

But people are continuing to develop new treadmills, and sooner or later they may hit on a winner. The Strider treadmill is getting better reviews, using a bunch of rotating balls underfoot. The KatVR is close to launch, and again, seems to be getting better reviews than the Omni. And there are literally dozens more - including some really ambitious stuff like the giant-robot-arm plan from AxonVR, of whom more below, and entire floors made of robot-controlled mini-treadmills.

And there are other approaches too - step-trackers, where you walk in place and the software detects your movements. "Arm-Swinger" locomotion, where swinging your arms as if you're walking translates into movement. "Grab And Pull" movement where you grab the air and pull to move forward.

With so much research and experimentation going on, and some of these approaches showing promise, I'm pretty sure we'll have an acceptable walking solution sooner or later.


This one's a harder problem.

Basically, you can see things in VR, but you can't feel them. And of particular importance, they can't stop your movement or interfere with it.

Problem #1 for pretty much any VR game developer is "how do you stop the player sticking their head through a wall". That's a haptics issue. So is allowing monsters to parry a sword strike, or allowing players to feel the recoil of a gun, or any other physical sensation.

Most controllers have limited haptic feedback, like a games controller. They can vibrate more or less, and if that's used intelligently it can add a remarkable amount of depth. If you get the chance, play "Longbow" from Valve's VR showcase "The Lab" to get a feel of just how much you can do with a simple vibrating controller.

Haptics is a huge, huge field of research - there are at least two separate haptics conferences, and dozens if not hundreds of companies working in the space. And there are a number of approaches that are showing progress on at least some aspects.

For touch, there's an ultrasound-based approach which appears to work remarkably well. It only works for soft touches, and is more bringing the sensation of touch into the virtual world rather than putting any force there, but it's still a significant leap forward.

Likewise, Tactical Haptics has been working for years on controllers that will simulate limited inertia and force.

This is actually a huge deal - I've been working on swordplay in VR recently for my next project, and the fact that the swords are weightless is a real immersion-killer, and rather hard to solve. From what I've read in hands-on reports, the Tactical Haptics addons do quite a good job adding the sensation that you're holding a heavy, hard to move object, or getting force feedback as you use that object in VR.

One really neat haptics device sounds pretty strange initially - a VR fan. Yep, it's a fan hooked up to your computer that simulates airflow whilst you're in VR. Weird, but users report it's a surprisingly significant addition to immersion in racing games or similar experiences.

In terms of actual "hard" feedback - a swing of a sword being parried or a wall blocking your progress - the only active project I'm aware of is the somewhat terrifying HaptX exoskeleton. This uses - or will eventually use - a giant robot arm and attached exoskeleton to provide a two-in-one solution to walking (you walk, your feet move, the robot compensates) and haptic feedback up to and including movement-stopping feedback. However, I can't find any videos of a working suit yet so this one's definitely some distance off.


You might assume that smell in VR is definitely in the "yeah, no progress" category. You'd be wrong.

8k VR headset Pimax has promised a "scent module" as an optional addon to their headset. No real news on how it'll work, but they say that it's coming.

Meanwhile, in New York there's a company dedicated to nothing but making VR smell real.

And there are more. There's a porn company also working on a scent module. Multiple other startups. Here's a roundup of the state of the fart - erm, I mean, art.

I look forward to testing the "scorched flesh", "squamous, dripping slime", "magically-opened demonic rift" and "rotting corpse that should have died ages since" smells for the upcoming Left-Hand Path smell-o-vision edition.

And Beyond!

There are more problems left to solve, of course.

We've still got to figure out how to create convincing human NPCs for VR - although there's plenty of movement on all fronts there, from programatically-created movement (either using reinforcement learning or clever algorithms) to natural language. Or we could just hire actors.

We've got to increase the visual fidelity of the environments - but that's going rather well at the moment. On the hardware front, the first 8k headset was recently Kickstarted, and whilst Oculus have said in the past we'll need to go past 8k for really realistic displays, we're clearly heading in the right direction.

There are plenty of problems that you might not even think of standing between reality and the Holodeck too. For example, equiloproprioception is surprisingly important in Actual Reality, and aside from the aforementioned magnets-affecting-your-brain tech, we're still in the very early stages of figuring out how to mess with that. How will we simulate humidity? What about air pressure? And so on. But there are smart people working on virtually (no pun intended) all of these.

And of course we've still got to invent the revolutionary technology which lets the entire thing malfunction and dump you in a real Western where you have to become an outlaw to survive, and bullets can really kill.

But I'm OK with waiting for that one.



Brave New World
"Every hair on the bearskin rug" ( Whilst, IIRC, having sex at the same time )


Ah, yeah, I didn't cover the wonderful world of VR sex peripherals in this article, but I believe there's a lot of active development going on there too :)


I'd add accessibility/inclusion to the list of problems that not many VR/AR folk seem to be poking at.

For example:

* I don't have stereo depth perception - so I need other depth queues

* A friend of mine in a wheelchair couldn't interact with elements of a game because they'd been designed "out of reach"

* Similar issues for somebody with dwarfism here


Interesting question! Let me think about that - those thoughts might turn into a separate post!


Stereo depth perception - this is particularly interesting!

What sort of cues can VR devs use for people who don't have stereo depth perception? I'll bear it in mind in future games.


Regarding movement I'm quite surprised at the amount of VR games that are first person shooters/adventure games. Perhaps it's just because the FPS is such a mainstay of AAA games but flight sims (like elite dangerous) or even genres like RTS seem far more appropriate.

VR gaming hasn't exactly taken off with most games feeling more like tech demos than actual fully fleshed games. I hope the technology doesn't stutter out and die on the FPS hill when it could have found a much more engaging niche.


Great description of the state of the art!
I think my only real quibble would be the notion that VR needs to support walking around and other forms of movement. That's a specific subclass of VR -- or perhaps a superset, with limited motion being the subset?

For the sake of argument, it seems we could define two extremes of a VR spectrum. First, we have "weak VR" -- things like Minecraft or The Sims that are sufficiently absorbing ("immersive") that you can lose yourself in them and fool yourself that you're actually there. But in no way do they feel real: no haptics, no smell, crude graphics, etc. etc. At the other extreme, "strong VR" would be more like a Trek holodeck.

We already have things that are semi-strong VR: some of my colleagues developed a training simulator for one of those big excavator-mounted tree-harvesting and -processing machines that included all the machine's controls, and provided sufficiently realistic force-feedback that operators made the leap to real machines quite quickly -- but it used crude vector graphics for the actual harvesting because the key design goal was the haptics, not the visuals. I've seen some interesting descriptions of tele-surgery applications ( that seem likely to be widely implemented in the "near" future. They have obvious military applications, so I imagine there's lots of funding.

Oh, and flight simulators -- the big ones they use to train and test pilots -- are already in the semi-strong or stronger state of development. (I base this on my father's descriptions of having "flown" a couple at the local CAE R&D facility.)

The semi-strong applications are available now, or coming soon, because they're what I'd call "bounded problems": all the parameters of the environment are "easily" (for small values of "easy") defined and constrained in software.

Something like a true holodeck would have to solve an unbounded problem -- or a large series of bounded problems -- which is hard to imagine happening any time soon. We'll undoubtedly get much closer to that solution. But it seems likely that a truly nonbounded application will require some kind of neural interface that takes advantage of your brain's encyclopedic knowledge of the problem environment to generate the feedbacks and constraints, and input them directly via your sensorium. But that doesn't seem close.


Coincidentally I came across this article yesterday

The relative ease of reproducing things like the "rubber hand illusion" in a virtual environment should have obvious implications for horror game designers. If I'm reading it correctly all you need is a controller that vibrates and appropriate visual feedback to set it up, then something lunges out of the shadows and bites their arms off :)


I don't have stereoscopic vision either, never have had. Parallax and scale are the most noticeable cues for me, but that doesn't mean they're the only or the most important ones (although they probably are the most important ones).

Also knowledge of real scenes. It took me many years to figure out what people meant by a photo of someone with a lamp post growing out of their head, because it just looks like someone standing in front of a lamp post to me.

But my brain does tend to synthesise parallax for itself anyway, which is probably a complicating factor. Photos sometimes look like they're displaying parallax to me. There is an optics museum in Keswick where one of the exhibits is a 360° hologram, and I found I had to view it from much more extreme angles than people with two eyes before it became impressive compared to a photo.


I was diagnosed with amblyopia as a child but never wore the prescribed glasses as they gave me headaches. However since I spent a lot of time outdoors and a lot of time reading what I ended up with was one short-sighted eye and one long-sighted eye. It does mean I don't perceive stereo normally, but I found Viewmaster stereoscope images quite vivid as a child and the 3D version of Avatar was more 3D than real life for me. On the other hand I still don't need glasses to read books or numberplates, I just switch eyes unconsciously. I expect I'd find any kind of 3D headset too intense.


Re: '... one short-sighted eye and one long-sighted eye'

If not too great a difference, that's actually 'normal' as having one dominant and one non-dominant eye helps the brain code for depth perception.

(BTW, seems that too much equality between the eyes in one particular eye area is associated with dyslexia in some people.)

'In the University of Rennes study, published in the journal Proceedings of the Royal Society B, scientists looked into the eyes of 30 non-dyslexics and 30 dyslexics.

They discovered differences in the shape of spots deep in the eye where red, green and blue cones - responsible for colour - are located.

In non-dyslexics, they found that the blue cone-free spot in one eye was round and in the other eye it was oblong or unevenly shaped, making the round one more dominant.

But in dyslexic people, both eyes had the same round-shaped spot, which meant neither eye was dominant.

This would result in the brain being confused by two slightly different images from the eyes.

Researchers Guy Ropars and Albert le Floch said this lack of asymmetry "might be the biological and anatomical basis of reading and spelling disabilities".

They added: "For dyslexic students, their two eyes are equivalent and their brain has to successively rely on the two slightly different versions of a given visual scene."

No single cause

Prof John Stein, dyslexia expert and emeritus professor in neuroscience at the University of Oxford, said having a dominant spot in one eye meant there were better connections between the two sides of the brain and therefore clearer vision.

He said the study was "really interesting" because it stressed the importance of eye dominance in reading.

But he said the research gave no indication of why these differences occurred in some people's eyes.

He said the French test appeared to be more objective than current tests, but was unlikely to explain everyone's dyslexia.

Dyslexia is usually an inherited condition which affects 10% of the population, but environmental factors are also thought to play a role.

"No one problem is necessary to get dyslexia and no one problem is behind it," Prof Stein said.'


Re: VR tech

Personally, the gaming application is less interesting than the medical/rehab potential of this tech. One parent had a stroke and a VR device that has a programmed sensory perception benchmark (normal) range that could be used to compare against could provide much more info than neurologists and physiotherapists currently offer. Post-stroke, it took the family quite a long time to figure out what our parent could and couldn't perceive, as well as how things were being perceived*. Such a device would also help measure the effectiveness of physio/rehab therapy which for my parent has been a complete waste of time and only caused more frustration.

Ditto as an application for testing pre-school kids - much better than the current vision and hearing tests in GPs' offices. Plus, most GPs do not test for balance.

Also, combining the VR (perception) with a servo-suit (action) would be useful for correcting/mitigating some of the damage done by a stroke. Very useful considering our aging population.

* When asked to draw a clock face, the result was something out of Picasso which made me wonder whether Picasso had had a stroke or brain injury. Why not - art historians eventually figured out that Van Gogh's starry nightscapes were symptomatic of neuropathy possibly brought on by the digitalis he was prescribed. Ditto for Monet and his gardens: he accurately painted what he saw so that you can trace the progression of his cataracts. (In contrast, El Greco's style is not attributable to astigmatism.)


Interesting. Tinted or coloured glasses and overlays help a lot of dyslexics cope with text better.


Thanks for, among other things, the round-up of current day VR tech. Lots of links to follow.

I do think smell is still in the going nowhere category. Until we get to direct neural input, all the smell systems both past and present have the problem that they persist. It's not obvious what you were watching or listening to on your VR system five minutes ago, but anything that pumps "squamous, dripping slime" into the atmosphere is going to lead to some heated conversations with family / roommates. (Or neighbours, if you're thinking of just opening a window.)


Add to that anyone who is never going to be a candidate for lasik surgery, who can't even use contact lenses. I'm always going to require corrective lenses (eyeglasses).

Plus I'm old and have bad knees. You won't find me crawling through the duct-work.


I might follow that up with "basically, I have a holodeck".

The Holodeck is a good cultural reference for explaining VR to non developers.

Is there a common reference work for VR/AR developers today? I ask because when I was lucky enough to do a couple of years VR programming around the turn of the century, Neuromancer by William Gibson was our common vision and reference. If you got a group of VR developers together, most or all of them would have read Neuromancer and it was a common shorthand for what we were trying to achieve. (I hasten to add just the networked cyberspace bit, not the dysopian cyberpunk future.) I believe this was still true for people like John Carmack and Michael Abrash.

Among VR developers today, what book or film is it safe to assume everyone knows? Or has the field grown sufficiently for fragmentation?


What, all this VR debate, and no-one has mentioned the elephant in the room?

It's a VR game that focusses on co-operation, rather than first-person shooting. My beloved decided to invest in PlayStation VR, and it's actually the game that youngest plays by choice (firstborn prefers the non-VR "Overwatch" and "Destiny" first-person shooters, in collaborative mode with his friends). I haven't had much time to play it, but it's rather engaging... ;)

Crew-based simulations have legs, although I suspect that it will take the next generation of game to take off - to operate as a vehicle crew, rather than several vehicle's solo pilots.


" First, we have "weak VR" -- things like Minecraft or The Sims that are sufficiently absorbing ("immersive") that you can lose yourself in them and fool yourself that you're actually there. But in no way do they feel real:"

I think there's an important insight in that.

Which is that what makes works of art immersive (including as works of art paintings, books, comic, movies, computer games, table top role-playing, LARPs, sculptures, etc) is *not* necessarily the same as making them more "realistic".

I think what you need to do is remove "frictions" that pull people out of the VR experience. That's not necessarily the same as making it "more realistic". For example, a non-realistic way of doing without haptic feedback that works smoothly is likely to be more "immersive" than a shitty haptic feedback system.


I don't know how fragmented it is, but my hope is "very".

My one current foray into VR is the beginnings of an attempt to build the VR love-child of Duolingo and Sesame Street, for pre-schoolers, using Unity. It has become very obvious that this far too big a job for one chap fiddling about vaguely in his spare time, but I may play with it for a while more for fun.

(Not sure about Unity either. But I'd need to put a lot more hours in to judge. Very much just twiddling about with it, with little chance of finishing anything).

But "realism" isn't really what I'm after. Or rather: my ideal is the realism of puppets, dolls, make-believe and cartoons.

Which puts me in a different space than an "I want a holodeck" VR developer. Which is fine.


Re: 'VR love-child of Duolingo and Sesame Street'

The first universal translator for kiddies? Surely Disney is already working on cartoon/comic book related VR by now for their theme parks. Interesting ... says here that Duolingo already offer English-High Valerian lessons and are working on English-Klingon:


"Ready player one" will be a good cultural reference especially once the movie hits


> Surely Disney is already working on cartoon/comic book related VR by now for their theme parks.

Disney actually opened theme parks with VR attractions almost 20 years ago:

...though I certainly wouldn't be surprised if they are working on something new now.


Not a translator, an environment that encourages interaction with language, by copying how others use it (because kids learn best by seeing their peers use language). Though with virtual puppets as one's peers.

Think language immersion, in VR, crossed with Muppets. With individualised progress tracking so we can do the operant conditioning more efficiently, because keeping the kiddies addicte... er, keen, is important.

Ridiculously over-ambitious as a project, but interesting.


I sometimes hear people talk about VR e-sports, and I have to admit I really don't think that's going to work well outside of things like RTS.

Latency is a major issue when it comes to "players doing flashy things" - heck, in fighting games longer inputs for moves are both a deliberate nerf and a tell-tale sign one can bluff with. Those of you who're familiar with EVO Moment 37 and Street Fighter's mechanics may have noticed that poor Justin Wong actually smells a rat on some level and feints the super several times!

Tournament play is already gruelling. With sticks for input, we see things like the EVO 2013 KOF XIII Grand Final happen. Or the sheer exhaustion on display in this year's KOF XIV Top 8. Yet we still see feats that to a spectator who knows the game are spectacular.

Add the stress of even your own motions in a physical fight and you get into situations where professionals just don't fight "for real" that much in one night or even one month. Equivalents are true for shooters even if you ignore how critical height (and thus sometimes airborne motion) are in many of them.

There's plenty to do for players in general with VR here, but I don't think VR is going to dominate e-sports - at least not with a naive translation. There are definite possibilities for adding new kinds of information and bluff, I'm a lot warier about input schemes etc. If you've seen how technical the fighting game crowd gets with a stick and 4-6 buttons you might appreciate why. Counting frames is the least of it.


I'd guess cues from accomodation are another form of depth perception (especially short range), but that's out until we can get lightfield displays.


Lightfield displays suffer from the same problems as lightfield cameras unfortunately - you need a large number of pixels between each microlens to get a fairly low resolution output.

OTOH I think that desktop hardware is probably capable of doing the rendering without too much trouble. For simple scenes you usually have GPU to burn so jittering the camera and drawing multiple passes is easy enough.

Mobile is a different kettle of fish though. Whichever way you look at it you are running on a computer the size of a cream cracker that has a limited power supply, a crappy heat sink and grinds to a halt if you drive it hard for more than a few tens of seconds.


Elite: Dangerous is actually hugely popular in VR. You don't hear much about it because the discussion is limited to ED boards and subreddits rather than the main VR ones, but it's a pretty significant draw.

RTSes less so, largely because there's not a good one in VR yet. Out Of Ammo did OK, but it's not full-fledged yet. But when one arrives I do expect it to be massively popular.

There's one non-obvious issue with RTS in VR, though, which I discovered by watching DOTA in VR: the viewpoint leads to lots and lots of looking downward. VR headsets are quite well-balanced, but they're also not super-light: if you spend a lot of time looking downward, you end up with a sore neck.


I completely agree.

I make games and game-like experiences in VR for the same reason Charlie writes books: I love telling stories and creating worlds. The money's also nice, but I'd be doing this anyway.

If you're looking at VR as a technology, and you don't have the "I have to use this to tell stories" urge, then there are applications for it which are likely to be considerably more lucrative and arguably more world-changing. From medicine to simulation, the possibilities are huge.


Visualisation is something that's going to be massive in VR. You can do SO much by just taking something that was on a screen and transferring it to a navigable virtual space, let alone getting more sophisticated.

So this looks really cool from that point of view.

It also makes a lot of sense. Audio is considerably more significant in VR than it is in "flat" content, and it's hard to get right. Authoring that experience right in VR sounds like a smart way to go.


Philippa Cowderoy noted: "I sometimes hear people talk about VR e-sports, and I have to admit I really don't think that's going to work well"

Not very well indeed. If you consider the frequency of IRL injury among aging jocks (me, for instance) and couch potatoes (many video game players) who aren't trying to perform at a competitive level, the mind boggles at the death toll when people try to emulate their favorite athletes in VR. *G*

Less facetiously, it takes serious training to move your limbs quickly and under precise control, like a professional athlete, and trying to do this in VR just isn't going to happen unless you're a professional athlete. For such people VR might be a very effective training tool; for the rest of us, it's likely to be a painful lesson in humility. Consider martial arts moves, which take quite a bit of training to execute effectively -- and you'll have to do them fast unless your VR opponent is so dumbed down to the point that there's no challenge. Then imagine being tackled in a football simulation or body-checked in a hocky sim -- my Dad told me once that he ached from head to toe for days after crashing a plane in a high-end flight simulator. It's going to take many hours of strength, flexibility, and endurance training to do any kind of sport justice.

That's actually one of my biggest objections to the whole SF/F cliché of being poured into another person's body and acting like you're still in your own body -- or having them poured into your body and trying to use their professional-level skills in your barely amateur-level body. There'd be so many muscle tears and stretched ligaments I wince just thinking of it.


Yeah Elite Dangerous is probably the best VR game out there

It fits within the platform limitations really well since it's s cockpit game and the low resolution isn't noticeable

Oculus + a thrust master + voice attack and you get a glimpse of the future


"Visualisation is something that's going to be massive in VR."

Finding good visualizations for data is really hard. I mean, *really* hard. Despite literally centuries of experience, we're still working on how to do that well in 2D.

If you'll forgive a digression: we had similar issues with the movie industry. It took a couple of decades for our current ideas about cuts between shots and of how cameras pan and move, not to mention standards like "close up of the head and shoulders" and "wide shot", or the subtleties of depth of field and focus. Things we just take for granted these days about how movies work, took a lot of time, a lot of clever people, a lot of money, and trial and error. The problems weren't really technical (though they had them too), they were conceptual problems of how to use the medium.

So I'm not a pessimist about VR technical advances for visualization, but I am a bit of pessimist about how long it will take us to really figure out how to leverage the technology.

Still, there's gold in them there hills for those who succeed.


As a keen esport player and also someone who has played a lot of VR games that require precise, forceful movements, I don't think this will be as much of a problem as you believe.

One of the most popular games in VR at the moment is GORN, which is literally a gladiatorial simulator. It has a semi-competitive mode, too. I've so far heard of very few people injuring themselves playing it - I've played it for hours myself and haven't injured myself. (I'm not unfit, and I've studied martial arts and body mechanics, but I'm by no means a professional athlete.

Another very popular game is Thrill Of The Fight, a decidedly non-dumbed-down boxing game. I'm at least semi-competent as a boxer, and I haven't completed it yet (albeit I've not put a lot of time into it). Again, it's popular, again, a lot of people who play it aren't trained, and again, I haven't heard of an epidemic of injuries.

Finally, there's an actual nascent VR e-Sport. In fact, there's two of them: Echo Arena and Onward. The former is basically zero-G Frisbee, whilst the latter is for all intents and purposes Counterstrike VR. Both very popular in the VR crowd, not all of whom are highly trained athletes (although a few people are), neither producing many reports of injuries.

Hmm - an article on VR and esports would be interesting. I might do that...


I understand why it's an issue in VR (and I'm guessing the speed of players' strikes is fixed rather than based on how fast you move), but I always feel pretty snarky about 2D fighters where supposedly proficient strikers punch slower than I do.

Admittedly I studied wing chun for a while, but I've been stuck in couch potato mode for a few years and I'm on daily sedatives. With you that injuries shouldn't be an issue designed correctly - I don't know how much you can follow of what's going on in the EVO vid I linked though? And Counterstrike is unfortunately the default boring attractor for VR FPS - unlike Overwatch et al, it's also in territory covered by physical sports.

Take me off the sedatives for a while and I'm still pretty damn twitchy for my mid-30s though - obviously the reaction time gap between touch, sound and sight matters and I'm gaining maybe a couple of frames and a little less execution jitter, but we'd have to have a pretty solid conversation just to compare Thrill of the Fight to, say, a Super Street Fighter 2 Turbo Ryu mirror match for complexity (that's a '94 release refining from SF2: The World Warrior and one of the first designs to really stand up long-term).

Certain things are grossly simplified and abstracted in conventional 2D fighters, of course. But others... well, there's room enough for "fun but serious" research in the programming language/formal theorem proving space that I'm hoping I find the spoons to give a talk :-)


How much research is being done on motion sickness and nausea that many people experience with VR (and even sometimes just first person games)? I feel like this is one of the biggest barriers to more widespread adoption of VR.


A lot. A lot.

It's already been significantly improved - the number of people who experience nausea with the Vive is very small, for example. It's night-and-day compared with phone-in-headset VR or earlier generations of the Oculus. Gabe Newell is on record saying it's around zero percent, and I wouldn't be surprised to find out that less people get sick in the Vive than in regular flat-screen FPSes.

But I know it's also still an active area of research. I don't know too many details of the specifics, though.


I think there might be something to do with the translation of the human field view to a flat screen that exaggerates the effect in FPS games. Certainly I can see how the VR hardware should allow for the presentation of environments in a way that is more sympathetic to how humans actually perceive things and doesn't lead to the dissonance that causes motion sickness.


Also @Hugh Hancock - "Motion sickness" is frequently caused by your inner ear correctly detecting that the vehicle you're in is moving in roll, whilst your eyes saw that the furthest thing you can see is stationary relative to them (this is the usual mechanism in air and sea sickness).

If the VR headset does this the other way about (the display horizon moves but your inner ear says it's stationary) it might trigger the same response?


Motion sickness in VR is usually caused by poor tracking and latency. The view is supposed to follow your head position, but the sensors that track the movement are imperfect and it takes a finite amount of time to generate a frame and present it to the eyes.

In the old days VR kit used magnetic and ultrasonic trackers which were frankly terrible, and often struggled to get to the double digit frames per second. You had no chance.

These days tracking tech is much improved, consumer PC hardware can easily manage 120fps on a reasonably complex scene and many of the VR rigs have libraries that extrapolate head position. so you know it will take you .01 seconds to render the next frame, and in .01 seconds the users head will be *there*.

Between those advances the situation is much improved but still falls short of perfect.


Re: 'For such people VR might be a very effective training tool;'

Combine with TMS via the helmet plus a work-out program and you have next-gen anti depression behavioral therapy. Would also need sensors to monitor some physiological signals. Also potentially useful as a pre-study aid provided the workout includes aerobic/HIIT units.


Well, this is a timely book review:

Aldo Faisal explores the immersive journey of technology pioneer Jaron Lanier.

'Dawn of the New Everything: A Journey Through Virtual Reality
Jaron Lanier'


This seems appropriate:

Vortx is the world’s first 4D simulator that can physically recreate any virtual environment in the comfort of your home or office. It reads and analyzes audio and video data in real-time to create physical effects that you can feel to match what you see and hear; because Vortx processes live data that means it is compatible with ANY PC game title and every digital video platform.

Imagine feeling the effects of bullets whizzing by and battlefield explosions in Battlefield. Or what about the swirling inferno of Drogon’s fiery breath in Game of Thrones?

With Vortx you won’t just see and hear the action -- you’ll feel it.

Not a PC gamer, so not something I'm interested in, but though some people here might find in worth a look-see.


I'm skeptical of their claim that they can infer the appropriate effect to generate only from video and audio data. Seems more likely that it will work OK for a few specific movies and games that they used as test samples but poorly for most others.

And that's before considering all the obvious context-based limitations: it can't know if there's a fire behind you, it can't know if the fire is behind a glass wall that should be blocking the heat, it can't know if your character has heat-immunity, etc.


I think choosing what should be, and should not to be, realistic will be one of the hardest parts of the VR genres as they develop.

Because why would most people want *realistic* martial arts or e-sports in VR?

Consider Guitar Hero.

Guitar Hero was a huge success as a game because it wasn't at all realistic. People don't want to spend all their spare hours locked away in their room practicing for years to become a rock guitar virtuoso. They just want to play at being one.

People love playing the Madden computer game. They don't love doing 120 sit-ups before breakfast.

Yes, there are martial arts enthusiasts who'd really love realistic VR martial arts, where their actual physical speed translates to game speed. My Kendo friends are talking about Kendo for the Vive, I'm sure they'd play it for hours and hours and hours.

But (and I mean no offence to people who could beat me senseless using just one little finger): I don't think these people are very important as an audience. Because there just aren't that many elite martial artists out there.

Maybe I'm wrong, and maybe the VR world will fill with old duffers playing very realistic VR golf, walking about VR golf courses together and chatting about slicing their drives. But my guess is that most people will want to play games where they transcend their mundane abilities, and where their real level of physical training isn't at all reflected in the game.

We don't really want virtual reality. We want virtual fantasy.


OK, I follow all that, but I'd suggest that it's additional to my comment that you replied to?


We don't really want virtual reality. We want virtual fantasy.

I'd agree; the sorts of games I play typically have your avatar as a "superhero" rather than a spurtsman, even one capable of world class performances.


OK it's a bit gloomy, but have VR developers any kind of coordinated plan or doctrine worked out for the inevitable lawsuits? Suppose someone has a heart attack in Left Hand Path, or more likely someone's parents blame Left Hand Path for their kid's disturbed sleep patterns and underperformance at school?

I'm not sure "it's just a computer game" will work when there is so much hype about VR being a revolutionary new medium offering hitherto unattainable realism...


Survival horror haptics leading to diagnosed PTSD seems a likely one, too - especially as you really can't predict what'll mess you up until you're familiar with something.


This is pretty much what I'm getting at with e-sports - as a spectator sport, the latest Street Fighter incarnation (let alone KOF XIII) is just not like actual martial arts at all.

All the strategy is in how you're going to take their head apart, not their body. Though there's pretty major physical skill required, much as it's specialised. But nobody survives being on tilt for more than a few seconds at tournament level.


Because why would most people want *realistic* martial arts or e-sports in VR?

...and not just VR, 2D games as well.

Consider first-person shooters; they are utterly unrealistic in terms of handling a firearm. The ability to aim and fire goes so far beyond the ability of even an Olympic rifle shooter, it's not even close. Yet this aspect of any such game goes without comment.

As you say, it's about the fantasy of it. Thankfully, most people will never want, need, or try to shoot at another being.


Survival horror haptics leading to diagnosed PTSD seems a likely one, too

Not necessarily - the irony being that they're using VR, apparently with some success, to treat PTSD as part of Exposure Therapy...


Going back to the original point about Walkin' On (Virtual) Sunshine and the And Beyond!, I think there's an implicit assumption in the article that needs challenged. Namely, that the desired outcomes are VR that can be experienced within the home.


Surely, the short to medium term answer (for scenario-based experiences, at least) is to Augment reality, not to Virtualise it. As an example, consider indoor Laser Tag sites. You wear a harness with a gun; you move around inside a dedicated arena full of walls and ramps and steps and windowframes. Those that I've attended (yay, parenthood, being lasered by a bunch of eight-year-olds at a birthday party) have minimised the interior decoration and lighting. Haptic feedback doesn't need simulated, it just needs some augmentation. With wireless VR headsets and tailored controllers, you can do a far better job of simulating your FPS or horror scenario or game.

The interesting one will be if the various door-kickers in Police and Armed Forces start to use AR within their training facilities, or whether they just stick to Simunition and hidden speakers.

It comes back to the balance between the desire for Augmented Reality versus Virtual Fantasy, as expressed by Icehawk...


The way things work, the fact you can use it for treatment in a guided environment is evidence it reaches the right places to cause when there's an inevitable SNAFU though.

If you want to look at it another way: where do you consent to be touched by a VR system, when do you get the chance and can you withdraw consent at least as quickly as during BDSM play given that the VR system can't do aftercare?


I'll be interested to see if something akin to the Uncanny Valley effect (the Uncanny Island, perhaps?) develops with VR.

For me, I just don't see the appeal of having a full-immersion experience of swinging a zweihander in the 14th Century Italian Wars--the blood, the wounds, the smell, all my comrades dying around me--even if I get to swing a zweihander (which would be hard for me in real life).

Would I want to play through a realistic VR version of the Siege of Leningrad? Nope. Reenacting the Shackleton expedition? Nope. Launching with Armstrong and experiencing 3 gees? Nope. Exploring the Amazon with Alfred Russel Wallace? Nope. Sailing with Darwin? Nope (guy vomited all the time, although he was reportedly a friendly chap). There are some things where the abstraction of a game is welcome.

Worse, with realistic recreational VR, you've got any obvious problems in the VR detracting from the experience, the ever-present bandwidth issues involved in taking terabytes of reality and shoving it through an abstraction pipe, and reality being generally unsatisfactory in general (per Buddha).

I wonder if, at least with recreational VR, the happy medium will be around where Pixar is with cartoons: not too real, but fun. In this environment, some inconsistencies in sensory inputs are interpreted as play rather than as a defect, and the real point with recreational VR is playing.

As for other applications, VR's probably most useful for abstracting and rescaling space in useful ways. It might help biochemists with the perennial problem of trying to build a drug molecule that binds to a receptor and only that receptor (they've found that just running the physical experiment with huge numbers of candidate molecules tends to work better than modeling, at least so far), or help builders work in 3-D a bit better.

On the flip side, I suspect that, if VR becomes a bigger part of the business world, we'll see How to Lie with VR join the bookshelf next to How to Lie with Statistics and How to Lie with Maps. For instance, you may think that you'll get everything you need to know by touring a house in VR before you buy it. But did they erase the water damage and sagging floors from the VR tour? Easy to do, hard to catch.


Random thought, since I don't do VR: has anyone capitalized on the problems with VR and made a game where you play a ghost? Then issues like not being able to touch the world and being able to stick your head through walls would be a feature, not a bug. It could get challenging too: you could possess in-game characters, but you would have to stay with them. Stick your ghost head through a wall, and they dispossess you.


Augmented reality has its own set of technical challenges, mostly to do with getting the virtual elements to correctly line up with the real-world elements. You need to detect your real-world position with great speed and accuracy, and possibly recognize and react to changes in your real-world environment.

People are working on this too, of course, but it's not obviously easier than pure VR. (Well, depending on what range of experiences you expect each to replicate.)

Augmenting a specific, highly controlled environment like a laser-tag arena would be easier, but also greatly reduces your potential number of customers.


I'd also be more interested in AR than VR.
For one, there is more real world to augment than there are VR worlds. :-)
There is also research on creating VR worlds from real, either from photo + laserscans or just from photos.


> there is more real world to augment than there are VR worlds

By what metric?


I was inclined to say "volume", but I'll amend that to recognizable complexity.
VR worlds have to be build, so either you rely on algorithms which deliver less diversion, or you create them manually, which consumes a lot of time and imagination. Real world will also always have more ways to interact with it. Just imagine what it would take to recreate the experience of a kid exploring a small wood: trees to climb, leaves, stone and sticks on the ground, ability to use or break a stick, or throw a stone. Find bird nests with eggs in it, experience moss, collect flowers, build a damn in a little stream, play with sand or mud, .... VR always loses when it comes to breadth of experience. VR are good at telling stories and providing alternate environments (similar to books).
With AR you can overlay the real world with imagined objects, characters and stories (a la Ingress and Pokemon Go), thus combining both worlds.


Consider Guitar Hero.

Knowing how to play guitar may actually be a detriment here. I'm no virtuoso, but I've been playing for 50 years. I just can't get the hang of Guitar Hero



About this Entry

This page contains a single entry by Hugh Hancock published on November 13, 2017 1:32 PM.

Burn The Programmer! was the previous entry in this blog.

Are VR esports going to become a thing? is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Search this blog