by James Mathers
Cinematographer and Founder of the Digital Cinema Society

(Excerpted from the Digital Cinema Society eNewsletter, July 2016)

Into the Future: Harnessing the Power of VR for Storytelling

Although I’ve written about Virtual Reality previously, it is such a quickly evolving space, that I think it deserves an update.  I’ve been researching the latest developments in preparation for SIGGRAPH, coming up later this month.  I’ll be moderating a panel produced by NVIDIA on The Future of Storytelling in VR, and following are some of my recent observations. 

There is still a lot of confusion about what exactly VR is.  It’s a term that means different things to different people, but can broadly be described as a means of placing the viewer into an environment by surrounding them both visually and aurally.  One thing for sure is that it is hard to convey the experience if you haven’t tried it.

The various formats for entering that environment can be via head-mounted display, whether that be an Oculus Rift-type system interfaced with a more powerful computer, or a simple attachment added to your smart phone such as Google’s Cardboard. 

However, resolution needs to increase in order for images to look photo realistic when they are being magnified and viewed only inches from your face.  Even if the master image is 4K, it has to be wrapped around, and “stretched” to cover the potential field of view, since you are only looking at a small percentage of that overall resolution as you scan around the environment.  Seeing pixels is a turn-off that tends to quickly take the viewer our of the virtual environment.

Although VR can also be experienced in various theatrical viewing environments such as a dome or panoramic screen, for now it seems that headsets are the most practical delivery medium.  The world of gaming is currently the most advanced forum, probably due to the fact that game developers already have experience creating interactive immersive environments, and the capabilities for spacial awareness are quite advanced, so that is where we are seeing the most progress.

As evidenced by Paul Walker’s appearance in “Furious 7,” where much of the coverage was created after his untimely death, we already have the ability to create photorealistic human beings in 2D.  However, creating that same illusion in VR is a much bigger challenge, especially considering the myriad of moving perspectives that need to be maintained.  It takes an enormous amount of computer horsepower, but the good news is that while calculating the interaction of every light ray and every material on an object has been impractical in the past, now GPUs are getting so fast that it will soon be possible to do it interactively in near real time.

While I’m not a big fan of higher frame rates for Digital Cinema as used, for example, in Peter Jackson’s “The Hobbit,” I do feel that there is a benefit for VR.  The smoother the movement, the more natural, realistic, and immersive the environment becomes.  Of course, that is just one more element adding to the data payload and computational challenge.

The rate of technological evolution in these areas is exponential, however, and as computer generated imagery becomes ever more photorealistic and resolution and rendering speeds increase, the power to create an engaging environment will only get better and better.   Moving forward, it is easy to see the vast array of potential applications, from education to various forms of entertainment. Nevertheless, even as we conquer the technological hurdles, cinematic storytelling remains an even bigger challenge.

A whole new storytelling language needs to be invented.  Where traditionally, the Director, through the selective eye of the two dimensional camera and editing, could control where the audience looks, in VR the viewer is free to meander around the whole environment.  This could lead to one viewer experiencing a completely different scenario than the next.

Done poorly, the experience can cause fatigue, discomfort and even motion sickness.  Traditional cutting can be quite jarring in VR, and many of our much relied on tools, like the close up or shallow depth of field, are out the window.   New ways of cueing the audience where to look will need to be developed.

Another challenge is that while many educational applications might benefit from having the viewer isolated from any outside distraction, cinema is usually best enjoyed as a social experience.  A comedy is never funnier than when the whole theater erupts in laugher and the tension in a scary movie is always heightened when the audience screams or gasps in horror.

VR is a nascent medium that will only get better and become more visceral as we learn how to use it.  I’ll be able to see where the technology currently stands when I visit SIGGRAPH in a couple of weeks, but I think the bigger challenge will be on the creative side of the equation.  I’m guessing the only way we’re really going to learn to tell stories in VR is to get out there and try it.  I’ll be looking for opportunities to further explore this world and hone my skills in real world content creation situations.  As I do, you can be sure you’ll be able to read about my experiences here…So, stay tuned to DCS; we are just starting on the path to VR storytelling.