Introducing Immersive Cinema

The idea of an interactive story is one that I've been fascinated by since I was a kid. Acting out the plots of our favorite films such as Star Wars, Ben Hur, or Monty Python and the Holy Grail, we became a part of those story worlds. The fun we had became the basis of my career in filmmaking and set me on a journey to discover new ways of creating an interactive narrative through visual effects, virtual reality, and the amazing tool of the real-time game engine. Combined together, these techniques are allowing us to create what I like to call Immersive Cinema.

If you are new to the concept of game engines, it’s just a word for a piece of software that allows you to build and create a whole variety of interactions for the viewer or player. Any game out there, whether it’s Super Mario Brothers or Fortnite uses a system of programming and logic to allow the player to interact with what is happening on screen. This allows for a rich experience for the viewer, where they are not just a passive viewer of the content but now they are an active participant where actions can have both meaning and consequences.

Immersive Cinema takes the flexibility of a game engine and when combined with Full Motion Video, a.k.a. FMV, you can allow your viewer to interact and manipulate what is happening on screen. This new approach allows us as content creators to build stories that are responsive to the actions of the viewer. This opportunity to bend and affect the story becomes a tool that we can leverage in ways that are surprising, reflective, and encourage the viewer to experience the story in different ways on repeated play throughs.

When you build interactions into the story it becomes more dynamic, reacting to the viewer's desires or inputs and giving creators another tool for subverting expectations while encouraging deeper immersion into the world of the story. In the past few years more studios and creators have been using a blend of video and interactive elements to explore how these stories might work.

You may have heard or played the Black Mirror episode “Bandersnatch” that was produced by Netflix. Their approach was to build a branching narrative, where the viewer was presented a handful of choices as a list of text based options that would then trigger different video clips based on what was chosen. Echoing the “Choose Your Own Adventure” books of the 1980’s, this video experience had many of the same pitfalls of the books where certain choices lead to early dead ends and only by choosing specific paths would the viewer be presented with a satisfying ending to the episode.

More interesting experiments with FMV have been explored by studios such as Flavorworks, with their project “Erica”. This story takes you through the mystery surrounding the protagonists family and allows the user to not only choose from text options or responses to trigger different branches of the story, but also allows for limited interactions with specific objects and the ability for the user to choose different directions such as going down a hallway or not. The interactions are more complex and provide the user with a better sense of agency as you feel like you are really affecting the story as you pull open drawers and get to know the other characters that inhabit this world.

Another approach altogether for an FMV experience is “Immortality” by Sam Barlow. Instead of being a character inside the world of the story, the viewer takes on the role of a video editor of sorts, who is able to play different clips from three lost films that have an actor who has also disappeared. You are able to discover new pieces of footage and learn more about her mysterious disappearance as you scrub the footage back and forth and click on objects that are mirrored in other clips. While not presented as a traditional narrative, the act of piecing together the information in the video clips gives the viewer the sense of discovering the story in a personal and unique way while the FMV video allows for some of the best performances in a game-like experience that have ever been produced.

Building on the ideas found within projects like Erica and Immortality, VOYRE has been developing Immersive Cinema techniques based on our expertise in 360 and 180 video. We start by capturing video content that is intended for virtual reality. This format allows us to create an immersive environment all around our viewer, giving them a sense of presence in the film where we can simultaneously capture authentic performances with actors and create scenes and environments that look and feel real.

As VOYRE creates Immersive Cinema specifically for Virtual Reality headsets, the advantages of using video based content allows us to optimize the relatively low processing power of stand alone VR headsets and Spatial Computing devices to deliver a high fidelity and immersive experience that feels real. If you have had experience with recent VR devices, you may have noticed that games and other applications tend to have a cartoony or low-fidelity look to them. The reason games and apps look a bit old-fashioned is the challenges of rendering high quality computer generated graphics on what is essentially a souped up smart phone. The technical challenge is very hard to show a high resolution image that runs fast enough to give a smooth and comfortable experience for the viewer. VR video allows us to sidestep this issue to some extent, by being able to playback pre-captured and pre-rendered content at a comfortable frame rate. VR video content is a much more efficient method for reproducing photorealistic visuals that efficiently utilize the resources of the headset to create the best experience possible.

Another big advantage is being able to capture human actors as they appear. There is a problem called the “uncanny valley” that rears its ugly head anytime you try to create a synthetic or CGI human character. While the technology has greatly improved over the years, many people still find CGI humans to seem weird and not completely believable. When you are trying to tell an immersive story, CGI characters can become a distraction that breaks the suspension of disbelief within the viewer and they will either consciously or subconsciously be trying to figure out why that CGI character seems off. By capturing actors with VR cameras we are able to capture a performance in the most realistic way possible. While we are excited to continue exploring the roles and uses of CGI based human characters in our stories, the cost and technical challenge they present to do well will most likely take many more years to realize when considering making content that is feature length or episodic. The additional benefit is to allow the actor to perform in a way that they are already familiar with. To capture a performance to use with a CGI character, you need to use motion capture technology which requires special suits and equipment and even the state of art capture systems currently available still tend to not be able to capture the beautiful subtlety that an actor can bring to a character.

Example of a digital human from Mass Effect: Andromeda (2017)

By using 360 and 180 degree video capture, we can solve that issue in a cost effective way, but we still need to consider how the viewer is going to interact with the story. When you display video in a VR headset, it’s essentially video that is playing back in a sphere. You can think of it as a fishbowl that has been placed over your head that you can look around in. We can even make it appear 3D by capturing it in stereoscopic. Stereoscopic is a fancy way of saying that we have two different viewpoints that we capture and are displaying to the viewer. Your right eye gets one image, and your left eye gets a very similar image but that is captured from a slightly different perspective. When these two images are displayed to the viewer their mind combines the two into one experience and because the images have parallax, a slight difference in perspective, your mind can detect and see the depth cues in a scene. This is the same general principle that has powered 3D movies since the early days of cinema, and the modern take on this uses the same principles but allows the user wearing a VR headset to also be able to look around the scene. When captured correctly the stereoscopic effect is an impressive sense of presence in the viewer, where they can experience and understand the depth in the scene and they feel like they are in the same place that the camera had recorded from. We like to say this is the super power of VR video, that the VR camera becomes like a transporter from Star Trek, the viewer can see and experience a place just as if they were standing there in real life and we can beam them to any location we can dream of.

As creators we can leverage this sense of presence to take our viewer to places they could never have imagined, a sci-fi world becomes tangible and real in a way that only VR can reproduce at scale. A life of luxury or poverty can be communicated and experienced, or daring feats and scary situations become heart-stopping as you just might believe it’s actually happening in that moment. The sense of presence that VR video can reproduce is unlike any other medium that has come before it. Our challenge then becomes how can we creatively use this sense of presence to move the story forward?

The solution we are encouraging brings in 3D objects into the game engine that the user can reach out and manipulate. By limiting a bulk of the interaction to objects that are within arms reach we can design each interactive element to complement the narrative that encourages deeper immersion by the viewer. For example instead of choosing from a text list of options that are crudely presented, a more diegetic approach to decision making could include choosing to break a window or not, breaking the window could allow the story to branch one way and might say something about the character we are inhibiting while not breaking a window could reveal an alternative path that compliments a more timid or stealthy engagement.

By designing interactions that feel a part of our story world, we can challenge our viewers in surprising ways while not blocking the flow of pacing of the story by having text dialogs or other immersion breaking decision mechanics. The pacing and editing of the narrative is a core element of cinematic storytelling that also needs to be seriously considered in the design of an immersive cinema experience. Too often the user can be stuck in a loop where they are required to solve a puzzle-like mechanic or where there is a pause in the action to allow the user time to choose a response from a list of options. To keep the pacing of the story moving and exciting, we propose creators to develop what we call “windows of opportunity” where the viewer can choose actions but with limited time to make a choice. When properly accounted for, these window of opportunity decision points can encourage the viewer to act while allowing for a passive or non-choice to be a valid path to the next scene. The pacing and timing of a story is a critical element that needs to be designed into any interactive element, we should strive to encourage interaction where appropriate while never impeding the pace of the story.

Designing a branching narrative comes with difficulties, rooted in a concept called the “Paradox Of Choice” coined by psychologist Barry Schwartz which posits that as the number of choices increases, the satisfaction of the user decreases. Related to this, is the other concept of the “Narrative Paradox” by Stephanie Riggs that the more agency you provide the viewer through the less control the creator has over the narrative the author wishes to express. These paradoxes need to be evaluated and leveraged as an Immersive Cinema project is designed. How much freedom and choice a creator gives their audience is a tool that you can use to your advantage, but a poorly considered choice or too much choice could work against the goals of the project. During development of the story each interaction should be considered from this perspective. Should the user have more freedom of choice at this point in the story? How would that affect the pacing of the narrative? And how actively do we want the user to participate in this moment?

The question of how much interaction is required and how often will we require the viewer to interact, should both be questions asked of each moment of the story. Not every scene needs to be interactive and not every choice needs to dramatically affect the story. Also as the creators will need to build or design every possible outcome that is offered to the user, so it becomes necessary to only allow the user to explore specific branches of the story, just to keep the production and scope of the project manageable.

Taken altogether combining the elements of VR video and interactive design become a new medium for expressing ideas. Our ongoing research and development in this area has become the most promising and important work that VOYRE has been involved with. We see Immersive Cinema as the natural evolution for narratives and believe that by laying the foundation for this new medium we’ll be able to empower the next generation of storytellers.

Scott Lynch
26/08/2024