Site-specific Immersive Storytelling and Near-Field Stop Motion for Head-mounted Displays
Howard Cook, CMTC Fellow

cmtc-immersive-storytelling-01.jpeg

Summary:
Research in Immersive Storytelling, the base of this project, focuses on researching the mutual influence of story, technology, and immersion. This pilot project critically explores the design challenges offered by the multitude of new opportunities in how we tell visual stories, how we use new technology and how we engage with our audience.

The project combines efforts across disciplines at the University of Colorado Denver with students from the College of Arts & Media, the Digital Animation Center (DAC) and the School of Engineering Computer Sciences. Students, faculty and researchers produce unique Immersive Cinematic Experiences that introduce the art of stop-motion animation and physical objects to the field of virtual reality. The goal is to create a surreal narrative experience in which the viewer is "immersed" in the story.

Problem that Inspired Research:
With the availability of consumer VR technology, the demand for meaningful content is rapidly expanding. Consumers using this tech cannot be expected to "figure out" how to experience a story. The stories must be crafted such that the viewer is "guided" through the experience. Like with traditional filmmaking, an artist must be in-between the user and the story to shape and make the visual story work for the viewer. Questions to be answered through this pilot project include: How do filmmakers tell visual stories in which the viewer can actively participate in the storytelling experience? Do the traditional rules of making a film apply to the new technology available to the filmmaker and consumer alike or is there a new language and grammar need to be developed? How does the visual storyteller use new types of narrative practices and technology to tell stories for this new media?

Objective / Proposed Solution:
Research in Immersive Storytelling is centered around the idea that in recent years we have seen the rise of numerous new types of narrative practices and purposes alongside traditional media. Rather than developing stories for one medium, today's professionals are very likely to design stories for all kinds of media, including computer and pervasive games, interactive web documentaries, virtual reality films, interactive installations, and performances. The goals are to begin to understand the challenges to telling immersive stories, both narratively and technologically, and then develop a methodology that expands on the traditions of filmmaking language to provide a new visual narrative toolkit.

Greatest Challenge to Overcome:
While the language and grammar of traditional filmmaking are well known, how that applies in the realm of the immersive experience is much less understood. Only recently has technology begun to emerge that is stable, usable and accessible to both consumer and artist. Sorting through the myriad of possible technological solutions and platforms has proven to be a challenge to developing a standard methodology and language that grounds the artist-designer, thus freeing them to create meaningful and sophisticated visual stories as you would find in traditional filmmaking.

Benefits of Research: 
By focusing on the producers of content, work in this area targets the development of a standard way to approach how immersive stories are created and told to shape the consumer's experience. Outcomes are aimed at helping artists take the foundations of traditional filmic storytelling and move them into the realm of creating content in which the viewer is moved beyond being a passive participant to instead be fully engaged in the experience of the story.

Real-World Application(s):
The goal of the project will be to accelerate innovation in the field of immersive storytelling and educate the next generation of researchers and practitioners. This pilot project brings together an interdisciplinary team of CU Denver faculty, graduate students and undergraduates working in 3D computer vision and perception, object recognition, graphics, storytelling and education, distributed computing, stream processing, databases, computer architecture, and privacy and security. The research and content produced will be positioned to service a wide range of fields such as Location Based Entertainment, motion graphics and design, medical and scientific visualization and augmentation, journalism and museums.

Innovations to Media and Technology:
Research in Immersive Storytelling focusses on researching the mutual influence of story, technology, and immersion. This pilot project explores the design challenges offered by the multitude of new opportunities in how we tell stories, how we use new technology and how we engage with our audience. The project is aimed at advancing state of the art in mixed, augmented and virtual reality.

Cutting-edge Technology Being Used:
While there is not any "cutting edge technology" used in this research project, the work will explore "cutting edge" applications of "bleeding edge" consumer and prosumer tech to develop new approaches and to creating immersive content. The goal is to find methods and strategies for including this consumer/prosumer tech in the immersive artist designer's toolkit.

Transdisciplinary Collaboration:
Research in Immersive Storytelling will critically explore the design challenges offered by the multitude of new opportunities in how we tell stories, how we use new technology and how we engage with our audience. CU Denver's College of Arts and Media, Comcast Technology and Media Center, College of Engineering and Applied Sciences and others will partner with leading technology companies on this pilot project to explore ways of advancing state of the art in mixed, augmented and virtual reality. This will pave the way for new academic and industry collaborations.