Tracking
All virtual performance relies on digital systems that capture and transmit a user’s movements. When performing as an Avatar this can best be thought of as digital puppetry. The OnBoardXR Community has experimented with a variety of systems and techniques for tracking and expressing a performer’s movements in virtual reality and welcome all artists to bring their own creative practive to this emerging medium, but strongly recommend Creator’s root their prototypes in the tools and methods already authentically integrated into their performance work.
Keystrokes
The simplest expression of any user is their use of a keyboard and mouse in the 2D Desktop experience. Several [Case Studies] by Creators were designed and performed without any advanced virtual reality equipment by careful practice to manipulate their Avatar using their existing computer or tablet. Combined with idle animations or non linear animations, these performances can become even more emotionally resonate.
Webcam Tracking
We can think of web cameras as another method of tracking a performer by capturing and transmiting the video of their performance as a “live feed.”
Some Creators have experimented [applying filters to a performer’s web camera] to create various effects. Community Member Koryn Wicks has even added filters to 360 video
Community Member Matt Rossman has explored using a performer’s web camera to track facial movements to puppet their avatar’s expressions, based on Bryan Pratte’s Join Hallway project.
VR Headset (HMD)
Virtual reality head mounted displays are the most consumer-friendly method for efficient avatar puppetry in virtual reality. Many of these systems offer users three (3) points of tracking using hardware worn on the user’s face and controllers to be held in each hand. Early studies have already shown that the biometric data captured by only these 3-points is expressive and distinct enough to identify a specific user by their movements. Dozens of OnBoardXR prototypes have demonstrated this method as sufficient to capture and community emotionally resonate performances in virtual relaity.
Hand, Eye, and Lip Tracking
Some head mounted displays offer accessories or updates to track more user data, including finger movements, eyes, and lips.
Full Body Capture
Community Member Clemence Debaig is recognized for her achievements in capturing and streaming full body motion caputure data from a Perception Neuron body suit into Mozilla Hubs. Many other artists across other VR platforms have similar integrations, all relying on bespoke data management systems and avatar rigging.
Below are some resources on various examples and integrations of motion caputure hardware and software for virtual performance.
- Unwanted Waters from Unwired Dance Theatre
- Christmas Carol from Heavenue
- VRChat tracking tested by Thrillist
Volumetric Streaming
Community Member Valencia James is celebrated for her dance performances using volumetric capture, which her team shared with OnBoardXR during her time at the New York University Innovation Lab for students to develop and perform prototypes using her techniques. While captured from an individual depth sensor, this device can generate a real-time, three-dimensional map of the user and their environment. Creators can choose how much of this data to transmit, resulting in the ability to separate a Performer from objects in the room that are closer or further from the sensor than the Performer’s body.