As XR technologies are not yet widely used in the performing arts, there is still a need for expertise and resources in employing 3D modelling and VR technologies among creatives in theatre and dance CCIs. While there are a lot of solutions out there, the options of using them are not yet standardized, making their adoption an experimental and often costly endeavor. We focus on proposing a process for their use and developing an environment for their deployment, to make the use of XR technologies in the performing arts more accessible to venues and creators

The first aspect of what we are working on has to do with performance capture, and the creation of a protocol for the use of the different solutions. We have been looking around and testing several solutions that allow us to capture – record in a more traditional terminology – the different assets of a work: the performer’s motion, speech and facial expressions, the props and the lights of the stage, the music/sound that is played. 

The challenge here relies on making the right choice for the specific needs of a work and its production, as well as the distinct requirements for dance and theatre captures. Fundamental considerations have to do with available financial resources, the level of intrusiveness associated with a performance capture system that may affect the freedom of movement, the scale of a work and the number of performers, or the need for facial fidelity, are aspects that have to inform the final choice. 

Specifically, about motion capture, the CYENS team conducted tests with commercial motion capture suits (Xsens, Rokoko) together with dancers and choreographers from ICK, while the UJM team, focused on marker less motion capture techniques, with external cameras, and on creating a live Data Motion Dataset that introduces occlusion, lightening and facial expressions variations. 

Overall, performance capture in PREMIERE involves a) data collection of performers and their environment from the different physical locations, b) local synchronization of these data, c) transmission of data in standardized formats through a network to the virtual theatre state synchronization server which then d) ensures all performers experience the same state and e) storage of appropriate data into the CMS.

The 3D virtual theatre is the venue that will host all the applications of Premiere, including the VR media streaming platform, a streaming channel for the transmission of live VR events. In other words, it provides the stage to stream a performance with the audience connected from a remote location, or for the encounter of performers.

Our approach includes developing a networking architecture to facilitate real-time interaction between remote performers and the virtual environment, implementing efficient streaming capabilities to distribute the content to remote viewers via VR and web browsers, integrating human avatars animation and 3D scene modelling. Moreover, we’re tailoring user experiences based on their roles; audience members will enjoy distinct privileges and access rights compared to performers. The challenges include synchronisation of users, synchronisation of different data type, and finding the balance between image quality, latency and system resiliency.

Institutions and creators will find in Premiere both a guide for adopting and combining systems for capturing a theatre or dance performance, and an infrastructure that supports interaction between performers and VR streaming for the audience. This is how we wish to empower the organisations to host virtual environments, as an extended set for the creation and distribution, and to provide a stage where artists can create and the audience enjoy.