Experiments in real-time hybrid performance: Innovating in the capture of facial expressions for live hybrid performances for the metaverse

MetaMirror is a technical framework created by Piotr Nierobisz at the creative technology studio, MunchingSquare, which was implemented through a collaborative research project in late 2022. The MetaMirror tool improves artists’ ability to communicate nonverbally to in-person and virtual audiences. MetaMirror projects a performer’s real-time facial expression on to an avatar in the virtual space, delivering improvements over flat 2D video representations or expressionless avatar figures moving in the space. The MetaMirror prototype is an experiment in allowing artists to perform in two spaces at once – a live performance venue and a virtual performance venue, accessed via Quest 2 headset or browser. This report presents the research conducted with the performers and audiences of the MetaMirror concert by Dr Laryssa Whittaker, StoryFutures Senior Audience Researcher. It finds that in-person and remote audiences do not necessarily want the same experience, and hybridity offers something unique to each audience, and the possibility of contact between them. Each environment affords different possibilities, and the hybrid space creates a portal between these possibilities for audiences.