Photorealistic avatars for more immersive VR experiences
Immersive is easily the first term that comes to mind when you think about Virtual Reality (VR). But there is a second one not too far behind: social. As the recent boom of social VR apps shows, immersion goes hand-in-hand with the capacity of users to interact just like they would in the real world. The EU-funded project VRTogether (An end-to-end system for the production and delivery of photorealistic social immersive virtual reality experiences) stands at this much-coveted crossroads between immersive and social VR. The project promises groundbreaking VR experiences built around ‘social photorealistic immersive content’, at an affordable cost. To cut a long story short, VRTogether wants VR users to just forget about the avatars they’ve had to deal with so far, and rather enter VR experiences looking exactly as they do in the real world. “We have focused our work on the creation of the tools necessary to take multiple users, capture them in real time as volumetric videos, and bring them in virtual environments. We now have an end-to-end lightweight holo-conferencing platform that pushes technological boundaries to seamlessly integrate people – or rather a digital version of themselves – in VR or Extended Reality (XR) environments,” Sergi Fernandez, Media and Internet director at i2CAT and coordinator of VRTogether, summarises. On the path to their photorealistic avatars, the project team left no stones unturned. “A platform, per se, is composed of multiple components that can both be used individually and add value to specific parts of the chain. We have generated tools for capture, streaming and communication between multiple users using volumetric video. These components may also serve for other purposes like pure volumetric video streaming for the next-generation contents or realistic capture of characters, to be later post-produced as VFX,” says Fernandez. Beyond the 3D capture and reconstruction of users as a point cloud, the consortium also developed: solutions to ensure low latency; a new compression format to save bandwidth; media orchestration to create consistent and synchronised experiences for all users; the capacity to broadcast content in real time; and protocols to make object interactions more worldly. Icing on the cake: the support of home usage with a low entry burden for users. The technology can be compared to that used in Hollywood CGI blockbusters, only at a fraction of the cost.
From gaming to dating applications
Potential applications are legion. In Fernandez’ own words, “VRTogether is the seed for the next generation of communication and collaboration tools where the boundaries between real and digital become blurred. Our technology can be applied to content watching, gaming, training, education, culture, industry, or anything in which human collaboration in hybrid or virtual environments requires a higher quality of experience than what current framed video calls offer.” To ensure these experiences will eventually come to life, the project team has developed three content pilots to demonstrate the technology and attract potential customers. In these pilots, the users can participate in a murder investigation. “It’s a threefold story with one chapter per pilot,” Fernandez explains. “Users have to discover the perpetrator of a crime and collaborate in groups of two to six to interrogate suspects at a police station, participate in a TV show, and inspect a crime scene.” Collaboration with partners interested in applying VRTogether technology to specific use cases has already begun, and the team expects first product commercialisation as early as 2022. Meanwhile, project partners have started working on project follow-ups. i2CAT for instance is developing two virtual escape rooms, as well as a watch-together-in-VR experience for a broadcaster. All in all, VRTogether brings us closer to live-like experiences in VR, where realism is pushed to the extreme.
Keywords
VRTogether, avatar, photorealistic, virtual reality, VR, social VR