Periodic Reporting for period 2 - Immersify (Audiovisual Technologies for Next Generation Immersive Media)
Période du rapport: 2018-10-01 au 2020-06-30
Virtual reality (VR) has emerged as a new form of content and experience where the user is completely involved in the content and can interact with elements of the virtual world. VR is able to provide immersive and interactive experiences that are not possible with current content formats.
As the production of immersive media content is continuously increasing, the type of platforms for consuming this new type of content is increasing as well. Not only Head-Mounted Displays for 360° video are already available, but new immersive display technologies for groups of audiences are emerging including multi-display setups, immersive cinemas, and dome projections.
Several advances in media technologies are being developed in order to deliver the next generation of immersive media experiences including UHD video (8K and beyond), High Frame Rate, High Dynamic Range, Wide Color Gamut, higher Field of View, seamless interactive personalized data, Immersive Audio, etc.
However, the video and audio compression and playback tools must also be enhanced to properly process this new immersive content. And this is the aim of Immersify: to advance the tools required for the distribution and exhibition of next-generation immersive media.
- To improve the quality of immersive media using advanced compression technology we implemented and optimized HEVC encoders and decoders, video rendering engines for 8K, 16K and 360° and Ambisonic sound.
- To enable immersive media for multiple display environments and multiple devices we extended rendering engines and video processing tools as well as implemented synchronization mechanisms.
- To support new ways in interactive experiences by providing the required tools for personalized and interactive non-linear storytelling we focused on integration with game engines such as Unity3d and Unreal.
- To promote immersive media content and tools in the creative and media industries in Europe we performed many real-life demonstrations of next generation immersive media during global events like art and film festivals, industry trade-fairs or conferences, we created a set of new immersive content freely available for stakeholders and shared experiences on producing immersive content.
As the production of immersive media content is continuously increasing, the type of platforms for consuming this new type of content is increasing as well. Not only Head-Mounted Displays for 360° video are already available, but new immersive display technologies for groups of audiences are emerging including multi-display setups, immersive cinemas, and dome projections.
Several advances in media technologies are being developed in order to deliver the next generation of immersive media experiences including UHD video (8K and beyond), High Frame Rate, High Dynamic Range, Wide Color Gamut, higher Field of View, seamless interactive personalized data, Immersive Audio, etc.
However, the video and audio compression and playback tools must also be enhanced to properly process this new immersive content. And this is the aim of Immersify: to advance the tools required for the distribution and exhibition of next-generation immersive media.
- To improve the quality of immersive media using advanced compression technology we implemented and optimized HEVC encoders and decoders, video rendering engines for 8K, 16K and 360° and Ambisonic sound.
- To enable immersive media for multiple display environments and multiple devices we extended rendering engines and video processing tools as well as implemented synchronization mechanisms.
- To support new ways in interactive experiences by providing the required tools for personalized and interactive non-linear storytelling we focused on integration with game engines such as Unity3d and Unreal.
- To promote immersive media content and tools in the creative and media industries in Europe we performed many real-life demonstrations of next generation immersive media during global events like art and film festivals, industry trade-fairs or conferences, we created a set of new immersive content freely available for stakeholders and shared experiences on producing immersive content.
The work performed in the Immersify project covered several main topics:
- Encoding and streaming: Enhancements in the HEVC/H.265 encoder for visual quality and encoding speed; support for 360° videos; testing the quality of experience; comparison with other 8K encoders; development and integration of Spin Enc Live, NVenc and Advantech 8K encoders; CDN live distribution of 8K and low-latency streaming using RTP/SRT.
- Immersive displays: Performance optimization of the HEVC video decoder and video render engine to enable fluent playback of video formats in 8K and beyond; testing and integrations of the enhanced decoding and rendering capabilities of display systems; and development of the first 16Kp60 playback system on a single workstation; HOA decoding and rendering.
- Creative development and demonstration: 50 video clips in various technologies have been produced in various technologies - 16K, 8K, 360°, 3D, interactive, Ambisonics. Additionally, several improvements and extensions have been done for interactive media applications based on the Unity3D and Unreal.
- Dissemination: The project is being promoted through its website, social media and press releases. The developed tools have been presented at key trade shows (NAB in Las Vegas, IBC and ISE in Amsterdam) and festivals (Festival de Cannes and Ars Electronica Festival). More than 375.000 people have been reached – including many professionals in the media industry. Six technical papers on international conferences were published and several non-technical articles have been released.
- Exploitation: Immersify has developed three main results that have been identified as exploitable foreground: codec and player tools, immersive spaces and installations, and immersive content.
- Encoding and streaming: Enhancements in the HEVC/H.265 encoder for visual quality and encoding speed; support for 360° videos; testing the quality of experience; comparison with other 8K encoders; development and integration of Spin Enc Live, NVenc and Advantech 8K encoders; CDN live distribution of 8K and low-latency streaming using RTP/SRT.
- Immersive displays: Performance optimization of the HEVC video decoder and video render engine to enable fluent playback of video formats in 8K and beyond; testing and integrations of the enhanced decoding and rendering capabilities of display systems; and development of the first 16Kp60 playback system on a single workstation; HOA decoding and rendering.
- Creative development and demonstration: 50 video clips in various technologies have been produced in various technologies - 16K, 8K, 360°, 3D, interactive, Ambisonics. Additionally, several improvements and extensions have been done for interactive media applications based on the Unity3D and Unreal.
- Dissemination: The project is being promoted through its website, social media and press releases. The developed tools have been presented at key trade shows (NAB in Las Vegas, IBC and ISE in Amsterdam) and festivals (Festival de Cannes and Ars Electronica Festival). More than 375.000 people have been reached – including many professionals in the media industry. Six technical papers on international conferences were published and several non-technical articles have been released.
- Exploitation: Immersify has developed three main results that have been identified as exploitable foreground: codec and player tools, immersive spaces and installations, and immersive content.
Results of the project and potential impacts:
- The main impact of Immersify will be to facilitate the adoption of the new generation of VR and immersive media by the media industry by providing tools (video compression and media playback), creative content development tools and guidelines for content creation, delivery and playback.
- The development of innovative solutions that will have an impact across the media value chain including: production companies, creative studios and content exhibitors
- Tools that allow to combine ultra-high quality video with interactive elements for creating more interactive and personalized experiences.
Progress beyond the state-of-the-art
- HEVC decoder and video render for 16K at 60 fps, also for 360° content.
- HEVC encoder for high-quality immersive content (8K and beyond)
with minimal compression losses,
- Live 8K streaming for the European industry to advance towards a future deployment of new immersive formats for live events, including 8K broadcasting and ILE. During the project, the 8K encoding, streaming, and playback technologies were tested and validated with several live demonstrations.
- Immersive media visualisation: support of playback up to 16Kx8K/60 fps on a single workstation as well as providing multi-machine synchronization methods. This brought us close to the final goal of development of compact rendering solutions for heterogeneous immersive spaces.
- Ambisonics: an end-to-end workflow has been developed, from production to playback, based on Ambisonics of up to 7th order, on heterogeneous loudspeaker layouts with user interaction.
Exploitation and use
The consortium partners are exploiting the current results of the project (performance optimized HEVC decoder and video renderer, enhanced HEVC encoder, new immersive content) making impacts in different industry areas including: museums and art festivals, 8K broadcast, 360° video, and large screen visualization.
Spin Digital is launching new versions of its commercial video solutions (Spin Player, Spin Enc, Spin SDK) including the enhancements being developed in the project. Spin Digital is actively promoting the new tools through demos at international trade-fairs, and by establishing partnerships with key industry stakeholders.
Ars Electronica and PSNC are using the new SDK to enhance their playback systems and the new encoder to produce high quality immersive content. Several demonstrations were performed at some of the key dissemination events including: NAB, IBC, InterBEE, Festival de Cannes, Ars Electronica Festival, etc. At some of these events the consortium has presented unique demonstrations such as the first 16K video playback or HOA.
Socio-economic impact and wider societal implications
Apart from the commercial exploitation, Immersify also expects to make a social impact on the audiences of the film festivals, museum, and science centers. The Deep Space installation at Ars Electronica and the dome installation at the Visualization Center C attract hundreds of thousands of visitors per year, who enjoy scientific, cultural, and artistics experiences. With the enhancements developed in this project, the user experience is becoming richer: higher quality content and more user-content interactivity.
- The main impact of Immersify will be to facilitate the adoption of the new generation of VR and immersive media by the media industry by providing tools (video compression and media playback), creative content development tools and guidelines for content creation, delivery and playback.
- The development of innovative solutions that will have an impact across the media value chain including: production companies, creative studios and content exhibitors
- Tools that allow to combine ultra-high quality video with interactive elements for creating more interactive and personalized experiences.
Progress beyond the state-of-the-art
- HEVC decoder and video render for 16K at 60 fps, also for 360° content.
- HEVC encoder for high-quality immersive content (8K and beyond)
with minimal compression losses,
- Live 8K streaming for the European industry to advance towards a future deployment of new immersive formats for live events, including 8K broadcasting and ILE. During the project, the 8K encoding, streaming, and playback technologies were tested and validated with several live demonstrations.
- Immersive media visualisation: support of playback up to 16Kx8K/60 fps on a single workstation as well as providing multi-machine synchronization methods. This brought us close to the final goal of development of compact rendering solutions for heterogeneous immersive spaces.
- Ambisonics: an end-to-end workflow has been developed, from production to playback, based on Ambisonics of up to 7th order, on heterogeneous loudspeaker layouts with user interaction.
Exploitation and use
The consortium partners are exploiting the current results of the project (performance optimized HEVC decoder and video renderer, enhanced HEVC encoder, new immersive content) making impacts in different industry areas including: museums and art festivals, 8K broadcast, 360° video, and large screen visualization.
Spin Digital is launching new versions of its commercial video solutions (Spin Player, Spin Enc, Spin SDK) including the enhancements being developed in the project. Spin Digital is actively promoting the new tools through demos at international trade-fairs, and by establishing partnerships with key industry stakeholders.
Ars Electronica and PSNC are using the new SDK to enhance their playback systems and the new encoder to produce high quality immersive content. Several demonstrations were performed at some of the key dissemination events including: NAB, IBC, InterBEE, Festival de Cannes, Ars Electronica Festival, etc. At some of these events the consortium has presented unique demonstrations such as the first 16K video playback or HOA.
Socio-economic impact and wider societal implications
Apart from the commercial exploitation, Immersify also expects to make a social impact on the audiences of the film festivals, museum, and science centers. The Deep Space installation at Ars Electronica and the dome installation at the Visualization Center C attract hundreds of thousands of visitors per year, who enjoy scientific, cultural, and artistics experiences. With the enhancements developed in this project, the user experience is becoming richer: higher quality content and more user-content interactivity.