Periodic Reporting for period 2 - ENHANCEplayer (An Artificial Intelligence Enhancing Video Quality Locally to Limit Internet Traffic Tied to Video Streaming)
Berichtszeitraum: 2019-11-01 bis 2021-01-31
What is the problem/issue being addressed?
As live streaming on phones becomes more and more popular across the world, broadcasters and live stream content creators have a bandwidth challenge. ENHANCEplayer address this by moving some of the cost away from bandwidth to local processing using upres models developed by Artomatix.
Why is it important for society?
More information can be shared more widely by more people in real time. This has very clear ramifications in mobile first developing economies, where bandwidth is often a strong barrier to entry for companies sharing content that the majority of users will watch on mobile devices.
What are the overall objectives?
Overall objectives is to create upres models that will enable broadcasters to push livestream content at a lower resolution and then have the ENHANCEplayer feature upres it in real time at a high quality.
As live streaming on phones becomes more and more popular across the world, broadcasters and live stream content creators have a bandwidth challenge. ENHANCEplayer address this by moving some of the cost away from bandwidth to local processing using upres models developed by Artomatix.
Why is it important for society?
More information can be shared more widely by more people in real time. This has very clear ramifications in mobile first developing economies, where bandwidth is often a strong barrier to entry for companies sharing content that the majority of users will watch on mobile devices.
What are the overall objectives?
Overall objectives is to create upres models that will enable broadcasters to push livestream content at a lower resolution and then have the ENHANCEplayer feature upres it in real time at a high quality.
We were successfully able to achieve the following result:
We achieved an upscale of 240p to 480p at 25fps on a wide range of Android phones.
We achieved an upscale of 360p to 720p at 25fps on an iPhone 11, thanks to the performing neural processing unit in the iPhone 11.
We established an MVP, and a Marketing and Sales strategy, and developed all supporting material, and reached out to a first set of parties to pitch the ENHANCEplayer concept.
We prepared for a public demonstration at NAB, but unfortunately had to replace this by a virtual NAB with a pre-recorded demonstration. The same holds for IBC.
We prepared a final MVP definition, including roadmap going forward.
We collected feedback from real users.
Unfortunately, we were not able to overcome the technical difficulties on Android devices, and after trying a variety of potential solutions, we concluded that the technology is not yet ready for this solution.
We've put together a selection of recommendations for future research, as we see technology developing in the next two to three years that should enable this technology to be successful and have a strong market impact.
We achieved an upscale of 240p to 480p at 25fps on a wide range of Android phones.
We achieved an upscale of 360p to 720p at 25fps on an iPhone 11, thanks to the performing neural processing unit in the iPhone 11.
We established an MVP, and a Marketing and Sales strategy, and developed all supporting material, and reached out to a first set of parties to pitch the ENHANCEplayer concept.
We prepared for a public demonstration at NAB, but unfortunately had to replace this by a virtual NAB with a pre-recorded demonstration. The same holds for IBC.
We prepared a final MVP definition, including roadmap going forward.
We collected feedback from real users.
Unfortunately, we were not able to overcome the technical difficulties on Android devices, and after trying a variety of potential solutions, we concluded that the technology is not yet ready for this solution.
We've put together a selection of recommendations for future research, as we see technology developing in the next two to three years that should enable this technology to be successful and have a strong market impact.
Recommendations for future research:
These recommendations are not only for Android but the Enhance Player project.
Overall, we have encountered following challenges during the 2 years:
Upres Quality
Temporal Consistency
Limitation on model size
Model capacity
Compression Artifacts
Face definition
Text definition
Audience attention
Performance: real time processing for high definition targets (like 1080p)
25 FPS
HD targets
Our experiments successfully solved all challenges in Upres Quality category and achieved acceptable results for Performance on iPhone devices. For future research, our suggestion is to focus on performance to allow quicker market occupation.
Research direction 1: Integrate upres as part of video codec. Lots of codec like MPEG4 have key frames which can be utilised to reduce the needs of upres frame by frame. Before decoding the video, upres the key frame in the video stream only.
Research direction 2: Integrate upres as part of video displaying pipeline with customised acceleration chipset. Instead of upres the frames at computer side, this approach upres them at monitor or TV side with customised chipset just before displaying. Same hardware design can be placed in the graphic card as well.
Research direction 3: Customise the streaming pipeline to allow upresing on part of the video on older mobile devices and browsers. As the goal is to reduce bandwidth cost, given the large user base number of low-medium range mobile devices and browser players, enabling partial upresing will save a significant amount of bandwidth in overall.
These recommendations are not only for Android but the Enhance Player project.
Overall, we have encountered following challenges during the 2 years:
Upres Quality
Temporal Consistency
Limitation on model size
Model capacity
Compression Artifacts
Face definition
Text definition
Audience attention
Performance: real time processing for high definition targets (like 1080p)
25 FPS
HD targets
Our experiments successfully solved all challenges in Upres Quality category and achieved acceptable results for Performance on iPhone devices. For future research, our suggestion is to focus on performance to allow quicker market occupation.
Research direction 1: Integrate upres as part of video codec. Lots of codec like MPEG4 have key frames which can be utilised to reduce the needs of upres frame by frame. Before decoding the video, upres the key frame in the video stream only.
Research direction 2: Integrate upres as part of video displaying pipeline with customised acceleration chipset. Instead of upres the frames at computer side, this approach upres them at monitor or TV side with customised chipset just before displaying. Same hardware design can be placed in the graphic card as well.
Research direction 3: Customise the streaming pipeline to allow upresing on part of the video on older mobile devices and browsers. As the goal is to reduce bandwidth cost, given the large user base number of low-medium range mobile devices and browser players, enabling partial upresing will save a significant amount of bandwidth in overall.