Realatars will soon be making their way to our screens
Ensuring that those in the deaf community have access to information, communication and knowledge is vital for their full and equal participation in society. The Audiovisual Media Services Directive aims to ensure that EU Member States make certain that media services contribute to equality and accessibility regarding content distribution and accessibility. However, producing sign language content is costly, with many companies offering only a small number of sign-presented programmes that are usually aired late at night. The EU-funded Content4All project set out to change this and make content more accessible for the deaf community through an automatic sign translation workflow with a photorealistic 3D human avatar for TV broadcasting. “We had two main goals. The first was providing a low-cost solution to create sign-interpreted versions of content produced for hearing audiences, without impacting on the viewing experience of current users. The second was to create data sets and algorithms to explore automated sign-interpreted content creation,” notes Giacomo Inches, project coordinator and innovation technologist at Fincons Group.
Meet realatar, the avatar
The avatar is achieved through an innovative multicamera studio where the movements of real people are captured and then processed with the help of AI algorithms. The end result is a 3D photorealistic avatar, named realatar, that can be distributed through video stream. Inches further explains: “While automatic sign language generation was a technological exploration in the lab, not meant for the market, we looked at the new concept, ‘remote studio’ for broadcasters, in which sign language interpreters could perform their job without needing to travel to the main premise of the broadcaster.” The time saved could then be employed to generate more signed content. The technologies used in the project relied on advanced deep learning and machine learning algorithms. They enable a computer to observe a large amount of data, and from instructions via examples they provide a desired outcome. “In the project, the algorithms helped reproduce a real person as a virtual human in real time, on a dedicated HbbTV app,” adds Inches. This solution gives TV broadcasters an affordable and sustainable way of developing sign language programmes, ultimately opening the door to increased production.
Moving forward with notable achievements
“The innovative nature of this concept was awarded the prestigious NAB Technology Innovation Award at the Broadcast Engineering and Information Technology Conference in 2020,” says Inches. Additionally, with the help of their partner representative organisations, the project was able to engage with members from the deaf community in Belgium and Switzerland to validate the technological components. Discussions from this enabled the project to launch an initiative under the umbrella of the COST Action, LEAD-ME, that brings together European stakeholders in the field of media accessibility. Two other projects, EASIER and SignON, will exploit and extend the Content4All project legacy, directly employing the signed collection. Furthermore, the Fincons Group, coordinator and lead industrial partner, is exploring concrete ways to exploit the project results for their network. Inches concludes: “In the long term, a concrete outcome of the project would be the employment of the collection of text aligned with sign language videos for research and pre-commercial purposes, and therefore contribute to the future developments of all the algorithms for sign language recognition.”
Keywords
Content4All, sign language, avatar, deaf community, media services, realatar, 3D human avatar, TV broadcasting, media accessibility