Periodic Reporting for period 2 - BrainControl (The first mental joystick based on Artificial Intelligence for Multi-Modal Human Machine Interaction)
Période du rapport: 2021-06-01 au 2022-10-31
People suffering from severe disabilities have problems to communicate via the normally available channels (voice, writing, signs) due to motor degeneration or brain damage. Those problems to communicate, decrease their autonomy, independence and other daily life activities.
As disabled people have specific requirements for their home environment and its functionalities, their demands increase and change with growing age - especially when their health status starts to worsen- and they experience the evolution of the impairment. An important aspect for all people having the need to be supported in their daily-life-activities is to remain integrated into social life despite their age and existing disabilities
Specifically we are addressing the following points:
• Technological void for most locked-in patients, those who cannot use eye-tracking systems or other assistive technologies available on the market.
• Lack of personalized and adaptive interfaces meeting user’s needs, which change in the undetermined-unknown way as the disease evolves.
• Lack of accurate tools for diagnosis to support physicians, hospitals and national health systems to provide value care services to their patients.
• Lack of tools to support relatives and informal caregivers to attend the patients, improving also their quality of life and working towards improving patient’s social integration and access to digital services.
All the above points have been met during the project period thanks to release of following solutions, all addressed specifically to LIS/CLIS (locked-in) people:
• New BrainControl AAC (communicator) with increased stability and improved user experience, both in BCI as well as Sensory version. The version is already available on the market and used by few patients.
• Software solution (API) to allow people in LIS (locked-in) to pilot a wheelchair: solution presented to 2 wheelchairs producers during 2021.
• BrainControl Avatar, the solution to drive a robotic alter ego in an independent way
• BrainControl Smart, a diagnostic platform containing tests and exercise to perform assessment and rehabilitation in the cognitive and visual ability area
The overall objective is to expand the usage of BrainControl software and solutions to as many people (specifically addressing the ones in Locked-in status or with higher mobility impairment) as possible in order to reduce the gap that the impairment is causing, both for communication, which is definitely vital, as well as for a better mobility and also to give a better instrument to doctors to perform a better diagnosis.
As disabled people have specific requirements for their home environment and its functionalities, their demands increase and change with growing age - especially when their health status starts to worsen- and they experience the evolution of the impairment. An important aspect for all people having the need to be supported in their daily-life-activities is to remain integrated into social life despite their age and existing disabilities
Specifically we are addressing the following points:
• Technological void for most locked-in patients, those who cannot use eye-tracking systems or other assistive technologies available on the market.
• Lack of personalized and adaptive interfaces meeting user’s needs, which change in the undetermined-unknown way as the disease evolves.
• Lack of accurate tools for diagnosis to support physicians, hospitals and national health systems to provide value care services to their patients.
• Lack of tools to support relatives and informal caregivers to attend the patients, improving also their quality of life and working towards improving patient’s social integration and access to digital services.
All the above points have been met during the project period thanks to release of following solutions, all addressed specifically to LIS/CLIS (locked-in) people:
• New BrainControl AAC (communicator) with increased stability and improved user experience, both in BCI as well as Sensory version. The version is already available on the market and used by few patients.
• Software solution (API) to allow people in LIS (locked-in) to pilot a wheelchair: solution presented to 2 wheelchairs producers during 2021.
• BrainControl Avatar, the solution to drive a robotic alter ego in an independent way
• BrainControl Smart, a diagnostic platform containing tests and exercise to perform assessment and rehabilitation in the cognitive and visual ability area
The overall objective is to expand the usage of BrainControl software and solutions to as many people (specifically addressing the ones in Locked-in status or with higher mobility impairment) as possible in order to reduce the gap that the impairment is causing, both for communication, which is definitely vital, as well as for a better mobility and also to give a better instrument to doctors to perform a better diagnosis.
The achievements at the end of the project are the following:
• Release in September 2021 for the current users and in October 2021 for the market of the new version of BrainControl BCI, an Augmentative and Alternative Communication software allowing locked-in patients to be able to communicate again with the world. The new release includes feature like: Social network integration, Instant messaging, Web browsing, Entertainment (eg Kindle access), Home Control (eg interaction with Alexa and GoogleMini) etc.
• API related wheelchair finalized to ensure possibility to drive a wheelchair through thoughts or gaze and has been presented to 2 important producers, though the interest is rather limited for the moment. Still the solution is ready and further producers will be contacted in the close future.
• The solution regarding Robotics control (BrainControl Avatar) is completed, tested, launched and already sold to two customers (2 Museums in Siena). The solution allows people with severe disabilities to drive a robotic alter ego in an independent way.
• Finally in September 2022 the new diagnostic platform, BrainControl Smart, was released. The new solution includes tests to carry out an assessment of cognitive aspects and visual skills. It has been specifically designed for people with high impairment (LIS/CLIS patient) but can be used by any type of patient thanks to a number of interaction modes included, from a traditional mouse to a BCI helmet.
• In the development process of all the different solutions internal testing have been carried out both for debug purposes as well as to check the adherence to the use cases. This has also allowed us to launch new version of BrainControl AAC 2.0 in October 2021 and the new BrainControl Smart in September 2022 and to have them on the market.
• Administrative and Organisational Management: a new organization has been designed, with clear roles and responsibilities, with high focus on the R&D Department; a new CEO has been appointed with solid management and commercial experience (as recommended); 2 senior consultants have been hired to support the creation and implementation of a comprehensive marketing and communication plan; a scientific committee has been created to increase the competence and speed-up the development; finally some activities has been outsourced to 2 Universities in order to boost the development of the solutions.
• Release in September 2021 for the current users and in October 2021 for the market of the new version of BrainControl BCI, an Augmentative and Alternative Communication software allowing locked-in patients to be able to communicate again with the world. The new release includes feature like: Social network integration, Instant messaging, Web browsing, Entertainment (eg Kindle access), Home Control (eg interaction with Alexa and GoogleMini) etc.
• API related wheelchair finalized to ensure possibility to drive a wheelchair through thoughts or gaze and has been presented to 2 important producers, though the interest is rather limited for the moment. Still the solution is ready and further producers will be contacted in the close future.
• The solution regarding Robotics control (BrainControl Avatar) is completed, tested, launched and already sold to two customers (2 Museums in Siena). The solution allows people with severe disabilities to drive a robotic alter ego in an independent way.
• Finally in September 2022 the new diagnostic platform, BrainControl Smart, was released. The new solution includes tests to carry out an assessment of cognitive aspects and visual skills. It has been specifically designed for people with high impairment (LIS/CLIS patient) but can be used by any type of patient thanks to a number of interaction modes included, from a traditional mouse to a BCI helmet.
• In the development process of all the different solutions internal testing have been carried out both for debug purposes as well as to check the adherence to the use cases. This has also allowed us to launch new version of BrainControl AAC 2.0 in October 2021 and the new BrainControl Smart in September 2022 and to have them on the market.
• Administrative and Organisational Management: a new organization has been designed, with clear roles and responsibilities, with high focus on the R&D Department; a new CEO has been appointed with solid management and commercial experience (as recommended); 2 senior consultants have been hired to support the creation and implementation of a comprehensive marketing and communication plan; a scientific committee has been created to increase the competence and speed-up the development; finally some activities has been outsourced to 2 Universities in order to boost the development of the solutions.
Besides the achievement, briefly explained in previous sections, there are other projects which are ongoing now and will go beyond the project objectives both when it comes to the communicator as well as to the telepresence/avatar system.
Communicator: after the launch of BrainControl 2.0 (WP1 of the Grant) we have decided to further improve its accessibility by making it available directly on the web, thus avoiding any issue with pc type or programs installed etc. This will make easier for people to use our system and also expand our target including people with disabilities who wants to improve the usage of web thanks to our software. The will be delivered in Q3-2023
Diagnostic: the version which was recently released, part of this project, has been the base to develop 2 other important and exciting solutions that will be developed together with 2 Universities (with which we are already co-operating for current solutions in Siena and in London). We are currently developing new solutions to monitor patients' emotional and consciousness level. This will be the only system allowing doctors to have an insight of the state of patients who are in severe conditions (including coma). Both solutions will be patented, requests will be submitted in the coming months (therefore we will not disclose more than this)
Telepresence: the good success of our solution, which is highly appreciated as inclusive solution, has been proven in different occasions, showing how people with high mobility impairment are now able to "be present" through a robotic alter ego in events and interact with other people, having both a full immersive experience. This has also inspired 2 new projects that are moving their first steps now, in co-operation with other Companies working in the area of virtual assistance, to extend it to disabled people, thus increasing their inclusion and giving them further possibility to live a normal life.
Communicator: after the launch of BrainControl 2.0 (WP1 of the Grant) we have decided to further improve its accessibility by making it available directly on the web, thus avoiding any issue with pc type or programs installed etc. This will make easier for people to use our system and also expand our target including people with disabilities who wants to improve the usage of web thanks to our software. The will be delivered in Q3-2023
Diagnostic: the version which was recently released, part of this project, has been the base to develop 2 other important and exciting solutions that will be developed together with 2 Universities (with which we are already co-operating for current solutions in Siena and in London). We are currently developing new solutions to monitor patients' emotional and consciousness level. This will be the only system allowing doctors to have an insight of the state of patients who are in severe conditions (including coma). Both solutions will be patented, requests will be submitted in the coming months (therefore we will not disclose more than this)
Telepresence: the good success of our solution, which is highly appreciated as inclusive solution, has been proven in different occasions, showing how people with high mobility impairment are now able to "be present" through a robotic alter ego in events and interact with other people, having both a full immersive experience. This has also inspired 2 new projects that are moving their first steps now, in co-operation with other Companies working in the area of virtual assistance, to extend it to disabled people, thus increasing their inclusion and giving them further possibility to live a normal life.