Periodic Reporting for period 2 - Digital Iris (Bringing the human being into the digital loop)
Berichtszeitraum: 2018-12-01 bis 2019-09-30
Beyond the initial focus on gaming and certain training applications, the potential of headworn AR, Mixed Reality (MR) and Virtual Reality (VR) technologies lies in the B2B market with capabilities that will reshape business processes or fundamentally recast customer experiences. The biggest disruptive digitizing step aims at delocalizing the smart content availability and connecting it to a person itself and its environment; wherever, whenever, in whatever situational awareness.
Unlike VR, which creates a totally artificial environment, AR uses the real environment and overlays new information on top of it, shifting attention from a hand-held display to the real world around us. MR is even more ambitious, merging and mapping artificial data with real life. Unfortunately, all this “XR”-technologies are fairly limited in their impact still today as they all have a joint flaw: not reacting to natural human gaze behaviour.
Therefore, an additional step is needed in order to bridge the gap between human being and technology: Eye Tracking technology allows to measure the eye activity in order to better understand sight, perception and reaction including emotions, providing a complete bi-directional communication with the environment and leading to much smarter user interactions.
The competitive edge of companies and economies is expected to jump forward once the organic digital integration of the individual into a digital workflow is feasible. However, barriers for that integration are high as the systems have to work reliably under nearly any conditions.
Digital Iris is the definitive tool to integrate the human being into the digital world. Through the most accurate Eye Tracking technology, Digital Iris allows to completely understand what a person sees, perceives and feels in real time under any environmental condition, providing for the first-time-ever Eye Tracking outdoor usability and creating a bidirectional interaction via video and audio with the user based on personalized situations. Adding to a head worn, functionally robust Eye Hyper Tracking (EHT) a robust Augmented Reality (AR) element, provides the most authentic experience, generating useful information reliably for the user depending on the environment and his/her reactions via the strongest sense: the eye. This intuitive ‘Human Machine Interface’ (i-HMI) will immerse the user into a complete digital loop, opening up complete new and disruptive applications in industry, mobility and medicine, among others.
The main objectives to be achieved in course of the project are:
- To increase Eye Tracking software robustness.
- To move from a manual End-of-Line (EOL) calibration process to an automated EOL calibration process, reduce the initial user calibration from the current 3-minute process to a 10-second process.
- To implement a high-resolution front view camera (13 MP), large viewing angle (90º) and object recognition for user feedback.
- To provide 4G+/5G technology through an integrated chip included in the Smart Unit.
- To implement bi-directional audio and video communication under harsh conditions.
- To increase the overall physical robustness, especially for the key functional 3Cam-Tripod, using reinforced materials and enforced snap-mounts.
- To implement HDR/High-Resolution CMOS nose pad sensors for high functional robustness and reliability level at gaze tracking.
Unlike VR, which creates a totally artificial environment, AR uses the real environment and overlays new information on top of it, shifting attention from a hand-held display to the real world around us. MR is even more ambitious, merging and mapping artificial data with real life. Unfortunately, all this “XR”-technologies are fairly limited in their impact still today as they all have a joint flaw: not reacting to natural human gaze behaviour.
Therefore, an additional step is needed in order to bridge the gap between human being and technology: Eye Tracking technology allows to measure the eye activity in order to better understand sight, perception and reaction including emotions, providing a complete bi-directional communication with the environment and leading to much smarter user interactions.
The competitive edge of companies and economies is expected to jump forward once the organic digital integration of the individual into a digital workflow is feasible. However, barriers for that integration are high as the systems have to work reliably under nearly any conditions.
Digital Iris is the definitive tool to integrate the human being into the digital world. Through the most accurate Eye Tracking technology, Digital Iris allows to completely understand what a person sees, perceives and feels in real time under any environmental condition, providing for the first-time-ever Eye Tracking outdoor usability and creating a bidirectional interaction via video and audio with the user based on personalized situations. Adding to a head worn, functionally robust Eye Hyper Tracking (EHT) a robust Augmented Reality (AR) element, provides the most authentic experience, generating useful information reliably for the user depending on the environment and his/her reactions via the strongest sense: the eye. This intuitive ‘Human Machine Interface’ (i-HMI) will immerse the user into a complete digital loop, opening up complete new and disruptive applications in industry, mobility and medicine, among others.
The main objectives to be achieved in course of the project are:
- To increase Eye Tracking software robustness.
- To move from a manual End-of-Line (EOL) calibration process to an automated EOL calibration process, reduce the initial user calibration from the current 3-minute process to a 10-second process.
- To implement a high-resolution front view camera (13 MP), large viewing angle (90º) and object recognition for user feedback.
- To provide 4G+/5G technology through an integrated chip included in the Smart Unit.
- To implement bi-directional audio and video communication under harsh conditions.
- To increase the overall physical robustness, especially for the key functional 3Cam-Tripod, using reinforced materials and enforced snap-mounts.
- To implement HDR/High-Resolution CMOS nose pad sensors for high functional robustness and reliability level at gaze tracking.
The Digital Iris System is completely developed.
In particular the project achieved the important progress in the following relevant fields:
Software:
• New and improved pupil and eye detection algorithms (improvements in measurement accuracy and usability) developed
• New and improved calibration algorithms developed
• Augmented user feedback interface developed
• Setup of Video and Audio streaming communication developed
• A new ego motion algorithm to match the viewing direction of the user with his head orientation to enable to measure the viewing orientation in the 3-dimensional world (Viewing Direction + Head Orientation = Viewing Orientation) developed.
• Biometric sensors implemented in the ear pieces and merged with the Eye Tracking algorithms.
• Perception algorithms optimized (based on the new HDR sensors with the high frames rates)
• Modifications of the Fact Finder Software (development and test video database done, open source API implemented, demo application “bar code scanner” implemented).
• Latency reduction in the streaming application achieved (<250ms)
Electronic:
• DI glasses PCBs done (D-Sample state reached)
• Smart Unit PCBs done (D-Sample state reached)
• Display Unit PCBs done (D-Sample state reached)
Mechanic:
• Glasses optimization in geometry and for production purpose done (D-Sample state reached)
• Smart Unit update to high computing power and year 2019 style done (D-Sample state reached)
• Display Unit including the click on function done (D-Sample state reached)
• Optical development done
• Development of a micro assembly machine done
Production:
• Modification of the Micro-Assembly machine done
• Semi-automatic calibration process implemented in production
• Implementation of the End of Line Tester in the production process done
• Production processes verified and approved
• First production test runs done
Testing:
• Component verification completed
• Functional verification done
• CE certification pending (due to battery supplier issue)
• UL certification pending (due to battery supplier issue)
• Production test results validated and verified
In particular the project achieved the important progress in the following relevant fields:
Software:
• New and improved pupil and eye detection algorithms (improvements in measurement accuracy and usability) developed
• New and improved calibration algorithms developed
• Augmented user feedback interface developed
• Setup of Video and Audio streaming communication developed
• A new ego motion algorithm to match the viewing direction of the user with his head orientation to enable to measure the viewing orientation in the 3-dimensional world (Viewing Direction + Head Orientation = Viewing Orientation) developed.
• Biometric sensors implemented in the ear pieces and merged with the Eye Tracking algorithms.
• Perception algorithms optimized (based on the new HDR sensors with the high frames rates)
• Modifications of the Fact Finder Software (development and test video database done, open source API implemented, demo application “bar code scanner” implemented).
• Latency reduction in the streaming application achieved (<250ms)
Electronic:
• DI glasses PCBs done (D-Sample state reached)
• Smart Unit PCBs done (D-Sample state reached)
• Display Unit PCBs done (D-Sample state reached)
Mechanic:
• Glasses optimization in geometry and for production purpose done (D-Sample state reached)
• Smart Unit update to high computing power and year 2019 style done (D-Sample state reached)
• Display Unit including the click on function done (D-Sample state reached)
• Optical development done
• Development of a micro assembly machine done
Production:
• Modification of the Micro-Assembly machine done
• Semi-automatic calibration process implemented in production
• Implementation of the End of Line Tester in the production process done
• Production processes verified and approved
• First production test runs done
Testing:
• Component verification completed
• Functional verification done
• CE certification pending (due to battery supplier issue)
• UL certification pending (due to battery supplier issue)
• Production test results validated and verified
Digital Iris is a mobile Eye Hyper-Tracking system consisting of sophisticated smart glasses with eye camera sensors for eye movement recognition and a front camera for visual field observation and recognition. These glasses are connected to a Smart Unit that records, compresses and stores data, allowing video live streaming to laptops and tablets, where a SW solution is used to exploit the full potential of the data. The social acceptance design, the very light glasses and the robust function in nearly all environmental light conditions is unique in the market.
Digital Iris significantly improves product performance, including new impacting features that bring innovation to the market: Digital Iris is both an evolution and a revolution. The main advantages of this innovative product are:
- A revolutionary interface between the user and the digital world (eye gestures control; look, blink and control via the eyes).
- Fast + High Dynamic Range (HDR) gaze image processing. It allows to access different analyses in real-time, including perception analysis (only product in the market that provides this feature).
- Emotion detection, as an add on function and a revolutionary feature completely new in the market.
- The glasses work in a natural environment, both indoors and outdoors: a certified functional reliability and robustness is implemented.
- Light glasses, social acceptance design and endless runtime due to hot plug battery exchange.
- Micro Display Integration for Augmented Reality and Fast Focus Very High-Resolution front-view camera for object recognition to provide real-time visual feedback to the wearer.
- Full bi-directional audio and video communication with a very latency.
- On the fly video annotation feedback from the expert to the wearer of the DI glasses.
- Easy, intuitive and quick 4-point calibration.
Digital Iris significantly improves product performance, including new impacting features that bring innovation to the market: Digital Iris is both an evolution and a revolution. The main advantages of this innovative product are:
- A revolutionary interface between the user and the digital world (eye gestures control; look, blink and control via the eyes).
- Fast + High Dynamic Range (HDR) gaze image processing. It allows to access different analyses in real-time, including perception analysis (only product in the market that provides this feature).
- Emotion detection, as an add on function and a revolutionary feature completely new in the market.
- The glasses work in a natural environment, both indoors and outdoors: a certified functional reliability and robustness is implemented.
- Light glasses, social acceptance design and endless runtime due to hot plug battery exchange.
- Micro Display Integration for Augmented Reality and Fast Focus Very High-Resolution front-view camera for object recognition to provide real-time visual feedback to the wearer.
- Full bi-directional audio and video communication with a very latency.
- On the fly video annotation feedback from the expert to the wearer of the DI glasses.
- Easy, intuitive and quick 4-point calibration.