Final Report Summary - HARKEN (Heart and respiration in-car embedded nonintrusive sensors)
Executive Summary:
Road accidents caused by fatigue are an important societal and economical problem, that accounts for 20-35% of serious crashes, and over 6,000 fatalities every year. Many public and private organisations and companies are committed to address this problem, including the manufacturers of cars and their components, who strive for developing new systems to monitor the state of the drivers and help them prevent risky situations.
To achieve this, such systems have to recognize the symptoms of drowsiness and fatigue. One of the approaches aimed at by the developers of drowsiness detectors consists of measuring the signals of the body which are related to the sympathetic nervous system, like heart rate or respiration. Thanks to recent technological advances, nowadays it is possible to measure such signals with wearable devices, but still remains the challenge of integrating that technology in a way that is acceptable by drivers and the automotive industry.
The HARKEN project has been the framework for the development of such sensors by a consortium of small and medium enterprises. The solution developed in this project consists of a nonintrusive sensing system of driver’s heart and respiration, embedded in the seat cover and the safety belt of a car, that detects the mechanical effect of heart and respiration activity, filters and cancels the noise and artifacts expected in a moving vehicle (vibration and body movements), and calculates the relevant physiological parameters.
Five main results have been obtained from the project: (1) the formulation and design of a smart fabric that is sensitive to the small mechanic effects of respiration and heart rate, and can be integrated in textile products; (2) a sensing seat cover made of that material, designed for the driver's seat; (3) a sensing safety belt; (4) a signal processing unit that collects and processes the data of the two sensing components (seat and safety belt); and (5) an integrated unit that has been validated in a driving simulator and also tested on road.
The resulting HARKEN system can capture the heart rhythm and respiration waveforms in real time, in a completely unobtrusive manner. The results of the tests have shown the high sensitivity of the sensors, and the effectiveness of the filtering algorithms, that make the overall solution a potential tool to be integrated in cars in a near future. Once the functional requirements of the sensing system have been met, the following work will be based on improving the industrialization of the developed prototypes and the accomplishment of the specific standards of the sector, to be the technological base of future driver drowsiness detectors.
Project Context and Objectives:
Road accidents and casualties caused by fatigue are an important societal and economical problem for the EU. In 2008 there were 1.2 million road accidents in the EU, which resulted in 1.5 million casualties and 38,000 fatalities. Driver fatigue accounts for 20-35% of serious accidents, meaning that there may be over 7,000 annual fatalities due to fatigue-related accidents in the EU. Besides the incalculable human and social costs of such accidents, their economic cost can be estimated between €10B and €24B annually.
In-vehicle systems that warn drivers when they are becoming fatigued are the most direct countermeasure, and the one that automotive industry and technology component suppliers (mostly SMEs) may contribute to. Such systems need an input of the driver’s state, which may be recorded by the car interior parts in contact with driver’s body. BORGSTENA, an SME that designs and manufactures automotive interior textiles, has found in this concept an opportunity to add value to their products. They have joined up a consortium with other complementary SMEs and RTD performers to create a novel product based on smart sensing textiles, with high added-value for the automotive industry, which is looking for such technologies to enhance their Advanced Driving Assistance Systems (ADAS).
The solution proposed in this project to address the stated need is a nonintrusive sensing system of driver’s heart and respiration embedded in the seat cover and the safety belt of a car. It will detect the mechanical effect of heart and respiration activity, filter and cancel the noise and artefacts expected in a moving vehicle (vibration and body movements), and calculate the relevant parameters, which will be delivered in a readable format to integrate it in a fatigue detector.
This concept has been created by joining the complementary technical expertise of the four SMEs of the consortium. The technological core of the HARKEN system will be the smart fabrics created by SENSINGTEX, with sensing properties that will detect the effects of respiration and heart activity upon the parts of the body in contact with the seat (chiefly driver’s back) and the safety belt (chest and abdomen). Those fabrics must be integrated into the textiles of the car parts, which will be done by BORGSTENA (seat cover) and ALATEX (safety belt). The signal measured by this device will be treated by a processing unit created by PLUX, to remove noise caused by vibration and movements, and compute the relevant biological parameters. The output data will be configurable by any of the TIER1 suppliers of the automotive industry (represented in the consortium by FICOMIRRORS). In order to create this product, the companies count with the service of RTD-performers who are specialised in the fields necessary to fulfil the objectives of the project: IBV, a leading technological centre in the study of human body’s behaviour, EII, expert in information and communication technologies and development of electronic components, and the School of Materials of the University of Manchester.
The scientific and technical objectives that have been defined in this project are:
- Gain understanding of intelligent materials to increase their sensing and mechanical properties.
- Investigate the nature of biometric and noise signals that may be sensed by the materials.
- Define an algorithm for signal processing with adaptive filters, to remove noise and artifacts.
- Develop a new smart fabric to sense heart and respiration without direct contact.
- Design a processing unit that handles the signals
- Achieve a benefit-cost ratio that is acceptable to the market.
Project Results:
After two years of work, the project HARKEN has achieved all its objectives, with the creation of a system of sensors that is able to detect heart rate and respiration signals of drivers in a car, integrated in the seat cover and the safety belt of the vehicle.
The work done during the project was divided into three phases:
1) A first scientific phase, focused on acquiring a deeper scientific understanding about the physiological and dynamic actions that had to be measured by the HARKEN system (Work Package 1), about the characteristics of smart materials and how can they be modified to achieve the required sensing properties (Work Package 2), and about filtering and data fusion techniques, adapted to the purposes of the project (Work Package 3).
2) A development phase, that included the creation, integration, and demonstration of the system. The three principal components of the system (sensing seat cover, sensing safety belt, and signal processing unit) were created in specific work packages (WP4, 5, and 6, respectively). Then they were be integrated in functional prototypes for testing in laboratory (Work Package 7), and a field trial demonstration (Work Package 8).
3) Finally, the innovation and management phase, running in parallel with the other activities, was focused on the dissemination and planning of exploitation of the project results (Work Package 9), and the management of the project (Work Package 10).
Each one of these phases gave different scientific and technical results and foregrounds. According to the plan of the project, there were five main expected results of the project, linked with the project phases as follows:
RESULT #1. A smart textile material with adequate sensitiveness for measuring heart beats and respiration in seats, belts, and similar products. This result was created during the scientific phase of the project for the partner SensingTex, a Spanish company whose business is the development and commercialisation of textiles with lighting and electronic properties.
RESULT #2. A sensing seat cover made of the smart material, to detect the physiological signals of the driver, made during the development phase for the partner and project coordinator Borgstena, a Portuguese worldwide supplier of technical textiles for seat covers.
RESULT #3. A sensing safety belt with similar characteristics to the seat cover, created for the partner Alatex, a German company dedicated to manufacturing belt straps for different industry sectors.
RESULT #4. The signal processing unit (SPU), a hardware/software module that collects the signals of the seat cover and the safety belt, and processes it according to the algorithms defined in the scientific phase. This result has been created for Plux, a Portuguese company dedicated to the business of wearable biomonitoring.
RESULT #5. The integrated HARKEN system, that puts together all the previous results in a functional equipment, which has been validated as an effective tool to measure the vital signs of drivers.
The achievement and details of these results are given next, in relation with the outcomes of the different work packages of the project, as described in their deliverables.
The GLOBAL CONCEPT of the system and its individual components was defined in the beginning of the project, starting with a thorough revision of the state-of-the-art about unobtrusive physiological sensing, including relevant publications and related projects (specially those related to the SFIT European cluster), in order to detect the best technologies and options that can be transferred to HARKEN.
Main related concepts were seen in the OFSETH project (2006-2009, cfr. http://www.ofseth.org/) that developed several technologies based on silica and polymer optical fibres for sensing respiration with belts around the chest and the abdomen; the HeartCycle project (2008-2012, http://www.heartcycle.eu/) with results that included sensors of ECG in automobile seats; and also particular studies in the field of nonintrusive biomonitoring using different technologies integrated on seats, based on the measurement of mechanical effects of respiration and heart rate (ballistocardiography - BCG, instead of ECG).
That research gave the fundamental ideas for the technology that could be used in the project, and also the variables and parameters of the vital signals that should be monitored by the system. The measurement of heart activity would be concentrated on the detection and analysis of inter-beat intervals, which are the most highlighted feature, the one that is more likely to be detected in noisy environments, and is also valuable to give information about cognitive states - analysing heart rate (HR) and heart rate variability (HRV). Regarding respiration, there are different characteristics of its waveform that are interesting for the analysis: breath amplitude and frequency, signal area, spikes over the mean, and also apneic events (intervals in which some degree of reduction of the waveform occurs over a significant period of time).
Another key element of the global concept is the procedure for signal processing. The research done in the scientific phase of the project included measures of the vital signs (monitored with standard medical monitors) in subjects in different situations: static, driving, and in both cases with and without vibrations (artificially generated by a vibratory platform), in order to characterise the signals that shall be measured by the Harken system, and the error sources that shall be dealt with by our system. The results of those experiments gave information about the signal/noise contents that could be expected in the measures with the sensors of the HARKEN system, and how to deal with them.
The SMART MATERIAL FOR THE SENSORS (result #1) was obtained after the research of Work Package 2. Sensing Tex's state-of-the-art technology was carefully studied by the experts of the University of Manchester, who created a solution based on their materials, that is capable of measuring the target signals, with the sensors on the subject's thorax an abdomen, without direct contact with the skin. The information provided by that research gave information about the composition of the textile components, and variations in the electric properties (resistance, conductance, etc.) in relation with pressures, deformations, and changes in the chemical composition, layers of textile, type and number of electrodes, and characteristics of the circuit.
The concept has been made reality during the development phase in the second year of the project, that has been dedicated to the production of the three principal components of the HARKEN system: the sensing seat cover, the sensing safety belt, and the SPU.
The SENSING SEAT COVER (result #2) has been designed after a thorough revision of different concepts. These concepts consisted in diverse configurations of sensor matrices, with different dimensions and distributions of the sensing elements in the areas of the backrest and the bottom cushion of the seat, materials used for the conductive terminal lines, and processes for manufacturing the complex that integrates the sensing textile with the fabrics used in standard seat covers.
As a starting point, the research team investigated the anthropometric measures of users, and the areas of interest of the seat where it is more likely to obtain the physiological signals by contact of the body. These areas have been adapted to the particular dimensions of the Hyundai Kia IX35 seat, which has been the model used for the tests during the whole development of the project.
In each iteration of the design, the research team investigated several aspects. One was the sensitiveness of the sensors and the signal/noise ratio in their picking up heart beats and respiration, with tests of users sitting quietly on a seat with the sensors. We also measured the correlation of such measures with a gold standard obtained by synchronized measures of a medical monitor, and compared the quality of those measures for different configurations and thickness of the layers of fabric/clothes between the sensors and the user’s skin. Finally, also the characteristics of the material, and its appropriateness to integrate it in standard production processes (stretchability and respirability), were investigated.
The final design provided very positive results regarding the extraction of respiration waveforms. After removing noise by signal processing, the signal was strongly correlated with the signal measured by a medical monitor. The algorithm to extract cardiac parameters achieved to isolate a BCG-like waveform with visible peaks around 1 Hz, corresponding to heart beats, with a Quality Index equal to 0.65 for the R-peak matching.
The SENSING SAFETY BELT (result #3) was designed as a concept similar to what would be the final solution of the seat cover, with the same structure of sensors and electrodes, although a different distribution of the sensors and another way of connecting them, due to the differences between the geometry of the components, and the way they are in contact with the driver’s body. The sensors have been integrated in regions that cover the chest of the driver and the abdomen, which are more likely to receive strong signals of the cardiorespiratory activity.
The same type of tests performed for the seat cover were applied to the prototypes of the safety belt, in order to test their sensitiveness and adequacy to pick up heart beats and respiration waveforms. Those experiments provided very positive results regarding the extraction of respiration waveforms, comparable to the ones obtained for the seat sensor, and even better results regarding the extraction of heart beats. The denoised signal was likewise correlated with the respiration signal measured by a medical monitor, specially in the lower thoracic sensor, which was also the optimal region to detect heart beats. The quality index of inter-beat intervals increased to 0.75 for this component, in a tight configuration, achieved by positioning the D-ring of the belt in a rear location.
The SIGNAL PROCESSING UNIT (result #4) was developed in parallel with the other components, in order to facilitate the correct integration between them, and with the algorithms for signal denoising and extraction of the physiological signals. The algorithms were developed after the analysis of the vital signals in users, first with commercial respiratory bands for breathing, and piezoelectric polyvinylidene fluoride (PVDF) for heart beats (ballistocardiograms), and afterwards with prototypes of the sensing materials used in the components of the HARKEN system, in order to obtain the characteristics of the signals that were going to be analyzed. The signal processing algorithms have various phases for: (1) filtering noise based on the frequency spectrum, (2) cancelling of the motion artifacts, (3) separation of heart beats from the respiratory signal, and (4) extraction of the relevant parameters.
A key issue in the processing of the signal is the cancellation of artifacts, due to the several sources of noise that affect the sensors. The main source are the movements of the car and the driver while driving. The technique chosen for that is a multi-stage adaptive filtering, that needs a reference signal correlated with the source of noise. Such reference signal has been obtained from the HARKEN sensors, focusing on global patterns of the distribution of the user’s weight on the seat.
The design of the SPU has been guided by the characteristics of PLUX's commercial catalogue, which marked the specifications of hardware and power. The unit developed in the project extends the capabilities of PLUX’s device, with the possibility of capturing the data from the HARKEN sensors, and adding software for signal processing. The SPU has been produced with a front end that can be run in different desktop environments, instead of a closed solution bond to the electronic network of a specific car. This improves the power of the system for processing and displaying results, and provides a more flexible platform for demonstration to potential clients.
The INTEGRATED SYSTEM (result #5) has been developed in the last months of the project, assembling together all the components in two settings: a driving simulator, and a real car. The former setting has been used to provide a controlled environment, where the outcome of the system could be compared with the physiological variables measured by a medical monitor. This has been used for a validation experiment, similar to the ones used to test the individual components, but with more complex conditions, with the subjects driving and moving as would be normally done in real use.
The experiments done in the laboratory, comparing the signals of the HARKEN system with physiological measures, demonstrate that such sensors and algorithms give excellent results regarding the respiration signal in any condition.
Heart rate can also be detected, with varying levels of quality. In optimal conditions, the sensors are sensitive enough to detect a virtually perfect BCG signal. When the driver is driving the car and moving his body, the quality of the signal degrades, but it is still useful to calculate some parameters (quality index up to 70%, even in noisy conditions). Heart rate is hardly affected by the errors, whereas heart rate variability is higher than the gold standard measured by an electrocardiogram, since the kind of activity detected by the HARKEN sensors is related to ballistocardiogram
The setup in the car was used in a field trial on a closed track circuit, where the performance of the system was successfully tested in real conditions. The integration of the HARKEN system in the test car was done similarly to the construction of the prototype in the laboratory driving simulator, and a driver was hired to conduct the test in the closed track circuit, during a 20-minute trial. The kind of signals obtained in this test resembled the ones that had been observed in the laboratory trial. Sample videos and the results of the tests were used for the creation of the videoclip of the project, and other dissemination actions in TV.
During the Final Meeting of the project, the partners also revised the prototype, in a live demonstration prepared in the facilities of the Project Coordinator Borgstena. In that session, the Coordinator and the other SME partners evaluated further aspects of the integration, in addition to the performance of the sensors and the data processing. That evaluation from the professional perspective of the companies that will exploit the results of the project, gave important insights about the readiness of the system, in relation to the possibilities to introduce it in the market. It was particularly useful to highlight other aspects of the integration, more related to the finishing of the prototype, and usability issues. As a result of that evaluation, the Consortium made plans to further improve the product that SME can use, to demonstrate more effectively in commercial and demonstration actions the added value of the HARKEN sensor system.
Potential Impact:
From a wide point of view, the system developed in this project will be an important contribution to the market of fatigue detectors, which will be a key factor in the reduction of road accidents. These types of devices have a total potential capacity to reduce accidents of 50%, according to the EC Directorate for General Energy and Transport (EC DGET). In terms of human cost, this potential implies about 4,000 lives saved, and tens of thousand injuries prevented every year. This improvement in road safety will have an important economic impact as well, as the EU will save billions of Euros every year.
There is also a more specific, economic positive impact in the SMEs that have participated in the project and their business sector. According to the EC DGET, the market share of FD forecasted for 20 years after their introduction is 1 of every 10 cars. Aiming at 10% of that potential market, the consortium expects to sell about 200,000 units of the system in the following 5 years, which could mean an accumulated profit for the enterprises of 2,5 M€. . Moreover, the companies will enhance their business in the very competitive automotive industry.
Such a new technology that monitors the physiological signals of a driver (or a seated person, in more general contexts), will also enlarge the knowledge-based economy, and it may also have an impact on the development of standards in varied fields, like human-machine interfaces, safety systems, and biomonitoring in other applications.
To achieve those objectives, the consortium has developed several innovation and dissemination activities during the project. Sensing Tex has published a patent of extensible pressure sensor for textiles surfaces (EP2682724), and the other companies are working in future patents for their respective results. Besides, there has been a considerable effort in scientific, technical, and mass media communication to disseminate the work done in the project.
Its progress and results have been presented in three scientific and technical conferences (Driver Distraction and Innatention Conference 2013, Transport Research Arena 2014, and the Electronic Conference on Sensors and Applications 2014), and an early prototype was exhibited in Frankfurt's Motor Show in September 2013. Different presentations and technical articles have been also delivered in technical forums and magazines specific of the automotive sector. The project has also had an important impact in electronic and mass media, including different news websites, newspapers, radio, and national and international TV.
From now on, the companies will concentrate in developing a business model for the integrated HARKEN system and its components. Borgstena, project coordinator and integrator of the system, will approach the main OEM and TIER1 potentially interested in the product, and will present the product in workshops and fairs to them. There will also be contacts with transport sector associations and road safety organisations, in order to promote the demand of fatigue detectors that may implement the HARKEN system.
Within the scope of this project, the HARKEN system will be developed to the pre-production prototype level, so that we will need further development work during 12 months after the end of the project, to perfect the prototype and achieve a marketable product. This period will be necessary to end certification procedures, as well as to perform minor adjustments, according to TIER1’s feedback after the demonstration and preliminary marketing activities included in the project
List of Websites:
Project website: http://harken.ibv.org
Road accidents caused by fatigue are an important societal and economical problem, that accounts for 20-35% of serious crashes, and over 6,000 fatalities every year. Many public and private organisations and companies are committed to address this problem, including the manufacturers of cars and their components, who strive for developing new systems to monitor the state of the drivers and help them prevent risky situations.
To achieve this, such systems have to recognize the symptoms of drowsiness and fatigue. One of the approaches aimed at by the developers of drowsiness detectors consists of measuring the signals of the body which are related to the sympathetic nervous system, like heart rate or respiration. Thanks to recent technological advances, nowadays it is possible to measure such signals with wearable devices, but still remains the challenge of integrating that technology in a way that is acceptable by drivers and the automotive industry.
The HARKEN project has been the framework for the development of such sensors by a consortium of small and medium enterprises. The solution developed in this project consists of a nonintrusive sensing system of driver’s heart and respiration, embedded in the seat cover and the safety belt of a car, that detects the mechanical effect of heart and respiration activity, filters and cancels the noise and artifacts expected in a moving vehicle (vibration and body movements), and calculates the relevant physiological parameters.
Five main results have been obtained from the project: (1) the formulation and design of a smart fabric that is sensitive to the small mechanic effects of respiration and heart rate, and can be integrated in textile products; (2) a sensing seat cover made of that material, designed for the driver's seat; (3) a sensing safety belt; (4) a signal processing unit that collects and processes the data of the two sensing components (seat and safety belt); and (5) an integrated unit that has been validated in a driving simulator and also tested on road.
The resulting HARKEN system can capture the heart rhythm and respiration waveforms in real time, in a completely unobtrusive manner. The results of the tests have shown the high sensitivity of the sensors, and the effectiveness of the filtering algorithms, that make the overall solution a potential tool to be integrated in cars in a near future. Once the functional requirements of the sensing system have been met, the following work will be based on improving the industrialization of the developed prototypes and the accomplishment of the specific standards of the sector, to be the technological base of future driver drowsiness detectors.
Project Context and Objectives:
Road accidents and casualties caused by fatigue are an important societal and economical problem for the EU. In 2008 there were 1.2 million road accidents in the EU, which resulted in 1.5 million casualties and 38,000 fatalities. Driver fatigue accounts for 20-35% of serious accidents, meaning that there may be over 7,000 annual fatalities due to fatigue-related accidents in the EU. Besides the incalculable human and social costs of such accidents, their economic cost can be estimated between €10B and €24B annually.
In-vehicle systems that warn drivers when they are becoming fatigued are the most direct countermeasure, and the one that automotive industry and technology component suppliers (mostly SMEs) may contribute to. Such systems need an input of the driver’s state, which may be recorded by the car interior parts in contact with driver’s body. BORGSTENA, an SME that designs and manufactures automotive interior textiles, has found in this concept an opportunity to add value to their products. They have joined up a consortium with other complementary SMEs and RTD performers to create a novel product based on smart sensing textiles, with high added-value for the automotive industry, which is looking for such technologies to enhance their Advanced Driving Assistance Systems (ADAS).
The solution proposed in this project to address the stated need is a nonintrusive sensing system of driver’s heart and respiration embedded in the seat cover and the safety belt of a car. It will detect the mechanical effect of heart and respiration activity, filter and cancel the noise and artefacts expected in a moving vehicle (vibration and body movements), and calculate the relevant parameters, which will be delivered in a readable format to integrate it in a fatigue detector.
This concept has been created by joining the complementary technical expertise of the four SMEs of the consortium. The technological core of the HARKEN system will be the smart fabrics created by SENSINGTEX, with sensing properties that will detect the effects of respiration and heart activity upon the parts of the body in contact with the seat (chiefly driver’s back) and the safety belt (chest and abdomen). Those fabrics must be integrated into the textiles of the car parts, which will be done by BORGSTENA (seat cover) and ALATEX (safety belt). The signal measured by this device will be treated by a processing unit created by PLUX, to remove noise caused by vibration and movements, and compute the relevant biological parameters. The output data will be configurable by any of the TIER1 suppliers of the automotive industry (represented in the consortium by FICOMIRRORS). In order to create this product, the companies count with the service of RTD-performers who are specialised in the fields necessary to fulfil the objectives of the project: IBV, a leading technological centre in the study of human body’s behaviour, EII, expert in information and communication technologies and development of electronic components, and the School of Materials of the University of Manchester.
The scientific and technical objectives that have been defined in this project are:
- Gain understanding of intelligent materials to increase their sensing and mechanical properties.
- Investigate the nature of biometric and noise signals that may be sensed by the materials.
- Define an algorithm for signal processing with adaptive filters, to remove noise and artifacts.
- Develop a new smart fabric to sense heart and respiration without direct contact.
- Design a processing unit that handles the signals
- Achieve a benefit-cost ratio that is acceptable to the market.
Project Results:
After two years of work, the project HARKEN has achieved all its objectives, with the creation of a system of sensors that is able to detect heart rate and respiration signals of drivers in a car, integrated in the seat cover and the safety belt of the vehicle.
The work done during the project was divided into three phases:
1) A first scientific phase, focused on acquiring a deeper scientific understanding about the physiological and dynamic actions that had to be measured by the HARKEN system (Work Package 1), about the characteristics of smart materials and how can they be modified to achieve the required sensing properties (Work Package 2), and about filtering and data fusion techniques, adapted to the purposes of the project (Work Package 3).
2) A development phase, that included the creation, integration, and demonstration of the system. The three principal components of the system (sensing seat cover, sensing safety belt, and signal processing unit) were created in specific work packages (WP4, 5, and 6, respectively). Then they were be integrated in functional prototypes for testing in laboratory (Work Package 7), and a field trial demonstration (Work Package 8).
3) Finally, the innovation and management phase, running in parallel with the other activities, was focused on the dissemination and planning of exploitation of the project results (Work Package 9), and the management of the project (Work Package 10).
Each one of these phases gave different scientific and technical results and foregrounds. According to the plan of the project, there were five main expected results of the project, linked with the project phases as follows:
RESULT #1. A smart textile material with adequate sensitiveness for measuring heart beats and respiration in seats, belts, and similar products. This result was created during the scientific phase of the project for the partner SensingTex, a Spanish company whose business is the development and commercialisation of textiles with lighting and electronic properties.
RESULT #2. A sensing seat cover made of the smart material, to detect the physiological signals of the driver, made during the development phase for the partner and project coordinator Borgstena, a Portuguese worldwide supplier of technical textiles for seat covers.
RESULT #3. A sensing safety belt with similar characteristics to the seat cover, created for the partner Alatex, a German company dedicated to manufacturing belt straps for different industry sectors.
RESULT #4. The signal processing unit (SPU), a hardware/software module that collects the signals of the seat cover and the safety belt, and processes it according to the algorithms defined in the scientific phase. This result has been created for Plux, a Portuguese company dedicated to the business of wearable biomonitoring.
RESULT #5. The integrated HARKEN system, that puts together all the previous results in a functional equipment, which has been validated as an effective tool to measure the vital signs of drivers.
The achievement and details of these results are given next, in relation with the outcomes of the different work packages of the project, as described in their deliverables.
The GLOBAL CONCEPT of the system and its individual components was defined in the beginning of the project, starting with a thorough revision of the state-of-the-art about unobtrusive physiological sensing, including relevant publications and related projects (specially those related to the SFIT European cluster), in order to detect the best technologies and options that can be transferred to HARKEN.
Main related concepts were seen in the OFSETH project (2006-2009, cfr. http://www.ofseth.org/) that developed several technologies based on silica and polymer optical fibres for sensing respiration with belts around the chest and the abdomen; the HeartCycle project (2008-2012, http://www.heartcycle.eu/) with results that included sensors of ECG in automobile seats; and also particular studies in the field of nonintrusive biomonitoring using different technologies integrated on seats, based on the measurement of mechanical effects of respiration and heart rate (ballistocardiography - BCG, instead of ECG).
That research gave the fundamental ideas for the technology that could be used in the project, and also the variables and parameters of the vital signals that should be monitored by the system. The measurement of heart activity would be concentrated on the detection and analysis of inter-beat intervals, which are the most highlighted feature, the one that is more likely to be detected in noisy environments, and is also valuable to give information about cognitive states - analysing heart rate (HR) and heart rate variability (HRV). Regarding respiration, there are different characteristics of its waveform that are interesting for the analysis: breath amplitude and frequency, signal area, spikes over the mean, and also apneic events (intervals in which some degree of reduction of the waveform occurs over a significant period of time).
Another key element of the global concept is the procedure for signal processing. The research done in the scientific phase of the project included measures of the vital signs (monitored with standard medical monitors) in subjects in different situations: static, driving, and in both cases with and without vibrations (artificially generated by a vibratory platform), in order to characterise the signals that shall be measured by the Harken system, and the error sources that shall be dealt with by our system. The results of those experiments gave information about the signal/noise contents that could be expected in the measures with the sensors of the HARKEN system, and how to deal with them.
The SMART MATERIAL FOR THE SENSORS (result #1) was obtained after the research of Work Package 2. Sensing Tex's state-of-the-art technology was carefully studied by the experts of the University of Manchester, who created a solution based on their materials, that is capable of measuring the target signals, with the sensors on the subject's thorax an abdomen, without direct contact with the skin. The information provided by that research gave information about the composition of the textile components, and variations in the electric properties (resistance, conductance, etc.) in relation with pressures, deformations, and changes in the chemical composition, layers of textile, type and number of electrodes, and characteristics of the circuit.
The concept has been made reality during the development phase in the second year of the project, that has been dedicated to the production of the three principal components of the HARKEN system: the sensing seat cover, the sensing safety belt, and the SPU.
The SENSING SEAT COVER (result #2) has been designed after a thorough revision of different concepts. These concepts consisted in diverse configurations of sensor matrices, with different dimensions and distributions of the sensing elements in the areas of the backrest and the bottom cushion of the seat, materials used for the conductive terminal lines, and processes for manufacturing the complex that integrates the sensing textile with the fabrics used in standard seat covers.
As a starting point, the research team investigated the anthropometric measures of users, and the areas of interest of the seat where it is more likely to obtain the physiological signals by contact of the body. These areas have been adapted to the particular dimensions of the Hyundai Kia IX35 seat, which has been the model used for the tests during the whole development of the project.
In each iteration of the design, the research team investigated several aspects. One was the sensitiveness of the sensors and the signal/noise ratio in their picking up heart beats and respiration, with tests of users sitting quietly on a seat with the sensors. We also measured the correlation of such measures with a gold standard obtained by synchronized measures of a medical monitor, and compared the quality of those measures for different configurations and thickness of the layers of fabric/clothes between the sensors and the user’s skin. Finally, also the characteristics of the material, and its appropriateness to integrate it in standard production processes (stretchability and respirability), were investigated.
The final design provided very positive results regarding the extraction of respiration waveforms. After removing noise by signal processing, the signal was strongly correlated with the signal measured by a medical monitor. The algorithm to extract cardiac parameters achieved to isolate a BCG-like waveform with visible peaks around 1 Hz, corresponding to heart beats, with a Quality Index equal to 0.65 for the R-peak matching.
The SENSING SAFETY BELT (result #3) was designed as a concept similar to what would be the final solution of the seat cover, with the same structure of sensors and electrodes, although a different distribution of the sensors and another way of connecting them, due to the differences between the geometry of the components, and the way they are in contact with the driver’s body. The sensors have been integrated in regions that cover the chest of the driver and the abdomen, which are more likely to receive strong signals of the cardiorespiratory activity.
The same type of tests performed for the seat cover were applied to the prototypes of the safety belt, in order to test their sensitiveness and adequacy to pick up heart beats and respiration waveforms. Those experiments provided very positive results regarding the extraction of respiration waveforms, comparable to the ones obtained for the seat sensor, and even better results regarding the extraction of heart beats. The denoised signal was likewise correlated with the respiration signal measured by a medical monitor, specially in the lower thoracic sensor, which was also the optimal region to detect heart beats. The quality index of inter-beat intervals increased to 0.75 for this component, in a tight configuration, achieved by positioning the D-ring of the belt in a rear location.
The SIGNAL PROCESSING UNIT (result #4) was developed in parallel with the other components, in order to facilitate the correct integration between them, and with the algorithms for signal denoising and extraction of the physiological signals. The algorithms were developed after the analysis of the vital signals in users, first with commercial respiratory bands for breathing, and piezoelectric polyvinylidene fluoride (PVDF) for heart beats (ballistocardiograms), and afterwards with prototypes of the sensing materials used in the components of the HARKEN system, in order to obtain the characteristics of the signals that were going to be analyzed. The signal processing algorithms have various phases for: (1) filtering noise based on the frequency spectrum, (2) cancelling of the motion artifacts, (3) separation of heart beats from the respiratory signal, and (4) extraction of the relevant parameters.
A key issue in the processing of the signal is the cancellation of artifacts, due to the several sources of noise that affect the sensors. The main source are the movements of the car and the driver while driving. The technique chosen for that is a multi-stage adaptive filtering, that needs a reference signal correlated with the source of noise. Such reference signal has been obtained from the HARKEN sensors, focusing on global patterns of the distribution of the user’s weight on the seat.
The design of the SPU has been guided by the characteristics of PLUX's commercial catalogue, which marked the specifications of hardware and power. The unit developed in the project extends the capabilities of PLUX’s device, with the possibility of capturing the data from the HARKEN sensors, and adding software for signal processing. The SPU has been produced with a front end that can be run in different desktop environments, instead of a closed solution bond to the electronic network of a specific car. This improves the power of the system for processing and displaying results, and provides a more flexible platform for demonstration to potential clients.
The INTEGRATED SYSTEM (result #5) has been developed in the last months of the project, assembling together all the components in two settings: a driving simulator, and a real car. The former setting has been used to provide a controlled environment, where the outcome of the system could be compared with the physiological variables measured by a medical monitor. This has been used for a validation experiment, similar to the ones used to test the individual components, but with more complex conditions, with the subjects driving and moving as would be normally done in real use.
The experiments done in the laboratory, comparing the signals of the HARKEN system with physiological measures, demonstrate that such sensors and algorithms give excellent results regarding the respiration signal in any condition.
Heart rate can also be detected, with varying levels of quality. In optimal conditions, the sensors are sensitive enough to detect a virtually perfect BCG signal. When the driver is driving the car and moving his body, the quality of the signal degrades, but it is still useful to calculate some parameters (quality index up to 70%, even in noisy conditions). Heart rate is hardly affected by the errors, whereas heart rate variability is higher than the gold standard measured by an electrocardiogram, since the kind of activity detected by the HARKEN sensors is related to ballistocardiogram
The setup in the car was used in a field trial on a closed track circuit, where the performance of the system was successfully tested in real conditions. The integration of the HARKEN system in the test car was done similarly to the construction of the prototype in the laboratory driving simulator, and a driver was hired to conduct the test in the closed track circuit, during a 20-minute trial. The kind of signals obtained in this test resembled the ones that had been observed in the laboratory trial. Sample videos and the results of the tests were used for the creation of the videoclip of the project, and other dissemination actions in TV.
During the Final Meeting of the project, the partners also revised the prototype, in a live demonstration prepared in the facilities of the Project Coordinator Borgstena. In that session, the Coordinator and the other SME partners evaluated further aspects of the integration, in addition to the performance of the sensors and the data processing. That evaluation from the professional perspective of the companies that will exploit the results of the project, gave important insights about the readiness of the system, in relation to the possibilities to introduce it in the market. It was particularly useful to highlight other aspects of the integration, more related to the finishing of the prototype, and usability issues. As a result of that evaluation, the Consortium made plans to further improve the product that SME can use, to demonstrate more effectively in commercial and demonstration actions the added value of the HARKEN sensor system.
Potential Impact:
From a wide point of view, the system developed in this project will be an important contribution to the market of fatigue detectors, which will be a key factor in the reduction of road accidents. These types of devices have a total potential capacity to reduce accidents of 50%, according to the EC Directorate for General Energy and Transport (EC DGET). In terms of human cost, this potential implies about 4,000 lives saved, and tens of thousand injuries prevented every year. This improvement in road safety will have an important economic impact as well, as the EU will save billions of Euros every year.
There is also a more specific, economic positive impact in the SMEs that have participated in the project and their business sector. According to the EC DGET, the market share of FD forecasted for 20 years after their introduction is 1 of every 10 cars. Aiming at 10% of that potential market, the consortium expects to sell about 200,000 units of the system in the following 5 years, which could mean an accumulated profit for the enterprises of 2,5 M€. . Moreover, the companies will enhance their business in the very competitive automotive industry.
Such a new technology that monitors the physiological signals of a driver (or a seated person, in more general contexts), will also enlarge the knowledge-based economy, and it may also have an impact on the development of standards in varied fields, like human-machine interfaces, safety systems, and biomonitoring in other applications.
To achieve those objectives, the consortium has developed several innovation and dissemination activities during the project. Sensing Tex has published a patent of extensible pressure sensor for textiles surfaces (EP2682724), and the other companies are working in future patents for their respective results. Besides, there has been a considerable effort in scientific, technical, and mass media communication to disseminate the work done in the project.
Its progress and results have been presented in three scientific and technical conferences (Driver Distraction and Innatention Conference 2013, Transport Research Arena 2014, and the Electronic Conference on Sensors and Applications 2014), and an early prototype was exhibited in Frankfurt's Motor Show in September 2013. Different presentations and technical articles have been also delivered in technical forums and magazines specific of the automotive sector. The project has also had an important impact in electronic and mass media, including different news websites, newspapers, radio, and national and international TV.
From now on, the companies will concentrate in developing a business model for the integrated HARKEN system and its components. Borgstena, project coordinator and integrator of the system, will approach the main OEM and TIER1 potentially interested in the product, and will present the product in workshops and fairs to them. There will also be contacts with transport sector associations and road safety organisations, in order to promote the demand of fatigue detectors that may implement the HARKEN system.
Within the scope of this project, the HARKEN system will be developed to the pre-production prototype level, so that we will need further development work during 12 months after the end of the project, to perfect the prototype and achieve a marketable product. This period will be necessary to end certification procedures, as well as to perform minor adjustments, according to TIER1’s feedback after the demonstration and preliminary marketing activities included in the project
List of Websites:
Project website: http://harken.ibv.org