Final Report Summary - ABCIT (Advancing Binaural Cochlear Implant Technology)
Executive Summary:
1.0 Executive Summary
In 2012, a consortium of researchers and commercial partners in the UK, France and Germany secured €4M from the European Union’s ‘Framework 7’ research fund to develop a research programme aimed at progressing bilateral cochlear implantation towards true binaural performance. Binaural (two-eared) hearing is critical for localizing the source of a sound, and for hearing out conversations in background noise (cocktail party listening). Compared to normal-hearing individuals, users of cochlear implants (CIs) are at a disadvantage when facing binaural tasks. Bilateral cochlear implantees (most children born deaf and an increasing number of adult recipients) are fitted with essentially two independent devices, each without regard to the sound processing or electrical stimulation of the other ear. The aim of this programme, Advancing Binaural Cochlear Implant Technology (ABCIT), therefore, was to develop a framework in which binaural hearing was central to the function of CI technology. The programme of work was divided into six work packages (WPs), each with a named lead partner. Critical aspects of the programme were to ensure the successful delivery of the milestones and deliverables within each WP, and to ensure compatibility across WPs. In order to provide CI researchers with the tools necessary to assess binaural function, a portable, real-time research platform was developed, capable of processing signals from 4 microphones simultaneously. A crucial aspect of this research platform lies in its use of the Master Hearing Aid (MHA) technology of Hörtech, the hearing technologies centre allied to Oldenburg Univerity. The new platform is well suited to assessing sensitivity to binaural spatial cues, including interaural time differences (ITDs), in real-time, and supports instantaneously variable stimulation rates. Software processing can be changed on the fly. A simulation tool was developed enabling researchers to explore the effects of electrical stimulation on the temporal pattern of nerve impulses generated by CIs and, from this, to develop computer models of neural sensitivity to binaural electrical stimulation. New electrical stimulation strategies were developed that seek to reduce the spread of electrical activity in the implanted cochlea and to convey the temporal information necessary for preserving ITD cues in binaural hearing. These stimulation strategies were assessed physiologically in vitro and in vivo in experimental animals. A key aspect of ensuring binaural information is maximised in CI listeners lies in developing objective measures of brain function. The ABCIT project therefore developed new, and assessed existing, measures of binaural brain function in normal-hearing and CI listeners, with the aim of providing a diagnostically feasible repertoire of neurophysiological measurements to aid the fitting of bilateral CIs. Another critical development was the merging of hearing-aid and CI technologies, particularly through the development and exploitation of sound-processing algorithms that enhance speech understanding by removing background noise from acoustic signals before these signals are converted to electrical pulses in CI devices. A final WP integrated these ideas into a workable binaural CI ‘development kit’ expressed in hardware and software. The acquisition of Neurelec by William Demant Holdings (WDH) in April 2013 and its ensuing incorporation into Oticon Medical should help to bring binaural CI technology to market and has had an energizing effect on the project. Overall, ABCIT was highly successful, bringing together researchers and commercial partners into true partnership, and providing the means of developing CI technology beyond the state-of-the-art. As well as many publications and presentations, ABCIT generated four joint patents and a range of associated projects and collaborations that will generate new advances in discovery science and implementation of these advances into technologies, diagnostics and interventions aimed at transforming the lives of the profoundly deaf.
Project Context and Objectives:
2.0 Project Context and Objectives
2.1 Context of the ABCIT Project
Hearing loss is the most common sensory deficit; one in seven of the global population has a hearing problem and with this population increasingly ageing, the issue of hearing loss and its financial and health cost to society is increasingly evident. Deaf and hard-of-hearing people are often socially isolated, preferring not to mix in social gatherings where background noise makes communication impossible. To this end, the advent of cochlear implantation, or electrical hearing, has revolutionized the lives of the severe and profoundly deaf. Cochlear implants (CI) restore sound perception by directly stimulating residual auditory nerve fibres with patterns of electrical pulses delivered via an electrode array implanted into the cochlea – the hearing organ of the inner ear. Employing up to 22 electrode contacts, CIs attempt to replace the function of all, or nearly all, of the 3000 or so sensory cells of the inner ear that are damaged or absent in individuals with severe and profound hearing loss.
Despite its undoubted success, much remains to be achieved if CI listeners are to perform adequately in even moderately-challenging listening conditions, especially so as CI is largely a ‘monaural’ technology at present, with CI listeners unable to exploit the known benefits of binaural (‘two-eared’) hearing. Binaural hearing refers to the brain’s ability to compare information about sounds arriving at the two ears, is essential for locating the source of a sound and a vital component of ‘cocktail party’ listening (understanding speech in noisy environments); monaural listeners are at a distinct disadvantage under such conditions, being unable effectively to segregate speech patterns originating from different talkers. Currently, even where CIs are provided to both ears (bilateral implantation) they remain independent devices, incapable of exploiting the important differences in sounds at the two ears that render cocktail party listening possible Neurelec is the only manufacturer of CIs that deliver synchronized signals to the two ears, using one processor serving both cochleae. This provides the opportunity of generating synchronised signals to the two ears (truly binaural) compared with using two unsynchronized monaural devices (bilateral). Nevertheless, exploiting the great potential of such technology has not yet been fully realized, including the possibility of exploiting specific binaural processing strategies that make use of the brain’s remarkable ability to compare small timing differences in sounds at the two ears to hear out speech in noisy conditions. The CI prototype to be developed will for the first time provide true binaural stimulation.
2.2 Aims and Objectives
The overall aim of ABCIT, therefore, was to enhance the lives of the profoundly deaf through the development of a binaural cochlear implant (CI) that exploits the spatial information available in the sound input to improve the listening experience of CI patients. Exploitation of spatial information has the potential to enable CI patients to perceive the direction of sound sources and enhance their ‘signal-to-noise ratio’ with the result that their speech perception in adverse listening conditions will be greatly enhanced. The original SME partner within the consortium (Neurelec) offered, with its current CI technology, a unique therapeutic intervention – the closest of all available CI designs to being capable of true binaural processing. Initially aimed at being cost effective, Neurelec’s CI device processes the left and right microphone signals on a single processor, allowing the interaural (‘two-eared”) cues essential for spatial hearing to be extracted and manipulated through the delivery of interaurally-synchronized electrical pulses to both cochleae. The design of this implant, with its galvanic connection across both sides of the brain, also offers the unique opportunity to record neurally evoked responses from the auditory brainstem (‘EABRs’) with the CI device itself, without the requirement of additional electrodes attached to the skin. In contrast to responses generated by the primary auditory nerve (‘ECAPs’), EABRs are informative as to the activity of the binaural brainstem nuclei and can therefore be employed to ensure objective measures are used in establishing and enhancing binaural fitting procedures.
At its outset, ABCIT had 5 main objectives that sought to develop bilateral cochlear implantation beyond the state-of-the-art:-
1. to develop the ability to exploit the full range of binaural hearing cues in cochlear implantation (CI), and to gear stimulation strategies towards enhancing binaural information
2. to develop a research platform, including a speech pre-processing module, to enhance the development of CI processors
3. to adapt currently successful hearing-aid (pre-processing) algorithms to meet the special demands of cochlear implants
4. to develop the means of measuring from the auditory brain neural signals that will provide an objective means of assessing binaural performance in CI users. This goes far beyond the capabilities of all current CI systems, and is based on Neurelec’s unique implant design
5. to develop a low-power, wireless audio link between the devices at both ears so as to enhance users’ access to binaural information
Originally, a 6th objective was to develop a pre-commercial prototype of a ‘second generation’ binaural CI that would provide the first true binaural hearing for CI users. However, during the course of the project it became clear that it was too early to harden all the newly discovered binaural CI knowledge into such a pre-commercial prototype. Instead it was decided to build a flexible system that could demonstrate the feasibility of the new ideas and could also provide a platform for further binaural CI research and development—a binaural CI ‘development kit’.
The WPs were therefore geared towards meeting these objectives. WP1 provided the framework for developing a new hardware device and software to permit research in experimental animals and human subjects to be undertaken by CI researchers. In two-stage process, WP2 developed a simulation tool, one stage involving development of a cochlear model of electrical stimulation and spread of excitation, and the other stage models of binaural processing based on neural firing patterns. Model outputs included predictions of binaural performance in different acoustic scenes. WP3 developed new stimulation strategies for CI, including strategies that take more-informed account of the biophysical properties of cochlear nerve fibres, and the potential for exploiting these properties to limit spread of excitation within the implanted cochlea. It also assessed a range of new stimulation strategies for enhancing binaural information in CIs, including through experimental recordings in small mammals and through behavioural assessment in normal-hearing listeners and CI users. The purpose of WP4 was to develop objective measures of binaural brain function to improve fitting for monaural and binaural performance. By assessing neural activity using electro-encephalography (EEG) in normal-hearing listeners and CI users, potential diagnostic tools can be compared to subjective measures of performance. The combination of bilateral CI technologies with pre-processing algorithms designed to enhance speech processing in noise was assessed for CI users. WP5 assessed the performance of these algorithms in an instrumental evaluation as well as in normal-hearing listeners. WP6 spoke to an original goal of the ABCIT programme, which was to integrate information from WP1-5 into a formal prototype of a new binaural device. At the time of the proposal, the potential of Neurelec’s CI design to underpin our objective to create a truly binaural CI device, capable of autonomous acquisition of EABR responses, was the over-arching goal. Originally, the outcome of meeting the objectives by the end of the programme was to develop a prototype ‘second generation’ binaural CI device that will eventually be brought to market, providing the possibility of true binaural hearing in CI users. However, during the course of the project it became clear that many of the individual elements being generated could be applied to current CI devices, or used to undertake research projects, including to explore the limits of binaural hearing in CI subjects, without the development of the prototype. Further, the acquisition of Neurelec by William Demant Holdings (WDH) – the Danish Foundation dedicated to researching and commercializing solutions for deafness and other forms of hearing impairment and the development of Oticon Medical to bring binaural CI technology rapidly to market, had an energizing effect on the project.
Overall, the ABCIT project has been an outstanding success for the individual partners and for the collective. It has imbued a strong sense of engagement with the commercial partner by the academic partners, and provided significantly more rapid translation of basic scientific discoveries into commercial products and clinical therapies.
There are four cochlear implant companies in the world, two based in Europe. Neurelec is the smallest of all of the companies (10% market share) but has been growing strongly due to recent innovations. In particular, their world's first (and still only) binaural device offers significant value to patients and health providers by providing the benefits of two implants for the cost of a single implant, as well as the opportunity to restore truly binaural hearing. Although Neurelec’s market and market share have been growing, the company now needs to increase its research efforts in order to establish itself as a major industry competitor. The current proposal is specifically designed to expand Neurelec's market reach by providing industry-leading technological solutions for better CIs, and includes provision for testing and demonstrating the benefits of the solutions in patient groups. These solutions are vital if Neurelec is to continue to be competitive in the worldwide CI market. The advances being targeted include: tools to improve rapidly the design of cochlear electrodes and stimulation strategies, the development and testing of new CI-specific speech processing algorithms, the development of specific binaural algorithms for Neurelec’s existing and future-generation binaural devices, and investigations into the use of objective measures (e.g. advanced CAP analysis and EABR) for automated fitting - something that is now (within the last 5 years) considered essential for continued operation in the paediatric implant market. Ultimately, the goal is to gear these solutions towards improving hearing health outcomes, improving Neurelec’s current market position, and expanding into the US market, a market from which Neurelec is currently completely absent. The time-to-market for these solutions comes in two stages: (1) New input algorithms or binaural algorithms could be implemented in year 2 of the programme, following the development and testing of the algorithms by HörTech/Neurelec. These advances are also potentially backwards compatible to current generation CIs. If next generation implants are required, the benefits will follow the timeline of those releases (typically new processors released every 3 years). (2) The development of new electrode arrays and measurement of objective responses would benefits next-generation implant users at 3 years.
Project Results:
4.0 Description of the main S & T results/foregrounds
Binaural hearing confers considerable advantages in everyday listening environments. By comparing the timing and intensity of a sound at each ear, listeners can locate its source on the horizontal plane, and exploit these differences to hear out signals in background noise – an important component of ‘cocktail party listening’. Sensitivity to interaural time differences (ITDs) in particular has received much attention, due in part to the exquisite temporal performance observed; for sound frequencies lower than about 1.3 kHz, ITDs of just a few tens of microseconds are discriminable, observed both at the neural and behavioural levels. Sensitivity to ITDs decreases with ageing (Herman et al., 1977; Abel et al., 2000; Babkoff et al., 2002), is typically impaired in hearing loss, and sensitivity to ITDs is typically poor in users of hearing technologies such as bilateral cochlear implants (CIs). Thus, although CIs are the most successful sensory prosthetic devices developed to date, as judged by their ability to restore sensory and motor function (i.e. hearing and normal speech patterns) in the profoundly deaf, considerable progress remains to be made if CI users are to be able to hear and communicate in even moderately challenging ‘cocktail party’ environments where sources must be localized and individual conversations heard out from a background of multiple talkers, room reverberations and extraneous noise sources. The advent of bilateral CI (an implant in each ear), therefore, provides the perspective for achieving this improvement in performance but, to date, the expected improvements have not materialized. At least part of the reason for this impasse is that CI technology was never designed to take account of binaural hearing; from the persistence of monaural implantation, to the lack of any post-surgical fitting procedures that take account of binaural or spatial listening skills, or the lack of diagnostics for assessing performance in the increasing number of listeners (particularly young children) implanted bilaterally, the lack of a binaural perspective in CI hinders improvements in listening performance in CI users. The purpose of the ABCIT (Advancing Binaural Cochlear Implant Technology) project, then, is to progress CI technologies to the point that the potential benefits of binaural hearing for CI users are maximized, providing engineers, clinicians and researchers with the tools to explore. Divided into 6 inter-related Work Packages (WPs), the programme of research and development undertaken in the ABCIT project generated new research tools for investigating binaural hearing in BiCI users, provided diagnostic tools for assessing binaural performance, and assessed novel and existing ‘pre-processing algorithms’ that exploit binaural cues to ‘clean’ (i.e. remove background noise from) speech signals before they are processed by the implanted devices. Below, under the framework of the 6 WPs, we outline the research outcomes from the ABCIT project, highlighting some of the major advances made in the project, and illustrating how these advances will be used to improve listening performance in users of bilateral CIs.
1.0 Developing a Research Platform for Binaural CI (WP1)
WP1 developed a new research platform - the Neurelec ‘cHIRP’ – a research tool designed to generate data to enable researchers to maximize the benefits of binaural listening in BiCI listeners. The cHIRP research platform is specifically designed to focus on binaural CI processing by synchronously driving two ‘Neurelec SP’ implants to produce binaural cues (interaural time and level differences, ITDs and ILDs, respectively). In addition, the platform also supports variable stimulation rate processing for studying the effect of stimulation rate on perception (e.g. the effect of this rate on the perception of soundpitch). The cHIRP platform consists of bespoke hardware and associated PC software joined via USB connection. The hardware is responsible for: (1) digitizing the 4-channel input audio signals coming from two ear-worn microphone systems and (2) generating the final electric outputs needed to drive the two antenna coils. The software is responsible for processing the four audio signals and then generating two synchronized electrodograms from these signals. The software includes a flexible environment for the development of sound pre-processing (“speech processing”) and stimulation strategies. This architecture allows a diversity of approaches - including spectral and spatial signal enhancement and novel stimulation strategies (within the limits of the SP implant) - to be implemented and tested on real CI users without the need to create new hardware. The cHIRP integrates the Master Hearing Aid (MHA) developed by Hörtech.
1.1 Overview of the ‘cHIRP’ Research Platform
The cHIRP research platform provides a complete signal processing platform for cochlear implants capable of implementing and evaluating state-of-the-art pre-processing algorithms or novel ones in real-time. Figure 1.1 shows a block diagram of the hardware and software parts and indicates their tasks and interconnections.
The cHIRP hardware part is a PCB board (Figure 1.2). It is responsible for handling all ‘real-world’ signals and translating to and from these signals. The main tasks of this board include: (1) digitizing the acoustic input, (2) providing a first-in-first-out (FIFO) buffer and robust communication protocol for receiving commands, (3) producing the outputs needed to drive two CIs and (4) providing triggering and synchronization of all signals. The board receives 4-channels of acoustic input through two standard 3.5mm stereo jacks. Typically 4 microphones are used (two each from left and a right behind-the-ear (BTE) device worn by the subject). These signals are all digitized synchronously, packetized, and then transferred via USB to the PC-side MHA software for processing. The cHIRP platform has two Neurelec SP implant coil drivers that are driven from the same clock, and stimulation command packets are sent via USB from the PC-side MHA software. The exact timing of outputs to external events is controlled by selecting a specific trigger. Timing of relative interaural output ITDs (left vs. right stimulation) is encoded directly within the stimulation command packet that always contains binaural output data.
The PC-side software part of the research platform is a real-time signal processing system - the MHA (Grimm et. al., 2006). Originally designed for conventional hearing-aid research, the MHA allows many different signal-processing algorithms (e.g. noise reduction, beam-forming etc.) to be run in software at low latency. These algorithms can be cascaded to produce complex systems, and new algorithms can be coded at any time through the MHA’s ‘plug-in’ architecture. Originally designed for conventional hearing-aid research, the MHA allows many different signal-processing algorithms (e.g. noise reduction, beam-forming etc.) to be run in software at low latency. These algorithms can be cascaded to produce complex systems, and new algorithms can be coded at any time through the MHA’s ‘plug-in’ architecture.
1.2 Capabilities of the ‘cHIRP’
Outlined below are some of the capabilities and features of the cHIRP that make it well suited as a research tool to investigate real-time function in cochlear implants.
1.2.1 Latency
Latency between audio input and electrical output needs to be kept low in order to prevent ‘lip sync’ errors or the deterioration of speech production for test subjects. The minimum system delay for avoiding visual lip reading to auditory speech reception is ~17ms. Disruption to speech from delayed auditory feedback is maximum at about ~200ms [2-4] and is not expected to be a factor below 20ms.The actual minimum latency achieved for the various stages of the platform between the input acquisition and the stimulation without using any specific signal-processing algorithm (using a pass-through identity plug-in) are listed below.
1.2.2 Synchronization Accuracy
Because the system is focused on the binaural CI processing, the ear-to-ear (ITD) synchronization target was 15 µs based on the just noticeable difference (JND) of ITD for normal-hearing individuals at low frequencies (500-1kHz) [5]. However, one sample at 16.667 kHz corresponds to 60μs. Therefore, the entire system was designed to be sample accurate and synchronous. Since ITD information is carried within the stimulation packet itself, synchronization of coil outputs can be completely controlled and was measured to be better than 1µs with 0.5µs resolution. However variations in the processing delay between the reception of coil information and stimulation output of the implants means the synchronization cannot be guaranteed and must be assessed individually for each subject using equipment used to measure eABRs (electrical auditory brainstem responses).
1.2.3 Triggers
Neurelec SP implants stimulate all electrodes sequentially upon receiving a coil command. Hence, for the cHIRP platform, all electrodes must operate from the same global trigger. Four different stimulation output triggers are available: (1) an internal trigger based on a hardware timer, (2) an external trigger via BNC, (3) an audio synchronized trigger activated on audio packet transmission initiation, (4) and an immediate trigger that stimulates immediately upon receiving a stimulation packet. The internal trigger timer has a range from 1 Hz to 1000 Hz, settable in increments of 1 Hz. For the internal trigger, care must be taken not to over-run or starve the hardware FIFO. This is addressed by the communication protocol and, on the PC-side software, via an MHA output plugin. An external trigger is provided for interfacing with third party hardware such as clinical eABR machines or precision timers. The audio synchronized trigger creates an outputted stimulation for every audio packet initiated. This corresponds to the default rate for Neurelec SP implants of 520 Hz. It allows the lowest possible latency because no buffering is needed. The immediate trigger allows variable rate strategies to be tested. Since the entire array is sequentially stimulated on each trigger, dynamically changing the rate affects all electrodes (although electrodes can be disabled). It can be used to evaluate the effect of rate on pitch for example.
1.3 Signal processing and implementation of algorithms
The PC-side software, based on the MHA [1], is real-time capable software platform for developing and evaluating signal-processing algorithms employed in hearing aids. It supports algorithm developers with an extensive ‘toolbox’ that can be used to develop quickly new signal-processing algorithms. It comes with a collection of ready-to-use hearing-aid algorithms that can be combined with user-developed algorithms to form a complete software-based hearing aid capable of processing signals in real-time.
1.4 Intended Use of the Research Platform
The cHRIP research platform will be made available to researchers who are interested in investigating CI signal processing. Because the CI signal processing is performed on a PC, it is accessible to inspection and alteration. The platform can perform extensive data logging, replay of stored input signals, and alteration of existing algorithms and insertion of additional algorithms into the processing chain. New algorithms for the MHA are developed as plug-ins, and the signal path through the algorithms as well as algorithm settings can be altered at run time, making it possible to compare different algorithms or algorithm settings with subjective measures such as speech intelligibility, listening effort scaling, paired comparisons etc. Often in developing a hearing system trade-offs must be made between increasing the SNR of a signal and preserving binaural cues. Exploitation of spatial information will enable CI patients to perceive the direction of sound sources and will increase their ‘signal-to-noise ratio’ with the result that their speech perception in adverse listening conditions will be enhanced.
The PC running the MHA can be a portable computer, allowing the subject to carry the research platform with him or her and thereby use it outside of a laboratory. In a laboratory setting, the platform may additionally be connected to an audiological workstation, performing audiological tests by replacing the microphone input signal with the output of the audiological workstation, if desired. The MHA has been used to test different combinations of state-of-the-art speech enhancement algorithms on CI patients [6, 7, 8]. On these tests, the incoming signal was enhanced in the MHA and the enhanced signal was sent to the CI device as an audio signal. The CI device converted the audio signal into electric stimulation and performed the stimulation independent of the platform.
1.5 Examples of the use of the cHIRP Research Platform
In order to demonstrate the capabilities of the cHIRP, we highlight two examples using the research platform running in real-time. The first example shows the simultaneous production of ITD and ILD cues. The second example shows a variable stimulation rate. In both examples, we used simulated audio inputs to more clearly demonstrate them, however, the platform is equally capable of running on real-world microphone input.
1.5.1 Example 1: ITD/ILD Encoding
Figure 1.3 shows two snapshots of an oscilloscope depicting two instances of stimulation frames. These snapshots were taken during the same run of the platform at two different time instances. The bottom (yellow) traces show the digital output of the research platform for the left ear and the upper (red) traces shows the analog the coil voltages on the right ear at two different time instances. These traces encode the intensities of the stimulation in each frequency band. The audio input used to generate these stimulation frames was moving in the frontal hemisphere continuously and is composed of one low frequency (255 Hz) and one high frequency (4 kHz) component. The time delay between the upper and lower traces is caused by the calculated ITD. The ILD is encoded within the trace, so is not visible directly.
1.5.2 Example 2: Variable Stimulation Rates
Figure 1.5 also shows two snapshots of the oscilloscope taken while varying the stimulation rate on the fly. The time interval between two successive traces changes between the left and right panel depending on the fundamental frequency of the audio input. Here, the fundamental frequency was pre-determined, but zero crossings or other techniques could be used to extract the timing information. Here the stimulation rate alternates between ~2ms (500 Hz) and ~1.2ms (833 Hz).
1.6 User Safety
A clear consideration of any device employed to assess human performance is safety. Protection from over stimulation is ensured on the PC-side MHA software by way of a stimulation plug-in that cannot be circumvented. This plug-in reads clinical patient fitting files and blocks any stimulation that exceeds ‘C’ levels specified within the files. To ensure data are not corrupted in transmission (via the USB), the plugin contains a 16-bit CRC key for each stimulation packet. Any detected CRC errors on the hardware side results in the packet being dropped and no stimulation.
1.7 Conclusions
The cHIRP research platform provides researchers pursuing research on bilateral CIs the complete processing chain of a cochlear implant. It enables them to reconfigure this chain easily according to their needs and thereby filling an important gap in CI research by combining hardware and software within the same platform. The cHIRP will be useful for performing psycho-acoustical experiments on CI patients, including the electric stimulation of the cochlea for measuring the improvement in their speech intelligibility for different combinations of pre-processing algorithms. In the future, the platform will be extended to support wireless connections of microphones and will be made smaller in size for better portability, enabling researchers (and patients) to perform tests more easily in real-world environments. As any algorithm with real time capabilities can be implemented as a new plug-in into the Master Hearing Aid, it allows an arbitrary combination of different speech enhancement algorithms. The electric stimulation patterns are also generated in the Master Hearing Aid and transferred to the research board, which performs the stimulation.
2.0 A Simulation Tool for Developing and Assessing Binaural CI (WP2)
Computer simulations that model electrical stimulation of the auditory nerve have significant implications for CI research. First, they represent a cost-effective solution for product development and validation. Second, by describing accurately the effects of the different variables in play, they allow for new hypothesis to be formulated and, later, tested. Finally, they can illuminate potentially less-thoroughly described elements of the information chain. As such, simulation tools increase research efficiency by pre-validating clinical trials, saving time and resources. At the commercial level, CI simulation models can also be used to evaluate different processing and stimulation schemes and selecting promising candidates for animal or human testing. The simulation model can also be used as a reference model for developing and verifying new devices. The simulation model can be used as a front end for a research hardware interface to test new stimulation and processing schemes on existing patients
2.1 Modelling design
Different approaches have been undertaken when modelling the effect of electrical stimulation on the nerve. The essential reason for all these approaches is to find an acceptable trade-off between the details of the models and the complexity or computational cost required. The trade-off issue faced by a CI simulation tool is to find an optimal level of modelling accuracy that describes neural encoding by electrical stimulation, but without resorting to excessive computational resources. This trade-off requires that a ‘modular architecture’ be employed in the design. To this end, the simulation tool includes several modules each corresponding to a single element of the auditory pathway, from the electrode array positioned within the cochlea, to electrophysiological measurements of auditory-brain activity (Figure 2.1). By building the simulation tool as a chain or pipeline of modules, each element can be refined independently from data obtained during fitting, or they can be optimized for computation once correct accuracy is achieved.
Modeling and simulations can have different purposes and uses, depending on computational complexity and the desired inputs and outputs. To this end, tABCIT developed a realistic model of electrical stimulation by a CI - one linked to a specific Neurelec device for sound processing and electrical stimulation, rather than a generic implant. It also developed a model of binaural processing by auditory neurons to assess binaural responses to electrical stimulation of both ears through a Neurelec device. Key features of the model were its flexibility (allowing ‘evolution’ of the model), a capacity for fast simulations (low computational complexity) and ease of use for research and commercial purposes (e.g. through development of user-friendly tool on the Matlab programming platform).
2.2 Modeling Neurelec devices
Several Neurelec devices and processing strategies were simulated in the model, which included all processing parameters, as well as precision errors related to hardware, for example. This simulation is therefore a realistic and precise model of the Neurelec CI system, and is now employed within the company for verification of in-house ‘firmware’. The simulation time is also limited, as sound processing is required to be performed in real-time by external CI processors using minimal power. One outcome of this approach was that the development of firmware at Neurelec was altered so that all devices manufactured by the company are simulated with the new Matlab simulation tool. New ideas and coding strategies under development are all first simulated on the tool, and newly developed coding schemes can be encrypted and compiled as dynamic link libraries (DLLs) usable in Matlab or in the ‘C’ programme language confidentiality, but also easily shared and used as ‘black box’ model components by any of the research partner using the simulation tool.
2.3 Modeling cochlear geometry and the electrode array
A 2D model of electric field/current spread was first designed to provide quantitative predictions of the interaction between electrode contacts. This model is sufficient to produce a coarse approximation of those interactions and to provide the inputs necessary to calibrate the model of the auditory nerve fibres, with very fast computation. The level of detail, however, is not sufficiently accurate enough to provide critical information such as stimulation performance depending on intra-cochlear electrode array position, optimal design of apical electrodes, etc.
A 3D model of the cochlea was also developed and is currently being added to the simulation tool. PhD projects are still running to work on the generation of 3D cochleae based on patient data (obtained from CT-scans of the cochlea prior to operations to insert CI devices), with low calculation complexity. In order to achieve this, a parametric cochlear model was developed from excised human temporal bones, and this a priori knowledge of cochlear geometry can be used on low-quality clinical CT scans to reconstruct realistic cochleae using 3D modeling. The model is parametric and is configured with the Neurelec EVO electrode array. Thanks to parameterization of the 3D model, the position of the electrode can be freely chosen and the surface mesh can be modified to ensure that the electrode edges are properly modelled.
2.3 Neural models
A stochastic model by Goldwyn et al., (2012) has been used to model the responses of auditory nerve fibres to electrical stimulation (Figure 2.4). The model parameters have been fitted (by the authors) to recordings of single-fibre responses to electric stimulation. The model includes the effects of neural refractoriness and summation, and reproduces the difference in spike-timing precision observed for various stimulation rates. The model does not include the effects of low-threshold potassium channel or hyperpolarized activated currents, both of which are known to produce specific type of adaptation without spiking at high rates of electrical stimulation. However this model is realistic enough to generate spike patterns in response to a given electrical stimulation with fast computation.
Spike patterns are analyzed to evaluate perceived sound localization (angle and azimuth). The model is now binaural, and is sensitive to ITDs. In terms of estimating the azimuthal direction of arrival of a sound with the current Neurelec stimulation technique, ITD coding is of very limited use, because pulse timing is independent of the temporal fine structure. Therefore, the focus of the model was placed on assessing performance in response to interaural level differences (ILDs), expressed as the ratio of neural activity (spikes) of the right and left brain hemispheres. The model can track ILD changes over short (e.g. 200-ms) time intervals, can output several ratios for specified blocks of fibres along the cochlea, and can be calibrated and operated through a flexible and user-friendly graphical user interface, as well as an interactive 3D visualization interface of the azimuth localization data (Figure 2.5).
2.4 Assessing localization abilities in CI users
The model was used to simulate the spatial hearing abilities of bilateral CI listeners, with predictions similar to the subjective performance of bilateral CI listeners, not only in terms of the average localization error but also in the occurrence of systematic errors in localization judgments. In general, the model confirmed a range of hypotheses from experimental studies, including the dominance of ILD cues in lateralization judgments, and the detrimental effect of level-dependent compression. The model also enables identification of the origin of any unexpected prediction along the processing chain. Beyond that, the model could be useful in predicting performance of individual CI subjects by customizing the model to their clinical profile, and in developing algorithms to enhance spatial hearing performance.
2.5 Modeling cochlear anatomy and electric fields
The 3D modeling of the cochlear anatomy, as well as electric field generated by a given electrode-array, are critical topic for electrode design. Two PhD projects were initiated on those two topics during the course of the ABCIT project, in collaboration with local (to Neurelec) academic partners outside the consortium. The parametric model of cochlear geometry was developed by one of these projects (Figure 2.6) with the long-term objective of generating patient-specific cochlear geometry in order to predict electrode insertion and placement. Eventually, electrode array insertion might also be dynamically modeled, in order to estimate cochlear damage and trauma upon insertion. For a given electrode position, the combination of bone, soft tissues and liquids, all with different conductivities, makes the electric fields difficult to predict. In order to better employ multipolar stimulation, or current steering/current focusing techniques, the second PhD has modeled electric fields within the cochlea, for a given combination of anatomical configuration and electrode-array (Figure 2.7).
2.6 Product development
The tool has been already use by Neurelec to evaluate coding strategy ideas and new concepts. Without such a model, it would be necessary to undertake clinical investigations or animal experiments, both of which require ethical approval and, in the case of human trials, verification and validation tests to obtain other regulatory clearance for non-CE approved prototypes and software. Both prototype and study designs have to be set, and study protocols and technical dossiers advertised. Depending on prototype complexity and study design, all clearances may take 6 months to a year, and costs are around 5-10k€. Further, the study itself requires coordination and project management (especially if the trial is conducted at several hospital sites), and participants have to be informed prior to inclusion. Trial budgets are also required to cover hospitals additional costs, estimated at 0.5-2k€ per subject, depending on study design (number of visits especially). A pilot clinical study may use 10 participants, up to 50 participants for a high-proofed trial, with study duration of 3 months to 1 year per subject. Altogether, a clinical trial may long 1 to 2 years in total, and may cost 10 to 100k€, with very limited possibility of fine-tuning. In comparison, model using fast prototyping using the Matlab model is very convenient, and first results are generated rapidly. Even though clinical tests are mandatory for final verification, all fine-tuning of CI stimulation algorithms can be performed with the model.
The model has been was used to evaluate new coding strategies and fitting parameters: spike patterns are generated with commercially available coding, with new prototype coding, and for the normal-hearing auditory system, as well as to estimate the battery life of the prototype CI system. In fact, as the charge delivered by the electrodes can be linked to the generation of neural action potentials (and, therefore, sound perception), the model helps estimate the useful energy with realistic clinical maps (thresholds and comfort clinical levels observed in real patients, data available to Neurelec). Future perspectives and product development plans with use of the simulation tool, following completion of the ABCIT project, include the connection to the research hardware platform developed as part of WP1, further tests and verification on signal processing optimization and stimulation strategies.
2.7 Conclusions
The development of a simulation tool for exploring neural responses to bilateral stimulation of CIs is a critical change on the commercial side, and has changed the development process within Neurelec/Oticon Medical. Now, all signal processing schemes and stimulation strategies developed and under development are designed and benchmarked through the model. Electrode-arrays design also benefit from modeling, in order to better predict placement, and electric fields in the cochlea. This allows a dramatic reduction of animal and clinical tests, with first hypothesis tested and compared numerically. Therefore, more risks can be taken and foreseen in terms of product development, as more ideas can be tested. Neurelec/Oticon Medical is now more proactive in development process, and our engineers no longer wait for published concept to initiate product development but rather propose new processing and coding schemes for development based on the model output.
References
Daanouni, E., Segovia-Martinez, M., Frater, A., Gnansia, D., Hoen, M. (2015). Residual hearing preservation, simulated and objective benefits from electroacoustic stimulation with the Evo-Zebra system.
Dang, K., Clerc, M., Vandersteen, C., Guevara, N. and Gnansia, D. (2015). In situ validation of a parametric model of electrical field distribution in an implanted cochlea. Proceedings of the 7th International IEEE EMBS Neural Engineering Conference (NER'15), April 22-24, Montpellier.
Frater, A., Maas, P., Undurraga, J., Kamaric Riis, S. (2015). Electro-acoustic cochlear implant simulation model.
Frater, A. (2014). Development and objective evaluation of an EAS cochlear implant model. Aalborg University Master thesis.
Goldwyn, J., Rubinstein, J., Shea Brown, E. (2012). A point process framework for modeling electrical stimulation of the auditory nerve. Journal of Neurophysiology, 5:1430-1452.
Hines, A., Harte, N. (2012). Speech intelligibility prediction using a neurogram similarity index measure. Speech Communication, 54(2):306-320.
Huang, T., Frater, A., Segovia-Martinez, M. (2015). A Gammatone filterbank and zero-crossing detection approach to extract TFS information for cochlear implant patients
Kelvasa, D., Dietz, M. (Submitted). Auditory model-based sound direction estimation with bilateral cochlear implants
Maretic, P., Frater, A., Kamaric Riis, S., Widell, A., Segovia-Martinez, M. (2015). Optimizing compression stimulation strategies in cochlear implants for music perception.
Segovia-Martinez, M., Daanouni, E., Gnansia, D., Hoen, M. (2015). Automatic post-spectral compression for Oticon Medical’s sound processor
Vatti, M., Frater, A., Huang, T., Segovia-Martinez, M., Pontoppidan, N. H. (2015). A phase-lock loop approach to extract temporal-fine structure information for cochlear implant patients.
3.0 Developing Novel Electrical Stimulation Strategies (WP3)
Many issues impact on the efficacy of cochlear implants. A general problem is the uncontrolled spread of electrical current within the cochlea when stimulating with even a single electrode contact. This results in many more auditory nerve fibres being stimulated than is desirable, and contributes to the poor sound-frequency resolution observed in CI users. In terms of bilateral CI users, the restoration of binaural hearing is hampered by the relative insensitivity of neurons to temporal information. Finding ways to reduce the spread of electrical current and enhance temporal coding so that interaural time differences are more readily conveyed in implants, were major goals of the ABCIT project. WP3 also assessed the extent to which ITD cues might most readily be delivered to CI users, and established new means of testing binaural performance, based on experimental work undertaken throughout the ABCIT project.
3.1 Ramped pulses: a novel stimulation strategy
At least some of the limitations in CI performance come about because of the design of the devices themselves. CIs work by electrically stimulating the primary auditory neurons (the spiral ganglion neurons, SGNs) that constitute the auditory nerve, through an electrode array inserted into scala vestibuli of the cochlea. A major factor limiting improved performance in CI is uncontrolled spread of electrical current within the cochlea itself (Middlebrooks 2004). This current spread is an intrinsic consequence of the implant design; the size of the electrodes, their distance from SGNs, and the fact that they are immersed in a highly conductive fluid environment, leads to a wide population of SGNs being activated with each electrode. Controlling the spread of excitation in situ remains problematic, and severely limits the number of independent channels of acoustic information that can be represented in electrical stimulation. We addressed this engineering-to-biology mismatch with an innovative stimulation strategy using novel pulse shapes based both on biophysical principles of SGNs and on the interaction of the electrical stimulus with the cochlear environment.
3.1.1 Theoretical considerations for employing ramped pulses in CI
We designed a new ‘ramped’ electrical pulse for CI stimulation (Fig 3.1A) to replace the square pulses (Figure 3.1B) most commonly employed in current CI technologies. Figure 3.1A shows the ramp pulse and its parameters, including the Foot (FA) and Peak (PA) amplitudes where FA is the amplitude at the start of the slope and the PA the maximum amplitude of the slope. In addition, the ramp pulse can also be defined in duration of phase (PD). and the inter phase-time duration (IPI).
The rationale behind this new stimulus shape relies on two important features of auditory neurons and the cochlea itself - the biophysical properties of the SGNs, and the principle that stimulation in aqueous media (here, the fluids of the inner ear) results in diffusion of the electrical current (Figure 3.2). The theoretical outcome of ramped pulses is that they will reduce the influence of the spread of current on neurons, without the need for an actual reduction in the spatial current spread. For two current pulses with equal peak value, one square (Figure 3.1A) the other with a ramp of defined slope (Figure 3.1B) the SGN at the focus of the stimulation will have a higher firing threshold for the ramped pulse. However, moving further from the stimulation point, the combination of the reduction in amplitude and in slope will result in a faster decrease in firing for the ramped stimulus compared to the square one, therefore effectively reducing the spread of excitation (Figure 3.2). Ramped current pulses can be defined according to their slope, specifically the rate at which the slope increases. Considering the dynamics of ion channels in SGNs, the rate of change of current will result in increased or decreased excitation of the neural population proportional to the slope.
3.1.2 Responses of SGNs to injections of square current pulses
We assessed whether or not ramped pulses are more efficient than square pulses for modulating the neural firing patterns by recording responses of SGNs from cochlear cultures from P12-15 mice to injected current pulses with either the traditional, square shape or with the new ramped design. A key outcome was to characterize the effect of the different stimulus parameters on SGN firing in order to better predict the effect of this new stimulation paradigm in the context of electrical hearing. We focused on how changing the different parameters, including the FA and the PA, might affect the efficacy with which current pulses evoke action potentials in SGNs. Our data (Figure 3.3) demonstrate that all the parameters proposed as part of the ramp stimulus influence SGN firing and could be beneficial to electrical stimulation in CI. Importantly, for populations of SGNs, the ramped pulses showed a greater range of responses (more variability in firing for a given input current level) meaning that the dynamic range (the range of sound intensities encoded by the neurons) is increased by the ramped stimulus.
3.2 Designing stimuli to convey ITD information
We also sought to establish which stimulus shapes are most effective in conveying ITD information in CI listeners. Currently, most CI listeners have access only to ITDs conveyed in the envelope of modulated sounds, and not those conveyed in the temporal fine structure of low-frequency sounds – the latter is the type of ITD information that provides significant benefit in noisy listening situations. Do specific envelope shapes exist that more readily convey ITD information, and can these be provided to CI users to enhance their spatial listening? Envelope ITDs are the only ITD cues available through today’s CI devices, and modification of the envelope, e.g. by enhancing the rising slopes, might improve users’ perception of auditory space.
3.2.1 Determining the best envelope shapes for conveying ITDs
Following a study by Klein-Henning et al (2011) in normal-hearing human listeners, we assessed the responses of neurons recorded from the midbrain of anaesthetised guinea pigs to a range of different envelope shapes. These shapes varied in four envelope components – attack, sustain, decay and pause (see Figure 3.4) were systematically varied, and confirmed the importance of the attack and pause components in generating ITD-sensitive responses. We also used a simulation model (such as that described in Section 2 above) to assess the neural responses we obtained. Figure 3.5 shows ITD tuning of neurons in the guinea pig midbrain (2nd row) for various types of the stimulus envelope (upper row). The lower row shows the ITD tuning mimicked by the simulation tool.
This study showed the importance of the rising slope in conveying envelope ITDs, and that their enhancement could potentially benefit neural sensitivity to ITDs sensitivity. Other studies of human performance found that binaural sounds with steep attack phases, and long pauses were those to which listeners were most able to lateralize to one side or the other (Klein-Hennig et al, 2015).
3.2.2 Determining where in the sound envelope CI users are sensitive to ITDs
Sound localization in reverberant environments is complicated by the fact that sound reflections carry ITDs and ILDs that can be very different to those actually directly from the source. CI listeners have great difficulty with reverberation – they are unable to separate out the mixture of direct sound and later-arriving reflections. To this end, a beneficial strategy would be to provide greater weight to the interaural differences at the onset of the stimulus or at the onset of each modulation cycle. During these onsets, the direct-to-reverberant ratio is typically optimal, and the interaural differences in this short moment are informative as to the source location. We tested where in the modulation cycle CI users actually extract ITD information compared to normal-hearing listeners. We found that in amplitude-modulated sounds, such as speech, normal-hearing listeners show strongly enhanced sensitivity to ITDs during the early rising portion of the modulation cycle, reflecting a higher weight on temporal read out for that signal portion. This is were ITD information in reverberant rooms is most reliable as to the true location of the sound source BiCI listeners, however, appear to be most sensitive to ITDs later in the sound wave-form, meaning that the later-arriving spurious ITDs – the result of reflections – will dominate their perception. The information from these experiments is now being used to develop new algorithms that will enhance ITD sensitivity in CI listeners.
3.3 Conclusions
The novel approach to stimulation we propose for CI stimulation suggests shaping individual electrical pulses to take account of the biophysical properties of SGNs and, in so doing, reduce the spread of excitation within the cochlea that comes from electrical stimulation. The two key features of this new strategy are: 1) the introduction of a slope in the stimulation pulse and the consequent effect this will have on the current spread, and 2) the sensitivity of auditory neurons to the rate at which the magnitude of current pulses is increased. Our in vitro data from SGNs represents a proof-of-concept for a new stimulation strategy for CI. We now plan to assess the potential for ramped pulses to limit the spread of neural excitation using Oticon Medical’s Animal Stimulation Platform (ASP), developed on the basis of a new generation stimulation chip. The binaural psychophysics, neural recordings and modeling demonstrate the important of the rising, or attack, phase of sound envelopes in conveying ITD information. Since these envelope ITDs are the only cues available to CI listeners, it suggests ways in which the envelope attack might be sharpened to provide better spatial hearing in BiCI users.
4.0 Objective Measures of Binaural Hearing (WP4)
The potential to provide CI listeners with true binaural hearing and to exploit the sensitivity of the auditory brain to differences in the sound presented to the two ears as a tool for fitting (matching) the populations of nerve fibres activated by electrical stimulation of the two ears are key goals of bilateral CI research. To this end, considerable effort was put into developing objective measures of binaural activity – measurements of brain activity that report sensitivity to binaural cues such as ITDs (e.g. He et al. 2010) – and in comparing these measures to traditional, subjective measures (e.g. patient-reported perceptual comparisons of the pitch evoked in each ear by stimulation of different electrodes). A critical outcome of this aspect of the ABCIT project, therefore, was to develop the tools required to assess factors known to be critical to binaural hearing, including how well matched is the electrical stimulation across the two ears (Hu at al. 2015a). Two broad research themes were investigated. The first developed new measures of ITD sensitivity using electroencephalography (EEG) in normal-hearing listeners, and sought to apply these measures to bilateral CI users. The second employed traditional EEG measures of binaural integration (the binaural interaction component, or BIC) and assessed the magnitude of the BIC as a function of stimulated electrode pairings across the ears in CI users, comparing the preferred electrode configuration with those suggested by purely subjective (perceptual) measures of electrode pairs. A key goal of both themes was to develop truly objective fitting procedures for matching implant stimulation across the ears, including in situations (e.g. children) where it is difficult or impossible to obtain subjective assessments of CI performance.
4.1 Developing an objective measure of ITD sensitivity
We developed an objective measure to assess neural sensitivity to ITDs conveyed in the temporal fine structure (TFS) of low-frequency sounds, and to distinguish it from neural sensitivity to ITDs conveyed in the temporal envelope of amplitude-modulated (AM’ed) high-frequency sounds. Such a measure would be useful in developing stimulus strategies to improve spatial listening in CI users (who currently can only access binaural cues conveyed in the temporal envelope). Using EEG, we recorded brain activity to sounds in which the interaural phase difference (IPD) of the TFS (or the modulated temporal envelope) was repeatedly switched between leading in one ear or the other. When the amplitude of the tones is modulated equally in the two ears at 41 Hz, the interaural phase modulation (IPM) generates an intracranial percept of a sound moving from one side to the other – and evokes an IPM following-response (IPM-FR) in the EEG signal (see Figure 4.1).
4.1.1 Sensitivity to IPDs conveyed in the temporal fine structure of low-frequency sounds
EEG recordings were obtained for different rates and depths of IPM, generating a total of 30 conditions (21 dichotic, 9 diotic), each lasting ≈5 min, so that the total recording time was approximately 2.5 hours. For low-frequency signals (520-Hz carrier), IPM-FRs were reliably obtained for a wide range of modulation rates and IPDs, and were largest for an IPM rate of 6.8 Hz and when IPD switches (around 0°) were in the range 45-90°. Increasing the modulation frequency increased the magnitude of IPM-FRs. Figure 4.2A plots the spectral magnitude of a typical recording for a dichotic condition with IPM of ±67.8° and IPM rate of 6.8 Hz, in which a significant response was observed for the frequency bin corresponding to the IPM rate (black arrow), and corresponding diotic conditions are shown in Figure 4.3 - for which no response to the IPM rate was observed (see Ross, 2008). Note that in both conditions the ASSR to the AM rate (41 Hz) was clearly observed.
4.1.2 Sensitivity to IPDs conveyed in the modulated envelope of high-frequency sounds
With current state-of-the art devices, BiCI listeners would not be expected to be sensitive to ITDs conveyed in low-frequency sounds, but many appear to show some behavioural sensitivity to ITDs conveyed in AM’ed high-frequency sounds. To develop an objective measure of high-frequency envelope ITD sensitivity, we recorded EEG recordings in normal-hearing listeners to ITDs conveyed in the stimulus envelope; 3000-Hz tones were modulated with a transposed envelope (Bernstein and Trahiotis 2002) at 128 Hz, and a second-order AM (41 Hz) applied diotically. IPDs of the 128-Hz envelopes were switched between ±90° IPD (around 0°) or from systematically between 0° and 180° (i.e. transposed phase of -90° in one ear and +90° in the other). In contrast to low-frequency tones, the magnitude of the IPM-FR for IPDs conveyed in the modulated envelopes of high-frequency (3-kHz) tones was relatively low for IPMs switching between ±90° IPD. Therefore, we also assessed EEG responses to IPMs switching between 0° (in phase) and 180° (anti-phasic) IPD conditions. These latter switches evoked considerably larger EEG responses (Figure 4, left). The magnitude of the IPM-FR decreased with increasing carrier frequency, and was sensitive to an offset in the carrier frequency between the ears (Figure 4, right).
IPDs conveyed in envelope of high-frequency tones (3-kHz tones AM’ed at 128 Hz, with a second-order modulation of 41 Hz) also generated a reliable pattern of IPM-FRs, but one in which response maxima occurred for IPDs switched between 0° and 180° IPD. The data are consistent with the interpretation that distinct binaural mechanisms generate the IPM-FR at low and high frequencies, and with the reported physiological responses of medial superior olive (MSO) and lateral superior olive (LSO) neurons in other mammals.
4.2 Objective measures of binaurally matched electrodes
Although users of BiCIs can take advantage of some of the benefits of having two sound receivers - enjoying better sound localization by virtue of being sensitive to ILDs, for example - there is little evidence that BiCIs employing commercial speech processing strategies are successful in restoring the advantages of ITD sensitivity, particularly the binaural unmasking that enables listeners to hear out speech in background noise. One reason for the large variability across bilateral CI subjects in spatial hearing performance is the potential for mismatches in the electrodes stimulated between the left and right CIs. Such mismatches might be the result of different surgical insertion depths, different implant lengths, or by differences in how the electrodes interface with the auditory nerve in each ear. In normal-hearing listeners, inputs from the two ears to binaural brainstem neurons are assumed to be well matched, and we expect this factor to be very important in BiCI listeners - matched pairs of electrodes should process the same acoustic frequency band, effectively compensating for any differences between the two implanted cochleae. Interaural electrode pairing (IEP) is likely to be increasingly important in future technological developments, such as in developing truly binaural CI coding strategies, which will preserve, enhance, and even optimize interaural cues such as ITDs. We compared three techniques to try to establish a ‘gold-standard’ objective measure for matching electrodes across the two ears. We hypothesized that the electrode pair that elicited the largest BIC amplitude would be the pair that also resulted in best subjective performance in discriminating ITDs (here, listeners determined whether the sound was heard to the left or heard to the right). Our second hypothesis was that the matched pair for evoking the largest BIC and the best ITD sensitivity might not be the best pitch-matched pair.
4.2.1 Matching electrodes across the ears for an individual BiCI listener
We assessed 7 different BiCI users to determine how well matched their devices are across the ears. In order to do this, we stimulated electrode 4 in the left ear, and systematically assessed the response to simultaneous stimulation of one of the electrodes in the right ear. The data for an individual listener are shown in Figure 4.5 and in Hu et al. (2015b).
4.2.2 Comparing best electrode pairs across all BiCI listeners
Table 4.1 lists the three IEP results for each subject (S). In general, the data indicate well-tuned pitch matching, ITD sensitivity, and BIC amplitudes as a function of electrode number. However, the pairing results were not consistent across methods. All but S2 was assessed using electrode 4 in the left ear. Generally, the comparison that produced the best pitch match was centered on electrode 4 in the right ear, whereas the comparison that produced the largest BIC or the best subjective sensitivity to ITDs, were not well matched to each other. The two binaural measures – the (objective) BIC and the perceptual (subjective) assessment of ITD sensitivity, however, were better matched. See Hu and Dietz (2015) for further details.
4.3 Conclusions
We demonstrated that an objective measure of binaural sensitivity – the IPM-FR - can be reliably obtained and used as an objective measure of binaural processing, at least in normal-hearing listeners. The EEG responses we recorded are consistent with evoked responses occurring in cortical and sub-cortical sources along the auditory pathway. Further, we have identified differences in the preferred IPDs for EEG signals conveying IPDs in the low-frequency TFS compared with the modulated envelopes of high-frequency tones, and these differences are consistent with the underlying physiological mechanisms of neurons in the brainstem pathways sub-serving these two cues. The IPM-FR, therefore, appears able to distinguish between these pathways based on the preferred IPDs employed to evoke the response, and provides a possible means of distinguishing which of the binaural pathways is activated in bilateral CI users, as well as a means of objective fitting of devices. Work is currently underway to assess the IPM-FR in bilateral CI users. In BiCI users, we observed a systematic discrepancy between the pair of electrodes that were best matched for pitch and those for which the best ITD sensitivity was found. We also observed a strong match between electrode pairs that were best matched for ITD sensitivity measured subjectively, and those pairs that generated the largest BIC response, suggesting that the subjective measure of binaural sensitivity provide a better measure of ‘true’ electrode matching than does pitch matching across the ears.
References
He, S., Brown,C. J., Abbas, P. J. (2010), Effects of stimulation level and electrode pairing on the binaural interaction component of the electrically evoked auditory brain stem response, Ear and Hearing, vol. 31, pp. 457-70.
Hu, H., Kollmeier, B., Dietz, M. (2015a). Reduction of stimulation coherent artifacts in electrically evoked auditory brainstem responses. Biomedical Signal Processing and Control, 21, 74-81.
Hu, H., Kollmeier, B., Dietz, M. (2015b). Suitability of the binaural interaction component for interaural electrode pairing of bilateral cochlear implants, ISH 2015 (in press).
Hu, H., Dietz, M. (2015). Comparison of interaural electrode pairing methods for bilateral cochlear implantation, Trends in Hearing (submitted).
McAlpine, D Haywood N, Undurraga J, Marquardt T (2015) Objective measures of neural processing of interaural time differences, Advances in Experimental Medicine and Technology (in press)
Ross, B., Tremblay, K. L., and Picton, T. W. (2007). “Physiological detection of interaural phase differences”. J. Acoust. Soc. Am. 121, p. 1017.
5.0 Developing and assessing algorithms for enhancing speech in background noise (WP5)
A critical outcome for the ABCIT project was to enhance the spatial-listening performance of CI users, in particular through the exploitation of binaural cues, especially ITDs which not only enable listeners to determine the location of the sound source, or sources, but also to hear out sounds in background noise. CI listeners are unable to perform binaural hearing in any way like normal-hearing listeners, and are particularly vulnerable to interference by background noise or competing talkers. In electrical (CI) hearing, therefore, two different stages of processing have to be considered when seeking to exploit binaural cues for listening in noisy conditions. The first stage is pre-processing – using sound-processing strategies to remove background noise from speech signals (a process sometimes referred to as ‘speech enhancement’). Computer algorithms that seek to achieve this are widely implemented in hearing aids so that the acoustic signal provided to hearing-impaired listeners is ‘cleaned up’, making it easier for hearing-aids users to comprehend speech in background noise. Implementing such algorithms before sounds are converted into electrical signals by CI devices has the potential to provide CI listeners with considerable benefit. The second stage is stimulation strategy – the shaping of patterns of electrical stimulation over time or over space (i.e. over different electrode contacts on the CI) to maximise the transmission of information by the auditory nerve. In this regard, the key question for the ABCIT project is ‘Is it possible to stimulate electrically each auditory nerve so as to maximise the benefits of having two CI devices, one in each ear?’. To this end, the main objective of WP5 was to improve speech and spatial perception in CI users by combining state-of-the-art pre-processing algorithms developed for use in hearing aids as well as novel algorithms, in combination with stimulation strategies employed in CI devices.
5.1 Testing algorithms for enhancing listening-in-noise
In general, assessment of existing and novel pre-processing algorithms by CI users indicated significant improvement in speech intelligibility. To date, these have not been implemented in combination with stimulation strategies due to the delay in the finalisation of the WP1 real-time research platform. Nevertheless, due to the keen interest in maximizing benefits to CI listeners receiving the new Neurelec/Oticon Medical CI devices, the best and most efficient of these algorithms will be made available for research and implementation. To this end, the consortium focused on estimating and enhancing spatial information from the noisy multi-microphone signals by making use of true binaural methods. The limitations of the monaural and bilateral methods motivated the consortium for working on binaural approaches. Besides, binaural approaches incorporated in other audio signal processing domains, e.g. hearing aids signal processing, yielded significant improvement in terms of instrumental (i.e. using objective, technical measures) as well as perceptual evaluation compared to other approaches. Finally, the unique binaural design of the NEU implants offered a natural motivation to the consortium to develop and evaluate this type of algorithms directly on NEU implants.
5.1.1 State-of-the-Art Pre-Processing Algorithms
We selected and evaluated state-of-the-art pre-processing algorithms specialised for hearing aids. The most promising candidates were implemented in the real-time signal-processing platform, i.e. the MHA, for further assessment. The list of algorithms ranges from differential microphones combined with coherence filters to spatial ‘beamformer’ methods together with single-channel noise reduction schemes. These algorithms were assessed in three different realistic environments, a multi-talker babble noise (20T), a cafeteria ambient noise (CAN) and a single competing talker (SCT). The perceptual evaluation performed with CI users showed a substantial improvement of speech intelligibility in terms of speech reception thresholds (SRT50) as shown in Figure 5.1 (left). In the multi-talker babble condition, the highest improvement yielded in SRT50 is around 6.9 dB ± 1.2 dB, a substantial improvement Similarly, in the realistic cafeteria ambient noise scenario, SRT50 improvements of 5.3 dB ± 2.0 dB were achieved. Finally, in the spatially separated single competing talker scenario, maximum SRT50 improvements of 15.2 dB ± 3.6 dB were achieved. These results are very promising in terms of the improvement of the speech reception threshold. However, the consortium investigated whether CI users prefer one algorithm over the others, in particular over the unprocessed signal (Figure 5.1 right).
In general, the CI users who participated in the evaluation studies preferred all algorithms compared to the unprocessed signal. We performed extensive evaluation experiments in the three different realistic listening conditions. Only in the third noise scenario (SCT), two of the algorithms were not preferred. Furthermore, novel pre-processing algorithms were developed in particular for enhancing the spatial listening abilities of CI users. These algorithms incorporating and combining approaches from different research domains depicted significant improvements compared to state-of-the-art approaches and showed large potential for benefit in the next generation CIs.
5.1.2 Steering Beamformer
In one study, we worked on developing a ‘steering beamformer’ approach to enhance a moving speech source in a diffuse noise environment. The proposed system estimates the direction of arrival (DOA) of the target speech source (Kayser and Anemüller, 2014) and of a spatial filter (Bitzer and Simmer, 2001). We evaluated the proposed algorithm in three different angular velocities for the target speech source, 15°/s, 30°/s and 60°/s. Experiments performed on normal-hearing listeners and CI users proved that the proposed steering beamformer is superior to its state-of-the-art competitors in tracking and enhancing moving speech sources. In these experiments, we measured the speech reception thresholds (SRT50) for the proposed algorithm as well as for a fixed beamformer setup (0° beamformer), where the target speech source is assumed to be in front, for a steering beamformer with the true target direction (perfect DOA beamformer) and for the unprocessed signal. The results are shown in Figure 5.2. The proposed steering beamformer approach achieved a 3.1dB SRT50 improvement in the 15°/s case compared to the unprocessed signal, whereas the fixed beamformer degraded the performance by 1.5dB. The proposed method performed similarly to the steering beamformer with the true DOA.
5.1.3 Extent of Lateralisation
We also performed experiments on spatial listening by investigating the extent of lateralization (how far to the left or right a sound is heard) of click trains with large ITDs on normal-hearing listeners. In these measurements, the consortium focussed on the extent of lateralization of large ITDs (> 600µs – close to the maximum range of ITDs generated by the human head). Stimuli consisted of unfiltered and band-pass filtered trains of clicks at different inter- click intervals (ICI), which acoustically mimic the effect of electrical (CI) pulses. An acoustic pointer procedure (Bernstein and Trahiotis, 1985) was employed to determine the extent of lateralisation of click trains. Results show that an increase in lateralisation percept continues beyond the physiological limit (~600µs); see Figure 5.3. These results are an indicator of bilateral CI percepts.
5.1.3 Estimating the Inter-channel Phase Differences (IcPD)
We also developed novel methods incorporating statistical approaches in audio-signal enhancement. In one of these studies, we computed so-called ‘short-term cross-correlation functions’ between the channels of a given multi-channel multi-source audio input signal and extracted the non-linear inter-channel phase differences (IcPDs) as well as a measure of activation for each source. The source separation performance of the proposed method was compared with the unprocessed signal as well as with the linear IPD estimation (TDOA) and with the true IPDs. The results are shown in Table 5.1. The linear IPDs are estimated based on the main direction of arrival estimation evaluating the peaks in the cross-correlation matrix. Both linear and non-linear IPD estimation methods lead to a significant improvement compared to the unprocessed signal. Furthermore, the proposed method clearly outperforms the state-of-the-art linear method.
5.1.4 Sparsity-based Noise Reduction
In another study, the consortium extended a non-negative matrix factorization (NMF) based noise reduction algorithm by automatically detecting and reducing noise activations within a given noisy audio signal. In the NMF approach, Hu et. al. (2012) decomposed the envelope matrix of CI frequency channels into the two matrices, which we can name as basic envelope patterns and their activations. Subsequently, Hu et. al. eliminated the noise activations manually. The consortium incorporated a sparse method for automatically detecting and reducing the noise activations within the given noisy audio signal. For this, the consortium incorporated a method for sparsely coding only the activation of the speech segments using a dictionary (code-book) and thereby reducing the noise. The first technical tests were performed using normalised covariance metric (NCM). Table 5.2 shows the results of these preliminary experiments using this metric. Here, higher values imply better speech intelligibility. The consortium performed sparse coding experiments using different number of dictionary atoms extracted. Increasing the number of atoms used for coding increased the speech intelligibility in terms of the used metric.
The preliminary test of the sparsity-based noise reduction algorithm in combination with the NMF algorithm depicted an improvement in the speech intelligibility compared to a noise reduction method based only on the NMF method in terms of the normalised covariance metric.
5.2 Conclusions
The data indicate that CI users can benefit from binaural pre-processing algorithms in a range of different listening environments. As these algorithms consider the left and right ear together, the spatial listening is substantially improved using binaural setups compared to simply ‘bilateral’ algorithms, where left and right ear are handled independently. Incorporating these types of algorithms in next-generation CIs will have a significant impact on users’s abilities to communicate in challenging environments e.g. environments with background noise, reverberation and/or multiple competing talkers. By combining these pre-processing strategies with better stimulation strategies, the promise of improved speech intelligibility may be realized.
References
Bitzer J, Simmer K (2001). Superdirective Microphone Arrays. In Microphone Arrays, Digital Signal Processing, 19-38, Springer, Berlin, Heidelberg.
Bernstein LR, Trahiotis C (1985). Lateralization of Low-Frequency, Complex Waveforms: The Use of Envelope-Based Temporal Disparities. J. Acoust. Soc. Am., 77(5), 1868-1880.
Hu H, Mohammadiha N, Taghia J, Leijon A, Lutman ME, Wang S (2012). Sparsity Level in a Non-Negative Matrix Factorization based Speech Strategy in Cochlear Implants. Proceedings 20th European Signal Processing Conference (EUSIPCO), Bucharest, Rumania.
Kayser H, Anemüller J (2014). A Discriminative Learning Approach to Probabilistic Acoustic Source Localization. Proceedings 14th International Workshop on Acoustic Signal Enhancement, Antibes, France.
6.0 Binaural Device Prototype (WP6)
WP6 was designed to combine and integrate the information gleaned from WP1-5 into a collection of hardware demonstrators that, together, form a novel binaural CI. We tailored the hardware and software solutions specifically to take advantage of binaural cues and the influences from the previous work-packages. A sketch of the major influences is shown in Figure 6.1. A new binaural codec was developed to support these contributions and hardware was designed and built around this codec to test individual parts of the system. Ultimately, a feasible binaural CI solution is presented from these pieces that produces an expected performance gain of 3-10 dB SNR when compared with unsynchronized bilateral implants (see WP5) with a lower cost than is currently available from traditional bilateral solutions. The benefits of an advanced CI come from three sources: (1) the availability of pre-processing algorithms such as binaural beamforming (WP5) that can access information from 4 synchronized microphones located on both sides of the head, (2) novel stimulation strategies that can transmit ITD as well as ILD information, (3) the ability to measure objectively eABR and cortical responses which could impact fitting. To realize all these benefits, the problem of synchronization must be overcome, an appropriate binaural electrode array must be created, and a scheme for transmitting the data must be devised. All of these issues were addressed in this prototype.
6.1 Developing Elements of the Prototype
6.1.1 Chosen Binaural implant Architecture
Many CI companies are adding binaural capabilities to their monaural offerings, however the ABCIT binaural CI is designed to be binaural from the ground up. The architecture leverages the current Neurelec binaural design. In this design, instead of using separate CI processors, a single binaural processor is used. The ABCIT design builds on this architecture. Here, a single stimulation engine running from a single clock serves both ears which means only 1 synchronization point is needed—the initial wireless ear-to-ear audio synchronization. This contrasts with all other designs (using separate processors) that require synchronization both at the input wireless data stream and again at the output before stimulation. Traditional designs, therefore, do not generally have the capacity to transmit binaural stimulation with sufficient ITD accuracy (~10 µs). In addition, with this architecture only a single transcutaneous communication channel is needed thereby reducing power.
6.1.2 Wireless Synchronized Ear-to-Ear Audio Streaming
A wireless ear-to-ear synchronized digital audio streaming radio was developed for (1) high synchronization accuracy, (2) low power, (3) low latency, and (4) and high reliability. This technology called Nearlink is now available. Nearlink transmits accurately without gaps at a sampling rate of Fs = 16.67 kHz to 20 kHz using a standard audio codec (722.2). The system bandwidth was measured at 8 kHz, with a latency of 10.4ms.
6.1.3 Binaural Processor
A binaural processor was developed using a combination of hardware and PC software components (MHA). This processor received 4 synchronized microphone signals and produced a special binaural codec output for transmission to the implant. Binaural pre-processing algorithms discovered from previous work (WP5) were implemented on the processor. The whole software processing chain is shown in Figure 6.3..
A proprietary binaural codec was developed that treats the data stream as a binaural to begin with. These streamed packets encode relative ITDs (and ILDs) within the data packet itself, leading to an effective sub 10 µs ITD accuracy and resolution and allowing 100% consistency between ITD/ILD cues. Because the received packet is decoded by a single, binaural CI processor (running a single clock, e.g. 1 MHz), these data can be converted to stimulation timings with absolute precision. A second advance, involved transmitting TFS information asynchronously and without reference to any master clock was also introduced. The required ear-to-ear synchronization was achieved via the synchronized wireless microphone system and audio codec (mentioned above).
In order for the system to achieve a ‘true binaural’ processing status, the processing had to combine information from the two ears in super-additive way. What do we mean by this? In general, normal hearing listeners show performance advantages in difficult listening conditions of more than a factor of 2 ( > 3dB improvement in SRT) when listening through two ear rather than one. This additional advantage (beyond the expected additive advantage) is a ‘true binaural’ advantage. We set out to develop a system that would have the best chance of producing this effect for CI users who to date never show such a true binaural advantage. Although claims are sometimes made that super-additive results—called binaural squelch--have been observed, these results can be explained using normally additive ILD processing strategies (Backus, 2015)).
The binaural processor focused on two strategies to achieve ‘true-binaural’ super-additive processing:
(1) A binaural beam-former was developed for the BTE processor in order to take advantage of the synchronized bilaterally placed microphones and thereby improve input SNR.
(2) ITD cues (artificial or real) were integrated with ILD cues and a packaged in a binaural codec designed to transmit this information accurately to the CI user.
The two strategies differ in an important way. In the first, the binaural processing is done in the BTE processor (to increase the SNR of the output stimulation) while in the second, the binaural cues are passed to the brain and the binaural processing is done in the brain (to increase the brain’s internal SNR).
For the first strategy, a fixed MVDR binaural beam-former with common post filter was chosen. This choice was made based on results from WP5. A schematic of this beam-former architecture is shown in Figure 6.4. The performance of this solution was measured at ~5 dB SRT improvement. This algorithm was chosen over better performing adaptive ones due to user preference. A challenge remains to understand how to make an adaptive algorithm that adapts in a pleasing way. For example, in a car situation where the user wishes to listen to a passenger rather than the noise from the window but still be able to hear incoming road alerts.
For the second strategy ITD cues were preserved and integrated with ILD cues and transmitted to the implant for stimulation. The ITD cues chosen to transmit were based on basic knowledge of ITD sensitivity in CI users found in research. Primarily the following aspects:
(1) It is known that ITD sensitivity exists for CI users along the entire cochlea (or electrode array) and that the resolution of this cue varies with stimulation rate.
(2) At low stimulation rates the ITD cue is more salient and detection threshold for the best CI users is 25–50 µs corresponding roughly to 5 degrees in the acoustic fovea and 10 degrees laterally.
(3) The differentiable ITD range for CI users is extended beyond the natural range (750 µs) to double that (1.5 s).
(4) ITDs appear to not be additive across electrodes for sequential stimulation.
According to these ITD observations, the binaural processor was developed to allow variable rate stimulation (which could transmit temporal fine structure as other devices already do) and which would satisfy the low rates required for high ITD sensitivity (1 above). It was assumed that channels with naturally low rates (according to TFS extraction) will capture the ITD perception, but all channels were designed to deliver ITDs such that different ITDs could be presented at different frequency electrode pairs. In addition, the ITD range was extended to 1.5 s (beyond the normal physiological range) to take advantage of CI users observed extended range. Figure 6.5 shows the locations encoded by the binaural codec. A lookup table was implemented in the implant to map these positions to physical ITDs (µs) used during stimulation. This strategy was not tested on CI users, however, because no currently implanted CIs could support it. We don’t yet know if it will produce the ‘true binaural’ performance gains we seek or if it will be preferred.
6.1.4 Transcutaneous Data Transfer
The ABCIT binaural codec (WP3) was created to efficiently encode both ILD and ITD cues using a single data channel. The digital packet format contains enough accuracy and electrode data to encode most proposed binaural coding strategies. In this sense the codec is agnostic to the type of binaural processing chosen for the binaural processor. Because the chosen binaural CI architecture has only 1 data transmission channel which must serve both ears, the minimum speed and bandwidth requirements of this channel are 2x higher than for monaural devices sending the same type of information (320 kbps vs. 160 kbps). The transmission must have low error rates, be robust, and minimize the power consumption particularly for the implant’s receiver. For robustness an on-off keying (OOK) method was used, such that only a single level change needed to be detected. A digital communication channel with CRC checking was built around the OOK physical layer.
For the OOK digital transmission, the key specifications of (1) robustness/reliability, (2) energy/bit, and (3) data rate can all be calculated analytically and compared for different modulation schemes providing they are applied to the same channel with the same physical channel properties. These properties are fixed by the physical method of transfer (generally RF) and by the thickness of a given users skin. Theoretical calculations comparing various modulation types were done. The standard RZI (return to zero inverted), 4-PPM (4 position pulse modulation), and a novel EPM (edge position modulation) were compared. All modulation techniques demonstrated sufficient bit-rate to meet the 320 kbps required (based on our physical channel’s properties). The EPM modulation provided the best bandwidth efficiency and transmit-power efficiency, but required a higher frequency, a more accurate sample clock, and added additional decoding complexity (state machine with 1-internal state-vs.-straight decode). However, because EPM is based on edge detection, elongated pulses did not create errors as they do for 4-PPM as long as the elongations do no run into the next edge.
Because the transmit energy required may change (depending on the skin thickness variations) an automatic adaptive algorithm that receives return data (e.g. no response + CRC failure on reception) and then updates the modulation parameters by changing the transmit VOH voltage (or pulse duration) is possible. This technology is already in use. In this design (Figure 6.6) we separated the fast 320 kbps channel (Channel A) from an intermittent return channel (Channel B) because these channels had very different speed requirements. A variable transmit voltage block was used (similar to the one currently used in existing designs) to adapt the transmit levels.
A complete system was built and was able to transmit bursts of data at 1 Mbps using an average of 2mA for transmit and 1mA for receive. This was higher than expected and further optimizations may be needed to reduce this (see Figure 6.7).
6.1.5 Stimulation
A new binaural implant stimulation engine was developed using an FPGA to drive two monaural current stimulation chips. The chips each have 20 configurable current sources and are very flexible so can produce many different types of stimulation. We specified the binaural codec to address 15 binaural channels (15 electrodes on each side); for comparison, the current binaural implant addresses 12. Several common pulse types were implemented: (1) monopolar common-ground monophasic with passive charge recovery, (2) bi-polar (e.g. e+1) biphasic rectangular pulses with inverted charge recovery, (3) bipolar biphasic rectangular pulses with elongated charge recovery. In addition to these common pulse strategies, a novel ramped pulse was introduced based on work done from WP3. Ramped pulse have the potential to reduce the spread of excitation and thereby improve frequency resolution. However these ramps are very new stimuli and it is not yet known what the best ramp method might be—a topic of ongoing animal experiments. We chose to implement the conceptually ‘simplest’ approach outlined in Figure 6.9. This was planned using a lookup table within the implant so that this ‘simplest’ approach could easily be extended to more complex ramp shapes in the future.
6.1.6 Objective Measures
Objective measures can help in ‘pitch’ matching electrodes across two ears to improve binaural CI performance. Brainstem responses such as the Binaural Interaction component (BIC) investigated by UOL or the novel Interaural Phase Modulation Following Response (IPM-FRs) developed at UCL (see WP4) each shows it may be a better way to match electrodes across two ears than the current state of the art, subjective pitch matching (asking patients which electrodes sound similar in pitch). These methods require the ability to gather eABRs. In addition eABR have been shown to be as effective as eCAPs for determining C-levels.
To take advantage of these techniques, a prototype eABR amplifier circuit and averager was created. The amplifier was designed for low signal and low noise, with 4-kHz bandwidth, measuring electrically evoked auditory brainstem responses (eABRs) and cortical responses from within the implant. The system includes low power amplifier, a DSP-based artefact rejection and averaging engine, and built-in impedance measurement. To evaluate the system’s performance it was built and tested using external surface electrodes on normal adults. An investigation of the test amplifier’s internal noise (measured as en=17.4nV/√Hz, in=0.2pA/√Hz) was carried out as well as an investigation on real-world noise contributions from biological (Figure 6.10) power-line, and RF sources. Algorithms built to monitor and reduce as far as possible these noises were refined.
The whole system was able to capture standard eABR (shown below) and advanced cortical measures. Impedances were automatically measured and artefacts were rejected using a combination of reversing polarity, thresholding to remove clipping, and alternated interleaved trace comparisons.
6.2 Conclusions
The original goal of the ABCIT programme was to develop a prototype binaural device that might one day be implemented as a device used in patients. However, over the course of the programme of work, each of the elements of the presumed prototype have more rapidly found their way to utility in isolation or in combination, meaning that the whole ABCIT project has also been assessing aspects of a binaural device. WP6 has provided the means of bringing many of these elements together, for both research and future clinical application.
Potential Impact:
3.0 The potential impact (including the socio-economic impact and the wider societal implications of the project so far) and the main dissemination activities and exploitation of results
Considerable scientific and societal impact, including economic impact, is envisaged from the ABCIT project. Each of WP1-6 has specific benefits, and these are outlined below. In addition, the description of the overall benefit (including to European competitiveness through commercial success) were delivered as part of WP7, and are provided in some detail below.
3.1 Impact of WP1-WP6
The ABCIT WP1 research platform – which has been given the name ‘cHIRP’ and placed on the product roadmap of Neurelec - ultimately allows the wider distribution of this new tool to qualified CI researchers. This plan expands its use beyond the original consortium and was not originally planned. With its hardware and software parts cHIRP provides researchers pursuing research on cochlear implants the complete processing chain of a cochlear implant with novel capabilities. Importantly it enables real-time, synchronized, 4-channel (4-microphones) binaural CI systems to be evaluated using software (MHA). As any algorithm with real time capabilities can be implemented as a new plug-in into the Master Hearing Aid, it allows an arbitrary combination of different speech enhancement algorithms to be written in software only without the need of hardware effort. The electric stimulation patterns are also generated in the Master Hearing Aid and transferred to the research board, which performs the stimulation. This architecture enables researchers to reconfigure the chain easily according to their experimental needs. It fills an important gap in cochlear implant research by combining hardware and software within the same platform. This represents a significant advance for the CI research field and should propel investigations into binaural CI for years to come. Questions that have remained unanswered can be answered with this platform (e.g. ITD and ILD trading for CI users in real environments, probing strategies for auditory scene analysis and the effect of binaural stimulation on listening effort etc.).
WP4 assessed a critical factor in the future development of bilateral cochlear implants and their efficacy, objective measures and developed new software and hardware tools for recording brain activity to bilateral CI. In particular the first multichannel brainstem EEG for CI listeners was developed. This exceeded the initial expectations by allowing to record brainstem EEG, even from subjects for whom a facial nerve artifact was present – previously these patients could not be assessed in a binaural setup. A custom-made EEG cap with holes at the CI-coil positions was developed and presented to scientific audiences as well as to the general public in a regional newspaper. The artifact removal techniques including the cap were published in a peer-reviewed journal. Subsequently the brainstem EEG measurements were employed to objectively determine left-right electrode pairs for binaural fitting. Results indicate the shortcomings of existing subjective techniques and have the potential to revolutionize binaural CI fitting, in particular for the growing number of paediatric patients. With a related basic research study on objective binaural measures performed on normal-hearing subjects, WP4 was highly visible at the international Objective Measures Conference (Toronto 2014), the International Symposium of Hearing (Groningen, 2015), and the Conference on Implantable Auditory Prostheses (Tahoe City, California, 2015) with two contributions each.
The outcomes of WP5 will have real impact on bringing the benefits hearing-aid devices have enjoyed for some time into the CI domain. This was achieved by adapting state-of-the-art binaural pre-processing algorithms, as well as by developing novel ones, for CI devices. The evaluations of these algorithms for speech enhancement have shown that the tested algorithms improve speech intelligibility of CI users significantly, especially by means of spatial filtering (i.e. binaural beamforming algorithms). These evaluations were performed in several reverberant environments for a more reliable assessment of the algorithms in realistic listening situations. The algorithms were implemented as real-time plug-ins on the Master Hearing Aid (MHA), i.e. on the software part of the ABCIT WP1 research platform. The wider distribution of the ABCIT WP1 research platform offers many advantages to the qualified researchers all over the world. The real-time capability enables the researchers to evaluate the algorithms in daily life environments as well as in laboratory conditions using different types of stimuli e.g. online speech, recorded speech etc. Furthermore, using the research board, the algorithms can be tested in combination with different stimulation strategies. Because the MHA can also be hosted by other hardware platforms such as standard PCs, ARM-processor-based computers as well as iOS smartphones, the algorithms can be widely and easily applied in real-world environments.
The impact of WP6 has both direct and indirect impacts. Although the ABCIT WP6 binaural prototype was originally envisioned as a hardware precursor to an eventual binaural CI, the WP6 platform has now been expanded to include a new animal research platform that will be made available to qualified animal researchers, enabling testing of some of the ideas for improvements in binaural hearing included here, and others to come. These include ideas around incorporating eABR or other objective measures (WP4) for automated fitting, novel stimulation strategies including pulse shapes (ramps), TFS, ITD, and others can now be studied in animals. This should also speed the adoption of the most promising results.
Indirectly, the WP6 feasibility study has been successful in proving the capacity of the technology, but further research is required to quantify the benefits (both in humans using the WP1 ‘cHIRP’ platform and in animals using the WP6 platform) before an eventual binaural implant product based on this will be developed. What ABCIT has delivered to date indicates that this device could provide >6 dB of improvement in speech-reception thresholds (by using novel pre-processing algorithms such as binaural beamforming) and would cost about 40% less than purchasing separate processors to create a binaural system with the same type of performance.
3.2 Exploiting the outcomes – WP7
As part of WP7, the ABCIT project developed a ‘Market Exploitation Plan’ to aid in the commercialization of marketable findings and discoveries. In its original conception, it was thought that the principle product for leveraging findings of the research would be the Neurelec Binaural device. However, since the acquisition of Neurelec by Oticon Medical, access to new wireless technologies enabled by custom chipsets developed by Oticon, has provided an alternative path for bilateral device communication and synchronization that may be more commercially viable than a single implant powering two ears (i.e. binaural device). In terms of preservation of bilateral timing cues, a chipset driven by a crystal will allow synchronization between left and right devices with high timing accuracy. It is possible that providing such timing information to infants may lead to improved development of neural pathways capable of extracting salient timing information between ears, and this may improve outcomes when using two cochlear implants. Outcomes in post-lingually deafened adults where such neural pathways may still be usable may also improve.
At present more information is needed to determine if a binaural device should be pursued in parallel to a synchronized bilateral solution. Specifically, technology for high density feed-throughs (2x20 electrodes) and a single device with two stimulation engine chips demonstrated in the ABCIT project needs to be developed and matured further before feasibility of such a design can be fully assessed. Historically, increasing feed-through density tends to reduce device reliability. Even though concepts exist demonstrating the feasibility of this binaural approach, substantial R&D is still needed to make the binaural solution commercially more attractive compared to an Aurora synchronized bilateral solution. Specifically, feedback from KOL meetings from nearly 70 surgeons from around the world since autumn 2013 has highlighted the following technological challenges of the binaural option: a) reliability must be at least as good as with a unilateral device because, at a minimum, in the event of a device failure, two ears are put at risk rather than one, b) the device is not feasible in small children due to substantial growth of the head during development and 3) reimbursement for bilateral implantation is more and more becoming the norm, especially for children. These three observations effectively mean that the business rationale of delivering binaural devices as a reduced cost alternative to bilateral implantation is not evident in all geographic regions.
In terms of product development, therefore, Oticon Medical will leverage the hardware platform for extended research on aspects of synchronized stimulation on bilateral patients. Thus, as a first commercial path we will pursue the direction of a synchronized bilateral stimulation leveraging bilateral BTEs using a crystal-clocked chipset. It should be noted that the chipset has already been developed and the principle work needed is to adapt the electronics to drive a new BTE that supports a new internal CI with improved timing capabilities. The new internal CI is called the Neuro ZTi and Oticon Medical received CE mark for this new device in June 2015. It should also be noted that having patients implanted with the Neuro Zti is critical for being able to fully test the potential benefits of the findings of the ABCIT project. First implantations of the Neuro Zti are expected to begin the fourth quarter of 2015.
Market Exploitation Activities
The Neuro System will be seen as the first fruit of the co-operation resulting from the acquisition of Neurelec by Oticon Medical in 2013, and will include several specific aspects derived from the ABCIT project. As the first Oticon Medical CI system, this launch comes with several innovations:
1. Launch of the Neuro name as Oticon Medical’s cochlear implant system brand.
2. Launch of Neuro Zti – the industry’s most recent implant technology platform with a new compact and safe design
3. Launch of Neuro One – the first sound processor with full integration of Oticon HA and CI technologies. Taking the success of the XDP sound processing and the Inium platform technology one step further though automation of environment driven mapping of acoustic to electrical dynamic range.
4. Major upgrade of the DigiMap fitting software. Digimap 4.0 includes improved fitting flow, tested pre-sets, ECAP possibilities and full 64-bit support.
5. Perhaps most relevant to the present discussion, the Neuro Zti internal platform delivers a capable substrate to provide the precision neuro-stimulation necessary to test and eventually implement research findings regarding bilateral stimulation resulting from the ABCIT project.
The whole communication, service and support around the Neuro System will be built to reinforce our mission to become the world leader in bilateral, bimodal and binaural hearing solutions. The new Neuro Zti implant is innovative and brings a new standard to the market. The changes are very visible (new high impact resistant material, compact, thin) and the advantages tangible (new stimulation chip, programmable stimulation capabilities, eCAP measurements, MRI peace of mind, minimal invasive surgery, reliability). We have a very competitive implant, which will make us stand out. The new Neuro One sound processor will be an update of the design of the current Saphyr Neo collection. We will build on the success of the XDP sound processing; bring the first integrated HA and CI sound treatment that will be providing real advantages to the CI users in terms of speech focus, automatic adaptive listening and on many points WITH regard to feeling at ease with the processor. XDP processing and its automated version, Voice Guard, provide several advantages to bilateral CI recipients over traditional processing. Traditional designs use broadband AGCs which have the effect of nearly eliminating loudness differences between the two ears as well as introducing temporal distortion that could negatively impact interpretation of inter-aural timing differences. In contrast, XDP and Voice guard preserve these interaural differences without introducing temporal artifacts or cross-modulation frequency distortion. We believe that this processing will quickly put us in the forefront of bilateral hearing.
Going forward, we will advance two-eared hearing with the aim of becoming the world leader in binaural/bilateral solutions in CI by undertaking the following:
1) Continue to improve bilateral hearing outcomes with refinements to XDP and Voice Guard supported by findings from the ABCIT project
2) Incorporate the already developed Chip-set that has both low-power wireless and high-temporal precision capabilities into our next generation BTE to allow new possibilities in communication and synchronization across not only two cochlear implants but also for the increasing population of bi-modal users to enable optimization of hearing between a CI and a contralateral hearing aid
3) Continue to advance our research platforms to empower researchers to better enable us to exploit our processing power to improve patient outcomes
4) Continue to expand internal KOL meetings with the aim of facilitating studies with key clinics to validate and improve our product developments including those derived from the ABCIT project
5) Continue feasibility studies regarding a future binaural device leveraging an Aurora chip-set empowered external processor
In summary, Oticon Medical has now undertaken two parallel paths designed to produce products to exploit viable findings of the ABCIT project, these being 1) two properly synchronized bilateral devices and 2) a binaural solution with a single internal processor driving both ears. Given that advanced capability of a new crystal clocked chipset as an off-the-shelf solution when compared to the technical risks associated with the binaural solution, we have decided to focus the development priority on the former. Critical to success of the project, a more capable internal device is required. In June 2015 Oticon Medical received CE mark for the Neuro Zti, this is the new internal device that will provide the platform flexibility necessary to exploit the new crystal-clocked chipset. Future efforts will involve in-patient research with the Neuro Zti as well as the development of improved research tools critical for understanding how to optimize the new chip-set driven external processors to deliver bilateral information.
List of Websites:
http://www.ucl.ac.uk/abcit
1.0 Executive Summary
In 2012, a consortium of researchers and commercial partners in the UK, France and Germany secured €4M from the European Union’s ‘Framework 7’ research fund to develop a research programme aimed at progressing bilateral cochlear implantation towards true binaural performance. Binaural (two-eared) hearing is critical for localizing the source of a sound, and for hearing out conversations in background noise (cocktail party listening). Compared to normal-hearing individuals, users of cochlear implants (CIs) are at a disadvantage when facing binaural tasks. Bilateral cochlear implantees (most children born deaf and an increasing number of adult recipients) are fitted with essentially two independent devices, each without regard to the sound processing or electrical stimulation of the other ear. The aim of this programme, Advancing Binaural Cochlear Implant Technology (ABCIT), therefore, was to develop a framework in which binaural hearing was central to the function of CI technology. The programme of work was divided into six work packages (WPs), each with a named lead partner. Critical aspects of the programme were to ensure the successful delivery of the milestones and deliverables within each WP, and to ensure compatibility across WPs. In order to provide CI researchers with the tools necessary to assess binaural function, a portable, real-time research platform was developed, capable of processing signals from 4 microphones simultaneously. A crucial aspect of this research platform lies in its use of the Master Hearing Aid (MHA) technology of Hörtech, the hearing technologies centre allied to Oldenburg Univerity. The new platform is well suited to assessing sensitivity to binaural spatial cues, including interaural time differences (ITDs), in real-time, and supports instantaneously variable stimulation rates. Software processing can be changed on the fly. A simulation tool was developed enabling researchers to explore the effects of electrical stimulation on the temporal pattern of nerve impulses generated by CIs and, from this, to develop computer models of neural sensitivity to binaural electrical stimulation. New electrical stimulation strategies were developed that seek to reduce the spread of electrical activity in the implanted cochlea and to convey the temporal information necessary for preserving ITD cues in binaural hearing. These stimulation strategies were assessed physiologically in vitro and in vivo in experimental animals. A key aspect of ensuring binaural information is maximised in CI listeners lies in developing objective measures of brain function. The ABCIT project therefore developed new, and assessed existing, measures of binaural brain function in normal-hearing and CI listeners, with the aim of providing a diagnostically feasible repertoire of neurophysiological measurements to aid the fitting of bilateral CIs. Another critical development was the merging of hearing-aid and CI technologies, particularly through the development and exploitation of sound-processing algorithms that enhance speech understanding by removing background noise from acoustic signals before these signals are converted to electrical pulses in CI devices. A final WP integrated these ideas into a workable binaural CI ‘development kit’ expressed in hardware and software. The acquisition of Neurelec by William Demant Holdings (WDH) in April 2013 and its ensuing incorporation into Oticon Medical should help to bring binaural CI technology to market and has had an energizing effect on the project. Overall, ABCIT was highly successful, bringing together researchers and commercial partners into true partnership, and providing the means of developing CI technology beyond the state-of-the-art. As well as many publications and presentations, ABCIT generated four joint patents and a range of associated projects and collaborations that will generate new advances in discovery science and implementation of these advances into technologies, diagnostics and interventions aimed at transforming the lives of the profoundly deaf.
Project Context and Objectives:
2.0 Project Context and Objectives
2.1 Context of the ABCIT Project
Hearing loss is the most common sensory deficit; one in seven of the global population has a hearing problem and with this population increasingly ageing, the issue of hearing loss and its financial and health cost to society is increasingly evident. Deaf and hard-of-hearing people are often socially isolated, preferring not to mix in social gatherings where background noise makes communication impossible. To this end, the advent of cochlear implantation, or electrical hearing, has revolutionized the lives of the severe and profoundly deaf. Cochlear implants (CI) restore sound perception by directly stimulating residual auditory nerve fibres with patterns of electrical pulses delivered via an electrode array implanted into the cochlea – the hearing organ of the inner ear. Employing up to 22 electrode contacts, CIs attempt to replace the function of all, or nearly all, of the 3000 or so sensory cells of the inner ear that are damaged or absent in individuals with severe and profound hearing loss.
Despite its undoubted success, much remains to be achieved if CI listeners are to perform adequately in even moderately-challenging listening conditions, especially so as CI is largely a ‘monaural’ technology at present, with CI listeners unable to exploit the known benefits of binaural (‘two-eared’) hearing. Binaural hearing refers to the brain’s ability to compare information about sounds arriving at the two ears, is essential for locating the source of a sound and a vital component of ‘cocktail party’ listening (understanding speech in noisy environments); monaural listeners are at a distinct disadvantage under such conditions, being unable effectively to segregate speech patterns originating from different talkers. Currently, even where CIs are provided to both ears (bilateral implantation) they remain independent devices, incapable of exploiting the important differences in sounds at the two ears that render cocktail party listening possible Neurelec is the only manufacturer of CIs that deliver synchronized signals to the two ears, using one processor serving both cochleae. This provides the opportunity of generating synchronised signals to the two ears (truly binaural) compared with using two unsynchronized monaural devices (bilateral). Nevertheless, exploiting the great potential of such technology has not yet been fully realized, including the possibility of exploiting specific binaural processing strategies that make use of the brain’s remarkable ability to compare small timing differences in sounds at the two ears to hear out speech in noisy conditions. The CI prototype to be developed will for the first time provide true binaural stimulation.
2.2 Aims and Objectives
The overall aim of ABCIT, therefore, was to enhance the lives of the profoundly deaf through the development of a binaural cochlear implant (CI) that exploits the spatial information available in the sound input to improve the listening experience of CI patients. Exploitation of spatial information has the potential to enable CI patients to perceive the direction of sound sources and enhance their ‘signal-to-noise ratio’ with the result that their speech perception in adverse listening conditions will be greatly enhanced. The original SME partner within the consortium (Neurelec) offered, with its current CI technology, a unique therapeutic intervention – the closest of all available CI designs to being capable of true binaural processing. Initially aimed at being cost effective, Neurelec’s CI device processes the left and right microphone signals on a single processor, allowing the interaural (‘two-eared”) cues essential for spatial hearing to be extracted and manipulated through the delivery of interaurally-synchronized electrical pulses to both cochleae. The design of this implant, with its galvanic connection across both sides of the brain, also offers the unique opportunity to record neurally evoked responses from the auditory brainstem (‘EABRs’) with the CI device itself, without the requirement of additional electrodes attached to the skin. In contrast to responses generated by the primary auditory nerve (‘ECAPs’), EABRs are informative as to the activity of the binaural brainstem nuclei and can therefore be employed to ensure objective measures are used in establishing and enhancing binaural fitting procedures.
At its outset, ABCIT had 5 main objectives that sought to develop bilateral cochlear implantation beyond the state-of-the-art:-
1. to develop the ability to exploit the full range of binaural hearing cues in cochlear implantation (CI), and to gear stimulation strategies towards enhancing binaural information
2. to develop a research platform, including a speech pre-processing module, to enhance the development of CI processors
3. to adapt currently successful hearing-aid (pre-processing) algorithms to meet the special demands of cochlear implants
4. to develop the means of measuring from the auditory brain neural signals that will provide an objective means of assessing binaural performance in CI users. This goes far beyond the capabilities of all current CI systems, and is based on Neurelec’s unique implant design
5. to develop a low-power, wireless audio link between the devices at both ears so as to enhance users’ access to binaural information
Originally, a 6th objective was to develop a pre-commercial prototype of a ‘second generation’ binaural CI that would provide the first true binaural hearing for CI users. However, during the course of the project it became clear that it was too early to harden all the newly discovered binaural CI knowledge into such a pre-commercial prototype. Instead it was decided to build a flexible system that could demonstrate the feasibility of the new ideas and could also provide a platform for further binaural CI research and development—a binaural CI ‘development kit’.
The WPs were therefore geared towards meeting these objectives. WP1 provided the framework for developing a new hardware device and software to permit research in experimental animals and human subjects to be undertaken by CI researchers. In two-stage process, WP2 developed a simulation tool, one stage involving development of a cochlear model of electrical stimulation and spread of excitation, and the other stage models of binaural processing based on neural firing patterns. Model outputs included predictions of binaural performance in different acoustic scenes. WP3 developed new stimulation strategies for CI, including strategies that take more-informed account of the biophysical properties of cochlear nerve fibres, and the potential for exploiting these properties to limit spread of excitation within the implanted cochlea. It also assessed a range of new stimulation strategies for enhancing binaural information in CIs, including through experimental recordings in small mammals and through behavioural assessment in normal-hearing listeners and CI users. The purpose of WP4 was to develop objective measures of binaural brain function to improve fitting for monaural and binaural performance. By assessing neural activity using electro-encephalography (EEG) in normal-hearing listeners and CI users, potential diagnostic tools can be compared to subjective measures of performance. The combination of bilateral CI technologies with pre-processing algorithms designed to enhance speech processing in noise was assessed for CI users. WP5 assessed the performance of these algorithms in an instrumental evaluation as well as in normal-hearing listeners. WP6 spoke to an original goal of the ABCIT programme, which was to integrate information from WP1-5 into a formal prototype of a new binaural device. At the time of the proposal, the potential of Neurelec’s CI design to underpin our objective to create a truly binaural CI device, capable of autonomous acquisition of EABR responses, was the over-arching goal. Originally, the outcome of meeting the objectives by the end of the programme was to develop a prototype ‘second generation’ binaural CI device that will eventually be brought to market, providing the possibility of true binaural hearing in CI users. However, during the course of the project it became clear that many of the individual elements being generated could be applied to current CI devices, or used to undertake research projects, including to explore the limits of binaural hearing in CI subjects, without the development of the prototype. Further, the acquisition of Neurelec by William Demant Holdings (WDH) – the Danish Foundation dedicated to researching and commercializing solutions for deafness and other forms of hearing impairment and the development of Oticon Medical to bring binaural CI technology rapidly to market, had an energizing effect on the project.
Overall, the ABCIT project has been an outstanding success for the individual partners and for the collective. It has imbued a strong sense of engagement with the commercial partner by the academic partners, and provided significantly more rapid translation of basic scientific discoveries into commercial products and clinical therapies.
There are four cochlear implant companies in the world, two based in Europe. Neurelec is the smallest of all of the companies (10% market share) but has been growing strongly due to recent innovations. In particular, their world's first (and still only) binaural device offers significant value to patients and health providers by providing the benefits of two implants for the cost of a single implant, as well as the opportunity to restore truly binaural hearing. Although Neurelec’s market and market share have been growing, the company now needs to increase its research efforts in order to establish itself as a major industry competitor. The current proposal is specifically designed to expand Neurelec's market reach by providing industry-leading technological solutions for better CIs, and includes provision for testing and demonstrating the benefits of the solutions in patient groups. These solutions are vital if Neurelec is to continue to be competitive in the worldwide CI market. The advances being targeted include: tools to improve rapidly the design of cochlear electrodes and stimulation strategies, the development and testing of new CI-specific speech processing algorithms, the development of specific binaural algorithms for Neurelec’s existing and future-generation binaural devices, and investigations into the use of objective measures (e.g. advanced CAP analysis and EABR) for automated fitting - something that is now (within the last 5 years) considered essential for continued operation in the paediatric implant market. Ultimately, the goal is to gear these solutions towards improving hearing health outcomes, improving Neurelec’s current market position, and expanding into the US market, a market from which Neurelec is currently completely absent. The time-to-market for these solutions comes in two stages: (1) New input algorithms or binaural algorithms could be implemented in year 2 of the programme, following the development and testing of the algorithms by HörTech/Neurelec. These advances are also potentially backwards compatible to current generation CIs. If next generation implants are required, the benefits will follow the timeline of those releases (typically new processors released every 3 years). (2) The development of new electrode arrays and measurement of objective responses would benefits next-generation implant users at 3 years.
Project Results:
4.0 Description of the main S & T results/foregrounds
Binaural hearing confers considerable advantages in everyday listening environments. By comparing the timing and intensity of a sound at each ear, listeners can locate its source on the horizontal plane, and exploit these differences to hear out signals in background noise – an important component of ‘cocktail party listening’. Sensitivity to interaural time differences (ITDs) in particular has received much attention, due in part to the exquisite temporal performance observed; for sound frequencies lower than about 1.3 kHz, ITDs of just a few tens of microseconds are discriminable, observed both at the neural and behavioural levels. Sensitivity to ITDs decreases with ageing (Herman et al., 1977; Abel et al., 2000; Babkoff et al., 2002), is typically impaired in hearing loss, and sensitivity to ITDs is typically poor in users of hearing technologies such as bilateral cochlear implants (CIs). Thus, although CIs are the most successful sensory prosthetic devices developed to date, as judged by their ability to restore sensory and motor function (i.e. hearing and normal speech patterns) in the profoundly deaf, considerable progress remains to be made if CI users are to be able to hear and communicate in even moderately challenging ‘cocktail party’ environments where sources must be localized and individual conversations heard out from a background of multiple talkers, room reverberations and extraneous noise sources. The advent of bilateral CI (an implant in each ear), therefore, provides the perspective for achieving this improvement in performance but, to date, the expected improvements have not materialized. At least part of the reason for this impasse is that CI technology was never designed to take account of binaural hearing; from the persistence of monaural implantation, to the lack of any post-surgical fitting procedures that take account of binaural or spatial listening skills, or the lack of diagnostics for assessing performance in the increasing number of listeners (particularly young children) implanted bilaterally, the lack of a binaural perspective in CI hinders improvements in listening performance in CI users. The purpose of the ABCIT (Advancing Binaural Cochlear Implant Technology) project, then, is to progress CI technologies to the point that the potential benefits of binaural hearing for CI users are maximized, providing engineers, clinicians and researchers with the tools to explore. Divided into 6 inter-related Work Packages (WPs), the programme of research and development undertaken in the ABCIT project generated new research tools for investigating binaural hearing in BiCI users, provided diagnostic tools for assessing binaural performance, and assessed novel and existing ‘pre-processing algorithms’ that exploit binaural cues to ‘clean’ (i.e. remove background noise from) speech signals before they are processed by the implanted devices. Below, under the framework of the 6 WPs, we outline the research outcomes from the ABCIT project, highlighting some of the major advances made in the project, and illustrating how these advances will be used to improve listening performance in users of bilateral CIs.
1.0 Developing a Research Platform for Binaural CI (WP1)
WP1 developed a new research platform - the Neurelec ‘cHIRP’ – a research tool designed to generate data to enable researchers to maximize the benefits of binaural listening in BiCI listeners. The cHIRP research platform is specifically designed to focus on binaural CI processing by synchronously driving two ‘Neurelec SP’ implants to produce binaural cues (interaural time and level differences, ITDs and ILDs, respectively). In addition, the platform also supports variable stimulation rate processing for studying the effect of stimulation rate on perception (e.g. the effect of this rate on the perception of soundpitch). The cHIRP platform consists of bespoke hardware and associated PC software joined via USB connection. The hardware is responsible for: (1) digitizing the 4-channel input audio signals coming from two ear-worn microphone systems and (2) generating the final electric outputs needed to drive the two antenna coils. The software is responsible for processing the four audio signals and then generating two synchronized electrodograms from these signals. The software includes a flexible environment for the development of sound pre-processing (“speech processing”) and stimulation strategies. This architecture allows a diversity of approaches - including spectral and spatial signal enhancement and novel stimulation strategies (within the limits of the SP implant) - to be implemented and tested on real CI users without the need to create new hardware. The cHIRP integrates the Master Hearing Aid (MHA) developed by Hörtech.
1.1 Overview of the ‘cHIRP’ Research Platform
The cHIRP research platform provides a complete signal processing platform for cochlear implants capable of implementing and evaluating state-of-the-art pre-processing algorithms or novel ones in real-time. Figure 1.1 shows a block diagram of the hardware and software parts and indicates their tasks and interconnections.
The cHIRP hardware part is a PCB board (Figure 1.2). It is responsible for handling all ‘real-world’ signals and translating to and from these signals. The main tasks of this board include: (1) digitizing the acoustic input, (2) providing a first-in-first-out (FIFO) buffer and robust communication protocol for receiving commands, (3) producing the outputs needed to drive two CIs and (4) providing triggering and synchronization of all signals. The board receives 4-channels of acoustic input through two standard 3.5mm stereo jacks. Typically 4 microphones are used (two each from left and a right behind-the-ear (BTE) device worn by the subject). These signals are all digitized synchronously, packetized, and then transferred via USB to the PC-side MHA software for processing. The cHIRP platform has two Neurelec SP implant coil drivers that are driven from the same clock, and stimulation command packets are sent via USB from the PC-side MHA software. The exact timing of outputs to external events is controlled by selecting a specific trigger. Timing of relative interaural output ITDs (left vs. right stimulation) is encoded directly within the stimulation command packet that always contains binaural output data.
The PC-side software part of the research platform is a real-time signal processing system - the MHA (Grimm et. al., 2006). Originally designed for conventional hearing-aid research, the MHA allows many different signal-processing algorithms (e.g. noise reduction, beam-forming etc.) to be run in software at low latency. These algorithms can be cascaded to produce complex systems, and new algorithms can be coded at any time through the MHA’s ‘plug-in’ architecture. Originally designed for conventional hearing-aid research, the MHA allows many different signal-processing algorithms (e.g. noise reduction, beam-forming etc.) to be run in software at low latency. These algorithms can be cascaded to produce complex systems, and new algorithms can be coded at any time through the MHA’s ‘plug-in’ architecture.
1.2 Capabilities of the ‘cHIRP’
Outlined below are some of the capabilities and features of the cHIRP that make it well suited as a research tool to investigate real-time function in cochlear implants.
1.2.1 Latency
Latency between audio input and electrical output needs to be kept low in order to prevent ‘lip sync’ errors or the deterioration of speech production for test subjects. The minimum system delay for avoiding visual lip reading to auditory speech reception is ~17ms. Disruption to speech from delayed auditory feedback is maximum at about ~200ms [2-4] and is not expected to be a factor below 20ms.The actual minimum latency achieved for the various stages of the platform between the input acquisition and the stimulation without using any specific signal-processing algorithm (using a pass-through identity plug-in) are listed below.
1.2.2 Synchronization Accuracy
Because the system is focused on the binaural CI processing, the ear-to-ear (ITD) synchronization target was 15 µs based on the just noticeable difference (JND) of ITD for normal-hearing individuals at low frequencies (500-1kHz) [5]. However, one sample at 16.667 kHz corresponds to 60μs. Therefore, the entire system was designed to be sample accurate and synchronous. Since ITD information is carried within the stimulation packet itself, synchronization of coil outputs can be completely controlled and was measured to be better than 1µs with 0.5µs resolution. However variations in the processing delay between the reception of coil information and stimulation output of the implants means the synchronization cannot be guaranteed and must be assessed individually for each subject using equipment used to measure eABRs (electrical auditory brainstem responses).
1.2.3 Triggers
Neurelec SP implants stimulate all electrodes sequentially upon receiving a coil command. Hence, for the cHIRP platform, all electrodes must operate from the same global trigger. Four different stimulation output triggers are available: (1) an internal trigger based on a hardware timer, (2) an external trigger via BNC, (3) an audio synchronized trigger activated on audio packet transmission initiation, (4) and an immediate trigger that stimulates immediately upon receiving a stimulation packet. The internal trigger timer has a range from 1 Hz to 1000 Hz, settable in increments of 1 Hz. For the internal trigger, care must be taken not to over-run or starve the hardware FIFO. This is addressed by the communication protocol and, on the PC-side software, via an MHA output plugin. An external trigger is provided for interfacing with third party hardware such as clinical eABR machines or precision timers. The audio synchronized trigger creates an outputted stimulation for every audio packet initiated. This corresponds to the default rate for Neurelec SP implants of 520 Hz. It allows the lowest possible latency because no buffering is needed. The immediate trigger allows variable rate strategies to be tested. Since the entire array is sequentially stimulated on each trigger, dynamically changing the rate affects all electrodes (although electrodes can be disabled). It can be used to evaluate the effect of rate on pitch for example.
1.3 Signal processing and implementation of algorithms
The PC-side software, based on the MHA [1], is real-time capable software platform for developing and evaluating signal-processing algorithms employed in hearing aids. It supports algorithm developers with an extensive ‘toolbox’ that can be used to develop quickly new signal-processing algorithms. It comes with a collection of ready-to-use hearing-aid algorithms that can be combined with user-developed algorithms to form a complete software-based hearing aid capable of processing signals in real-time.
1.4 Intended Use of the Research Platform
The cHRIP research platform will be made available to researchers who are interested in investigating CI signal processing. Because the CI signal processing is performed on a PC, it is accessible to inspection and alteration. The platform can perform extensive data logging, replay of stored input signals, and alteration of existing algorithms and insertion of additional algorithms into the processing chain. New algorithms for the MHA are developed as plug-ins, and the signal path through the algorithms as well as algorithm settings can be altered at run time, making it possible to compare different algorithms or algorithm settings with subjective measures such as speech intelligibility, listening effort scaling, paired comparisons etc. Often in developing a hearing system trade-offs must be made between increasing the SNR of a signal and preserving binaural cues. Exploitation of spatial information will enable CI patients to perceive the direction of sound sources and will increase their ‘signal-to-noise ratio’ with the result that their speech perception in adverse listening conditions will be enhanced.
The PC running the MHA can be a portable computer, allowing the subject to carry the research platform with him or her and thereby use it outside of a laboratory. In a laboratory setting, the platform may additionally be connected to an audiological workstation, performing audiological tests by replacing the microphone input signal with the output of the audiological workstation, if desired. The MHA has been used to test different combinations of state-of-the-art speech enhancement algorithms on CI patients [6, 7, 8]. On these tests, the incoming signal was enhanced in the MHA and the enhanced signal was sent to the CI device as an audio signal. The CI device converted the audio signal into electric stimulation and performed the stimulation independent of the platform.
1.5 Examples of the use of the cHIRP Research Platform
In order to demonstrate the capabilities of the cHIRP, we highlight two examples using the research platform running in real-time. The first example shows the simultaneous production of ITD and ILD cues. The second example shows a variable stimulation rate. In both examples, we used simulated audio inputs to more clearly demonstrate them, however, the platform is equally capable of running on real-world microphone input.
1.5.1 Example 1: ITD/ILD Encoding
Figure 1.3 shows two snapshots of an oscilloscope depicting two instances of stimulation frames. These snapshots were taken during the same run of the platform at two different time instances. The bottom (yellow) traces show the digital output of the research platform for the left ear and the upper (red) traces shows the analog the coil voltages on the right ear at two different time instances. These traces encode the intensities of the stimulation in each frequency band. The audio input used to generate these stimulation frames was moving in the frontal hemisphere continuously and is composed of one low frequency (255 Hz) and one high frequency (4 kHz) component. The time delay between the upper and lower traces is caused by the calculated ITD. The ILD is encoded within the trace, so is not visible directly.
1.5.2 Example 2: Variable Stimulation Rates
Figure 1.5 also shows two snapshots of the oscilloscope taken while varying the stimulation rate on the fly. The time interval between two successive traces changes between the left and right panel depending on the fundamental frequency of the audio input. Here, the fundamental frequency was pre-determined, but zero crossings or other techniques could be used to extract the timing information. Here the stimulation rate alternates between ~2ms (500 Hz) and ~1.2ms (833 Hz).
1.6 User Safety
A clear consideration of any device employed to assess human performance is safety. Protection from over stimulation is ensured on the PC-side MHA software by way of a stimulation plug-in that cannot be circumvented. This plug-in reads clinical patient fitting files and blocks any stimulation that exceeds ‘C’ levels specified within the files. To ensure data are not corrupted in transmission (via the USB), the plugin contains a 16-bit CRC key for each stimulation packet. Any detected CRC errors on the hardware side results in the packet being dropped and no stimulation.
1.7 Conclusions
The cHIRP research platform provides researchers pursuing research on bilateral CIs the complete processing chain of a cochlear implant. It enables them to reconfigure this chain easily according to their needs and thereby filling an important gap in CI research by combining hardware and software within the same platform. The cHIRP will be useful for performing psycho-acoustical experiments on CI patients, including the electric stimulation of the cochlea for measuring the improvement in their speech intelligibility for different combinations of pre-processing algorithms. In the future, the platform will be extended to support wireless connections of microphones and will be made smaller in size for better portability, enabling researchers (and patients) to perform tests more easily in real-world environments. As any algorithm with real time capabilities can be implemented as a new plug-in into the Master Hearing Aid, it allows an arbitrary combination of different speech enhancement algorithms. The electric stimulation patterns are also generated in the Master Hearing Aid and transferred to the research board, which performs the stimulation.
2.0 A Simulation Tool for Developing and Assessing Binaural CI (WP2)
Computer simulations that model electrical stimulation of the auditory nerve have significant implications for CI research. First, they represent a cost-effective solution for product development and validation. Second, by describing accurately the effects of the different variables in play, they allow for new hypothesis to be formulated and, later, tested. Finally, they can illuminate potentially less-thoroughly described elements of the information chain. As such, simulation tools increase research efficiency by pre-validating clinical trials, saving time and resources. At the commercial level, CI simulation models can also be used to evaluate different processing and stimulation schemes and selecting promising candidates for animal or human testing. The simulation model can also be used as a reference model for developing and verifying new devices. The simulation model can be used as a front end for a research hardware interface to test new stimulation and processing schemes on existing patients
2.1 Modelling design
Different approaches have been undertaken when modelling the effect of electrical stimulation on the nerve. The essential reason for all these approaches is to find an acceptable trade-off between the details of the models and the complexity or computational cost required. The trade-off issue faced by a CI simulation tool is to find an optimal level of modelling accuracy that describes neural encoding by electrical stimulation, but without resorting to excessive computational resources. This trade-off requires that a ‘modular architecture’ be employed in the design. To this end, the simulation tool includes several modules each corresponding to a single element of the auditory pathway, from the electrode array positioned within the cochlea, to electrophysiological measurements of auditory-brain activity (Figure 2.1). By building the simulation tool as a chain or pipeline of modules, each element can be refined independently from data obtained during fitting, or they can be optimized for computation once correct accuracy is achieved.
Modeling and simulations can have different purposes and uses, depending on computational complexity and the desired inputs and outputs. To this end, tABCIT developed a realistic model of electrical stimulation by a CI - one linked to a specific Neurelec device for sound processing and electrical stimulation, rather than a generic implant. It also developed a model of binaural processing by auditory neurons to assess binaural responses to electrical stimulation of both ears through a Neurelec device. Key features of the model were its flexibility (allowing ‘evolution’ of the model), a capacity for fast simulations (low computational complexity) and ease of use for research and commercial purposes (e.g. through development of user-friendly tool on the Matlab programming platform).
2.2 Modeling Neurelec devices
Several Neurelec devices and processing strategies were simulated in the model, which included all processing parameters, as well as precision errors related to hardware, for example. This simulation is therefore a realistic and precise model of the Neurelec CI system, and is now employed within the company for verification of in-house ‘firmware’. The simulation time is also limited, as sound processing is required to be performed in real-time by external CI processors using minimal power. One outcome of this approach was that the development of firmware at Neurelec was altered so that all devices manufactured by the company are simulated with the new Matlab simulation tool. New ideas and coding strategies under development are all first simulated on the tool, and newly developed coding schemes can be encrypted and compiled as dynamic link libraries (DLLs) usable in Matlab or in the ‘C’ programme language confidentiality, but also easily shared and used as ‘black box’ model components by any of the research partner using the simulation tool.
2.3 Modeling cochlear geometry and the electrode array
A 2D model of electric field/current spread was first designed to provide quantitative predictions of the interaction between electrode contacts. This model is sufficient to produce a coarse approximation of those interactions and to provide the inputs necessary to calibrate the model of the auditory nerve fibres, with very fast computation. The level of detail, however, is not sufficiently accurate enough to provide critical information such as stimulation performance depending on intra-cochlear electrode array position, optimal design of apical electrodes, etc.
A 3D model of the cochlea was also developed and is currently being added to the simulation tool. PhD projects are still running to work on the generation of 3D cochleae based on patient data (obtained from CT-scans of the cochlea prior to operations to insert CI devices), with low calculation complexity. In order to achieve this, a parametric cochlear model was developed from excised human temporal bones, and this a priori knowledge of cochlear geometry can be used on low-quality clinical CT scans to reconstruct realistic cochleae using 3D modeling. The model is parametric and is configured with the Neurelec EVO electrode array. Thanks to parameterization of the 3D model, the position of the electrode can be freely chosen and the surface mesh can be modified to ensure that the electrode edges are properly modelled.
2.3 Neural models
A stochastic model by Goldwyn et al., (2012) has been used to model the responses of auditory nerve fibres to electrical stimulation (Figure 2.4). The model parameters have been fitted (by the authors) to recordings of single-fibre responses to electric stimulation. The model includes the effects of neural refractoriness and summation, and reproduces the difference in spike-timing precision observed for various stimulation rates. The model does not include the effects of low-threshold potassium channel or hyperpolarized activated currents, both of which are known to produce specific type of adaptation without spiking at high rates of electrical stimulation. However this model is realistic enough to generate spike patterns in response to a given electrical stimulation with fast computation.
Spike patterns are analyzed to evaluate perceived sound localization (angle and azimuth). The model is now binaural, and is sensitive to ITDs. In terms of estimating the azimuthal direction of arrival of a sound with the current Neurelec stimulation technique, ITD coding is of very limited use, because pulse timing is independent of the temporal fine structure. Therefore, the focus of the model was placed on assessing performance in response to interaural level differences (ILDs), expressed as the ratio of neural activity (spikes) of the right and left brain hemispheres. The model can track ILD changes over short (e.g. 200-ms) time intervals, can output several ratios for specified blocks of fibres along the cochlea, and can be calibrated and operated through a flexible and user-friendly graphical user interface, as well as an interactive 3D visualization interface of the azimuth localization data (Figure 2.5).
2.4 Assessing localization abilities in CI users
The model was used to simulate the spatial hearing abilities of bilateral CI listeners, with predictions similar to the subjective performance of bilateral CI listeners, not only in terms of the average localization error but also in the occurrence of systematic errors in localization judgments. In general, the model confirmed a range of hypotheses from experimental studies, including the dominance of ILD cues in lateralization judgments, and the detrimental effect of level-dependent compression. The model also enables identification of the origin of any unexpected prediction along the processing chain. Beyond that, the model could be useful in predicting performance of individual CI subjects by customizing the model to their clinical profile, and in developing algorithms to enhance spatial hearing performance.
2.5 Modeling cochlear anatomy and electric fields
The 3D modeling of the cochlear anatomy, as well as electric field generated by a given electrode-array, are critical topic for electrode design. Two PhD projects were initiated on those two topics during the course of the ABCIT project, in collaboration with local (to Neurelec) academic partners outside the consortium. The parametric model of cochlear geometry was developed by one of these projects (Figure 2.6) with the long-term objective of generating patient-specific cochlear geometry in order to predict electrode insertion and placement. Eventually, electrode array insertion might also be dynamically modeled, in order to estimate cochlear damage and trauma upon insertion. For a given electrode position, the combination of bone, soft tissues and liquids, all with different conductivities, makes the electric fields difficult to predict. In order to better employ multipolar stimulation, or current steering/current focusing techniques, the second PhD has modeled electric fields within the cochlea, for a given combination of anatomical configuration and electrode-array (Figure 2.7).
2.6 Product development
The tool has been already use by Neurelec to evaluate coding strategy ideas and new concepts. Without such a model, it would be necessary to undertake clinical investigations or animal experiments, both of which require ethical approval and, in the case of human trials, verification and validation tests to obtain other regulatory clearance for non-CE approved prototypes and software. Both prototype and study designs have to be set, and study protocols and technical dossiers advertised. Depending on prototype complexity and study design, all clearances may take 6 months to a year, and costs are around 5-10k€. Further, the study itself requires coordination and project management (especially if the trial is conducted at several hospital sites), and participants have to be informed prior to inclusion. Trial budgets are also required to cover hospitals additional costs, estimated at 0.5-2k€ per subject, depending on study design (number of visits especially). A pilot clinical study may use 10 participants, up to 50 participants for a high-proofed trial, with study duration of 3 months to 1 year per subject. Altogether, a clinical trial may long 1 to 2 years in total, and may cost 10 to 100k€, with very limited possibility of fine-tuning. In comparison, model using fast prototyping using the Matlab model is very convenient, and first results are generated rapidly. Even though clinical tests are mandatory for final verification, all fine-tuning of CI stimulation algorithms can be performed with the model.
The model has been was used to evaluate new coding strategies and fitting parameters: spike patterns are generated with commercially available coding, with new prototype coding, and for the normal-hearing auditory system, as well as to estimate the battery life of the prototype CI system. In fact, as the charge delivered by the electrodes can be linked to the generation of neural action potentials (and, therefore, sound perception), the model helps estimate the useful energy with realistic clinical maps (thresholds and comfort clinical levels observed in real patients, data available to Neurelec). Future perspectives and product development plans with use of the simulation tool, following completion of the ABCIT project, include the connection to the research hardware platform developed as part of WP1, further tests and verification on signal processing optimization and stimulation strategies.
2.7 Conclusions
The development of a simulation tool for exploring neural responses to bilateral stimulation of CIs is a critical change on the commercial side, and has changed the development process within Neurelec/Oticon Medical. Now, all signal processing schemes and stimulation strategies developed and under development are designed and benchmarked through the model. Electrode-arrays design also benefit from modeling, in order to better predict placement, and electric fields in the cochlea. This allows a dramatic reduction of animal and clinical tests, with first hypothesis tested and compared numerically. Therefore, more risks can be taken and foreseen in terms of product development, as more ideas can be tested. Neurelec/Oticon Medical is now more proactive in development process, and our engineers no longer wait for published concept to initiate product development but rather propose new processing and coding schemes for development based on the model output.
References
Daanouni, E., Segovia-Martinez, M., Frater, A., Gnansia, D., Hoen, M. (2015). Residual hearing preservation, simulated and objective benefits from electroacoustic stimulation with the Evo-Zebra system.
Dang, K., Clerc, M., Vandersteen, C., Guevara, N. and Gnansia, D. (2015). In situ validation of a parametric model of electrical field distribution in an implanted cochlea. Proceedings of the 7th International IEEE EMBS Neural Engineering Conference (NER'15), April 22-24, Montpellier.
Frater, A., Maas, P., Undurraga, J., Kamaric Riis, S. (2015). Electro-acoustic cochlear implant simulation model.
Frater, A. (2014). Development and objective evaluation of an EAS cochlear implant model. Aalborg University Master thesis.
Goldwyn, J., Rubinstein, J., Shea Brown, E. (2012). A point process framework for modeling electrical stimulation of the auditory nerve. Journal of Neurophysiology, 5:1430-1452.
Hines, A., Harte, N. (2012). Speech intelligibility prediction using a neurogram similarity index measure. Speech Communication, 54(2):306-320.
Huang, T., Frater, A., Segovia-Martinez, M. (2015). A Gammatone filterbank and zero-crossing detection approach to extract TFS information for cochlear implant patients
Kelvasa, D., Dietz, M. (Submitted). Auditory model-based sound direction estimation with bilateral cochlear implants
Maretic, P., Frater, A., Kamaric Riis, S., Widell, A., Segovia-Martinez, M. (2015). Optimizing compression stimulation strategies in cochlear implants for music perception.
Segovia-Martinez, M., Daanouni, E., Gnansia, D., Hoen, M. (2015). Automatic post-spectral compression for Oticon Medical’s sound processor
Vatti, M., Frater, A., Huang, T., Segovia-Martinez, M., Pontoppidan, N. H. (2015). A phase-lock loop approach to extract temporal-fine structure information for cochlear implant patients.
3.0 Developing Novel Electrical Stimulation Strategies (WP3)
Many issues impact on the efficacy of cochlear implants. A general problem is the uncontrolled spread of electrical current within the cochlea when stimulating with even a single electrode contact. This results in many more auditory nerve fibres being stimulated than is desirable, and contributes to the poor sound-frequency resolution observed in CI users. In terms of bilateral CI users, the restoration of binaural hearing is hampered by the relative insensitivity of neurons to temporal information. Finding ways to reduce the spread of electrical current and enhance temporal coding so that interaural time differences are more readily conveyed in implants, were major goals of the ABCIT project. WP3 also assessed the extent to which ITD cues might most readily be delivered to CI users, and established new means of testing binaural performance, based on experimental work undertaken throughout the ABCIT project.
3.1 Ramped pulses: a novel stimulation strategy
At least some of the limitations in CI performance come about because of the design of the devices themselves. CIs work by electrically stimulating the primary auditory neurons (the spiral ganglion neurons, SGNs) that constitute the auditory nerve, through an electrode array inserted into scala vestibuli of the cochlea. A major factor limiting improved performance in CI is uncontrolled spread of electrical current within the cochlea itself (Middlebrooks 2004). This current spread is an intrinsic consequence of the implant design; the size of the electrodes, their distance from SGNs, and the fact that they are immersed in a highly conductive fluid environment, leads to a wide population of SGNs being activated with each electrode. Controlling the spread of excitation in situ remains problematic, and severely limits the number of independent channels of acoustic information that can be represented in electrical stimulation. We addressed this engineering-to-biology mismatch with an innovative stimulation strategy using novel pulse shapes based both on biophysical principles of SGNs and on the interaction of the electrical stimulus with the cochlear environment.
3.1.1 Theoretical considerations for employing ramped pulses in CI
We designed a new ‘ramped’ electrical pulse for CI stimulation (Fig 3.1A) to replace the square pulses (Figure 3.1B) most commonly employed in current CI technologies. Figure 3.1A shows the ramp pulse and its parameters, including the Foot (FA) and Peak (PA) amplitudes where FA is the amplitude at the start of the slope and the PA the maximum amplitude of the slope. In addition, the ramp pulse can also be defined in duration of phase (PD). and the inter phase-time duration (IPI).
The rationale behind this new stimulus shape relies on two important features of auditory neurons and the cochlea itself - the biophysical properties of the SGNs, and the principle that stimulation in aqueous media (here, the fluids of the inner ear) results in diffusion of the electrical current (Figure 3.2). The theoretical outcome of ramped pulses is that they will reduce the influence of the spread of current on neurons, without the need for an actual reduction in the spatial current spread. For two current pulses with equal peak value, one square (Figure 3.1A) the other with a ramp of defined slope (Figure 3.1B) the SGN at the focus of the stimulation will have a higher firing threshold for the ramped pulse. However, moving further from the stimulation point, the combination of the reduction in amplitude and in slope will result in a faster decrease in firing for the ramped stimulus compared to the square one, therefore effectively reducing the spread of excitation (Figure 3.2). Ramped current pulses can be defined according to their slope, specifically the rate at which the slope increases. Considering the dynamics of ion channels in SGNs, the rate of change of current will result in increased or decreased excitation of the neural population proportional to the slope.
3.1.2 Responses of SGNs to injections of square current pulses
We assessed whether or not ramped pulses are more efficient than square pulses for modulating the neural firing patterns by recording responses of SGNs from cochlear cultures from P12-15 mice to injected current pulses with either the traditional, square shape or with the new ramped design. A key outcome was to characterize the effect of the different stimulus parameters on SGN firing in order to better predict the effect of this new stimulation paradigm in the context of electrical hearing. We focused on how changing the different parameters, including the FA and the PA, might affect the efficacy with which current pulses evoke action potentials in SGNs. Our data (Figure 3.3) demonstrate that all the parameters proposed as part of the ramp stimulus influence SGN firing and could be beneficial to electrical stimulation in CI. Importantly, for populations of SGNs, the ramped pulses showed a greater range of responses (more variability in firing for a given input current level) meaning that the dynamic range (the range of sound intensities encoded by the neurons) is increased by the ramped stimulus.
3.2 Designing stimuli to convey ITD information
We also sought to establish which stimulus shapes are most effective in conveying ITD information in CI listeners. Currently, most CI listeners have access only to ITDs conveyed in the envelope of modulated sounds, and not those conveyed in the temporal fine structure of low-frequency sounds – the latter is the type of ITD information that provides significant benefit in noisy listening situations. Do specific envelope shapes exist that more readily convey ITD information, and can these be provided to CI users to enhance their spatial listening? Envelope ITDs are the only ITD cues available through today’s CI devices, and modification of the envelope, e.g. by enhancing the rising slopes, might improve users’ perception of auditory space.
3.2.1 Determining the best envelope shapes for conveying ITDs
Following a study by Klein-Henning et al (2011) in normal-hearing human listeners, we assessed the responses of neurons recorded from the midbrain of anaesthetised guinea pigs to a range of different envelope shapes. These shapes varied in four envelope components – attack, sustain, decay and pause (see Figure 3.4) were systematically varied, and confirmed the importance of the attack and pause components in generating ITD-sensitive responses. We also used a simulation model (such as that described in Section 2 above) to assess the neural responses we obtained. Figure 3.5 shows ITD tuning of neurons in the guinea pig midbrain (2nd row) for various types of the stimulus envelope (upper row). The lower row shows the ITD tuning mimicked by the simulation tool.
This study showed the importance of the rising slope in conveying envelope ITDs, and that their enhancement could potentially benefit neural sensitivity to ITDs sensitivity. Other studies of human performance found that binaural sounds with steep attack phases, and long pauses were those to which listeners were most able to lateralize to one side or the other (Klein-Hennig et al, 2015).
3.2.2 Determining where in the sound envelope CI users are sensitive to ITDs
Sound localization in reverberant environments is complicated by the fact that sound reflections carry ITDs and ILDs that can be very different to those actually directly from the source. CI listeners have great difficulty with reverberation – they are unable to separate out the mixture of direct sound and later-arriving reflections. To this end, a beneficial strategy would be to provide greater weight to the interaural differences at the onset of the stimulus or at the onset of each modulation cycle. During these onsets, the direct-to-reverberant ratio is typically optimal, and the interaural differences in this short moment are informative as to the source location. We tested where in the modulation cycle CI users actually extract ITD information compared to normal-hearing listeners. We found that in amplitude-modulated sounds, such as speech, normal-hearing listeners show strongly enhanced sensitivity to ITDs during the early rising portion of the modulation cycle, reflecting a higher weight on temporal read out for that signal portion. This is were ITD information in reverberant rooms is most reliable as to the true location of the sound source BiCI listeners, however, appear to be most sensitive to ITDs later in the sound wave-form, meaning that the later-arriving spurious ITDs – the result of reflections – will dominate their perception. The information from these experiments is now being used to develop new algorithms that will enhance ITD sensitivity in CI listeners.
3.3 Conclusions
The novel approach to stimulation we propose for CI stimulation suggests shaping individual electrical pulses to take account of the biophysical properties of SGNs and, in so doing, reduce the spread of excitation within the cochlea that comes from electrical stimulation. The two key features of this new strategy are: 1) the introduction of a slope in the stimulation pulse and the consequent effect this will have on the current spread, and 2) the sensitivity of auditory neurons to the rate at which the magnitude of current pulses is increased. Our in vitro data from SGNs represents a proof-of-concept for a new stimulation strategy for CI. We now plan to assess the potential for ramped pulses to limit the spread of neural excitation using Oticon Medical’s Animal Stimulation Platform (ASP), developed on the basis of a new generation stimulation chip. The binaural psychophysics, neural recordings and modeling demonstrate the important of the rising, or attack, phase of sound envelopes in conveying ITD information. Since these envelope ITDs are the only cues available to CI listeners, it suggests ways in which the envelope attack might be sharpened to provide better spatial hearing in BiCI users.
4.0 Objective Measures of Binaural Hearing (WP4)
The potential to provide CI listeners with true binaural hearing and to exploit the sensitivity of the auditory brain to differences in the sound presented to the two ears as a tool for fitting (matching) the populations of nerve fibres activated by electrical stimulation of the two ears are key goals of bilateral CI research. To this end, considerable effort was put into developing objective measures of binaural activity – measurements of brain activity that report sensitivity to binaural cues such as ITDs (e.g. He et al. 2010) – and in comparing these measures to traditional, subjective measures (e.g. patient-reported perceptual comparisons of the pitch evoked in each ear by stimulation of different electrodes). A critical outcome of this aspect of the ABCIT project, therefore, was to develop the tools required to assess factors known to be critical to binaural hearing, including how well matched is the electrical stimulation across the two ears (Hu at al. 2015a). Two broad research themes were investigated. The first developed new measures of ITD sensitivity using electroencephalography (EEG) in normal-hearing listeners, and sought to apply these measures to bilateral CI users. The second employed traditional EEG measures of binaural integration (the binaural interaction component, or BIC) and assessed the magnitude of the BIC as a function of stimulated electrode pairings across the ears in CI users, comparing the preferred electrode configuration with those suggested by purely subjective (perceptual) measures of electrode pairs. A key goal of both themes was to develop truly objective fitting procedures for matching implant stimulation across the ears, including in situations (e.g. children) where it is difficult or impossible to obtain subjective assessments of CI performance.
4.1 Developing an objective measure of ITD sensitivity
We developed an objective measure to assess neural sensitivity to ITDs conveyed in the temporal fine structure (TFS) of low-frequency sounds, and to distinguish it from neural sensitivity to ITDs conveyed in the temporal envelope of amplitude-modulated (AM’ed) high-frequency sounds. Such a measure would be useful in developing stimulus strategies to improve spatial listening in CI users (who currently can only access binaural cues conveyed in the temporal envelope). Using EEG, we recorded brain activity to sounds in which the interaural phase difference (IPD) of the TFS (or the modulated temporal envelope) was repeatedly switched between leading in one ear or the other. When the amplitude of the tones is modulated equally in the two ears at 41 Hz, the interaural phase modulation (IPM) generates an intracranial percept of a sound moving from one side to the other – and evokes an IPM following-response (IPM-FR) in the EEG signal (see Figure 4.1).
4.1.1 Sensitivity to IPDs conveyed in the temporal fine structure of low-frequency sounds
EEG recordings were obtained for different rates and depths of IPM, generating a total of 30 conditions (21 dichotic, 9 diotic), each lasting ≈5 min, so that the total recording time was approximately 2.5 hours. For low-frequency signals (520-Hz carrier), IPM-FRs were reliably obtained for a wide range of modulation rates and IPDs, and were largest for an IPM rate of 6.8 Hz and when IPD switches (around 0°) were in the range 45-90°. Increasing the modulation frequency increased the magnitude of IPM-FRs. Figure 4.2A plots the spectral magnitude of a typical recording for a dichotic condition with IPM of ±67.8° and IPM rate of 6.8 Hz, in which a significant response was observed for the frequency bin corresponding to the IPM rate (black arrow), and corresponding diotic conditions are shown in Figure 4.3 - for which no response to the IPM rate was observed (see Ross, 2008). Note that in both conditions the ASSR to the AM rate (41 Hz) was clearly observed.
4.1.2 Sensitivity to IPDs conveyed in the modulated envelope of high-frequency sounds
With current state-of-the art devices, BiCI listeners would not be expected to be sensitive to ITDs conveyed in low-frequency sounds, but many appear to show some behavioural sensitivity to ITDs conveyed in AM’ed high-frequency sounds. To develop an objective measure of high-frequency envelope ITD sensitivity, we recorded EEG recordings in normal-hearing listeners to ITDs conveyed in the stimulus envelope; 3000-Hz tones were modulated with a transposed envelope (Bernstein and Trahiotis 2002) at 128 Hz, and a second-order AM (41 Hz) applied diotically. IPDs of the 128-Hz envelopes were switched between ±90° IPD (around 0°) or from systematically between 0° and 180° (i.e. transposed phase of -90° in one ear and +90° in the other). In contrast to low-frequency tones, the magnitude of the IPM-FR for IPDs conveyed in the modulated envelopes of high-frequency (3-kHz) tones was relatively low for IPMs switching between ±90° IPD. Therefore, we also assessed EEG responses to IPMs switching between 0° (in phase) and 180° (anti-phasic) IPD conditions. These latter switches evoked considerably larger EEG responses (Figure 4, left). The magnitude of the IPM-FR decreased with increasing carrier frequency, and was sensitive to an offset in the carrier frequency between the ears (Figure 4, right).
IPDs conveyed in envelope of high-frequency tones (3-kHz tones AM’ed at 128 Hz, with a second-order modulation of 41 Hz) also generated a reliable pattern of IPM-FRs, but one in which response maxima occurred for IPDs switched between 0° and 180° IPD. The data are consistent with the interpretation that distinct binaural mechanisms generate the IPM-FR at low and high frequencies, and with the reported physiological responses of medial superior olive (MSO) and lateral superior olive (LSO) neurons in other mammals.
4.2 Objective measures of binaurally matched electrodes
Although users of BiCIs can take advantage of some of the benefits of having two sound receivers - enjoying better sound localization by virtue of being sensitive to ILDs, for example - there is little evidence that BiCIs employing commercial speech processing strategies are successful in restoring the advantages of ITD sensitivity, particularly the binaural unmasking that enables listeners to hear out speech in background noise. One reason for the large variability across bilateral CI subjects in spatial hearing performance is the potential for mismatches in the electrodes stimulated between the left and right CIs. Such mismatches might be the result of different surgical insertion depths, different implant lengths, or by differences in how the electrodes interface with the auditory nerve in each ear. In normal-hearing listeners, inputs from the two ears to binaural brainstem neurons are assumed to be well matched, and we expect this factor to be very important in BiCI listeners - matched pairs of electrodes should process the same acoustic frequency band, effectively compensating for any differences between the two implanted cochleae. Interaural electrode pairing (IEP) is likely to be increasingly important in future technological developments, such as in developing truly binaural CI coding strategies, which will preserve, enhance, and even optimize interaural cues such as ITDs. We compared three techniques to try to establish a ‘gold-standard’ objective measure for matching electrodes across the two ears. We hypothesized that the electrode pair that elicited the largest BIC amplitude would be the pair that also resulted in best subjective performance in discriminating ITDs (here, listeners determined whether the sound was heard to the left or heard to the right). Our second hypothesis was that the matched pair for evoking the largest BIC and the best ITD sensitivity might not be the best pitch-matched pair.
4.2.1 Matching electrodes across the ears for an individual BiCI listener
We assessed 7 different BiCI users to determine how well matched their devices are across the ears. In order to do this, we stimulated electrode 4 in the left ear, and systematically assessed the response to simultaneous stimulation of one of the electrodes in the right ear. The data for an individual listener are shown in Figure 4.5 and in Hu et al. (2015b).
4.2.2 Comparing best electrode pairs across all BiCI listeners
Table 4.1 lists the three IEP results for each subject (S). In general, the data indicate well-tuned pitch matching, ITD sensitivity, and BIC amplitudes as a function of electrode number. However, the pairing results were not consistent across methods. All but S2 was assessed using electrode 4 in the left ear. Generally, the comparison that produced the best pitch match was centered on electrode 4 in the right ear, whereas the comparison that produced the largest BIC or the best subjective sensitivity to ITDs, were not well matched to each other. The two binaural measures – the (objective) BIC and the perceptual (subjective) assessment of ITD sensitivity, however, were better matched. See Hu and Dietz (2015) for further details.
4.3 Conclusions
We demonstrated that an objective measure of binaural sensitivity – the IPM-FR - can be reliably obtained and used as an objective measure of binaural processing, at least in normal-hearing listeners. The EEG responses we recorded are consistent with evoked responses occurring in cortical and sub-cortical sources along the auditory pathway. Further, we have identified differences in the preferred IPDs for EEG signals conveying IPDs in the low-frequency TFS compared with the modulated envelopes of high-frequency tones, and these differences are consistent with the underlying physiological mechanisms of neurons in the brainstem pathways sub-serving these two cues. The IPM-FR, therefore, appears able to distinguish between these pathways based on the preferred IPDs employed to evoke the response, and provides a possible means of distinguishing which of the binaural pathways is activated in bilateral CI users, as well as a means of objective fitting of devices. Work is currently underway to assess the IPM-FR in bilateral CI users. In BiCI users, we observed a systematic discrepancy between the pair of electrodes that were best matched for pitch and those for which the best ITD sensitivity was found. We also observed a strong match between electrode pairs that were best matched for ITD sensitivity measured subjectively, and those pairs that generated the largest BIC response, suggesting that the subjective measure of binaural sensitivity provide a better measure of ‘true’ electrode matching than does pitch matching across the ears.
References
He, S., Brown,C. J., Abbas, P. J. (2010), Effects of stimulation level and electrode pairing on the binaural interaction component of the electrically evoked auditory brain stem response, Ear and Hearing, vol. 31, pp. 457-70.
Hu, H., Kollmeier, B., Dietz, M. (2015a). Reduction of stimulation coherent artifacts in electrically evoked auditory brainstem responses. Biomedical Signal Processing and Control, 21, 74-81.
Hu, H., Kollmeier, B., Dietz, M. (2015b). Suitability of the binaural interaction component for interaural electrode pairing of bilateral cochlear implants, ISH 2015 (in press).
Hu, H., Dietz, M. (2015). Comparison of interaural electrode pairing methods for bilateral cochlear implantation, Trends in Hearing (submitted).
McAlpine, D Haywood N, Undurraga J, Marquardt T (2015) Objective measures of neural processing of interaural time differences, Advances in Experimental Medicine and Technology (in press)
Ross, B., Tremblay, K. L., and Picton, T. W. (2007). “Physiological detection of interaural phase differences”. J. Acoust. Soc. Am. 121, p. 1017.
5.0 Developing and assessing algorithms for enhancing speech in background noise (WP5)
A critical outcome for the ABCIT project was to enhance the spatial-listening performance of CI users, in particular through the exploitation of binaural cues, especially ITDs which not only enable listeners to determine the location of the sound source, or sources, but also to hear out sounds in background noise. CI listeners are unable to perform binaural hearing in any way like normal-hearing listeners, and are particularly vulnerable to interference by background noise or competing talkers. In electrical (CI) hearing, therefore, two different stages of processing have to be considered when seeking to exploit binaural cues for listening in noisy conditions. The first stage is pre-processing – using sound-processing strategies to remove background noise from speech signals (a process sometimes referred to as ‘speech enhancement’). Computer algorithms that seek to achieve this are widely implemented in hearing aids so that the acoustic signal provided to hearing-impaired listeners is ‘cleaned up’, making it easier for hearing-aids users to comprehend speech in background noise. Implementing such algorithms before sounds are converted into electrical signals by CI devices has the potential to provide CI listeners with considerable benefit. The second stage is stimulation strategy – the shaping of patterns of electrical stimulation over time or over space (i.e. over different electrode contacts on the CI) to maximise the transmission of information by the auditory nerve. In this regard, the key question for the ABCIT project is ‘Is it possible to stimulate electrically each auditory nerve so as to maximise the benefits of having two CI devices, one in each ear?’. To this end, the main objective of WP5 was to improve speech and spatial perception in CI users by combining state-of-the-art pre-processing algorithms developed for use in hearing aids as well as novel algorithms, in combination with stimulation strategies employed in CI devices.
5.1 Testing algorithms for enhancing listening-in-noise
In general, assessment of existing and novel pre-processing algorithms by CI users indicated significant improvement in speech intelligibility. To date, these have not been implemented in combination with stimulation strategies due to the delay in the finalisation of the WP1 real-time research platform. Nevertheless, due to the keen interest in maximizing benefits to CI listeners receiving the new Neurelec/Oticon Medical CI devices, the best and most efficient of these algorithms will be made available for research and implementation. To this end, the consortium focused on estimating and enhancing spatial information from the noisy multi-microphone signals by making use of true binaural methods. The limitations of the monaural and bilateral methods motivated the consortium for working on binaural approaches. Besides, binaural approaches incorporated in other audio signal processing domains, e.g. hearing aids signal processing, yielded significant improvement in terms of instrumental (i.e. using objective, technical measures) as well as perceptual evaluation compared to other approaches. Finally, the unique binaural design of the NEU implants offered a natural motivation to the consortium to develop and evaluate this type of algorithms directly on NEU implants.
5.1.1 State-of-the-Art Pre-Processing Algorithms
We selected and evaluated state-of-the-art pre-processing algorithms specialised for hearing aids. The most promising candidates were implemented in the real-time signal-processing platform, i.e. the MHA, for further assessment. The list of algorithms ranges from differential microphones combined with coherence filters to spatial ‘beamformer’ methods together with single-channel noise reduction schemes. These algorithms were assessed in three different realistic environments, a multi-talker babble noise (20T), a cafeteria ambient noise (CAN) and a single competing talker (SCT). The perceptual evaluation performed with CI users showed a substantial improvement of speech intelligibility in terms of speech reception thresholds (SRT50) as shown in Figure 5.1 (left). In the multi-talker babble condition, the highest improvement yielded in SRT50 is around 6.9 dB ± 1.2 dB, a substantial improvement Similarly, in the realistic cafeteria ambient noise scenario, SRT50 improvements of 5.3 dB ± 2.0 dB were achieved. Finally, in the spatially separated single competing talker scenario, maximum SRT50 improvements of 15.2 dB ± 3.6 dB were achieved. These results are very promising in terms of the improvement of the speech reception threshold. However, the consortium investigated whether CI users prefer one algorithm over the others, in particular over the unprocessed signal (Figure 5.1 right).
In general, the CI users who participated in the evaluation studies preferred all algorithms compared to the unprocessed signal. We performed extensive evaluation experiments in the three different realistic listening conditions. Only in the third noise scenario (SCT), two of the algorithms were not preferred. Furthermore, novel pre-processing algorithms were developed in particular for enhancing the spatial listening abilities of CI users. These algorithms incorporating and combining approaches from different research domains depicted significant improvements compared to state-of-the-art approaches and showed large potential for benefit in the next generation CIs.
5.1.2 Steering Beamformer
In one study, we worked on developing a ‘steering beamformer’ approach to enhance a moving speech source in a diffuse noise environment. The proposed system estimates the direction of arrival (DOA) of the target speech source (Kayser and Anemüller, 2014) and of a spatial filter (Bitzer and Simmer, 2001). We evaluated the proposed algorithm in three different angular velocities for the target speech source, 15°/s, 30°/s and 60°/s. Experiments performed on normal-hearing listeners and CI users proved that the proposed steering beamformer is superior to its state-of-the-art competitors in tracking and enhancing moving speech sources. In these experiments, we measured the speech reception thresholds (SRT50) for the proposed algorithm as well as for a fixed beamformer setup (0° beamformer), where the target speech source is assumed to be in front, for a steering beamformer with the true target direction (perfect DOA beamformer) and for the unprocessed signal. The results are shown in Figure 5.2. The proposed steering beamformer approach achieved a 3.1dB SRT50 improvement in the 15°/s case compared to the unprocessed signal, whereas the fixed beamformer degraded the performance by 1.5dB. The proposed method performed similarly to the steering beamformer with the true DOA.
5.1.3 Extent of Lateralisation
We also performed experiments on spatial listening by investigating the extent of lateralization (how far to the left or right a sound is heard) of click trains with large ITDs on normal-hearing listeners. In these measurements, the consortium focussed on the extent of lateralization of large ITDs (> 600µs – close to the maximum range of ITDs generated by the human head). Stimuli consisted of unfiltered and band-pass filtered trains of clicks at different inter- click intervals (ICI), which acoustically mimic the effect of electrical (CI) pulses. An acoustic pointer procedure (Bernstein and Trahiotis, 1985) was employed to determine the extent of lateralisation of click trains. Results show that an increase in lateralisation percept continues beyond the physiological limit (~600µs); see Figure 5.3. These results are an indicator of bilateral CI percepts.
5.1.3 Estimating the Inter-channel Phase Differences (IcPD)
We also developed novel methods incorporating statistical approaches in audio-signal enhancement. In one of these studies, we computed so-called ‘short-term cross-correlation functions’ between the channels of a given multi-channel multi-source audio input signal and extracted the non-linear inter-channel phase differences (IcPDs) as well as a measure of activation for each source. The source separation performance of the proposed method was compared with the unprocessed signal as well as with the linear IPD estimation (TDOA) and with the true IPDs. The results are shown in Table 5.1. The linear IPDs are estimated based on the main direction of arrival estimation evaluating the peaks in the cross-correlation matrix. Both linear and non-linear IPD estimation methods lead to a significant improvement compared to the unprocessed signal. Furthermore, the proposed method clearly outperforms the state-of-the-art linear method.
5.1.4 Sparsity-based Noise Reduction
In another study, the consortium extended a non-negative matrix factorization (NMF) based noise reduction algorithm by automatically detecting and reducing noise activations within a given noisy audio signal. In the NMF approach, Hu et. al. (2012) decomposed the envelope matrix of CI frequency channels into the two matrices, which we can name as basic envelope patterns and their activations. Subsequently, Hu et. al. eliminated the noise activations manually. The consortium incorporated a sparse method for automatically detecting and reducing the noise activations within the given noisy audio signal. For this, the consortium incorporated a method for sparsely coding only the activation of the speech segments using a dictionary (code-book) and thereby reducing the noise. The first technical tests were performed using normalised covariance metric (NCM). Table 5.2 shows the results of these preliminary experiments using this metric. Here, higher values imply better speech intelligibility. The consortium performed sparse coding experiments using different number of dictionary atoms extracted. Increasing the number of atoms used for coding increased the speech intelligibility in terms of the used metric.
The preliminary test of the sparsity-based noise reduction algorithm in combination with the NMF algorithm depicted an improvement in the speech intelligibility compared to a noise reduction method based only on the NMF method in terms of the normalised covariance metric.
5.2 Conclusions
The data indicate that CI users can benefit from binaural pre-processing algorithms in a range of different listening environments. As these algorithms consider the left and right ear together, the spatial listening is substantially improved using binaural setups compared to simply ‘bilateral’ algorithms, where left and right ear are handled independently. Incorporating these types of algorithms in next-generation CIs will have a significant impact on users’s abilities to communicate in challenging environments e.g. environments with background noise, reverberation and/or multiple competing talkers. By combining these pre-processing strategies with better stimulation strategies, the promise of improved speech intelligibility may be realized.
References
Bitzer J, Simmer K (2001). Superdirective Microphone Arrays. In Microphone Arrays, Digital Signal Processing, 19-38, Springer, Berlin, Heidelberg.
Bernstein LR, Trahiotis C (1985). Lateralization of Low-Frequency, Complex Waveforms: The Use of Envelope-Based Temporal Disparities. J. Acoust. Soc. Am., 77(5), 1868-1880.
Hu H, Mohammadiha N, Taghia J, Leijon A, Lutman ME, Wang S (2012). Sparsity Level in a Non-Negative Matrix Factorization based Speech Strategy in Cochlear Implants. Proceedings 20th European Signal Processing Conference (EUSIPCO), Bucharest, Rumania.
Kayser H, Anemüller J (2014). A Discriminative Learning Approach to Probabilistic Acoustic Source Localization. Proceedings 14th International Workshop on Acoustic Signal Enhancement, Antibes, France.
6.0 Binaural Device Prototype (WP6)
WP6 was designed to combine and integrate the information gleaned from WP1-5 into a collection of hardware demonstrators that, together, form a novel binaural CI. We tailored the hardware and software solutions specifically to take advantage of binaural cues and the influences from the previous work-packages. A sketch of the major influences is shown in Figure 6.1. A new binaural codec was developed to support these contributions and hardware was designed and built around this codec to test individual parts of the system. Ultimately, a feasible binaural CI solution is presented from these pieces that produces an expected performance gain of 3-10 dB SNR when compared with unsynchronized bilateral implants (see WP5) with a lower cost than is currently available from traditional bilateral solutions. The benefits of an advanced CI come from three sources: (1) the availability of pre-processing algorithms such as binaural beamforming (WP5) that can access information from 4 synchronized microphones located on both sides of the head, (2) novel stimulation strategies that can transmit ITD as well as ILD information, (3) the ability to measure objectively eABR and cortical responses which could impact fitting. To realize all these benefits, the problem of synchronization must be overcome, an appropriate binaural electrode array must be created, and a scheme for transmitting the data must be devised. All of these issues were addressed in this prototype.
6.1 Developing Elements of the Prototype
6.1.1 Chosen Binaural implant Architecture
Many CI companies are adding binaural capabilities to their monaural offerings, however the ABCIT binaural CI is designed to be binaural from the ground up. The architecture leverages the current Neurelec binaural design. In this design, instead of using separate CI processors, a single binaural processor is used. The ABCIT design builds on this architecture. Here, a single stimulation engine running from a single clock serves both ears which means only 1 synchronization point is needed—the initial wireless ear-to-ear audio synchronization. This contrasts with all other designs (using separate processors) that require synchronization both at the input wireless data stream and again at the output before stimulation. Traditional designs, therefore, do not generally have the capacity to transmit binaural stimulation with sufficient ITD accuracy (~10 µs). In addition, with this architecture only a single transcutaneous communication channel is needed thereby reducing power.
6.1.2 Wireless Synchronized Ear-to-Ear Audio Streaming
A wireless ear-to-ear synchronized digital audio streaming radio was developed for (1) high synchronization accuracy, (2) low power, (3) low latency, and (4) and high reliability. This technology called Nearlink is now available. Nearlink transmits accurately without gaps at a sampling rate of Fs = 16.67 kHz to 20 kHz using a standard audio codec (722.2). The system bandwidth was measured at 8 kHz, with a latency of 10.4ms.
6.1.3 Binaural Processor
A binaural processor was developed using a combination of hardware and PC software components (MHA). This processor received 4 synchronized microphone signals and produced a special binaural codec output for transmission to the implant. Binaural pre-processing algorithms discovered from previous work (WP5) were implemented on the processor. The whole software processing chain is shown in Figure 6.3..
A proprietary binaural codec was developed that treats the data stream as a binaural to begin with. These streamed packets encode relative ITDs (and ILDs) within the data packet itself, leading to an effective sub 10 µs ITD accuracy and resolution and allowing 100% consistency between ITD/ILD cues. Because the received packet is decoded by a single, binaural CI processor (running a single clock, e.g. 1 MHz), these data can be converted to stimulation timings with absolute precision. A second advance, involved transmitting TFS information asynchronously and without reference to any master clock was also introduced. The required ear-to-ear synchronization was achieved via the synchronized wireless microphone system and audio codec (mentioned above).
In order for the system to achieve a ‘true binaural’ processing status, the processing had to combine information from the two ears in super-additive way. What do we mean by this? In general, normal hearing listeners show performance advantages in difficult listening conditions of more than a factor of 2 ( > 3dB improvement in SRT) when listening through two ear rather than one. This additional advantage (beyond the expected additive advantage) is a ‘true binaural’ advantage. We set out to develop a system that would have the best chance of producing this effect for CI users who to date never show such a true binaural advantage. Although claims are sometimes made that super-additive results—called binaural squelch--have been observed, these results can be explained using normally additive ILD processing strategies (Backus, 2015)).
The binaural processor focused on two strategies to achieve ‘true-binaural’ super-additive processing:
(1) A binaural beam-former was developed for the BTE processor in order to take advantage of the synchronized bilaterally placed microphones and thereby improve input SNR.
(2) ITD cues (artificial or real) were integrated with ILD cues and a packaged in a binaural codec designed to transmit this information accurately to the CI user.
The two strategies differ in an important way. In the first, the binaural processing is done in the BTE processor (to increase the SNR of the output stimulation) while in the second, the binaural cues are passed to the brain and the binaural processing is done in the brain (to increase the brain’s internal SNR).
For the first strategy, a fixed MVDR binaural beam-former with common post filter was chosen. This choice was made based on results from WP5. A schematic of this beam-former architecture is shown in Figure 6.4. The performance of this solution was measured at ~5 dB SRT improvement. This algorithm was chosen over better performing adaptive ones due to user preference. A challenge remains to understand how to make an adaptive algorithm that adapts in a pleasing way. For example, in a car situation where the user wishes to listen to a passenger rather than the noise from the window but still be able to hear incoming road alerts.
For the second strategy ITD cues were preserved and integrated with ILD cues and transmitted to the implant for stimulation. The ITD cues chosen to transmit were based on basic knowledge of ITD sensitivity in CI users found in research. Primarily the following aspects:
(1) It is known that ITD sensitivity exists for CI users along the entire cochlea (or electrode array) and that the resolution of this cue varies with stimulation rate.
(2) At low stimulation rates the ITD cue is more salient and detection threshold for the best CI users is 25–50 µs corresponding roughly to 5 degrees in the acoustic fovea and 10 degrees laterally.
(3) The differentiable ITD range for CI users is extended beyond the natural range (750 µs) to double that (1.5 s).
(4) ITDs appear to not be additive across electrodes for sequential stimulation.
According to these ITD observations, the binaural processor was developed to allow variable rate stimulation (which could transmit temporal fine structure as other devices already do) and which would satisfy the low rates required for high ITD sensitivity (1 above). It was assumed that channels with naturally low rates (according to TFS extraction) will capture the ITD perception, but all channels were designed to deliver ITDs such that different ITDs could be presented at different frequency electrode pairs. In addition, the ITD range was extended to 1.5 s (beyond the normal physiological range) to take advantage of CI users observed extended range. Figure 6.5 shows the locations encoded by the binaural codec. A lookup table was implemented in the implant to map these positions to physical ITDs (µs) used during stimulation. This strategy was not tested on CI users, however, because no currently implanted CIs could support it. We don’t yet know if it will produce the ‘true binaural’ performance gains we seek or if it will be preferred.
6.1.4 Transcutaneous Data Transfer
The ABCIT binaural codec (WP3) was created to efficiently encode both ILD and ITD cues using a single data channel. The digital packet format contains enough accuracy and electrode data to encode most proposed binaural coding strategies. In this sense the codec is agnostic to the type of binaural processing chosen for the binaural processor. Because the chosen binaural CI architecture has only 1 data transmission channel which must serve both ears, the minimum speed and bandwidth requirements of this channel are 2x higher than for monaural devices sending the same type of information (320 kbps vs. 160 kbps). The transmission must have low error rates, be robust, and minimize the power consumption particularly for the implant’s receiver. For robustness an on-off keying (OOK) method was used, such that only a single level change needed to be detected. A digital communication channel with CRC checking was built around the OOK physical layer.
For the OOK digital transmission, the key specifications of (1) robustness/reliability, (2) energy/bit, and (3) data rate can all be calculated analytically and compared for different modulation schemes providing they are applied to the same channel with the same physical channel properties. These properties are fixed by the physical method of transfer (generally RF) and by the thickness of a given users skin. Theoretical calculations comparing various modulation types were done. The standard RZI (return to zero inverted), 4-PPM (4 position pulse modulation), and a novel EPM (edge position modulation) were compared. All modulation techniques demonstrated sufficient bit-rate to meet the 320 kbps required (based on our physical channel’s properties). The EPM modulation provided the best bandwidth efficiency and transmit-power efficiency, but required a higher frequency, a more accurate sample clock, and added additional decoding complexity (state machine with 1-internal state-vs.-straight decode). However, because EPM is based on edge detection, elongated pulses did not create errors as they do for 4-PPM as long as the elongations do no run into the next edge.
Because the transmit energy required may change (depending on the skin thickness variations) an automatic adaptive algorithm that receives return data (e.g. no response + CRC failure on reception) and then updates the modulation parameters by changing the transmit VOH voltage (or pulse duration) is possible. This technology is already in use. In this design (Figure 6.6) we separated the fast 320 kbps channel (Channel A) from an intermittent return channel (Channel B) because these channels had very different speed requirements. A variable transmit voltage block was used (similar to the one currently used in existing designs) to adapt the transmit levels.
A complete system was built and was able to transmit bursts of data at 1 Mbps using an average of 2mA for transmit and 1mA for receive. This was higher than expected and further optimizations may be needed to reduce this (see Figure 6.7).
6.1.5 Stimulation
A new binaural implant stimulation engine was developed using an FPGA to drive two monaural current stimulation chips. The chips each have 20 configurable current sources and are very flexible so can produce many different types of stimulation. We specified the binaural codec to address 15 binaural channels (15 electrodes on each side); for comparison, the current binaural implant addresses 12. Several common pulse types were implemented: (1) monopolar common-ground monophasic with passive charge recovery, (2) bi-polar (e.g. e+1) biphasic rectangular pulses with inverted charge recovery, (3) bipolar biphasic rectangular pulses with elongated charge recovery. In addition to these common pulse strategies, a novel ramped pulse was introduced based on work done from WP3. Ramped pulse have the potential to reduce the spread of excitation and thereby improve frequency resolution. However these ramps are very new stimuli and it is not yet known what the best ramp method might be—a topic of ongoing animal experiments. We chose to implement the conceptually ‘simplest’ approach outlined in Figure 6.9. This was planned using a lookup table within the implant so that this ‘simplest’ approach could easily be extended to more complex ramp shapes in the future.
6.1.6 Objective Measures
Objective measures can help in ‘pitch’ matching electrodes across two ears to improve binaural CI performance. Brainstem responses such as the Binaural Interaction component (BIC) investigated by UOL or the novel Interaural Phase Modulation Following Response (IPM-FRs) developed at UCL (see WP4) each shows it may be a better way to match electrodes across two ears than the current state of the art, subjective pitch matching (asking patients which electrodes sound similar in pitch). These methods require the ability to gather eABRs. In addition eABR have been shown to be as effective as eCAPs for determining C-levels.
To take advantage of these techniques, a prototype eABR amplifier circuit and averager was created. The amplifier was designed for low signal and low noise, with 4-kHz bandwidth, measuring electrically evoked auditory brainstem responses (eABRs) and cortical responses from within the implant. The system includes low power amplifier, a DSP-based artefact rejection and averaging engine, and built-in impedance measurement. To evaluate the system’s performance it was built and tested using external surface electrodes on normal adults. An investigation of the test amplifier’s internal noise (measured as en=17.4nV/√Hz, in=0.2pA/√Hz) was carried out as well as an investigation on real-world noise contributions from biological (Figure 6.10) power-line, and RF sources. Algorithms built to monitor and reduce as far as possible these noises were refined.
The whole system was able to capture standard eABR (shown below) and advanced cortical measures. Impedances were automatically measured and artefacts were rejected using a combination of reversing polarity, thresholding to remove clipping, and alternated interleaved trace comparisons.
6.2 Conclusions
The original goal of the ABCIT programme was to develop a prototype binaural device that might one day be implemented as a device used in patients. However, over the course of the programme of work, each of the elements of the presumed prototype have more rapidly found their way to utility in isolation or in combination, meaning that the whole ABCIT project has also been assessing aspects of a binaural device. WP6 has provided the means of bringing many of these elements together, for both research and future clinical application.
Potential Impact:
3.0 The potential impact (including the socio-economic impact and the wider societal implications of the project so far) and the main dissemination activities and exploitation of results
Considerable scientific and societal impact, including economic impact, is envisaged from the ABCIT project. Each of WP1-6 has specific benefits, and these are outlined below. In addition, the description of the overall benefit (including to European competitiveness through commercial success) were delivered as part of WP7, and are provided in some detail below.
3.1 Impact of WP1-WP6
The ABCIT WP1 research platform – which has been given the name ‘cHIRP’ and placed on the product roadmap of Neurelec - ultimately allows the wider distribution of this new tool to qualified CI researchers. This plan expands its use beyond the original consortium and was not originally planned. With its hardware and software parts cHIRP provides researchers pursuing research on cochlear implants the complete processing chain of a cochlear implant with novel capabilities. Importantly it enables real-time, synchronized, 4-channel (4-microphones) binaural CI systems to be evaluated using software (MHA). As any algorithm with real time capabilities can be implemented as a new plug-in into the Master Hearing Aid, it allows an arbitrary combination of different speech enhancement algorithms to be written in software only without the need of hardware effort. The electric stimulation patterns are also generated in the Master Hearing Aid and transferred to the research board, which performs the stimulation. This architecture enables researchers to reconfigure the chain easily according to their experimental needs. It fills an important gap in cochlear implant research by combining hardware and software within the same platform. This represents a significant advance for the CI research field and should propel investigations into binaural CI for years to come. Questions that have remained unanswered can be answered with this platform (e.g. ITD and ILD trading for CI users in real environments, probing strategies for auditory scene analysis and the effect of binaural stimulation on listening effort etc.).
WP4 assessed a critical factor in the future development of bilateral cochlear implants and their efficacy, objective measures and developed new software and hardware tools for recording brain activity to bilateral CI. In particular the first multichannel brainstem EEG for CI listeners was developed. This exceeded the initial expectations by allowing to record brainstem EEG, even from subjects for whom a facial nerve artifact was present – previously these patients could not be assessed in a binaural setup. A custom-made EEG cap with holes at the CI-coil positions was developed and presented to scientific audiences as well as to the general public in a regional newspaper. The artifact removal techniques including the cap were published in a peer-reviewed journal. Subsequently the brainstem EEG measurements were employed to objectively determine left-right electrode pairs for binaural fitting. Results indicate the shortcomings of existing subjective techniques and have the potential to revolutionize binaural CI fitting, in particular for the growing number of paediatric patients. With a related basic research study on objective binaural measures performed on normal-hearing subjects, WP4 was highly visible at the international Objective Measures Conference (Toronto 2014), the International Symposium of Hearing (Groningen, 2015), and the Conference on Implantable Auditory Prostheses (Tahoe City, California, 2015) with two contributions each.
The outcomes of WP5 will have real impact on bringing the benefits hearing-aid devices have enjoyed for some time into the CI domain. This was achieved by adapting state-of-the-art binaural pre-processing algorithms, as well as by developing novel ones, for CI devices. The evaluations of these algorithms for speech enhancement have shown that the tested algorithms improve speech intelligibility of CI users significantly, especially by means of spatial filtering (i.e. binaural beamforming algorithms). These evaluations were performed in several reverberant environments for a more reliable assessment of the algorithms in realistic listening situations. The algorithms were implemented as real-time plug-ins on the Master Hearing Aid (MHA), i.e. on the software part of the ABCIT WP1 research platform. The wider distribution of the ABCIT WP1 research platform offers many advantages to the qualified researchers all over the world. The real-time capability enables the researchers to evaluate the algorithms in daily life environments as well as in laboratory conditions using different types of stimuli e.g. online speech, recorded speech etc. Furthermore, using the research board, the algorithms can be tested in combination with different stimulation strategies. Because the MHA can also be hosted by other hardware platforms such as standard PCs, ARM-processor-based computers as well as iOS smartphones, the algorithms can be widely and easily applied in real-world environments.
The impact of WP6 has both direct and indirect impacts. Although the ABCIT WP6 binaural prototype was originally envisioned as a hardware precursor to an eventual binaural CI, the WP6 platform has now been expanded to include a new animal research platform that will be made available to qualified animal researchers, enabling testing of some of the ideas for improvements in binaural hearing included here, and others to come. These include ideas around incorporating eABR or other objective measures (WP4) for automated fitting, novel stimulation strategies including pulse shapes (ramps), TFS, ITD, and others can now be studied in animals. This should also speed the adoption of the most promising results.
Indirectly, the WP6 feasibility study has been successful in proving the capacity of the technology, but further research is required to quantify the benefits (both in humans using the WP1 ‘cHIRP’ platform and in animals using the WP6 platform) before an eventual binaural implant product based on this will be developed. What ABCIT has delivered to date indicates that this device could provide >6 dB of improvement in speech-reception thresholds (by using novel pre-processing algorithms such as binaural beamforming) and would cost about 40% less than purchasing separate processors to create a binaural system with the same type of performance.
3.2 Exploiting the outcomes – WP7
As part of WP7, the ABCIT project developed a ‘Market Exploitation Plan’ to aid in the commercialization of marketable findings and discoveries. In its original conception, it was thought that the principle product for leveraging findings of the research would be the Neurelec Binaural device. However, since the acquisition of Neurelec by Oticon Medical, access to new wireless technologies enabled by custom chipsets developed by Oticon, has provided an alternative path for bilateral device communication and synchronization that may be more commercially viable than a single implant powering two ears (i.e. binaural device). In terms of preservation of bilateral timing cues, a chipset driven by a crystal will allow synchronization between left and right devices with high timing accuracy. It is possible that providing such timing information to infants may lead to improved development of neural pathways capable of extracting salient timing information between ears, and this may improve outcomes when using two cochlear implants. Outcomes in post-lingually deafened adults where such neural pathways may still be usable may also improve.
At present more information is needed to determine if a binaural device should be pursued in parallel to a synchronized bilateral solution. Specifically, technology for high density feed-throughs (2x20 electrodes) and a single device with two stimulation engine chips demonstrated in the ABCIT project needs to be developed and matured further before feasibility of such a design can be fully assessed. Historically, increasing feed-through density tends to reduce device reliability. Even though concepts exist demonstrating the feasibility of this binaural approach, substantial R&D is still needed to make the binaural solution commercially more attractive compared to an Aurora synchronized bilateral solution. Specifically, feedback from KOL meetings from nearly 70 surgeons from around the world since autumn 2013 has highlighted the following technological challenges of the binaural option: a) reliability must be at least as good as with a unilateral device because, at a minimum, in the event of a device failure, two ears are put at risk rather than one, b) the device is not feasible in small children due to substantial growth of the head during development and 3) reimbursement for bilateral implantation is more and more becoming the norm, especially for children. These three observations effectively mean that the business rationale of delivering binaural devices as a reduced cost alternative to bilateral implantation is not evident in all geographic regions.
In terms of product development, therefore, Oticon Medical will leverage the hardware platform for extended research on aspects of synchronized stimulation on bilateral patients. Thus, as a first commercial path we will pursue the direction of a synchronized bilateral stimulation leveraging bilateral BTEs using a crystal-clocked chipset. It should be noted that the chipset has already been developed and the principle work needed is to adapt the electronics to drive a new BTE that supports a new internal CI with improved timing capabilities. The new internal CI is called the Neuro ZTi and Oticon Medical received CE mark for this new device in June 2015. It should also be noted that having patients implanted with the Neuro Zti is critical for being able to fully test the potential benefits of the findings of the ABCIT project. First implantations of the Neuro Zti are expected to begin the fourth quarter of 2015.
Market Exploitation Activities
The Neuro System will be seen as the first fruit of the co-operation resulting from the acquisition of Neurelec by Oticon Medical in 2013, and will include several specific aspects derived from the ABCIT project. As the first Oticon Medical CI system, this launch comes with several innovations:
1. Launch of the Neuro name as Oticon Medical’s cochlear implant system brand.
2. Launch of Neuro Zti – the industry’s most recent implant technology platform with a new compact and safe design
3. Launch of Neuro One – the first sound processor with full integration of Oticon HA and CI technologies. Taking the success of the XDP sound processing and the Inium platform technology one step further though automation of environment driven mapping of acoustic to electrical dynamic range.
4. Major upgrade of the DigiMap fitting software. Digimap 4.0 includes improved fitting flow, tested pre-sets, ECAP possibilities and full 64-bit support.
5. Perhaps most relevant to the present discussion, the Neuro Zti internal platform delivers a capable substrate to provide the precision neuro-stimulation necessary to test and eventually implement research findings regarding bilateral stimulation resulting from the ABCIT project.
The whole communication, service and support around the Neuro System will be built to reinforce our mission to become the world leader in bilateral, bimodal and binaural hearing solutions. The new Neuro Zti implant is innovative and brings a new standard to the market. The changes are very visible (new high impact resistant material, compact, thin) and the advantages tangible (new stimulation chip, programmable stimulation capabilities, eCAP measurements, MRI peace of mind, minimal invasive surgery, reliability). We have a very competitive implant, which will make us stand out. The new Neuro One sound processor will be an update of the design of the current Saphyr Neo collection. We will build on the success of the XDP sound processing; bring the first integrated HA and CI sound treatment that will be providing real advantages to the CI users in terms of speech focus, automatic adaptive listening and on many points WITH regard to feeling at ease with the processor. XDP processing and its automated version, Voice Guard, provide several advantages to bilateral CI recipients over traditional processing. Traditional designs use broadband AGCs which have the effect of nearly eliminating loudness differences between the two ears as well as introducing temporal distortion that could negatively impact interpretation of inter-aural timing differences. In contrast, XDP and Voice guard preserve these interaural differences without introducing temporal artifacts or cross-modulation frequency distortion. We believe that this processing will quickly put us in the forefront of bilateral hearing.
Going forward, we will advance two-eared hearing with the aim of becoming the world leader in binaural/bilateral solutions in CI by undertaking the following:
1) Continue to improve bilateral hearing outcomes with refinements to XDP and Voice Guard supported by findings from the ABCIT project
2) Incorporate the already developed Chip-set that has both low-power wireless and high-temporal precision capabilities into our next generation BTE to allow new possibilities in communication and synchronization across not only two cochlear implants but also for the increasing population of bi-modal users to enable optimization of hearing between a CI and a contralateral hearing aid
3) Continue to advance our research platforms to empower researchers to better enable us to exploit our processing power to improve patient outcomes
4) Continue to expand internal KOL meetings with the aim of facilitating studies with key clinics to validate and improve our product developments including those derived from the ABCIT project
5) Continue feasibility studies regarding a future binaural device leveraging an Aurora chip-set empowered external processor
In summary, Oticon Medical has now undertaken two parallel paths designed to produce products to exploit viable findings of the ABCIT project, these being 1) two properly synchronized bilateral devices and 2) a binaural solution with a single internal processor driving both ears. Given that advanced capability of a new crystal clocked chipset as an off-the-shelf solution when compared to the technical risks associated with the binaural solution, we have decided to focus the development priority on the former. Critical to success of the project, a more capable internal device is required. In June 2015 Oticon Medical received CE mark for the Neuro Zti, this is the new internal device that will provide the platform flexibility necessary to exploit the new crystal-clocked chipset. Future efforts will involve in-patient research with the Neuro Zti as well as the development of improved research tools critical for understanding how to optimize the new chip-set driven external processors to deliver bilateral information.
List of Websites:
http://www.ucl.ac.uk/abcit
