Final Report Summary - NANOBIOTOUCH (Nano-resolved multi-scale investigations of human tactile sensations and tissue engineered nanobiosensors)
Executive Summary:
A biomimetic approach has been taken to develop a robotic finger that is capable of sensing touch, able to explore surfaces with artificial curiosity, discriminate different textures and assess their tactile pleasantness. The development of a biomimetic approach required a greater understanding of the subjective and neurological mechanisms involved with discriminative and affective touch for both healthy human volunteers and patients. This included (a) the role of the peripheral receptor response using microneurography that allowed the spike trains from a mechanoreceptor in a finger pad to be recorded when subjected to a tactile stimulus, (b) brain imaging using electroencephalography (EEG) and functional magnetic resonace imaging (fMRI) in conjunction with microsimulation of mechanoreceptors in the finger pads, (c) microstimulation of single, identified tactile nerve afferents, and evaluation of the sensations reported by the participants in order to explore the discrimination of both the stimulation frequency from a series of target frequencies, and the variability of neural signals, and (d) psychophysical studies of the pleasantness of materials in order to develop a unidimensional scale and also studies of autistic subjects and stroke patients to evaluate their tactile abilities for diagnostic and therapy purposes. To carry out these studies, a novel dynamic platform for tactile stimulation and temperature control was developed in order to provide a controlled tactile stimulus. Multi-scale multi-physics modelling of the finger pad was developed that was capable of describing the deformation when in static or sliding contact with a surface. It was coupled to micro model of two of the tactile afferents (Meissener corpuscle and Merkell cell) so that the deformation caused spike trains to be emitted by a further coupling with a micro-mechanical model of ion channel opening. The spike trains were of similar form to those recorded by microneurography measurements. Arificial touch sensors were developed that were incorporated in a biomimetic artificial finger. By exploiting machine learning algorithms, it was possible to use the finger to identify textures, assess pleasantness and provide the ability for the finger to explore unknown surfaces. A bio-hybrid tactile sensor was also developed that was based on encapsulated cells. The sensor was able to detect both static and sliding stimuli. A number of applications of the technology developed in the project were examined that included enhancements to the Cutometer for measuring the mechanical properties of skin in vivo. The idea is to investigate the aging of skin during space flights in a joint project with the European Space Agency. Other applications included the use of the artificial finger in a prosthetic hand with direct neural coupling, microstimulation of a sensory nerve in an upper limb amputee to evoke the sense of touch, design of security cards and packaging and the evaluation of packaging surfaces and cosmetic creams.
Project Context and Objectives:
Tactile sensations reveal vital features of the environment of an organism. Human touch can be considered as a complex multi-scale interaction of biophysical and biochemical phenomena that transduce mechanical and thermal stimuli imposed by shape, texture, stiffness, temperature and movement of surrounding objects. The neural response is transmitted to the brain for recurrent sequence processing that influences our emotions and behaviour. Due to these high level impacts, investments into an improved understanding of human touch can bring many socio-economic, technological and scientific rewards. Tactile sensations can affect our well-being, support medical and psychiatric rehabilitation, assist blind people to gain access to books, maps and other documents and play a critical role in social interactions. Recent research has revealed that affective touch can yield valuable social benefits; the term affective touch relates to pleasantness and the preference of a particular tactile experience. Many industrial sectors would benefit from understanding and optimising the tactile perception of their products in order to achieve better consumer satisfaction. Furthermore, an understanding of mechano-transducting sub-systems can inspire many biomimetic engineering solutions related to sensor technologies, smart material composites and micro-tribological systems in addition to information processing. The human tactile sensation system illustrated above has been approached from many scientific and technological disciplines including medicine, biology, chemistry, physics, electronics and mechanics as well as numerical modelling and computer science, which are revealing the fascinating features of this biological mechano-transduction system.
On the above basis, NanoBioTouch was formulated to radically improve the understanding of the human mechanotransduction system in order to assist in the biomimetic design of tactile sensors incorporated in an artificial finger. This was achieved through systematic integration of new developments from converging scientific areas by involving academic and industrial participants who are experts in cognitive sciences, microneurography, brain imaging, cell biology and mechanics, tissue engineering, skin physics (tribology and mechanics), microengineering, multi-scale multi-physics modelling, information processing, robotics, prosthetics and medical rehabilitation. The project was built on existing discriminative touch research in order to understand affective touch mediated by the human fingerpad. Sensors capable of detecting directional force and temperature were to be developed as a combination of these modalities, which is critical to the affective component of the neurophysiological response evoked in taction. This next generation of sensors includes artificial sensor arrays and hybrid bio-artificial systems. A biorobotic finger with articulation was to be developed, which was controlled by neural network information processing that allowed artificial exploration of a surface to be achieved in ways that mimic human haptic behaviour and affective response. The required impact of the project included alleviating the effects of human tactile and visual disabilities, improving the quality of life, security printing, brand protection, smart packaging, space exploration and also the evaluation of products such as textiles and skin creams using the instrumented artificial finger. The consortium includes industrial participants to undertake specific technical innovation activities in order to maximise the commercial impact and exploitation of the research. The partners are listed in the table below:
Role Partner name
Coordinator University of Birmingham, UK
Industry & SME C3M, Slovenia
Rockfield Software Ltd., UK
CK Electronics, Germany
Optaglio, Czech Republic
Unilever R&D Port Sunlight, UK
Academic University of Gothenburg, Sweden
Universite Catholique de Louvain, Belgium
University of Ljubljana, Slovenia
University of Wales, Swansea, UK
Scuola Superiore Sant’Anna, Italy
Dalle Molle Institute for Artificial Intelligence, Switzerland
Objectives
The scientific aims of this project were to radically improve understanding of the human mechano-transduction system and develop an artificial finger incorporating a touch sensor. This was achieved by systematic integration of new developments from converging scientific areas (bio, nano, info and cogni) that were based on the following scientific objectives:
»Bio« To isolate and then grow mechanoreceptor co-cultures within a gel matrix. The production of such a structure was to allow the response of mechanoreceptors found in the skin to be monitored during the application of defined stress fields induced by normal and tangential contacts.
»Nano« To develop a new generation of artificial and nanobio tactile sensors for discriminative and affective touch. The nanobio sensor to involve the interfacing of cells and engineered mechanoreceptors with microelectrodes on the surface of a chip. The electrical signals generated by ion channel opening to be monitored and the resulting data to be employed in multi-scale multi-physics modelling of their response to applied mechanical stimuli.
»Cogni« To investigate the relationship between affective components of touch, such as pleasantness and preferences, and different discriminative touch modalities (e.g. roughness, temperature and softness) in order to provide data that can be employed to develop a model capable of predicting whether a material is pleasant to touch given the discriminative touch responses elicited from the artificial finger. To investigate pleasantness feeling in glabrous skin using the brain imaging techniques EEG and fMRI to analyze the physiological responses of single afferent units recorded using microneurography.
»Info« To develop a virtual multi-scale multi-physics model of mechano-transduction by bridging the length scales from a macroscopic fingertip contact to the deformation of mechanoreceptor cells with integral ion channels. In this way a previously-lacking computational framework for linking mechanical stimuli to neural response would be provided. To develop new artificial recurrent networks (ARNN) in order to process the neural sequences generated by the virtual model for multiple object, surface texture recognition and tactile pleasantness.
»Integration« To develop a multiphalangeal biorobotic (artificial) finger that is capable of tactile sensing and exploration. The sensing to be implemented by incorporating the nanoscale artificial silicon sensor arrays and bio-hybrid tactile sensors. The ARNN and multiscale model to be used to analyse the signals acquired from the sensors to generate discriminative and affective touch output (such as roughness, temperature and pleasantness). The ARNN to be implemented in the movement control software of the artificial finger so that it will be capable of human like exploration.
»Innovation« To use the new understanding of discriminative and affective touch for innovation of products for security printing, brand protection, smart packaging, personal care,
measurement equipment and robotics as well as to rehabilitate patients with neurological problems and amputees, and to investigate tactile phenomena in microgravity.
Project Results:
Cognitive investigations of tactile sensations
The main objective of this activity was to investigate mechanisms of tactile sensation in human healthy volunteers and patients, from the level of peripheral receptor response to cognitive and brain mechanisms. A particular objective was to uncover principles of higher-order or complex tactile experience such as pleasantness or preference. The work included (a) brain imaging studies that were carried out in conjunction with microstimulation of mechanoreceptors in the finger pads and (b) microneurography that allowed the spike trains from a mechanoreceptor in a finger pad to be recorded when it is subjected to a tactile stimulation. A dynamic platform for tactile stimulation and temperature control was developed for psychophysical and microneurographic measurements of the response of subjects to different tactile stimuli. In the latter case, it was necessary to design and construct a special mechatronic platform that would be suitable for orientating the hands of subject in way that measurements could be performed. Details of the thermal stage are shown in Fig. 1. A range of tactile surfaces were prepared for this stage including a set of aluminium surfaces with varying spatial roughness that were measured using profilometry.
Microneurographic recording of peripheral tactile receptor activity
The dynamic platform allowed the application of motion along the direction normal to the skin surface with a smooth and controlled application of forces. The surfaces used for stimulation were evaluated by psychophysical appraisal (see below) to range from surfaces “unpleasant” to “pleasant” to touch during active exploration. The different surfaces were tested with a range of normal loads and velocities, in order to explore issues of simultaneous mechanoreceptor coding of 1) force, 2) sliding velocity, and 3) normal force.
The results showed that surface texture is encoded by a modulation of the activity of both RA and SAI afferents. The RA units were strongly sensitive to the structural roughness of the surfaces, with higher firing rate for larger spatial periods in the tested textiles. Interestingly, RA firing was only broadly frequency modulated by the presence of spatial periodicities in the surfaces, instead showing a weak and broad modulation of firing at rather high frequency (150-250 Hz). Firing rates were lower and frequency modulation lacking for smooth surfaces, both the higher-friction surfaces deemed the most “unpleasant” in the sample, and for the very smooth paper surfaces that were considered “pleasant”. SAI activity broadly resembled RA, with strongest afferent activity for surfaces with large spatial period (“rough” textile surfaces). However, the SAI sample was smaller, and some SAI units showed a strong tendency to code also the tangential friction of the surfaces, which is interesting since this factor is a strong determinant for the perceived pleasantness of the surfaces. PC activity in a smaller sample was similar to RA. Overall, the results showed a complex but very interesting pattern of combined coding of all the different aspects of mechanical stimulation of skin during natural tactile events, and these results obtained for real-life textured surfaces (such as textiles) could not readily be predicted from previous experiments on human tactile receptor response during stimulation with more strictly controlled conditions, such as stimulation with periodic gratings or Braille type dot patterns.
Fig. 1. A) Example of one of the temperature stimulation platforms; B) thermographic IR-imaging of the tactile stimulus; C) 3D visualization of the temperature along the thermal stimuli plate; D) linear temperature curve along the thermal stimulation plate; E) examples of stimuli.
Microstimulation
This involves microstimulation of single, identified tactile nerve afferents, and evaluation of the sensations reported by the participants. The following issues explored included: (a) discrimination of stimulation frequency from a series of target frequencies, and (b) discrimination of variability of neural signal, measured as coefficient of variation while average frequency is kept constant. The results showed that graded tactile percepts can be encoded in the activity of single nerve fibres. Variations in patterns were also further tested, where it was demonstrated that small pauses in ongoing activity can be perceived down to the level of single missing impulses during an ongoing 60 Hz stimulation strain, although psychophysical testing of thresholds for resolving differences in patterns was not tested extensively.
Psychophysical studies of taction
The relationship between tactile pleasantness and four other tactile dimensions (cold/warm, hardness, stickiness and roughness) was investigated and it was found that roughness, stickiness and hardness explained 30%, 14% and 5% of the variance of tactile pleasantness respectively. When lateral movement between the fingertip and the surface was prevented (during static touch), the tactile pleasantness of some materials changed significantly. However, the rating along other tactile dimensions also changed significantly for some materials. Analysis showed that when variances of ratings along other tactile dimensions were taken into consideration, movement condition did not have any effect on variance of pleasantness rating. Therefore, the perception of pleasantness in glabrous skin is probably mediated through the perception of other tactile dimensions. However, contact force was significantly different for the movement conditions. This was overcome by a passive touch study. Participants were asked to rate the pleasantness of materials while being touched using the dynamic touch platform. Contact force and velocity were controlled. There were two contact force condition (0.5 N vs 2.5 N) and two velocity condition (static and 50mm/s). There were no significant effects on rating by force and movement condition. However, the variance of the rating was significantly smaller in static touch and therefore it was concluded that lateral movement is important in the perception of tactile pleasantness. An analysis of affective touch was carried out that allowed pleasantness to be represented on a unidimensional scale. To elaborate this Pleasant Touch Scale, ordinal pleasantness scores of 37 everyday life materials (sandpaper, paper, wood, fabrics, wax, latex,…) have logarithmically been transformed into linear and unidimensional measures (using the Rasch model). The scale has been calibrated on 200 subjects (Fig. 2). The results of this study indicated that materials being rougher or stickier were perceived as more unpleasant than those being smoother. Furthermore, the participants’ fingertip moisture levels had an effect on the surfaces linear pleasantness levels, suggesting the dynamic coefficient of friction influence pleasantness perception. This is a major breakthrough in being able to rank and select surfaces in this way.
The effects of regularity of frequency on pleasantness were investigated by participants rating the roughness of periodic gratings and aperiodic (pink noise) gratings. The median frequency and mean height of the gratings did not differ significantly with grating type. Pink noise gratings were perceived to be significantly rougher whereas no significant difference in pleasantness could be found after rms height and frequency of the gratings were taken into consideration. In another study, it was found that the subjects perceived surfaces to be more unpleasant when the sliding velocity in active touch was increased.
Patient studies
Autistic participants rated gratings (regular or irregular) rougher than control participants although there was no significant difference in pleasantness rating between groups. British pictorial verbal scale (BPVS), sensory profile, grating orientation threshold, monofilament threshold, fine and coarse grating thresholds and tactile rating of a range of materials and gratings were also measured. There was a higher monofilament threshold in autistic participants and the hypotactile group (as defined using the sensory profile questionnaire) in particular had a significantly higher monofilament thresholds. Therefore, there was psychophysics evidence of tactile sensory difficulties in autism.
Stroke patients with similar parietal lesions were tested for their ability to process bilateral tactile stimuli. In 4 of the 5 patients there was a clear pattern of tactile extinction. Responses to unilateral stimuli were fast and accurate. However, responses to contralesional stimuli were slowed or absent in the bilateral condition. They also experienced difficulty detecting the offset of a contralesional vibrotactile stimulus when it occurred in the presence of an ipsilesional stimulus. Interestingly, when feeling textures with the two hands simultaneously, the 4 patients’ judgments about a surface touched by the ipsilesional hand were heavily biased towards the roughness of surfaces touched by the contralesional hand. The results of the patients showing classical signs of tactile extinction suggested that information from the contralsional hand is not entirely lost under bilateral conditions in these patients.
Fig. 2. Illustration of the Pleasant Touch Scale. Upper part: Distribution of the participant sample. Black bars represent subjects having a low fingertip moisture level and white bars represent participants having a high fingertip moisture level. Middle part: Each line represents the most probable response of one participant to the different materials. Materials are ordered from the least pleasant (top) to the most pleasant (bottom). Bottom part: Illustration of the relationship between the ordinal pleasantness scores and the linear unidimensional pleasantness measures.
Imaging of brain activity
Experiments involving stimulation of single afferents during functional magnetic resonance imaging (fMRI) of brain activity were undertaken. This involved very extensive technical challenges in setting up the microneurography and microstimulation techniques in the very strong magnetic field environment of a 7T MR scanner. Sets of afferent units were successfully stimulated during scanning of brain activity. It was observed that, first, single afferent stimulation can result in detectable activation of S1 cortex, showing early stages of processing of tactile input from the finger tips with high temporal and spatial resolution. The results from microstimulation were compared to tactile stimulation of the finger tips, and of the receptive fields where the sensations were elicited during microstimulation. The results show a strong overlap, corroborating the interpretation that the observed brain activations resulted from stimulation of single afferents with sensations projected from almost punctate areas on the hands and fingers. Further analyses revealed gradually increased fMRI activations with increased stimulus intensity in terms of stimulation frequency. This strongly suggests that the observed activations resulted from bottom-up processing related to the activation of inputs in single afferents, although top-down processes such as the subjects’ attention to the ongoing stimulation and the tactile sensations presumably are important.
Single afferent microstimulation in conjunction with recording of high-resolution (64-channel) EEG were also performed. The results showed that single afferent stimulation can result in modulation of brain activity both in the time domain as evoked potentials that were time-locked to the start of the stimuli, but also clearly in the frequency domain, where stimulation resulted in enhancement or depression of ongoing rhythmic brain activity. The latter has been previously demonstrated for tactile stimulation to the fingers, such as vibration, and the patterns for single afferent stimulation resemble these previous results, with depression of both alpha and beta oscillations during stimulation, as well as effects in the theta and possibly also the higher gamma frequency bands. Similar to the fMRI results, variations in stimulation intensity resulted in graded amplitudes of the time- (evoked potentials) and frequency-domain (modulation of oscillations) measures of cortical activity.
Nano-resolved virtual taction model
Multi-scale model of the finger pad
The modelling of the finger pad at the smallest length scale provided information about the mechanosensitivity of the afferents. Models were developed to predict the rate of opening of ion channels that evoke the action potentials forming a spike train. The focus of the work was the Meissner corpuscle and Merkell cell, which are two of the four types of afferent present in the finger pad. Finite element models of these afferents were developed (Fig. 3)
Fig. 3. (A) Meissner corpuscle with five segments of the neurite used for extraction of mechanical deformations. (B) Merkel cell – polysynaptic complex.
The 3D model had two stages: the first stage is a micro-anatomic model containing a slab of epidermis, a slab of dermis, one Meissner corpuscle and three Merkel cells (Fig. 4A); the second stage is a macro model of a fingertip and a grating (Fig. 4B). For the micro-anatomic model, a Meissner corpuscle is in the dermis, among four epidermal papillae. The three Merkel cells are embedded into one of the epidermal papillae just above lamina densa. Their postsynaptic elements are in the dermis below the lamina densa. One Merkel cell is situated in the bottom of the papilla and two are positioned on the side of the papilla.
Virtual experiments of the multi-scale model were carried out by sliding the finger pad over a range of gratings as illustrated in Fig. 5. The micro-anatomic model allowed spike trains to be computed on the basis of the mechanical deformation of the afferents (Fig. 6). They were similar in form to those measured by microneurography experiments. The various models were integrated into a framework of three distinctive blocks i.e. steering, analysis and optimization (Fig. 7). The function of the steering application is to control the whole process depending on the task the framework is performing.
Fig. 4. Finite element models of (A) Meissner corpuscle, Merkel cells and lamina densa; Meissner corpuscle (blue) is nested among four epidermal papillae and three Merkel cells (red) with their postsynaptic element (orange) are embedded into one of the papillae. The transparent mesh represents the lamina densa which divides the epidermis (above lamina densa) and the dermis (below lamina densa), and (B) distribution of effective stress during a tactile contact of a finger pad.
Fig. 5. Examples of the multi-scale finite element model in contact with a fine (left) and coarse (right) grating.
Fig. 6. Examples of spike trains generated from the multi-scale model.
Fig. 7. Structure of the computational framework.
The model has been validated against different in vivo measurements on the finger pad including sliding, indentation and suction. It has been applied to a number of applications including numerical predictions of microneurography data, input data for information processing of discriminative and affective touch, and assisting in the design and optimisation of the artificial and bio-artificial tactile sensors.
Development of an artificial finger
Artificial tactile sensor
An artificial tactile sensor was developed that incorporated a temperature sensor and an array of force sensors based on capacitance measurements to detect the bending of nano-beams (Fig. 8A). A CAD design was developed for incorporating the sensor in a robotic finger (Fig. 8B). The sensor was coated with silicone elastomer to provide protection against damage and artificial finger prints.
Fig. 8. (A) SEM of NEMS tactile sensor showing force and temperature sensors. (B) CAD drawing for incorporating the sensor in the robotic finger.
A biomimetic robotic finger (Fig. 9) was developed that is human‐sized, tendon‐driven and underactuated, i.e. with more degrees of freedom (DoFs) than actuators. This property reduced the design complexity and allowed self‐adaptation and anthropomorphic movements similar to human exploratory tasks. The finger has three DoFs (as flexion/extension DoFs of the human finger) and two DC‐motor actuators. One motor actuates the flexion/extension of the metacarpophalangeal (MCP) joint by means of two lead screw pairs driven by the same motor, in order to achieve the active flexion and extension of the joint. The other is for the underactuated flexion of the proximal interphalangeal (PIP) and distal interphalangeal (DIP) coupled joints, while the extension is achieved by means of torsional springs housed in the joints.
Fig. 9. Pictures of the experimental setup. (a) Tactile platform with (1) the robotic finger, (2) actuator modules, (3) sensor processing facilities and (4) housing for replaceable surface blocks.
Bio-artificial tactile sensor
Numerical parametric studies of the biosensor, using the nano-resolved virtual taction model described above, were carried out to evaluate the performance and optimise the design. It confirmed that in principle, cells with a mechanosensory capability can be used in a biosensor. It was found that: (1) deformation of cells will be relatively greatest when the Poisson ratio is small, suggesting using a stiff alginate, (2) electrical changes were small at low volume fractions but became significant at higher cell volume fractions, (3) deformation could be best detected when the conductivity of the extracellular solution would be as small as possible; therefore the sensitivity could be improved if the original ionic content of the extra-cellular matrix was reduced, (4) the sensitivity of the cells would increase with an increased density of mechanosensory SA channels, or when the membrane-tension-dependent SA channel activation would be optimized (slope, offset of the Boltzmann sigmoid); this could be achieved using genetic tools, (5) the sensitivity of the system would increase if the cells included an intrinsic gain mechanism, e.g. voltage-gated channels, (perhaps a system of classical Hodgin-Huxley sodium and potassium channels; This could be achieved using genetic tools. Although the parameters used in the study were approximate, it seemed that a fibroblast tissue in the absence of intrinsic gain-boost mechanism in the cells (e.g. voltage-sensitive channels) is not exquisitely sensitive to mechanical stimulation.
As a first stage of the development of the bio-hybrid tactile sensor, the lifetime of cells on a microfluidic platform were investigated. Alginate hydrogels were prepared and encapsulated with 3T3 fibroblasts, the encapsulated hydrogels were integrated on a microfluidic device for 3 days with the constant supply of Supplemented- Dulbecco’s Modified Eagle’s Medium (S-DMEM). For comparative reasons, non-encapsulated hydrogels were integrated in the device and hydrogels that were not connected to the device were incubated as well. The cells viability, the alginate hydrogel dehydration, the encapsulated with cells gels and the evaluation of the microfluidic device were examined. The microfluidic system consists of micro channels etched into a silicon substrate designed to supply the cells with nutrients and therefore be capable of sustaining their long-term viability. The channels lead into a chamber with a regular array of 50 μm diameter micro-pillars, which allow flow and diffusion of the media for cell culture throughout the chamber whilst supporting a polycarbonate nanoporous membrane. The membrane is 100 μm thick with an irregular array of 200 nm diameter holes etched through. This membrane forms a layer upon which the cells or tissues can be cultured whilst allowing media to diffuse through to the cells, assisted by capillary effects. In essence this design emulates the epidermis layer in human skin which contains no blood vessels and cells are nourished by diffusion from capillaries below. On the membrane three local conductivity sensors were deposited composed of two electrodes with a 100 μm gap. The microfluidic channels were sealed and the cells are confined to a well using a thick 500 μm silicon cover plate. The device is shown in Fig. 11 except for the silicon cover plate.
The device was fabricated in three separate layers and then assembled. Initially, a 500 μm thick silicon wafer was patterned with SPR220-7 and etched twice to define the microfluidics and silicon pillars. Next a shadow mask was fabricated by patterning and through-etching another silicon wafer. The electrodes of the conductivity sensor were formed by thermally evaporating 20 nm Cr and 200 nm Au directly onto the membrane through the shadow mask. Another 500 μm thick silicon wafer was patterned and etched twice to form the cover plate; the first time to define a well to house the membrane and again to expose the tissue. The microfluidics, the nanoporous membrane and cover plate were then assembled. The medium is fed through the microfluidic channels via right-angled adapters integrated with Luer lock connectors (Fig. 11). The syringes are connected to these connectors by gauge 20 PTFE tubing with an internal diameter of 800 μm.
After the gel formation, the cellular and a-cellular hydrogels were washed with PBS. Then, cellular and a-cellular gels were integrated onto the microfluidic system. They were continuously fed with S-DMED with a rate of 0.8 ml/hr for 3 days. As control samples cellular and acellular hydrogels were placed in 6-well plates and were half immersed in S-DMED. In order to examine the cellular viability 3 days post encapsulation, sections of 1mm thickness were taken from the centre of alginate discs containing 3T3’s using a sterilized blade. The sections were immersed in 0.2 μl calceinacetoxymethylester (calcein-AM) for 15 min and 2.5 μl propidium iodide (PI) for 5 min in supplemented DMEM at 37C. The calcein-AM was cleaved to form calcein in the presence of esterases in live cells resulting in green fluorescence. In the case of dead-cells, the PI penetrated the cell and nuclear membrane and intercalated the DNA resulting in a red fluorescence. All samples were visualized using fluorescence microscopy.
Following this initial study the form of the microfluidic platform was modified to incorporate a PDMS cover layer that sealed the microfluidic channels and confined the cells to a well defined in the PDMS cover plate; resulting in a bio-hybrid tactile sensor. An exploded schematic diagram of the bio-artificial construct is shown in full in Fig. 10. It comprises a first layer based on a conductance sensor; a second layer composed of alginate gel in which cells are grown; a third PDMS layer that acts as covering and sealing of the structure and as interface with the external world. The cells are to be cultured and encapsulated within 5% alginate, a hydrogel with mechanical properties comparable to the dermal layer of human skin. The system of micro channels etched into the silicon substrate (described above) are designed to supply the cells with nutrients and maintain their long-term viability (Fig. 11). The PDMS effectively becomes the epidermis layer, possibly completed with finger print ridges, providing a robust surface to conduct experiments, while transmitting stresses and protecting the encapsulated alginate-fibroblast tissue. The PDMS also serves as a seal containing any excess fluid.
Fig. 10. Exploded schematic diagram of the bio-hybrid tactile sensor.
Fig. 11. Cross-section of the bio-hybrid tactile sensor.
A final iteration of the bio-hybrid sensor replaces the three conductivity sensors with an array of electrodes that provide improved spatial resolution of the tactile stimulus. This impedance sensor array is a ten channel linear array of electrode pairs interspersed with ground lines to minimise cross talk between the sensors. The electrodes consisted of tin plated copper conductors insulated between polyester based tape. Each electrode was 0.3 mm wide with a pitch of 0.5 mm. The array was fabricated by means of chemical processes applied to a 210mm flat flexible cable (FFC): a section of the FFC insulation layers was etched using a methyl trichloride/phenol solution at a ratio of 2:3 by weight at 110°C. This was done to expose a section of the conductors, nominally 18 mm long. This section was placed onto the nanoporous membrane in direct contact with the tissue. The end of the FFC is connected to the interface circuitry. One electrode of the sensing pair is connected to the input signal via a high power buffer amplifier to ensure the voltage remains constant should the resistance between the electrodes drop significantly. The input voltage was maintained as an AC signal with a frequency of 250 Hz and amplitude of 1 V. The other electrode is connected to the negative terminal of a transimpedance amplifier. This converts the current travelling between the electrode pairs into a voltage with a gain factor proportional (with negated sign) to the resistance value in the transimpedance circuit, in this case -10 kΩ. The output of all the transimpedance amplifiers were streamed directly into a DAQ and compared to the input signal in order to calculate the phase difference.
A custom sensor box was designed to hold the sensor and PDMS cover layer during tactile stimulation measurements. The silicone elastomer (PDMS) surface layer, which acts as a protective layer, was designed to cover the upper external side of the box containing the conductivity/impedance sensor. The PDMS cover is elastic, soft and gas permeable, thus allows cells’ respiration; moreover, the PDMS compatibility with cells was already proven. Different shapes and dimensions were designed for the PDMS cover: a simply flat layer and prototypes with ridges (Fig. 12).
Fig.12. (Left) PDMS cover biosensor: A. flat design; B. design with ridges; C. PDMS prototype; D. particular of the ridges geometry. (Right) Test assembly for biosensor.
The biosensor was integrated and tested with the tactile stimulation platform, which was also used for human touch studies (via electrophysiological and psychophysical methods). The platform (Fig. 13) was developed as a 2 DoF Cartesian manipulator, to indent and slide stimuli against the biosensor.
In order to test the biosensor, the complete system (upside-down) takes the place of the finger in human tests. The stimuli are attached to the bottom part of the dynamic platform, above the actuator, in order to be moved vertically along the indentation axis and translated horizontally along the tangential direction. Thus, the biosensor receives the same stimulation profiles that are applied to the human finger during electrophysiological and psychophysical passive-touch studies while using the platform and machine learning methods. During such dynamic passive-touch experiments the sensor is interfaced to the readout electronics in order to record impedance data during the test.
Fig. 13. Biosensor integrated in dynamic platform.
A typical data set is shown in Fig. 14, which was generated by sliding a cylindrical probe across the sensing area, retracting the stimulus and return to start location, and finally repeating the test. The two indentation phases are clearly visible in the figure. This behaviour was registered for all the different stimuli (cylinder and hemisphere) during the indentation tests. The data demonstrate that the biosensor clearly responds to the stimulus.
Fig. 14. Biosensor output data of sliding test for 2.5 mm single cylinder stimulus.
Simpler experiments were also carried out involving the indentation depth of a spherical probe, where the amplitude of the output signal was shown to increase according to the stimulation. This increase continues until the indentation depth is kept constant when it is observed that the signal appears to relax at an exponential rate for a time scale of several minutes. When the sphere is retracted and immediately indented to the previous depth, the signal is observed to decrease rapidly and then return to its former value at a rate consistent with the sphere speed. It is also observed that the phase of the signal appears to remain constant throughout the experiment. The increase in signal magnitude during indentation appears to increase with the sphere radius.
Information processing
The main objectives of information processing were to use machine learning techniques to analyse data produced from physiological and psychophysical measurements and virtual data from the finite element model of the finger pad. The aim was to increase the understanding of the causal and correlative relationship between different neural signals and behaviour metrics. In addition, the aim was to exploit such insights into the mechanisms underlying the encoding of sensory information to produce a believable and efficient active learning system for an artificial robotic finger.
In order to evaluate different information processing methods, data from the artifical tactile sensor were analysed. All methods were quite successful in classifying textiles but the weighted k‐nearest neighbor (kNN) method was more successful for gratings (Fig. 15)
Fig. 15. Results of several classification algorithms for passive touch with the artificial sensor on textile (top) and grating (bottom) datasets. The data provided to the classification algorithms consisted of the location of the highest peak in frequency spectrum obtained from a Fourier transform of the sensor measurements.
Using the artificial finger, active exploration driven by artificial curiosity was developed. The ultimate goal of the active robotic finger platform is to learn to infer the properties of the touched surfaces, for example, classify different materials. However, it is intractable to learn behaviour for finger movement directly from teacher signals provided for correctly identified surface properties. Instead, an intermediate learning stage is used during which a reinforcement learning (RL) algorithm learns to move the finger such that the sensors generate compressible signals. Compressing sensor signals implies finding regularities that allow for storing the signals in shorter form than the raw data. These regularities can arise from moving the sensor over a surface with corresponding regularities, for example regular gratings or textiles. Regular signals were considered that contained repetitive increases and decreases in voltage (e.g. a sine wave). Such signals will produce frequency spectra with clearly identifiable peaks. A significant part of the raw sensor data could be represented by the height and location of the peaks in frequency domain, which is requires much less storage than the original signal, and hence, the signal is compressible (e.g. Fig. 16). In contrast, random signals arising from sensor noise generated in the absence of sliding sensor-surface contact cannot be easily compressed. The reinforcement learning module is driven to generate motor command that lead to further learning progress of the compressor, allowing for quick learning of the relevant movements for detecting surface properties from the sensor data.
Fig. 16. From top to bottom: detrended sensor readings; frequency spectrum; regularity measure; Morlet-based wavelet transform of sensor readings, during a finger movement with sliding sensor-surface contact. Results are indicated for all sensors (from left to right) with four channels per sensor.
Integrating the machine learning methods with the drivers for the robotic platform was successful in initial experiments. The robotic finger learned how to move in order to pick up the most relevant (compressible) information from its sensor (Fig. 17).
Fig. 17. States of the robotic finger in the reinforcement learning experiments: (a) ‘extend’, (b) ‘flex’, (c) ‘extend touch’ and (d) ‘flex touch’. Only transitions between ‘extend touch’ and ‘flex touch’ and vice versa involve sensor-surface contact. The RL algorithm learns to move the finger from ‘extend touch’ to ‘flex touch’, involving the longest sensor-surface contact.
Applications
Microgravity
SKIN-B project is an ESA supported action joining the efforts of two research groups both targeting changes of human skin tissue during long term space mission. The project consist of two parts:
• Validation of skin physiological changes in space and skin as a model for other body systems.
• In vivo biomechanical measurements of human skin properties under accelerated aging conditions during ISS mission.
The first part is lead by Prof. Ulrike Heinrich (DermaTronnier GmbH & Co.KG and University of Witten-Herdecke) while the second part is the responsibly of the current project. There are two objectives to this combined study i.e. increasing the scientific knowledge of skin physiology and skin “aging” process and use those parameters in a skin mathematical model. The list of measurements include:
• Hydration-measurements
• Measuring the TEWL - transepidermal water loss
• SELS - Surface evaluation of living skin
• Microcirculation
• Ultrasound
• Pseudo biopsies (multiphoton tomography)
• Skin Elasticity
The main objective for the current project is to study the influences of the space environment on human skin by measuring changes in viscoelastic properties of the skin in-vivo before and after the ISS mission. The non-invasive experiment using the Cutometer device will be performed at different body sites (fingertips and volar side of the forearm) of selected astronauts before and after the ISS mission. The force-displacement versus time measurements of the epidermal layer of the skin will be recorded.
The Skin-B experiment will help to develop a mathematical model of aging skin and improve understanding of skin-aging mechanisms, which are accelerated in weightlessness. It will also provide a model for the adaptive processes for other tissues in the body. The project has started in 2013 and the first measurements were performed as presented in Fig. 18.
Fig. 18. Astronaut Luca Parmitano is performing SKIN-B measurements .
The Cutometer device has been extensively used also during the NanoBioTouch project. The experimental technique has been improved in order to include the suction tactile stimuli in microneurography experiments. The device has been modified to include synchronization.
The numerical model of suction test developed within the project simulates the Cutometer experiment. Although the complexity of the full 3D micro-macro model exceeds the requirements of the SKIN-B measurement protocol it can be used for evaluation of SKIN-B experimental results by utilizing the 3D macro-model part. The SKIN-B project experimental program has started and pre-flight and some on-board ISS measurements were performed.
Microstimulation of a sensory nerve in an upper limb amputated patient.
The possibility of evoking a sense of touch in the missing limb of a patient with an amputated upper limb was investigated even though at the time of testing the patient was 40 years old. As a result of a tumour growth in his right forearm, a trans-humeral amputation was performed in 2003. In early 2013, the patient went through surgery where electrodes were chronically implanted directly onto the muscles of the upper arm (epimysial electrodes) allowing the amputated patient to control the movements of the prosthetic limb. In the same surgical procedure, a cuff electrode was chronically attached around the ulnar nerve in the upper right arm (Fig. 19). The cuff electrode has three stimulation sites with a surface area of 1 mm2. From a knowledge of microstimulation during microneurography, the same technique in stimulating parts of the ulnar nerve through the cuff electrode was investigated. It was found that the stimulation of the ulnar nerve gave rise to similar sensations as experimental microstimulation, as during microneurographical microstimulation using intra-neural electrodes in healthy subjects. At lower stimulating currents, some of the sensations generated in the illusory/phantom hand of the patient were very similar to what was found when stimulating single mechanoreceptive afferents in the hand. For example, the patient could clearly indicate a sensation that was well localized to a small dot on the missing tip of a finger, and he had no problem indicating the precise location on a drawing of a hand. The sensation strictly coincided with the duration of the train of stimulation impulses delivered to the electrode. The current intensity at the threshold for this sensation was just above that normally used with intraneural electrodes. The character of the sensations resembled sensation commonly elicited when stimulating a single unit identified as an SAI (Merkel) or RA (Meissner) type afferent during microneurography. Increasing the stimulus intensity resulted in a spatial elongation of the sensation, as expected when more afferent nerve fibres are recruited.
Fig. 19. A close-up of the patient’s right arm during experiments with microstimulation through an implanted nerve cuff electrode. Attached to the upper limb is the preamplifier and shielded connectors for the signals to and from an integrated amplifier and isolated stimulator for human microneurography (not shown). The signals from the microneurography unit are connected to the implanted electrodes through a connector that is integrated in the osseointegrated titanium screw that can be seen to protrude from the end of the amputated upper limb. The scar on the lateral side of the patients’ arm is from the surgical procedure for implantation of epimysial (muscle) and nerve cuff electrodes, which was performed one month prior to the microstimulation experiment.
Bidirectional hand prostheses
Nanobiotouch also contributed to the advancements towards the restoration of natural sensory feedback in real-time bidirectional hand prostheses (Fig 20). The results were published by “Science Translational Medicine” journal, with the paper entitled “Restoring Natural Sensory Feedback in Real-Time Bidirectional Hand Prostheses”. The study was carried out in Rome and involved various members of the Nanobiotouch team of The BioRobotics Institute of Scuola Superiore Sant’Anna, Pisa, Italy. The paper describing the work had a high impact on media worldwide, and it was the result of a synergy between various projects and funding bodies:
- the EU Grant CP-FP-INFSO 224012 (TIME project);
- the project NEMESIS (NEurocontrolled MEchatronic hand prostheSIS) funded by the Italian Ministry of Health;
- the EU Grant FP7-611687 NEBIAS (NEurocontrolled BIdirectional Artificial upper limb and hand prosthesiS);
- the EU Grant FP7-NMP 228844 NANOBIOTOUCH (Nanoresolved multi-scan investigations of human tactile sensations and tissue engineered nanobiosensors);
- the Swiss National Science Foundation through the National Centre of Competence in Research Robotics.
Fig. 20. Bidirectional prosthetic hand during experimental evaluation in the LifeHand2 project, that involved NanoBioTouch researchers.
Biophysical and cognitive tests performed on packaging surfaces, textiles and cosmetic creams
The underlying hypothesis in this study was that the sensation of pleasantness by the human fingertips is associated with particular features in the temporal and spatial patterns of force (and temperature) produced during touching and rubbing. Hence the data from an artificial finger that can record such patterns should be analyzable in such a way as to predict pleasantness as assessed by sliding against a range of packaging surfaces, textiles and cosmetic creams. The artificial sensor response to different textures known for eliciting different levels of (un)pleasantness were analysed and relevant and meaningful differences in the temporal and spatial data were identified. In the temporal data, differences in the stick-slip behaviour between materials were found and in the mean amplitude and fluctuations in the data that relate to friction. The data from each sensor were also transformed into a neuromorphic code, i.e. spikes that emulate the coding mechanisms that are observed in human mechanoreceptors (Fig. 21). This allowed the data to be analysed using the information processing techniques developed in the project.
Fig. 21. Each plot represents the average response across trials in each channel to different textures (indicated by different colours, see legend) for a given speed (indicated by the title of the plot expressed in mm/s). Error bars indicate the whole range of the response across trials.
In the spatial data, the spike trains generated by the different sensors were examined. The first attempts at decoding the texture identity were hypothesis-free, i.e. testing if there were some features in the responses able to discriminate between different realistic textures like those under investigation. Note that most studies of this kind involve regular geometrical patterns so the current study was already breaking new ground. Reliable discrimination of the commercial textures was achieved. On this basis, it was possible to identify more advanced measures of discrimination for future work.
Potential Impact:
Innovative S&T research
Tissue engineered mechanoreceptor cells have been isolated, stimulated and interfaced to measure the neural responses, which represents a significant advance for nanobiosensors and also for studies of the molecular basis of mechanotransduction at the nano-scale. This understanding has been underpinned by the development of micro-anatomic mathematical models of mechanoreceptor afferents that allow the relationship between their deformation behaviour during tactile contact and the evoked action potentials forming a spike train. The deformation characteristics of the afferents were computed by developing a micro finite element model that was coupled to a macro finite element model of a finger pad. The spike trains were similar in form to those measured directly using microneurography. These latter measurements showed that surface texture is strongly coded by modulation of the activity of both RA and SAI afferents. Further understanding of the human tactile system was derived by brain imaging that strongly suggested that it is a bottom up process related to the activation of input from single afferents. A novel unidimensional scale has been developed for ranking the pleasantness of surfaces. The friction of the finger pad is a key factor and significant advances have been made in understanding the tribological mechanisms involved. Novel machine learning algorithms have allowed information, which was obtained from the micro-macro model, microneurography and the artificial finger that was developed, to be used to classify surfaces and assess their pleasantness. Thus the work has made major progress in defining and understanding affective touch.
Innovations in European industry:
The project has generated substantial innovations for European industry in the areas of security printing, packaging, consumer goods, biomedical measurement equipment and robotics. Security printing has been developed that allow visually impaired people to read authentication information, for example. The packaging and consumer goods industries will be able to exploit the improved understanding of affective touch by the possibility to identify beneficial combinations of tactile properties that will increase customer satisfaction. This will be assisted by novel equipment that has been developed for microneurography and tactile system characterisation. The fundamental investigations of mechanoreceptors and skin layers are expected to lead to innovations in nanobiosensors, biomimetic materials, tribological systems.
Strengthening partnerships with European industry
Strong collaborations have been developed between the academic and industrial partners. As result of the current project a new EU funded project has been successfully started that exploits the understanding of the human tactile system to develop touch screens with haptic feedback. The companies in NanoBioTouch include the Anglo-Dutch company Unilever plc, one of the largest consumer products companies in the world with global annual sales of more than €40 billion to 150 countries. In addition, there are four high-tech SME companies: C3M specialises in the development of the most advanced virtual M5 models for sensitivity analyses, inverse modelling and optimisation, CK-electronics is the world market leader in producing skin testing equipment for clinical diagnosis in dermatology as well as cosmetic, pharmaceutical, raw material industries; Rockfield Software Limited is world leader in the development of advanced Finite and Discrete Element computer systems; and Optaglio is a high-tech SME specialising in security printing and brand protection with markets in more than 30 countries.
Economic and societal impacts
The project will contribute to the emerging appreciation of the importance of touch and tactile experience for emotions, well-being, social interaction between con-specifics, and normal development. The improved understanding of human touch and details of mechanotransduction will provide valuable insights to those tactile scenarios that yield pleasant emotions and have beneficial social effects. Enriched tactile perception of consumer products can have a decisive impact on the success of the products and bring considerable added value and economic benefits, especially when applied to the large markets of packaging, home and personal care sectors, for example. The under-exploited potential of the tactile dimension of products represents a strong potential for a step-change innovation rather than just incremental improvements, and this will assist industry to deliver improved benefits that consumers demand, leading to maintained or increased market share. Another large size market relevant to the results of the project is the food and beverage packaging industry with annual turnover of around 24 billion USD. A massive 235 billion beverage cans are produced annually worldwide (45 billion cans in Europe). The trends in the packaging industry are determined by the requirements for individualism, flexibility, complexity, specialisation and new product experiences where important drivers are lifestyle, indulgence, wellness, health, functional and emotional added value, convenience and cost efficiency. A better understanding of how tactile perception is influenced by the physical properties of products will contribute significantly to multi-objective optimisation of packaging products. Due to the S&T breakthroughs of this project Optaglio, CK electronics, Rockfield Software and C3M will enhance their leading positions in specialised markets as well as enlarging their business network to newly emerging areas of converging technologies. Finally, the development of an artificial finger with sophisticated touch capabilities will have an enormous economic and societal impact on prosthetics and robotics that will assist disabled and elderly people to lead more independent lives and improve their quality of life.
Impact on medical rehabilitation
The fundamental understanding of touch in the project will have beneficial effects on tactile stimulation practices that are used in medical and psychological rehabilitation. Touch is a key part of physical examination procedures in medicine. As a sensory system, touch is prone to a variety of peripheral and central neurological disorders and new approaches to the diagnosis and treatment of such disorders may be expected from the improved scientific understanding and the innovation embodied in this project. As a very robust and widely distributed sensory system, touch is also very important in its potential to offer substitute information pathways to compensate for the loss of other sensory systems. This may be particularly true in neurologically impaired and elderly people in whom sensory substitution by touch may be less efficient. The focus on affective as well as discriminative touch has the potential for significant impact on understanding contributions to emotional well-being, which can be critical for effective medical intervention. Touch also plays an important role in the wider domain of patient-carer relations, including for instance nursing and physiotherapy. Thus stimulation of tactile pathways is an important adjunct to physiotherapy for the treatment of muscle weakness following stroke. Tactile stimulation is also used by physiotherapists in the rehabilitation of peripheral sensory impairment, for instance arising from median nerve damage in accidents involving falls onto glass. There are also examples of therapies, complementary to medicine, based around touch. For example, healing touch (HT; www.healingtouchprogram.com) is a complementary therapy fostering nurse-patient connection. Nurses are beginning to use HT with their patients to assist in easing pain and anxiety, promote relaxation, accelerate wound healing, diminish depression, and increase a patient's sense of well-being. The research may be expected to support new studies and clinical tests in several of these areas. According to the World Health Organization (WHO), in 2002 an estimated 161 million people worldwide were visually impaired, of whom 124 million people had low vision and 37 million were blind. The innovative approaches to generate security printing will assist visually impaired people in many ways. The improved understanding of human touch especially the exploratory process in discriminative touch has the potential to help develop rehabilitation methods for patients with strokes and peripheral nerve injuries. The insights gained from studying autistic subjects should assist in diagnosing the condition and developing improved therapies.
Main dissemination actives
The main dissemination channels were by journal publications, the website (http://nanobiotouch.org/) and the mass media, which include the following:
• Video interview released to “E se domain”, a TV Show on the Italian national Television Rai3, which discusses the impact of modern technology to life. The interview will be transmitted on TV in 2013.
• Video interview released to “TG3 Pixel”, a Science TV Show on the Italian national Television Rai3 (October 13, 2012): http://www.rai.tv/dl/RaiTV/programmi/media/ContentItem-e9388b11-ad52-4ab2-b51c-962b1f1c0fcd-tg3.html
• Interview released to "ilSole24Ore" (first page of the weekly supplement "Nòva", dealing with science and technology) about Nanobiotouch: http://www.ilsole24ore.com/art/tecnologie/2012-03-24/sesto-senso-robot-121457.shtml?uuid=AbiYLQDF
• A purposely-edited video (including UG, SSSA and Nanobiotouch logos) on joint UG-SSSA research activities is shown in the on-line version of the article: http://www.ilsole24ore.com/art/tecnologie/2012-03-24/sesto-senso-robot-121457.shtml?video&uuid=AbiYLQDF
• Several images and pictures resulting from the Nanobiotouch project are shown in the on-line version of the article: http://www.ilsole24ore.com/art/tecnologie/2012-03-24/sesto-senso-robot-121457.shtml?foto&uuid=AbiYLQDF
• A long report by the “Crash” programme of the Italian National Television Rai3, dealing with hybrid-bionic systems
• Particularly, results of the NMP-Smarthand and of the NMP-Nanobiotouch projects, among others, were shown: http://www.sssup.it/webtv_channel_rass_stampa.jsp?id_context=28631&ID_LINK=10156&area=193 The full programme is available via the website of the Rai Italian National Television: http://www.crash.rai.it/sito/scheda_puntata.asp?progid=1545
In addition Prof. Maria Chiara Carrozza at released an interview within the programme “Unomattina” of the principal Italian public television RAI1 that included some results of NanoBioTouch. During the interview, Prof. Carrozza was supported by Dr. Calogero Maria Oddo and Mr. Marco Controzzi, who contributed to NanoBioTouch, and reported and demonstrated the research activities carried out at The BioRobotics Institute of the Scuola Superiore Sant’Anna. The interview is online in the RAI website: http://www.rai.tv/dl/RaiTV/programmi/media/ContentItem-2076577d-f7f4-4a11-bc53-d8337b53abbe.html
Screenshots of the interview released by Prof. Maria Chiara Carrozza within the programme showing the artificial finger pad (left).
The journal publications and communications at conferences are listed below:
Journal Papers
Tactile discrimination threshold is unrelated to tactile spatial acuity
X. Libouton, O. Barbier, L. Plaghki, J-L.Thonnard
Behavioral Brain Research 208, 473-478, (2010)
Finger pad friction and its role in grip and touch
MJ Adams, SA Johnson, P Lefevre, V Levesque, V Hayward, T Andre, J-L Thonnard
Journal of the Royal Society interface, 10, 20120467 (2013)
Fingertip Moisture is Optimally Modulated during Object Manipulation
T. André, P. Lefèvre, J-L. Thonnard
J. Neurophysiol. 103 , 402-408, (2010)
Optimised determination of viscoelastic properties using compliant measurement system
JW Andrews, J Bowen, D Cheneler
Soft Matter 2013, 9, 5581
This paper received the award for best submitted publication at the University of Birmingham, June 2013
Alginate hydrogel has a negative impact on in vitro Collagen 1 deposition by fibroblasts
AM Smith, NC Hunt, RM Shelton, G Birdi, LM Grover
Bio Macromolecules 13, 4032-4038 (2012)
Development and characterisation of a bio-hybrid skin-like stretchable electrode
E. Buselli, AM Smith, LM Grover, A Levi, R Allman, V Mattoli, A Menciassi, L Beccai
Microelectronic Engineering, 88, 1676-1680, 2011
Friction of the human fingerpad; influence of moisture, occlusion and velocity
SM Pasumarty, SA Johnson, SA Watson, MJ Adams
Tribology Letters 44, 117-137, (2011)
Effect of skin hydration on the dynamics of fingertip gripping contact
T. André, V. Lévesque, V. Hayward, P. Lefèvre, J-L. Thonnard
J. R. Soc. Interface 8, 1574–1583, (2011)
Active touch sensing
Tony J Prescott, Matthew E Diamond, Alan M Wing
Phil. Trans. R. Soc. B 366, 2989-2995 (2011)
Light touch for balance: influence of a time-varying external driving signal
Alan M Wing, Leif Johannsen. Satoshi Endo
Phil. Trans. R. Soc. B 366, 3133-3141 (2011)
Learning to touch through active exploration.
L. Pape, C.M. Oddo, M. Controzzi, C. Cipriani, A. Förster, M.C. Carozza, J. Schmidhuber.
Frontiers in NeuroRobotics, 2012, 6(6).
Texture-induced vibrations in the forearm during tactile exploration
B. Delhaye, V. Hayward, P. Lefèvre and J‐L. Thonnard
Front. Behav. Neurosci. 6:37. doi: 10.3389/fnbeh.2012.00037 (2012)
Roughness encoding in human and biomimetic artificial touch: spatiotemporal frequency modulation and structural anisotropy of fingerprints.
C.M. Oddo, L. Beccai, J. Wessberg, H. Backlund Wasling, F. Mattioli, M.C. Carrozza.
Sensors, 11, 5596-5615 (2011)
Roughness Encoding for Discrimination of Surfaces in Artificial Active Touch.
C.M. Oddo, M. Controzzi, L. Beccai, C. Cipriani, M.C. Carrozza.
IEEE Transactions on Robotics, 27, 522-533 (2011)
Mechatronic Platform for Human Touch Studies
C.M. Oddo, L Beccai, N. Vitiello, H Backlund Wasling, J Wessberg, M.C. Carrozza.
Mechatronics, 21, 604-613 (2011).
Calcium-alginate hydrogel-encapsulated fibroblasts provide sustained release of vascular endothelial growth factor. NC Hunt, RM Shelton, DJ Henderson, LM Grover. Tissue Engineering Part A, 2012
Synthetic and bio-artificial tactile sensing: A Review
C Lucarotti, CM Oddo, N Vitiello, MC Carozza. Sensors, 2012
Degradation of polysaccharide hydrogels in cell culture conditions
SH Jahromi, LM Grover, JZ Paxton, AM Smith
Journal of the Mechanical Behaviour of Biomedical Materials 2012
Rasch-built measure of pleasant touch through active fingertip explorations
A. Klöcker, C. Arnould, M. Penta, J-L. Thonnard
Front. Neurorobot. 6:5. doi: 10.3389/fnbot.2012.00005 (2012)
Tactile roughness discrimination of the finer pad relies primarily on vibration sensitive afferents not necessarily located in the hand
X Libouton, O Barbier, Y Berger, L Plaghki, J-L Thonnard, Behavioural Brain Research 229, 273-279 (2012)
Physical factors influencing pleasant touch during tactile exploration
A Klocker, M Wiertlewski, V Theate, V Hayward, J-L Thonnard, PLoS One, November 2013, Vol 8, Issue 11, www.plosone.org
Restoring natural sensory feedback in real-time bi-directional hand prosthesis
Science Transitional Medicine, Vol 6, Issue 222, Feb 2014
Communications at conferences
Finger pad tribology: grip function and tactile perception
M J Adams, S A Johnson, P Lefèvre, V Lévesque, V Hayward, T André and J-L Thonnard
37th Leeds-Lyon Symposium on Tribology, Leeds, September 2010
Tactile discrimination threshold is unrelated to tactile spatial acuity
X. Libouton, O. Barbier, L. Plaghki, J-L.Thonnard
18th Brussels Hand-Upper Limb International Symposium, Brussels, Belgium, January 2010
The role of moisture in the optimization of stick-to-slide dynamics in the human grip.
T. André, V. Lévesque, V. Hayward, P. Lefèvre, J-L.Thonnard
20th Annual Meeting of the Neural Control of Movement Society, Naples, USA , April 2010
The motor control of haptic perception: getting more out of it than what you put into it.
A. Smith, V. Hayward, G.Loeb J-L.Thonnard
20th Annual Meeting of the Neural Control of Movement Society, Naples, USA April 2010
Optimal modulation of fingertip moisture during object manipulation.
J-L.Thonnard University College of Dublin, Health Sciences Centre, Dublin, Ireland, September 2010
Optimal modulation of fingertip moisture during object manipulation.
J-L.Thonnard , T. André, P. Lefèvre
40th Annual Meeting of the Society for Neuroscience, San Diego, USA November 2010
Textural vibrations in the forearm during tactile exploration
B. Delhaye, V. Hayward, P. Lefèvre, J-L.Thonnard
40th Annual Meeting of the Society for Neuroscience, San Diego, USA November 2010
A comparison of the frictional behaviour of human glabrous skin with synthetic glassy polymers and elastomers
M. J. Adams and S. A. Johnson
International Conference on BioTribology, Imperial College London, September 2011
Bio-hybrid tactile sensor for the study of the role of mechanoreceptors in human tactile perception
D. Cheneler a*, M. C. L. Ward a, C. J. Anthony a, 37th International Conference on Micro and Nano Engineering, MNE 2011 19-23 September 2011, Berlin, Germany
Poster presentation at the Active touch sensing meeting at the "Kavli Royal Society Centre", February 2011
Pleasant touch and the role of the fingertip moisture level, A Klöcker, C Arnould, M Penta, JL Thonnard
Poster presentation at the Active touch sensing meeting at the "Kavli Royal Society Centre", February 2011
Vibrations in the forearm during active touch of rough textures, B Delhaye, V Hayward, P Lefevre, JL Thonnard
Fingerpad sensitivity is not required for Tactile Roughness Discrimination
X. Libouton, O. Barbier, L. Plaghki, J-L.Thonnard 19th Brussels Hand-Upper Limb International Symposium, Brussels, Belgium, 2011
Rasch built measure of pleasant touch elicited through active exploration
A. Klöcker, C. Arnould, M. Penta, J-L. Thonnard, 19th Brussels Hand-Upper Limb International Symposium, Brussels, Belgium, February 2011
Textural vibrations in the forearm during tactile exploration
B. Delhaye, V. Hayward, P. Lefèvre, J-L. Thonnard
19th Brussels Hand-Upper Limb International Symposium, Brussels, Belgium, February 2011
Brain Imaging and touch
HF Kwok, Presentation given during Science Week, Moseley, Birmingham, UK, 2011
Novelty-based restarts for evolution strategies
G Cuccu, F Gomez, T Glasmachers, 2011 IEEE Congress on Evolutionary Computation (CEC2011)
Modular deep belief networks that do not forget
L Pape, F Gomez, M Ring, J Schmidhuber, International Joint Conference on Neural Networks, 2011 (IJCNN2011)
Vibrotactile sensory substitution in multi-fingered hand prostheses: Evaluation studies
M D’Alonzo, C Cipriani, MC Carrozza
Proc of International Conference on Rehabilitation Robotics, Zurich, Switzerland, June 2011 (ICORR2011)
Tactile detection of slip. Fine characterization of skin deformation during the onset of slip.
B. Delhaye, P. Lefèvre, J-L Thonnard
Brussels Hand/Upper Limb International Symposium, Brussels, Belgium January 2012
Tactile detection of slip: Fine characterisation of skin deformation during the onset of slip
B Delhaye, P Lefevre, J-L Thonnard, Poster presented at “NCM” Congress, Venice, Italy, April 2012
Bio-hybrid tactile sensor and experimental set-up for investigating and mimicking the human sense of touch
D Cheneler, E Buselli, CM Oddo, G Kaklamani, L Beccai, MC Carrozza, LM Grover, CJ Anthony, MC Ward, MJ Adams. Workshop on “Advances in tactile sensing and touch based human-robot interaction”, Boston, USA, March 2012
Physical properties that contribute to roughness discrimination of textures with randomly distributed asperities
M D’Alonzo, N Vitiello, L Beccai, HF Kwok, CM Oddo, AM Wing, MC Carrozza. Workshop on “Advances in tactile sensing and touch based human-robot interaction”, Boston, USA, March 2012
Soft-neuromorphic artificial touch for applications in neuro-robotics.
G. Spigler, C.M. Oddo, M.C. Carrozza. IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob 2012), Rome, Italy, June 2012
Physical factors influencing pleasant touch during tactile exploration
A Klocker, M Wiertlewski, V Theate, V Hayward, J-L Thonnard
Poster presented at Annual Meeting of the Society for Neuroscience, New Orleans, USA, October 2012
Functional micro-anatomical finite element model of Meissner corpuscle
T Vodlak, Z Vidrih, P Pirih, J Presern, T Rodic, Abstract submitted to Eurohaptics, Paris 2014
Exploitation of results
• Use of an array of microelectrodes on a silicon substrate providing real-time information on cell response during the application of applied loads. This would involve integration of the bio-hybrid tactile sensor and of the machine learning algorithms in the dynamic platform that was developed.
• An element to be developed that is visually flat, but produces a quickly recognizable 3D effects when touched. Applications would be mainly in security printing companies particularly aimed at the visually impaired.
• Development of an alginate/S-DMEM hydrogel for use in wound management targeted at the health care market.
Exploitable results that would be of interest to large retailers, large packagers ‘high-tech’ markets of security printing, biomedical measurement equipment, virtual modelling software companies, healthcare companies:
• Development of a bio-inspired robotic finger with an adequate number of DoFs and actuators (degrees of mobility) to allow for exploratory touch.
• Nanoscale sensitivity of taction and concomitant advances in signal processing. This would involve integration of the NEMS tactile array and of the machine learning algorithms in the dynamic platform that was developed.
• Development of a demonstrator of exploratory touch using the biorobotic finger with the embedded NEMS tactile array. This would involve integration of the NEMS tactile array and the machine learning algorithms in the biorobotic finger.
• Design and fabrication of a two-degree-of-freedom platform, with integrated thermal cycle stimulation control. Target commercial sectors include instrument manufacturers, security printing, packaging, consumer goods, biomedical measurement equipment and robotics.
Conclusive remarks
In addition to the patents arising from the project, there are a number of potential foreground applications that are wide ranging and include enhancing the tactile dimension of personal care products, diagnosing Carpal Tunnel Syndrome, restoration of touch in amputees, robotic fingers with NEMS and bio-hybrid tactile sensing that incorporate machine learning a demonstration platform for the robotic finger, and a 2D tactile mechanical and thermal stimulator for studying in vivo tactile responses. These applications are summarised in the table below together with their Technological Readiness Level.
Exploitable product or measure. Technology Readiness Level Description Supporting Information
Potential to enhance the tactile dimension of personal care products, food and beverage,smart packaging, security printing,medical rehabilitation sectors 1. Basic principles observed and reported In a first step, a Pleasant Touch Scale has been elaborated. This scale defines reliable and objective quantification of the (i) pleasantness levels of 37 everyday life materials (fabrics, wood, wax, latex, paper,…) and (ii) pleasantness satisfaction levels of 200 subjects, induced by exploration of the 37 surfaces have been determined (Klöcker et al. 2012). In a second step these pleasantness levels could be significantly related to various surface physical properties (such as the surface topography) (Klöcker et al. 2013).
These results are particular of interest for various industries using, for example, the “tactile branding marketing strategy”. The aim of this strategy is to generate a “connection” between the consumer’s emotional feeling and the brand through a stimulation of the tactile sense. Indeed, what a product feels like can influence people’s decision on buying this product (Spence and Gallace 2001). As a consequence, it is of importance to extract factors intervening with the perceiver’s “tactile evaluation” of specific objects. For example, the textile industry typically aims to create comfortable and pleasant products, inducing the consumers to buy these products. Other examples of industries which might be interested by the results described in the studies of Klöcker et al. (2012, 2013) are the electronic industry and the cosmetic industry. With respect to the electronic industry, it is interesting to note that they use more and more the tactile branding to promote their products (e.g. the Apple iPod can easily be recognized by only touching it). The cosmetic industry tries also to elaborate, for example, hand lotions after the application of which consumers have a smooth tactile sensation.
The results might also be of importance for the health care sector. Indeed, the affective modality of the sense of touch is completely ignored in sensory assessments of pathologies affecting touch. This ignorance is probably due to the fact that no objective scale existed to measure pleasant touch. Klöcker A, Arnould C, Penta M and Thonnard JL (2012) Rasch-built measure of pleasant touch through active fingertip exploration. Front Neurorobot 6, 1-9.
Klöcker A, Wiertlewski M, Théate V, Hayward V and Thonnard JL (2013) Physical factors influencing pleasant touch during tactile exploration. PLoS ONE 8, e79085.
Friction test to diagnose Carpal Tunnel Syndrome and other medical conditions. 2. Technology concept and application formulated Finger friction measurements on relatively smooth surfaces have been shown to be very sensitive to skin moisture. Carpal Tunnel Syndrome causes some fingers to not sweat as much as others. Hence comparative tests should aid diagnosis. Could be extended to support diagnosis and monitoring of other conditions that effect skin moisture such as diabetes, xerosis, hyperhidrosis and anxiety disorders. Governing principles described in
http://dx.doi.org/10.1098/rsif.2012.0467
Application being considered:
http://dx.doi.org/10.1177/1753193408087231
Tactile perceptions from stimulation of single afferents for restoration of touch in amputees 1. Basic principles observed and reported Using the techniques of microneurography and microstimulation, we stimulated single, identified tactile receptors in human healthy volunteers. This gives rise to discrete and highly repeatable sensations of pressure (from SAI afferents) or a sense of vibration (RA, Pacinian afferents). Variations in stimulus frequency results in graded variations in the intensity of the sensations, and variations in stimulus patterns are clearly distinguishable. The corresponding brain activity can be studied using functional imaging techniques.
These basic scientific findings are directly applicable for experiments on replacement of tactile sensory function in arm and hand amputees, using implanted neural electrodes. Initial experiments have been initiated. Importantly, the aim is to go beyond crude detection of impact or force intensity, but aim towards replacement of the discriminatory details of tactile sensations. Results reported at the Society for Neuroscience (USA) meeting:
2012, Paper 677.26:
R Ackerley et al. Microstimulation of single mechanoreceptor afferents in humans.
2013, Paper 465.17:
R Ackerley et al. EEG activity evoked from microstimulation of single mechanoafferents in humans.
2013, Paper 644.23:
FP McGlone et al. Intraneural microstimulation of somatosensory afferents during fMRI at 7T.
Presentation at Janelia Conference, Howard Hughes Medical Institute, “Mammalian Circuits Underlying Touch Sensation” 22-25 Sept. 2013. R. Ackerley, Microstimulation of single mechanoafferents in humans.
Robotic finger 6. System demonstration in a relevant environment. The robotic finger is part of a long-term scientific and technological strategy, implemented with the federation of various national and European research projects, that is driving towards the integration of an artificial sense of touch in multi-fingered robotic hands for main use in industrial manipulation and upper limb prosthetics.
Implementation of a neuromorphic architecture mimicking the coding of tactile information by human mechanoreceptors.
The integrated system has been experimented in a controlled operational environment (i.e. in a clinical facility) in the framework of a pilot study with a transradial hand amputee. A high impact journal publication (in Science Translational Medicine) related in part to the integrated system has been achieved, and other high-impact publications are being prepared.
S. Raspopovic, M. Capogrosso, F. M. Petrini, M. Bonizzato, J. Rigosa, G. Di Pino, J. Carpaneto, M. Controzzi, T. Boretius, E. Fernandez, G. Granata, C. M. Oddo, L. Citi, A. L. Ciancio, C. Cipriani, M. C. Carrozza, W. Jensen, E. Guglielmelli, T. Stieglitz, P. M. Rossini, S. Micera, Restoring Natural Sensory Feedback in Real-Time Bidirectional Hand Prostheses. Sci. Transl. Med. 6, 222ra19 (2014). http://stm.sciencemag.org/content/6/222/222ra19
Integration of the NEMS tactile array and of the machine learning algorithms in the Dynamic Platform 6. System demonstration in a relevant environment. The integrated system has been experimented in a controlled operational environment (i.e. in a clinical facility) in the framework of a pilot study with a transradial hand amputee. CM Oddo, L Beccai, J Wessberg, H Backlund Wasling, F Mattioli, MC Carrozza, Roughness encoding in human and biomimetic artificial touch: spatiotemporal frequency modulation and structural anisotropy of fingerprints. Sensors, 11, 5596-5615 (2011). http://www.mdpi.com/1424-8220/11/6/5596
G Spigler, CM Oddo, MC Carrozza, Soft-neuromorphic artificial touch for applications in neuro-robotics. IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob 2012), Rome, Italy, June 2012. http://ieeexplore.ieee.org/xpl/abstractAuthors.jsp?arnumber=6290849
Intellectual property rights have been raised with the following patent application claiming specific technological solutions. PCT/IB2014/060223 patent application is an extension of a previous Italian national application that obtained positive responses to all claims.
G. Muscolo, C.M. Oddo, L. Beccai, M.C. Carrozza, Method and Device for Making Fingerprints Associated to Artificial Fingers. PCT Application Number PCT/IB2014/060223.
Integration of the NEMS tactile array and of the machine learning algorithms in the biorobotic finger 5. Component validation in relevant environment. Experiments of the integrated actuated biorobotic system have been carried out under realistic scenarios (e.g. discrimination of materials and textures, softness coding) in controlled conditions emulating daily life activities and with the implementation of machine learning methods (artificial curiosity). Experiments involving sensorimotor loops with neuronal control architectures have been carried out. L Pape, CM Oddo, M Controzzi, C Cipriani, A Förster, MC Carozza, J Schmidhuber, Learning tactile skills through curious exploration. Frontiers in NeuroRobotics, 6:6 (2012). http://journal.frontiersin.org/Journal/10.3389/fnbot.2012.00006/full
CM. Oddo, M Controzzi, L Beccai, C Cipriani, MC Carrozza. Roughness encoding for discrimination of surfaces in artificial active touch. IEEE Transactions on Robotics, 27, 522-533 (2011). http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5740988&tag=1
G Spigler, CM Oddo, M Controzzi, D Camboni, C Cipriani, MC Carrozza. Haptic sensing in a robotic hand, Workshop Presentation at the IEEE World Haptics Conference, Daejeon, South Korea, April 14th 2013.
Intellectual property rights have been raised with the following patent application claiming specific technological solutions.
G. Spigler, C.M. Oddo, D. Camboni, M.C. Carrozza, Metodo per trasmettere sensazioni tattili ad un utente e apparecchiatura che attua tale metodo (Method and apparatus for transmitting tactile sensations to an user). Italian Patent Application Number PI2013A000028.
D6.4 describes the results of tests on some representative packaging materials, cosmetics and textiles. Some further optimisation and validation of the data analysis procedures is necessary before appropriate machine learning algorithms could be integrated for routine use in product improvement scenarios.
An artificial bio-hybrid integrated system having an enhanced level of integration of the control, actuation and tactility that will provide affective as well as a discriminatory exploratory touch capability. 3. Analytical and experimental critical function and characteristic proof of concept. The feasibility of the bio-hybrid tactile sensing technology has been verified through experiments in controlled laboratory conditions with the dynamic tactile stimulation platform. C Lucarotti, CM Oddo, N Vitiello, MC Carozza, Synthetic and bio-artificial tactile sensing: a review. Sensors 13, 1435-1466 (2013). http://www.mdpi.com/1424-8220/13/2/1435
D Cheneler, E Buselli, CM Oddo, G Kaklamani, L Beccai, MC Carrozza, LM Grover, CJ Anthony, MC Ward, MJ Adams, Bio-hybrid tactile sensor and experimental set-up for investigating and mimicking the human sense of touch. Workshop on “Advances in tactile sensing and touch based human-robot interaction”, Boston, USA, March 2012
D Cheneler, MCL Ward, CJ Anthony , Bio-hybrid tactile sensor for the study of the role of mechanoreceptors in human tactile perception. 37th International Conference on Micro and Nano Engineering, MNE 2011 19-23 September 2011, Berlin, Germany.
D5.4 presents the relevant achievements. Other scientific publications are in progress.
The demonstrator consists in active and passive touch tasks performed consecutively by using different tactile stimuli. 6. System demonstration in a relevant environment. The system has been demonstrated in controlled scenarios during scientific events as well as during open-days involving society at large. G Spigler, CM Oddo, M Controzzi, D Camboni, C Cipriani, MC Carrozza. Haptic sensing in a robotic hand, Demonstration and Workshop Presentation at the IEEE World Haptics Conference, Daejeon, South Korea, April 14th 2013. http://www.haptics2013.org/download/WHC2013_Final_Program.pdf
Public demonstrations with citizens during open days at SSSA, as detailed in D8.2 http://sssa.bioroboticsinstitute.it/events/sanfaustino13
A two-degree-of-freedom tactile mechanical stimulator able to stimulate a human (or an artificial) fingertip along the tangential and normal axis with different surfaces and materials, with integrated thermal cycle stimulation control, to be used in touch experimental paradigms with both human subjects and artificial/hybrid tactile sensors 6. System demonstration in a relevant environment. Mechanical and thermal stimulator deployed to a number of project partners and used as experimental tool for human and artificial touch studies. Particularly of interest to industries concerned with improving the feel of metallic components, where sensory cues associated with heat transfer are important, through coating or surface texturing. CM Oddo, L Beccai, N Vitiello, H Backlund Wasling, J Wessberg, MC Carrozza, A mechatronic platform for human touch studies. Mechatronics, 21, 604-613 (2011). http://www.sciencedirect.com/science/article/pii/S0957415811000456
D3.3 describes the design and operation of the equipment. Particularly well-suited to thin thermally conductive samples whose surface temperature can be rapidly altered using the Peltier devices.
Artificial curiosity software. 2. Technology concept and application formulated The machine learning methods developed and applied in NanoBioTouch will become increasingly useful, as companies begin to deploy robotic and neuroprosthetic systems. The principle of artificial curiosity, was shown to enable a robotic finger to actively explore its environment, such that it could better discriminate between the types of surface textures it was likely to encounter, without the need for any pre-programmed information. This is a critical requirement if robots are to be trusted to behave autonomously in complex and changing environments. Learning tactile skills through curious exploration
L Pape, CM Oddo, M Controzzi, C Cipriani, A Förster, MC Carozza, J Schmidhuber
Frontiers in NeuroRobotics, 6:6 (2012).
List of Websites:
www.nanobiotouch.org contact Kathleen Hynes, k.m.hynes@bham.ac.uk
A biomimetic approach has been taken to develop a robotic finger that is capable of sensing touch, able to explore surfaces with artificial curiosity, discriminate different textures and assess their tactile pleasantness. The development of a biomimetic approach required a greater understanding of the subjective and neurological mechanisms involved with discriminative and affective touch for both healthy human volunteers and patients. This included (a) the role of the peripheral receptor response using microneurography that allowed the spike trains from a mechanoreceptor in a finger pad to be recorded when subjected to a tactile stimulus, (b) brain imaging using electroencephalography (EEG) and functional magnetic resonace imaging (fMRI) in conjunction with microsimulation of mechanoreceptors in the finger pads, (c) microstimulation of single, identified tactile nerve afferents, and evaluation of the sensations reported by the participants in order to explore the discrimination of both the stimulation frequency from a series of target frequencies, and the variability of neural signals, and (d) psychophysical studies of the pleasantness of materials in order to develop a unidimensional scale and also studies of autistic subjects and stroke patients to evaluate their tactile abilities for diagnostic and therapy purposes. To carry out these studies, a novel dynamic platform for tactile stimulation and temperature control was developed in order to provide a controlled tactile stimulus. Multi-scale multi-physics modelling of the finger pad was developed that was capable of describing the deformation when in static or sliding contact with a surface. It was coupled to micro model of two of the tactile afferents (Meissener corpuscle and Merkell cell) so that the deformation caused spike trains to be emitted by a further coupling with a micro-mechanical model of ion channel opening. The spike trains were of similar form to those recorded by microneurography measurements. Arificial touch sensors were developed that were incorporated in a biomimetic artificial finger. By exploiting machine learning algorithms, it was possible to use the finger to identify textures, assess pleasantness and provide the ability for the finger to explore unknown surfaces. A bio-hybrid tactile sensor was also developed that was based on encapsulated cells. The sensor was able to detect both static and sliding stimuli. A number of applications of the technology developed in the project were examined that included enhancements to the Cutometer for measuring the mechanical properties of skin in vivo. The idea is to investigate the aging of skin during space flights in a joint project with the European Space Agency. Other applications included the use of the artificial finger in a prosthetic hand with direct neural coupling, microstimulation of a sensory nerve in an upper limb amputee to evoke the sense of touch, design of security cards and packaging and the evaluation of packaging surfaces and cosmetic creams.
Project Context and Objectives:
Tactile sensations reveal vital features of the environment of an organism. Human touch can be considered as a complex multi-scale interaction of biophysical and biochemical phenomena that transduce mechanical and thermal stimuli imposed by shape, texture, stiffness, temperature and movement of surrounding objects. The neural response is transmitted to the brain for recurrent sequence processing that influences our emotions and behaviour. Due to these high level impacts, investments into an improved understanding of human touch can bring many socio-economic, technological and scientific rewards. Tactile sensations can affect our well-being, support medical and psychiatric rehabilitation, assist blind people to gain access to books, maps and other documents and play a critical role in social interactions. Recent research has revealed that affective touch can yield valuable social benefits; the term affective touch relates to pleasantness and the preference of a particular tactile experience. Many industrial sectors would benefit from understanding and optimising the tactile perception of their products in order to achieve better consumer satisfaction. Furthermore, an understanding of mechano-transducting sub-systems can inspire many biomimetic engineering solutions related to sensor technologies, smart material composites and micro-tribological systems in addition to information processing. The human tactile sensation system illustrated above has been approached from many scientific and technological disciplines including medicine, biology, chemistry, physics, electronics and mechanics as well as numerical modelling and computer science, which are revealing the fascinating features of this biological mechano-transduction system.
On the above basis, NanoBioTouch was formulated to radically improve the understanding of the human mechanotransduction system in order to assist in the biomimetic design of tactile sensors incorporated in an artificial finger. This was achieved through systematic integration of new developments from converging scientific areas by involving academic and industrial participants who are experts in cognitive sciences, microneurography, brain imaging, cell biology and mechanics, tissue engineering, skin physics (tribology and mechanics), microengineering, multi-scale multi-physics modelling, information processing, robotics, prosthetics and medical rehabilitation. The project was built on existing discriminative touch research in order to understand affective touch mediated by the human fingerpad. Sensors capable of detecting directional force and temperature were to be developed as a combination of these modalities, which is critical to the affective component of the neurophysiological response evoked in taction. This next generation of sensors includes artificial sensor arrays and hybrid bio-artificial systems. A biorobotic finger with articulation was to be developed, which was controlled by neural network information processing that allowed artificial exploration of a surface to be achieved in ways that mimic human haptic behaviour and affective response. The required impact of the project included alleviating the effects of human tactile and visual disabilities, improving the quality of life, security printing, brand protection, smart packaging, space exploration and also the evaluation of products such as textiles and skin creams using the instrumented artificial finger. The consortium includes industrial participants to undertake specific technical innovation activities in order to maximise the commercial impact and exploitation of the research. The partners are listed in the table below:
Role Partner name
Coordinator University of Birmingham, UK
Industry & SME C3M, Slovenia
Rockfield Software Ltd., UK
CK Electronics, Germany
Optaglio, Czech Republic
Unilever R&D Port Sunlight, UK
Academic University of Gothenburg, Sweden
Universite Catholique de Louvain, Belgium
University of Ljubljana, Slovenia
University of Wales, Swansea, UK
Scuola Superiore Sant’Anna, Italy
Dalle Molle Institute for Artificial Intelligence, Switzerland
Objectives
The scientific aims of this project were to radically improve understanding of the human mechano-transduction system and develop an artificial finger incorporating a touch sensor. This was achieved by systematic integration of new developments from converging scientific areas (bio, nano, info and cogni) that were based on the following scientific objectives:
»Bio« To isolate and then grow mechanoreceptor co-cultures within a gel matrix. The production of such a structure was to allow the response of mechanoreceptors found in the skin to be monitored during the application of defined stress fields induced by normal and tangential contacts.
»Nano« To develop a new generation of artificial and nanobio tactile sensors for discriminative and affective touch. The nanobio sensor to involve the interfacing of cells and engineered mechanoreceptors with microelectrodes on the surface of a chip. The electrical signals generated by ion channel opening to be monitored and the resulting data to be employed in multi-scale multi-physics modelling of their response to applied mechanical stimuli.
»Cogni« To investigate the relationship between affective components of touch, such as pleasantness and preferences, and different discriminative touch modalities (e.g. roughness, temperature and softness) in order to provide data that can be employed to develop a model capable of predicting whether a material is pleasant to touch given the discriminative touch responses elicited from the artificial finger. To investigate pleasantness feeling in glabrous skin using the brain imaging techniques EEG and fMRI to analyze the physiological responses of single afferent units recorded using microneurography.
»Info« To develop a virtual multi-scale multi-physics model of mechano-transduction by bridging the length scales from a macroscopic fingertip contact to the deformation of mechanoreceptor cells with integral ion channels. In this way a previously-lacking computational framework for linking mechanical stimuli to neural response would be provided. To develop new artificial recurrent networks (ARNN) in order to process the neural sequences generated by the virtual model for multiple object, surface texture recognition and tactile pleasantness.
»Integration« To develop a multiphalangeal biorobotic (artificial) finger that is capable of tactile sensing and exploration. The sensing to be implemented by incorporating the nanoscale artificial silicon sensor arrays and bio-hybrid tactile sensors. The ARNN and multiscale model to be used to analyse the signals acquired from the sensors to generate discriminative and affective touch output (such as roughness, temperature and pleasantness). The ARNN to be implemented in the movement control software of the artificial finger so that it will be capable of human like exploration.
»Innovation« To use the new understanding of discriminative and affective touch for innovation of products for security printing, brand protection, smart packaging, personal care,
measurement equipment and robotics as well as to rehabilitate patients with neurological problems and amputees, and to investigate tactile phenomena in microgravity.
Project Results:
Cognitive investigations of tactile sensations
The main objective of this activity was to investigate mechanisms of tactile sensation in human healthy volunteers and patients, from the level of peripheral receptor response to cognitive and brain mechanisms. A particular objective was to uncover principles of higher-order or complex tactile experience such as pleasantness or preference. The work included (a) brain imaging studies that were carried out in conjunction with microstimulation of mechanoreceptors in the finger pads and (b) microneurography that allowed the spike trains from a mechanoreceptor in a finger pad to be recorded when it is subjected to a tactile stimulation. A dynamic platform for tactile stimulation and temperature control was developed for psychophysical and microneurographic measurements of the response of subjects to different tactile stimuli. In the latter case, it was necessary to design and construct a special mechatronic platform that would be suitable for orientating the hands of subject in way that measurements could be performed. Details of the thermal stage are shown in Fig. 1. A range of tactile surfaces were prepared for this stage including a set of aluminium surfaces with varying spatial roughness that were measured using profilometry.
Microneurographic recording of peripheral tactile receptor activity
The dynamic platform allowed the application of motion along the direction normal to the skin surface with a smooth and controlled application of forces. The surfaces used for stimulation were evaluated by psychophysical appraisal (see below) to range from surfaces “unpleasant” to “pleasant” to touch during active exploration. The different surfaces were tested with a range of normal loads and velocities, in order to explore issues of simultaneous mechanoreceptor coding of 1) force, 2) sliding velocity, and 3) normal force.
The results showed that surface texture is encoded by a modulation of the activity of both RA and SAI afferents. The RA units were strongly sensitive to the structural roughness of the surfaces, with higher firing rate for larger spatial periods in the tested textiles. Interestingly, RA firing was only broadly frequency modulated by the presence of spatial periodicities in the surfaces, instead showing a weak and broad modulation of firing at rather high frequency (150-250 Hz). Firing rates were lower and frequency modulation lacking for smooth surfaces, both the higher-friction surfaces deemed the most “unpleasant” in the sample, and for the very smooth paper surfaces that were considered “pleasant”. SAI activity broadly resembled RA, with strongest afferent activity for surfaces with large spatial period (“rough” textile surfaces). However, the SAI sample was smaller, and some SAI units showed a strong tendency to code also the tangential friction of the surfaces, which is interesting since this factor is a strong determinant for the perceived pleasantness of the surfaces. PC activity in a smaller sample was similar to RA. Overall, the results showed a complex but very interesting pattern of combined coding of all the different aspects of mechanical stimulation of skin during natural tactile events, and these results obtained for real-life textured surfaces (such as textiles) could not readily be predicted from previous experiments on human tactile receptor response during stimulation with more strictly controlled conditions, such as stimulation with periodic gratings or Braille type dot patterns.
Fig. 1. A) Example of one of the temperature stimulation platforms; B) thermographic IR-imaging of the tactile stimulus; C) 3D visualization of the temperature along the thermal stimuli plate; D) linear temperature curve along the thermal stimulation plate; E) examples of stimuli.
Microstimulation
This involves microstimulation of single, identified tactile nerve afferents, and evaluation of the sensations reported by the participants. The following issues explored included: (a) discrimination of stimulation frequency from a series of target frequencies, and (b) discrimination of variability of neural signal, measured as coefficient of variation while average frequency is kept constant. The results showed that graded tactile percepts can be encoded in the activity of single nerve fibres. Variations in patterns were also further tested, where it was demonstrated that small pauses in ongoing activity can be perceived down to the level of single missing impulses during an ongoing 60 Hz stimulation strain, although psychophysical testing of thresholds for resolving differences in patterns was not tested extensively.
Psychophysical studies of taction
The relationship between tactile pleasantness and four other tactile dimensions (cold/warm, hardness, stickiness and roughness) was investigated and it was found that roughness, stickiness and hardness explained 30%, 14% and 5% of the variance of tactile pleasantness respectively. When lateral movement between the fingertip and the surface was prevented (during static touch), the tactile pleasantness of some materials changed significantly. However, the rating along other tactile dimensions also changed significantly for some materials. Analysis showed that when variances of ratings along other tactile dimensions were taken into consideration, movement condition did not have any effect on variance of pleasantness rating. Therefore, the perception of pleasantness in glabrous skin is probably mediated through the perception of other tactile dimensions. However, contact force was significantly different for the movement conditions. This was overcome by a passive touch study. Participants were asked to rate the pleasantness of materials while being touched using the dynamic touch platform. Contact force and velocity were controlled. There were two contact force condition (0.5 N vs 2.5 N) and two velocity condition (static and 50mm/s). There were no significant effects on rating by force and movement condition. However, the variance of the rating was significantly smaller in static touch and therefore it was concluded that lateral movement is important in the perception of tactile pleasantness. An analysis of affective touch was carried out that allowed pleasantness to be represented on a unidimensional scale. To elaborate this Pleasant Touch Scale, ordinal pleasantness scores of 37 everyday life materials (sandpaper, paper, wood, fabrics, wax, latex,…) have logarithmically been transformed into linear and unidimensional measures (using the Rasch model). The scale has been calibrated on 200 subjects (Fig. 2). The results of this study indicated that materials being rougher or stickier were perceived as more unpleasant than those being smoother. Furthermore, the participants’ fingertip moisture levels had an effect on the surfaces linear pleasantness levels, suggesting the dynamic coefficient of friction influence pleasantness perception. This is a major breakthrough in being able to rank and select surfaces in this way.
The effects of regularity of frequency on pleasantness were investigated by participants rating the roughness of periodic gratings and aperiodic (pink noise) gratings. The median frequency and mean height of the gratings did not differ significantly with grating type. Pink noise gratings were perceived to be significantly rougher whereas no significant difference in pleasantness could be found after rms height and frequency of the gratings were taken into consideration. In another study, it was found that the subjects perceived surfaces to be more unpleasant when the sliding velocity in active touch was increased.
Patient studies
Autistic participants rated gratings (regular or irregular) rougher than control participants although there was no significant difference in pleasantness rating between groups. British pictorial verbal scale (BPVS), sensory profile, grating orientation threshold, monofilament threshold, fine and coarse grating thresholds and tactile rating of a range of materials and gratings were also measured. There was a higher monofilament threshold in autistic participants and the hypotactile group (as defined using the sensory profile questionnaire) in particular had a significantly higher monofilament thresholds. Therefore, there was psychophysics evidence of tactile sensory difficulties in autism.
Stroke patients with similar parietal lesions were tested for their ability to process bilateral tactile stimuli. In 4 of the 5 patients there was a clear pattern of tactile extinction. Responses to unilateral stimuli were fast and accurate. However, responses to contralesional stimuli were slowed or absent in the bilateral condition. They also experienced difficulty detecting the offset of a contralesional vibrotactile stimulus when it occurred in the presence of an ipsilesional stimulus. Interestingly, when feeling textures with the two hands simultaneously, the 4 patients’ judgments about a surface touched by the ipsilesional hand were heavily biased towards the roughness of surfaces touched by the contralesional hand. The results of the patients showing classical signs of tactile extinction suggested that information from the contralsional hand is not entirely lost under bilateral conditions in these patients.
Fig. 2. Illustration of the Pleasant Touch Scale. Upper part: Distribution of the participant sample. Black bars represent subjects having a low fingertip moisture level and white bars represent participants having a high fingertip moisture level. Middle part: Each line represents the most probable response of one participant to the different materials. Materials are ordered from the least pleasant (top) to the most pleasant (bottom). Bottom part: Illustration of the relationship between the ordinal pleasantness scores and the linear unidimensional pleasantness measures.
Imaging of brain activity
Experiments involving stimulation of single afferents during functional magnetic resonance imaging (fMRI) of brain activity were undertaken. This involved very extensive technical challenges in setting up the microneurography and microstimulation techniques in the very strong magnetic field environment of a 7T MR scanner. Sets of afferent units were successfully stimulated during scanning of brain activity. It was observed that, first, single afferent stimulation can result in detectable activation of S1 cortex, showing early stages of processing of tactile input from the finger tips with high temporal and spatial resolution. The results from microstimulation were compared to tactile stimulation of the finger tips, and of the receptive fields where the sensations were elicited during microstimulation. The results show a strong overlap, corroborating the interpretation that the observed brain activations resulted from stimulation of single afferents with sensations projected from almost punctate areas on the hands and fingers. Further analyses revealed gradually increased fMRI activations with increased stimulus intensity in terms of stimulation frequency. This strongly suggests that the observed activations resulted from bottom-up processing related to the activation of inputs in single afferents, although top-down processes such as the subjects’ attention to the ongoing stimulation and the tactile sensations presumably are important.
Single afferent microstimulation in conjunction with recording of high-resolution (64-channel) EEG were also performed. The results showed that single afferent stimulation can result in modulation of brain activity both in the time domain as evoked potentials that were time-locked to the start of the stimuli, but also clearly in the frequency domain, where stimulation resulted in enhancement or depression of ongoing rhythmic brain activity. The latter has been previously demonstrated for tactile stimulation to the fingers, such as vibration, and the patterns for single afferent stimulation resemble these previous results, with depression of both alpha and beta oscillations during stimulation, as well as effects in the theta and possibly also the higher gamma frequency bands. Similar to the fMRI results, variations in stimulation intensity resulted in graded amplitudes of the time- (evoked potentials) and frequency-domain (modulation of oscillations) measures of cortical activity.
Nano-resolved virtual taction model
Multi-scale model of the finger pad
The modelling of the finger pad at the smallest length scale provided information about the mechanosensitivity of the afferents. Models were developed to predict the rate of opening of ion channels that evoke the action potentials forming a spike train. The focus of the work was the Meissner corpuscle and Merkell cell, which are two of the four types of afferent present in the finger pad. Finite element models of these afferents were developed (Fig. 3)
Fig. 3. (A) Meissner corpuscle with five segments of the neurite used for extraction of mechanical deformations. (B) Merkel cell – polysynaptic complex.
The 3D model had two stages: the first stage is a micro-anatomic model containing a slab of epidermis, a slab of dermis, one Meissner corpuscle and three Merkel cells (Fig. 4A); the second stage is a macro model of a fingertip and a grating (Fig. 4B). For the micro-anatomic model, a Meissner corpuscle is in the dermis, among four epidermal papillae. The three Merkel cells are embedded into one of the epidermal papillae just above lamina densa. Their postsynaptic elements are in the dermis below the lamina densa. One Merkel cell is situated in the bottom of the papilla and two are positioned on the side of the papilla.
Virtual experiments of the multi-scale model were carried out by sliding the finger pad over a range of gratings as illustrated in Fig. 5. The micro-anatomic model allowed spike trains to be computed on the basis of the mechanical deformation of the afferents (Fig. 6). They were similar in form to those measured by microneurography experiments. The various models were integrated into a framework of three distinctive blocks i.e. steering, analysis and optimization (Fig. 7). The function of the steering application is to control the whole process depending on the task the framework is performing.
Fig. 4. Finite element models of (A) Meissner corpuscle, Merkel cells and lamina densa; Meissner corpuscle (blue) is nested among four epidermal papillae and three Merkel cells (red) with their postsynaptic element (orange) are embedded into one of the papillae. The transparent mesh represents the lamina densa which divides the epidermis (above lamina densa) and the dermis (below lamina densa), and (B) distribution of effective stress during a tactile contact of a finger pad.
Fig. 5. Examples of the multi-scale finite element model in contact with a fine (left) and coarse (right) grating.
Fig. 6. Examples of spike trains generated from the multi-scale model.
Fig. 7. Structure of the computational framework.
The model has been validated against different in vivo measurements on the finger pad including sliding, indentation and suction. It has been applied to a number of applications including numerical predictions of microneurography data, input data for information processing of discriminative and affective touch, and assisting in the design and optimisation of the artificial and bio-artificial tactile sensors.
Development of an artificial finger
Artificial tactile sensor
An artificial tactile sensor was developed that incorporated a temperature sensor and an array of force sensors based on capacitance measurements to detect the bending of nano-beams (Fig. 8A). A CAD design was developed for incorporating the sensor in a robotic finger (Fig. 8B). The sensor was coated with silicone elastomer to provide protection against damage and artificial finger prints.
Fig. 8. (A) SEM of NEMS tactile sensor showing force and temperature sensors. (B) CAD drawing for incorporating the sensor in the robotic finger.
A biomimetic robotic finger (Fig. 9) was developed that is human‐sized, tendon‐driven and underactuated, i.e. with more degrees of freedom (DoFs) than actuators. This property reduced the design complexity and allowed self‐adaptation and anthropomorphic movements similar to human exploratory tasks. The finger has three DoFs (as flexion/extension DoFs of the human finger) and two DC‐motor actuators. One motor actuates the flexion/extension of the metacarpophalangeal (MCP) joint by means of two lead screw pairs driven by the same motor, in order to achieve the active flexion and extension of the joint. The other is for the underactuated flexion of the proximal interphalangeal (PIP) and distal interphalangeal (DIP) coupled joints, while the extension is achieved by means of torsional springs housed in the joints.
Fig. 9. Pictures of the experimental setup. (a) Tactile platform with (1) the robotic finger, (2) actuator modules, (3) sensor processing facilities and (4) housing for replaceable surface blocks.
Bio-artificial tactile sensor
Numerical parametric studies of the biosensor, using the nano-resolved virtual taction model described above, were carried out to evaluate the performance and optimise the design. It confirmed that in principle, cells with a mechanosensory capability can be used in a biosensor. It was found that: (1) deformation of cells will be relatively greatest when the Poisson ratio is small, suggesting using a stiff alginate, (2) electrical changes were small at low volume fractions but became significant at higher cell volume fractions, (3) deformation could be best detected when the conductivity of the extracellular solution would be as small as possible; therefore the sensitivity could be improved if the original ionic content of the extra-cellular matrix was reduced, (4) the sensitivity of the cells would increase with an increased density of mechanosensory SA channels, or when the membrane-tension-dependent SA channel activation would be optimized (slope, offset of the Boltzmann sigmoid); this could be achieved using genetic tools, (5) the sensitivity of the system would increase if the cells included an intrinsic gain mechanism, e.g. voltage-gated channels, (perhaps a system of classical Hodgin-Huxley sodium and potassium channels; This could be achieved using genetic tools. Although the parameters used in the study were approximate, it seemed that a fibroblast tissue in the absence of intrinsic gain-boost mechanism in the cells (e.g. voltage-sensitive channels) is not exquisitely sensitive to mechanical stimulation.
As a first stage of the development of the bio-hybrid tactile sensor, the lifetime of cells on a microfluidic platform were investigated. Alginate hydrogels were prepared and encapsulated with 3T3 fibroblasts, the encapsulated hydrogels were integrated on a microfluidic device for 3 days with the constant supply of Supplemented- Dulbecco’s Modified Eagle’s Medium (S-DMEM). For comparative reasons, non-encapsulated hydrogels were integrated in the device and hydrogels that were not connected to the device were incubated as well. The cells viability, the alginate hydrogel dehydration, the encapsulated with cells gels and the evaluation of the microfluidic device were examined. The microfluidic system consists of micro channels etched into a silicon substrate designed to supply the cells with nutrients and therefore be capable of sustaining their long-term viability. The channels lead into a chamber with a regular array of 50 μm diameter micro-pillars, which allow flow and diffusion of the media for cell culture throughout the chamber whilst supporting a polycarbonate nanoporous membrane. The membrane is 100 μm thick with an irregular array of 200 nm diameter holes etched through. This membrane forms a layer upon which the cells or tissues can be cultured whilst allowing media to diffuse through to the cells, assisted by capillary effects. In essence this design emulates the epidermis layer in human skin which contains no blood vessels and cells are nourished by diffusion from capillaries below. On the membrane three local conductivity sensors were deposited composed of two electrodes with a 100 μm gap. The microfluidic channels were sealed and the cells are confined to a well using a thick 500 μm silicon cover plate. The device is shown in Fig. 11 except for the silicon cover plate.
The device was fabricated in three separate layers and then assembled. Initially, a 500 μm thick silicon wafer was patterned with SPR220-7 and etched twice to define the microfluidics and silicon pillars. Next a shadow mask was fabricated by patterning and through-etching another silicon wafer. The electrodes of the conductivity sensor were formed by thermally evaporating 20 nm Cr and 200 nm Au directly onto the membrane through the shadow mask. Another 500 μm thick silicon wafer was patterned and etched twice to form the cover plate; the first time to define a well to house the membrane and again to expose the tissue. The microfluidics, the nanoporous membrane and cover plate were then assembled. The medium is fed through the microfluidic channels via right-angled adapters integrated with Luer lock connectors (Fig. 11). The syringes are connected to these connectors by gauge 20 PTFE tubing with an internal diameter of 800 μm.
After the gel formation, the cellular and a-cellular hydrogels were washed with PBS. Then, cellular and a-cellular gels were integrated onto the microfluidic system. They were continuously fed with S-DMED with a rate of 0.8 ml/hr for 3 days. As control samples cellular and acellular hydrogels were placed in 6-well plates and were half immersed in S-DMED. In order to examine the cellular viability 3 days post encapsulation, sections of 1mm thickness were taken from the centre of alginate discs containing 3T3’s using a sterilized blade. The sections were immersed in 0.2 μl calceinacetoxymethylester (calcein-AM) for 15 min and 2.5 μl propidium iodide (PI) for 5 min in supplemented DMEM at 37C. The calcein-AM was cleaved to form calcein in the presence of esterases in live cells resulting in green fluorescence. In the case of dead-cells, the PI penetrated the cell and nuclear membrane and intercalated the DNA resulting in a red fluorescence. All samples were visualized using fluorescence microscopy.
Following this initial study the form of the microfluidic platform was modified to incorporate a PDMS cover layer that sealed the microfluidic channels and confined the cells to a well defined in the PDMS cover plate; resulting in a bio-hybrid tactile sensor. An exploded schematic diagram of the bio-artificial construct is shown in full in Fig. 10. It comprises a first layer based on a conductance sensor; a second layer composed of alginate gel in which cells are grown; a third PDMS layer that acts as covering and sealing of the structure and as interface with the external world. The cells are to be cultured and encapsulated within 5% alginate, a hydrogel with mechanical properties comparable to the dermal layer of human skin. The system of micro channels etched into the silicon substrate (described above) are designed to supply the cells with nutrients and maintain their long-term viability (Fig. 11). The PDMS effectively becomes the epidermis layer, possibly completed with finger print ridges, providing a robust surface to conduct experiments, while transmitting stresses and protecting the encapsulated alginate-fibroblast tissue. The PDMS also serves as a seal containing any excess fluid.
Fig. 10. Exploded schematic diagram of the bio-hybrid tactile sensor.
Fig. 11. Cross-section of the bio-hybrid tactile sensor.
A final iteration of the bio-hybrid sensor replaces the three conductivity sensors with an array of electrodes that provide improved spatial resolution of the tactile stimulus. This impedance sensor array is a ten channel linear array of electrode pairs interspersed with ground lines to minimise cross talk between the sensors. The electrodes consisted of tin plated copper conductors insulated between polyester based tape. Each electrode was 0.3 mm wide with a pitch of 0.5 mm. The array was fabricated by means of chemical processes applied to a 210mm flat flexible cable (FFC): a section of the FFC insulation layers was etched using a methyl trichloride/phenol solution at a ratio of 2:3 by weight at 110°C. This was done to expose a section of the conductors, nominally 18 mm long. This section was placed onto the nanoporous membrane in direct contact with the tissue. The end of the FFC is connected to the interface circuitry. One electrode of the sensing pair is connected to the input signal via a high power buffer amplifier to ensure the voltage remains constant should the resistance between the electrodes drop significantly. The input voltage was maintained as an AC signal with a frequency of 250 Hz and amplitude of 1 V. The other electrode is connected to the negative terminal of a transimpedance amplifier. This converts the current travelling between the electrode pairs into a voltage with a gain factor proportional (with negated sign) to the resistance value in the transimpedance circuit, in this case -10 kΩ. The output of all the transimpedance amplifiers were streamed directly into a DAQ and compared to the input signal in order to calculate the phase difference.
A custom sensor box was designed to hold the sensor and PDMS cover layer during tactile stimulation measurements. The silicone elastomer (PDMS) surface layer, which acts as a protective layer, was designed to cover the upper external side of the box containing the conductivity/impedance sensor. The PDMS cover is elastic, soft and gas permeable, thus allows cells’ respiration; moreover, the PDMS compatibility with cells was already proven. Different shapes and dimensions were designed for the PDMS cover: a simply flat layer and prototypes with ridges (Fig. 12).
Fig.12. (Left) PDMS cover biosensor: A. flat design; B. design with ridges; C. PDMS prototype; D. particular of the ridges geometry. (Right) Test assembly for biosensor.
The biosensor was integrated and tested with the tactile stimulation platform, which was also used for human touch studies (via electrophysiological and psychophysical methods). The platform (Fig. 13) was developed as a 2 DoF Cartesian manipulator, to indent and slide stimuli against the biosensor.
In order to test the biosensor, the complete system (upside-down) takes the place of the finger in human tests. The stimuli are attached to the bottom part of the dynamic platform, above the actuator, in order to be moved vertically along the indentation axis and translated horizontally along the tangential direction. Thus, the biosensor receives the same stimulation profiles that are applied to the human finger during electrophysiological and psychophysical passive-touch studies while using the platform and machine learning methods. During such dynamic passive-touch experiments the sensor is interfaced to the readout electronics in order to record impedance data during the test.
Fig. 13. Biosensor integrated in dynamic platform.
A typical data set is shown in Fig. 14, which was generated by sliding a cylindrical probe across the sensing area, retracting the stimulus and return to start location, and finally repeating the test. The two indentation phases are clearly visible in the figure. This behaviour was registered for all the different stimuli (cylinder and hemisphere) during the indentation tests. The data demonstrate that the biosensor clearly responds to the stimulus.
Fig. 14. Biosensor output data of sliding test for 2.5 mm single cylinder stimulus.
Simpler experiments were also carried out involving the indentation depth of a spherical probe, where the amplitude of the output signal was shown to increase according to the stimulation. This increase continues until the indentation depth is kept constant when it is observed that the signal appears to relax at an exponential rate for a time scale of several minutes. When the sphere is retracted and immediately indented to the previous depth, the signal is observed to decrease rapidly and then return to its former value at a rate consistent with the sphere speed. It is also observed that the phase of the signal appears to remain constant throughout the experiment. The increase in signal magnitude during indentation appears to increase with the sphere radius.
Information processing
The main objectives of information processing were to use machine learning techniques to analyse data produced from physiological and psychophysical measurements and virtual data from the finite element model of the finger pad. The aim was to increase the understanding of the causal and correlative relationship between different neural signals and behaviour metrics. In addition, the aim was to exploit such insights into the mechanisms underlying the encoding of sensory information to produce a believable and efficient active learning system for an artificial robotic finger.
In order to evaluate different information processing methods, data from the artifical tactile sensor were analysed. All methods were quite successful in classifying textiles but the weighted k‐nearest neighbor (kNN) method was more successful for gratings (Fig. 15)
Fig. 15. Results of several classification algorithms for passive touch with the artificial sensor on textile (top) and grating (bottom) datasets. The data provided to the classification algorithms consisted of the location of the highest peak in frequency spectrum obtained from a Fourier transform of the sensor measurements.
Using the artificial finger, active exploration driven by artificial curiosity was developed. The ultimate goal of the active robotic finger platform is to learn to infer the properties of the touched surfaces, for example, classify different materials. However, it is intractable to learn behaviour for finger movement directly from teacher signals provided for correctly identified surface properties. Instead, an intermediate learning stage is used during which a reinforcement learning (RL) algorithm learns to move the finger such that the sensors generate compressible signals. Compressing sensor signals implies finding regularities that allow for storing the signals in shorter form than the raw data. These regularities can arise from moving the sensor over a surface with corresponding regularities, for example regular gratings or textiles. Regular signals were considered that contained repetitive increases and decreases in voltage (e.g. a sine wave). Such signals will produce frequency spectra with clearly identifiable peaks. A significant part of the raw sensor data could be represented by the height and location of the peaks in frequency domain, which is requires much less storage than the original signal, and hence, the signal is compressible (e.g. Fig. 16). In contrast, random signals arising from sensor noise generated in the absence of sliding sensor-surface contact cannot be easily compressed. The reinforcement learning module is driven to generate motor command that lead to further learning progress of the compressor, allowing for quick learning of the relevant movements for detecting surface properties from the sensor data.
Fig. 16. From top to bottom: detrended sensor readings; frequency spectrum; regularity measure; Morlet-based wavelet transform of sensor readings, during a finger movement with sliding sensor-surface contact. Results are indicated for all sensors (from left to right) with four channels per sensor.
Integrating the machine learning methods with the drivers for the robotic platform was successful in initial experiments. The robotic finger learned how to move in order to pick up the most relevant (compressible) information from its sensor (Fig. 17).
Fig. 17. States of the robotic finger in the reinforcement learning experiments: (a) ‘extend’, (b) ‘flex’, (c) ‘extend touch’ and (d) ‘flex touch’. Only transitions between ‘extend touch’ and ‘flex touch’ and vice versa involve sensor-surface contact. The RL algorithm learns to move the finger from ‘extend touch’ to ‘flex touch’, involving the longest sensor-surface contact.
Applications
Microgravity
SKIN-B project is an ESA supported action joining the efforts of two research groups both targeting changes of human skin tissue during long term space mission. The project consist of two parts:
• Validation of skin physiological changes in space and skin as a model for other body systems.
• In vivo biomechanical measurements of human skin properties under accelerated aging conditions during ISS mission.
The first part is lead by Prof. Ulrike Heinrich (DermaTronnier GmbH & Co.KG and University of Witten-Herdecke) while the second part is the responsibly of the current project. There are two objectives to this combined study i.e. increasing the scientific knowledge of skin physiology and skin “aging” process and use those parameters in a skin mathematical model. The list of measurements include:
• Hydration-measurements
• Measuring the TEWL - transepidermal water loss
• SELS - Surface evaluation of living skin
• Microcirculation
• Ultrasound
• Pseudo biopsies (multiphoton tomography)
• Skin Elasticity
The main objective for the current project is to study the influences of the space environment on human skin by measuring changes in viscoelastic properties of the skin in-vivo before and after the ISS mission. The non-invasive experiment using the Cutometer device will be performed at different body sites (fingertips and volar side of the forearm) of selected astronauts before and after the ISS mission. The force-displacement versus time measurements of the epidermal layer of the skin will be recorded.
The Skin-B experiment will help to develop a mathematical model of aging skin and improve understanding of skin-aging mechanisms, which are accelerated in weightlessness. It will also provide a model for the adaptive processes for other tissues in the body. The project has started in 2013 and the first measurements were performed as presented in Fig. 18.
Fig. 18. Astronaut Luca Parmitano is performing SKIN-B measurements .
The Cutometer device has been extensively used also during the NanoBioTouch project. The experimental technique has been improved in order to include the suction tactile stimuli in microneurography experiments. The device has been modified to include synchronization.
The numerical model of suction test developed within the project simulates the Cutometer experiment. Although the complexity of the full 3D micro-macro model exceeds the requirements of the SKIN-B measurement protocol it can be used for evaluation of SKIN-B experimental results by utilizing the 3D macro-model part. The SKIN-B project experimental program has started and pre-flight and some on-board ISS measurements were performed.
Microstimulation of a sensory nerve in an upper limb amputated patient.
The possibility of evoking a sense of touch in the missing limb of a patient with an amputated upper limb was investigated even though at the time of testing the patient was 40 years old. As a result of a tumour growth in his right forearm, a trans-humeral amputation was performed in 2003. In early 2013, the patient went through surgery where electrodes were chronically implanted directly onto the muscles of the upper arm (epimysial electrodes) allowing the amputated patient to control the movements of the prosthetic limb. In the same surgical procedure, a cuff electrode was chronically attached around the ulnar nerve in the upper right arm (Fig. 19). The cuff electrode has three stimulation sites with a surface area of 1 mm2. From a knowledge of microstimulation during microneurography, the same technique in stimulating parts of the ulnar nerve through the cuff electrode was investigated. It was found that the stimulation of the ulnar nerve gave rise to similar sensations as experimental microstimulation, as during microneurographical microstimulation using intra-neural electrodes in healthy subjects. At lower stimulating currents, some of the sensations generated in the illusory/phantom hand of the patient were very similar to what was found when stimulating single mechanoreceptive afferents in the hand. For example, the patient could clearly indicate a sensation that was well localized to a small dot on the missing tip of a finger, and he had no problem indicating the precise location on a drawing of a hand. The sensation strictly coincided with the duration of the train of stimulation impulses delivered to the electrode. The current intensity at the threshold for this sensation was just above that normally used with intraneural electrodes. The character of the sensations resembled sensation commonly elicited when stimulating a single unit identified as an SAI (Merkel) or RA (Meissner) type afferent during microneurography. Increasing the stimulus intensity resulted in a spatial elongation of the sensation, as expected when more afferent nerve fibres are recruited.
Fig. 19. A close-up of the patient’s right arm during experiments with microstimulation through an implanted nerve cuff electrode. Attached to the upper limb is the preamplifier and shielded connectors for the signals to and from an integrated amplifier and isolated stimulator for human microneurography (not shown). The signals from the microneurography unit are connected to the implanted electrodes through a connector that is integrated in the osseointegrated titanium screw that can be seen to protrude from the end of the amputated upper limb. The scar on the lateral side of the patients’ arm is from the surgical procedure for implantation of epimysial (muscle) and nerve cuff electrodes, which was performed one month prior to the microstimulation experiment.
Bidirectional hand prostheses
Nanobiotouch also contributed to the advancements towards the restoration of natural sensory feedback in real-time bidirectional hand prostheses (Fig 20). The results were published by “Science Translational Medicine” journal, with the paper entitled “Restoring Natural Sensory Feedback in Real-Time Bidirectional Hand Prostheses”. The study was carried out in Rome and involved various members of the Nanobiotouch team of The BioRobotics Institute of Scuola Superiore Sant’Anna, Pisa, Italy. The paper describing the work had a high impact on media worldwide, and it was the result of a synergy between various projects and funding bodies:
- the EU Grant CP-FP-INFSO 224012 (TIME project);
- the project NEMESIS (NEurocontrolled MEchatronic hand prostheSIS) funded by the Italian Ministry of Health;
- the EU Grant FP7-611687 NEBIAS (NEurocontrolled BIdirectional Artificial upper limb and hand prosthesiS);
- the EU Grant FP7-NMP 228844 NANOBIOTOUCH (Nanoresolved multi-scan investigations of human tactile sensations and tissue engineered nanobiosensors);
- the Swiss National Science Foundation through the National Centre of Competence in Research Robotics.
Fig. 20. Bidirectional prosthetic hand during experimental evaluation in the LifeHand2 project, that involved NanoBioTouch researchers.
Biophysical and cognitive tests performed on packaging surfaces, textiles and cosmetic creams
The underlying hypothesis in this study was that the sensation of pleasantness by the human fingertips is associated with particular features in the temporal and spatial patterns of force (and temperature) produced during touching and rubbing. Hence the data from an artificial finger that can record such patterns should be analyzable in such a way as to predict pleasantness as assessed by sliding against a range of packaging surfaces, textiles and cosmetic creams. The artificial sensor response to different textures known for eliciting different levels of (un)pleasantness were analysed and relevant and meaningful differences in the temporal and spatial data were identified. In the temporal data, differences in the stick-slip behaviour between materials were found and in the mean amplitude and fluctuations in the data that relate to friction. The data from each sensor were also transformed into a neuromorphic code, i.e. spikes that emulate the coding mechanisms that are observed in human mechanoreceptors (Fig. 21). This allowed the data to be analysed using the information processing techniques developed in the project.
Fig. 21. Each plot represents the average response across trials in each channel to different textures (indicated by different colours, see legend) for a given speed (indicated by the title of the plot expressed in mm/s). Error bars indicate the whole range of the response across trials.
In the spatial data, the spike trains generated by the different sensors were examined. The first attempts at decoding the texture identity were hypothesis-free, i.e. testing if there were some features in the responses able to discriminate between different realistic textures like those under investigation. Note that most studies of this kind involve regular geometrical patterns so the current study was already breaking new ground. Reliable discrimination of the commercial textures was achieved. On this basis, it was possible to identify more advanced measures of discrimination for future work.
Potential Impact:
Innovative S&T research
Tissue engineered mechanoreceptor cells have been isolated, stimulated and interfaced to measure the neural responses, which represents a significant advance for nanobiosensors and also for studies of the molecular basis of mechanotransduction at the nano-scale. This understanding has been underpinned by the development of micro-anatomic mathematical models of mechanoreceptor afferents that allow the relationship between their deformation behaviour during tactile contact and the evoked action potentials forming a spike train. The deformation characteristics of the afferents were computed by developing a micro finite element model that was coupled to a macro finite element model of a finger pad. The spike trains were similar in form to those measured directly using microneurography. These latter measurements showed that surface texture is strongly coded by modulation of the activity of both RA and SAI afferents. Further understanding of the human tactile system was derived by brain imaging that strongly suggested that it is a bottom up process related to the activation of input from single afferents. A novel unidimensional scale has been developed for ranking the pleasantness of surfaces. The friction of the finger pad is a key factor and significant advances have been made in understanding the tribological mechanisms involved. Novel machine learning algorithms have allowed information, which was obtained from the micro-macro model, microneurography and the artificial finger that was developed, to be used to classify surfaces and assess their pleasantness. Thus the work has made major progress in defining and understanding affective touch.
Innovations in European industry:
The project has generated substantial innovations for European industry in the areas of security printing, packaging, consumer goods, biomedical measurement equipment and robotics. Security printing has been developed that allow visually impaired people to read authentication information, for example. The packaging and consumer goods industries will be able to exploit the improved understanding of affective touch by the possibility to identify beneficial combinations of tactile properties that will increase customer satisfaction. This will be assisted by novel equipment that has been developed for microneurography and tactile system characterisation. The fundamental investigations of mechanoreceptors and skin layers are expected to lead to innovations in nanobiosensors, biomimetic materials, tribological systems.
Strengthening partnerships with European industry
Strong collaborations have been developed between the academic and industrial partners. As result of the current project a new EU funded project has been successfully started that exploits the understanding of the human tactile system to develop touch screens with haptic feedback. The companies in NanoBioTouch include the Anglo-Dutch company Unilever plc, one of the largest consumer products companies in the world with global annual sales of more than €40 billion to 150 countries. In addition, there are four high-tech SME companies: C3M specialises in the development of the most advanced virtual M5 models for sensitivity analyses, inverse modelling and optimisation, CK-electronics is the world market leader in producing skin testing equipment for clinical diagnosis in dermatology as well as cosmetic, pharmaceutical, raw material industries; Rockfield Software Limited is world leader in the development of advanced Finite and Discrete Element computer systems; and Optaglio is a high-tech SME specialising in security printing and brand protection with markets in more than 30 countries.
Economic and societal impacts
The project will contribute to the emerging appreciation of the importance of touch and tactile experience for emotions, well-being, social interaction between con-specifics, and normal development. The improved understanding of human touch and details of mechanotransduction will provide valuable insights to those tactile scenarios that yield pleasant emotions and have beneficial social effects. Enriched tactile perception of consumer products can have a decisive impact on the success of the products and bring considerable added value and economic benefits, especially when applied to the large markets of packaging, home and personal care sectors, for example. The under-exploited potential of the tactile dimension of products represents a strong potential for a step-change innovation rather than just incremental improvements, and this will assist industry to deliver improved benefits that consumers demand, leading to maintained or increased market share. Another large size market relevant to the results of the project is the food and beverage packaging industry with annual turnover of around 24 billion USD. A massive 235 billion beverage cans are produced annually worldwide (45 billion cans in Europe). The trends in the packaging industry are determined by the requirements for individualism, flexibility, complexity, specialisation and new product experiences where important drivers are lifestyle, indulgence, wellness, health, functional and emotional added value, convenience and cost efficiency. A better understanding of how tactile perception is influenced by the physical properties of products will contribute significantly to multi-objective optimisation of packaging products. Due to the S&T breakthroughs of this project Optaglio, CK electronics, Rockfield Software and C3M will enhance their leading positions in specialised markets as well as enlarging their business network to newly emerging areas of converging technologies. Finally, the development of an artificial finger with sophisticated touch capabilities will have an enormous economic and societal impact on prosthetics and robotics that will assist disabled and elderly people to lead more independent lives and improve their quality of life.
Impact on medical rehabilitation
The fundamental understanding of touch in the project will have beneficial effects on tactile stimulation practices that are used in medical and psychological rehabilitation. Touch is a key part of physical examination procedures in medicine. As a sensory system, touch is prone to a variety of peripheral and central neurological disorders and new approaches to the diagnosis and treatment of such disorders may be expected from the improved scientific understanding and the innovation embodied in this project. As a very robust and widely distributed sensory system, touch is also very important in its potential to offer substitute information pathways to compensate for the loss of other sensory systems. This may be particularly true in neurologically impaired and elderly people in whom sensory substitution by touch may be less efficient. The focus on affective as well as discriminative touch has the potential for significant impact on understanding contributions to emotional well-being, which can be critical for effective medical intervention. Touch also plays an important role in the wider domain of patient-carer relations, including for instance nursing and physiotherapy. Thus stimulation of tactile pathways is an important adjunct to physiotherapy for the treatment of muscle weakness following stroke. Tactile stimulation is also used by physiotherapists in the rehabilitation of peripheral sensory impairment, for instance arising from median nerve damage in accidents involving falls onto glass. There are also examples of therapies, complementary to medicine, based around touch. For example, healing touch (HT; www.healingtouchprogram.com) is a complementary therapy fostering nurse-patient connection. Nurses are beginning to use HT with their patients to assist in easing pain and anxiety, promote relaxation, accelerate wound healing, diminish depression, and increase a patient's sense of well-being. The research may be expected to support new studies and clinical tests in several of these areas. According to the World Health Organization (WHO), in 2002 an estimated 161 million people worldwide were visually impaired, of whom 124 million people had low vision and 37 million were blind. The innovative approaches to generate security printing will assist visually impaired people in many ways. The improved understanding of human touch especially the exploratory process in discriminative touch has the potential to help develop rehabilitation methods for patients with strokes and peripheral nerve injuries. The insights gained from studying autistic subjects should assist in diagnosing the condition and developing improved therapies.
Main dissemination actives
The main dissemination channels were by journal publications, the website (http://nanobiotouch.org/) and the mass media, which include the following:
• Video interview released to “E se domain”, a TV Show on the Italian national Television Rai3, which discusses the impact of modern technology to life. The interview will be transmitted on TV in 2013.
• Video interview released to “TG3 Pixel”, a Science TV Show on the Italian national Television Rai3 (October 13, 2012): http://www.rai.tv/dl/RaiTV/programmi/media/ContentItem-e9388b11-ad52-4ab2-b51c-962b1f1c0fcd-tg3.html
• Interview released to "ilSole24Ore" (first page of the weekly supplement "Nòva", dealing with science and technology) about Nanobiotouch: http://www.ilsole24ore.com/art/tecnologie/2012-03-24/sesto-senso-robot-121457.shtml?uuid=AbiYLQDF
• A purposely-edited video (including UG, SSSA and Nanobiotouch logos) on joint UG-SSSA research activities is shown in the on-line version of the article: http://www.ilsole24ore.com/art/tecnologie/2012-03-24/sesto-senso-robot-121457.shtml?video&uuid=AbiYLQDF
• Several images and pictures resulting from the Nanobiotouch project are shown in the on-line version of the article: http://www.ilsole24ore.com/art/tecnologie/2012-03-24/sesto-senso-robot-121457.shtml?foto&uuid=AbiYLQDF
• A long report by the “Crash” programme of the Italian National Television Rai3, dealing with hybrid-bionic systems
• Particularly, results of the NMP-Smarthand and of the NMP-Nanobiotouch projects, among others, were shown: http://www.sssup.it/webtv_channel_rass_stampa.jsp?id_context=28631&ID_LINK=10156&area=193 The full programme is available via the website of the Rai Italian National Television: http://www.crash.rai.it/sito/scheda_puntata.asp?progid=1545
In addition Prof. Maria Chiara Carrozza at released an interview within the programme “Unomattina” of the principal Italian public television RAI1 that included some results of NanoBioTouch. During the interview, Prof. Carrozza was supported by Dr. Calogero Maria Oddo and Mr. Marco Controzzi, who contributed to NanoBioTouch, and reported and demonstrated the research activities carried out at The BioRobotics Institute of the Scuola Superiore Sant’Anna. The interview is online in the RAI website: http://www.rai.tv/dl/RaiTV/programmi/media/ContentItem-2076577d-f7f4-4a11-bc53-d8337b53abbe.html
Screenshots of the interview released by Prof. Maria Chiara Carrozza within the programme showing the artificial finger pad (left).
The journal publications and communications at conferences are listed below:
Journal Papers
Tactile discrimination threshold is unrelated to tactile spatial acuity
X. Libouton, O. Barbier, L. Plaghki, J-L.Thonnard
Behavioral Brain Research 208, 473-478, (2010)
Finger pad friction and its role in grip and touch
MJ Adams, SA Johnson, P Lefevre, V Levesque, V Hayward, T Andre, J-L Thonnard
Journal of the Royal Society interface, 10, 20120467 (2013)
Fingertip Moisture is Optimally Modulated during Object Manipulation
T. André, P. Lefèvre, J-L. Thonnard
J. Neurophysiol. 103 , 402-408, (2010)
Optimised determination of viscoelastic properties using compliant measurement system
JW Andrews, J Bowen, D Cheneler
Soft Matter 2013, 9, 5581
This paper received the award for best submitted publication at the University of Birmingham, June 2013
Alginate hydrogel has a negative impact on in vitro Collagen 1 deposition by fibroblasts
AM Smith, NC Hunt, RM Shelton, G Birdi, LM Grover
Bio Macromolecules 13, 4032-4038 (2012)
Development and characterisation of a bio-hybrid skin-like stretchable electrode
E. Buselli, AM Smith, LM Grover, A Levi, R Allman, V Mattoli, A Menciassi, L Beccai
Microelectronic Engineering, 88, 1676-1680, 2011
Friction of the human fingerpad; influence of moisture, occlusion and velocity
SM Pasumarty, SA Johnson, SA Watson, MJ Adams
Tribology Letters 44, 117-137, (2011)
Effect of skin hydration on the dynamics of fingertip gripping contact
T. André, V. Lévesque, V. Hayward, P. Lefèvre, J-L. Thonnard
J. R. Soc. Interface 8, 1574–1583, (2011)
Active touch sensing
Tony J Prescott, Matthew E Diamond, Alan M Wing
Phil. Trans. R. Soc. B 366, 2989-2995 (2011)
Light touch for balance: influence of a time-varying external driving signal
Alan M Wing, Leif Johannsen. Satoshi Endo
Phil. Trans. R. Soc. B 366, 3133-3141 (2011)
Learning to touch through active exploration.
L. Pape, C.M. Oddo, M. Controzzi, C. Cipriani, A. Förster, M.C. Carozza, J. Schmidhuber.
Frontiers in NeuroRobotics, 2012, 6(6).
Texture-induced vibrations in the forearm during tactile exploration
B. Delhaye, V. Hayward, P. Lefèvre and J‐L. Thonnard
Front. Behav. Neurosci. 6:37. doi: 10.3389/fnbeh.2012.00037 (2012)
Roughness encoding in human and biomimetic artificial touch: spatiotemporal frequency modulation and structural anisotropy of fingerprints.
C.M. Oddo, L. Beccai, J. Wessberg, H. Backlund Wasling, F. Mattioli, M.C. Carrozza.
Sensors, 11, 5596-5615 (2011)
Roughness Encoding for Discrimination of Surfaces in Artificial Active Touch.
C.M. Oddo, M. Controzzi, L. Beccai, C. Cipriani, M.C. Carrozza.
IEEE Transactions on Robotics, 27, 522-533 (2011)
Mechatronic Platform for Human Touch Studies
C.M. Oddo, L Beccai, N. Vitiello, H Backlund Wasling, J Wessberg, M.C. Carrozza.
Mechatronics, 21, 604-613 (2011).
Calcium-alginate hydrogel-encapsulated fibroblasts provide sustained release of vascular endothelial growth factor. NC Hunt, RM Shelton, DJ Henderson, LM Grover. Tissue Engineering Part A, 2012
Synthetic and bio-artificial tactile sensing: A Review
C Lucarotti, CM Oddo, N Vitiello, MC Carozza. Sensors, 2012
Degradation of polysaccharide hydrogels in cell culture conditions
SH Jahromi, LM Grover, JZ Paxton, AM Smith
Journal of the Mechanical Behaviour of Biomedical Materials 2012
Rasch-built measure of pleasant touch through active fingertip explorations
A. Klöcker, C. Arnould, M. Penta, J-L. Thonnard
Front. Neurorobot. 6:5. doi: 10.3389/fnbot.2012.00005 (2012)
Tactile roughness discrimination of the finer pad relies primarily on vibration sensitive afferents not necessarily located in the hand
X Libouton, O Barbier, Y Berger, L Plaghki, J-L Thonnard, Behavioural Brain Research 229, 273-279 (2012)
Physical factors influencing pleasant touch during tactile exploration
A Klocker, M Wiertlewski, V Theate, V Hayward, J-L Thonnard, PLoS One, November 2013, Vol 8, Issue 11, www.plosone.org
Restoring natural sensory feedback in real-time bi-directional hand prosthesis
Science Transitional Medicine, Vol 6, Issue 222, Feb 2014
Communications at conferences
Finger pad tribology: grip function and tactile perception
M J Adams, S A Johnson, P Lefèvre, V Lévesque, V Hayward, T André and J-L Thonnard
37th Leeds-Lyon Symposium on Tribology, Leeds, September 2010
Tactile discrimination threshold is unrelated to tactile spatial acuity
X. Libouton, O. Barbier, L. Plaghki, J-L.Thonnard
18th Brussels Hand-Upper Limb International Symposium, Brussels, Belgium, January 2010
The role of moisture in the optimization of stick-to-slide dynamics in the human grip.
T. André, V. Lévesque, V. Hayward, P. Lefèvre, J-L.Thonnard
20th Annual Meeting of the Neural Control of Movement Society, Naples, USA , April 2010
The motor control of haptic perception: getting more out of it than what you put into it.
A. Smith, V. Hayward, G.Loeb J-L.Thonnard
20th Annual Meeting of the Neural Control of Movement Society, Naples, USA April 2010
Optimal modulation of fingertip moisture during object manipulation.
J-L.Thonnard University College of Dublin, Health Sciences Centre, Dublin, Ireland, September 2010
Optimal modulation of fingertip moisture during object manipulation.
J-L.Thonnard , T. André, P. Lefèvre
40th Annual Meeting of the Society for Neuroscience, San Diego, USA November 2010
Textural vibrations in the forearm during tactile exploration
B. Delhaye, V. Hayward, P. Lefèvre, J-L.Thonnard
40th Annual Meeting of the Society for Neuroscience, San Diego, USA November 2010
A comparison of the frictional behaviour of human glabrous skin with synthetic glassy polymers and elastomers
M. J. Adams and S. A. Johnson
International Conference on BioTribology, Imperial College London, September 2011
Bio-hybrid tactile sensor for the study of the role of mechanoreceptors in human tactile perception
D. Cheneler a*, M. C. L. Ward a, C. J. Anthony a, 37th International Conference on Micro and Nano Engineering, MNE 2011 19-23 September 2011, Berlin, Germany
Poster presentation at the Active touch sensing meeting at the "Kavli Royal Society Centre", February 2011
Pleasant touch and the role of the fingertip moisture level, A Klöcker, C Arnould, M Penta, JL Thonnard
Poster presentation at the Active touch sensing meeting at the "Kavli Royal Society Centre", February 2011
Vibrations in the forearm during active touch of rough textures, B Delhaye, V Hayward, P Lefevre, JL Thonnard
Fingerpad sensitivity is not required for Tactile Roughness Discrimination
X. Libouton, O. Barbier, L. Plaghki, J-L.Thonnard 19th Brussels Hand-Upper Limb International Symposium, Brussels, Belgium, 2011
Rasch built measure of pleasant touch elicited through active exploration
A. Klöcker, C. Arnould, M. Penta, J-L. Thonnard, 19th Brussels Hand-Upper Limb International Symposium, Brussels, Belgium, February 2011
Textural vibrations in the forearm during tactile exploration
B. Delhaye, V. Hayward, P. Lefèvre, J-L. Thonnard
19th Brussels Hand-Upper Limb International Symposium, Brussels, Belgium, February 2011
Brain Imaging and touch
HF Kwok, Presentation given during Science Week, Moseley, Birmingham, UK, 2011
Novelty-based restarts for evolution strategies
G Cuccu, F Gomez, T Glasmachers, 2011 IEEE Congress on Evolutionary Computation (CEC2011)
Modular deep belief networks that do not forget
L Pape, F Gomez, M Ring, J Schmidhuber, International Joint Conference on Neural Networks, 2011 (IJCNN2011)
Vibrotactile sensory substitution in multi-fingered hand prostheses: Evaluation studies
M D’Alonzo, C Cipriani, MC Carrozza
Proc of International Conference on Rehabilitation Robotics, Zurich, Switzerland, June 2011 (ICORR2011)
Tactile detection of slip. Fine characterization of skin deformation during the onset of slip.
B. Delhaye, P. Lefèvre, J-L Thonnard
Brussels Hand/Upper Limb International Symposium, Brussels, Belgium January 2012
Tactile detection of slip: Fine characterisation of skin deformation during the onset of slip
B Delhaye, P Lefevre, J-L Thonnard, Poster presented at “NCM” Congress, Venice, Italy, April 2012
Bio-hybrid tactile sensor and experimental set-up for investigating and mimicking the human sense of touch
D Cheneler, E Buselli, CM Oddo, G Kaklamani, L Beccai, MC Carrozza, LM Grover, CJ Anthony, MC Ward, MJ Adams. Workshop on “Advances in tactile sensing and touch based human-robot interaction”, Boston, USA, March 2012
Physical properties that contribute to roughness discrimination of textures with randomly distributed asperities
M D’Alonzo, N Vitiello, L Beccai, HF Kwok, CM Oddo, AM Wing, MC Carrozza. Workshop on “Advances in tactile sensing and touch based human-robot interaction”, Boston, USA, March 2012
Soft-neuromorphic artificial touch for applications in neuro-robotics.
G. Spigler, C.M. Oddo, M.C. Carrozza. IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob 2012), Rome, Italy, June 2012
Physical factors influencing pleasant touch during tactile exploration
A Klocker, M Wiertlewski, V Theate, V Hayward, J-L Thonnard
Poster presented at Annual Meeting of the Society for Neuroscience, New Orleans, USA, October 2012
Functional micro-anatomical finite element model of Meissner corpuscle
T Vodlak, Z Vidrih, P Pirih, J Presern, T Rodic, Abstract submitted to Eurohaptics, Paris 2014
Exploitation of results
• Use of an array of microelectrodes on a silicon substrate providing real-time information on cell response during the application of applied loads. This would involve integration of the bio-hybrid tactile sensor and of the machine learning algorithms in the dynamic platform that was developed.
• An element to be developed that is visually flat, but produces a quickly recognizable 3D effects when touched. Applications would be mainly in security printing companies particularly aimed at the visually impaired.
• Development of an alginate/S-DMEM hydrogel for use in wound management targeted at the health care market.
Exploitable results that would be of interest to large retailers, large packagers ‘high-tech’ markets of security printing, biomedical measurement equipment, virtual modelling software companies, healthcare companies:
• Development of a bio-inspired robotic finger with an adequate number of DoFs and actuators (degrees of mobility) to allow for exploratory touch.
• Nanoscale sensitivity of taction and concomitant advances in signal processing. This would involve integration of the NEMS tactile array and of the machine learning algorithms in the dynamic platform that was developed.
• Development of a demonstrator of exploratory touch using the biorobotic finger with the embedded NEMS tactile array. This would involve integration of the NEMS tactile array and the machine learning algorithms in the biorobotic finger.
• Design and fabrication of a two-degree-of-freedom platform, with integrated thermal cycle stimulation control. Target commercial sectors include instrument manufacturers, security printing, packaging, consumer goods, biomedical measurement equipment and robotics.
Conclusive remarks
In addition to the patents arising from the project, there are a number of potential foreground applications that are wide ranging and include enhancing the tactile dimension of personal care products, diagnosing Carpal Tunnel Syndrome, restoration of touch in amputees, robotic fingers with NEMS and bio-hybrid tactile sensing that incorporate machine learning a demonstration platform for the robotic finger, and a 2D tactile mechanical and thermal stimulator for studying in vivo tactile responses. These applications are summarised in the table below together with their Technological Readiness Level.
Exploitable product or measure. Technology Readiness Level Description Supporting Information
Potential to enhance the tactile dimension of personal care products, food and beverage,smart packaging, security printing,medical rehabilitation sectors 1. Basic principles observed and reported In a first step, a Pleasant Touch Scale has been elaborated. This scale defines reliable and objective quantification of the (i) pleasantness levels of 37 everyday life materials (fabrics, wood, wax, latex, paper,…) and (ii) pleasantness satisfaction levels of 200 subjects, induced by exploration of the 37 surfaces have been determined (Klöcker et al. 2012). In a second step these pleasantness levels could be significantly related to various surface physical properties (such as the surface topography) (Klöcker et al. 2013).
These results are particular of interest for various industries using, for example, the “tactile branding marketing strategy”. The aim of this strategy is to generate a “connection” between the consumer’s emotional feeling and the brand through a stimulation of the tactile sense. Indeed, what a product feels like can influence people’s decision on buying this product (Spence and Gallace 2001). As a consequence, it is of importance to extract factors intervening with the perceiver’s “tactile evaluation” of specific objects. For example, the textile industry typically aims to create comfortable and pleasant products, inducing the consumers to buy these products. Other examples of industries which might be interested by the results described in the studies of Klöcker et al. (2012, 2013) are the electronic industry and the cosmetic industry. With respect to the electronic industry, it is interesting to note that they use more and more the tactile branding to promote their products (e.g. the Apple iPod can easily be recognized by only touching it). The cosmetic industry tries also to elaborate, for example, hand lotions after the application of which consumers have a smooth tactile sensation.
The results might also be of importance for the health care sector. Indeed, the affective modality of the sense of touch is completely ignored in sensory assessments of pathologies affecting touch. This ignorance is probably due to the fact that no objective scale existed to measure pleasant touch. Klöcker A, Arnould C, Penta M and Thonnard JL (2012) Rasch-built measure of pleasant touch through active fingertip exploration. Front Neurorobot 6, 1-9.
Klöcker A, Wiertlewski M, Théate V, Hayward V and Thonnard JL (2013) Physical factors influencing pleasant touch during tactile exploration. PLoS ONE 8, e79085.
Friction test to diagnose Carpal Tunnel Syndrome and other medical conditions. 2. Technology concept and application formulated Finger friction measurements on relatively smooth surfaces have been shown to be very sensitive to skin moisture. Carpal Tunnel Syndrome causes some fingers to not sweat as much as others. Hence comparative tests should aid diagnosis. Could be extended to support diagnosis and monitoring of other conditions that effect skin moisture such as diabetes, xerosis, hyperhidrosis and anxiety disorders. Governing principles described in
http://dx.doi.org/10.1098/rsif.2012.0467
Application being considered:
http://dx.doi.org/10.1177/1753193408087231
Tactile perceptions from stimulation of single afferents for restoration of touch in amputees 1. Basic principles observed and reported Using the techniques of microneurography and microstimulation, we stimulated single, identified tactile receptors in human healthy volunteers. This gives rise to discrete and highly repeatable sensations of pressure (from SAI afferents) or a sense of vibration (RA, Pacinian afferents). Variations in stimulus frequency results in graded variations in the intensity of the sensations, and variations in stimulus patterns are clearly distinguishable. The corresponding brain activity can be studied using functional imaging techniques.
These basic scientific findings are directly applicable for experiments on replacement of tactile sensory function in arm and hand amputees, using implanted neural electrodes. Initial experiments have been initiated. Importantly, the aim is to go beyond crude detection of impact or force intensity, but aim towards replacement of the discriminatory details of tactile sensations. Results reported at the Society for Neuroscience (USA) meeting:
2012, Paper 677.26:
R Ackerley et al. Microstimulation of single mechanoreceptor afferents in humans.
2013, Paper 465.17:
R Ackerley et al. EEG activity evoked from microstimulation of single mechanoafferents in humans.
2013, Paper 644.23:
FP McGlone et al. Intraneural microstimulation of somatosensory afferents during fMRI at 7T.
Presentation at Janelia Conference, Howard Hughes Medical Institute, “Mammalian Circuits Underlying Touch Sensation” 22-25 Sept. 2013. R. Ackerley, Microstimulation of single mechanoafferents in humans.
Robotic finger 6. System demonstration in a relevant environment. The robotic finger is part of a long-term scientific and technological strategy, implemented with the federation of various national and European research projects, that is driving towards the integration of an artificial sense of touch in multi-fingered robotic hands for main use in industrial manipulation and upper limb prosthetics.
Implementation of a neuromorphic architecture mimicking the coding of tactile information by human mechanoreceptors.
The integrated system has been experimented in a controlled operational environment (i.e. in a clinical facility) in the framework of a pilot study with a transradial hand amputee. A high impact journal publication (in Science Translational Medicine) related in part to the integrated system has been achieved, and other high-impact publications are being prepared.
S. Raspopovic, M. Capogrosso, F. M. Petrini, M. Bonizzato, J. Rigosa, G. Di Pino, J. Carpaneto, M. Controzzi, T. Boretius, E. Fernandez, G. Granata, C. M. Oddo, L. Citi, A. L. Ciancio, C. Cipriani, M. C. Carrozza, W. Jensen, E. Guglielmelli, T. Stieglitz, P. M. Rossini, S. Micera, Restoring Natural Sensory Feedback in Real-Time Bidirectional Hand Prostheses. Sci. Transl. Med. 6, 222ra19 (2014). http://stm.sciencemag.org/content/6/222/222ra19
Integration of the NEMS tactile array and of the machine learning algorithms in the Dynamic Platform 6. System demonstration in a relevant environment. The integrated system has been experimented in a controlled operational environment (i.e. in a clinical facility) in the framework of a pilot study with a transradial hand amputee. CM Oddo, L Beccai, J Wessberg, H Backlund Wasling, F Mattioli, MC Carrozza, Roughness encoding in human and biomimetic artificial touch: spatiotemporal frequency modulation and structural anisotropy of fingerprints. Sensors, 11, 5596-5615 (2011). http://www.mdpi.com/1424-8220/11/6/5596
G Spigler, CM Oddo, MC Carrozza, Soft-neuromorphic artificial touch for applications in neuro-robotics. IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob 2012), Rome, Italy, June 2012. http://ieeexplore.ieee.org/xpl/abstractAuthors.jsp?arnumber=6290849
Intellectual property rights have been raised with the following patent application claiming specific technological solutions. PCT/IB2014/060223 patent application is an extension of a previous Italian national application that obtained positive responses to all claims.
G. Muscolo, C.M. Oddo, L. Beccai, M.C. Carrozza, Method and Device for Making Fingerprints Associated to Artificial Fingers. PCT Application Number PCT/IB2014/060223.
Integration of the NEMS tactile array and of the machine learning algorithms in the biorobotic finger 5. Component validation in relevant environment. Experiments of the integrated actuated biorobotic system have been carried out under realistic scenarios (e.g. discrimination of materials and textures, softness coding) in controlled conditions emulating daily life activities and with the implementation of machine learning methods (artificial curiosity). Experiments involving sensorimotor loops with neuronal control architectures have been carried out. L Pape, CM Oddo, M Controzzi, C Cipriani, A Förster, MC Carozza, J Schmidhuber, Learning tactile skills through curious exploration. Frontiers in NeuroRobotics, 6:6 (2012). http://journal.frontiersin.org/Journal/10.3389/fnbot.2012.00006/full
CM. Oddo, M Controzzi, L Beccai, C Cipriani, MC Carrozza. Roughness encoding for discrimination of surfaces in artificial active touch. IEEE Transactions on Robotics, 27, 522-533 (2011). http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5740988&tag=1
G Spigler, CM Oddo, M Controzzi, D Camboni, C Cipriani, MC Carrozza. Haptic sensing in a robotic hand, Workshop Presentation at the IEEE World Haptics Conference, Daejeon, South Korea, April 14th 2013.
Intellectual property rights have been raised with the following patent application claiming specific technological solutions.
G. Spigler, C.M. Oddo, D. Camboni, M.C. Carrozza, Metodo per trasmettere sensazioni tattili ad un utente e apparecchiatura che attua tale metodo (Method and apparatus for transmitting tactile sensations to an user). Italian Patent Application Number PI2013A000028.
D6.4 describes the results of tests on some representative packaging materials, cosmetics and textiles. Some further optimisation and validation of the data analysis procedures is necessary before appropriate machine learning algorithms could be integrated for routine use in product improvement scenarios.
An artificial bio-hybrid integrated system having an enhanced level of integration of the control, actuation and tactility that will provide affective as well as a discriminatory exploratory touch capability. 3. Analytical and experimental critical function and characteristic proof of concept. The feasibility of the bio-hybrid tactile sensing technology has been verified through experiments in controlled laboratory conditions with the dynamic tactile stimulation platform. C Lucarotti, CM Oddo, N Vitiello, MC Carozza, Synthetic and bio-artificial tactile sensing: a review. Sensors 13, 1435-1466 (2013). http://www.mdpi.com/1424-8220/13/2/1435
D Cheneler, E Buselli, CM Oddo, G Kaklamani, L Beccai, MC Carrozza, LM Grover, CJ Anthony, MC Ward, MJ Adams, Bio-hybrid tactile sensor and experimental set-up for investigating and mimicking the human sense of touch. Workshop on “Advances in tactile sensing and touch based human-robot interaction”, Boston, USA, March 2012
D Cheneler, MCL Ward, CJ Anthony , Bio-hybrid tactile sensor for the study of the role of mechanoreceptors in human tactile perception. 37th International Conference on Micro and Nano Engineering, MNE 2011 19-23 September 2011, Berlin, Germany.
D5.4 presents the relevant achievements. Other scientific publications are in progress.
The demonstrator consists in active and passive touch tasks performed consecutively by using different tactile stimuli. 6. System demonstration in a relevant environment. The system has been demonstrated in controlled scenarios during scientific events as well as during open-days involving society at large. G Spigler, CM Oddo, M Controzzi, D Camboni, C Cipriani, MC Carrozza. Haptic sensing in a robotic hand, Demonstration and Workshop Presentation at the IEEE World Haptics Conference, Daejeon, South Korea, April 14th 2013. http://www.haptics2013.org/download/WHC2013_Final_Program.pdf
Public demonstrations with citizens during open days at SSSA, as detailed in D8.2 http://sssa.bioroboticsinstitute.it/events/sanfaustino13
A two-degree-of-freedom tactile mechanical stimulator able to stimulate a human (or an artificial) fingertip along the tangential and normal axis with different surfaces and materials, with integrated thermal cycle stimulation control, to be used in touch experimental paradigms with both human subjects and artificial/hybrid tactile sensors 6. System demonstration in a relevant environment. Mechanical and thermal stimulator deployed to a number of project partners and used as experimental tool for human and artificial touch studies. Particularly of interest to industries concerned with improving the feel of metallic components, where sensory cues associated with heat transfer are important, through coating or surface texturing. CM Oddo, L Beccai, N Vitiello, H Backlund Wasling, J Wessberg, MC Carrozza, A mechatronic platform for human touch studies. Mechatronics, 21, 604-613 (2011). http://www.sciencedirect.com/science/article/pii/S0957415811000456
D3.3 describes the design and operation of the equipment. Particularly well-suited to thin thermally conductive samples whose surface temperature can be rapidly altered using the Peltier devices.
Artificial curiosity software. 2. Technology concept and application formulated The machine learning methods developed and applied in NanoBioTouch will become increasingly useful, as companies begin to deploy robotic and neuroprosthetic systems. The principle of artificial curiosity, was shown to enable a robotic finger to actively explore its environment, such that it could better discriminate between the types of surface textures it was likely to encounter, without the need for any pre-programmed information. This is a critical requirement if robots are to be trusted to behave autonomously in complex and changing environments. Learning tactile skills through curious exploration
L Pape, CM Oddo, M Controzzi, C Cipriani, A Förster, MC Carozza, J Schmidhuber
Frontiers in NeuroRobotics, 6:6 (2012).
List of Websites:
www.nanobiotouch.org contact Kathleen Hynes, k.m.hynes@bham.ac.uk
