Final Report Summary - NOMS (Nano-optical mechanical systems)
Project Context and Objectives:
WP1 System specification and risk assessment
Parameters such as speed requirements for specific information display (semi-permanent or fast refreshed display) and the minimum upward force the material must sustain as well as early designs will be identified and discussed in WP1. A tentative design will be agreed on. Contingency plans will also be drafted on the tablet and blister design by considering two or three alternatives, identifying risks to maximize success. Establishing a common base-line will promote early identification of technical problems that could be encountered later on.
WP2 Characterization and optimization of photoactive nanomaterials
WP2 is aimed at developing the CNT polymer composite materials for use in the NOMS tablet, with the parallel development of photo-actuating nematic elastomers as a fall-back option.
Fundamental studies of CNT photo-actuation:
Initial effort in WP1 will identify optimal material combinations within the fields of polymers and nanocomposites. Proper optical actuation is based on a reversible (equilibrium) shape-memory effect at the molecular level. In this shape-memory effect, the actuation is caused by individual tubes in polymer-carbon composite responding to light. The project will undertake a fundamental study of CNT photo-actuation using real-time TEM/AFM methods in order to understand the mechanism in more detail. This will provide a further exploitable outcome of the project.
Functionalisation of CNTs, dispersion and cross-linking:
One key task is to prepare a long polymer : CNT composite, operating within a temperature range of 0 and 40?°C and over hundreds of actuating cycles without significant degradation. The first two requirements limit the choice of polymer matrix to low-Tg UV-resistant materials, while the elastic repeatability demands cross-linking of the polymer. The choice of CNT is important for scaling-up the production in the future, and will therefore focus on multi-walled CNTs, the large reported sales of which suggest that they are becoming the industry standard.
As part of the development, a standard CNT dispersion procedure will be established. Although this is done in dozens of research group around the world, there is still no standard process, which could be quantified and reproduced. Specially developed surfactants will be used, to ensure high affinity of the CNT surface, with a long flexible moiety tailored to the specific polymer matrix to be dispersed. The effect of CNT length on photo-mechanical response in an aligned polymer composite has never been studied. In addition, not only can the length distribution of the CNTs vary greatly, but also there could be some CNT breaking during sonication. Therefore, the CNT length distribution will be determined using a new DLS technique available at UCAM.
After mixing of functionalized CNT in polymer melt (and alignment, see below), the cross-linking has to be initiated to produce a mechanically stable elastic rubber nanocomposite. Ideally this could be achieved by tri-block copolymer self-assembly. This would allow fewer expensive chemical steps and also let the nanocomposite material be efficiently re-moulded into shapes appropriate to individual tasks. This line of research has a direct bearing on the rest of this project, but also is of fundamental importance to the broad scientific and engineering community involved in various aspects of carbon nanocomposites where there is a great need for the advanced and quantitative dispersion and characterization methods, and specially designed surfactants.
CNT alignment and composite characterization:
To achieve high CNT alignment the melt shear-spreading method will be applied to the polymer-CNT mixture. The degree of alignment will strongly depend on geometric parameters of shear-spreading, the polymer viscosity and the tube parameters (length / persistence length ratio). After shearing, the polymer films will be cross-linked. X-ray analysis will be carried out to confirm the CNT orientational order. Mechanical testing on cross-linked nanocomposite elastomers with controlled CNT alignment will follow the protocols established at UCAM and a stress relaxation study will be carried out to establish the accessible range of times / frequencies for the future actuators.
The photo-actuation in aligned nanocomposites will be characterized. First, by constraining a taut strip of the sample in a load-cell rig and illuminating it with a light source, we shall measure the actuating force and use this experiment to optimize the light wavelength / intensity, as well as the sample composition (CNT concentration, length and alignment) Secondly, we shall test the actuation stroke (strain induced by illumination) and response time. Quantities of optimized CNT composites will be provided to WP3 for final blister fabrication.
All of the results of this part of the project are of fundamental importance to a broad field, and will result in a number of high-profile publications.
Liquid crystalline elastomer actuators:
This will be a parallel line of research, aiming to explore an alternative actuating mechanism and also provide a backup line of materials for the NOMS design. Nematic elastomers (NE) doped with azobenzene-based chromophore molecules undergoing trans-cis photo-isomerisation on photon absorption has been shown (by UCAM, and several other research groups worldwide) to be a promising photo-actuating material. In many cases, adding a small concentration of CNTs, aligned by the orientational order of a liquid crystal, is known to enhance actuation properties of NE. These will be investigated.
WP3 NOMS Blister and tablet design
The aim of WP3 is the design and optimization of the blister and the design of the overall NOMS tablet which will be integrated with the other components in WP5. In order to achieve the general objective, the following tasks and objectives have been implemented.
Task 3.1: Prototype blister concept and design:
The objective of this task is to define the concept and design a prototype for the blister according to the characteristics of the photo-actuating materials, such as magnitude of stroke, Young??s modulus, etc. This blister must fit the requirement in terms of force and size defined in the specifications. Two alternative geometries will be investigated, one where the walls contract and the blister flattens and one where the blister is effectively inverted and a centre pin protrudes on exposure to light.
Task 3.2: Microtechnology for blister fabrication:
The objective of this task is to develop the required technology to obtain the correctly sized and shaped blister from a basic shear-aligned film of photo-actuating polymer material. First approach will be based in developing a master stamp, but different microtechnologies will be considered.
Task 3.3: Simulation, characterization and optimization of blister:
FEM analysis will help optimize the geometry. Once technical specifications are provided, several prototypes will be fabricated for integration and evaluation. Parameters to be evaluated and optimized include response time, lateral and vertical resolution
Task 3.4: Overall tablet design and assembly process:
The initial NOMS tablet will be formed by a light emitting substrate, followed by microlenses to focus the emitted light, a photo-actuating layer with assembled blisters, and finally a flexible polymeric layer for protection. The initial tactile display will be a 10x10 tactel array of blisters which will be compatible with the display and tactile representation of Braille text. A second, more challenging design will aim to produce an 80x80 tactel array with improved resolution.
Task 3.5: Protection layer for tactile display surface:
The NOMS tablet will require a protective polymer layer on top of the photoactive layer. It should be capable of withstanding any temperature gradients generated by the light emitting layer and sufficiently robust to withstand direct contact by the user.
WP4 Communication and control
WP4 has developed the transformation software and the control electronics needed to transform images from a computer into tactile information on the tablet. The three main objectives were to develop computer software that can transform any given image into binary information that can be sent to the device, to develop the control electronics needed to transform this information into tactile information and to develop the communication protocol to send information from the computer to the device.
The control electronics developed by WP4 is connected to an LED board developed by Philips that transforms the electric signals into light, to actuate the material and thus move a tactile element to be felt by the user.
WP5 Optics and microsystem integration
An important milestone will be a demonstrator of a polymer surface with regular array of vertical columns contracting and expanding on illumination from below to enable proper 3D representation. Actuating light sources will be selected in terms of power, intensity, integration, resilience, ability to be focused etc.
WP5 will take an incremental approach, initially developing a single blister system before fabrication of an array for the NOMS tablet. To form the single blister with integrated light source, a suitable surface-mount, high intensity LED will be integrated with an appropriately scaled single microlens to form a small, mounted single element light source. The illumination intensities and light directionality will be measured and subsequently the light emitter will be modified to ensure that these meet the :blister: requirements. The resulting unit will be integrated with the blister layer from WP3 to complete a single blister prototype. The prototype will be tested for deformation, including magnitude and temporal resolution, temperature, and finally tactile quality.
The technology developed for the single blister design will be expanded to produce an initial 10x10 tactel array. Two parallel activities are required, the first is the design and fabrication of a light emitting layer and the second is the fabrication of microlenses to focus the light on to the photo-active material. The light emitting layer will be based on experience gained in fabricating a single element. This will either use an arraying out of the single element (e.g. an array of 10x10 surface-mount LEDs with suitable row and column controller) or by using the alternative display approaches : rear projection using a digital light mirror device or a high intensity backlight and monochromatic LCD. These latter approaches will likely be adopted for the 80x80 array if analysis and tests indicate that they can achieve the necessary intensities at target wavelengths. Issues of thermal load and dissipation, suitable use of optical films to increase backlight intensity etc. will be included in the design, testing and fabrication.
The polymer microlens array will be based upon the tablet design (interpixel distances, focal lengths, wavelength of light) and optical design of the blisters. The goal is a thin layer film with an array of microstructure elements implementing the lenses. Depending on the lens depth required, this may be achieved using actual microlenses cast in a UV curable polymer or the lens may be restructured as a Fresnel with multiple prismatic zones in order to minimise the depth whilst achieving a large equivalent curvature. The resulting designs will be first constructed as master moulds, using either laser ablation or diamond fly cutting to create the microstructures, followed by replication onto :tiles:, tiling of these replicates on a drum shim and manufacture of optical film using a UV casting process. Choice of base film (the clear plastic film on which the microstructured optics are cast) and the polymer for structure casting will be matched to the required illumination wavelength and the thermal environment anticipated. The resulting optical film will be tested and characterised for physical and optical properties.
The light-emitting layer (LED arrays for the low resolution displays, and either LED arrays or some other high intensity display type for the high resolution display) and polymer microlenses will be brought together with the blister technology from WP3 and the communication and control hardware & software from WP4 in an integrating activity to produce the final 10x10 and 80x80 NOMS tablets. Careful control of alignment between the light emitting display, microlens layer and blister layer will be required. The resulting tactile displays will be tested for thermal and mechanical issues, including mechanical deformation magnitude, temporal response and tactile properties. The results of testing the 10x10 display in WP6 and WP7 will be used to inform the development of the 80x80 display.
Subsequent to the prototype tablets being produced, a design for manufacture task will be undertaken which will examine methods to scale up the production of the tactile displays components capable of achieving the required quality, along with techniques which are capable of integrating the elements to required tolerances at suitable cost.
WP6 End-user evaluation and neuro-cognitive study
End-users will conduct preliminary testing, providing feedback on material performance and blister design. These first-degree tactile cues will be addressed early-on during the project, providing prompt feedback on the material and tablet design. Some of this work can be done using traditional methods (e.g. raised dots on paper) and dummy blisters on PDMS, but will transfer to using the NOMS tablet once available. This will include tests with both healthy and, if appropriate, visually impaired adults. Prior to such tests with the NOMS tablet, the necessary ethical approvals will be obtained. Psychophysical studies (including absolute thresholds, two-point discrimination, roughness discrimination) and studies testing the use of NOMS for complex spatial tests will be conducted by neuropsychologists. Psychophysical experiments provide information on whether the tactile stimulation provided by NOMS fits with the capacities of the tactile system of humans. Thus, in the initial phase (with a first prototype) the tactile properties of NOMS are iteratively adapted to the need of humans. Subsequently, the capabilities of the NOMS tablet as a research tool in combination with the use of event-related brain potentials will allow studying the neural correlates of language processing in a tactile medium. Finally, it will be tested whether the NOMS tablet can be used to administer psychological tests for verbal and spatial abilities. Simple administration of such cognitive tests would be of great help in vocational guidance and education consulting.
WP7 Dissemination, exploitation and management of intellectual property
Our findings will be disseminated in international conferences and area-specific workshops as well as through a dedicated project website. NOMS will adopt The Web Content Accessibility Guidelines (WCAG)?©. WCAG offers documentation and guidelines to make web content accessible to people with disabilities. Additionally, the consortium will engage in outreach activities, both in primary and high schools. As for academic advancement, research assistants will complete their Ph.D. thesis, thoroughly developing novel research topics lying amidst the breath of the project. Professional development of senior researchers will also be encouraged by project management and executive training programs. The NOMS project will produce a substantial number of publications early-on, given the incipient interest for photo-actuating systems.
The inclusion of industrial partners (Microsharp, Philips and iXscient) in the consortium will form the basis of the exploitation and technology-transfer. Thorough market studies for NOMS tablets or other applications revolving around optical-actuation will be conducted, as well as a cost-benefit analysis.
Project Results:
WP1 System specification and risk assessment
An initial risk assessment was performed early in the project, following the guidelines set out in deliverable D8.2 (Risk assessment procedure), and deliverable D1.1 (Initial risk assessment and contingency plans) was written. Risks have been continuously updated and monitored through internal quarterly reports, leading to deliverables D1.3 & D1.4.
The early provision of the draft specifications document at the beginning of the project allowed the technical work to progress as planned. An update of the system specifications was completed allowing the final components for the 10x10 tablet (e.g. light source, optical design etc.) to be defined and D1.2 was completed in January 2010.
WP2 Characterization and optimization of photoactive nanomaterials
Task 2.1: Fundamental studies of CNT photo-actuation (CNM, UCAM):
The performance of the actuating material is of crucial importance to NOMS. The actuating properties of CNT-nematic elastomers (often referred to as LCE-CNT) have consistently been superior to those of other polymer-CNT systems. For that reason, to provide the WP3 (blister design) with interim working systems, UCAM have continued to study and improve the LCE CNT material. In particular, the main earlier difficulty with photo-actuation of LCE (which required UV light) has been removed by sensitizing the system to visible/IR light by adding a small amount of CNTs or by doping the material with special dyes. Independently, the effort to develop the CNT elastomeric composites using commodity materials have continued at PISAS and UCAM. The main impetus here is to work with thermoplastic (self-assembling) elastomers such that CNT segments would be aligned under shear flow at a high temperature and then the system frozen into an elastic rubber composite on rapid cooling. This a much simpler route for industrial use.
UCAM and CNM studied how the actuating behaviour changes with the composition of the LCE-CNT system. Attempts to quantify the degree of alignment present in our samples were made using X-ray analysis. The results indicate that we managed to achieve a fairly uniform degree of alignment over several samples, thus allowing a quantitative study of how the polymer composition changes the actuation properties.
Studies were carried out by SEM and TEM imaging and X-ray analysis, on both LCE-CNT and EVA-CNT samples. One of the main objectives of this analysis is to study the CNT distribution in the polymeric matrix and the observation of the material contraction phenomenon in situ, at micro- and nanoscale.
Firstly, we started with the examination of LCE-CNT composites by Scanning Electron Microscopy (SEM). We examined the surface of the samples searching for CNT presence, but even after surface carbon coating, it was very difficult to localize the nanotubes.
After that, we used Focused Ion Beam (FIB) milling to micromachine a lamella for TEM examination. The thinned lamella shows heavy irradiation damage, suggesting excessive irradiation. The complex contrast observed in TEM imaging leaves the CNTs still unresolved.
An alternative method of cryo-microtomy for sample preparation was more successful. Figure 1 shows a few TEM images obtained from one mono-domain side-chain LCE sample with 0.1% wt. of CNT content, where the CNTs can be easily observed. This gave us confidence that the protocols for TEM sample preparation were suitable.
A TEM holder was retrofitted with an optical fibre and sealed with a vacuum-friendly epoxy in order to allow in situ actuation and analysis. Initial tests with the JEOL-2010 housed at the Penn Regional Nanotechnology Facility confirmed the vacuum held upon holder insertion. In addition, the optical conductivity of the fibre has been tested and laser light injection on the tip end results in light emission at the other end of the fibre. These results suggest that the fibre has endured the axial compression imposed during vacuum-sealing. These experiments are being further pursued at the moment.
Task 2.2: Functionalisation of CNTs, dispersion and cross-linking and Task 2.3: CNT alignment and composite characterization (UCAM, PISAS):
For the polymeric matrix, PISAS initially used two types of elastomers, namely Styrene-Isoprene-Styrene block copolymer (SIS, Kraton) and ethylene-vinylacetate copolymer (EVA).
To improve compatibility of carbon nanotubes (MWCNT) with Kraton, the MWCNT were modified either with cholesteryl chloroformate (MWCNT-chol), expecting that the aliphatic structure can improve interactions with polyisoprene phase, or with short PS chains (MWCNT-PS), for improving compatibility with the Kraton polystyrene phase. Studies showed that the presence of covalently bonded polystyrene chains in MWCNT-PS significantly improved the filler/matrix interaction, leading to well dispersed CNTs.
Strips of this composite material were elongated at temperatures of 70 ?°C and 80 ?°C. Subsequently, the photo-actuation response was monitored by Dynamic Mechanical Analyser (DMA) in iso-strain mode. The clamped strip was illuminated by red LED at 300 mA for 10 or 30 seconds and the changes in stress were recorded. A negative stress was recorded, pointing to an elongation of the strip in response to light. However, the response was not sufficiently large for the requirements of the project. Therefore, nanocomposites based on Styrene-Isoprene-Styrene block copolymer and containing carbon nanotubes were not considered suitable for preparation of the prototype tablet or for further investigation.
Two kinds of ethylene-vinylacetate copolymers (EVA) were used for nanocomposite preparation, Levapren 500 containing 50% wt. of vinylacetate and EVATANE 28-25, with 28% wt. of vinylacetate.
EVA/MWCNT nanocomposites based on Levapren were prepared by casting from solution using liquid crystalline polymers called :pyrene main chainß? (PyMC) with a PyMC:MWCNT ratio of 10:1, 5:1 or 1:1.
Analysis by dynamometry of stretched samples indicated that the materials thus fabricated were responsive to an applied light source, and that the surfactant:CNT ratio affects the magnitude of this response. It is clear that the sample prepared at a surfactant:CNT ratio of 5:1 shows a significant stress response to an applied light, whereas samples prepared at ratios of 10:1 and 1:1 show no response. It is possible that a ratio of 1:1 may not be sufficient to disperse CNTs sufficiently well in the polymer, while a ratio of 10:1 may lead to crystallization of the phase-separating excess surfactant.
It is also clear that the concentration of CNTs in the polymer can affect the stress response to an applied light. The data indicate that there is a :threshold: concentration of MWCNTs, below which the material is not actuating.
However, when the measurements were repeated two weeks later, there was very little photo-actuation response in the 5:1 sample. This could be due to a loss of orientation of carbon nanotubes on each heating cycle during measurement.
Next, the EVA copolymer containing 28% wt. vinyl-acetate, (EVATANE 28-25) was selected for further investigation. The nanocomposite was prepared by casting from solution using cholesteryl 1-pyrencaroxylate (PyChol) compatibilizer. The ratio of MWCNT/PyChol was 1:5.
The extent of MWCNT dispersion within the polymeric matrix was characterized using TEM. The strip samples in the original unstretched and stretched form with 50% deformation were studied. The main results from the TEM micrographs obtained for EVA composites containing 0.3 and 3% wt. MWCNT-PyChol are the following:
- a good dispersion was obtained due to the use of a cholesteryl 1-pyrenecarboxylate compatibilizer (ratio CNT/compatibilizer was 1:5)
- single carbon nanotubes and a minimal amount of their agglomerates were obtained but this is usual for composite materials
- a good indication of CNT orientation was observed in the stretched samples
A detailed study of these nanocomposite materials was done using DMA analysis. The photo-actuation response was monitored by DMA equipped with red LED (?» = 627 nm) by iso-strain mode. The strip was placed between two clamps and illuminated by the LED at 300 mA (power 58 mW) for 10 or 30 seconds and the changes in stress over time were recorded simultaneously.
Experiments were performed with a stretched strip based on an EVA composite containing 0.3% wt. MWCNT-PyChol with 10%, 20%, 30% and 50% deformation. For the 10% deformation samples, an expansion (negative stress) was obtained upon illumination. For all other stretched samples, a contraction (positive stress) was obtained. In addition, measurements were done for 50% deformation EVA samples containing various other amounts of MWCNTs or SWCNTs. The measured stress changes are summarised in Table 1.
According to the previous results performed at PISAS, a sample of the EVA/0.3% MWCNT composite was sent to CNM for further testing. The materials were characterized by force measurements using a white Luxeon LED (the same as used in the 10x10 NOMS tablet) as the actuating light source. The maximum force obtained was 17 mN. It is worth noting, however, that in contrast to LCE samples, the actuation time increases from the order of a few seconds to minutes.
To prove the repeatability of the actuation of those samples (no sample damage), the same measurements were repeated few days later. In contrast to previous samples measured, no force variations were observed. In fact, the results obtained with the EVA/MWCNT samples were comparable to the ones obtained using LCE dye-doped ones (see task 2.4) making this material a candidate for the blister configuration.
Task 2.4: Liquid crystalline elastomer (LCE) actuators (UCAM, PISAS):
Initially, composites of CNTs with commercial polymers were favoured because of the expected decreased manufacturing costs. However, actuation parameters (magnitude of the stroke, needed exerted force in tactile applications, etc.) in CNT-nematic elastomer (LCE-CNT) composites have outperformed those of polymer-CNT systems. As a result of this, a decision was made to pursue the LCE-CNT actuator material for the initial 10x10 tablet (milestone M2.2).
The incorporation of MWCNTs into side-chain LCEs in order to sensitise these materials to IR or visible light was demonstrated. However, the disadvantages of this method for haptic technologies include mechanical weakness of the material as well as a complex synthesis procedure.
In order to improve the actuation response and the mechanical strength of the polymer, the incorporation of main-chain LCE materials (in which LC units are incorporated into the polymer backbone, rather than into side-chains) was investigated. Hydrosilylation reactions between the divinyl mesogens and siloxane polymers containing Si-H end-groups can lead to polymerization reactions in which the LC units are incorporated into the polymer backbone. Unfortunately, while these have been shown to improve the mechanical strength of the material, they also proved to inhibit the photo-response, perhaps partly due to imperfect coupling to the siloxane polymer matrix. This remains a definite area of future improvement, but given the severe time constraints and pressures in the project, we decided to abandon this route.
The next step, therefore, was to attempt the fabrication of new LCEs made purely from main-chain LCs (thus removing the possibility that the main-chain mesogens are not properly coupled to the polymer). In addition, using siloxane spacers between rod-like mesogenic units would reduce the glass transition and make the elastomer more deformable. The properties of the materials created so far are an issue, however; these polymers do not show the elasticity required for good actuation (being either hard and inextensible, or too soft). We believe that optimisation of the cross-linking ratio may improve this. However, due to the time pressures in the project and the need to deliver the actuating materials immediately, this line of polymers was also abandoned.
The main problem with composites of side-chain LCE with MWCNTs, which were originally used at CNM for prototypes, is that their consistent fabrication in large quantities is extremely difficult (a large amount of attempted reactions do not proceed as required, almost in random fashion -- and from those that do proceed, most actuating strips break during fabrication: at most 1 in 10 strips survives the fabrication process). This was the cause of restricted availability of these samples: sometimes there was no suitable material generated from a full month of synthetic work.
As we started to believe the unreliable chemistry and easy to fracture elastomers are due to the CNTs disrupting the polymer matrix, another method under investigation for simplifying this procedure was now followed. We used a special light-absorbing dye rather than MWCNTs for the photo-sensitising component. Our previous results indicate that MWCNTs are causing photo-sensitisation by conversion of light into heat and so we selected the so-called laser-welding dyes that are very efficient in converting absorbed infrared light into local heat. Selection of such dyes is not a trivial process, since many dyes will not produce enough heat to raise the local temperature of the material by tens of degrees; however, some industrial applications (e.g. laser plastic welding, laser ablation of printing plates) rely on highly absorbing dyes that can heat a surrounding matrix. The dye must be compatible with our polymer (ideally, the dye must be soluble in similar organic solvents) in order to be incorporated into the LCE.
IR1310 dye was obtained. The solubility of the dye in toluene was found to be sufficiently high to achieve high levels of doping in our LCE. We fabricated strips containing dye concentrations (by mass) of 0.1% and 1%. The UV-vis absorption spectra of these samples were measured.
As was to be expected, under a white light source the actuation stress generated by these samples was lower than reported previously for LCE-CNT materials (because the material is largely transparent to light between 500-650 nm). CNM also carried out the measurement using a light source at 980 nm and found that the actuation was comparable to that reported for LCE-CNT samples.
Based on our expectations for the MWCNT materials, the NOMS tablet was designed to work with a white-light LED source. Therefore, in order to avoid changing the tablet design, we also created materials containing a commercial dye, Disperse Red 1 (DR1), which absorbs in the visible region. See Figure 2 for a comparison between the absorption spectra of these and of the IR dye samples.
The heat-activated actuation of these materials was good, with the samples exhibiting similar temperature response to :un-doped: LCE samples.
This material was used to fabricate the blisters in the 2x2 prototype tablet. CNM measured the force produced by the pins in the tablet and found that the material performed better than the original LCE-CNT material with a force up to 35mN (see WP3 for more details).
In general, the results obtained seem relevant and encouraging to continue exploring the dye-doped material as an alternative to LCE-CNTs. In order to create a larger quantity of dye-doped LCE material, experiments were attempted by UCAM using a larger centrifuge reactor in order to create wider (and longer) films. This was achieved, but these films showed a lesser photo-response than previous ones.
WP3 NOMS Blister and tablet design
Task 3.1: Prototype blister concept and design:
According to the prerequisites established for the NOMS device, a new concept of actuator should be designed to accomplish not only Braille and graphical representation restrictions (reduced dimensions with a pitch of 2.5 mm between consecutive actuators for appropriate reading) but also tactile acuity specifications (large vertical displacements (at least 0.3 mm) as well as high forces (minimum values around 15 mN) for correct tactile perception.
Following the above mentioned requirements as well as the material characteristics (see WP2), a :U-shape: configuration was proposed for the fabrication of the actuators to make the most of stress generated during the material actuation. The working principle of the device is based on the light-induced mechanical actuation of a nematic LCE. More concretely, the stress gradient generated into an LCE film which contracts under illumination is used to generate a vertical displacement of a pin. Figure 3 shows an illustration of this concept.
Task 3.2: Microtechnology for blister fabrication:
Two different technological approaches based on moulding alignment of LCE materials were investigated with the main objective to obtain a high density of actuators, thinking of the fabrication of the 10x10 tablet. These two technologies allow local actuation of the material, due to the localized stress gradient generated during the alignment. The first approach proposed is a pure mechanical moulding system (see Figure 4a); the second one is a gas pressure system (Figure 4b). Both are new approaches, with great potential for the fabrication of small actuators as microfluidic valves and pumps. However, due to the restrictive blister requirements (large forces and displacements in a reduced area, with small pitch between actuators), it has not so far been possible to adapt this fabrication technology for the NOMS actuators. Thus, U shaped blisters were the most suitable approach for NOMS tablet design.
To extend the NOMS tablet from 10x10 to 80x80, the proposed approach of optically actuated U-shaped blisters proved not to be possible, due to the high power consumption of the light source necessary to actuate the material. To try to solve this problem, a third approach was proposed changing from optical to thermal actuation of the material through an array of heaters. Figure 4c shows the first array of heaters fabricated with its corresponding circuitry.
Task 3.3: Simulation, characterization and optimization of blister:
Almost all the effort around this task was focused on the characterization and optimization of the designed blisters. Through different setup configurations with one or various assembled actuators (see Figure 5 a-c for an example), different parameters related to blister behaviour were studied: the stress produced when material contracts (blister illuminated), the actuation time, the displacement of the pin, etc., all of them as a function of the power supplied for the light source. By using an auto-translational stage, an accurate dynamometer and a specific programming code, actuation curves of each blister were recorded for subsequent analysis. Comparison of blisters assembled with different material samples (LCE-CNT, and LCE with various dyes types and EVA) was done. Force was successfully measured, reaching higher values than required for the application in most cases. Figure 5d shows an example of the force measurements of three different blisters assembled by using different material samples, for the same power applied. To check the viability of the actuator and the whole system, some cyclic actuation tests were performed, adjusting the operation parameters as the intensity of current and time.
ANSYS?® Finite element analysis software was used for the simulations, which were basically focused on the study of basic characteristics of LCE (temperature distribution and contraction under excitation); tasks related to WP2. Within WP3, some studies were carried out to determine the force and stress distributions inside the elastomer during the actuation of blisters.
Task 3.4: Overall tablet design and assembly process:
Two different 10x10 prototype tablets were assembled and tested, one using LCE material with embedded CNTs and the other one using LCE doped with dye, to increase the absorption of the light (see WP2). Each prototype has a fixed part that contains the LED array with the corresponding circuitry together with the hardware electronics, and a mobile part with the assembled blisters (the actuators), that can be swapped. Blisters of both materials were measured over different periods of time to verify their stability. No actuation differences or material damage was observed, maintaining the good force values measured over several months.
According to UHamburg requirements, a specific prototype version with only four blisters was assembled for perception and acuity testing (Figure 6). The blisters were assembled accurately to guarantee the correct position of the pin (they should not protrude from the top surface of the NOMS tablet at rest to do not hinder the correct finger reading). For the fabrication of these blisters, LCE dye-doped material from UCAM was used. The maximum force reached for each blister was measured individually, as a function of the intensity applied. The graph of Figure 7 shows the force measurements obtained for each blister, with the deviations. In spite of existing significant differences between the force values reached by each single blister, all of them can be perceived by the finger and distinguished from the other ones. However, pin number 3 presented some difficulties.
Figure 8 shows the assembly process for the NOMS tablet and photos of the completed device.
Due to high power requirements of the photo-actuating materials, only 10x10 prototypes were assembled. The 80x80 tablet requires improvements in the photo-actuating materials and a new technological approach for the blisters and light platform. This was not possible to be optimized within the project framework. However, a feasible technology based on thermal actuation of LCE material was presented and explored.
Task 3.5: Protection layer for tactile display surface:
Tests of flexible materials for the protection layer were completed. Several thin silicone materials appear suitable. There is clearly a trade-off between robustness of any cover layer and unimpeded functioning of the tactile device. A small number of very thin materials were selected. Following further testing of thin silicone samples which provide a suitable compromise between robustness and functionality in a tactile device, the best candidate material for incorporation into the demonstrator was Silex (basic silicone) with a thickness of 25 - 50 ?µm.
WP4 Communication and control
WP4 has developed three main results: the transformation software, the control electronics and communications.
Transformation software:
Special software has been developed to transform the visual images into tactile information, and to send this information to the tablet. Two versions of the software have been developed. The first one (Figure 9) is designed to be used with a tablet of 80x80 high resolution tactile elements (tactels).
A second version of the software (Figure 10) is designed to operate with the 10x10 version of the tablet (with Braille standard resolution), as this is the first developed prototype of the NOMS tablet.
The 80x80 version of the software has the following features:
- It can transform any given image into pixelated information, adjusting the threshold of this transformation.
- It allows the combination of several tablets, forming a matrix of up to 6 tablets (two columns and three rows). This combination can then be automatically sent to the corresponding tablets, adjusting the position of each screen to the corresponding device automatically.
The 10x10 version of the software has the following features:
- This version can operate with up to 10 images at the same time, and can send them to the tablet one by one or in a batch.
- It can perform power ramps, adjusting the power of the LEDs in a very precise way over a given period of time. Also, the general power of the LEDs can be programmed to different values.
- The software can also update the device:s firmware when needed, using a simple User Interface. With this, the device can be updated by the user, without having to send it to the developer.
Both versions of the software have the following common features:
- They allow the user to draw simple shapes with the mouse, and can also transform written text into Braille characters.
- The transformed images and/or Braille text can be stored and retrieved from the user:s computer.
- They can send the tablet information to the device using either Bluetooth or USB ports, depending on the convenience of the user.
- They have been developed under the Microsoft Windows platform to be used by the majority of computers.
- They can be updated automatically via Internet, whenever new versions are published. This allows the user to have the latest version with new features and/or bug fixes without having to search and install a new version.
Control electronics and communications:
A device has been developed to control the 10x10 LED board developed by Philips, and to receive information from the computer. This device communicates with the computer either by Bluetooth or USB protocols.
A special protocol has been developed to send the transformed image information to the device. This protocol can operate through Bluetooth or USB communication architectures. It is also capable of sending information related to power ramps and also to update the device firmware.
Four different prototypes have been developed, tailored for the use of each partner (CNM, UHamburg, Microsharp and Philips). CNM needed to perform power ramps in order to test different materials and/or stress the material in different ways, so the ability to perform such operations has been added to their prototype. UHamburg needed to control the timing when showing information on the tablet in a very precise way, so a parallel port has been added to control the triggering of the screens from a central computer. This allows them to perform neuro-psychology experiments with different users. The prototypes for Microsharp and Philips are the same as the prototype for CNM, with only minor changes.
These prototypes can be updated from the Transformation Software, using a special loader stored in the microcontroller memory. This allows the device to be updated in an easy way by the final user.
The design of these prototypes has been made to be easily scalable to the 80x80 version of the NOMS tablet, in case it is developed in the future.
WP5 Optics and microsystem integration
Task 5.1: Design and fabrication of light emitting layer (Philips, Microsharp):
Philips performed an initial scan of the available LED light sources on the market today, for both single blister and 10x10 arrays. Several SMD-LED package sizes were identified that could realistically be used for the illumination array. Smaller form factors are available, down to 1x1 mm, and in the (near) future probably even less. However optical output typically drops one or two decades when scaling down. Since it was unclear in the early stages of the project what would be the illumination requirements for the actuating materials, a decision was made to select the Luxeon C white-light LED for the 10x10 tablet. This would allow for maximum light output over a broad spectrum while still being capable of arraying the devices on a 2.5 mm pitch in order to meet the Braille standard.
For the initial 10x10 tactile display, Philips then analysed the requirements, as relevant for the electronic control (driving) of the LED array that will provide the light. Based on these requirements, Philips has proposed a suitable LED driving electronics architecture, based on previous experience with LED television backlights.
Philips made the complete light emitting layer based on an array of 10x10 LEDs. This layer was successfully implemented in a tactile demonstrator during the first Technical Review meeting in Bratislava. Subsequently, four additional copies of the light emitting layers were built for further testing and incorporation into prototype 10x10 tablets.
The photo on the left in Figure 11 shows the 10x10 light source array in operating mode. Even though no valid data has been given to the system, LEDs illuminate (at low power) corresponding to the random data present at start up. Via the two fans at the bottom side, forced air cooling is provided. A view of the inside of the mounted boards is provided by the photo on the right. As can be seen, the forced air is not obstructed and will cool the heatsink as well as the driver ICs on the driver board.
An internal workshop was held to discuss ideas for the 80x80 light emitting layer. It is not possible to simply scale up the 10x10 LED array to 80x80 (even at the same pitch) because of the very large power requirements, and moving to a higher resolution (1.25 mm pitch) would require much smaller, and hence lower power, LEDs.
With respect to resolution, power and size, pico- or so called micro-projectors were considered worthwhile to investigate as a potential light source for the 80x80 tablet. Only one powerful light source is needed while a dedicated spatial light modulator takes care of redirecting the light to the individual pixels with the desired amount of light intensity. This was investigated by Philips. In addition, Microsharp investigated a transmissive LCD coupled with a pixelated backlight.
A DLP projector was modified with a custom side-entrance to be able to use with any white light source. A laser was used to measure the efficiency of the projector: the laser power was measured at the entrance and exit of the projector. The modified DLP projector was founde to have an efficiency of 46%. Based on the photo-actuating material requirements, around 14000 lumen would be needed as a light source. This is very high. Therefore, it was concluded that a light source based on DLP for the 80x80 tablet is not practical.
In parallel, an investigation was initiated at Microsharp into the use of a monochrome LCD display to modulate a high brightness LED array in order to achieve the resolution and small pixel size required for the 80x80 array. In effect a similar arrangement to the 10x10 LED array is produced. Each of the LEDs is then subdivided by blocks of the monochrome LCD pixels which are switched on or off in order to :shape: the LED to produce smaller, :virtualß? pixels.
When a virtual pixel needs to be illuminated, that virtual pixel (or its centre) are set to be transmissive - that is in the on state and all other pixels are set to be in the off state. At the same time the LED associated with that virtual pixel is set to be on. This is an appropriate illumination system if the following are the case:
1. The intensity of the LED illumination through the on state of the transmissive display is sufficiently bright to actuate the associated blister
2. The transmissive display has sufficient contrast to ensure that the illuminated LED with a virtual pixel in the off state does NOT actuate the associated blister
3. The tactile display is built in a manner which sufficiently limits cross-talk between adjacent :tactels: (associated with virtual pixels) - that is an adjacent on state virtual pixel does not actuate a blister (even partially) which should be in the off state.
This initial experimental work evaluated the feasibility of the 80x80 illumination solution being based on a low resolution modulated LED array gated by high resolution transmissive LCD display. The LED array from the 10x10 demonstrator was used as a backlight. A monochrome STN 320 x 240 resolution transmissive LCD with pixel size of 300 x 300 ?µm was chosen as the evaluation display.
The conclusion of this experimental work was that the intensity of the LED illumination through the on state of the transmissive LCD display is sufficiently bright to actuate the associated blister in the current 10x10 tablet. The transmissive display has sufficient contrast to ensure that the illuminated LED with a virtual pixel in the off state does NOT actuate the associated blister. The tactile display is also built in a manner which sufficiently limits cross-talk between adjacent :tactels: (associated with virtual pixels) : that is an on state virtual pixel does not actuate an adjacent blister (even partially) which should be in the off state.
Therefore the overall conclusion is that, providing the smaller blisters associated with the 80x80 array can be actuated by 40% of the light within the blister area from an LED, then the display should in principle work. However, this is not possible for the current photo-actuating materials, and significant improvements would need to be made to these materials.
Task 5.2: Design and fabrication of microlens array (Microsharp):
Based on the configuration of the U-shape blister and LED source, a full optical design of a single blister element was performed. This will eventually consist of the lens arrangement shown in Figure 12. Two lenses are designed, since redirecting the light in a single stage exceeds the manufacturing limits on facet angles. In addition, by having two stages in the light redirection, the top lens can either be radial (converging to a circularly symmetrical structure) or linear (converging to a linearly symmetrical structure).
The collimating lens has been designed with a novel arrangement. The central part of the lens is a collimating refractive lens and the outer part a collimating TIR lens (with symmetrical prismatic feature). This results in correct collimation (assuming a point source) and all prismatic features have peak angles of less than 48 degrees and thus are manufacturable.
Task 5.3: Integrate light source with single blister (Microsharp, Philips):
The Luxeon C LED has been integrated with the single blister. This has been shown to achieve good actuation even without microlenses between the light source and blister.
CNM used a conventional photodetector to check the efficiency of the microlenses when coupled with the Luxeon C LED. The light intensity was measured at a distance 5 mm from the LED without lenses, with collimating lens only and with the both lenses. Figure 13 shows the relative improvement of the light intensity for the different lens configurations over the case with no lens.
The results indicate that the lenses are functioning as required, though care needs to be taken to accurately position the lenses with regard to each other, the LED and the actuator ribbon position.
Task 5.4: Overall integration of NOMS tablet (Philips):
The overall 10x10 tablet design consists of a number of different layers as described below, from the bottom layer upwards:
- Light emitting layer including LED array, drive electronics and cooling fans (see task 5.1)
- Lens foil, 250 ?µm thick, incorporating both microlens array mounted on either side of a carrier film
- Lower PMMA block with square holes on top (to accommodate the photo-actuating material) leading to reflective chambers on the bottom
- Strips of photo-actuating material formed into rows of blisters
- Thin aluminium plate for heat dissipation (100?µ thick)
- Upper PMMA block, 3mm thick, which includes holes for the pins which are raised by the blisters
- Flexible protective layer, 25 - 50 ?µm thick
The protective layer was omitted on the first prototypes. The individual components were sent to CNM along with two small :tools: to assist in the shaping of the actuator ribbon, one an individual blister :shaper: and one a shaper for a row of 10 blisters.
Subsequently, CNM assembled the first prototype. However, the amount of LCE-CNT photo-actuating material available was only sufficient to partially complete the tablet; 60 of 100 blisters were mounted (Figure 14). Nevertheless, this was suitable for initial testing. Further prototypes incorporating the dye-doped LCE material were subsequently also assembled.
Philips have tested the 10x10 light source and certified it as safe for the user tests at UHamburg. These are the final tests and paperwork necessary before UHamburg can obtain ethical approval for testing the tablet with visually impaired users.
An additional tablet with a 2x2 array of working blisters was subsequently assembled and fully characterized. This tablet has been delivered to UHamburg in June 2012 for them to start testing.
Extending the 10x10 prototype into an 80x80 prototype is not as straightforward as it may seem. There are two main problems: the first is being able to pack the high power LEDs into a small area and be able to manage the heat generated by them in the situation when they are all on at the same time; and the second problem is being able to source an appropriate monochrome LCD with the right pixel size and pitch. While compromises may have been struck to source a display, the fact that a suitable backlight was unavailable caused the partners to decide that the 80x80 display is not feasible with the currently available photo- actuated materials. This was always expected to be a major challenge in the high-risk NOMS project.
WP6 End-user evaluation and neuro-cognitive study
Development of a new tactile acuity measure using dummy blisters showed suitability of polymer material for tactile perception:
Psychophysical testing was performed to assess tactile spatial acuity with polymer material using samples of PDMS pieces with dummy blisters (Braille-like dot patterns and Landolt ring patterns with varying spacing). Character amplitude (0.2 mm) remained constant, but the spacing of the characters decreased by 0.1 log units (~26%) per line, following the common design principles for modern logMAR visual acuity charts. Tactile acuity measured with the PDMS material was compared to acuity measured with traditional thermo-sensitive paper in a sample of young sighted adults. In line with previous studies, perceptual thresholds were lower for the ring patterns. Importantly, however, performance did not differ between the two materials, verifying that polymer material had no adverse effect on tactile perception. Additional studies have been conducted that aim at comparing the active acuity measures obtained with the acuity charts with more traditional passive measures such as two-point discrimination and grating orientation resolution. The results show that tactile acuity measured with the polymer dot charts has high test-retest reliability and agrees well with passive tactile acuity measured as grating orientation resolution. Apart from the direct implications for the design and material properties of the NOMS tablet, these findings have important theoretical value for the advancement of tactile acuity charts as a measure of tactile spatial resolution in applied and research settings.
Suitability of NOMS control board and computer interface for neuroscience studies has been achieved
Upon receipt of a first 2x2 NOMS prototype at month 34, an experiment was prepared to test tactile discrimination performance of simple patterns (akin to the dot patterns used with dummy blisters). This experiment had to be postponed due to the on-going ethical approval procedure (as a consequence of the delay in the delivery of the first prototype). However, note that the major limitation of the 2x2 prototype appears to be the unequal height and force of the blisters, with one blister being hardly detectable at all. Nevertheless, the software and hardware interface between PC and tablet implemented at UAB proved to be well suited for the special requirements in laboratory research settings (precise timing, interface with standard experimental control software, flexibility). Thus, as a proof of concept the suitability of the NOMS tablet as a research tool in studies on tactile perception could be demonstrated. Moreover, the completely noiseless operation of the individual blisters would mean a considerable improvement of existing technology in research settings.
Basic neuroscience studies on the influence of tactile stimulation on auditory localization have been conducted that shed new light on the role of developmental vision and involved cortical circuits in audio-tactile spatial integration
Due to the delay in the delivery of the first NOMS prototype and the low resolution of the device (2x2 only), the original plan to use the NOMS tablet in combination with event-related brain potentials to study the neural correlates of tactile language processing could not be followed up. Instead, basic neuroscience studies on fundamental aspects of tactile perception and multisensory integration of tactile input with input from other sensory modalities were continued using existing tactile stimulators corresponding to single blisters that stimulate for example the index fingers of either hand. In the basic paradigm established previously in our laboratory, participants are asked to indicate the perceived location of sounds presented from different loudspeaker locations while they receive a task irrelevant touch to either the left or the right hand. A group of congenitally blind and a group of late blind participants were tested with this paradigm. The aim of this study was to test the contribution of developmental vision to audio-tactile spatial integration in later life. While both sighted controls and late blind participants similarly mislocalized auditory stimuli toward the external position of the concurrent tactile stimuli, the congenitally blind showed a markedly reduced influence of touch on auditory localization. This finding suggests that the absence of vision leads to a lasting impairment of cross-modal integration in adult life.
Previous studies have suggested that the putative homologue of the ventral intraparietal area in humans (hVIP) is crucially involved in the remapping of tactile information into external spatial coordinates and in the realignment of tactile and visual maps. In order to test whether hVIP is essential for the spatial remapping process during audio-tactile interactions as well, we used transcranial magnetic stimulation (TMS). TMS allows to briefly interfering with neural processing in a focal region of the cortex below the coil through which the pulses are applied. The hVIP was localized on individual MRI scans of a sample of healthy young sighted adults. Then, the participants performed the same audio-tactile localization task as described above, while single TMS pulses were applied either to the hVIP or a control site (the primary somatosensory cortex). hVIP stimulation, compared to the control condition, interfered with the remapping of touch into an external reference frame, i.e. with crossed hands sound localization was shifted to the anatomical side of the stimulated hand, rather than toward the external location of the tactile stimulus. This result supports the idea that TMS selectively modulated spatial remapping prior to cross-modal integration, and suggests that hVIP is crucially involved in coordinate transformations in audio-tactile processing.
Additional studies showed that brief exposure to a consistent spatial disparity between tactile and auditory stimuli for about 10 minutes can induce a lasting shift of auditory spatial localization in the direction of the tactile stimuli. Moreover, this shift of subjective auditory space was associated with a relatively early modulation of auditory ERPs around 100 ms post-stimulus, suggesting that this effect reflects a relatively rapid form of cross-modal plasticity in the mature brain. Taken together, these basic neuroscience studies extended previous work on the role of tactile information in multisensory spatial integration and showed that tactile input is integrated with auditory input in the same manner as visual input, but that the capability to integrate audio-tactile spatial information itself is modulated by developmental vision.
Suitability of tactile versions of standard psychodiagnostic tests of spatial abilities to predict orienting and mobility skills in the blind could be demonstrated
Finally, preliminary studies (i.e. without a NOMS prototype with sufficient resolution being available) on tactile versions of standard tests of spatial abilities converted to the tactile modality were conducted. In particular, several sub-tests of standard intelligence tests (IST-2000R, MWT, Wilde) and additional custom-designed tasks measuring spatial abilities have been converted to the tactile modality (including Braille instructions) and were produced as :hard-copiesß?. Tests intended for later use with the NOMS tablet included the MWT, which is a quick IQ screening based on verbal material, and the subtest :mirror imagesß? from the Wilde Intelligence Test (WIT), in which simple drawings are presented in different orientation and the subject has to identify the drawing that is a mirror image of the others. Both tests are easily transferable to the NOMS tablet and would then allow a neuropsychological assessment of spatial and cognitive abilities without the need for special test materials being available on site. The tactile test versions have been tested with blind participants to estimate their suitability to measure spatial abilities in the blind using tactile media. Results show that the performance in the WIT mirror images test predicts instructor-rated orientation and mobility performance in daily life very well, suggesting that the test is a valid predictor of spatial abilities in blindness. The MWT intelligence test proved to provide reliable estimates of general intelligence levels, making it a useful measure for quick IQ screenings in blind people where standard tests based on visual material cannot be applied.
Exploitation:
An initial exploitation plan was presented at the first Technical Review meeting in March 2011. The exploitation plan was further discussed and updated throughout the second period of the project. Benchmarking and overview is made for the Braille market. A typical Braille device costs about ß?¬4,000 (80 cells) and the price scales linearly with the number of Braille cells (more or less cells, depending on the application). There are several SMEs offering these devices. The main actuation technology here is based on piezo-actuators which are relatively expensive and need mechanical amplification in order to reach sufficient displacement of the Braille pins. The idea of a NOMS single blister element as a tactile indicator used in parallel to an LED indicator or their combination in one element was also formulated as a possible exploitable result of the NOMS project.
An Exploitation Strategy Seminar was held on 19th July 2011 in Brussels as part of a free service offered by the EC. All relevant NOMS partners attended. As part of the preparation process, a validated list of exploitable results has been generated and this has been included in the final version of the exploitation plan.
List of Websites:
http://www.noms-project.eu
Contact: David Wenn, iXscient Ltd. (davew@ixscient.com)
WP1 System specification and risk assessment
Parameters such as speed requirements for specific information display (semi-permanent or fast refreshed display) and the minimum upward force the material must sustain as well as early designs will be identified and discussed in WP1. A tentative design will be agreed on. Contingency plans will also be drafted on the tablet and blister design by considering two or three alternatives, identifying risks to maximize success. Establishing a common base-line will promote early identification of technical problems that could be encountered later on.
WP2 Characterization and optimization of photoactive nanomaterials
WP2 is aimed at developing the CNT polymer composite materials for use in the NOMS tablet, with the parallel development of photo-actuating nematic elastomers as a fall-back option.
Fundamental studies of CNT photo-actuation:
Initial effort in WP1 will identify optimal material combinations within the fields of polymers and nanocomposites. Proper optical actuation is based on a reversible (equilibrium) shape-memory effect at the molecular level. In this shape-memory effect, the actuation is caused by individual tubes in polymer-carbon composite responding to light. The project will undertake a fundamental study of CNT photo-actuation using real-time TEM/AFM methods in order to understand the mechanism in more detail. This will provide a further exploitable outcome of the project.
Functionalisation of CNTs, dispersion and cross-linking:
One key task is to prepare a long polymer : CNT composite, operating within a temperature range of 0 and 40?°C and over hundreds of actuating cycles without significant degradation. The first two requirements limit the choice of polymer matrix to low-Tg UV-resistant materials, while the elastic repeatability demands cross-linking of the polymer. The choice of CNT is important for scaling-up the production in the future, and will therefore focus on multi-walled CNTs, the large reported sales of which suggest that they are becoming the industry standard.
As part of the development, a standard CNT dispersion procedure will be established. Although this is done in dozens of research group around the world, there is still no standard process, which could be quantified and reproduced. Specially developed surfactants will be used, to ensure high affinity of the CNT surface, with a long flexible moiety tailored to the specific polymer matrix to be dispersed. The effect of CNT length on photo-mechanical response in an aligned polymer composite has never been studied. In addition, not only can the length distribution of the CNTs vary greatly, but also there could be some CNT breaking during sonication. Therefore, the CNT length distribution will be determined using a new DLS technique available at UCAM.
After mixing of functionalized CNT in polymer melt (and alignment, see below), the cross-linking has to be initiated to produce a mechanically stable elastic rubber nanocomposite. Ideally this could be achieved by tri-block copolymer self-assembly. This would allow fewer expensive chemical steps and also let the nanocomposite material be efficiently re-moulded into shapes appropriate to individual tasks. This line of research has a direct bearing on the rest of this project, but also is of fundamental importance to the broad scientific and engineering community involved in various aspects of carbon nanocomposites where there is a great need for the advanced and quantitative dispersion and characterization methods, and specially designed surfactants.
CNT alignment and composite characterization:
To achieve high CNT alignment the melt shear-spreading method will be applied to the polymer-CNT mixture. The degree of alignment will strongly depend on geometric parameters of shear-spreading, the polymer viscosity and the tube parameters (length / persistence length ratio). After shearing, the polymer films will be cross-linked. X-ray analysis will be carried out to confirm the CNT orientational order. Mechanical testing on cross-linked nanocomposite elastomers with controlled CNT alignment will follow the protocols established at UCAM and a stress relaxation study will be carried out to establish the accessible range of times / frequencies for the future actuators.
The photo-actuation in aligned nanocomposites will be characterized. First, by constraining a taut strip of the sample in a load-cell rig and illuminating it with a light source, we shall measure the actuating force and use this experiment to optimize the light wavelength / intensity, as well as the sample composition (CNT concentration, length and alignment) Secondly, we shall test the actuation stroke (strain induced by illumination) and response time. Quantities of optimized CNT composites will be provided to WP3 for final blister fabrication.
All of the results of this part of the project are of fundamental importance to a broad field, and will result in a number of high-profile publications.
Liquid crystalline elastomer actuators:
This will be a parallel line of research, aiming to explore an alternative actuating mechanism and also provide a backup line of materials for the NOMS design. Nematic elastomers (NE) doped with azobenzene-based chromophore molecules undergoing trans-cis photo-isomerisation on photon absorption has been shown (by UCAM, and several other research groups worldwide) to be a promising photo-actuating material. In many cases, adding a small concentration of CNTs, aligned by the orientational order of a liquid crystal, is known to enhance actuation properties of NE. These will be investigated.
WP3 NOMS Blister and tablet design
The aim of WP3 is the design and optimization of the blister and the design of the overall NOMS tablet which will be integrated with the other components in WP5. In order to achieve the general objective, the following tasks and objectives have been implemented.
Task 3.1: Prototype blister concept and design:
The objective of this task is to define the concept and design a prototype for the blister according to the characteristics of the photo-actuating materials, such as magnitude of stroke, Young??s modulus, etc. This blister must fit the requirement in terms of force and size defined in the specifications. Two alternative geometries will be investigated, one where the walls contract and the blister flattens and one where the blister is effectively inverted and a centre pin protrudes on exposure to light.
Task 3.2: Microtechnology for blister fabrication:
The objective of this task is to develop the required technology to obtain the correctly sized and shaped blister from a basic shear-aligned film of photo-actuating polymer material. First approach will be based in developing a master stamp, but different microtechnologies will be considered.
Task 3.3: Simulation, characterization and optimization of blister:
FEM analysis will help optimize the geometry. Once technical specifications are provided, several prototypes will be fabricated for integration and evaluation. Parameters to be evaluated and optimized include response time, lateral and vertical resolution
Task 3.4: Overall tablet design and assembly process:
The initial NOMS tablet will be formed by a light emitting substrate, followed by microlenses to focus the emitted light, a photo-actuating layer with assembled blisters, and finally a flexible polymeric layer for protection. The initial tactile display will be a 10x10 tactel array of blisters which will be compatible with the display and tactile representation of Braille text. A second, more challenging design will aim to produce an 80x80 tactel array with improved resolution.
Task 3.5: Protection layer for tactile display surface:
The NOMS tablet will require a protective polymer layer on top of the photoactive layer. It should be capable of withstanding any temperature gradients generated by the light emitting layer and sufficiently robust to withstand direct contact by the user.
WP4 Communication and control
WP4 has developed the transformation software and the control electronics needed to transform images from a computer into tactile information on the tablet. The three main objectives were to develop computer software that can transform any given image into binary information that can be sent to the device, to develop the control electronics needed to transform this information into tactile information and to develop the communication protocol to send information from the computer to the device.
The control electronics developed by WP4 is connected to an LED board developed by Philips that transforms the electric signals into light, to actuate the material and thus move a tactile element to be felt by the user.
WP5 Optics and microsystem integration
An important milestone will be a demonstrator of a polymer surface with regular array of vertical columns contracting and expanding on illumination from below to enable proper 3D representation. Actuating light sources will be selected in terms of power, intensity, integration, resilience, ability to be focused etc.
WP5 will take an incremental approach, initially developing a single blister system before fabrication of an array for the NOMS tablet. To form the single blister with integrated light source, a suitable surface-mount, high intensity LED will be integrated with an appropriately scaled single microlens to form a small, mounted single element light source. The illumination intensities and light directionality will be measured and subsequently the light emitter will be modified to ensure that these meet the :blister: requirements. The resulting unit will be integrated with the blister layer from WP3 to complete a single blister prototype. The prototype will be tested for deformation, including magnitude and temporal resolution, temperature, and finally tactile quality.
The technology developed for the single blister design will be expanded to produce an initial 10x10 tactel array. Two parallel activities are required, the first is the design and fabrication of a light emitting layer and the second is the fabrication of microlenses to focus the light on to the photo-active material. The light emitting layer will be based on experience gained in fabricating a single element. This will either use an arraying out of the single element (e.g. an array of 10x10 surface-mount LEDs with suitable row and column controller) or by using the alternative display approaches : rear projection using a digital light mirror device or a high intensity backlight and monochromatic LCD. These latter approaches will likely be adopted for the 80x80 array if analysis and tests indicate that they can achieve the necessary intensities at target wavelengths. Issues of thermal load and dissipation, suitable use of optical films to increase backlight intensity etc. will be included in the design, testing and fabrication.
The polymer microlens array will be based upon the tablet design (interpixel distances, focal lengths, wavelength of light) and optical design of the blisters. The goal is a thin layer film with an array of microstructure elements implementing the lenses. Depending on the lens depth required, this may be achieved using actual microlenses cast in a UV curable polymer or the lens may be restructured as a Fresnel with multiple prismatic zones in order to minimise the depth whilst achieving a large equivalent curvature. The resulting designs will be first constructed as master moulds, using either laser ablation or diamond fly cutting to create the microstructures, followed by replication onto :tiles:, tiling of these replicates on a drum shim and manufacture of optical film using a UV casting process. Choice of base film (the clear plastic film on which the microstructured optics are cast) and the polymer for structure casting will be matched to the required illumination wavelength and the thermal environment anticipated. The resulting optical film will be tested and characterised for physical and optical properties.
The light-emitting layer (LED arrays for the low resolution displays, and either LED arrays or some other high intensity display type for the high resolution display) and polymer microlenses will be brought together with the blister technology from WP3 and the communication and control hardware & software from WP4 in an integrating activity to produce the final 10x10 and 80x80 NOMS tablets. Careful control of alignment between the light emitting display, microlens layer and blister layer will be required. The resulting tactile displays will be tested for thermal and mechanical issues, including mechanical deformation magnitude, temporal response and tactile properties. The results of testing the 10x10 display in WP6 and WP7 will be used to inform the development of the 80x80 display.
Subsequent to the prototype tablets being produced, a design for manufacture task will be undertaken which will examine methods to scale up the production of the tactile displays components capable of achieving the required quality, along with techniques which are capable of integrating the elements to required tolerances at suitable cost.
WP6 End-user evaluation and neuro-cognitive study
End-users will conduct preliminary testing, providing feedback on material performance and blister design. These first-degree tactile cues will be addressed early-on during the project, providing prompt feedback on the material and tablet design. Some of this work can be done using traditional methods (e.g. raised dots on paper) and dummy blisters on PDMS, but will transfer to using the NOMS tablet once available. This will include tests with both healthy and, if appropriate, visually impaired adults. Prior to such tests with the NOMS tablet, the necessary ethical approvals will be obtained. Psychophysical studies (including absolute thresholds, two-point discrimination, roughness discrimination) and studies testing the use of NOMS for complex spatial tests will be conducted by neuropsychologists. Psychophysical experiments provide information on whether the tactile stimulation provided by NOMS fits with the capacities of the tactile system of humans. Thus, in the initial phase (with a first prototype) the tactile properties of NOMS are iteratively adapted to the need of humans. Subsequently, the capabilities of the NOMS tablet as a research tool in combination with the use of event-related brain potentials will allow studying the neural correlates of language processing in a tactile medium. Finally, it will be tested whether the NOMS tablet can be used to administer psychological tests for verbal and spatial abilities. Simple administration of such cognitive tests would be of great help in vocational guidance and education consulting.
WP7 Dissemination, exploitation and management of intellectual property
Our findings will be disseminated in international conferences and area-specific workshops as well as through a dedicated project website. NOMS will adopt The Web Content Accessibility Guidelines (WCAG)?©. WCAG offers documentation and guidelines to make web content accessible to people with disabilities. Additionally, the consortium will engage in outreach activities, both in primary and high schools. As for academic advancement, research assistants will complete their Ph.D. thesis, thoroughly developing novel research topics lying amidst the breath of the project. Professional development of senior researchers will also be encouraged by project management and executive training programs. The NOMS project will produce a substantial number of publications early-on, given the incipient interest for photo-actuating systems.
The inclusion of industrial partners (Microsharp, Philips and iXscient) in the consortium will form the basis of the exploitation and technology-transfer. Thorough market studies for NOMS tablets or other applications revolving around optical-actuation will be conducted, as well as a cost-benefit analysis.
Project Results:
WP1 System specification and risk assessment
An initial risk assessment was performed early in the project, following the guidelines set out in deliverable D8.2 (Risk assessment procedure), and deliverable D1.1 (Initial risk assessment and contingency plans) was written. Risks have been continuously updated and monitored through internal quarterly reports, leading to deliverables D1.3 & D1.4.
The early provision of the draft specifications document at the beginning of the project allowed the technical work to progress as planned. An update of the system specifications was completed allowing the final components for the 10x10 tablet (e.g. light source, optical design etc.) to be defined and D1.2 was completed in January 2010.
WP2 Characterization and optimization of photoactive nanomaterials
Task 2.1: Fundamental studies of CNT photo-actuation (CNM, UCAM):
The performance of the actuating material is of crucial importance to NOMS. The actuating properties of CNT-nematic elastomers (often referred to as LCE-CNT) have consistently been superior to those of other polymer-CNT systems. For that reason, to provide the WP3 (blister design) with interim working systems, UCAM have continued to study and improve the LCE CNT material. In particular, the main earlier difficulty with photo-actuation of LCE (which required UV light) has been removed by sensitizing the system to visible/IR light by adding a small amount of CNTs or by doping the material with special dyes. Independently, the effort to develop the CNT elastomeric composites using commodity materials have continued at PISAS and UCAM. The main impetus here is to work with thermoplastic (self-assembling) elastomers such that CNT segments would be aligned under shear flow at a high temperature and then the system frozen into an elastic rubber composite on rapid cooling. This a much simpler route for industrial use.
UCAM and CNM studied how the actuating behaviour changes with the composition of the LCE-CNT system. Attempts to quantify the degree of alignment present in our samples were made using X-ray analysis. The results indicate that we managed to achieve a fairly uniform degree of alignment over several samples, thus allowing a quantitative study of how the polymer composition changes the actuation properties.
Studies were carried out by SEM and TEM imaging and X-ray analysis, on both LCE-CNT and EVA-CNT samples. One of the main objectives of this analysis is to study the CNT distribution in the polymeric matrix and the observation of the material contraction phenomenon in situ, at micro- and nanoscale.
Firstly, we started with the examination of LCE-CNT composites by Scanning Electron Microscopy (SEM). We examined the surface of the samples searching for CNT presence, but even after surface carbon coating, it was very difficult to localize the nanotubes.
After that, we used Focused Ion Beam (FIB) milling to micromachine a lamella for TEM examination. The thinned lamella shows heavy irradiation damage, suggesting excessive irradiation. The complex contrast observed in TEM imaging leaves the CNTs still unresolved.
An alternative method of cryo-microtomy for sample preparation was more successful. Figure 1 shows a few TEM images obtained from one mono-domain side-chain LCE sample with 0.1% wt. of CNT content, where the CNTs can be easily observed. This gave us confidence that the protocols for TEM sample preparation were suitable.
A TEM holder was retrofitted with an optical fibre and sealed with a vacuum-friendly epoxy in order to allow in situ actuation and analysis. Initial tests with the JEOL-2010 housed at the Penn Regional Nanotechnology Facility confirmed the vacuum held upon holder insertion. In addition, the optical conductivity of the fibre has been tested and laser light injection on the tip end results in light emission at the other end of the fibre. These results suggest that the fibre has endured the axial compression imposed during vacuum-sealing. These experiments are being further pursued at the moment.
Task 2.2: Functionalisation of CNTs, dispersion and cross-linking and Task 2.3: CNT alignment and composite characterization (UCAM, PISAS):
For the polymeric matrix, PISAS initially used two types of elastomers, namely Styrene-Isoprene-Styrene block copolymer (SIS, Kraton) and ethylene-vinylacetate copolymer (EVA).
To improve compatibility of carbon nanotubes (MWCNT) with Kraton, the MWCNT were modified either with cholesteryl chloroformate (MWCNT-chol), expecting that the aliphatic structure can improve interactions with polyisoprene phase, or with short PS chains (MWCNT-PS), for improving compatibility with the Kraton polystyrene phase. Studies showed that the presence of covalently bonded polystyrene chains in MWCNT-PS significantly improved the filler/matrix interaction, leading to well dispersed CNTs.
Strips of this composite material were elongated at temperatures of 70 ?°C and 80 ?°C. Subsequently, the photo-actuation response was monitored by Dynamic Mechanical Analyser (DMA) in iso-strain mode. The clamped strip was illuminated by red LED at 300 mA for 10 or 30 seconds and the changes in stress were recorded. A negative stress was recorded, pointing to an elongation of the strip in response to light. However, the response was not sufficiently large for the requirements of the project. Therefore, nanocomposites based on Styrene-Isoprene-Styrene block copolymer and containing carbon nanotubes were not considered suitable for preparation of the prototype tablet or for further investigation.
Two kinds of ethylene-vinylacetate copolymers (EVA) were used for nanocomposite preparation, Levapren 500 containing 50% wt. of vinylacetate and EVATANE 28-25, with 28% wt. of vinylacetate.
EVA/MWCNT nanocomposites based on Levapren were prepared by casting from solution using liquid crystalline polymers called :pyrene main chainß? (PyMC) with a PyMC:MWCNT ratio of 10:1, 5:1 or 1:1.
Analysis by dynamometry of stretched samples indicated that the materials thus fabricated were responsive to an applied light source, and that the surfactant:CNT ratio affects the magnitude of this response. It is clear that the sample prepared at a surfactant:CNT ratio of 5:1 shows a significant stress response to an applied light, whereas samples prepared at ratios of 10:1 and 1:1 show no response. It is possible that a ratio of 1:1 may not be sufficient to disperse CNTs sufficiently well in the polymer, while a ratio of 10:1 may lead to crystallization of the phase-separating excess surfactant.
It is also clear that the concentration of CNTs in the polymer can affect the stress response to an applied light. The data indicate that there is a :threshold: concentration of MWCNTs, below which the material is not actuating.
However, when the measurements were repeated two weeks later, there was very little photo-actuation response in the 5:1 sample. This could be due to a loss of orientation of carbon nanotubes on each heating cycle during measurement.
Next, the EVA copolymer containing 28% wt. vinyl-acetate, (EVATANE 28-25) was selected for further investigation. The nanocomposite was prepared by casting from solution using cholesteryl 1-pyrencaroxylate (PyChol) compatibilizer. The ratio of MWCNT/PyChol was 1:5.
The extent of MWCNT dispersion within the polymeric matrix was characterized using TEM. The strip samples in the original unstretched and stretched form with 50% deformation were studied. The main results from the TEM micrographs obtained for EVA composites containing 0.3 and 3% wt. MWCNT-PyChol are the following:
- a good dispersion was obtained due to the use of a cholesteryl 1-pyrenecarboxylate compatibilizer (ratio CNT/compatibilizer was 1:5)
- single carbon nanotubes and a minimal amount of their agglomerates were obtained but this is usual for composite materials
- a good indication of CNT orientation was observed in the stretched samples
A detailed study of these nanocomposite materials was done using DMA analysis. The photo-actuation response was monitored by DMA equipped with red LED (?» = 627 nm) by iso-strain mode. The strip was placed between two clamps and illuminated by the LED at 300 mA (power 58 mW) for 10 or 30 seconds and the changes in stress over time were recorded simultaneously.
Experiments were performed with a stretched strip based on an EVA composite containing 0.3% wt. MWCNT-PyChol with 10%, 20%, 30% and 50% deformation. For the 10% deformation samples, an expansion (negative stress) was obtained upon illumination. For all other stretched samples, a contraction (positive stress) was obtained. In addition, measurements were done for 50% deformation EVA samples containing various other amounts of MWCNTs or SWCNTs. The measured stress changes are summarised in Table 1.
According to the previous results performed at PISAS, a sample of the EVA/0.3% MWCNT composite was sent to CNM for further testing. The materials were characterized by force measurements using a white Luxeon LED (the same as used in the 10x10 NOMS tablet) as the actuating light source. The maximum force obtained was 17 mN. It is worth noting, however, that in contrast to LCE samples, the actuation time increases from the order of a few seconds to minutes.
To prove the repeatability of the actuation of those samples (no sample damage), the same measurements were repeated few days later. In contrast to previous samples measured, no force variations were observed. In fact, the results obtained with the EVA/MWCNT samples were comparable to the ones obtained using LCE dye-doped ones (see task 2.4) making this material a candidate for the blister configuration.
Task 2.4: Liquid crystalline elastomer (LCE) actuators (UCAM, PISAS):
Initially, composites of CNTs with commercial polymers were favoured because of the expected decreased manufacturing costs. However, actuation parameters (magnitude of the stroke, needed exerted force in tactile applications, etc.) in CNT-nematic elastomer (LCE-CNT) composites have outperformed those of polymer-CNT systems. As a result of this, a decision was made to pursue the LCE-CNT actuator material for the initial 10x10 tablet (milestone M2.2).
The incorporation of MWCNTs into side-chain LCEs in order to sensitise these materials to IR or visible light was demonstrated. However, the disadvantages of this method for haptic technologies include mechanical weakness of the material as well as a complex synthesis procedure.
In order to improve the actuation response and the mechanical strength of the polymer, the incorporation of main-chain LCE materials (in which LC units are incorporated into the polymer backbone, rather than into side-chains) was investigated. Hydrosilylation reactions between the divinyl mesogens and siloxane polymers containing Si-H end-groups can lead to polymerization reactions in which the LC units are incorporated into the polymer backbone. Unfortunately, while these have been shown to improve the mechanical strength of the material, they also proved to inhibit the photo-response, perhaps partly due to imperfect coupling to the siloxane polymer matrix. This remains a definite area of future improvement, but given the severe time constraints and pressures in the project, we decided to abandon this route.
The next step, therefore, was to attempt the fabrication of new LCEs made purely from main-chain LCs (thus removing the possibility that the main-chain mesogens are not properly coupled to the polymer). In addition, using siloxane spacers between rod-like mesogenic units would reduce the glass transition and make the elastomer more deformable. The properties of the materials created so far are an issue, however; these polymers do not show the elasticity required for good actuation (being either hard and inextensible, or too soft). We believe that optimisation of the cross-linking ratio may improve this. However, due to the time pressures in the project and the need to deliver the actuating materials immediately, this line of polymers was also abandoned.
The main problem with composites of side-chain LCE with MWCNTs, which were originally used at CNM for prototypes, is that their consistent fabrication in large quantities is extremely difficult (a large amount of attempted reactions do not proceed as required, almost in random fashion -- and from those that do proceed, most actuating strips break during fabrication: at most 1 in 10 strips survives the fabrication process). This was the cause of restricted availability of these samples: sometimes there was no suitable material generated from a full month of synthetic work.
As we started to believe the unreliable chemistry and easy to fracture elastomers are due to the CNTs disrupting the polymer matrix, another method under investigation for simplifying this procedure was now followed. We used a special light-absorbing dye rather than MWCNTs for the photo-sensitising component. Our previous results indicate that MWCNTs are causing photo-sensitisation by conversion of light into heat and so we selected the so-called laser-welding dyes that are very efficient in converting absorbed infrared light into local heat. Selection of such dyes is not a trivial process, since many dyes will not produce enough heat to raise the local temperature of the material by tens of degrees; however, some industrial applications (e.g. laser plastic welding, laser ablation of printing plates) rely on highly absorbing dyes that can heat a surrounding matrix. The dye must be compatible with our polymer (ideally, the dye must be soluble in similar organic solvents) in order to be incorporated into the LCE.
IR1310 dye was obtained. The solubility of the dye in toluene was found to be sufficiently high to achieve high levels of doping in our LCE. We fabricated strips containing dye concentrations (by mass) of 0.1% and 1%. The UV-vis absorption spectra of these samples were measured.
As was to be expected, under a white light source the actuation stress generated by these samples was lower than reported previously for LCE-CNT materials (because the material is largely transparent to light between 500-650 nm). CNM also carried out the measurement using a light source at 980 nm and found that the actuation was comparable to that reported for LCE-CNT samples.
Based on our expectations for the MWCNT materials, the NOMS tablet was designed to work with a white-light LED source. Therefore, in order to avoid changing the tablet design, we also created materials containing a commercial dye, Disperse Red 1 (DR1), which absorbs in the visible region. See Figure 2 for a comparison between the absorption spectra of these and of the IR dye samples.
The heat-activated actuation of these materials was good, with the samples exhibiting similar temperature response to :un-doped: LCE samples.
This material was used to fabricate the blisters in the 2x2 prototype tablet. CNM measured the force produced by the pins in the tablet and found that the material performed better than the original LCE-CNT material with a force up to 35mN (see WP3 for more details).
In general, the results obtained seem relevant and encouraging to continue exploring the dye-doped material as an alternative to LCE-CNTs. In order to create a larger quantity of dye-doped LCE material, experiments were attempted by UCAM using a larger centrifuge reactor in order to create wider (and longer) films. This was achieved, but these films showed a lesser photo-response than previous ones.
WP3 NOMS Blister and tablet design
Task 3.1: Prototype blister concept and design:
According to the prerequisites established for the NOMS device, a new concept of actuator should be designed to accomplish not only Braille and graphical representation restrictions (reduced dimensions with a pitch of 2.5 mm between consecutive actuators for appropriate reading) but also tactile acuity specifications (large vertical displacements (at least 0.3 mm) as well as high forces (minimum values around 15 mN) for correct tactile perception.
Following the above mentioned requirements as well as the material characteristics (see WP2), a :U-shape: configuration was proposed for the fabrication of the actuators to make the most of stress generated during the material actuation. The working principle of the device is based on the light-induced mechanical actuation of a nematic LCE. More concretely, the stress gradient generated into an LCE film which contracts under illumination is used to generate a vertical displacement of a pin. Figure 3 shows an illustration of this concept.
Task 3.2: Microtechnology for blister fabrication:
Two different technological approaches based on moulding alignment of LCE materials were investigated with the main objective to obtain a high density of actuators, thinking of the fabrication of the 10x10 tablet. These two technologies allow local actuation of the material, due to the localized stress gradient generated during the alignment. The first approach proposed is a pure mechanical moulding system (see Figure 4a); the second one is a gas pressure system (Figure 4b). Both are new approaches, with great potential for the fabrication of small actuators as microfluidic valves and pumps. However, due to the restrictive blister requirements (large forces and displacements in a reduced area, with small pitch between actuators), it has not so far been possible to adapt this fabrication technology for the NOMS actuators. Thus, U shaped blisters were the most suitable approach for NOMS tablet design.
To extend the NOMS tablet from 10x10 to 80x80, the proposed approach of optically actuated U-shaped blisters proved not to be possible, due to the high power consumption of the light source necessary to actuate the material. To try to solve this problem, a third approach was proposed changing from optical to thermal actuation of the material through an array of heaters. Figure 4c shows the first array of heaters fabricated with its corresponding circuitry.
Task 3.3: Simulation, characterization and optimization of blister:
Almost all the effort around this task was focused on the characterization and optimization of the designed blisters. Through different setup configurations with one or various assembled actuators (see Figure 5 a-c for an example), different parameters related to blister behaviour were studied: the stress produced when material contracts (blister illuminated), the actuation time, the displacement of the pin, etc., all of them as a function of the power supplied for the light source. By using an auto-translational stage, an accurate dynamometer and a specific programming code, actuation curves of each blister were recorded for subsequent analysis. Comparison of blisters assembled with different material samples (LCE-CNT, and LCE with various dyes types and EVA) was done. Force was successfully measured, reaching higher values than required for the application in most cases. Figure 5d shows an example of the force measurements of three different blisters assembled by using different material samples, for the same power applied. To check the viability of the actuator and the whole system, some cyclic actuation tests were performed, adjusting the operation parameters as the intensity of current and time.
ANSYS?® Finite element analysis software was used for the simulations, which were basically focused on the study of basic characteristics of LCE (temperature distribution and contraction under excitation); tasks related to WP2. Within WP3, some studies were carried out to determine the force and stress distributions inside the elastomer during the actuation of blisters.
Task 3.4: Overall tablet design and assembly process:
Two different 10x10 prototype tablets were assembled and tested, one using LCE material with embedded CNTs and the other one using LCE doped with dye, to increase the absorption of the light (see WP2). Each prototype has a fixed part that contains the LED array with the corresponding circuitry together with the hardware electronics, and a mobile part with the assembled blisters (the actuators), that can be swapped. Blisters of both materials were measured over different periods of time to verify their stability. No actuation differences or material damage was observed, maintaining the good force values measured over several months.
According to UHamburg requirements, a specific prototype version with only four blisters was assembled for perception and acuity testing (Figure 6). The blisters were assembled accurately to guarantee the correct position of the pin (they should not protrude from the top surface of the NOMS tablet at rest to do not hinder the correct finger reading). For the fabrication of these blisters, LCE dye-doped material from UCAM was used. The maximum force reached for each blister was measured individually, as a function of the intensity applied. The graph of Figure 7 shows the force measurements obtained for each blister, with the deviations. In spite of existing significant differences between the force values reached by each single blister, all of them can be perceived by the finger and distinguished from the other ones. However, pin number 3 presented some difficulties.
Figure 8 shows the assembly process for the NOMS tablet and photos of the completed device.
Due to high power requirements of the photo-actuating materials, only 10x10 prototypes were assembled. The 80x80 tablet requires improvements in the photo-actuating materials and a new technological approach for the blisters and light platform. This was not possible to be optimized within the project framework. However, a feasible technology based on thermal actuation of LCE material was presented and explored.
Task 3.5: Protection layer for tactile display surface:
Tests of flexible materials for the protection layer were completed. Several thin silicone materials appear suitable. There is clearly a trade-off between robustness of any cover layer and unimpeded functioning of the tactile device. A small number of very thin materials were selected. Following further testing of thin silicone samples which provide a suitable compromise between robustness and functionality in a tactile device, the best candidate material for incorporation into the demonstrator was Silex (basic silicone) with a thickness of 25 - 50 ?µm.
WP4 Communication and control
WP4 has developed three main results: the transformation software, the control electronics and communications.
Transformation software:
Special software has been developed to transform the visual images into tactile information, and to send this information to the tablet. Two versions of the software have been developed. The first one (Figure 9) is designed to be used with a tablet of 80x80 high resolution tactile elements (tactels).
A second version of the software (Figure 10) is designed to operate with the 10x10 version of the tablet (with Braille standard resolution), as this is the first developed prototype of the NOMS tablet.
The 80x80 version of the software has the following features:
- It can transform any given image into pixelated information, adjusting the threshold of this transformation.
- It allows the combination of several tablets, forming a matrix of up to 6 tablets (two columns and three rows). This combination can then be automatically sent to the corresponding tablets, adjusting the position of each screen to the corresponding device automatically.
The 10x10 version of the software has the following features:
- This version can operate with up to 10 images at the same time, and can send them to the tablet one by one or in a batch.
- It can perform power ramps, adjusting the power of the LEDs in a very precise way over a given period of time. Also, the general power of the LEDs can be programmed to different values.
- The software can also update the device:s firmware when needed, using a simple User Interface. With this, the device can be updated by the user, without having to send it to the developer.
Both versions of the software have the following common features:
- They allow the user to draw simple shapes with the mouse, and can also transform written text into Braille characters.
- The transformed images and/or Braille text can be stored and retrieved from the user:s computer.
- They can send the tablet information to the device using either Bluetooth or USB ports, depending on the convenience of the user.
- They have been developed under the Microsoft Windows platform to be used by the majority of computers.
- They can be updated automatically via Internet, whenever new versions are published. This allows the user to have the latest version with new features and/or bug fixes without having to search and install a new version.
Control electronics and communications:
A device has been developed to control the 10x10 LED board developed by Philips, and to receive information from the computer. This device communicates with the computer either by Bluetooth or USB protocols.
A special protocol has been developed to send the transformed image information to the device. This protocol can operate through Bluetooth or USB communication architectures. It is also capable of sending information related to power ramps and also to update the device firmware.
Four different prototypes have been developed, tailored for the use of each partner (CNM, UHamburg, Microsharp and Philips). CNM needed to perform power ramps in order to test different materials and/or stress the material in different ways, so the ability to perform such operations has been added to their prototype. UHamburg needed to control the timing when showing information on the tablet in a very precise way, so a parallel port has been added to control the triggering of the screens from a central computer. This allows them to perform neuro-psychology experiments with different users. The prototypes for Microsharp and Philips are the same as the prototype for CNM, with only minor changes.
These prototypes can be updated from the Transformation Software, using a special loader stored in the microcontroller memory. This allows the device to be updated in an easy way by the final user.
The design of these prototypes has been made to be easily scalable to the 80x80 version of the NOMS tablet, in case it is developed in the future.
WP5 Optics and microsystem integration
Task 5.1: Design and fabrication of light emitting layer (Philips, Microsharp):
Philips performed an initial scan of the available LED light sources on the market today, for both single blister and 10x10 arrays. Several SMD-LED package sizes were identified that could realistically be used for the illumination array. Smaller form factors are available, down to 1x1 mm, and in the (near) future probably even less. However optical output typically drops one or two decades when scaling down. Since it was unclear in the early stages of the project what would be the illumination requirements for the actuating materials, a decision was made to select the Luxeon C white-light LED for the 10x10 tablet. This would allow for maximum light output over a broad spectrum while still being capable of arraying the devices on a 2.5 mm pitch in order to meet the Braille standard.
For the initial 10x10 tactile display, Philips then analysed the requirements, as relevant for the electronic control (driving) of the LED array that will provide the light. Based on these requirements, Philips has proposed a suitable LED driving electronics architecture, based on previous experience with LED television backlights.
Philips made the complete light emitting layer based on an array of 10x10 LEDs. This layer was successfully implemented in a tactile demonstrator during the first Technical Review meeting in Bratislava. Subsequently, four additional copies of the light emitting layers were built for further testing and incorporation into prototype 10x10 tablets.
The photo on the left in Figure 11 shows the 10x10 light source array in operating mode. Even though no valid data has been given to the system, LEDs illuminate (at low power) corresponding to the random data present at start up. Via the two fans at the bottom side, forced air cooling is provided. A view of the inside of the mounted boards is provided by the photo on the right. As can be seen, the forced air is not obstructed and will cool the heatsink as well as the driver ICs on the driver board.
An internal workshop was held to discuss ideas for the 80x80 light emitting layer. It is not possible to simply scale up the 10x10 LED array to 80x80 (even at the same pitch) because of the very large power requirements, and moving to a higher resolution (1.25 mm pitch) would require much smaller, and hence lower power, LEDs.
With respect to resolution, power and size, pico- or so called micro-projectors were considered worthwhile to investigate as a potential light source for the 80x80 tablet. Only one powerful light source is needed while a dedicated spatial light modulator takes care of redirecting the light to the individual pixels with the desired amount of light intensity. This was investigated by Philips. In addition, Microsharp investigated a transmissive LCD coupled with a pixelated backlight.
A DLP projector was modified with a custom side-entrance to be able to use with any white light source. A laser was used to measure the efficiency of the projector: the laser power was measured at the entrance and exit of the projector. The modified DLP projector was founde to have an efficiency of 46%. Based on the photo-actuating material requirements, around 14000 lumen would be needed as a light source. This is very high. Therefore, it was concluded that a light source based on DLP for the 80x80 tablet is not practical.
In parallel, an investigation was initiated at Microsharp into the use of a monochrome LCD display to modulate a high brightness LED array in order to achieve the resolution and small pixel size required for the 80x80 array. In effect a similar arrangement to the 10x10 LED array is produced. Each of the LEDs is then subdivided by blocks of the monochrome LCD pixels which are switched on or off in order to :shape: the LED to produce smaller, :virtualß? pixels.
When a virtual pixel needs to be illuminated, that virtual pixel (or its centre) are set to be transmissive - that is in the on state and all other pixels are set to be in the off state. At the same time the LED associated with that virtual pixel is set to be on. This is an appropriate illumination system if the following are the case:
1. The intensity of the LED illumination through the on state of the transmissive display is sufficiently bright to actuate the associated blister
2. The transmissive display has sufficient contrast to ensure that the illuminated LED with a virtual pixel in the off state does NOT actuate the associated blister
3. The tactile display is built in a manner which sufficiently limits cross-talk between adjacent :tactels: (associated with virtual pixels) - that is an adjacent on state virtual pixel does not actuate a blister (even partially) which should be in the off state.
This initial experimental work evaluated the feasibility of the 80x80 illumination solution being based on a low resolution modulated LED array gated by high resolution transmissive LCD display. The LED array from the 10x10 demonstrator was used as a backlight. A monochrome STN 320 x 240 resolution transmissive LCD with pixel size of 300 x 300 ?µm was chosen as the evaluation display.
The conclusion of this experimental work was that the intensity of the LED illumination through the on state of the transmissive LCD display is sufficiently bright to actuate the associated blister in the current 10x10 tablet. The transmissive display has sufficient contrast to ensure that the illuminated LED with a virtual pixel in the off state does NOT actuate the associated blister. The tactile display is also built in a manner which sufficiently limits cross-talk between adjacent :tactels: (associated with virtual pixels) : that is an on state virtual pixel does not actuate an adjacent blister (even partially) which should be in the off state.
Therefore the overall conclusion is that, providing the smaller blisters associated with the 80x80 array can be actuated by 40% of the light within the blister area from an LED, then the display should in principle work. However, this is not possible for the current photo-actuating materials, and significant improvements would need to be made to these materials.
Task 5.2: Design and fabrication of microlens array (Microsharp):
Based on the configuration of the U-shape blister and LED source, a full optical design of a single blister element was performed. This will eventually consist of the lens arrangement shown in Figure 12. Two lenses are designed, since redirecting the light in a single stage exceeds the manufacturing limits on facet angles. In addition, by having two stages in the light redirection, the top lens can either be radial (converging to a circularly symmetrical structure) or linear (converging to a linearly symmetrical structure).
The collimating lens has been designed with a novel arrangement. The central part of the lens is a collimating refractive lens and the outer part a collimating TIR lens (with symmetrical prismatic feature). This results in correct collimation (assuming a point source) and all prismatic features have peak angles of less than 48 degrees and thus are manufacturable.
Task 5.3: Integrate light source with single blister (Microsharp, Philips):
The Luxeon C LED has been integrated with the single blister. This has been shown to achieve good actuation even without microlenses between the light source and blister.
CNM used a conventional photodetector to check the efficiency of the microlenses when coupled with the Luxeon C LED. The light intensity was measured at a distance 5 mm from the LED without lenses, with collimating lens only and with the both lenses. Figure 13 shows the relative improvement of the light intensity for the different lens configurations over the case with no lens.
The results indicate that the lenses are functioning as required, though care needs to be taken to accurately position the lenses with regard to each other, the LED and the actuator ribbon position.
Task 5.4: Overall integration of NOMS tablet (Philips):
The overall 10x10 tablet design consists of a number of different layers as described below, from the bottom layer upwards:
- Light emitting layer including LED array, drive electronics and cooling fans (see task 5.1)
- Lens foil, 250 ?µm thick, incorporating both microlens array mounted on either side of a carrier film
- Lower PMMA block with square holes on top (to accommodate the photo-actuating material) leading to reflective chambers on the bottom
- Strips of photo-actuating material formed into rows of blisters
- Thin aluminium plate for heat dissipation (100?µ thick)
- Upper PMMA block, 3mm thick, which includes holes for the pins which are raised by the blisters
- Flexible protective layer, 25 - 50 ?µm thick
The protective layer was omitted on the first prototypes. The individual components were sent to CNM along with two small :tools: to assist in the shaping of the actuator ribbon, one an individual blister :shaper: and one a shaper for a row of 10 blisters.
Subsequently, CNM assembled the first prototype. However, the amount of LCE-CNT photo-actuating material available was only sufficient to partially complete the tablet; 60 of 100 blisters were mounted (Figure 14). Nevertheless, this was suitable for initial testing. Further prototypes incorporating the dye-doped LCE material were subsequently also assembled.
Philips have tested the 10x10 light source and certified it as safe for the user tests at UHamburg. These are the final tests and paperwork necessary before UHamburg can obtain ethical approval for testing the tablet with visually impaired users.
An additional tablet with a 2x2 array of working blisters was subsequently assembled and fully characterized. This tablet has been delivered to UHamburg in June 2012 for them to start testing.
Extending the 10x10 prototype into an 80x80 prototype is not as straightforward as it may seem. There are two main problems: the first is being able to pack the high power LEDs into a small area and be able to manage the heat generated by them in the situation when they are all on at the same time; and the second problem is being able to source an appropriate monochrome LCD with the right pixel size and pitch. While compromises may have been struck to source a display, the fact that a suitable backlight was unavailable caused the partners to decide that the 80x80 display is not feasible with the currently available photo- actuated materials. This was always expected to be a major challenge in the high-risk NOMS project.
WP6 End-user evaluation and neuro-cognitive study
Development of a new tactile acuity measure using dummy blisters showed suitability of polymer material for tactile perception:
Psychophysical testing was performed to assess tactile spatial acuity with polymer material using samples of PDMS pieces with dummy blisters (Braille-like dot patterns and Landolt ring patterns with varying spacing). Character amplitude (0.2 mm) remained constant, but the spacing of the characters decreased by 0.1 log units (~26%) per line, following the common design principles for modern logMAR visual acuity charts. Tactile acuity measured with the PDMS material was compared to acuity measured with traditional thermo-sensitive paper in a sample of young sighted adults. In line with previous studies, perceptual thresholds were lower for the ring patterns. Importantly, however, performance did not differ between the two materials, verifying that polymer material had no adverse effect on tactile perception. Additional studies have been conducted that aim at comparing the active acuity measures obtained with the acuity charts with more traditional passive measures such as two-point discrimination and grating orientation resolution. The results show that tactile acuity measured with the polymer dot charts has high test-retest reliability and agrees well with passive tactile acuity measured as grating orientation resolution. Apart from the direct implications for the design and material properties of the NOMS tablet, these findings have important theoretical value for the advancement of tactile acuity charts as a measure of tactile spatial resolution in applied and research settings.
Suitability of NOMS control board and computer interface for neuroscience studies has been achieved
Upon receipt of a first 2x2 NOMS prototype at month 34, an experiment was prepared to test tactile discrimination performance of simple patterns (akin to the dot patterns used with dummy blisters). This experiment had to be postponed due to the on-going ethical approval procedure (as a consequence of the delay in the delivery of the first prototype). However, note that the major limitation of the 2x2 prototype appears to be the unequal height and force of the blisters, with one blister being hardly detectable at all. Nevertheless, the software and hardware interface between PC and tablet implemented at UAB proved to be well suited for the special requirements in laboratory research settings (precise timing, interface with standard experimental control software, flexibility). Thus, as a proof of concept the suitability of the NOMS tablet as a research tool in studies on tactile perception could be demonstrated. Moreover, the completely noiseless operation of the individual blisters would mean a considerable improvement of existing technology in research settings.
Basic neuroscience studies on the influence of tactile stimulation on auditory localization have been conducted that shed new light on the role of developmental vision and involved cortical circuits in audio-tactile spatial integration
Due to the delay in the delivery of the first NOMS prototype and the low resolution of the device (2x2 only), the original plan to use the NOMS tablet in combination with event-related brain potentials to study the neural correlates of tactile language processing could not be followed up. Instead, basic neuroscience studies on fundamental aspects of tactile perception and multisensory integration of tactile input with input from other sensory modalities were continued using existing tactile stimulators corresponding to single blisters that stimulate for example the index fingers of either hand. In the basic paradigm established previously in our laboratory, participants are asked to indicate the perceived location of sounds presented from different loudspeaker locations while they receive a task irrelevant touch to either the left or the right hand. A group of congenitally blind and a group of late blind participants were tested with this paradigm. The aim of this study was to test the contribution of developmental vision to audio-tactile spatial integration in later life. While both sighted controls and late blind participants similarly mislocalized auditory stimuli toward the external position of the concurrent tactile stimuli, the congenitally blind showed a markedly reduced influence of touch on auditory localization. This finding suggests that the absence of vision leads to a lasting impairment of cross-modal integration in adult life.
Previous studies have suggested that the putative homologue of the ventral intraparietal area in humans (hVIP) is crucially involved in the remapping of tactile information into external spatial coordinates and in the realignment of tactile and visual maps. In order to test whether hVIP is essential for the spatial remapping process during audio-tactile interactions as well, we used transcranial magnetic stimulation (TMS). TMS allows to briefly interfering with neural processing in a focal region of the cortex below the coil through which the pulses are applied. The hVIP was localized on individual MRI scans of a sample of healthy young sighted adults. Then, the participants performed the same audio-tactile localization task as described above, while single TMS pulses were applied either to the hVIP or a control site (the primary somatosensory cortex). hVIP stimulation, compared to the control condition, interfered with the remapping of touch into an external reference frame, i.e. with crossed hands sound localization was shifted to the anatomical side of the stimulated hand, rather than toward the external location of the tactile stimulus. This result supports the idea that TMS selectively modulated spatial remapping prior to cross-modal integration, and suggests that hVIP is crucially involved in coordinate transformations in audio-tactile processing.
Additional studies showed that brief exposure to a consistent spatial disparity between tactile and auditory stimuli for about 10 minutes can induce a lasting shift of auditory spatial localization in the direction of the tactile stimuli. Moreover, this shift of subjective auditory space was associated with a relatively early modulation of auditory ERPs around 100 ms post-stimulus, suggesting that this effect reflects a relatively rapid form of cross-modal plasticity in the mature brain. Taken together, these basic neuroscience studies extended previous work on the role of tactile information in multisensory spatial integration and showed that tactile input is integrated with auditory input in the same manner as visual input, but that the capability to integrate audio-tactile spatial information itself is modulated by developmental vision.
Suitability of tactile versions of standard psychodiagnostic tests of spatial abilities to predict orienting and mobility skills in the blind could be demonstrated
Finally, preliminary studies (i.e. without a NOMS prototype with sufficient resolution being available) on tactile versions of standard tests of spatial abilities converted to the tactile modality were conducted. In particular, several sub-tests of standard intelligence tests (IST-2000R, MWT, Wilde) and additional custom-designed tasks measuring spatial abilities have been converted to the tactile modality (including Braille instructions) and were produced as :hard-copiesß?. Tests intended for later use with the NOMS tablet included the MWT, which is a quick IQ screening based on verbal material, and the subtest :mirror imagesß? from the Wilde Intelligence Test (WIT), in which simple drawings are presented in different orientation and the subject has to identify the drawing that is a mirror image of the others. Both tests are easily transferable to the NOMS tablet and would then allow a neuropsychological assessment of spatial and cognitive abilities without the need for special test materials being available on site. The tactile test versions have been tested with blind participants to estimate their suitability to measure spatial abilities in the blind using tactile media. Results show that the performance in the WIT mirror images test predicts instructor-rated orientation and mobility performance in daily life very well, suggesting that the test is a valid predictor of spatial abilities in blindness. The MWT intelligence test proved to provide reliable estimates of general intelligence levels, making it a useful measure for quick IQ screenings in blind people where standard tests based on visual material cannot be applied.
Exploitation:
An initial exploitation plan was presented at the first Technical Review meeting in March 2011. The exploitation plan was further discussed and updated throughout the second period of the project. Benchmarking and overview is made for the Braille market. A typical Braille device costs about ß?¬4,000 (80 cells) and the price scales linearly with the number of Braille cells (more or less cells, depending on the application). There are several SMEs offering these devices. The main actuation technology here is based on piezo-actuators which are relatively expensive and need mechanical amplification in order to reach sufficient displacement of the Braille pins. The idea of a NOMS single blister element as a tactile indicator used in parallel to an LED indicator or their combination in one element was also formulated as a possible exploitable result of the NOMS project.
An Exploitation Strategy Seminar was held on 19th July 2011 in Brussels as part of a free service offered by the EC. All relevant NOMS partners attended. As part of the preparation process, a validated list of exploitable results has been generated and this has been included in the final version of the exploitation plan.
List of Websites:
http://www.noms-project.eu
Contact: David Wenn, iXscient Ltd. (davew@ixscient.com)