Periodic Reporting for period 3 - H-Unique (In search of uniqueness - harnessing anatomical hand variation)
Période du rapport: 2022-01-01 au 2023-06-30
H-unique will be the first multimodal automated interrogation of visible hand anatomy, through analysis and interpretation of human variation. It will be an interdisciplinary project, supported by anatomists, anthropologists, geneticists, bioinformaticians, image analysts and computer scientists. We will investigate inherent and acquired variation in search of uniqueness, as the hand retains and displays a multiplicity of anatomical variants formed by different aetiologies (genetics, development, environment, accident etc).
Hard biometrics, such as fingerprints, are well understood and some soft biometrics are gaining traction within both biometric and forensic domains (e.g. superficial vein pattern, skin crease pattern, morphometry, scars, tattoos and pigmentation pattern). A combinatorial approach of soft and hard biometrics has not been previously attempted from images of the hand. We will pioneer the development of new methods that will release the full extent of variation locked within the visible anatomy of the human hand and reconstruct its discriminatory profile as a retro-engineered multimodal biometric. A significant step change is required in the science to both reliably and repeatably extract and compare anatomical information from large numbers of images especially when the hand is not in a standard position or when either the resolution or lighting in the image is not ideal.
Large datasets are vital for this work to be legally admissible. Through citizen engagement with science, this research will collect images from over 5,000 participants, creating an active, open source, ground-truth dataset. It will examine and address the effects of variable image conditions on data extraction and will design algorithms that permit auto-pattern searching across large numbers of stored images of variable quality. This will provide a major novel breakthrough in the study of anatomical variation, with wide ranging, interdisciplinary and transdisciplinary impact.
Our key objectives are (i) To establish variability in the human hand to better understand variation. (ii) To create new algorithms to both reliably and repeatedly extract anatomical features from images. (iii) To determine the extent to which variation in hand position and image quality alters the ability to recognise features of hand anatomy. (iv) To undertake black and white box testing, to establish a hierarchy of hand biometrics. (v) To retro-engineer a multimodal biometric to represent and visualise hand variation thereby establishing uniqueness.
Hard biometrics, such as fingerprints, are well understood and some soft biometrics are gaining traction within both biometric and forensic domains (e.g. superficial vein pattern, skin crease pattern, morphometry, scars, tattoos and pigmentation pattern). A combinatorial approach of soft and hard biometrics has not been previously attempted from images of the hand. We will pioneer the development of new methods that will release the full extent of variation locked within the visible anatomy of the human hand and reconstruct its discriminatory profile as a retro-engineered multimodal biometric. A significant step change is required in the science to both reliably and repeatably extract and compare anatomical information from large numbers of images especially when the hand is not in a standard position or when either the resolution or lighting in the image is not ideal.
Large datasets are vital for this work to be legally admissible. Through citizen engagement with science, this research will collect images from over 5,000 participants, creating an active, open source, ground-truth dataset. It will examine and address the effects of variable image conditions on data extraction and will design algorithms that permit auto-pattern searching across large numbers of stored images of variable quality. This will provide a major novel breakthrough in the study of anatomical variation, with wide ranging, interdisciplinary and transdisciplinary impact.
Our key objectives are (i) To establish variability in the human hand to better understand variation. (ii) To create new algorithms to both reliably and repeatedly extract anatomical features from images. (iii) To determine the extent to which variation in hand position and image quality alters the ability to recognise features of hand anatomy. (iv) To undertake black and white box testing, to establish a hierarchy of hand biometrics. (v) To retro-engineer a multimodal biometric to represent and visualise hand variation thereby establishing uniqueness.
In the first 54 months of the project, we have established large teams at Lancaster University and the University of Dundee. Work has focussed on outreach, establishment of two large ground truth image datasets and a 3D dataset, development of automated feature extraction algorithms and comparison methodology. We have established the infrastructure, databases, mechanisms and collection systems for the required datasets and made excellent progress with collection. This includes development of a multi-camera image capture rig and complete data collection for Dataset 2 (High Quality Dataset), a web-based application to facilitate data collection for Dataset 1 (Large-Scale Citizen Science), resulting in the largest hand image dataset in the world, and a multi-camera rig for photogrammetry for Dataset 3 (3D Hand Dataset). We have developed and implemented our plans for continuous engagement with periodic events to ensure that submissions to our datasets continued and annotated datasets 1 and 2 with expert forensic anthropologists.
A key objective is to develop the ability to extract key features from photographs of the hand, including the superficial veins, knuckle and palmar creases, pigmentation, scars and lunules. We have developed approaches for vein extraction in two modalities (colour photograph and infrared), as well as crease and pigmentation extraction. We have developed localisation techniques to find and identify key regions of the hand from any image (regardless of scene, camera and quality) including knuckles and joints, punctate pigmentation, and fingernails, and lunules. Excellent accuracy is being achieved and this is being evaluated on our datasets. Several scientific papers have been published, already achieving 24 citations, in the top venues in computer vision and biometrics.
Work package 3 has been advanced in several directions at Lancaster and Dundee, well beyond the original plan. We have carried out work to develop the rig for collecting multiple images of the hand for high quality 3D reconstruction and methods of achieving this. Data collection is almost complete. We have developed a technique for determining the 3D surface representation of the hand from single 2D images. We have established refinements for the 3D reconstruction, targeted towards finer features, and sub-region quality assessment with multiple characteristics.
Work packages 4 and 5 have been significantly advanced with the development of feature comparison methodology for anatomical features, including punctate patterns, curvilinear structures, graph structures, and unstructured data. Several studies have been conducted to investigate the relative contribution of different anatomical constructs. Work is ongoing on the study of uncertainty, which is important for the development and adoption of the biometric.
A key objective is to develop the ability to extract key features from photographs of the hand, including the superficial veins, knuckle and palmar creases, pigmentation, scars and lunules. We have developed approaches for vein extraction in two modalities (colour photograph and infrared), as well as crease and pigmentation extraction. We have developed localisation techniques to find and identify key regions of the hand from any image (regardless of scene, camera and quality) including knuckles and joints, punctate pigmentation, and fingernails, and lunules. Excellent accuracy is being achieved and this is being evaluated on our datasets. Several scientific papers have been published, already achieving 24 citations, in the top venues in computer vision and biometrics.
Work package 3 has been advanced in several directions at Lancaster and Dundee, well beyond the original plan. We have carried out work to develop the rig for collecting multiple images of the hand for high quality 3D reconstruction and methods of achieving this. Data collection is almost complete. We have developed a technique for determining the 3D surface representation of the hand from single 2D images. We have established refinements for the 3D reconstruction, targeted towards finer features, and sub-region quality assessment with multiple characteristics.
Work packages 4 and 5 have been significantly advanced with the development of feature comparison methodology for anatomical features, including punctate patterns, curvilinear structures, graph structures, and unstructured data. Several studies have been conducted to investigate the relative contribution of different anatomical constructs. Work is ongoing on the study of uncertainty, which is important for the development and adoption of the biometric.
Most of our work has improved on the state-of-the-art (SOA), evidenced by our world-leading results and publications in top journals and conferences. We have built the world's largest database of hand images and this continues to grow. Our vein segmentation and vein map extraction algorithms improve considerably over the SOA. We have developed the first method for vein map extraction from colour photographs and infrared imaging, demonstrating its improvement over the current SOA on more commonly reported tasks such as crack detection and road mapping. We have developed the first method for simultaneously automatically detecting and labelling the regions of each of the knuckles of the hand and extended this to include fingernails, allowing us to map the skeleton of the hand and automatically break it down into key regions for analysis. We have contrasted this with a similar approach that aims to find the centre-points of the knuckles, demonstrating that our approach also outperforms the current SOA. We have also developed methods for graph matching (to facilitate identification of the vein patterns) and knuckle crease matching, both of which achieve superior results to the SOA. We have developed a method for the holistic comparison of knuckle creases. Our 3D mesh and texture reconstruction achieves the world-leading accuracy and we produced the first model in 3D hand texture reconstruction from a single image. Further, our work in weakly-supervised image segmentation improves beyond the existing SOA by a large margin.
Following on from this, we are further developing our work in weakly-supervised segmentation and uncertainty estimation in terms of biometric identification and image segmentation, which will be important in facilitating adoption of the resulting biometric. We will continue to evaluate our segmentation work on the full set of anatomic features using our datasets and build these into our multimodal biometric, while taking into account real-world factors including lighting variation, partial images and unexpected pose.
Following on from this, we are further developing our work in weakly-supervised segmentation and uncertainty estimation in terms of biometric identification and image segmentation, which will be important in facilitating adoption of the resulting biometric. We will continue to evaluate our segmentation work on the full set of anatomic features using our datasets and build these into our multimodal biometric, while taking into account real-world factors including lighting variation, partial images and unexpected pose.