Periodic Reporting for period 4 - SAHR (Skill Acquisition in Humans and Robots)
Reporting period: 2022-04-01 to 2024-03-31
For decades, robotics has developed simple gripping mechanisms to pick and place objects. Such actions were predetermined according to the object’s size, weight, and shape, all of which were deemed to be known. Robots would execute these tasks with no ability to adapt to a changing environment. As time passed by, it became evident that robots needed to handle errors and uncertainty at run time. Today, artificial intelligence has made it possible to deploy robots capable of picking up a variety of arbitrary objects stacked in random manner. Yet, industrial robots are still capable solely of simple pick and place actions, one object at a time and in isolation. It remains rare to see two robots work with one another and when they do, motion remains fully prescribed and both arms are merely mirroring each other’s actions.
In contrast, humans can skillfully carry numerous objects inside a single hand. They routinely sort objects within one hand, such as when extracting a key from a keyset. For tasks requiring two hands or more, they naturally coordinate their left and right hands and can seamlessly work in a four-handed manner with a colleague.
The Skill Acquisition in Humans and Robots (SAHR) project aimed to go well beyond traditional pick-and-place and endow robots with a dexterity comparable to that of humans. The project first sought to develop a better understanding of human dexterity through a careful study of the acquisition of watchmaking skills. It then developed new AI-based control methods to enable robots to hold and manipulate multiple objects within a single hand and for two robots to work in a coordinated bimanual manner.
In contrast, humans can skillfully carry numerous objects inside a single hand. They routinely sort objects within one hand, such as when extracting a key from a keyset. For tasks requiring two hands or more, they naturally coordinate their left and right hands and can seamlessly work in a four-handed manner with a colleague.
The Skill Acquisition in Humans and Robots (SAHR) project aimed to go well beyond traditional pick-and-place and endow robots with a dexterity comparable to that of humans. The project first sought to develop a better understanding of human dexterity through a careful study of the acquisition of watchmaking skills. It then developed new AI-based control methods to enable robots to hold and manipulate multiple objects within a single hand and for two robots to work in a coordinated bimanual manner.
The project started with the conduct of an eighteen-months long longitudinal study of apprentices in watchmaking at the Ecole Technique de la Vallee de Joux in Switzerland. Apprentices' manual capacities were assessed in six sessions, each a month apart, over two semesters to determine the evolution of their skills during the first year of training. Skill retention was further assessed by administering one more recording session at the beginning of the school year after the summer vacation. Apprentices' skills were contrasted with those of professionals.
We followed an optimal control approach for our analysis and modeling of human skill acquisition. Our findings showed that as novices progressed, they modified their objectives, moving away from comfortable hand postures to favor finger postures that were more aligned with task demands, progressively resembling those displayed by experts. Distinct patterns of force at the fingertips could be extracted and related to levels of expertise. Following these observations, we conducted a controlled study to systematically study and contrast levels of complexity in bimanual coordination as watchmaking tasks increased in complexity. We found that novices tended to explore more hand postures and new role distributions across the two hands when faced with tasks of low complexity rather than the converse. These results suggest that humans search for different hand poses in an effort to improve performance, possibly seeking postures that minimize the influence of sensorimotor noise on precision control, in line with an earlier study by Dagmar Sternad at Northeastern University, who collaborated with us on this study.
To address the question of how to endow robots with algorithm for dexterous fine manipulation, the SAHR project made fundamental advances in machine learning with application to robot control. A first issue we tackled was to enable swift obstacle avoidance in high dimensions with preserving theoretical guarantees such as non-penetration of the obstacle and convergence to the target. This was crucial as dexterous manipulation of one or multiple objects in hand requires to move fingers along and around the objects and other fingers while controlling accurately for contacts, to not destabilize the objects.
SAHR made important advances on the use of dynamical systems to control robots, showing the one could learn dynamical systems with by embedding simultaneously periodic and discrete dynamics of motion in a single control law. This technique was applied to replicate swift switching from polishing to pick and place as displayed in watchmaking with a robot arm manipulator. Additionally, SAHR developed novel approaches to learn and classify control laws with the design of new methods for spectral clustering and manifold learning. These methods found application to enable to enable two or more robot fingers and hands to work in synchronous or asynchronous manner.
We followed an optimal control approach for our analysis and modeling of human skill acquisition. Our findings showed that as novices progressed, they modified their objectives, moving away from comfortable hand postures to favor finger postures that were more aligned with task demands, progressively resembling those displayed by experts. Distinct patterns of force at the fingertips could be extracted and related to levels of expertise. Following these observations, we conducted a controlled study to systematically study and contrast levels of complexity in bimanual coordination as watchmaking tasks increased in complexity. We found that novices tended to explore more hand postures and new role distributions across the two hands when faced with tasks of low complexity rather than the converse. These results suggest that humans search for different hand poses in an effort to improve performance, possibly seeking postures that minimize the influence of sensorimotor noise on precision control, in line with an earlier study by Dagmar Sternad at Northeastern University, who collaborated with us on this study.
To address the question of how to endow robots with algorithm for dexterous fine manipulation, the SAHR project made fundamental advances in machine learning with application to robot control. A first issue we tackled was to enable swift obstacle avoidance in high dimensions with preserving theoretical guarantees such as non-penetration of the obstacle and convergence to the target. This was crucial as dexterous manipulation of one or multiple objects in hand requires to move fingers along and around the objects and other fingers while controlling accurately for contacts, to not destabilize the objects.
SAHR made important advances on the use of dynamical systems to control robots, showing the one could learn dynamical systems with by embedding simultaneously periodic and discrete dynamics of motion in a single control law. This technique was applied to replicate swift switching from polishing to pick and place as displayed in watchmaking with a robot arm manipulator. Additionally, SAHR developed novel approaches to learn and classify control laws with the design of new methods for spectral clustering and manifold learning. These methods found application to enable to enable two or more robot fingers and hands to work in synchronous or asynchronous manner.
The SAHR project has significantly advanced our understanding of human skill acquisition. It conducted the first in-situ longitudinal study of watchmaking apprentices, providing unique, detailed measurements of finger motion and pressure. These real-world data differ from traditional lab-based studies and offer a historical account of watchmaking, a highly complex craft.
The SAHR project offered new insight on human dexterity. Specifically, we documented a novel taxonomy of bimanual hand postures, revealing the role of handedness in fine manipulation. While handedness is well documented for arm movements, its role in fine manipulation has received much less attention.Moreover our work provided evidence that, while humans rarely exploit the full potential of the hands and fingers dexterity, they can do so easily when the task requires to go beyond the usual routine.
The SAHR project contributed important theoretical and experimental advances in machine learning and robotics. It pushed forward the use of dynamical systems as a mode to control robots. While learning dynamical systems was so far performed in a fully supervised manner, results from SAHR show that unsupervised learning techniques can also be used for the identification of the systems. This is advantageous, as the approach does not necessitate explicit labelling which is time consuming and relies on expert data.
The SAHR project was interdisciplinary, with human behavior studies informing robot controller design. Observations from human skill acquisition, such as prioritizing feasibility over optimality, influenced the design of algorithms and robot controllers.
Ultimately, the SAHR project achieved its goal, producing robots with high dexterity. Examples include a robotic hand picking up and holding four objects simultaneously, skillfully balancing a champagne glass, and rotating everyday objects such as a pencil, fork, cable and banana with its fingers.
The SAHR project offered new insight on human dexterity. Specifically, we documented a novel taxonomy of bimanual hand postures, revealing the role of handedness in fine manipulation. While handedness is well documented for arm movements, its role in fine manipulation has received much less attention.Moreover our work provided evidence that, while humans rarely exploit the full potential of the hands and fingers dexterity, they can do so easily when the task requires to go beyond the usual routine.
The SAHR project contributed important theoretical and experimental advances in machine learning and robotics. It pushed forward the use of dynamical systems as a mode to control robots. While learning dynamical systems was so far performed in a fully supervised manner, results from SAHR show that unsupervised learning techniques can also be used for the identification of the systems. This is advantageous, as the approach does not necessitate explicit labelling which is time consuming and relies on expert data.
The SAHR project was interdisciplinary, with human behavior studies informing robot controller design. Observations from human skill acquisition, such as prioritizing feasibility over optimality, influenced the design of algorithms and robot controllers.
Ultimately, the SAHR project achieved its goal, producing robots with high dexterity. Examples include a robotic hand picking up and holding four objects simultaneously, skillfully balancing a champagne glass, and rotating everyday objects such as a pencil, fork, cable and banana with its fingers.