Periodic Reporting for period 3 - EUROBENCH (EUropean ROBotic framework for bipedal locomotion bENCHmarking)
Reporting period: 2020-07-01 to 2022-06-30
Human-centred robots such as prostheses, exoskeletons and humanoids, are becoming increasingly relevant worldwide. Many prototypes are moving out of the lab into everyday applications in a wide range of market domains. Several roadblocks exist in this process. Some of these are technical, while others are related to the lack of reliable performance/safety indicators for these devices to meet international certifications and standardization requirements. The Robotic Research Agenda has emphasized benchmarking as an important instrument to assess the Technology Readiness Level (TRL) and to quantify how robotic solutions match user needs. Recent international efforts (e.g. RockIn, Robocup, Cybathlon, DARPA Robotic Challenge) have confirmed the interest of the scientific and industrial communities in evaluating (and comparing) the performance of robotic systems in real-life environments.
However, a consolidated benchmarking methodology for Robotics does not exist yet. In bipedal robotics, current benchmarking approaches are focused on generic, goal-level performance indicators. They are usually based on competitions, which can hardly describe performance in sufficient detail to allow recognising and quantifying the technical causes of performance. When existing, metrics are numerous and overlapping, and experimental procedures are defined from each lab independently and chosen according to lab-specific functionalities. This situation undermines the basis of scientific (and industrial) progress, i.e. the evaluation of the state of the art, and building on other’s research works. This situation is aggravated by the lack of standard terminology, databases, regulations across domains and research areas.
In order to fill these gaps, EUROBENCH created the first European framework for the application of benchmarking methodology on robotic systems. The framework, specifically focused on bipedal robotic technologies, will include methods and tools to measure system abilities on a rigorous, quantitative and replicable way. The successful achievement of this goal will put Europe at the forefront of the evaluation of robotic systems, facilitating the process of bringing innovative robotic technologies forward to market.
However, a consolidated benchmarking methodology for Robotics does not exist yet. In bipedal robotics, current benchmarking approaches are focused on generic, goal-level performance indicators. They are usually based on competitions, which can hardly describe performance in sufficient detail to allow recognising and quantifying the technical causes of performance. When existing, metrics are numerous and overlapping, and experimental procedures are defined from each lab independently and chosen according to lab-specific functionalities. This situation undermines the basis of scientific (and industrial) progress, i.e. the evaluation of the state of the art, and building on other’s research works. This situation is aggravated by the lack of standard terminology, databases, regulations across domains and research areas.
In order to fill these gaps, EUROBENCH created the first European framework for the application of benchmarking methodology on robotic systems. The framework, specifically focused on bipedal robotic technologies, will include methods and tools to measure system abilities on a rigorous, quantitative and replicable way. The successful achievement of this goal will put Europe at the forefront of the evaluation of robotic systems, facilitating the process of bringing innovative robotic technologies forward to market.
The EUROBENCH project achieved the following results:
- Identification of the needs of stakeholders.
- In-depth analysis of the international State of the Art of the assessment of robotics technologies.
- Priorities on the system ability levels established.
- FSTP procedures created.
- Intense collaboration and interaction with other EU initiatives, e.g. COVR, COST, INBOTS.
- Technical and business success indicators defined.
- Technical specifications of the benchmarks outlined.
- Contents and software architecture of the benchmarking algorithms defined.
- Existing human reference data for the relevant motions identified and made available to the project.
- Development of algorithms for the calculation of walking Performance Indicators initiated.
- FSTP-1 Open Call successfully closed, 17 projects selected and finalized.
- Integration of all the algorithms developed by the FSTP-1 sub-projects and the Consortium into a unified benchmarking Software.
- Testbeds from both EUROBENCH Consortium and Third Parties succesfully integrated into the two testing facilities.
- Assembly of the humanoid robot Reem-C finalized and integrated in the Humanoids facility.
- Facility of Wearable Robots and Humanoids built and operative.
- Validation of the framework with the collaboration of FSTP-2 Third Parties and with the technical support of FSTP-1 sub-project.
- Preparation of a business plan for the Wearable Robot Facility and definition of several actions for the future exploitation of the framework.
- Organization of online Webinars and a final event to involve the scientific community and consolidating the FSTP network.
- Identification of the needs of stakeholders.
- In-depth analysis of the international State of the Art of the assessment of robotics technologies.
- Priorities on the system ability levels established.
- FSTP procedures created.
- Intense collaboration and interaction with other EU initiatives, e.g. COVR, COST, INBOTS.
- Technical and business success indicators defined.
- Technical specifications of the benchmarks outlined.
- Contents and software architecture of the benchmarking algorithms defined.
- Existing human reference data for the relevant motions identified and made available to the project.
- Development of algorithms for the calculation of walking Performance Indicators initiated.
- FSTP-1 Open Call successfully closed, 17 projects selected and finalized.
- Integration of all the algorithms developed by the FSTP-1 sub-projects and the Consortium into a unified benchmarking Software.
- Testbeds from both EUROBENCH Consortium and Third Parties succesfully integrated into the two testing facilities.
- Assembly of the humanoid robot Reem-C finalized and integrated in the Humanoids facility.
- Facility of Wearable Robots and Humanoids built and operative.
- Validation of the framework with the collaboration of FSTP-2 Third Parties and with the technical support of FSTP-1 sub-project.
- Preparation of a business plan for the Wearable Robot Facility and definition of several actions for the future exploitation of the framework.
- Organization of online Webinars and a final event to involve the scientific community and consolidating the FSTP network.
The EUROBENCH project aims to reach the following global objectives, each representing an advance on the State of the Art in the evaluation of robotic performance:
1. Achieving an international consensus in the industrial and academic community on the benchmarking requirements. The involvement of the international community is a required step to convert benchmarks into future standards. The EUROBENCH project is leading several efforts to create a worldwide network of industrial and academic entities in discussing on the specific requirements on standardization and terminology of robotic systems. This is expected to facilitate the inclusion of good practice and regulations for the smooth introduction of robotics systems in the society.
2. Identifying a scientific method to quantitatively assess the system abilities of bipedal robots and establish the causal relationships with the technologies composing them. The EUROBENCH is gathering together algorithms and data from many Third Parties involved through the cascade funding Open Calls. This massive software and data sharing has been never achieved before in the bipedal locomotion field, and is expected to advance the knowledge and habilitate to application of artificial intelligence algorithms to establish reliable predictors of robot performance, and help developers identify technical shortcomings efficiently.
3. Providing practical tools to industry and academy to efficiently test robotic platforms at any stage of development, from early prototyping to commercial products. The novel value proposition of EUROBENCH stands on three main pillars, i.e. facilities, benchmarking algorithms, and datasets, which are being designed to be easy accessible by anyone worldwide. Current Testing facilities (in Europe and US) only provide very simple benchmarks, based on completion time and do/not-do metrics. EUROBENCH, instead, will include a comprehensive set of test benches and metrics to evaluate bipedal robotic performance under a multi-faceted perspective. This will increase the adoption of benchmarking methods as a common practice, and therefore permit to achieve systems with higher quality and lower costs.
4. Achieving an effective utilisation of the benchmarking methodology in the bipedal robotics community and an efficient transferability to other application domains. The main advance in this respect will be to have flexible tools, focused on sub-tasks, which can be used across technological fields and application domains. For this reason the Benchmarking Software and Database will be modular and prone to be extended over time. This approach is not in conflict with the competition approaches, but represents a complementary tool that can help researchers to find causal relationships between the performance during competition and the performance in each of the sub-functions, taken separately.
5. Create a sustainable “benchmarking infrastructure” for the European robotics community. The EUROBENCH project contains a sustainability plan that aims to convert the EUROBENCH outcomes (facilities, software, database) into a business model, based on pay-per-use model, which will offer users different kinds of services. This model, partly based on existing Testing Facilities (e.g. NIST-RACE), includes additional services U8e.g. calculation of Perfromance Indicators, or access to a common Database) which have never been offered before by any other facility for Robot testing worldwide.
1. Achieving an international consensus in the industrial and academic community on the benchmarking requirements. The involvement of the international community is a required step to convert benchmarks into future standards. The EUROBENCH project is leading several efforts to create a worldwide network of industrial and academic entities in discussing on the specific requirements on standardization and terminology of robotic systems. This is expected to facilitate the inclusion of good practice and regulations for the smooth introduction of robotics systems in the society.
2. Identifying a scientific method to quantitatively assess the system abilities of bipedal robots and establish the causal relationships with the technologies composing them. The EUROBENCH is gathering together algorithms and data from many Third Parties involved through the cascade funding Open Calls. This massive software and data sharing has been never achieved before in the bipedal locomotion field, and is expected to advance the knowledge and habilitate to application of artificial intelligence algorithms to establish reliable predictors of robot performance, and help developers identify technical shortcomings efficiently.
3. Providing practical tools to industry and academy to efficiently test robotic platforms at any stage of development, from early prototyping to commercial products. The novel value proposition of EUROBENCH stands on three main pillars, i.e. facilities, benchmarking algorithms, and datasets, which are being designed to be easy accessible by anyone worldwide. Current Testing facilities (in Europe and US) only provide very simple benchmarks, based on completion time and do/not-do metrics. EUROBENCH, instead, will include a comprehensive set of test benches and metrics to evaluate bipedal robotic performance under a multi-faceted perspective. This will increase the adoption of benchmarking methods as a common practice, and therefore permit to achieve systems with higher quality and lower costs.
4. Achieving an effective utilisation of the benchmarking methodology in the bipedal robotics community and an efficient transferability to other application domains. The main advance in this respect will be to have flexible tools, focused on sub-tasks, which can be used across technological fields and application domains. For this reason the Benchmarking Software and Database will be modular and prone to be extended over time. This approach is not in conflict with the competition approaches, but represents a complementary tool that can help researchers to find causal relationships between the performance during competition and the performance in each of the sub-functions, taken separately.
5. Create a sustainable “benchmarking infrastructure” for the European robotics community. The EUROBENCH project contains a sustainability plan that aims to convert the EUROBENCH outcomes (facilities, software, database) into a business model, based on pay-per-use model, which will offer users different kinds of services. This model, partly based on existing Testing Facilities (e.g. NIST-RACE), includes additional services U8e.g. calculation of Perfromance Indicators, or access to a common Database) which have never been offered before by any other facility for Robot testing worldwide.