Periodic Reporting for period 3 - AEROARMS (AErial RObotic system integrating multiple ARMS and advanced manipulation capabilities for inspection and maintenance)
Reporting period: 2018-06-01 to 2019-08-31
AEROARMS has developed the first aerial robotic manipulators with multiple arms and advanced manipulation capabilities to be applied in industrial inspection and maintenance, particularly in works at height that involve significant risks for humans and costs (Fig. 1). Special attention has been paid to oil and gas industries (Fig. 2), which has a very important economic impact with 600 Million every year only in Europe and 2 Billion in the world. Only one large refinery has 40.000 Kms of pipes, and 50.000 measurements are needed.
The objectives of AEROARMS have been:
1) Research and innovation in aerial robotic manipulation including multi-directional thrust platforms and dual arm manipulation systems for complex inspection and maintenance tasks requiring dexterity
2) Validation in the industrial environment, including contact sensing while flying (Fig. 2), permanent sensor installation and deployment of a mobile robotic systems
The objectives of AEROARMS have been:
1) Research and innovation in aerial robotic manipulation including multi-directional thrust platforms and dual arm manipulation systems for complex inspection and maintenance tasks requiring dexterity
2) Validation in the industrial environment, including contact sensing while flying (Fig. 2), permanent sensor installation and deployment of a mobile robotic systems
We have developed research activities in control, teleoperation, perception and planning. The following scientific results were obtained:
– Aerial manipulators with integrated position and force control, dual arms (Fig. 3 and Fig. 4), multidirectional-thrust control (Fig. 5), behavioural coordinated control and visual servoing. We developed model based controllers and studied the aerodynamic effects near surfaces such as pipes. We applied behavioural coordinated control methods and admittance controllers for aerial manipulators with multi-directional thrusters, with robustness to rotor failure assured. Different approaches for visual servoing were developed and validated. Controllers for the autonomous helicopters (Fig. 6) were also developed.
– Telemanipulation with force feedback, stability and bilateral control with passivity checks (Fig. 7) and also methods for dynamically constrained tele-aerial manipulation.
– Adaptive vision based on deep-learning in crawler detection and localization, multi sensor 3D mapping and localization including validation in fully autonomous navigation using only environment perception, relative position estimation for manipulation and cooperative perception methods for crawler detection.
– Control aware planning (Fig. 8) and on-board trajectory re-planning with dynamic awareness and reactivity validated indoor (Fig. 9) and outdoor with the long reach manipulator
The project also obtained very important technological achievements, including:
– Fully autonomous aerial robotic manipulators with multiple arms demonstrated indoor and outdoor in grasping and transportation with embedded autonomous perception and planning
– Aerial robotic manipulators with multi-directional thrust demonstrated indoor and outdoor in contact inspection (AEROX industrial contact inspection system)
– Aerial manipulator with an industrial robotic arm suspended of a multirotor platform (SAM) (Fig. 10)
– Adaptation of an existing magnetic inspection crawler robot which can be safely deployed and retrieved by an UAV and perform the inspection in a remote location.
– Adaptation of existing eddy current sensor and ultrasonic sensor for the integration in the end-effector of the aerial manipulator to be used in contact inspection (Fig. 11) and sensor deployment (Fig. 12)
Moreover, we developed fully autonomous navigation, and simultaneous localization and mapping (SLAM) in GNSS industrial denied environment by using on-board 3D LIDAR (Fig. 13)
We have integrated and demonstrated outdoor the different control, perception and planning modules developed in the context of AEROARMS.
We have also integrated the high TRL aerial systems for inspection and maintenance applications including the multirotor system AEROX, and the Suspended Aerial Manipulator (SAM) (see Figure 10) which was used for crawler deployment (Fig. 14)
AEROARMS has been validated in two industrial scenarios: A refinery in North Germany and a Cement Factory in South Spain. The results that achieved TRL 7 are:
– AEROX 2, aerial manipulator for industrial inspection and maintenance which is an evolution of the AEROX in Figure 1, which demonstrated wall thickness measurements (Fig. 11) and sensor deployment (Fig. 12).
– Autopilot of the helicopter system
– TRIC mobile robot for inspection and maintenance, deployable by the aerial robot
– Eddy current sensor for the end-effector of the aerial manipulator and the mobile robot
– Aerial manipulators with integrated position and force control, dual arms (Fig. 3 and Fig. 4), multidirectional-thrust control (Fig. 5), behavioural coordinated control and visual servoing. We developed model based controllers and studied the aerodynamic effects near surfaces such as pipes. We applied behavioural coordinated control methods and admittance controllers for aerial manipulators with multi-directional thrusters, with robustness to rotor failure assured. Different approaches for visual servoing were developed and validated. Controllers for the autonomous helicopters (Fig. 6) were also developed.
– Telemanipulation with force feedback, stability and bilateral control with passivity checks (Fig. 7) and also methods for dynamically constrained tele-aerial manipulation.
– Adaptive vision based on deep-learning in crawler detection and localization, multi sensor 3D mapping and localization including validation in fully autonomous navigation using only environment perception, relative position estimation for manipulation and cooperative perception methods for crawler detection.
– Control aware planning (Fig. 8) and on-board trajectory re-planning with dynamic awareness and reactivity validated indoor (Fig. 9) and outdoor with the long reach manipulator
The project also obtained very important technological achievements, including:
– Fully autonomous aerial robotic manipulators with multiple arms demonstrated indoor and outdoor in grasping and transportation with embedded autonomous perception and planning
– Aerial robotic manipulators with multi-directional thrust demonstrated indoor and outdoor in contact inspection (AEROX industrial contact inspection system)
– Aerial manipulator with an industrial robotic arm suspended of a multirotor platform (SAM) (Fig. 10)
– Adaptation of an existing magnetic inspection crawler robot which can be safely deployed and retrieved by an UAV and perform the inspection in a remote location.
– Adaptation of existing eddy current sensor and ultrasonic sensor for the integration in the end-effector of the aerial manipulator to be used in contact inspection (Fig. 11) and sensor deployment (Fig. 12)
Moreover, we developed fully autonomous navigation, and simultaneous localization and mapping (SLAM) in GNSS industrial denied environment by using on-board 3D LIDAR (Fig. 13)
We have integrated and demonstrated outdoor the different control, perception and planning modules developed in the context of AEROARMS.
We have also integrated the high TRL aerial systems for inspection and maintenance applications including the multirotor system AEROX, and the Suspended Aerial Manipulator (SAM) (see Figure 10) which was used for crawler deployment (Fig. 14)
AEROARMS has been validated in two industrial scenarios: A refinery in North Germany and a Cement Factory in South Spain. The results that achieved TRL 7 are:
– AEROX 2, aerial manipulator for industrial inspection and maintenance which is an evolution of the AEROX in Figure 1, which demonstrated wall thickness measurements (Fig. 11) and sensor deployment (Fig. 12).
– Autopilot of the helicopter system
– TRIC mobile robot for inspection and maintenance, deployable by the aerial robot
– Eddy current sensor for the end-effector of the aerial manipulator and the mobile robot
AEROARMS has developed new methods and technologies for aerial robotic manipulation. Particularly,
It has developed the first fully autonomous aerial manipulators with dual arms, involving autonomous grasping by means of visual servoing without markers, autonomous navigation of long reach dual arms with on-board perception and planning and re-planning capabilities with dynamic awareness.
It also developed new aerial manipulators, with multi-directional thrusts aerial platform, that have been demonstrated in contact inspection and the installation of sensors.
AEROARMS also developed an aerial manipulator suspended from a multirotor platform (SAM) for the deployment and retrieval of an inspection crawler
We also integrated and demonstrated for the first time the integrated planning, perception and control of aerial manipulators in an outdoor environment and also in an industrial environment where we demonstrated SLAM in GNSS industrial denied environment by means of an on-board LIDAR (Fig. 13) by using the map for the planning of the industrial manipulator.
AEROARMS robots demonstrated for the first time the capability to fly to elevated structures and perform manipulation tasks such as the ultrasonic of the eddy current measurements of the wall thickness of a pipe or a tank, which is very important to know the corrosion and avoid potential damages and accidents. They can be also used to install permanent sensors to perform measurements in the plant or to deploy an inspection crawler in inaccessible sites. These tasks currently need people manipulating at height, and are dangerous for the workers who need scaffolding, anchor cables, cranes or cherry pickers. Moreover, they involve huge costs.
With the AEROARMS flying robots the work will be cheaper, faster and safer. Cheaper because the savings could be 700K per refinery per year, 10 times faster, more competitive with 25% percent of bill reduction and avoiding accidents of works at height.
AEROARMS has combined scientific and technological excellence with high industrial relevance. Thus, AEROARMS and its teams have been extensively recognized:
– First time that a robotics project obtains the Overall ICT innovation award of the European Commission (see Figure 15)
– Individuals with relevant awards and recognitions due to the achievements in the project
Moreover, AEROARMS had a very important scientific and technology impact generating up to the date of the Final review 123 papers, including 51 journal papers and 72 conference papers, and the Springer STAR book “Aerial Robotic Manipulation” published in August 2019 with 27 Chapters and 385 pages. Moreover, we have organized 14 Workshops and a Tutorial, we presented the project in 27 industry and end users events and 27 educational events. Moreover, AEROARMS had 139 appearances in the Media
AEROARMS has obtained excellent results situating Europe in the leadership position of Aerial Robotic Manipulation and Application to Inspection and Maintenance.
It has developed the first fully autonomous aerial manipulators with dual arms, involving autonomous grasping by means of visual servoing without markers, autonomous navigation of long reach dual arms with on-board perception and planning and re-planning capabilities with dynamic awareness.
It also developed new aerial manipulators, with multi-directional thrusts aerial platform, that have been demonstrated in contact inspection and the installation of sensors.
AEROARMS also developed an aerial manipulator suspended from a multirotor platform (SAM) for the deployment and retrieval of an inspection crawler
We also integrated and demonstrated for the first time the integrated planning, perception and control of aerial manipulators in an outdoor environment and also in an industrial environment where we demonstrated SLAM in GNSS industrial denied environment by means of an on-board LIDAR (Fig. 13) by using the map for the planning of the industrial manipulator.
AEROARMS robots demonstrated for the first time the capability to fly to elevated structures and perform manipulation tasks such as the ultrasonic of the eddy current measurements of the wall thickness of a pipe or a tank, which is very important to know the corrosion and avoid potential damages and accidents. They can be also used to install permanent sensors to perform measurements in the plant or to deploy an inspection crawler in inaccessible sites. These tasks currently need people manipulating at height, and are dangerous for the workers who need scaffolding, anchor cables, cranes or cherry pickers. Moreover, they involve huge costs.
With the AEROARMS flying robots the work will be cheaper, faster and safer. Cheaper because the savings could be 700K per refinery per year, 10 times faster, more competitive with 25% percent of bill reduction and avoiding accidents of works at height.
AEROARMS has combined scientific and technological excellence with high industrial relevance. Thus, AEROARMS and its teams have been extensively recognized:
– First time that a robotics project obtains the Overall ICT innovation award of the European Commission (see Figure 15)
– Individuals with relevant awards and recognitions due to the achievements in the project
Moreover, AEROARMS had a very important scientific and technology impact generating up to the date of the Final review 123 papers, including 51 journal papers and 72 conference papers, and the Springer STAR book “Aerial Robotic Manipulation” published in August 2019 with 27 Chapters and 385 pages. Moreover, we have organized 14 Workshops and a Tutorial, we presented the project in 27 industry and end users events and 27 educational events. Moreover, AEROARMS had 139 appearances in the Media
AEROARMS has obtained excellent results situating Europe in the leadership position of Aerial Robotic Manipulation and Application to Inspection and Maintenance.