Final Report Summary - FACTORY-IN-A-DAY (Factory-in-a-day)
Executive Summary:
Factory-in-a-Day was aimed at improving the competitiveness of European manufacturing SMEs by removing the primary obstacle for robot automation; installation time and installation cost. The high costs result in payback periods, making the investment in robotized automation economically unattractive. Factory-in-a-Day has helped reduce the installation time (and the related cost) by bringing together hardware, software, and business innovations with a diverse project team consisting of academic partners (TU Delft, TU Munchen, KU Leuven), research institutes (Fraunhofer IPA and IPT, CNRS-LAAS), large companies (Philips, Randstad, Siemens, Materialise, Universal Robots) and SME’s (Lacquey, FactoryControl, EMP Tooling, Delft Robotics, PAL robotics). All of the project milestones and results have been reached according to plan. Below, the most salient events and results are highlighted per workpackage.
In Workpackage 2 we aimed to develop the business models and standards and certifications that are necessary for practical implementation of the factory-in-a-day robots. Various business models have been analysed and as a result the spin-off company Delft Robotics was started, which later became a partner in Factory-in-a-Day. Temp agency Randstad determined the requirements and created the blueprints for setting up a combined robot/human rental/temp agency, and various Factory-in-a-Day partners contributed to the new ISO/TS 15066 standard for collaborative industrial robots.
Workpackage 3 focused on robot system hardware, resulting in a patented gripper design, a library for quick design of 3D printed robot grippers, 3D printed polishing tools, and a Workplace Simulation Tool for quick assessment of the customer requirements. All of these results help speed up the installation process of new industrial robot systems.
The goal of Workpackage 4 was to create robot arms that are aware of all (dynamic) obstacles in their environment, and that respond by moving around these obstacles while still continuing their work. This resulted in several integrated systems for Philips, one by Siemens (who quit Factory-in-a-Day after 18 months due to severe internal reorganizations), reactive motion planning demonstrators by CNRS-LAAS. Additionally, we delivered an Augmented Reality system to transmit robot intentions to human coworkers, in order to reduce unintended motion obstructions. Arguably the most visible result was winning the first prize at the Amazon Picking Challenge 2016. These innovations in automated 3D vision and motion planning makes it quicker to install a new robot system because the programming time is significantly reduced.
The focus of Workpackage 5 was on learnable skills, i.e. to make it easy and fast to teach robots how to execute a new task. An extensive list of scientific publications has resulted from this work, demonstrating how the robot understands the order of execution and learns the proper motions after only a few examples from a human. Simultaneously, Universal Robots developed the UR+ programme (later commercialized into their UR Caps programme) which allows easy connection and integration of hardware and software components into a full robot system.
Workpackage 6 was fully aimed at a framework of software tools for the rapid installation of new robot systems. Through the work in this workpackage, we have achieved a leading position in the internationally accepted standard framework ROS-Industrial, as evidenced by our coordinatorship of the new H2020 project “ROSIN: ROS-Industrial Quality-Assured Robot Software Components”.
All technical innovations were brought together in a series of ‘robothons’ (2-day focused development events) and a dozen demonstrators in Workpackage 7, disseminated through over 40 scientific publications and more than 200 dissemination activities (exhibitions, presentations, press articles, etc.) in Workpackage 8.
Project Context and Objectives:
This section contains the project context and objectives. The text is adapted from the text in the original Description of Work.
The problem
Factory-in-a-Day aimed at reducing the installation time of a new hybrid robot-human production line, from the weeks or months that current industrial systems now take, down to 1 day. The ability to rapidly install (and reconfigure) production lines where robots work alongside humans, will strongly reduce operating cost and open a range of new opportunities for industry – especially manufacturing SMEs – to implement robotic systems which improve productivity, flexibility and competitiveness, while strongly reducing investments and pay-back times.
Typically, a robot installation in a traditional industrial setting used to take 3 months and often more, when measuring the time between first customer contact up to full operability. In optimistic cases it may take about 3 months (roughly 100 days). A survey preceding the Factory-in-a-Day project showed that the long installation time (and associated cost) was the main reason for SME’s to not invest in robotics, because their production batch sizes are too small to justify the investment, see Figure 1. This project therefore targets a 100-fold improvement over current industrial practice, which would benefit not only large industries looking for additional flexibility in their production lines, but particularly to SMEs who will be able to take advantage of the enhanced productivity and much-improved return-on-investment.
Objectives
The objective of this project is to marginalize the system integration cost by reducing the system integration time to one single day. The resultant 50% price reduction of fully integrated systems is not even the main effect. The really significant impact will be that the SME’s no longer have to earn back the investment through only one of their short production batches. In one day, the machines can be re-installed for another temporary product line and continue to be useful. To achieve this radical improvement in installation times, the project will bring together, develop to near-market stage and integrate a number of key technology innovations under development at project partner organisations, that will make a ‘Factory-in-a-day’ concept fully realizable within a few years. The development of these core technological breakthroughs in parallel to the organizational innovations they enable, are the main goal of this project and will radically change the robot automation sector and be a key driver for improving the competitiveness of European manufacturing SMEs.
The key breakthrough technologies are:
1. Safe robot arms with novel proximity-sensing skin and dynamic contact-avoiding behaviours, complemented with underlying inherent mechanical safety, allowing ubiquitous use of robots in shared workspaces with humans. The work builds upon state-of-the-art skin technology from TU München, world recognized path planning algorithms from CNRS-LAAS, and novel depth-perception algorithms of TU Delft.
2. Platform-independent harmonized robot software components for seamless integration with existing machinery and robots, including rapid self-calibration to operate in un-altered environments. The work builds on Fraunhofer IPA’s leading role in the ROS Industrial consortium aiming at such software systems.
3. Standard core hardware modules (e.g. adaptive grippers and arms) plus a procedure to use Additive Manufacturing (3D printing) for task specific parts, brought together in a novel high-speed hardware development-and-installation procedure. The work builds upon Materialises world leading position in additive manufacturing as well as Lacquey’s prominent gripping technology and Universal Robots highly popular safe and intuitive robot arms.
4. Fast teaching software for on-site robot “programming”, using domain-specific models and languages (e.g. optimized for mould finishing or for snap-on assembly) so that only essential parameters and trajectories need to be taught by humans. The work builds on KU Leuven’s novel iTask framework and leading open-source robotics control software.
Consortium
At the start of the project, there were State of the Art initiatives that were already targeting shorter installation periods for production lines in SMEs. For example, project partner Universal Robots already developed low cost robotic arms that can be installed in a matter of hours, that will also contribute to much faster production line installation. EU R&D project SMErobotics, claims to have developed a system that can be installed in 3 days; however this excludes the design and component customisation cycle. ‘Factory-in-a-day’ will build further on these developments to create the first complete hybrid robotic systems that allow for 1-day preparation and installation in a factory line, easy adaptation to changes in the production process and intuitive cooperation with humans.
The project was initiated with a consortium combining these existing initiatives with an extended set of industrial users and leading European universities and research centres on robotics. In addition to the research institutes and companies listed in the previous section (categorized per breakthrough), there were key contributions from Randstad, the world’s second-largest temp agency, Fraunhofer IPT, Philips, Siemens (who unfortunately left the project early), and several smaller companies.
Vision
At the start of the project, the following visual description was made to illustrate the possible timeline of an installation day. Although the research results in Deliverable 3.1 and 3.3 later showed that the installation ultimately would need to be separated in at least two days with some allowance for production time in between, the original illustration still accurately visualizes the core concept of the project.
Project Results:
The content of this section is adapted from all deliverables of the project. It is organized according to the original project structure, per Workpackage and subdivided per Deliverable. All of the technical innovations come together in the Demonstration workpackage (WP7) at the end of this section.
WP 2: New business models and certification procedures
The objective of Workpackage 2, according to the Description of Work, is to develop the business models and standards and certifications that are necessary for practical implementation of the factory-in-a-day robots. In effect, a next-generation type of robot service provider will have to emerge, a combination of a temporary work agency and a classical system integrator. This work package will provide the blueprint for such new companies.
We completed the work by producing the five Deliverables as detailed below. In addition, a spin-off company called Delft Robotics was created to demonstrate the new concepts described in the deliverables. During the run-time of Factory-in-a-Day, Delft Robotics became a partner in the project, contributing to the deliverables in this Workpackage. The other main contributors to this Workpackage were Randstad (WP lead), TU Delft, Fraunhofer IPA, and Philips.
The work started with a detailed task and market analysis, to determine which manual tasks are suitable for automation with the next generation industrial robots (Deliverable 2.1). Next, we performed first a quick scan and then a more extensive study into human factors involved in human-robot co-production (Deliverable 2.2). Simultaneously, we developed the blueprints for new businesses which can leverage the quick-install technologies of Factory-in-a-Day in Deliverable 2.3. The two final deliverables contain theoretical recommendations (Deliverable 2.4) and the practical implementation (Deliverable 2.5) of safety standards and certifications in the context of the quick-install robots developed within Factory-in-a-Day.
Deliverable 2.1: Detailed task and market analysis
Which manual tasks could be robotized first, and which tasks were still too hard for current robot technology? Moreover, which tasks were most economically viable for robotization? To answer these questions, we developed an online quick-scan tool. Business owners could complete a survey, resulting in a report about the robotization feasibility of their proposed task. The analysis of the technical complexity was based on several complexity charts such as the one depicted in Figure 2.
In a similar fashion, the business drivers (economic analysis) was performed to obtain a well-balanced overall indicator for robotization feasibility. In total, over 40 quick-scan reports were issued. The most promising cases, such as box-filling, machine tending, polishing, and yucca-stem planting have been taken as example cases for the entire Factory-in-a-Day project. The section on Workpackage 7 will contain the details of the example cases, but they will emerge throughout this report, forming the connection between the work in all of the workpackages.
Deliverable 2.2: Human Factors quick scan
This section contains not only a summary of the Human Factors quick scan that we have performed, but also a more extensive study into human-robot collaboration, which resulted in three scientific publications. The work started with the definition of four roles played by humans during the different stages of development of a Factory-in-a-Day robot system, shown in Figure 3. The focus of our human factors analysis was mostly on the role of the operator.
There is a large body of textbook knowledge on human factors with respect to the envisioned Factory in a Day system. This considers both physical and cognitive ergonomics. Deliverable 2.2 contains a comprehensive overview of the following topics, including a thorough literature list:
• Engineering physiology: the inclusion of physical ergonomics to assess physical workload and workplace design guidelines. This specifically applies to the operator and operator roles within the envisioned FiaD system.
• Situation awareness: “demons” and tools to improve awareness (feedback, affordances, feed forward). This requires additional attention while exploring the different responsibilities the operator might have during coproduction (Remote controller, Supervisor, Co-worker, or Teammate).
• The notions of assessing usability, which is an extension of the field of cognitive ergonomics including topics like learnability and satisfaction for the system operators.
This deliverable also contained a first overview of safety norms regarding operation and training robots and robotic systems. We briefly discussed the ISO norms 11161, 10218, 13482 and the ISO technical specification 15066. The standards and norms are discussed in more detail in the section on Deliverables 2.4 and 2.5 below.
The literature on actual human-robot co-production is rather sparse. To understand exactly when and how the human worker interacts with the collaborative robot in terms of physical interaction and information exchange, we developed experimental setups to analyze machine tending tasks and to test human-robot collaborative box packaging tasks in a realistic setting , see Figure 4.
The analyses resulted in the “Human-Machine-Product-System framework (HMPS)”. The proposed framework defines explicit and fixed number of actors in the system and forces the expression of the system using these actors: Human, Product, Machine, and System. Furthermore, the types of interaction between the actors are made explicit and are clearly divided between physical transfers and transfers of information. Using this framework, we performed a review of the current state-of-the-art in collaborative human-robot co-production . We concluded that there was surprisingly little actual interaction between robot and human; in most cases, the tasks were clearly divided between the human and the robot and not much interaction was reported, even though the robot and the human shared the same workspace and collaborated on the same tasks. For the remainder of the Factory-in-a-Day project (and beyond), this leads to the decision to focus not much on human-robot interaction during the operational phase, but to focus mostly on developing technologies to speed up the installation phase, while adhering to (but not developing further) the safety requirements for human-robot workspace sharing, see Deliverables 2.4 and 2.5.
Deliverable 2.3: High-level business concept and organizational structure scenarios
The goal of this deliverable was to analyse several potential business models for the quick-install robots created in Factory-in-a-Day. For the analysis, we used the Business Canvas tool as depicted in Figure 5. Three business models showed to have great potential. The first is a business model for a “normal” robot systems integrator, yet one that aims at maximizing the speed of installation. The blueprint for this business model was so promising that the spin-off company Delft Robotics (www.delftrobotics.com) was started. The second business model was called “Combined
Labour/Technology/Process QuickScan (LTPQuickScan)”. Our proposal is that a temp agency such as project partner Randstad collaborates with a technology provider such as Delft Robotics or project partner FactoryControl. Together, they provide the customer with an integral advice on which (sub)tasks to robotize for a new factory installation. This proposal has inspired the start of the Dutch company Smart Robotics, which collaborates with a local temp agency to provide exactly such a service.
The third business model deemed promising at first, was what we called a “Human-Robot-Team Labour Service Provider”. We envisioned a joint venture providing both the robot technology and the human labor. We extensively experimented with this model, trying to create a viable case for customer Bausch and Lomb. Eventually, the results showed that the state of the Factory-in-a-Day technology was not yet sufficiently advanced. The main conclusion, drawn by partner Randstad, is that such business models will only become feasible once the robot system development and installation phases together will take less than one week. Admittedly, the state of technology is not yet ready for that. Therefore, the focus of the Factory-in-a-Day project from this point onward has taken a more technology-oriented approach to maximise progress on the technologies required for quick installation.
Deliverable 2.4: Safety standards assessment and safety training inventory
Current safety regulations and safety standards relevant for robots have not been drafted with the highly modular and reconfigurable robots of Factory-in-a-Day in mind. As a consequence the necessary procedures to achieve compliance are time-consuming and require a large amount of paperwork. In addition alteration of a robot system usually requires almost the full process to be repeated. In order to enable setting up an automation system with the time scale intended in Factory in a Day, it is essential to speed up these procedures.
This deliverable identified approaches to reduce the workload to comply with regulations and especially to speed up the deployment process by certifying parts of the automation system and preparing technical documentation application-independent in advance. These strategies can be supported by the development of software tools that can generate parts of the technical documentation as well as user manuals and training material automatically based on the components of the automation system. Additional approaches are discussed to reduce the amount of training by designing machines in a way they can be intuitively operated or by the use of experienced personnel.
The deliverable furthermore gives an outlook on the possibility to change regulations in order to simplify the way modular safety components are safety certified. One was to establish connection to an ISO standardisation committee dealing with modularity for service robots. This recommendation was followed up in the next Deliverable.
Deliverable 2.5: Novel certification structure for both robot and human labor side
Following the recommendations from Deliverable 2.4 project partner Delft Robotics has implemented novel certification procedures. Within this deliverable, we reported the results, observations, and modifications to the recommendations from a practical point of view. Also following the recommendations, several partners have contributed to the ISO standardization committees dealing with the relevant safety standards. Although the standardization processes are very slow, such that individual contributions are not identifiable, we have provided important input from the Factory-in-a-Day perspective into the development of new or updated standards, as detailed in Deliverable 2.5. In Figure 6, we list all relevant standards for Factory-in-a-Day type of robot systems.
WP 3: System hardware in a day
The objective of Workpackage 3 was to develop the hardware components and the workflow to produce customer-specific hardware components for a complete production line. The first two sections below describe the workflow results, followed by sections on hardware developments, mostly revolving around Additive Manufacturing. The workpackage was led by Materialise with significant contributions from Lacquey, TU Delft, Fraunhofer IPT, Philips, and smaller contributions from the remaining project partners. Note that part of the work in WP3 was in service of other workpackages, delivering the 3D printed components as required.
Deliverable 3.1 + 3.3: Overall factory-in-a-day workflow diagram – including Description of Additive Manufacturing workflow
Deliverable 3.1 describes the robot system installation workflow that, according to our insights, should be adopted by a new type of systems integrator for proper and quick installation of robots in SMEs. The new type of systems integrator should be software-focused, and should develop and maintain a proprietary software architecture and a surprisingly small set of hardware components to select from. This should allow them to install a robot in two site visits, each of one day duration, with some development time between the two days. The main contribution in Deliverable 3.1 is the 4-step procedure depicted in Figure 7.
Note that the result of Deliverable 3.3 (description of additive manufacturing workflow) is part of the overall workflow. Its main result is a decision flow chart to identify if Additive Manufacturing should be used or not, which parts of the gripper should be 3D printed, which printing technologies/materials should be used, and which IP issues should be taken into account. Please consult Deliverable 3.3 for more details on additive manufacturing, and Deliverable 3.1 for detailed explanations of each blue underlined keyword in Figure 7. Although we have not completely succeeded in following the proposed installation workflow, it is in use now as a blueprint for the operations of project partner Delft Robotics.
Deliverable 3.2: Workflow Simulation Tool
One of the most time consuming aspects of new robot system installation, is the communication with the client. Oftentimes months go by, creating a proposed system design, learning of new implicit restrictions from the client, and redesigning the system once again. To alleviate this process, Deliverable 3.2 describes the development and validation of a Workflow Simulation Tool (WST). The tool was developed and tested in close collaboration with system integrators yet offers a fast and intuitive modelling solution to reason on automation scenarios. To enable this, it entails a portable tablet that runs a visual modelling environment, entitled Visual Components, combined with a handheld 3D scanning solution. Complementary to the online files, this report encompasses a description of the tool, installation and user manual, and rudimentary background to introduce underlying concepts of modelling workflows and the selection process of the contributing technologies. Because the tool itself considers commercial hardware and software, a bill of materials is included in Deliverable 3.2. The tool is recommended for all robot system integrators. The workflow with the tool is depicted in Figure 8.
Deliverables 3.4 + 3.5: Library of generic 3d printable components and Design template for 3D printable gripper fingers
Deliverables 3.4 and 3.5 are focused on 3D printed parts. We use the terms “3D printing” and “Additive Manufacturing” as equal alternatives throughout the text.
Deliverable 3.4 concludes that 3D printing is only economically interesting for the production of complete grippers or parts thereof, and not for most other system components. For moving parts within the gripper, the Laser Sintering method is the only one that results in sufficiently durable solutions. The usable materials are PA 12 and TPU. There is a food-safe PA12 available and a food-safe silicone coating. TPU is not food-safe but very flexible. The motion can only be made durable with “living hinges”, i.e. hinges depending on the flexible deformation of the material and not on sliding surfaces (wear through friction). For living hinges, we have found good design guidelines to obtain a proper tradeoff between durability, flexibility, and motion range.
In Deliverable 3.5 we report extensive studies into various types of 3D printed gripper fingers. The original design was to use bellows, which we have modeled and measured, see Figure 9. This type of gripper finger has been patented, however the studies show that it is intractable to accurately model the bending behavior of such fingers, which limits the scope of applications.
The work in Deliverable 3.5 continues with an overview on how to design grippers for 3D-printing. The main benefit of 3D printing is our concept of “printable functionality” for grippers. From hinges over valves, it can all be printed without having to make a compromise by having to select a specific, possibly suboptimal gripper. We list the gripper finger designs for the Philips cases, for various food grippers, and the (for safety) magnetically connected gripper for box filling applications. As a final example, the “Venturi Finger” was invented (patent pending ) showing the combination of clog-free vacuum gripping with mechanical gripping, all in a single finger, see Figure 10. Note that the original intention of this deliverable was to create standard gripper designs with modifiable parameters to address all types of applications. Our extensive experiments have taught us that such “god-type” design attempts are not successful, while a much more effective way is to create a library of design features to be combined quickly in a new design. The library contains for example the Venturi Finger, living hinges, standard flange designs, etc. This approach has been used successfully to create new gripper designs in less than an hour. A final noteworthy remark is that we created an extensive IP map, since there are so many patents already present in the field of gripping (with 3D printed components) that one could almost call it a “patent mine field”.
Deliverable 3.6: Evaluation on adaptive gripper client tests
Task 3.6 started with exploring the methods of 3D printing for adaptive grippers. Several cases were investigated, but turned out to have more simple and effective vacuum alternatives. The fresh food bin picking case required a 3D Ultra adaptive Gripper for 35% of the products. A gripper was developed within the FIAD scope (3D printed, modular, and scalable) and integrated in a test set up with user interface. Successful gripping was achieved and tested for 3 different products (Egg plant, apple, and pear). Speeds were 8 seconds/pick and can be further improved. This system could potentially replace 35% of the order pickers and reduce (human) errors.
Deliverable 3.7: Design template for mould polishing components
Mould polishing was one of the feasible tasks as concluded in Deliverable 2.1. Nevertheless, it is a specific task which requires domain-specific knowledge, so this task was addressed separately by partner Fraunhofer IPT in Deliverable 3.7. The first discovery that was made, together with partner Materialise, was that the material ALUMIDE was perfect to 3D print the polishing tools. The next step was to dismiss the current approach of complete manual polishing, create an automated integral system (Figure 12, top row), and through extensive testing (Figure 12, bottom) determine the minimum number of tools to still obtain properly polished surfaces for any random product design. The automatization of the polishing process not only brings many advantages as repeatability and homogeneity to a process that is used to be manual, but it also brings a few main disadvantages that includes the need of a preparation using CAD/CAM software. More than that, there is also the problem regarding the geometrical limitation of the tools and the machinery used at the process. Since the focus in FiaD is to reduce the setup time on factory environment, the aim for the creation of a tool template should be defined, so that the robotic polishing preparation time could be as short as possible. Along with a template definition, the standardization of the interfaces of the hardware is also mandatory.
WP 4: Dynamic obstacle avoidance
The goal of this work package was to create robot arms that are aware of all (dynamic) obstacles in their environment, and that respond by moving around these obstacles while still continuing their work. To create this awareness, not only the robot’s own sensors is to be used, but also information from existing machinery is to be used, e.g. a second camera (overview) and the position sensor of axis controlled by the production line. The various sensors must be combined into one consistent model of the work environment. Current robots were safe (sometimes), but use obstacle detection only to stop moving. Here we aimed to re-plan the motion trajectory and continue working. At the same time, human co-workers will be made aware of the planned motions, such that they themselves can predict and avoid the robot as well. Additionally, we aimed to create motion plans that fulfill various task specific constraints for typical industrial applications. The automatic consideration of these constraints will drastically simplify and speed-up the deployment of a robot as part of the factory-in-a-day concept.
We have worked toward this goal through various demo setups. This includes the demo setup “TOMM” described in more detail in the videos of WP5, a PR2 mobile manipulator at the site of partner CNRS-LAAS, our final box-picking demonstrator setup and our well-known winning system for the 2016 Amazon Picking Challenge, see Figure 13. One of the main challenges in this workpackage was to integrate all of the (mostly) software technologies: the path-planning component by partner SIEMENS-PLM, the proximity-sensing skin by partner TUM, the reactive motion planner “Stack-of-Tasks” by partner CNRS-LAAS, the framework ROS with the motion library “MoveIT!” by partner TUD, and the (mostly vision) sensor data integration framework (with Deep Learning) from partner Delft Robotics. Through intensive exchange and collaboration, we are proud to have successfully coupled all of these technologies and by doing so winning (ahead of MIT, amongst others) the world renowned Amazon Picking Challenge. Finally, this workpackage has also resulted in a large number of scientific publications as listed in the Dissemination chapter of this report.
Deliverable 4.1: Prototype Robotic Skin to Partners
The main objective was to deliver a prototype of the artificial skin (Figure 14) to the partners in an early stage of the project (month 6), before the final full robot skin becomes available (month 36). This way, other partners can evaluate the skin’s functionality, software integration and application in small scale before its final distribution. TUM therefore developed a hardware and software “Demo Kit” package. In extension to the raw deliverable, TUM hosted a hands-on workshop for all interested partners, in order to transfer the required knowhow to install, operate and apply the skin. The workshop has been combined with topics from the previous workshop on “learnable skills” of WP5, due to an overlap of interests and affiliations.
Deliverable 4.2: Website Video showing Path Planning with Proximity Sensing Data
This deliverable consisted of a first video that illustrates the ability of mobile manipulator robot executing a motion plan task with proximity sensing for reactivity, see Figure 15 (left). The robot is equipped with state of the art control task based reactive controller giving the flexibility to install in any robot platform reducing time for configurations. The proposed simple architecture exploits the hierarchical property of the controller to handle trajectory tracking tasks without compromising on safety and the final goal. The video (https://youtu.be/y-6Oyi21ioQ) shows how a robot autonomously plans a motion towards a target bottle on a table, and executes the plan robustly while evading contact measured with the (demo patch) of proximity-sensing skin.
Deliverable 4.3: Delivery of hardware and prototypes
This deliverable consisted of three components. The first was a follow-up on D4.1 i.e. the delivery of the completed robotic skin to the project partners, as shown in Figure 13 (right). This fault-tolerant low-cost robot skin by TUM, the CellulARSkin, allows to cover a complete robot manipulator, featuring around 300 cells with distance, temperature, force sensor and accelerometers. It has successfully been installed in two UR5 robot arms at TUM and TUD.
The second component was a prototype of our integrated Dynamic Obstacle Avoidance Framework developed by CNRS-LAAS in collaboration with Siemens PLM. A prototype has been released and validated with a simulation of the TOMM robotic setup from TUM, see Figure 15 (right). Finally the third component of this deliverable was our Industrial Sensor Integration Fusion Framework. A prototype of this framework has been developed by TUD using the ROS framework, and validated with the development of the Team Delft robot that won the Amazon Picking Challenge 2016 (Figure 13).
Deliverable 4.4: Website Video Contribution: Reactive Path Planning and Motion Control
The three components of the previous deliverable (4.3) are combined into an integrated demo system which is reported here. The presented deliverable video focuses on the dynamic obstacle avoidance which is an essential component to ensure safety in the robot environment and get the robots collaborate with fellow human beings thus improving the efficiency of the processes in the factory environment which is one of the goals of the Factory-in-a-Day project.
In terms of technology, Skin Sensors from TUM, Reactive Path Planner from SIEMENS-PLM, and Reactive Controller from LAAS are combined into a manipulation scenario to illustrate the dynamical obstacle avoidance capability. The simulation video is a proof-of-concept for the future deployment in industrial robot setups. The illustration is done on the TOMM setup with skin sensors on the right forearm of the robot.
The video (https://www.youtube.com/watch?v=uLStjR7mpOI) has two parts. First, it shows the reactive dynamic obstacle avoidance behavior using the simulated skin sensors with 'Stack of Tasks'(SOT), the reactive controller driving the robot. Secondly, it contains the manipulation scenario which shows the use of the Point Cloud Library - based planner and the reactive SOT controller to avoid obstacles. The techniques shown in the video are used in the final demonstrator which is shown at the end of the project, see WP7.
Finally, Deliverable 4.4 also consisted of a video related to a slightly different topic, namely the first exploration on how Augmented Reality can be used in production planning and during deployment & operation, as shown here: https://youtu.be/uOh7mG5tmnQ. This topic will be elaborated in the next deliverable.
Here, we also report the significant but incomplete contribution from former project partner Siemens AG. Although their newly acquired subsidiary SIEMENS-PLM remained an active partner, the main company Siemens AG decided to stop with Factory-in-a-Day due to severe internal reorganizations. An important contribution was their demonstration of vision sensor data integration into their robot control framework, allowing instantaneous re-positioning of objects by the robot operator. The video is shown here: https://www.youtube.com/watch?v=_6rn-dfz1Sc. Factory-in-a-Day partners have taken over the tasks of Siemens and continued the development of easy-to-use vision-guided robots.
Deliverable 4.5: Report on the effect of awareness augmentation
This deliverable of the Factory in a Day project is concerning the development and validation of the Augmented Awareness Toolkit of human co-workers, as part of work package ‘Dynamic Obstacle Avoidance’. The toolkit was developed and tested in collaboration with system integrators, offering multiple interaction modes for use of augmented reality technology in context of knowledge exchange and decision making in manufacturing environments.
To this end, it entails the use of both traditional 2D displays and head mounted display (HMD) devices to immerse and engage the users in the augmented environment, thus placing virtual simulated content within a real, physical context. The proposed solution features collaborative as well as remote assistance capabilities, presented in three distinct interaction modes. The user experience has been tested in user studies with relevant immersion and situational awareness assessment techniques.
Complementary to the online files, this report encompasses a description of the tool, installation and user manual, and rudimentary background to introduce underlying concepts of environment augmentation and the design rationale for the contributing technologies. Because the tool itself considers commercial hardware and software, a bill of materials is included.
WP 5: Learnable skills
The objective of this work package was briefly formulated as to make it easy and fast to teach robots how to execute a new task.
Deliverable 5.1: Learnable skill model
Deliverable 5.1 is an extensive document which lays the fundaments for the software developments in successive deliverables. It describes the fundamental models for the specification of robot tasks. These models allow us to describe learnable skills in a generic way. One of the first main results was the identification of various types of stakeholders, each with their own contribution to the development of a new robot system, see Figure 18. It shows that the system developer must use or create a Domain Specific Language (DSL) within which it is easy to describe a certain type of robot task. The Application developer can use that to program the robot behavior, which in turn can be used by the Deployer to actually install the robot and set the correct parameters such as pick/place locations. The operator finally can easily monitor the system status and perform basic operations.
The deliverable describes the models, the underlying mathematics, and a proof-of principle implementation in our newly defined task specification language called eTaSL (for expressiongraph-based Task Specification Language) .
Deliverable 5.2: Update (Month 48) Learnable skill model
The work preliminary reported in D5.1 has continued throughout the project and the updated result is reported in Deliverable D5.2. New concepts were added to the eTaSL language and implementation, such as instantaneously coinciding expressions and a way to integrate sensors into eTaSL and thus extend eTaSL beyond geometric constraints. With regards to the software architecture and implementation, more detail was given on the different Orocos components involved in an application and the ROS layer that provides the application. Deliverable D5.2 also described the different approaches by which eTaSL can be extended and listed a number extensions such as extensions for the TUM skin, extensions for point cloud distance computations, extensions for the computation of distances between convex objects, and extensions to provide spline and velocity profile functionality. The resulting overall architecture provides adaptability at different levels:
• A system developer can add new functionality to eTaSL/eTC (e.g. the skin extension, using a new solver, etc.)
• The system can be configured for new Robot hardware and sensors at the Orocos level by adding new hardware driver components and configuring new connections between the components. This is independent of the actual application running at the system.
• New applications can be configured at the level of the ROS-layer.
• The applications can be deployed to new sites using the techniques of programming by demonstration (cf. deliverable D5.4).
More elaborate documentation on eTaSL is given at https://people.mech.kuleuven.be/~eaertbel/etasl.
Deliverable 5.3: Website video contributions
Deliverable 5.3 was intended to show implementation progress for our work on Learnable Skills. Four videos were delivered showing various software components at work to make it easy to teach the robot how to move.
Deliverable 5.4: A model based task specification
This deliverable contains videos representing the work in Task 5.4 which was aimed at two approaches; a scientific approach continuing the work in the previous deliverables, and a direct application-oriented approach based on Factory-in-a-Day insights and existing systems of partner Universal Robots. Specifically, according to the Description of Work, UR developed “the general principles behind an configurable assembly skill in the context of their existing robot controller and GUI. Intuitive programming of this assembly skill is a challenge. It will be investigated whether partial application of already modelled constraints/forces are a solution for this.” We did investigate partial application of already modelled constraints/forces (i.e. the scientific approach) and decided that this was not already a solution. Therefore, the deliverable contains separate videos for the scientific approach (video 1-2) and for the application-oriented work (video 3-4).
Deliverable 5.5: Description of concrete learnable skills
The ultimate result of Workpackage 5 was the implementation of examples of concrete learnable skills. The journey started with the design of a new language and then new tools (see all previous deliverables of Workpackage 5), and these have been used together with tools from other workpackages (mainly WP4 and WP6) to implement two concrete examples.
Concrete learnable skill 1: pick-and-place application
The skill of “learning to pick and place objects while avoiding collisions” is actually a combination of low-level skills which all have been implemented with the eTaSL framework and which have been combined to provide the demonstration videos reported in Deliverable 5.4. Although the low-level skills are quite different as listed below, we are very proud that they have all been implemented in a uniform way. This makes the eTaSL framework highly extensible and maintainable. The concrete low-level skills that make up the pick-and-place application are:
Skill 1: Motion learning from demonstration (see kinesthetic teaching GUI in inset figure →)
Skill 2: Force admittance for interaction
Skill 3: Precise pick positioning
Skill 4: Precise velocity tracking
Skill 5: Limitation avoidance to stay within user-defined workspace
Skill 6: Collision avoidance for known obstacles (using shape primitives)
Skill 7: Collision avoidance for unknown obstacles (using point clouds)
Skill 8: Grasping affordances for different types of objects
Concrete learnable skill 2: polishing application
For polishing applications, highly detailed CAD-models of the workpieces are generally available. The work cell typically contains a tool-changer with storage, and a specifically engineered work cell environment of which also a detailed CAD-model is available. Therefore, kinesthetic teaching as reported for the pick-and-place skill does not lead to added value for this application. However, in this application, a lot of skill knowledge is present (e.g. in selecting tools, lubricants, etc. and the order in which you apply them for a given desired result). Therefore, another approach was followed in which this skill knowledge is encapsulated in a knowledge database/expert system and accessible in a graphical user interface where the user can select the desired finishing of indicated areas of the workpiece. The implementation of the overall skill once again consists of various low-level skills which have been implemented using Factory-in-a-Day results and common technologies (such as ROS, URDF-files, SMACH, Move-It! ) as were used in work package 4 and 6. A screenshot of the user-friendly GUI is shown here.
WP 6: Rapid installation software framework
The goal of this work package was to develop a complete framework for the rapid installation of a mobile manipulation system in new environments to cooperate with onsite workers. In order to focus on the acceleration of installation time the aim is to use and integrate existing software components as much as possible. The “Robot Operation System” (ROS) community that holds a huge collection of open available functionality and drivers will be the main source in this project. This workpackage has resulted in very strong results, captured partly in the new H2020 project “ROSIN: ROS-Industrial Quality-Assured Robot Software Components”.
Deliverable 6.1: Report on Quality metrics and evaluation infrastructure
A variety of service robots is available as standard platforms allowing a worldwide exchange of software for applications in the service sector or industrial environments. Open source software components enhance this sharing process, but require the maintenance of a certain level of quality. This deliverable presents an approach to an automated testing and evaluation platform which facilitates the sharing of capabilities over different robot types, environments and application cases.
Deliverable 6.2: Robot Deployment Toolbox Prototype
This deliverable presents a deployment toolbox which facilitates continuous deployment in a distributed development environment. Based on the architecture shown in Figure 20, we have created a deployment toolbox which allows quick installation and combination of several robot software components here depicted as “App repository”. The toolbox is available at the repositories of partner Fraunhofer IPA, and it has been validated with two versions of the Factory-in-a-Day demo setup (the Philips case). The particular software for that setup is hosted at (https://github.com/fiad-project/fiad_hackathon_philips and https://github.com/fiad-project/bandl_robothon).
Deliverable 6.3: Final Robot Deployment Toolbox And Working Robot Deployment Process
This deliverable presents the final version of the deployment toolbox which facilitates continuous deployment in a distributed development environment. With the vast amount of packages available in the open-source ROS community, it is hard for a user to select the appropriate or best one for his application. There are various approaches for solving this selection problem: Test-Driven-Development, benchmarking and continuous integration are widely adopted approaches in general software development processes, whereas in the domain of robotics, especially for distributed open source development environments like ROS, Test-Driven-Development is not widely used today.
The deployment toolbox presented in this deliverable addresses this situation by also allowing testing at integration and system levels.The test framework is available as running implementation at Fraunhofer IPA. It is running in multiple test instances at internal Fraunhofer servers which can be made available for project partners upon request. The main result is the set of deployment environment scrips available at https://github.com/ipa-mdl/ipde_utils accompanied by three scientific contributions.
WP 7: Demonstration/validation
The objective of this work package was to validate the robotic systems and the business implementation procedures. This work package formed the central point of coordination of all of the tests and demonstrations that have taken place throughout the project, and as such it also served to connect all of the technical innovations.
Deliverable 7.1: Task list, evaluation metrics and test protocols
The first task within Workpackage 7 was to create a framework for evaluating cases and progress in the Factory-in-a-Day project. Since Factory-in-a-Day was the first project to focus on the installation process, we had to develop a framework of measures how to analyse such installation processes. The framework is comprehensively documented in Deliverable 7.1. It consists of an extensive task list of tasks and cases relevant for the project, a list of metrics to ‘score’ tasks and measure the progress of the project, and the test protocols outlining in general terms how to measure the metrics. Furthermore, we subdivide the metrics in Key Performance Indicators (KPI’s), capturing the goal of the project, and PI’s, capturing the constraints of the project. The most important KPI’s for Factory-in-a-Day are listed in Figure 21 below. Throughout the project, we have reverted back to these KPI’s.
Deliverable 7.2-7.4: Demonstration reports
To obtain an overview, we decided to combine the three Demonstration reports (Deliverable 7.2-7.4) because the each build on top of each other. The demonstration cases are shown chronologically in Figure 22. The demonstrators can be categorized in a few “dependency lines”.
First, we evolved from a box-filling case (using fixed, pre-programmed positions) to a system which completely autonomously detects objects and determines how to move, ultimately leading to industrial quality results in the Philips 3 case and leading to our team winning the Amazon Picking Challenge. In the latter case, the robot was able to recognize various types of objects on cluttered shelves, and could pick and sort them as in an Amazon distribution centre, scoring better than top international teams from MIT, Mitsubishi, Bonn, etcetera.
Second, a more academic research setup was completed, successfully showing the combination of kinaesthetic teaching (show the robot what to do) with vision and motion planning. The proximity-sensing skin of partner TU Munich was an important component in that demonstrator, which was transferred also into the final box filling demonstration. Third, we have worked on a separate setup for the automation of mould polishing tasks, according to the Description of Work. Although the setup differs mechanically from the other setups (larger machine, more rigid placement of mould and tools), we have re-used ROS-based software and we have equally analysed the use of 3D printing for quick creation and setup of tools. Finally, separate demonstrations have been delivered by Siemens (quick-install vision-based pick-and-place actions), by CNRS-LAAS (mobile manipulator demonstrating reactive motion planning behaviour), and by Universal Robots (the now commercially available UR Caps platform, resulting from Factory-in-a-Day demo “UR+”). Videos of these demonstrators are online here: http://www.factory-in-a-day.eu/media/videos/.
An important remark is that the complexity of the robotic task was increased for each subsequent demonstrator, which had implications for the installation time. Especially for the “Philips 3” case in year 3, the installation time and effort was exceptionally high due to the fact that we wanted to break away from academic demonstrations and create industrial-quality robust performance. The four graphs in Figure 23 clearly show that the investment paid off; from the “Philips 3” case onward, we achieved 100% of the industrial robustness specs (i.e. uptime according to requirements, usually 95%).
The Amazon Picking Challenge demonstrator was the most visible result of Workpackage 7. The challenge 2016 included two parts: in the picking challenge a set of products from the Amazon product range needs to be picked from the shelf and placed in a tote. For the stowing challenge, it was the other way around: products are to be picked from a tote and stowed into the shelf. The robot system consisted of a 7 DoF manipulator, ROS software with MoveIT as a core component for motion planning and Deep Learning as a core algorithm for object recognition. The system completed the series of 12 item picks within 6.5 minutes, less than half of the allotted time thereby securing victory for the team.
Potential Impact:
The content of this section is largely based on Deliverable 8.9 and potentially overlaps with information in the tables that will (should) automatically be generated by the SESAM / Participant Portal interface for Section 4.2 of this report.
In this section, we give an overview of the main results of the Factory-in-a-day project and a draft idea on how consortium partners intend to use these project results in on-going and future activities, in their further R&D and product marketing strategies. We list in detail the exploitation activities implemented over the project’s four-year period. The results provide manifold possibilities for commercial exploitation, as well as take-up in scientific projects. Therefore, this deliverable overviews all the partners’ exploitation activities including: MSc/PhD thesis, teaching/training services, follow-up projects, open source software, product/service development, spinoff/start-up companies and IPR actions (e.g. patents).
To sum up, these are the highlights in terms of exploitation:
- 1 spin-off company directly from the project: Delft Robotics
- 2 patents (Materialise: Printable Venturi Valve & robot-Fingertip/ TUM: Interactive robot surface)
- 12 projects (different funding schemes) that build on results of the project
- Software components: several open-source ROS-control frameworks, also used in ROS-industrial, MoveIT, Kineo Path Planner ROS-component
- Products:
• developed gripper placement algorithms in latest product CoreTakr
• URCaps / UR+ has been a great success for UR and some third party hardware vendors. A number of registered active developers are working on new extensions for the UR robot platform.
Other partners are still in discussion about licensing or building spin-off companies (TUM/TU Delft), results might only be expected to be finalized after the end of the project.
Deliverable 8.8: Survey amongst SME’s
Before listing all of the Factory-in-a-Day dissemination and exploitation activities, it is useful to provide insight in the motivation of SMEs for using (not) robotic technology Therefore, as reported in Deliverable 8.8 we carried out a survey entitled “Market Analysis of Plug and Work Robots for SME’s”
In the deliverable, we describe the current market situation of industrial robots in Europe. Thereby, the definition of SME’s, the market volume and its players are introduced. This section aims to give a rough overview about the development and market sizes in the industrial robot industry.
The number of sold robots in Germany and Europe is expected to increase. Especially within the automotive industry, where industrial robots are used the most, medium-sized and large Small and medium-sized enterprises (SME’s) have increased their amount for new investments. Considering that personnel is one of the biggest cost blocks for those companies and traditional robots are quite expensive, new concepts such as ‘Plug and Work’ robots aim to reduce costs and help SME’s to ensure competitiveness.
However, to tailor Plug and Work robots for SME’s it is necessary to understand their needs and concerns. Therefore, an online questionnaire study in the three biggest SME industries (i.e. automotive, metal, and rubber and plastics) in Germany was conducted. The goal of the questionnaire was to figure out how SME’s perceive their current situation and how they assess Plug and Work robots.
The results confirm our notion that SME’s perceive the ongoing price competition as threat for their business. When being asked about the reasons why they are reluctant to implement robots, costs and maintenance were mentioned as barriers. In the same vein, they expect that robots are safe, flexible, and easy to program.
All of the mentioned points constitute an advantage in Plug and Work robots compared to traditional robots. Derived from our survey we found out that when promoting Plug and Work robots for SME’s it is necessary to create offers which include trainings for SME’s how to use them. As a medium to advertise, especially classical media such as tech magazines or fairs should be used.
Deliverable 8.9: Exploitation activities
The project’s results cover a range of different exploitable assets. During the time of writing the proposal, “at least two spin-off companies are expected to arise from Factory-in-a-Day. For both companies, preliminary business plans have already been investigated”, the current state of the spin-off aspect is the following:
- TUM is in ongoing discussions about licensing the skin-sensor technology and also discussing several options for the creation of a spin-off company. At the point of writing this deliverable, no decisions have been taken.
- The idea to create a company at TU Delft for renting out robots has not been followed, instead another spin-off company, Delft Robotics, has been founded in Delft (see Table 1 below)
- TU Delft is also looking into a another spin-off company with respect to the Workflow Simulation tool (see Table 1 below)
Two patents have been filed from different partners (Materialise/TUM, more details below).
Inventions and ideas from the projects can further be found in several products, for instance a machine for lettuce-handling and in the online platform UR+, an online showroom, providing you with cutting-edge products to customize a UR robot.
The industrial partners, Materialise, Philips, PAL Robotics and Siemens also used results from Factory-in-a-day for adjusting internal production processes or updating products.
In the last years, several projects have been created during the runtime of Factory-in-a-day, so that the knowledge created in our project will be further increased and also spread in different directions on a European as well as a national level. Factory-in-a-day partners participated or initiated several H2020-EU-funded projects: ROS-IN, Scalable, Robmosys, Co4Robots as well as several projects on a national level (Belgium/The Netherlands). Demonstrations and publications from these projects also contribute to a further enhancement of knowledge originally gained from Factory-in-a-day on an international level.
With respect to software developments - there are a number of open source repositories available, where the developments of Factory-in-a-day are accessible. The work of Factory-in-a-day has always been closely linked to ROS (Robot Operating System). Several partners contribute to the ROS community with different code, see the following links to the software repositories:
https://github.com/ipa-mdl/ipde_utils
https://github.com/ipa-fmw/atf
https://github.com/git-afsantos/haros
http://moveit.ros.org/
http://wiki.ros.org/ros_control
Furthermore, Siemens will update the latest Kineo Path Planning software with a ROS-component. This includes a major update on the Kineo Collision Detector (KCD) for faster and more configurable collision checks. PAL Robotics now uses Automated Test Framework and Docker as internal tools.
Below we provide two overviews of exploitable results, first as listed per workpackage (Table 1) and then listed per partner (Table 2).
List of Websites:
http://www.factory-in-a-day.eu/
Factory-in-a-Day was aimed at improving the competitiveness of European manufacturing SMEs by removing the primary obstacle for robot automation; installation time and installation cost. The high costs result in payback periods, making the investment in robotized automation economically unattractive. Factory-in-a-Day has helped reduce the installation time (and the related cost) by bringing together hardware, software, and business innovations with a diverse project team consisting of academic partners (TU Delft, TU Munchen, KU Leuven), research institutes (Fraunhofer IPA and IPT, CNRS-LAAS), large companies (Philips, Randstad, Siemens, Materialise, Universal Robots) and SME’s (Lacquey, FactoryControl, EMP Tooling, Delft Robotics, PAL robotics). All of the project milestones and results have been reached according to plan. Below, the most salient events and results are highlighted per workpackage.
In Workpackage 2 we aimed to develop the business models and standards and certifications that are necessary for practical implementation of the factory-in-a-day robots. Various business models have been analysed and as a result the spin-off company Delft Robotics was started, which later became a partner in Factory-in-a-Day. Temp agency Randstad determined the requirements and created the blueprints for setting up a combined robot/human rental/temp agency, and various Factory-in-a-Day partners contributed to the new ISO/TS 15066 standard for collaborative industrial robots.
Workpackage 3 focused on robot system hardware, resulting in a patented gripper design, a library for quick design of 3D printed robot grippers, 3D printed polishing tools, and a Workplace Simulation Tool for quick assessment of the customer requirements. All of these results help speed up the installation process of new industrial robot systems.
The goal of Workpackage 4 was to create robot arms that are aware of all (dynamic) obstacles in their environment, and that respond by moving around these obstacles while still continuing their work. This resulted in several integrated systems for Philips, one by Siemens (who quit Factory-in-a-Day after 18 months due to severe internal reorganizations), reactive motion planning demonstrators by CNRS-LAAS. Additionally, we delivered an Augmented Reality system to transmit robot intentions to human coworkers, in order to reduce unintended motion obstructions. Arguably the most visible result was winning the first prize at the Amazon Picking Challenge 2016. These innovations in automated 3D vision and motion planning makes it quicker to install a new robot system because the programming time is significantly reduced.
The focus of Workpackage 5 was on learnable skills, i.e. to make it easy and fast to teach robots how to execute a new task. An extensive list of scientific publications has resulted from this work, demonstrating how the robot understands the order of execution and learns the proper motions after only a few examples from a human. Simultaneously, Universal Robots developed the UR+ programme (later commercialized into their UR Caps programme) which allows easy connection and integration of hardware and software components into a full robot system.
Workpackage 6 was fully aimed at a framework of software tools for the rapid installation of new robot systems. Through the work in this workpackage, we have achieved a leading position in the internationally accepted standard framework ROS-Industrial, as evidenced by our coordinatorship of the new H2020 project “ROSIN: ROS-Industrial Quality-Assured Robot Software Components”.
All technical innovations were brought together in a series of ‘robothons’ (2-day focused development events) and a dozen demonstrators in Workpackage 7, disseminated through over 40 scientific publications and more than 200 dissemination activities (exhibitions, presentations, press articles, etc.) in Workpackage 8.
Project Context and Objectives:
This section contains the project context and objectives. The text is adapted from the text in the original Description of Work.
The problem
Factory-in-a-Day aimed at reducing the installation time of a new hybrid robot-human production line, from the weeks or months that current industrial systems now take, down to 1 day. The ability to rapidly install (and reconfigure) production lines where robots work alongside humans, will strongly reduce operating cost and open a range of new opportunities for industry – especially manufacturing SMEs – to implement robotic systems which improve productivity, flexibility and competitiveness, while strongly reducing investments and pay-back times.
Typically, a robot installation in a traditional industrial setting used to take 3 months and often more, when measuring the time between first customer contact up to full operability. In optimistic cases it may take about 3 months (roughly 100 days). A survey preceding the Factory-in-a-Day project showed that the long installation time (and associated cost) was the main reason for SME’s to not invest in robotics, because their production batch sizes are too small to justify the investment, see Figure 1. This project therefore targets a 100-fold improvement over current industrial practice, which would benefit not only large industries looking for additional flexibility in their production lines, but particularly to SMEs who will be able to take advantage of the enhanced productivity and much-improved return-on-investment.
Objectives
The objective of this project is to marginalize the system integration cost by reducing the system integration time to one single day. The resultant 50% price reduction of fully integrated systems is not even the main effect. The really significant impact will be that the SME’s no longer have to earn back the investment through only one of their short production batches. In one day, the machines can be re-installed for another temporary product line and continue to be useful. To achieve this radical improvement in installation times, the project will bring together, develop to near-market stage and integrate a number of key technology innovations under development at project partner organisations, that will make a ‘Factory-in-a-day’ concept fully realizable within a few years. The development of these core technological breakthroughs in parallel to the organizational innovations they enable, are the main goal of this project and will radically change the robot automation sector and be a key driver for improving the competitiveness of European manufacturing SMEs.
The key breakthrough technologies are:
1. Safe robot arms with novel proximity-sensing skin and dynamic contact-avoiding behaviours, complemented with underlying inherent mechanical safety, allowing ubiquitous use of robots in shared workspaces with humans. The work builds upon state-of-the-art skin technology from TU München, world recognized path planning algorithms from CNRS-LAAS, and novel depth-perception algorithms of TU Delft.
2. Platform-independent harmonized robot software components for seamless integration with existing machinery and robots, including rapid self-calibration to operate in un-altered environments. The work builds on Fraunhofer IPA’s leading role in the ROS Industrial consortium aiming at such software systems.
3. Standard core hardware modules (e.g. adaptive grippers and arms) plus a procedure to use Additive Manufacturing (3D printing) for task specific parts, brought together in a novel high-speed hardware development-and-installation procedure. The work builds upon Materialises world leading position in additive manufacturing as well as Lacquey’s prominent gripping technology and Universal Robots highly popular safe and intuitive robot arms.
4. Fast teaching software for on-site robot “programming”, using domain-specific models and languages (e.g. optimized for mould finishing or for snap-on assembly) so that only essential parameters and trajectories need to be taught by humans. The work builds on KU Leuven’s novel iTask framework and leading open-source robotics control software.
Consortium
At the start of the project, there were State of the Art initiatives that were already targeting shorter installation periods for production lines in SMEs. For example, project partner Universal Robots already developed low cost robotic arms that can be installed in a matter of hours, that will also contribute to much faster production line installation. EU R&D project SMErobotics, claims to have developed a system that can be installed in 3 days; however this excludes the design and component customisation cycle. ‘Factory-in-a-day’ will build further on these developments to create the first complete hybrid robotic systems that allow for 1-day preparation and installation in a factory line, easy adaptation to changes in the production process and intuitive cooperation with humans.
The project was initiated with a consortium combining these existing initiatives with an extended set of industrial users and leading European universities and research centres on robotics. In addition to the research institutes and companies listed in the previous section (categorized per breakthrough), there were key contributions from Randstad, the world’s second-largest temp agency, Fraunhofer IPT, Philips, Siemens (who unfortunately left the project early), and several smaller companies.
Vision
At the start of the project, the following visual description was made to illustrate the possible timeline of an installation day. Although the research results in Deliverable 3.1 and 3.3 later showed that the installation ultimately would need to be separated in at least two days with some allowance for production time in between, the original illustration still accurately visualizes the core concept of the project.
Project Results:
The content of this section is adapted from all deliverables of the project. It is organized according to the original project structure, per Workpackage and subdivided per Deliverable. All of the technical innovations come together in the Demonstration workpackage (WP7) at the end of this section.
WP 2: New business models and certification procedures
The objective of Workpackage 2, according to the Description of Work, is to develop the business models and standards and certifications that are necessary for practical implementation of the factory-in-a-day robots. In effect, a next-generation type of robot service provider will have to emerge, a combination of a temporary work agency and a classical system integrator. This work package will provide the blueprint for such new companies.
We completed the work by producing the five Deliverables as detailed below. In addition, a spin-off company called Delft Robotics was created to demonstrate the new concepts described in the deliverables. During the run-time of Factory-in-a-Day, Delft Robotics became a partner in the project, contributing to the deliverables in this Workpackage. The other main contributors to this Workpackage were Randstad (WP lead), TU Delft, Fraunhofer IPA, and Philips.
The work started with a detailed task and market analysis, to determine which manual tasks are suitable for automation with the next generation industrial robots (Deliverable 2.1). Next, we performed first a quick scan and then a more extensive study into human factors involved in human-robot co-production (Deliverable 2.2). Simultaneously, we developed the blueprints for new businesses which can leverage the quick-install technologies of Factory-in-a-Day in Deliverable 2.3. The two final deliverables contain theoretical recommendations (Deliverable 2.4) and the practical implementation (Deliverable 2.5) of safety standards and certifications in the context of the quick-install robots developed within Factory-in-a-Day.
Deliverable 2.1: Detailed task and market analysis
Which manual tasks could be robotized first, and which tasks were still too hard for current robot technology? Moreover, which tasks were most economically viable for robotization? To answer these questions, we developed an online quick-scan tool. Business owners could complete a survey, resulting in a report about the robotization feasibility of their proposed task. The analysis of the technical complexity was based on several complexity charts such as the one depicted in Figure 2.
In a similar fashion, the business drivers (economic analysis) was performed to obtain a well-balanced overall indicator for robotization feasibility. In total, over 40 quick-scan reports were issued. The most promising cases, such as box-filling, machine tending, polishing, and yucca-stem planting have been taken as example cases for the entire Factory-in-a-Day project. The section on Workpackage 7 will contain the details of the example cases, but they will emerge throughout this report, forming the connection between the work in all of the workpackages.
Deliverable 2.2: Human Factors quick scan
This section contains not only a summary of the Human Factors quick scan that we have performed, but also a more extensive study into human-robot collaboration, which resulted in three scientific publications. The work started with the definition of four roles played by humans during the different stages of development of a Factory-in-a-Day robot system, shown in Figure 3. The focus of our human factors analysis was mostly on the role of the operator.
There is a large body of textbook knowledge on human factors with respect to the envisioned Factory in a Day system. This considers both physical and cognitive ergonomics. Deliverable 2.2 contains a comprehensive overview of the following topics, including a thorough literature list:
• Engineering physiology: the inclusion of physical ergonomics to assess physical workload and workplace design guidelines. This specifically applies to the operator and operator roles within the envisioned FiaD system.
• Situation awareness: “demons” and tools to improve awareness (feedback, affordances, feed forward). This requires additional attention while exploring the different responsibilities the operator might have during coproduction (Remote controller, Supervisor, Co-worker, or Teammate).
• The notions of assessing usability, which is an extension of the field of cognitive ergonomics including topics like learnability and satisfaction for the system operators.
This deliverable also contained a first overview of safety norms regarding operation and training robots and robotic systems. We briefly discussed the ISO norms 11161, 10218, 13482 and the ISO technical specification 15066. The standards and norms are discussed in more detail in the section on Deliverables 2.4 and 2.5 below.
The literature on actual human-robot co-production is rather sparse. To understand exactly when and how the human worker interacts with the collaborative robot in terms of physical interaction and information exchange, we developed experimental setups to analyze machine tending tasks and to test human-robot collaborative box packaging tasks in a realistic setting , see Figure 4.
The analyses resulted in the “Human-Machine-Product-System framework (HMPS)”. The proposed framework defines explicit and fixed number of actors in the system and forces the expression of the system using these actors: Human, Product, Machine, and System. Furthermore, the types of interaction between the actors are made explicit and are clearly divided between physical transfers and transfers of information. Using this framework, we performed a review of the current state-of-the-art in collaborative human-robot co-production . We concluded that there was surprisingly little actual interaction between robot and human; in most cases, the tasks were clearly divided between the human and the robot and not much interaction was reported, even though the robot and the human shared the same workspace and collaborated on the same tasks. For the remainder of the Factory-in-a-Day project (and beyond), this leads to the decision to focus not much on human-robot interaction during the operational phase, but to focus mostly on developing technologies to speed up the installation phase, while adhering to (but not developing further) the safety requirements for human-robot workspace sharing, see Deliverables 2.4 and 2.5.
Deliverable 2.3: High-level business concept and organizational structure scenarios
The goal of this deliverable was to analyse several potential business models for the quick-install robots created in Factory-in-a-Day. For the analysis, we used the Business Canvas tool as depicted in Figure 5. Three business models showed to have great potential. The first is a business model for a “normal” robot systems integrator, yet one that aims at maximizing the speed of installation. The blueprint for this business model was so promising that the spin-off company Delft Robotics (www.delftrobotics.com) was started. The second business model was called “Combined
Labour/Technology/Process QuickScan (LTPQuickScan)”. Our proposal is that a temp agency such as project partner Randstad collaborates with a technology provider such as Delft Robotics or project partner FactoryControl. Together, they provide the customer with an integral advice on which (sub)tasks to robotize for a new factory installation. This proposal has inspired the start of the Dutch company Smart Robotics, which collaborates with a local temp agency to provide exactly such a service.
The third business model deemed promising at first, was what we called a “Human-Robot-Team Labour Service Provider”. We envisioned a joint venture providing both the robot technology and the human labor. We extensively experimented with this model, trying to create a viable case for customer Bausch and Lomb. Eventually, the results showed that the state of the Factory-in-a-Day technology was not yet sufficiently advanced. The main conclusion, drawn by partner Randstad, is that such business models will only become feasible once the robot system development and installation phases together will take less than one week. Admittedly, the state of technology is not yet ready for that. Therefore, the focus of the Factory-in-a-Day project from this point onward has taken a more technology-oriented approach to maximise progress on the technologies required for quick installation.
Deliverable 2.4: Safety standards assessment and safety training inventory
Current safety regulations and safety standards relevant for robots have not been drafted with the highly modular and reconfigurable robots of Factory-in-a-Day in mind. As a consequence the necessary procedures to achieve compliance are time-consuming and require a large amount of paperwork. In addition alteration of a robot system usually requires almost the full process to be repeated. In order to enable setting up an automation system with the time scale intended in Factory in a Day, it is essential to speed up these procedures.
This deliverable identified approaches to reduce the workload to comply with regulations and especially to speed up the deployment process by certifying parts of the automation system and preparing technical documentation application-independent in advance. These strategies can be supported by the development of software tools that can generate parts of the technical documentation as well as user manuals and training material automatically based on the components of the automation system. Additional approaches are discussed to reduce the amount of training by designing machines in a way they can be intuitively operated or by the use of experienced personnel.
The deliverable furthermore gives an outlook on the possibility to change regulations in order to simplify the way modular safety components are safety certified. One was to establish connection to an ISO standardisation committee dealing with modularity for service robots. This recommendation was followed up in the next Deliverable.
Deliverable 2.5: Novel certification structure for both robot and human labor side
Following the recommendations from Deliverable 2.4 project partner Delft Robotics has implemented novel certification procedures. Within this deliverable, we reported the results, observations, and modifications to the recommendations from a practical point of view. Also following the recommendations, several partners have contributed to the ISO standardization committees dealing with the relevant safety standards. Although the standardization processes are very slow, such that individual contributions are not identifiable, we have provided important input from the Factory-in-a-Day perspective into the development of new or updated standards, as detailed in Deliverable 2.5. In Figure 6, we list all relevant standards for Factory-in-a-Day type of robot systems.
WP 3: System hardware in a day
The objective of Workpackage 3 was to develop the hardware components and the workflow to produce customer-specific hardware components for a complete production line. The first two sections below describe the workflow results, followed by sections on hardware developments, mostly revolving around Additive Manufacturing. The workpackage was led by Materialise with significant contributions from Lacquey, TU Delft, Fraunhofer IPT, Philips, and smaller contributions from the remaining project partners. Note that part of the work in WP3 was in service of other workpackages, delivering the 3D printed components as required.
Deliverable 3.1 + 3.3: Overall factory-in-a-day workflow diagram – including Description of Additive Manufacturing workflow
Deliverable 3.1 describes the robot system installation workflow that, according to our insights, should be adopted by a new type of systems integrator for proper and quick installation of robots in SMEs. The new type of systems integrator should be software-focused, and should develop and maintain a proprietary software architecture and a surprisingly small set of hardware components to select from. This should allow them to install a robot in two site visits, each of one day duration, with some development time between the two days. The main contribution in Deliverable 3.1 is the 4-step procedure depicted in Figure 7.
Note that the result of Deliverable 3.3 (description of additive manufacturing workflow) is part of the overall workflow. Its main result is a decision flow chart to identify if Additive Manufacturing should be used or not, which parts of the gripper should be 3D printed, which printing technologies/materials should be used, and which IP issues should be taken into account. Please consult Deliverable 3.3 for more details on additive manufacturing, and Deliverable 3.1 for detailed explanations of each blue underlined keyword in Figure 7. Although we have not completely succeeded in following the proposed installation workflow, it is in use now as a blueprint for the operations of project partner Delft Robotics.
Deliverable 3.2: Workflow Simulation Tool
One of the most time consuming aspects of new robot system installation, is the communication with the client. Oftentimes months go by, creating a proposed system design, learning of new implicit restrictions from the client, and redesigning the system once again. To alleviate this process, Deliverable 3.2 describes the development and validation of a Workflow Simulation Tool (WST). The tool was developed and tested in close collaboration with system integrators yet offers a fast and intuitive modelling solution to reason on automation scenarios. To enable this, it entails a portable tablet that runs a visual modelling environment, entitled Visual Components, combined with a handheld 3D scanning solution. Complementary to the online files, this report encompasses a description of the tool, installation and user manual, and rudimentary background to introduce underlying concepts of modelling workflows and the selection process of the contributing technologies. Because the tool itself considers commercial hardware and software, a bill of materials is included in Deliverable 3.2. The tool is recommended for all robot system integrators. The workflow with the tool is depicted in Figure 8.
Deliverables 3.4 + 3.5: Library of generic 3d printable components and Design template for 3D printable gripper fingers
Deliverables 3.4 and 3.5 are focused on 3D printed parts. We use the terms “3D printing” and “Additive Manufacturing” as equal alternatives throughout the text.
Deliverable 3.4 concludes that 3D printing is only economically interesting for the production of complete grippers or parts thereof, and not for most other system components. For moving parts within the gripper, the Laser Sintering method is the only one that results in sufficiently durable solutions. The usable materials are PA 12 and TPU. There is a food-safe PA12 available and a food-safe silicone coating. TPU is not food-safe but very flexible. The motion can only be made durable with “living hinges”, i.e. hinges depending on the flexible deformation of the material and not on sliding surfaces (wear through friction). For living hinges, we have found good design guidelines to obtain a proper tradeoff between durability, flexibility, and motion range.
In Deliverable 3.5 we report extensive studies into various types of 3D printed gripper fingers. The original design was to use bellows, which we have modeled and measured, see Figure 9. This type of gripper finger has been patented, however the studies show that it is intractable to accurately model the bending behavior of such fingers, which limits the scope of applications.
The work in Deliverable 3.5 continues with an overview on how to design grippers for 3D-printing. The main benefit of 3D printing is our concept of “printable functionality” for grippers. From hinges over valves, it can all be printed without having to make a compromise by having to select a specific, possibly suboptimal gripper. We list the gripper finger designs for the Philips cases, for various food grippers, and the (for safety) magnetically connected gripper for box filling applications. As a final example, the “Venturi Finger” was invented (patent pending ) showing the combination of clog-free vacuum gripping with mechanical gripping, all in a single finger, see Figure 10. Note that the original intention of this deliverable was to create standard gripper designs with modifiable parameters to address all types of applications. Our extensive experiments have taught us that such “god-type” design attempts are not successful, while a much more effective way is to create a library of design features to be combined quickly in a new design. The library contains for example the Venturi Finger, living hinges, standard flange designs, etc. This approach has been used successfully to create new gripper designs in less than an hour. A final noteworthy remark is that we created an extensive IP map, since there are so many patents already present in the field of gripping (with 3D printed components) that one could almost call it a “patent mine field”.
Deliverable 3.6: Evaluation on adaptive gripper client tests
Task 3.6 started with exploring the methods of 3D printing for adaptive grippers. Several cases were investigated, but turned out to have more simple and effective vacuum alternatives. The fresh food bin picking case required a 3D Ultra adaptive Gripper for 35% of the products. A gripper was developed within the FIAD scope (3D printed, modular, and scalable) and integrated in a test set up with user interface. Successful gripping was achieved and tested for 3 different products (Egg plant, apple, and pear). Speeds were 8 seconds/pick and can be further improved. This system could potentially replace 35% of the order pickers and reduce (human) errors.
Deliverable 3.7: Design template for mould polishing components
Mould polishing was one of the feasible tasks as concluded in Deliverable 2.1. Nevertheless, it is a specific task which requires domain-specific knowledge, so this task was addressed separately by partner Fraunhofer IPT in Deliverable 3.7. The first discovery that was made, together with partner Materialise, was that the material ALUMIDE was perfect to 3D print the polishing tools. The next step was to dismiss the current approach of complete manual polishing, create an automated integral system (Figure 12, top row), and through extensive testing (Figure 12, bottom) determine the minimum number of tools to still obtain properly polished surfaces for any random product design. The automatization of the polishing process not only brings many advantages as repeatability and homogeneity to a process that is used to be manual, but it also brings a few main disadvantages that includes the need of a preparation using CAD/CAM software. More than that, there is also the problem regarding the geometrical limitation of the tools and the machinery used at the process. Since the focus in FiaD is to reduce the setup time on factory environment, the aim for the creation of a tool template should be defined, so that the robotic polishing preparation time could be as short as possible. Along with a template definition, the standardization of the interfaces of the hardware is also mandatory.
WP 4: Dynamic obstacle avoidance
The goal of this work package was to create robot arms that are aware of all (dynamic) obstacles in their environment, and that respond by moving around these obstacles while still continuing their work. To create this awareness, not only the robot’s own sensors is to be used, but also information from existing machinery is to be used, e.g. a second camera (overview) and the position sensor of axis controlled by the production line. The various sensors must be combined into one consistent model of the work environment. Current robots were safe (sometimes), but use obstacle detection only to stop moving. Here we aimed to re-plan the motion trajectory and continue working. At the same time, human co-workers will be made aware of the planned motions, such that they themselves can predict and avoid the robot as well. Additionally, we aimed to create motion plans that fulfill various task specific constraints for typical industrial applications. The automatic consideration of these constraints will drastically simplify and speed-up the deployment of a robot as part of the factory-in-a-day concept.
We have worked toward this goal through various demo setups. This includes the demo setup “TOMM” described in more detail in the videos of WP5, a PR2 mobile manipulator at the site of partner CNRS-LAAS, our final box-picking demonstrator setup and our well-known winning system for the 2016 Amazon Picking Challenge, see Figure 13. One of the main challenges in this workpackage was to integrate all of the (mostly) software technologies: the path-planning component by partner SIEMENS-PLM, the proximity-sensing skin by partner TUM, the reactive motion planner “Stack-of-Tasks” by partner CNRS-LAAS, the framework ROS with the motion library “MoveIT!” by partner TUD, and the (mostly vision) sensor data integration framework (with Deep Learning) from partner Delft Robotics. Through intensive exchange and collaboration, we are proud to have successfully coupled all of these technologies and by doing so winning (ahead of MIT, amongst others) the world renowned Amazon Picking Challenge. Finally, this workpackage has also resulted in a large number of scientific publications as listed in the Dissemination chapter of this report.
Deliverable 4.1: Prototype Robotic Skin to Partners
The main objective was to deliver a prototype of the artificial skin (Figure 14) to the partners in an early stage of the project (month 6), before the final full robot skin becomes available (month 36). This way, other partners can evaluate the skin’s functionality, software integration and application in small scale before its final distribution. TUM therefore developed a hardware and software “Demo Kit” package. In extension to the raw deliverable, TUM hosted a hands-on workshop for all interested partners, in order to transfer the required knowhow to install, operate and apply the skin. The workshop has been combined with topics from the previous workshop on “learnable skills” of WP5, due to an overlap of interests and affiliations.
Deliverable 4.2: Website Video showing Path Planning with Proximity Sensing Data
This deliverable consisted of a first video that illustrates the ability of mobile manipulator robot executing a motion plan task with proximity sensing for reactivity, see Figure 15 (left). The robot is equipped with state of the art control task based reactive controller giving the flexibility to install in any robot platform reducing time for configurations. The proposed simple architecture exploits the hierarchical property of the controller to handle trajectory tracking tasks without compromising on safety and the final goal. The video (https://youtu.be/y-6Oyi21ioQ) shows how a robot autonomously plans a motion towards a target bottle on a table, and executes the plan robustly while evading contact measured with the (demo patch) of proximity-sensing skin.
Deliverable 4.3: Delivery of hardware and prototypes
This deliverable consisted of three components. The first was a follow-up on D4.1 i.e. the delivery of the completed robotic skin to the project partners, as shown in Figure 13 (right). This fault-tolerant low-cost robot skin by TUM, the CellulARSkin, allows to cover a complete robot manipulator, featuring around 300 cells with distance, temperature, force sensor and accelerometers. It has successfully been installed in two UR5 robot arms at TUM and TUD.
The second component was a prototype of our integrated Dynamic Obstacle Avoidance Framework developed by CNRS-LAAS in collaboration with Siemens PLM. A prototype has been released and validated with a simulation of the TOMM robotic setup from TUM, see Figure 15 (right). Finally the third component of this deliverable was our Industrial Sensor Integration Fusion Framework. A prototype of this framework has been developed by TUD using the ROS framework, and validated with the development of the Team Delft robot that won the Amazon Picking Challenge 2016 (Figure 13).
Deliverable 4.4: Website Video Contribution: Reactive Path Planning and Motion Control
The three components of the previous deliverable (4.3) are combined into an integrated demo system which is reported here. The presented deliverable video focuses on the dynamic obstacle avoidance which is an essential component to ensure safety in the robot environment and get the robots collaborate with fellow human beings thus improving the efficiency of the processes in the factory environment which is one of the goals of the Factory-in-a-Day project.
In terms of technology, Skin Sensors from TUM, Reactive Path Planner from SIEMENS-PLM, and Reactive Controller from LAAS are combined into a manipulation scenario to illustrate the dynamical obstacle avoidance capability. The simulation video is a proof-of-concept for the future deployment in industrial robot setups. The illustration is done on the TOMM setup with skin sensors on the right forearm of the robot.
The video (https://www.youtube.com/watch?v=uLStjR7mpOI) has two parts. First, it shows the reactive dynamic obstacle avoidance behavior using the simulated skin sensors with 'Stack of Tasks'(SOT), the reactive controller driving the robot. Secondly, it contains the manipulation scenario which shows the use of the Point Cloud Library - based planner and the reactive SOT controller to avoid obstacles. The techniques shown in the video are used in the final demonstrator which is shown at the end of the project, see WP7.
Finally, Deliverable 4.4 also consisted of a video related to a slightly different topic, namely the first exploration on how Augmented Reality can be used in production planning and during deployment & operation, as shown here: https://youtu.be/uOh7mG5tmnQ. This topic will be elaborated in the next deliverable.
Here, we also report the significant but incomplete contribution from former project partner Siemens AG. Although their newly acquired subsidiary SIEMENS-PLM remained an active partner, the main company Siemens AG decided to stop with Factory-in-a-Day due to severe internal reorganizations. An important contribution was their demonstration of vision sensor data integration into their robot control framework, allowing instantaneous re-positioning of objects by the robot operator. The video is shown here: https://www.youtube.com/watch?v=_6rn-dfz1Sc. Factory-in-a-Day partners have taken over the tasks of Siemens and continued the development of easy-to-use vision-guided robots.
Deliverable 4.5: Report on the effect of awareness augmentation
This deliverable of the Factory in a Day project is concerning the development and validation of the Augmented Awareness Toolkit of human co-workers, as part of work package ‘Dynamic Obstacle Avoidance’. The toolkit was developed and tested in collaboration with system integrators, offering multiple interaction modes for use of augmented reality technology in context of knowledge exchange and decision making in manufacturing environments.
To this end, it entails the use of both traditional 2D displays and head mounted display (HMD) devices to immerse and engage the users in the augmented environment, thus placing virtual simulated content within a real, physical context. The proposed solution features collaborative as well as remote assistance capabilities, presented in three distinct interaction modes. The user experience has been tested in user studies with relevant immersion and situational awareness assessment techniques.
Complementary to the online files, this report encompasses a description of the tool, installation and user manual, and rudimentary background to introduce underlying concepts of environment augmentation and the design rationale for the contributing technologies. Because the tool itself considers commercial hardware and software, a bill of materials is included.
WP 5: Learnable skills
The objective of this work package was briefly formulated as to make it easy and fast to teach robots how to execute a new task.
Deliverable 5.1: Learnable skill model
Deliverable 5.1 is an extensive document which lays the fundaments for the software developments in successive deliverables. It describes the fundamental models for the specification of robot tasks. These models allow us to describe learnable skills in a generic way. One of the first main results was the identification of various types of stakeholders, each with their own contribution to the development of a new robot system, see Figure 18. It shows that the system developer must use or create a Domain Specific Language (DSL) within which it is easy to describe a certain type of robot task. The Application developer can use that to program the robot behavior, which in turn can be used by the Deployer to actually install the robot and set the correct parameters such as pick/place locations. The operator finally can easily monitor the system status and perform basic operations.
The deliverable describes the models, the underlying mathematics, and a proof-of principle implementation in our newly defined task specification language called eTaSL (for expressiongraph-based Task Specification Language) .
Deliverable 5.2: Update (Month 48) Learnable skill model
The work preliminary reported in D5.1 has continued throughout the project and the updated result is reported in Deliverable D5.2. New concepts were added to the eTaSL language and implementation, such as instantaneously coinciding expressions and a way to integrate sensors into eTaSL and thus extend eTaSL beyond geometric constraints. With regards to the software architecture and implementation, more detail was given on the different Orocos components involved in an application and the ROS layer that provides the application. Deliverable D5.2 also described the different approaches by which eTaSL can be extended and listed a number extensions such as extensions for the TUM skin, extensions for point cloud distance computations, extensions for the computation of distances between convex objects, and extensions to provide spline and velocity profile functionality. The resulting overall architecture provides adaptability at different levels:
• A system developer can add new functionality to eTaSL/eTC (e.g. the skin extension, using a new solver, etc.)
• The system can be configured for new Robot hardware and sensors at the Orocos level by adding new hardware driver components and configuring new connections between the components. This is independent of the actual application running at the system.
• New applications can be configured at the level of the ROS-layer.
• The applications can be deployed to new sites using the techniques of programming by demonstration (cf. deliverable D5.4).
More elaborate documentation on eTaSL is given at https://people.mech.kuleuven.be/~eaertbel/etasl.
Deliverable 5.3: Website video contributions
Deliverable 5.3 was intended to show implementation progress for our work on Learnable Skills. Four videos were delivered showing various software components at work to make it easy to teach the robot how to move.
Deliverable 5.4: A model based task specification
This deliverable contains videos representing the work in Task 5.4 which was aimed at two approaches; a scientific approach continuing the work in the previous deliverables, and a direct application-oriented approach based on Factory-in-a-Day insights and existing systems of partner Universal Robots. Specifically, according to the Description of Work, UR developed “the general principles behind an configurable assembly skill in the context of their existing robot controller and GUI. Intuitive programming of this assembly skill is a challenge. It will be investigated whether partial application of already modelled constraints/forces are a solution for this.” We did investigate partial application of already modelled constraints/forces (i.e. the scientific approach) and decided that this was not already a solution. Therefore, the deliverable contains separate videos for the scientific approach (video 1-2) and for the application-oriented work (video 3-4).
Deliverable 5.5: Description of concrete learnable skills
The ultimate result of Workpackage 5 was the implementation of examples of concrete learnable skills. The journey started with the design of a new language and then new tools (see all previous deliverables of Workpackage 5), and these have been used together with tools from other workpackages (mainly WP4 and WP6) to implement two concrete examples.
Concrete learnable skill 1: pick-and-place application
The skill of “learning to pick and place objects while avoiding collisions” is actually a combination of low-level skills which all have been implemented with the eTaSL framework and which have been combined to provide the demonstration videos reported in Deliverable 5.4. Although the low-level skills are quite different as listed below, we are very proud that they have all been implemented in a uniform way. This makes the eTaSL framework highly extensible and maintainable. The concrete low-level skills that make up the pick-and-place application are:
Skill 1: Motion learning from demonstration (see kinesthetic teaching GUI in inset figure →)
Skill 2: Force admittance for interaction
Skill 3: Precise pick positioning
Skill 4: Precise velocity tracking
Skill 5: Limitation avoidance to stay within user-defined workspace
Skill 6: Collision avoidance for known obstacles (using shape primitives)
Skill 7: Collision avoidance for unknown obstacles (using point clouds)
Skill 8: Grasping affordances for different types of objects
Concrete learnable skill 2: polishing application
For polishing applications, highly detailed CAD-models of the workpieces are generally available. The work cell typically contains a tool-changer with storage, and a specifically engineered work cell environment of which also a detailed CAD-model is available. Therefore, kinesthetic teaching as reported for the pick-and-place skill does not lead to added value for this application. However, in this application, a lot of skill knowledge is present (e.g. in selecting tools, lubricants, etc. and the order in which you apply them for a given desired result). Therefore, another approach was followed in which this skill knowledge is encapsulated in a knowledge database/expert system and accessible in a graphical user interface where the user can select the desired finishing of indicated areas of the workpiece. The implementation of the overall skill once again consists of various low-level skills which have been implemented using Factory-in-a-Day results and common technologies (such as ROS, URDF-files, SMACH, Move-It! ) as were used in work package 4 and 6. A screenshot of the user-friendly GUI is shown here.
WP 6: Rapid installation software framework
The goal of this work package was to develop a complete framework for the rapid installation of a mobile manipulation system in new environments to cooperate with onsite workers. In order to focus on the acceleration of installation time the aim is to use and integrate existing software components as much as possible. The “Robot Operation System” (ROS) community that holds a huge collection of open available functionality and drivers will be the main source in this project. This workpackage has resulted in very strong results, captured partly in the new H2020 project “ROSIN: ROS-Industrial Quality-Assured Robot Software Components”.
Deliverable 6.1: Report on Quality metrics and evaluation infrastructure
A variety of service robots is available as standard platforms allowing a worldwide exchange of software for applications in the service sector or industrial environments. Open source software components enhance this sharing process, but require the maintenance of a certain level of quality. This deliverable presents an approach to an automated testing and evaluation platform which facilitates the sharing of capabilities over different robot types, environments and application cases.
Deliverable 6.2: Robot Deployment Toolbox Prototype
This deliverable presents a deployment toolbox which facilitates continuous deployment in a distributed development environment. Based on the architecture shown in Figure 20, we have created a deployment toolbox which allows quick installation and combination of several robot software components here depicted as “App repository”. The toolbox is available at the repositories of partner Fraunhofer IPA, and it has been validated with two versions of the Factory-in-a-Day demo setup (the Philips case). The particular software for that setup is hosted at (https://github.com/fiad-project/fiad_hackathon_philips and https://github.com/fiad-project/bandl_robothon).
Deliverable 6.3: Final Robot Deployment Toolbox And Working Robot Deployment Process
This deliverable presents the final version of the deployment toolbox which facilitates continuous deployment in a distributed development environment. With the vast amount of packages available in the open-source ROS community, it is hard for a user to select the appropriate or best one for his application. There are various approaches for solving this selection problem: Test-Driven-Development, benchmarking and continuous integration are widely adopted approaches in general software development processes, whereas in the domain of robotics, especially for distributed open source development environments like ROS, Test-Driven-Development is not widely used today.
The deployment toolbox presented in this deliverable addresses this situation by also allowing testing at integration and system levels.The test framework is available as running implementation at Fraunhofer IPA. It is running in multiple test instances at internal Fraunhofer servers which can be made available for project partners upon request. The main result is the set of deployment environment scrips available at https://github.com/ipa-mdl/ipde_utils accompanied by three scientific contributions.
WP 7: Demonstration/validation
The objective of this work package was to validate the robotic systems and the business implementation procedures. This work package formed the central point of coordination of all of the tests and demonstrations that have taken place throughout the project, and as such it also served to connect all of the technical innovations.
Deliverable 7.1: Task list, evaluation metrics and test protocols
The first task within Workpackage 7 was to create a framework for evaluating cases and progress in the Factory-in-a-Day project. Since Factory-in-a-Day was the first project to focus on the installation process, we had to develop a framework of measures how to analyse such installation processes. The framework is comprehensively documented in Deliverable 7.1. It consists of an extensive task list of tasks and cases relevant for the project, a list of metrics to ‘score’ tasks and measure the progress of the project, and the test protocols outlining in general terms how to measure the metrics. Furthermore, we subdivide the metrics in Key Performance Indicators (KPI’s), capturing the goal of the project, and PI’s, capturing the constraints of the project. The most important KPI’s for Factory-in-a-Day are listed in Figure 21 below. Throughout the project, we have reverted back to these KPI’s.
Deliverable 7.2-7.4: Demonstration reports
To obtain an overview, we decided to combine the three Demonstration reports (Deliverable 7.2-7.4) because the each build on top of each other. The demonstration cases are shown chronologically in Figure 22. The demonstrators can be categorized in a few “dependency lines”.
First, we evolved from a box-filling case (using fixed, pre-programmed positions) to a system which completely autonomously detects objects and determines how to move, ultimately leading to industrial quality results in the Philips 3 case and leading to our team winning the Amazon Picking Challenge. In the latter case, the robot was able to recognize various types of objects on cluttered shelves, and could pick and sort them as in an Amazon distribution centre, scoring better than top international teams from MIT, Mitsubishi, Bonn, etcetera.
Second, a more academic research setup was completed, successfully showing the combination of kinaesthetic teaching (show the robot what to do) with vision and motion planning. The proximity-sensing skin of partner TU Munich was an important component in that demonstrator, which was transferred also into the final box filling demonstration. Third, we have worked on a separate setup for the automation of mould polishing tasks, according to the Description of Work. Although the setup differs mechanically from the other setups (larger machine, more rigid placement of mould and tools), we have re-used ROS-based software and we have equally analysed the use of 3D printing for quick creation and setup of tools. Finally, separate demonstrations have been delivered by Siemens (quick-install vision-based pick-and-place actions), by CNRS-LAAS (mobile manipulator demonstrating reactive motion planning behaviour), and by Universal Robots (the now commercially available UR Caps platform, resulting from Factory-in-a-Day demo “UR+”). Videos of these demonstrators are online here: http://www.factory-in-a-day.eu/media/videos/.
An important remark is that the complexity of the robotic task was increased for each subsequent demonstrator, which had implications for the installation time. Especially for the “Philips 3” case in year 3, the installation time and effort was exceptionally high due to the fact that we wanted to break away from academic demonstrations and create industrial-quality robust performance. The four graphs in Figure 23 clearly show that the investment paid off; from the “Philips 3” case onward, we achieved 100% of the industrial robustness specs (i.e. uptime according to requirements, usually 95%).
The Amazon Picking Challenge demonstrator was the most visible result of Workpackage 7. The challenge 2016 included two parts: in the picking challenge a set of products from the Amazon product range needs to be picked from the shelf and placed in a tote. For the stowing challenge, it was the other way around: products are to be picked from a tote and stowed into the shelf. The robot system consisted of a 7 DoF manipulator, ROS software with MoveIT as a core component for motion planning and Deep Learning as a core algorithm for object recognition. The system completed the series of 12 item picks within 6.5 minutes, less than half of the allotted time thereby securing victory for the team.
Potential Impact:
The content of this section is largely based on Deliverable 8.9 and potentially overlaps with information in the tables that will (should) automatically be generated by the SESAM / Participant Portal interface for Section 4.2 of this report.
In this section, we give an overview of the main results of the Factory-in-a-day project and a draft idea on how consortium partners intend to use these project results in on-going and future activities, in their further R&D and product marketing strategies. We list in detail the exploitation activities implemented over the project’s four-year period. The results provide manifold possibilities for commercial exploitation, as well as take-up in scientific projects. Therefore, this deliverable overviews all the partners’ exploitation activities including: MSc/PhD thesis, teaching/training services, follow-up projects, open source software, product/service development, spinoff/start-up companies and IPR actions (e.g. patents).
To sum up, these are the highlights in terms of exploitation:
- 1 spin-off company directly from the project: Delft Robotics
- 2 patents (Materialise: Printable Venturi Valve & robot-Fingertip/ TUM: Interactive robot surface)
- 12 projects (different funding schemes) that build on results of the project
- Software components: several open-source ROS-control frameworks, also used in ROS-industrial, MoveIT, Kineo Path Planner ROS-component
- Products:
• developed gripper placement algorithms in latest product CoreTakr
• URCaps / UR+ has been a great success for UR and some third party hardware vendors. A number of registered active developers are working on new extensions for the UR robot platform.
Other partners are still in discussion about licensing or building spin-off companies (TUM/TU Delft), results might only be expected to be finalized after the end of the project.
Deliverable 8.8: Survey amongst SME’s
Before listing all of the Factory-in-a-Day dissemination and exploitation activities, it is useful to provide insight in the motivation of SMEs for using (not) robotic technology Therefore, as reported in Deliverable 8.8 we carried out a survey entitled “Market Analysis of Plug and Work Robots for SME’s”
In the deliverable, we describe the current market situation of industrial robots in Europe. Thereby, the definition of SME’s, the market volume and its players are introduced. This section aims to give a rough overview about the development and market sizes in the industrial robot industry.
The number of sold robots in Germany and Europe is expected to increase. Especially within the automotive industry, where industrial robots are used the most, medium-sized and large Small and medium-sized enterprises (SME’s) have increased their amount for new investments. Considering that personnel is one of the biggest cost blocks for those companies and traditional robots are quite expensive, new concepts such as ‘Plug and Work’ robots aim to reduce costs and help SME’s to ensure competitiveness.
However, to tailor Plug and Work robots for SME’s it is necessary to understand their needs and concerns. Therefore, an online questionnaire study in the three biggest SME industries (i.e. automotive, metal, and rubber and plastics) in Germany was conducted. The goal of the questionnaire was to figure out how SME’s perceive their current situation and how they assess Plug and Work robots.
The results confirm our notion that SME’s perceive the ongoing price competition as threat for their business. When being asked about the reasons why they are reluctant to implement robots, costs and maintenance were mentioned as barriers. In the same vein, they expect that robots are safe, flexible, and easy to program.
All of the mentioned points constitute an advantage in Plug and Work robots compared to traditional robots. Derived from our survey we found out that when promoting Plug and Work robots for SME’s it is necessary to create offers which include trainings for SME’s how to use them. As a medium to advertise, especially classical media such as tech magazines or fairs should be used.
Deliverable 8.9: Exploitation activities
The project’s results cover a range of different exploitable assets. During the time of writing the proposal, “at least two spin-off companies are expected to arise from Factory-in-a-Day. For both companies, preliminary business plans have already been investigated”, the current state of the spin-off aspect is the following:
- TUM is in ongoing discussions about licensing the skin-sensor technology and also discussing several options for the creation of a spin-off company. At the point of writing this deliverable, no decisions have been taken.
- The idea to create a company at TU Delft for renting out robots has not been followed, instead another spin-off company, Delft Robotics, has been founded in Delft (see Table 1 below)
- TU Delft is also looking into a another spin-off company with respect to the Workflow Simulation tool (see Table 1 below)
Two patents have been filed from different partners (Materialise/TUM, more details below).
Inventions and ideas from the projects can further be found in several products, for instance a machine for lettuce-handling and in the online platform UR+, an online showroom, providing you with cutting-edge products to customize a UR robot.
The industrial partners, Materialise, Philips, PAL Robotics and Siemens also used results from Factory-in-a-day for adjusting internal production processes or updating products.
In the last years, several projects have been created during the runtime of Factory-in-a-day, so that the knowledge created in our project will be further increased and also spread in different directions on a European as well as a national level. Factory-in-a-day partners participated or initiated several H2020-EU-funded projects: ROS-IN, Scalable, Robmosys, Co4Robots as well as several projects on a national level (Belgium/The Netherlands). Demonstrations and publications from these projects also contribute to a further enhancement of knowledge originally gained from Factory-in-a-day on an international level.
With respect to software developments - there are a number of open source repositories available, where the developments of Factory-in-a-day are accessible. The work of Factory-in-a-day has always been closely linked to ROS (Robot Operating System). Several partners contribute to the ROS community with different code, see the following links to the software repositories:
https://github.com/ipa-mdl/ipde_utils
https://github.com/ipa-fmw/atf
https://github.com/git-afsantos/haros
http://moveit.ros.org/
http://wiki.ros.org/ros_control
Furthermore, Siemens will update the latest Kineo Path Planning software with a ROS-component. This includes a major update on the Kineo Collision Detector (KCD) for faster and more configurable collision checks. PAL Robotics now uses Automated Test Framework and Docker as internal tools.
Below we provide two overviews of exploitable results, first as listed per workpackage (Table 1) and then listed per partner (Table 2).
List of Websites:
http://www.factory-in-a-day.eu/
