Final Report Summary - MARWIN (Decision making and augmented reality support for automatic welding installations)
Executive Summary:
Welding tasks have been estimated to account for nearly a fourth of all industrial robots. However, the benefits decrease for the production of small batches of different products in small and medium-sized enterprises (SMEs). This is related directly to robot programming issues. Very often programming robots is time-consuming and requires qualified resources that SMEs often lack. Moreover, the latter companies often produce small batches of components; hence, setting-up and programming robots are perceived as an extremely costly activity.
The EU-funded project 'Decision making and augmented reality support for automatic welding installations' (MARWIN) developed a flexible, programmable system to optimise the robotic welding process. It enables the production of high-quality differentiated parts with increased productivity and reduced costs for the SMEs. The welding robot uses computer-based vision with an automatic path calculation and quality control system for enhanced production, flexibility and efficiency.
The project started with a survey sent to potential end-users, aiming to understand and identify real industrial needs. The MARWIN system was defined accordingly, including identification of suitable hardware and definition of communication specifications for the vision system, robot and personal computer controller.
A database of welding parameters, including welding torch positions, has been created. User information on the component to be produced is uploaded to the Marwin program via a graphical user interface (GUI). A weld parameters database enables the recommendation of suitable welding procedures. The operator and future end-users will also be able to upload their own welding procedures or modify existing ones.
MARWIN uses a vision system to compare the geometry and fit-up of the weld assembly to CAD assemblies uploaded earlier by the user and automatically adjusts welding parameters and weld robot trajectory. The vision system is mounted on the robot forearm and a tool changer allows it to be picked up as and when needed. A cognitive module enables communication between the robot and the vision system.
The MARWIN system is expected to significantly reduce the overall production costs of SMEs involved in production of industrial components and make robotic systems more accessible, while enhancing their potential product range, and boosting their competitiveness.
Project Context and Objectives:
Europe is a major player in the global machine tool and automation markets, providing some 52% of the world-wide value of machine tool production in 2002 (Japan 20%, China 9.6% and USA 6.1%) and is at the cutting edge of research and technological developments in that area . The past few years have witnessed a vigorous upsurge in the use of industrial robots for welded parts manufacturing. Notable reasons for the sharp increase in flexible automation with industrial robots are the shorter product cycles, shrinking batch sizes and a steadily rising level of wage costs.
Research and innovation in machine tools, robots and automation provides a key building block for economic performance, as it provides the intelligent production capabilities to make EU manufacturing sectors more competitive.
The use of industrial robotics in new application fields and industrial branches is considered a cornerstone in the development of many industries today. Giving impulses to the creation of new SMEs as providers of manufacturing equipment and services and the integration and support of small and medium enterprises in future manufacturing supply chains is a priority of current EC policies. Aware of the business opportunity and growing market in the welded parts industry, SMEs in the current project, with the help of RTD performers, intend to capitalise on this need by developing a product that will overcome the limitations of available systems for SME end users, thereby fulfilling a real market need.
Project objectives can be divided into three categories:
Scientific objectives
• Use cognitive vision (3D image reconstruction and registration) to calculate any deviations between a real assembly and a theoretical CAD file, and use this to automatically adjust welding and robotic parameters.
• Build a database of welding parameters and welding positions, to allow the Marwin system to choose the best parameter to obtain high quality welds (WP3).
Technological objectives
• Create an exact software model of the system and to define a communication scheme between the software modules as well as between the software and hardware modules, including classes, interfaces, data access restrictions, dependencies and database models (WP2).
• Create a vision system and 3D object reconstruction and registration software able to reconstruct the object with the accuracy of a minimum of 500microns as well as to find surface discrepancies on the work piece (WP4).
• Develop a software module that helps in the implementation of the technological instructions for welding by cognitive communication and is able to recognise and calculate the welding trajectory and welding parameters with minimal human intervention by using the knowledgebase (WP5).
• Create an intelligent learning environment based on Human-Computer Interaction (HCI) combined with Augmented Reality (AR) in order to visualise the welding parameters, trajectory and differences between the reconstructed and the designed model in a cutting-edge but intuitive graphical user environment aiding and speeding up the work of an operator (Task 6.1).
• Develop an easy to expand software framework that defines and implements the communication functions and protocols via Ethernet and RS232 port for industrial robot controllers in order to ensure the possibility to expand the communication skills of the product for industrial robot controller from different manufacturers. (Task 6.2)
• Expand existing welding robot systems (used as well as new) in order to deliver a flexible pre-competitive prototype by integrating all the software and hardware modules together and to transfer the methods to welding cell manufacturers of the consortium to be able to construct the system below the cost of € 30,000 (WP7).
• Test and validate the system at SMEs in laboratory and field tests by taking into account the object reconstruction, automatic program generation time, processing speed, consistency of weldment quality, joint location against manual welders by using ISO standard welding investigation techniques, first on simpler work-pieces and as a second stage on complex pieces(WP8).
Training, dissemination and exploitation objectives:
• Comprehensive training will be designed, prepared, tested, and delivered via well thought out training material, in order to ensure that the SME participant members will be able to assimilate the results of the project. Technology transfer will take place tri-monthly (Task 9.1).
• Non-confidential information about MARWIN project and its results will be disseminated to a wide and relevant audience to extend the impact of the project results (Task 9.2).
• A database of the target audience will be created for dissemination purposes. Seminars, workshops, conferences, and trade fairs will be identified / organized (at least 3 are planned); furthermore the public results will be disseminated via websites, mailings, articles, and papers in technical journals
(1000 SMEs to be addressed) (Task 9.2)
• Taking-up of results by the SMEs will be facilitated by training, dissemination, knowledge management, and IPR protection. Additionally, a business plan will be defined to increase the competitiveness
Project Results:
Marwin concept
The main idea of Marwin is to combine the human knowledge and problem handling skills with the accuracy and speed of robots, resulting in a flexible welding system with the ability to make decisions by a human welding expert and to create constantly high quality joints between two metal parts. The idea is originated from the fact of the cognitive psychology, which states that the information, which is enough to recognise the shape of an object, can be divided into two major parts, where 10% is given by our sensory organs (such as eyes) and the remaining is expanded by the knowledge, stored in the brain, which is gained from preliminary cognition. In MARWIN the eyes will be replaced by a vision system, which aims to reconstruct the visible part of the surface of the desired work piece, while the stored data will be replaced by a CAD model. This solution predicts a hybrid cognitive system, which will exploit the advantages of both human worker and robot.
MARWIN Software
The main result of the project is the MARWIN Control software itself, an offline robot programming solution with intelligent and flexible adjustment tools aiming to reduce welding robot programming time and to create part specific programs taking into consideration the real parts’ deviations from the CAD models. The main result can be broken down into several sub-modules as the following: Cognitive module, Simulation tools, Programming module, Communication module, Graphical User Interface and last but not least the Vision module.
Cognitive module and welding parameters database
The cognitive module is responsible for the calculation of the welding parameters for each joint in the assembly to be welded. For this purpose a special knowledgebase (i.e. welding parameters database) containing all essential welding parameters, from welding process to welding torch angle was created. These parameters included materials, plate thicknesses, gaps, torch angels, wire feed rate as well as the power source settings such as the welding speed, voltage and current. The database is prefilled with sample welding configurations and parameters selected and validated by TWI. As not every joint and welding process could be recorded in the duration of the project, the welding process, material and joint configuration were selected from feedback received on the survey at the beginning of the project.
The database was originally created as a remote database, so it could be reached from anywhere but for the final product, a local database was created as a mirror of the remote system.The database design was produced with MySQL WorkBench (WB). Once the tables, their relations and structures were identified, source codes could be generated. Although MySQL WB is capable to generate code automatically from existing designs, it was necessary to fine tune the generated code and restructure it. Due to the strict rules and constraints in the database structure some table must be created before a connecting table should be defined. Unfortunately MySQL WB does not take this into account and generates the code for the tables in ABC order.
The role of the database is to enable the user to find to optimal set of welding parameters as possible for the joints to be welded in his assembly. In order to select the best parameter combination from the database, a specific module is used that was implemented as a part of the project. A Neural Network based Decision Support System (DSS) was designed and implemented to aid the operators in decision making while selecting the most appropriate welding parameters. This DSS is considered as the one of the ‘brains’ of the MARWIN application. The system is a well know and widely applied Fuzzy Neural Network (FNN) so called Adaptive neuro-fuzzy inference system that is a neural network based on the Fuzzy logic. Since the system is able to handle partial truth or uncertainty and the advantages of neural network the benefits of both system is combined in one single framework.
Just like all neural networks the ANFIS has layers and nodes defined in the system. It is called adaptive due to the fact that each layer and node can have one or more parameters. The most important part of the system is the five layers in between with the following rules:
Layer 1: Contains parameterised or adaptive input nodes, which describe the complete domain of the input side of the system. The input domain is represented by membership functions which describes continuous intervals and / or set of discrete values. Parameters in this layer are referred to as premise parameters
Layer 2: Every node in this layer is fixed, and this is where the first T-norm (triangular norm) generalises intersection in a lattice and conjunction. More clearly this node used to ‘AND’ the membership grades.
Layer 3: The elements of this layer are fixed nodes which calculate the ratio of the firing strengths of the rules. This means that all the membership function of each fuzzy clause is being calculated and combined here. The outputs of the elements in this layer are called as normalised firing strength.
Layer 4: This layer holds adaptive nodes and performs consequent of the rules, or more clearly it adds the ‘THEN’ term to the expression. The parameters in Layer 4 are referred to as consequent parameters.
Layer 5: Layer 5 is a single-node layer and stands for to compute the overall output of the system.. For the proposed solution of the problem Type 2 had been chosen as it is capable to select a specific WPS based on the specified rules.
This means that the system is a multilayer system with defined nodes in each layer and well defined connection between the nodes. As a result of this structure a five-layer system was developed. The rules in this type of fuzzy neural network are the following: IF x is A1 and y is B1 THEN f1=p1x+q1y+r1; AND IF x is A2 and y is B2 THEN f2=p2x+q2y+r2.
From the programming point of view the system is based on the same parameters as the database. This means that the system is capable of handling problems up to a 24 dimensional space, meaning that each problem can be described by 24 different parameters. Each of the parameters belongs to a range so called Domain. The domain describes the possible values of parameters by using discrete or continuous values. These two values, DiscreteValue and ContinuousValue are the most basic building block of the system. Based on the DiscreteValues the system builds a so called WPS that is the definition of the problem and contain 24 values, each representing on parameter. Based on the knowledgebase the system creates a so called TrainingSet that is the collection of the known problem and solutions combination. A training set is also built up by WPSs but in this case the parameters can be described by Discrete and Continuous values as well. Based on the type the results of the parameter comparison the system return a value in the range of 0 and 1. The AnfisCore is responsible for handling the knowledgebase and for the calculation of responses.
Simulation environment
The simulation environment is the most important part of the system as this acts as the offline programming module of the system. This module is responsible for the simulation of the robot movement and to perform the calculations behind it. This includes the Inverse Kinematics (IK) as well as the Forward Kinematics (FK). In order to be able to calculate the TCP points that can be used to calculate the IK parameters a trajectory must be planned that includes the points on which the robot has to move through.
The trajectory planning is a fairly simple user interaction based method based on the popular A* path finding algorithm. The latter operates following a two-step behaviour. In order to represent a trajectory different waypoints and segments are defined. Waypoints are used to determine the location and orientation of the Tool Centre Point (TCP) of the robot. This means that the system can specify the alignment of the welding torch of the robot with respect to the work piece on specific points. Segments are used to group waypoints together according the type of movement. A movement can be a general positions movement, which is used when the robot moves at high speed big distances fairly far away from obstacles and work pieces. Operation movement is use during welding which has a certain speed taken from the knowledgebase according to the calculated welding parameters. There is also a low-speed trajectory movement that is used when the TCP and the robot moves close objects.
Two waypoints are always present in the environment: the StartPoint and the EndPoint. These points represent the beginning and the end of a movement or robotic operation. In many cases, these points are the same, but for specific purposes they can be different. This means that if the robot finishes the movement in an EndPoint different from the StartPoint the next program will start with a movement towards the StartPoint.
The trajectory is calculated in a simple way for the first time. The trajectory starts from the StartPoint and goes through all the WayPoints in the list in the order specified by the user. As a first step the system creates linear paths between the points and evaluates it. The evaluation criteria is simple, if any of the linear paths intersects with any object the path needs fine tuning, meaning that the system applies A* algorithm. This can be overridden if the user modifies the waypoint in way where none of the linear path intersects objects.
If users decide to apply the A*path finding technique, the system automatically creates a search space. This search space can be considered as a 3D grid. This is a necessary approach, as the A-star is able to operate only with discrete values. The size of the grids greatly affects the performance of the algorithm however the system give adequate responses. As first step the grid was created and the cells in the grid are marked as forbidden in which an obstacle is found. Following this the start point and target point is defined meaning. The algorithm FindPath then receives the space (including the forbidden cells), the start and target points and results a set of steps that is the shortest path between the start and target point.
Once the Path is defined, the system simulates the robot movement required to follow the latter trajectory. This requires solving of an IK problem for every point in the path. During the project, several IK solver solutions were tested including numerical and analytical solution, starting from the own developed CCD algorithm to 3rd party open source IK projects.
The current system applies the IkFast module of the OpenRave open source project. This library makes possible the use of IK library for any kind of industrial robots, the IK solvers are generated based on the OpenRave XML format that describes the dynamic and kinematic behaviour of the mechanical unit. This means that the IK solver is not universal, but needs to be generated for every mechanical configuration. Due to this reason a specific robot file format was created that encapsulates the IK solver’s DLL files, the robot model and the robot parameters as well. The file is a simple ZIP file with an altered file extension. As the IkFast module is implemented in C++ and have only Python interfaces a wrapper is implemented that makes the use of the library possible in the main .NET project. The wrapper is written in managed C++ that can be accessed from any .NET environments. In the implementation only two classes and a bunch of functions were wrapped, such as the IkSolution, IkSolutionList classes, or ComputeFk and ComputeIk fucntions.
For the simulation a background worker thread was developed that can be controlled over the main GUI just like a DVD player. It has simple playback, stop and pause functions and allows the user to perform step-by-step operation. The background worker was implemented in order to make sure that the calculations of the IK solutions will not block the application.
Communication module
The communication module used in the application is .NET; implementation of standard socket communication using UDP and TCP sockets for communication, while for file transfer standard FTP protocol is used. For communication purposes, the robot controller is also configured and a server program, so called User Task, was implemented in the SLIM language.
The TCP socket is used to communicate with the robot controller and exchange information between the robot and control PC. The module is based on the System.Net.Sockets.Socket class and builds a module around it. Once the object of the Socket is created by the system, a listener is created on a separate thread that is responsible for handling the incoming messages. There are four events wired for the class that are fired on Connect, Disconnect, Message Sent and Message Received events. The system is capable of sending raw byte streams as well as string messages and also able to receive the same format from the controller. The UDP socket is implemented on the same way however it does not maintain continuous connection with the other end point, therefore connect and disconnect events are removed. As the system does not check if the message that was sent has arrived, the message sent event handle was removed as well.
The communication flow is fairly simple. The PC application sends a command to the robot controller. On succesfull transmission the MessageSent event is fired. The robot controller then processes the message, and, according to the content of the command creates a response. If the command is an information request like STATUS or JOINT, the system returns the command with the requested information. If the system does not expect any data from the controller the controller responds with OK or ERROR based on the execution success of the command.
In order to be able to easily process messages a SocketMessage class was defined that can be created by using a string or a byte stream as input. The basic message contains only commands, and the extended messages contain data as well. For instance the message “Hello” only contains the message code while the Joint message has six parameters describing the angles of each mechanical joint. The following messages are interpreted: PING, CLOSE, JOINT, TCP, SET, RESET, SETSTOP, CLEARSTOP, STATUS, HELLO, BYEBYE, OK, ERROR, INVALID.
Socket messages can be sent also on the TCP as well as on the UDP socket.
Ftp connection is used to transmit programs between the Control computer and the robot controller as it is already available on every robot controllers. This makes the system available to transmit any of the programs to almost any kind of robot controllers. The module is built around the FtpWebRequest class implemented in the .NET framework. The FtpConnection class is able to connect to any FTP server if the username and password are specified and can switch directories. It is able to list the content of the actual directory and can upload and download files.
Programming module
The main goal of the programming module was to create a platform able to create robot programs for various robot platforms on multiple robot languages. For this purpose a middleware language is used that is a platform independent description using different objects and a configurable post-processor is applied to create the platform specific language.
The general robot program (MARWIN Generap Program – MGP) is built up from different modules. A module represents a set of instructions that can have multiple parameters according to instruction definition. The three main class of the programming are the GeneralCommand, RobotProgramModule and the RobotProgram.
The General command is built of a command type parameter and the respective parameters stored in dictionary that help to avoid duplications in the parameters list and enables the search between the parameters based on their types. During the creation of a command the type of the command must be specified and the constructor of the object creates a command prefilled with the available parameters. It is important to mention that parameters cannot be removed from the list and cannot be added however the existing values can be overridden by the user.
The RobotProgramModule is a collection of the commands introduced above. The main reason of applying modules is to be able to reuse parts of the program. The object holds a list of specified instructions that can be expanded and reduced as well. The most important feature is the save and load feature of the class that makes it possible to store the program including the commands and their parameters in an XML structure and to load them back. For this purpose SaveXml and LoadFromXml functions were developed that implement simple XML parsing functions. The result of the saving is a standard XML file that can be read by simple text editors with the extension of MPM (Marwin Program Module).
The robot program itself only contains the name and the file location of the program modules. The RobotProgram class is again also a list that contains the RobotProgramModules. It also have a load and a save function that again implements the XML parsing functions same as used in the RobotProgramModule class.
The term “Post Processing” is used in manufacturing and machining, and is part of the machining process. During post processing, a general language movement description is translated to a controller specific script, in which the commands, variables and parameters are in concert with the given machine’s characteristics.
Regarding post processing, a machine description which acts as a template, and describes the desired command and parameter mappings is created. The post processor then takes the general program and the machine description as input, and creates the machine specific program as output. The MARWIN Post Processor application is developed to streamline and automate this process. Its main purposes are the following:
• To provide an interface with which a user can configure a conversion template between MARWIN’s general program file structure and the desired machine’s specific program file structure
• To provide a convenient way of converting general language files to specific language files. Both machine description and mapping are stored in the standard XML format and can be edited by using the developed configuration tool.
Graphical User Interface
The Graphical User Interface (GUI) is the main contact between the user and the system. It was designed and developed in parallel with the backend modules taking into consideration the user requirements.
The main window can be broken down into different sections:
1. Ribbon and application menu – The ribbon is a standard Microsoft Office like ribbon that contains different menu points grouped by functions. Preferences can be access in the Application menu in the top left corner of the application.
2. Simulation view – The simulation view is responsible for displaying the content of the virtual scene created by the user and to visualise the different components. The scene can be zoomed, paned and rotated as well just like in any other CAD programs. Selection of objects can be made by pressing Shift button while clicking the mouse over the desired object.
3. Entity list – The list displays all the objects loaded into the simulation view, such as models, robots, work pieces and clamps. Object selection can be made through the list by selecting the desired entity from the list by clicking on its name.
4. Object properties – Anything selected in the simulation view or in the entity list has properties that can be accessed, monitored and modified over this interface. For instance the name, position and rotation of each object can be accessed; however, the robot has joint angles that can be modified.
5. Segments and waypoints – This area displays the points and paths where the robot moves its tooltip. Selection of waypoints can be made through this interface by selecting the names of the desired elements.
6. Logging and calculation results – Logging can be broken down into three parts: informal, warning and error messages. Informal messages are included for debugging the application behaviour and can be used to reveal software errors. Warning messages are displayed when a sub-module in the application return with an error but the operation of the software is not affected and this is highlighted in a message box too. Error messages are displayed when a critical error is occurred during the operation of the software. This means that the software needs to be restarted
For the development of the GUI, two 3rd party libraries were used. For the 3D visualisation and CAD handling, Devdept Eyeshot 6.0 was used while for the window and float component handling, DevExpress Winforms 12.1 was selected.
Vision system
As specified in the DoW, the vision system’s aim is to scan the component to be welded, in order to check if its local geometry and weld joint fit-up match these of the theoretical component.
The First vision system Hardware and software prototype was developed ahead of schedule, at M8. Since then, most of the work was spent on software implementation and on testing the light source and on bettering the quality of the acquired images as follow on to the deliverables D4.2 Algorithms for 3D Reconstruction for the Characteristics of the Camera and Projector, and D4.3 Methods to Normalize 3D Pose and Registration with CAD Models. Furthermore, we completed the deliverable D4.4 Benchmark 3D Reconstruction, Registration, and Error Analysis.
The outcomes of the work were that a large set of functions to manipulate 3D data were implemented and tested. Examples include functions to estimate local and global error analysis of registration, to automatically select visible surfaces from the 3D viewport, to automatically detect the reference stripe on images, to interactively define and project data points (e.g. welding control points) on the surface of the model, to automatically perform transformations (rotation and translation) according to Euler theorem to define scanned surfaces in relation to the robot base, methods to automatically save sequence of images using the robot, methods to automatically reconstruct models described in a prescribed XML format, methods to change mesh properties and visualisation, and a number of utility functions to support these main methods. Due to delay observed on work packages 5 and 6 affecting the integration phase, it is possible that middleware or utility functions are still required to make the system work seamlessly.
Regarding testing the light source and the quality of the acquired images: the vision system prototype was completed by month 8, concerns were observed on the brightness of light source, reflections on metallic surfaces and general levels of noise. The conclusion of sensitive studies we performed was that the laser light source (Microvision Pico Projector) should be replaced by white light (Aiptek Pocket Cinema Projector). The main reasons were related to the generated levels of noise. The laser Pico Projector generated high levels of “salt and pepper” noise and these are undesirable to the stripe detection algorithms. The white light projector emitted more lumens than the laser (50 lumens against 30) while, at the same time, generated less salt and pepper noise. Another aspect that was tested was the robustness of automatic detection of the reference stripe. In the earlier stages of the MARWIN project we manually clicked with the mouse on the reference stripe to guarantee correct 3D reconstruction. Initially a statistical method was tested to automatically find the reference stripe but it was not robust enough. In the end, the adopted solution was to generate a pattern of light using only black and green colours, and place a red mark (a small cross 3 pixels wide) right on the centre of the image. Upon casting this image with its stripe pattern over the target surface, the idea is that by detecting the red marker on the acquired image that would indicate the exact location of the reference stripe. Upon acquisition of the image by the camera, the image is broken into their red, blue and green channels. The green channel is used to detect all stripes in the image, the red channel is used to detect the location of the red cross, and the blue channel is ignored. This worked well for the tested scenarios but it is possible that the marker needs to be made larger, or even extra markers may be required. This is left as future work, following end-users feedback.
Issues were observed when implementing the vision system software. The first aspect that was implemented was a template matching algorithm to detect the marker within the red channel image. The red marker must be found in the centre of the image width, so the task is to detect the height of the marker. The algorithm only needs to search a narrow band of pixels along the middle width of the image and from about 1/3 from the top to 2/3 down the image. This works well but it may need further optimisation for different environments. The second aspect was the 3D manipulation functions. At M9, global registration algorithms were implemented but large chunks of other desired functionality were just empty functions.
The issue of registration needed to be tackled from a different perspective as registering a scanned surface to a CAD model in a least squares sense is deemed to fail. The reason is that while CAD models are full 3D models, scanned surfaces are just that, localised views of a 3D object. By trying to match a surface to a full 3D model, the condition of minimum error is invariably the one in which a surface penetrates the model and lies somewhere in the middle of the CAD model. This is the wrong solution. Thus, alternative solutions were investigated; a solution based on z-buffer visibility was finally implemented. For example, if the scanned surface covers a partial view of the CAD model such as one side alone, then the CAD model needs to be rotated on screen using the mouse until only that side corresponding to the scanned surface is visible. An automatic function to select visible surface vertices was successfully developed and implemented. Now, the visible vertices roughly correspond to the scanned surface and the two parts can be registered together.
The other aspect to comment upon is the selection of data points directly from the surface of the model. This serves two purposes, to test the projection method of a welding point over the surface and also to allow specifying welding paths directly on the scanned model when a CAD model is not available. This function has been extremely problematic generating program crashes for seemingly no apparent reason. While most of problems have now been solved, the function must be used with care. The issue of automatically perform transformations to align the scanned surface to the robot base has been successfully implemented. While the implementation performs all transformations correctly from a theoretical point of view, integration tests were not performed. We also developed methods to capture a sequence of images using the robot saving the images and transformation parameters into an xml file. This file can then be later loaded and 3D surfaces automatically generated. Methods were also developed to automatically fuse two or more surfaces into a single 3D model; this can then be used to register to the corresponding visible area of a CAD model.
The development of the vision system software followed the MVC (model-view-controller) software methodology in object-oriented programming. The MVC is a useful design pattern for object code re-use and allows software developers to significantly reduce the time for development of complex applications with user interfaces. The MVC obviously has a model, which represents the logical structure of the data. This has been developed independently in the MARWIN application from existing code within our research group. Generic functions to create 3D models from scanned data were developed, and operations on it defined such as filling holes, mesh smoothing, texture mapping, wire and shaded meshes, visible surfaces, and so on. The MVC has a View, which is related to the user interface. For the MARWIN interface these were the icons, buttons, display buttons, drop-down menus, listbox, and so on – all objects a user sees on the screen of the MARWIN interface. Finally, the MVC has a Controller, which connects the (empty) user interface to the classes in the Model. The controller is thus the glue between Model and View and controls what happens with the data and when it should be displayed.
Considerable effort was placed on the development of the MVC View component. A large number of classes were developed handling model manipulation such as zooming in, out, translate, rotate, and also the ability to handle models independently on the screen. A large number of utility functions were developed to change mesh properties such as colour, visibility, wire mesh, shaded mesh, and textured mesh among others. All major model manipulation functions such as for generating 3D data or manipulating 3D models (registration, fusion, visible surfaces, etc) are easily accessible from dropdown menus and also from shortcut icons on the toolbars. The ability to save and export models to two different formats have been implemented. A model can be exported as a wavefront OBJ file format or as striper scene (a GMPR designed format for 3D scenes). The vision system interface is highly sophisticated and has been developed to high professional standards.
Potential Impact:
Marwin, as a system, has had a significant impact on the economy and on the society throughout its project life; but its beneficial effects will be observed after the project end, once started to be used by end-users, in the industry.
Economic benefits to the SME consortium
Marwin will boost the competitiveness of the SME consortium by providing them with an intelligent robot controlling solution that incorporates a state of the art cognitive system and human-computer interaction. This innovative technology allows for cost-effective and efficient work, contributing to an overall greater manufacturing speed and higher quality of the produced components. Combining the performance of modern robot technology with the skill of the human operators resulted in a flexible and high-end technology, capable of being used in small to large components batch production to create simple or complex metal work pieces with semiautomatic process. Aware of the highly competitive nature of this industry, the business opportunity and growing markets for new welding technologies, SME partners of the current MARWIN project recognize the potential to increase their competitiveness by developing a flexible and affordable automation system, adaptable to small business needs, which can be quickly set up and easily switched over to other product lines. Beyond its impact to the SMEs represented by this project, the consortium is convinced that the wider economic and strategic impact across Europe for the technology proposed in this project will be considerable.
The SME consortium originally estimated the future benefits of this project to be just below 3 million € in the first 5 years of commercialisation, based on the following assumption: 235 pieces of MARWIN systems will be sold in this period, half of them as a complete system. The return of investment is calculated for 1:2.95 on consortium level. Initial sales plan and financial calculations presented in details in section 3.2.2. show that the development and commercialization of MARWIN results in long-term market opportunities and profit for the participant SMEs.
General economic benefits for the EU
Despite a minor setback experienced, but long term growth in energy, transportation and other types of global commerce has generated increased activity in the automotive, aerospace and shipbuilding industries, especially in the Asia-Pacific and Eastern European regions, which in turn have had a positive influence for the welding industry. Due to the global economic crisis, the European welding equipment and the consumables markets have experienced a decrease in their sales and consequently in their production. However, it is expected that this will have effect only in the short-run and the sector should be prepared to face in the long-run an upswing due to the dynamic developments in the shipbuilding industry, as welding equipment and consumables manufacturers have consistently managed to maintain relations with the shipyard sector. It is predicted that the expected growth of the shipbuilding industry within the next decade will favourably impact the welding equipment and consumables markets. Moreover, it is expected that the use of MARWIN system by the welding industry will have a “knock-on” effect to its interrelated sectors as their customers will also benefit in competitiveness by having suppliers offering cost-effective, flexible and high quality production technology.
As a consequence of the global economic crisis both Spain and the United Kingdom are forecast to account for a larger share of overall EU market revenue in the future. The growth in demand has not only been in welded equipment and parts for the heavy and consumable industries, but also for welding technology. According to current Kompass data, there are 35,324 welding workshops in Europe. The current $3.17 billion European welding equipment and consumables market is expected to grow as increasing demand for advanced welding equipment is projected to push revenues to $3.51 billion in 2009. Despite the power of the industry, there are important factors impacting the competitive advantage of the SME welding sector that could be addressed by the cost effective and adaptable MARWIN technology. First, Asia is becoming a world hub for the welding industry. Outsourcing to East and South Asia, where wages are low but investment in industry is high, is tightening competition for the EU industry. Nowadays this is making it increasingly difficult for welding equipment and consumables manufacturers to maintain their market share. In addition, in terms of the production of high-end technology, Japan is today the robot technology-producing centre of the world. Secondly, the rising demand on the welding industry for the production of sophisticated parts, such as those used in aeronautics and for just-in-time production, is driving growth of welding robotics. The overall advances in the welding industry have had a significant impact on SMEs, particularly since the capital required to install automated equipment that would counteract the current threats and allow greater competition can be prohibitive.
Community/society objectives
The metalworking sector represents 8% of the total EU business with a turnover of € 970 billion a year, with the welding industry providing 730,000 full time, and 5.5 million welding related jobs in Europe. The production and marketing of MARWIN will increase the availability of jobs in the welding industry, as the SME sector will be in a position to respond to demands for more differentiated and higher quality pieces from a wider range of industries. The MARWIN technology will contribute to the creation of employment on three fronts: manufacturing SMEs will require higher skilled workers to produce the technology for both the EU and international markets; end user SMEs will increase their levels of production thereby demanding more capital labour; distribution SMEs will market this innovative solution to a wider customer base, increasing level of sales and the need for personnel to provide customer and technical services to industrial consumers. This technology will ensure the quality of parts produced, thus gaining access to a far wider market of clients seeking low production costs and good reliability. Improvement will increase process quality and throughput and greatly reduce time-to-market and costs, making European SMEs consortium and end users more productive and efficient in the global marketplace.
The effects of MARWIN in job creation will grow exponentially as European automotive, aeronautical and other relevant industrial sectors become more competitive through greater access to lower cost, high performance welded parts. The future of the welding industry will depend on the ability to produce highly sophisticated parts and an increasingly well trained and educated workforce to produce such parts. MARWIN will have a great impact on the quality of life in the European Union, as more foreign direct investment will spur industrial activity. As the EU expands, industrial activity will create a demand for sophisticated welding equipment and consumables.
Improved standards
All project members recognise that it is imperative for MARWIN to be implemented at a European level, given the social and economic importance to the welding sector in member states and its potential effects on improving quality of life of all European citizens. MARWIN will spur the project partners and other end users across Europe towards best practice by minimizing energy loss, creating jobs, increasing safety and reducing worker exposure to trial and error-correcting, thereby contributing to raising standards at a European level, especially in light of the entrance of the accession countries, whereby it is vital to ensure the cohesion and the uniformity of standards. Whilst the accession countries have invested massive efforts into preparing themselves for EU accession, and have made major progress in every aspect, they are still lagging significantly behind in key areas that affect their competitive position in the EU and world market, worker safety and environmentally friendly technologies. A repetitive cycle, typical of the welding industry is tiring for human workers but ideal for robots, while humans are more efficient in doing complex tasks and taking and making decisions based on their knowledge and personal experience. In addition to creating jobs and improving the quality of work, welding is considered a low level polluting industry compared to other manufacturing processes. With respect to the entrance of countries to the EU success or failure of companies in adapting to EU requirements in health and safety will have a direct impact on the quality of life of the community as a whole, as it is vital to ensure cohesion of the industry and the uniformity of standards.
Contribution to policy developments
Nowadays, robot welding is fully automated with no human interaction. If there is a discrepancy in the work pieces, a robot with no human control is not able to recognise and correct it; and will continue to carry on welding in the same way for all samples. This results in industrial material and energy wastage, as existing systems lack automatic defects correction. MARWIN will have a beneficial impact on the environment from reduced material and energy consumption in manufacturing. By facilitating reduction in energy use, MARWIN will ensure that European SMEs and end-users are working to the highest practicable standard and optimising the outcome for the environment to strike the right balance between market competitiveness and use of energy resources. The welding innovation proposed will help industry and the private sector comply with EU legislation, such as Directive 2006/32 of the European Parliament and of the Council of April 2006, which aims to ensure energy end-use efficiency. Additionally, it will help the EU in meeting targets established through the Lisbon Strategy. At the Lisbon European Council, the EU set itself a new strategic goal for the next decade: to become the most competitive and dynamic knowledge-based economy in the world, capable of sustainable economic growth with more and better jobs and greater social cohesion.
To maximise the chances of achieving these impacts, the SMEs have devised a detailed plan for the dissemination of foreground, which will allow both an effective promotion of the technology and the communication of the results to the wider public. All along its development, the system has been shown and demonstrated at various exhibitions, and its scientific content has been disseminated in papers and discussed at international conferences. International exhibitions and fairs have been identified once the project ends to serve as optimal ground for further dissemination. The SMEs plan to protect the Foreground knowledge with copyright and database right; therefore dissemination activities to the wider public contained limited technical information but focused on commercial and practical aspects. The project web site provides general information about the project, a description of the partners, the major objectives and news. The website also contains a restricted section in order to facilitate exchange of information and communication between the partners. To help smoothen the entry of the system in the market, an essential part of project meeting focussed on dissemination and use of knowledge. Practical aspects such as system costs, etc were also discussed. Discussions were also centred on the needs of the consortium member SMEs to carry out the necessary complex demonstration activities (business plan, industrial validation, performance verification, market strategy) with the acquired results before being able to enter the market to welding industry.
Following dissemination activities in scientific papers, journals and exhibitions, a strong interest from future end-users was expressed both verbally and by phone and email contacts. We currently receive on average 2 invitations per week to international conferences, and received a considerable amount of email enquiries through our project website, asking when the system will be made available.
List of Websites:
www.marwin-welding.eu
marwin@twi.co.uk or clement.schuhler@twi.co.uk
Welding tasks have been estimated to account for nearly a fourth of all industrial robots. However, the benefits decrease for the production of small batches of different products in small and medium-sized enterprises (SMEs). This is related directly to robot programming issues. Very often programming robots is time-consuming and requires qualified resources that SMEs often lack. Moreover, the latter companies often produce small batches of components; hence, setting-up and programming robots are perceived as an extremely costly activity.
The EU-funded project 'Decision making and augmented reality support for automatic welding installations' (MARWIN) developed a flexible, programmable system to optimise the robotic welding process. It enables the production of high-quality differentiated parts with increased productivity and reduced costs for the SMEs. The welding robot uses computer-based vision with an automatic path calculation and quality control system for enhanced production, flexibility and efficiency.
The project started with a survey sent to potential end-users, aiming to understand and identify real industrial needs. The MARWIN system was defined accordingly, including identification of suitable hardware and definition of communication specifications for the vision system, robot and personal computer controller.
A database of welding parameters, including welding torch positions, has been created. User information on the component to be produced is uploaded to the Marwin program via a graphical user interface (GUI). A weld parameters database enables the recommendation of suitable welding procedures. The operator and future end-users will also be able to upload their own welding procedures or modify existing ones.
MARWIN uses a vision system to compare the geometry and fit-up of the weld assembly to CAD assemblies uploaded earlier by the user and automatically adjusts welding parameters and weld robot trajectory. The vision system is mounted on the robot forearm and a tool changer allows it to be picked up as and when needed. A cognitive module enables communication between the robot and the vision system.
The MARWIN system is expected to significantly reduce the overall production costs of SMEs involved in production of industrial components and make robotic systems more accessible, while enhancing their potential product range, and boosting their competitiveness.
Project Context and Objectives:
Europe is a major player in the global machine tool and automation markets, providing some 52% of the world-wide value of machine tool production in 2002 (Japan 20%, China 9.6% and USA 6.1%) and is at the cutting edge of research and technological developments in that area . The past few years have witnessed a vigorous upsurge in the use of industrial robots for welded parts manufacturing. Notable reasons for the sharp increase in flexible automation with industrial robots are the shorter product cycles, shrinking batch sizes and a steadily rising level of wage costs.
Research and innovation in machine tools, robots and automation provides a key building block for economic performance, as it provides the intelligent production capabilities to make EU manufacturing sectors more competitive.
The use of industrial robotics in new application fields and industrial branches is considered a cornerstone in the development of many industries today. Giving impulses to the creation of new SMEs as providers of manufacturing equipment and services and the integration and support of small and medium enterprises in future manufacturing supply chains is a priority of current EC policies. Aware of the business opportunity and growing market in the welded parts industry, SMEs in the current project, with the help of RTD performers, intend to capitalise on this need by developing a product that will overcome the limitations of available systems for SME end users, thereby fulfilling a real market need.
Project objectives can be divided into three categories:
Scientific objectives
• Use cognitive vision (3D image reconstruction and registration) to calculate any deviations between a real assembly and a theoretical CAD file, and use this to automatically adjust welding and robotic parameters.
• Build a database of welding parameters and welding positions, to allow the Marwin system to choose the best parameter to obtain high quality welds (WP3).
Technological objectives
• Create an exact software model of the system and to define a communication scheme between the software modules as well as between the software and hardware modules, including classes, interfaces, data access restrictions, dependencies and database models (WP2).
• Create a vision system and 3D object reconstruction and registration software able to reconstruct the object with the accuracy of a minimum of 500microns as well as to find surface discrepancies on the work piece (WP4).
• Develop a software module that helps in the implementation of the technological instructions for welding by cognitive communication and is able to recognise and calculate the welding trajectory and welding parameters with minimal human intervention by using the knowledgebase (WP5).
• Create an intelligent learning environment based on Human-Computer Interaction (HCI) combined with Augmented Reality (AR) in order to visualise the welding parameters, trajectory and differences between the reconstructed and the designed model in a cutting-edge but intuitive graphical user environment aiding and speeding up the work of an operator (Task 6.1).
• Develop an easy to expand software framework that defines and implements the communication functions and protocols via Ethernet and RS232 port for industrial robot controllers in order to ensure the possibility to expand the communication skills of the product for industrial robot controller from different manufacturers. (Task 6.2)
• Expand existing welding robot systems (used as well as new) in order to deliver a flexible pre-competitive prototype by integrating all the software and hardware modules together and to transfer the methods to welding cell manufacturers of the consortium to be able to construct the system below the cost of € 30,000 (WP7).
• Test and validate the system at SMEs in laboratory and field tests by taking into account the object reconstruction, automatic program generation time, processing speed, consistency of weldment quality, joint location against manual welders by using ISO standard welding investigation techniques, first on simpler work-pieces and as a second stage on complex pieces(WP8).
Training, dissemination and exploitation objectives:
• Comprehensive training will be designed, prepared, tested, and delivered via well thought out training material, in order to ensure that the SME participant members will be able to assimilate the results of the project. Technology transfer will take place tri-monthly (Task 9.1).
• Non-confidential information about MARWIN project and its results will be disseminated to a wide and relevant audience to extend the impact of the project results (Task 9.2).
• A database of the target audience will be created for dissemination purposes. Seminars, workshops, conferences, and trade fairs will be identified / organized (at least 3 are planned); furthermore the public results will be disseminated via websites, mailings, articles, and papers in technical journals
(1000 SMEs to be addressed) (Task 9.2)
• Taking-up of results by the SMEs will be facilitated by training, dissemination, knowledge management, and IPR protection. Additionally, a business plan will be defined to increase the competitiveness
Project Results:
Marwin concept
The main idea of Marwin is to combine the human knowledge and problem handling skills with the accuracy and speed of robots, resulting in a flexible welding system with the ability to make decisions by a human welding expert and to create constantly high quality joints between two metal parts. The idea is originated from the fact of the cognitive psychology, which states that the information, which is enough to recognise the shape of an object, can be divided into two major parts, where 10% is given by our sensory organs (such as eyes) and the remaining is expanded by the knowledge, stored in the brain, which is gained from preliminary cognition. In MARWIN the eyes will be replaced by a vision system, which aims to reconstruct the visible part of the surface of the desired work piece, while the stored data will be replaced by a CAD model. This solution predicts a hybrid cognitive system, which will exploit the advantages of both human worker and robot.
MARWIN Software
The main result of the project is the MARWIN Control software itself, an offline robot programming solution with intelligent and flexible adjustment tools aiming to reduce welding robot programming time and to create part specific programs taking into consideration the real parts’ deviations from the CAD models. The main result can be broken down into several sub-modules as the following: Cognitive module, Simulation tools, Programming module, Communication module, Graphical User Interface and last but not least the Vision module.
Cognitive module and welding parameters database
The cognitive module is responsible for the calculation of the welding parameters for each joint in the assembly to be welded. For this purpose a special knowledgebase (i.e. welding parameters database) containing all essential welding parameters, from welding process to welding torch angle was created. These parameters included materials, plate thicknesses, gaps, torch angels, wire feed rate as well as the power source settings such as the welding speed, voltage and current. The database is prefilled with sample welding configurations and parameters selected and validated by TWI. As not every joint and welding process could be recorded in the duration of the project, the welding process, material and joint configuration were selected from feedback received on the survey at the beginning of the project.
The database was originally created as a remote database, so it could be reached from anywhere but for the final product, a local database was created as a mirror of the remote system.The database design was produced with MySQL WorkBench (WB). Once the tables, their relations and structures were identified, source codes could be generated. Although MySQL WB is capable to generate code automatically from existing designs, it was necessary to fine tune the generated code and restructure it. Due to the strict rules and constraints in the database structure some table must be created before a connecting table should be defined. Unfortunately MySQL WB does not take this into account and generates the code for the tables in ABC order.
The role of the database is to enable the user to find to optimal set of welding parameters as possible for the joints to be welded in his assembly. In order to select the best parameter combination from the database, a specific module is used that was implemented as a part of the project. A Neural Network based Decision Support System (DSS) was designed and implemented to aid the operators in decision making while selecting the most appropriate welding parameters. This DSS is considered as the one of the ‘brains’ of the MARWIN application. The system is a well know and widely applied Fuzzy Neural Network (FNN) so called Adaptive neuro-fuzzy inference system that is a neural network based on the Fuzzy logic. Since the system is able to handle partial truth or uncertainty and the advantages of neural network the benefits of both system is combined in one single framework.
Just like all neural networks the ANFIS has layers and nodes defined in the system. It is called adaptive due to the fact that each layer and node can have one or more parameters. The most important part of the system is the five layers in between with the following rules:
Layer 1: Contains parameterised or adaptive input nodes, which describe the complete domain of the input side of the system. The input domain is represented by membership functions which describes continuous intervals and / or set of discrete values. Parameters in this layer are referred to as premise parameters
Layer 2: Every node in this layer is fixed, and this is where the first T-norm (triangular norm) generalises intersection in a lattice and conjunction. More clearly this node used to ‘AND’ the membership grades.
Layer 3: The elements of this layer are fixed nodes which calculate the ratio of the firing strengths of the rules. This means that all the membership function of each fuzzy clause is being calculated and combined here. The outputs of the elements in this layer are called as normalised firing strength.
Layer 4: This layer holds adaptive nodes and performs consequent of the rules, or more clearly it adds the ‘THEN’ term to the expression. The parameters in Layer 4 are referred to as consequent parameters.
Layer 5: Layer 5 is a single-node layer and stands for to compute the overall output of the system.. For the proposed solution of the problem Type 2 had been chosen as it is capable to select a specific WPS based on the specified rules.
This means that the system is a multilayer system with defined nodes in each layer and well defined connection between the nodes. As a result of this structure a five-layer system was developed. The rules in this type of fuzzy neural network are the following: IF x is A1 and y is B1 THEN f1=p1x+q1y+r1; AND IF x is A2 and y is B2 THEN f2=p2x+q2y+r2.
From the programming point of view the system is based on the same parameters as the database. This means that the system is capable of handling problems up to a 24 dimensional space, meaning that each problem can be described by 24 different parameters. Each of the parameters belongs to a range so called Domain. The domain describes the possible values of parameters by using discrete or continuous values. These two values, DiscreteValue and ContinuousValue are the most basic building block of the system. Based on the DiscreteValues the system builds a so called WPS that is the definition of the problem and contain 24 values, each representing on parameter. Based on the knowledgebase the system creates a so called TrainingSet that is the collection of the known problem and solutions combination. A training set is also built up by WPSs but in this case the parameters can be described by Discrete and Continuous values as well. Based on the type the results of the parameter comparison the system return a value in the range of 0 and 1. The AnfisCore is responsible for handling the knowledgebase and for the calculation of responses.
Simulation environment
The simulation environment is the most important part of the system as this acts as the offline programming module of the system. This module is responsible for the simulation of the robot movement and to perform the calculations behind it. This includes the Inverse Kinematics (IK) as well as the Forward Kinematics (FK). In order to be able to calculate the TCP points that can be used to calculate the IK parameters a trajectory must be planned that includes the points on which the robot has to move through.
The trajectory planning is a fairly simple user interaction based method based on the popular A* path finding algorithm. The latter operates following a two-step behaviour. In order to represent a trajectory different waypoints and segments are defined. Waypoints are used to determine the location and orientation of the Tool Centre Point (TCP) of the robot. This means that the system can specify the alignment of the welding torch of the robot with respect to the work piece on specific points. Segments are used to group waypoints together according the type of movement. A movement can be a general positions movement, which is used when the robot moves at high speed big distances fairly far away from obstacles and work pieces. Operation movement is use during welding which has a certain speed taken from the knowledgebase according to the calculated welding parameters. There is also a low-speed trajectory movement that is used when the TCP and the robot moves close objects.
Two waypoints are always present in the environment: the StartPoint and the EndPoint. These points represent the beginning and the end of a movement or robotic operation. In many cases, these points are the same, but for specific purposes they can be different. This means that if the robot finishes the movement in an EndPoint different from the StartPoint the next program will start with a movement towards the StartPoint.
The trajectory is calculated in a simple way for the first time. The trajectory starts from the StartPoint and goes through all the WayPoints in the list in the order specified by the user. As a first step the system creates linear paths between the points and evaluates it. The evaluation criteria is simple, if any of the linear paths intersects with any object the path needs fine tuning, meaning that the system applies A* algorithm. This can be overridden if the user modifies the waypoint in way where none of the linear path intersects objects.
If users decide to apply the A*path finding technique, the system automatically creates a search space. This search space can be considered as a 3D grid. This is a necessary approach, as the A-star is able to operate only with discrete values. The size of the grids greatly affects the performance of the algorithm however the system give adequate responses. As first step the grid was created and the cells in the grid are marked as forbidden in which an obstacle is found. Following this the start point and target point is defined meaning. The algorithm FindPath then receives the space (including the forbidden cells), the start and target points and results a set of steps that is the shortest path between the start and target point.
Once the Path is defined, the system simulates the robot movement required to follow the latter trajectory. This requires solving of an IK problem for every point in the path. During the project, several IK solver solutions were tested including numerical and analytical solution, starting from the own developed CCD algorithm to 3rd party open source IK projects.
The current system applies the IkFast module of the OpenRave open source project. This library makes possible the use of IK library for any kind of industrial robots, the IK solvers are generated based on the OpenRave XML format that describes the dynamic and kinematic behaviour of the mechanical unit. This means that the IK solver is not universal, but needs to be generated for every mechanical configuration. Due to this reason a specific robot file format was created that encapsulates the IK solver’s DLL files, the robot model and the robot parameters as well. The file is a simple ZIP file with an altered file extension. As the IkFast module is implemented in C++ and have only Python interfaces a wrapper is implemented that makes the use of the library possible in the main .NET project. The wrapper is written in managed C++ that can be accessed from any .NET environments. In the implementation only two classes and a bunch of functions were wrapped, such as the IkSolution, IkSolutionList classes, or ComputeFk and ComputeIk fucntions.
For the simulation a background worker thread was developed that can be controlled over the main GUI just like a DVD player. It has simple playback, stop and pause functions and allows the user to perform step-by-step operation. The background worker was implemented in order to make sure that the calculations of the IK solutions will not block the application.
Communication module
The communication module used in the application is .NET; implementation of standard socket communication using UDP and TCP sockets for communication, while for file transfer standard FTP protocol is used. For communication purposes, the robot controller is also configured and a server program, so called User Task, was implemented in the SLIM language.
The TCP socket is used to communicate with the robot controller and exchange information between the robot and control PC. The module is based on the System.Net.Sockets.Socket class and builds a module around it. Once the object of the Socket is created by the system, a listener is created on a separate thread that is responsible for handling the incoming messages. There are four events wired for the class that are fired on Connect, Disconnect, Message Sent and Message Received events. The system is capable of sending raw byte streams as well as string messages and also able to receive the same format from the controller. The UDP socket is implemented on the same way however it does not maintain continuous connection with the other end point, therefore connect and disconnect events are removed. As the system does not check if the message that was sent has arrived, the message sent event handle was removed as well.
The communication flow is fairly simple. The PC application sends a command to the robot controller. On succesfull transmission the MessageSent event is fired. The robot controller then processes the message, and, according to the content of the command creates a response. If the command is an information request like STATUS or JOINT, the system returns the command with the requested information. If the system does not expect any data from the controller the controller responds with OK or ERROR based on the execution success of the command.
In order to be able to easily process messages a SocketMessage class was defined that can be created by using a string or a byte stream as input. The basic message contains only commands, and the extended messages contain data as well. For instance the message “Hello” only contains the message code while the Joint message has six parameters describing the angles of each mechanical joint. The following messages are interpreted: PING, CLOSE, JOINT, TCP, SET, RESET, SETSTOP, CLEARSTOP, STATUS, HELLO, BYEBYE, OK, ERROR, INVALID.
Socket messages can be sent also on the TCP as well as on the UDP socket.
Ftp connection is used to transmit programs between the Control computer and the robot controller as it is already available on every robot controllers. This makes the system available to transmit any of the programs to almost any kind of robot controllers. The module is built around the FtpWebRequest class implemented in the .NET framework. The FtpConnection class is able to connect to any FTP server if the username and password are specified and can switch directories. It is able to list the content of the actual directory and can upload and download files.
Programming module
The main goal of the programming module was to create a platform able to create robot programs for various robot platforms on multiple robot languages. For this purpose a middleware language is used that is a platform independent description using different objects and a configurable post-processor is applied to create the platform specific language.
The general robot program (MARWIN Generap Program – MGP) is built up from different modules. A module represents a set of instructions that can have multiple parameters according to instruction definition. The three main class of the programming are the GeneralCommand, RobotProgramModule and the RobotProgram.
The General command is built of a command type parameter and the respective parameters stored in dictionary that help to avoid duplications in the parameters list and enables the search between the parameters based on their types. During the creation of a command the type of the command must be specified and the constructor of the object creates a command prefilled with the available parameters. It is important to mention that parameters cannot be removed from the list and cannot be added however the existing values can be overridden by the user.
The RobotProgramModule is a collection of the commands introduced above. The main reason of applying modules is to be able to reuse parts of the program. The object holds a list of specified instructions that can be expanded and reduced as well. The most important feature is the save and load feature of the class that makes it possible to store the program including the commands and their parameters in an XML structure and to load them back. For this purpose SaveXml and LoadFromXml functions were developed that implement simple XML parsing functions. The result of the saving is a standard XML file that can be read by simple text editors with the extension of MPM (Marwin Program Module).
The robot program itself only contains the name and the file location of the program modules. The RobotProgram class is again also a list that contains the RobotProgramModules. It also have a load and a save function that again implements the XML parsing functions same as used in the RobotProgramModule class.
The term “Post Processing” is used in manufacturing and machining, and is part of the machining process. During post processing, a general language movement description is translated to a controller specific script, in which the commands, variables and parameters are in concert with the given machine’s characteristics.
Regarding post processing, a machine description which acts as a template, and describes the desired command and parameter mappings is created. The post processor then takes the general program and the machine description as input, and creates the machine specific program as output. The MARWIN Post Processor application is developed to streamline and automate this process. Its main purposes are the following:
• To provide an interface with which a user can configure a conversion template between MARWIN’s general program file structure and the desired machine’s specific program file structure
• To provide a convenient way of converting general language files to specific language files. Both machine description and mapping are stored in the standard XML format and can be edited by using the developed configuration tool.
Graphical User Interface
The Graphical User Interface (GUI) is the main contact between the user and the system. It was designed and developed in parallel with the backend modules taking into consideration the user requirements.
The main window can be broken down into different sections:
1. Ribbon and application menu – The ribbon is a standard Microsoft Office like ribbon that contains different menu points grouped by functions. Preferences can be access in the Application menu in the top left corner of the application.
2. Simulation view – The simulation view is responsible for displaying the content of the virtual scene created by the user and to visualise the different components. The scene can be zoomed, paned and rotated as well just like in any other CAD programs. Selection of objects can be made by pressing Shift button while clicking the mouse over the desired object.
3. Entity list – The list displays all the objects loaded into the simulation view, such as models, robots, work pieces and clamps. Object selection can be made through the list by selecting the desired entity from the list by clicking on its name.
4. Object properties – Anything selected in the simulation view or in the entity list has properties that can be accessed, monitored and modified over this interface. For instance the name, position and rotation of each object can be accessed; however, the robot has joint angles that can be modified.
5. Segments and waypoints – This area displays the points and paths where the robot moves its tooltip. Selection of waypoints can be made through this interface by selecting the names of the desired elements.
6. Logging and calculation results – Logging can be broken down into three parts: informal, warning and error messages. Informal messages are included for debugging the application behaviour and can be used to reveal software errors. Warning messages are displayed when a sub-module in the application return with an error but the operation of the software is not affected and this is highlighted in a message box too. Error messages are displayed when a critical error is occurred during the operation of the software. This means that the software needs to be restarted
For the development of the GUI, two 3rd party libraries were used. For the 3D visualisation and CAD handling, Devdept Eyeshot 6.0 was used while for the window and float component handling, DevExpress Winforms 12.1 was selected.
Vision system
As specified in the DoW, the vision system’s aim is to scan the component to be welded, in order to check if its local geometry and weld joint fit-up match these of the theoretical component.
The First vision system Hardware and software prototype was developed ahead of schedule, at M8. Since then, most of the work was spent on software implementation and on testing the light source and on bettering the quality of the acquired images as follow on to the deliverables D4.2 Algorithms for 3D Reconstruction for the Characteristics of the Camera and Projector, and D4.3 Methods to Normalize 3D Pose and Registration with CAD Models. Furthermore, we completed the deliverable D4.4 Benchmark 3D Reconstruction, Registration, and Error Analysis.
The outcomes of the work were that a large set of functions to manipulate 3D data were implemented and tested. Examples include functions to estimate local and global error analysis of registration, to automatically select visible surfaces from the 3D viewport, to automatically detect the reference stripe on images, to interactively define and project data points (e.g. welding control points) on the surface of the model, to automatically perform transformations (rotation and translation) according to Euler theorem to define scanned surfaces in relation to the robot base, methods to automatically save sequence of images using the robot, methods to automatically reconstruct models described in a prescribed XML format, methods to change mesh properties and visualisation, and a number of utility functions to support these main methods. Due to delay observed on work packages 5 and 6 affecting the integration phase, it is possible that middleware or utility functions are still required to make the system work seamlessly.
Regarding testing the light source and the quality of the acquired images: the vision system prototype was completed by month 8, concerns were observed on the brightness of light source, reflections on metallic surfaces and general levels of noise. The conclusion of sensitive studies we performed was that the laser light source (Microvision Pico Projector) should be replaced by white light (Aiptek Pocket Cinema Projector). The main reasons were related to the generated levels of noise. The laser Pico Projector generated high levels of “salt and pepper” noise and these are undesirable to the stripe detection algorithms. The white light projector emitted more lumens than the laser (50 lumens against 30) while, at the same time, generated less salt and pepper noise. Another aspect that was tested was the robustness of automatic detection of the reference stripe. In the earlier stages of the MARWIN project we manually clicked with the mouse on the reference stripe to guarantee correct 3D reconstruction. Initially a statistical method was tested to automatically find the reference stripe but it was not robust enough. In the end, the adopted solution was to generate a pattern of light using only black and green colours, and place a red mark (a small cross 3 pixels wide) right on the centre of the image. Upon casting this image with its stripe pattern over the target surface, the idea is that by detecting the red marker on the acquired image that would indicate the exact location of the reference stripe. Upon acquisition of the image by the camera, the image is broken into their red, blue and green channels. The green channel is used to detect all stripes in the image, the red channel is used to detect the location of the red cross, and the blue channel is ignored. This worked well for the tested scenarios but it is possible that the marker needs to be made larger, or even extra markers may be required. This is left as future work, following end-users feedback.
Issues were observed when implementing the vision system software. The first aspect that was implemented was a template matching algorithm to detect the marker within the red channel image. The red marker must be found in the centre of the image width, so the task is to detect the height of the marker. The algorithm only needs to search a narrow band of pixels along the middle width of the image and from about 1/3 from the top to 2/3 down the image. This works well but it may need further optimisation for different environments. The second aspect was the 3D manipulation functions. At M9, global registration algorithms were implemented but large chunks of other desired functionality were just empty functions.
The issue of registration needed to be tackled from a different perspective as registering a scanned surface to a CAD model in a least squares sense is deemed to fail. The reason is that while CAD models are full 3D models, scanned surfaces are just that, localised views of a 3D object. By trying to match a surface to a full 3D model, the condition of minimum error is invariably the one in which a surface penetrates the model and lies somewhere in the middle of the CAD model. This is the wrong solution. Thus, alternative solutions were investigated; a solution based on z-buffer visibility was finally implemented. For example, if the scanned surface covers a partial view of the CAD model such as one side alone, then the CAD model needs to be rotated on screen using the mouse until only that side corresponding to the scanned surface is visible. An automatic function to select visible surface vertices was successfully developed and implemented. Now, the visible vertices roughly correspond to the scanned surface and the two parts can be registered together.
The other aspect to comment upon is the selection of data points directly from the surface of the model. This serves two purposes, to test the projection method of a welding point over the surface and also to allow specifying welding paths directly on the scanned model when a CAD model is not available. This function has been extremely problematic generating program crashes for seemingly no apparent reason. While most of problems have now been solved, the function must be used with care. The issue of automatically perform transformations to align the scanned surface to the robot base has been successfully implemented. While the implementation performs all transformations correctly from a theoretical point of view, integration tests were not performed. We also developed methods to capture a sequence of images using the robot saving the images and transformation parameters into an xml file. This file can then be later loaded and 3D surfaces automatically generated. Methods were also developed to automatically fuse two or more surfaces into a single 3D model; this can then be used to register to the corresponding visible area of a CAD model.
The development of the vision system software followed the MVC (model-view-controller) software methodology in object-oriented programming. The MVC is a useful design pattern for object code re-use and allows software developers to significantly reduce the time for development of complex applications with user interfaces. The MVC obviously has a model, which represents the logical structure of the data. This has been developed independently in the MARWIN application from existing code within our research group. Generic functions to create 3D models from scanned data were developed, and operations on it defined such as filling holes, mesh smoothing, texture mapping, wire and shaded meshes, visible surfaces, and so on. The MVC has a View, which is related to the user interface. For the MARWIN interface these were the icons, buttons, display buttons, drop-down menus, listbox, and so on – all objects a user sees on the screen of the MARWIN interface. Finally, the MVC has a Controller, which connects the (empty) user interface to the classes in the Model. The controller is thus the glue between Model and View and controls what happens with the data and when it should be displayed.
Considerable effort was placed on the development of the MVC View component. A large number of classes were developed handling model manipulation such as zooming in, out, translate, rotate, and also the ability to handle models independently on the screen. A large number of utility functions were developed to change mesh properties such as colour, visibility, wire mesh, shaded mesh, and textured mesh among others. All major model manipulation functions such as for generating 3D data or manipulating 3D models (registration, fusion, visible surfaces, etc) are easily accessible from dropdown menus and also from shortcut icons on the toolbars. The ability to save and export models to two different formats have been implemented. A model can be exported as a wavefront OBJ file format or as striper scene (a GMPR designed format for 3D scenes). The vision system interface is highly sophisticated and has been developed to high professional standards.
Potential Impact:
Marwin, as a system, has had a significant impact on the economy and on the society throughout its project life; but its beneficial effects will be observed after the project end, once started to be used by end-users, in the industry.
Economic benefits to the SME consortium
Marwin will boost the competitiveness of the SME consortium by providing them with an intelligent robot controlling solution that incorporates a state of the art cognitive system and human-computer interaction. This innovative technology allows for cost-effective and efficient work, contributing to an overall greater manufacturing speed and higher quality of the produced components. Combining the performance of modern robot technology with the skill of the human operators resulted in a flexible and high-end technology, capable of being used in small to large components batch production to create simple or complex metal work pieces with semiautomatic process. Aware of the highly competitive nature of this industry, the business opportunity and growing markets for new welding technologies, SME partners of the current MARWIN project recognize the potential to increase their competitiveness by developing a flexible and affordable automation system, adaptable to small business needs, which can be quickly set up and easily switched over to other product lines. Beyond its impact to the SMEs represented by this project, the consortium is convinced that the wider economic and strategic impact across Europe for the technology proposed in this project will be considerable.
The SME consortium originally estimated the future benefits of this project to be just below 3 million € in the first 5 years of commercialisation, based on the following assumption: 235 pieces of MARWIN systems will be sold in this period, half of them as a complete system. The return of investment is calculated for 1:2.95 on consortium level. Initial sales plan and financial calculations presented in details in section 3.2.2. show that the development and commercialization of MARWIN results in long-term market opportunities and profit for the participant SMEs.
General economic benefits for the EU
Despite a minor setback experienced, but long term growth in energy, transportation and other types of global commerce has generated increased activity in the automotive, aerospace and shipbuilding industries, especially in the Asia-Pacific and Eastern European regions, which in turn have had a positive influence for the welding industry. Due to the global economic crisis, the European welding equipment and the consumables markets have experienced a decrease in their sales and consequently in their production. However, it is expected that this will have effect only in the short-run and the sector should be prepared to face in the long-run an upswing due to the dynamic developments in the shipbuilding industry, as welding equipment and consumables manufacturers have consistently managed to maintain relations with the shipyard sector. It is predicted that the expected growth of the shipbuilding industry within the next decade will favourably impact the welding equipment and consumables markets. Moreover, it is expected that the use of MARWIN system by the welding industry will have a “knock-on” effect to its interrelated sectors as their customers will also benefit in competitiveness by having suppliers offering cost-effective, flexible and high quality production technology.
As a consequence of the global economic crisis both Spain and the United Kingdom are forecast to account for a larger share of overall EU market revenue in the future. The growth in demand has not only been in welded equipment and parts for the heavy and consumable industries, but also for welding technology. According to current Kompass data, there are 35,324 welding workshops in Europe. The current $3.17 billion European welding equipment and consumables market is expected to grow as increasing demand for advanced welding equipment is projected to push revenues to $3.51 billion in 2009. Despite the power of the industry, there are important factors impacting the competitive advantage of the SME welding sector that could be addressed by the cost effective and adaptable MARWIN technology. First, Asia is becoming a world hub for the welding industry. Outsourcing to East and South Asia, where wages are low but investment in industry is high, is tightening competition for the EU industry. Nowadays this is making it increasingly difficult for welding equipment and consumables manufacturers to maintain their market share. In addition, in terms of the production of high-end technology, Japan is today the robot technology-producing centre of the world. Secondly, the rising demand on the welding industry for the production of sophisticated parts, such as those used in aeronautics and for just-in-time production, is driving growth of welding robotics. The overall advances in the welding industry have had a significant impact on SMEs, particularly since the capital required to install automated equipment that would counteract the current threats and allow greater competition can be prohibitive.
Community/society objectives
The metalworking sector represents 8% of the total EU business with a turnover of € 970 billion a year, with the welding industry providing 730,000 full time, and 5.5 million welding related jobs in Europe. The production and marketing of MARWIN will increase the availability of jobs in the welding industry, as the SME sector will be in a position to respond to demands for more differentiated and higher quality pieces from a wider range of industries. The MARWIN technology will contribute to the creation of employment on three fronts: manufacturing SMEs will require higher skilled workers to produce the technology for both the EU and international markets; end user SMEs will increase their levels of production thereby demanding more capital labour; distribution SMEs will market this innovative solution to a wider customer base, increasing level of sales and the need for personnel to provide customer and technical services to industrial consumers. This technology will ensure the quality of parts produced, thus gaining access to a far wider market of clients seeking low production costs and good reliability. Improvement will increase process quality and throughput and greatly reduce time-to-market and costs, making European SMEs consortium and end users more productive and efficient in the global marketplace.
The effects of MARWIN in job creation will grow exponentially as European automotive, aeronautical and other relevant industrial sectors become more competitive through greater access to lower cost, high performance welded parts. The future of the welding industry will depend on the ability to produce highly sophisticated parts and an increasingly well trained and educated workforce to produce such parts. MARWIN will have a great impact on the quality of life in the European Union, as more foreign direct investment will spur industrial activity. As the EU expands, industrial activity will create a demand for sophisticated welding equipment and consumables.
Improved standards
All project members recognise that it is imperative for MARWIN to be implemented at a European level, given the social and economic importance to the welding sector in member states and its potential effects on improving quality of life of all European citizens. MARWIN will spur the project partners and other end users across Europe towards best practice by minimizing energy loss, creating jobs, increasing safety and reducing worker exposure to trial and error-correcting, thereby contributing to raising standards at a European level, especially in light of the entrance of the accession countries, whereby it is vital to ensure the cohesion and the uniformity of standards. Whilst the accession countries have invested massive efforts into preparing themselves for EU accession, and have made major progress in every aspect, they are still lagging significantly behind in key areas that affect their competitive position in the EU and world market, worker safety and environmentally friendly technologies. A repetitive cycle, typical of the welding industry is tiring for human workers but ideal for robots, while humans are more efficient in doing complex tasks and taking and making decisions based on their knowledge and personal experience. In addition to creating jobs and improving the quality of work, welding is considered a low level polluting industry compared to other manufacturing processes. With respect to the entrance of countries to the EU success or failure of companies in adapting to EU requirements in health and safety will have a direct impact on the quality of life of the community as a whole, as it is vital to ensure cohesion of the industry and the uniformity of standards.
Contribution to policy developments
Nowadays, robot welding is fully automated with no human interaction. If there is a discrepancy in the work pieces, a robot with no human control is not able to recognise and correct it; and will continue to carry on welding in the same way for all samples. This results in industrial material and energy wastage, as existing systems lack automatic defects correction. MARWIN will have a beneficial impact on the environment from reduced material and energy consumption in manufacturing. By facilitating reduction in energy use, MARWIN will ensure that European SMEs and end-users are working to the highest practicable standard and optimising the outcome for the environment to strike the right balance between market competitiveness and use of energy resources. The welding innovation proposed will help industry and the private sector comply with EU legislation, such as Directive 2006/32 of the European Parliament and of the Council of April 2006, which aims to ensure energy end-use efficiency. Additionally, it will help the EU in meeting targets established through the Lisbon Strategy. At the Lisbon European Council, the EU set itself a new strategic goal for the next decade: to become the most competitive and dynamic knowledge-based economy in the world, capable of sustainable economic growth with more and better jobs and greater social cohesion.
To maximise the chances of achieving these impacts, the SMEs have devised a detailed plan for the dissemination of foreground, which will allow both an effective promotion of the technology and the communication of the results to the wider public. All along its development, the system has been shown and demonstrated at various exhibitions, and its scientific content has been disseminated in papers and discussed at international conferences. International exhibitions and fairs have been identified once the project ends to serve as optimal ground for further dissemination. The SMEs plan to protect the Foreground knowledge with copyright and database right; therefore dissemination activities to the wider public contained limited technical information but focused on commercial and practical aspects. The project web site provides general information about the project, a description of the partners, the major objectives and news. The website also contains a restricted section in order to facilitate exchange of information and communication between the partners. To help smoothen the entry of the system in the market, an essential part of project meeting focussed on dissemination and use of knowledge. Practical aspects such as system costs, etc were also discussed. Discussions were also centred on the needs of the consortium member SMEs to carry out the necessary complex demonstration activities (business plan, industrial validation, performance verification, market strategy) with the acquired results before being able to enter the market to welding industry.
Following dissemination activities in scientific papers, journals and exhibitions, a strong interest from future end-users was expressed both verbally and by phone and email contacts. We currently receive on average 2 invitations per week to international conferences, and received a considerable amount of email enquiries through our project website, asking when the system will be made available.
List of Websites:
www.marwin-welding.eu
marwin@twi.co.uk or clement.schuhler@twi.co.uk