Periodic Reporting for period 1 - uDEVOPS (Software Quality Assurance for Microservice Development Operations Engineering)
Berichtszeitraum: 2020-03-01 bis 2023-05-31
Software Quality Assurance (SQA) aims at monitoring the engineering processes and methods used for software development and production so as to ensure proper quality. The EU-funded uDEVOPS project intends to form an international and intersectoral research network focusing on SQA for microservice development operations engineering processes. The objective is to promote a cultural shift that makes microservice development operations testing a cornerstone in development and operation phases. The testing process will integrate algorithms, techniques and methods for model-based context learning, in vivo reliability, performance and security sampling-based testing driven by the learnt context, and risk assessment. The project will result in the implementation of know-how and breakthroughs for novel products, methodologies and services and create strategic bases to deal with SQA challenges in modern software production practices.
From the beginning of the project, the work concerned with monitoring and SQA tasks such as testing, defect prediction, sustainability concerns (e.g. energy consumption characterization), process quality (e.g. tickets calssification), security (e.g. evaluation of cyber-ranges).
A characterization of tuning of solutions for monitoring in the context of microservices and DevOps, and the investigation of techniques and tools for “context modelling” in Microservice Architecture (MSA), has been extensively carried out in the first part of the project. The analysis pertained to monitoring strategies and tools for MSA to gather quality data about the reliabiltiy, performance, energy consumption.
We have completed a systematic mapping study to scan all relevant open-source repositories (hence, tools, plugins, libraries for monitoring) indexed in GitHub and in Google, so as to give a broader view of the artifacts available and take inspiration to improve our final solution. Moreover, we have developed our own prototypal solution for monitoring, called “uSauron”. We will enforce decoupling between monitoring and testing, so as results of our next WP on testing can work with any monitoring solution.
We are stressing the usage of Artificial Intelligence/Machine Learning (AI/ML) solutions for supporting SQA decision by using data gathered by monitoring. We have exploited AI/ML algorithms for supporting relevant SQA decisions, in particular by implementing: 1) learning-to-rank algorithms to support automatic (regression) testing, 2) causal structure discovery (CSD) algorithms for performance-related and energy-related characterization of an MSA, for supporting Root Cause Analysis (RCA) as well as preventive decisions (e.g. which microservice needs more test, which needs protection mechanisms, etc.), 3) algorithms for anomaly detection and for assessing the propagation paths of performance or energy consumption via multivariate transfer entropy, and, 4) binary classification for the so-called just-in-time software defect prediction, aimed at predicting if a code commit is likely to introduce a defect, as well as other machine learning algorithms for automatic user tickets classification, 5) user behaviour modelling from logs, useful for UX evaluation as well as for testing purposes.
In the context of testing, we developed "MacroHive," a novel tool for grey-box testing microservices architecture. Since automated test generation techniques for microservice architecture have received little attention in the literature, they frequently borrow SOTA techniques from RESTful services black-box testing (MacroHive is an important advancement in the SOTA for this task). We are currently working on several quality-related tasks, including 1) applying anomaly detection and root cause analysis (including causal reasoning) algorithms for energy consumption and performance bottleneck detection and energy/performance configurations improvement, 2) on continuous performance modelling, 3) on improving user behaviour modelling and tickets classification, on 4) tests generation from logs, 5) on sampling-based algorithms for quality attributes assessment, for which the consortium has explored statistical survey sampling theory for enabling a smart selection/generation of test cases.
Till the end of the project we plan to develop an unified proof-of-concept platform that support all these tasks and different objectives, for instance, by integrating these components together . Most of the work performed has been already publicly released via scientific publications, social media.
A characterization of tuning of solutions for monitoring in the context of microservices and DevOps, and the investigation of techniques and tools for “context modelling” in Microservice Architecture (MSA), has been extensively carried out in the first part of the project. The analysis pertained to monitoring strategies and tools for MSA to gather quality data about the reliabiltiy, performance, energy consumption.
We have completed a systematic mapping study to scan all relevant open-source repositories (hence, tools, plugins, libraries for monitoring) indexed in GitHub and in Google, so as to give a broader view of the artifacts available and take inspiration to improve our final solution. Moreover, we have developed our own prototypal solution for monitoring, called “uSauron”. We will enforce decoupling between monitoring and testing, so as results of our next WP on testing can work with any monitoring solution.
We are stressing the usage of Artificial Intelligence/Machine Learning (AI/ML) solutions for supporting SQA decision by using data gathered by monitoring. We have exploited AI/ML algorithms for supporting relevant SQA decisions, in particular by implementing: 1) learning-to-rank algorithms to support automatic (regression) testing, 2) causal structure discovery (CSD) algorithms for performance-related and energy-related characterization of an MSA, for supporting Root Cause Analysis (RCA) as well as preventive decisions (e.g. which microservice needs more test, which needs protection mechanisms, etc.), 3) algorithms for anomaly detection and for assessing the propagation paths of performance or energy consumption via multivariate transfer entropy, and, 4) binary classification for the so-called just-in-time software defect prediction, aimed at predicting if a code commit is likely to introduce a defect, as well as other machine learning algorithms for automatic user tickets classification, 5) user behaviour modelling from logs, useful for UX evaluation as well as for testing purposes.
In the context of testing, we developed "MacroHive," a novel tool for grey-box testing microservices architecture. Since automated test generation techniques for microservice architecture have received little attention in the literature, they frequently borrow SOTA techniques from RESTful services black-box testing (MacroHive is an important advancement in the SOTA for this task). We are currently working on several quality-related tasks, including 1) applying anomaly detection and root cause analysis (including causal reasoning) algorithms for energy consumption and performance bottleneck detection and energy/performance configurations improvement, 2) on continuous performance modelling, 3) on improving user behaviour modelling and tickets classification, on 4) tests generation from logs, 5) on sampling-based algorithms for quality attributes assessment, for which the consortium has explored statistical survey sampling theory for enabling a smart selection/generation of test cases.
Till the end of the project we plan to develop an unified proof-of-concept platform that support all these tasks and different objectives, for instance, by integrating these components together . Most of the work performed has been already publicly released via scientific publications, social media.
uDEVOPS addresses Testing and Software Quality Assurance (SQA) in Microservices and in DevOps. The project aim to tackle the main state of the art challenges of these two contexts by developing a joint approach that combines techniques developed separately for Microservices and DevOps in a unique workflow, thereby extending the benefits of combining both. In particular, by coupling the testing activities defined at development time with information from operation, via automatic learning of the context at operational time. Unlike existing approaches, (usage, behaviour, architecture, failure) modelling is supported by monitoring and AI/ML algorithms (e.g. conventional Machine Learning as well as causality-based AI) to exploit information about the architecture/behaviour of the system, coping with dynamic aspects of the application at runtime.
Impact: The project has the potential to significantly impact participant organizations' capacities to introduce innovation into their products/services and contribute to European excellence in: 1) SQA, 2) SbS systems, 3) agile & DevOps proactive, 4) AI/ML for SE, and 5) the cloud. These are arguably the most important enablers of modern software production and global market success. The outcomes of uDEVOPS will influence the decision-making process of modern IT-related companies due to its emphasis on SQA in modern development paradigm and infrastructure. In fact, the uDEVOPS solution will be able to automate continuous quality (and business risk) measurements based on attributes of interest such as reliability, performance, and security Key Performance Indicators, as well as automate a continuous testing process to spot defects or bottlenecks. The project is expected then to bring improvements in the quality of systems wherein software and ICTs (Information and Communications Technologies) are key enablers, reducing risks for people and assets in safety-critical scenarios, as well as reducing cost for companies to produce high-quality competitive systems. So far, the project is impacting both in terms of scientific advancement, especially in the area of SQA for Microservices, including testing, and in the companies know-how, especially regarding hot topic and emerging topics such as energy efficiency and sustainable computing, artificial intelligence, reasoning-based strategies.
Impact: The project has the potential to significantly impact participant organizations' capacities to introduce innovation into their products/services and contribute to European excellence in: 1) SQA, 2) SbS systems, 3) agile & DevOps proactive, 4) AI/ML for SE, and 5) the cloud. These are arguably the most important enablers of modern software production and global market success. The outcomes of uDEVOPS will influence the decision-making process of modern IT-related companies due to its emphasis on SQA in modern development paradigm and infrastructure. In fact, the uDEVOPS solution will be able to automate continuous quality (and business risk) measurements based on attributes of interest such as reliability, performance, and security Key Performance Indicators, as well as automate a continuous testing process to spot defects or bottlenecks. The project is expected then to bring improvements in the quality of systems wherein software and ICTs (Information and Communications Technologies) are key enablers, reducing risks for people and assets in safety-critical scenarios, as well as reducing cost for companies to produce high-quality competitive systems. So far, the project is impacting both in terms of scientific advancement, especially in the area of SQA for Microservices, including testing, and in the companies know-how, especially regarding hot topic and emerging topics such as energy efficiency and sustainable computing, artificial intelligence, reasoning-based strategies.