Periodic Reporting for period 3 - OpenDreamKit (Open Digital Research Environment Toolkit for the Advancement of Mathematics)
Reporting period: 2018-09-01 to 2019-08-31
In the last decades, vibrant communities of mathematicians have spearheaded the Open Science movement, developing a wide ecosystem of open tools covering all aspects of mechanized mathematics: computation, tabulation (data), deduction, and narration. The ecosystem suffered however from fragmentation and from lack of recognition when funding and career paths reward novel results rather than the daily work of fixing bugs and supporting users; this led to long standing technical issues which impeded its sustainability in a rapidly evolving landscape of computing.
Meanwhile early Virtual Research Environments (VRE) had proven their potential to better serve the growing needs of mathematicians, reduce entry barriers, and promote collaboration and Open Science.
The ecosystem offered the potential to cover a large area of pure mathematics, and a variety of use cases ranging from a one-system-one laptop to multi-system institution-wide cloud deployments. This could not however be covered by a single one-size-fits-all VRE.
The existing communities joined forces, and further joined with the interactive computing environment Jupyter to deliver instead a toolkit from which to build VREs.
Meanwhile early Virtual Research Environments (VRE) had proven their potential to better serve the growing needs of mathematicians, reduce entry barriers, and promote collaboration and Open Science.
The ecosystem offered the potential to cover a large area of pure mathematics, and a variety of use cases ranging from a one-system-one laptop to multi-system institution-wide cloud deployments. This could not however be covered by a single one-size-fits-all VRE.
The existing communities joined forces, and further joined with the interactive computing environment Jupyter to deliver instead a toolkit from which to build VREs.
By integrating or updating best practices, embracing emerging technologies (Docker containers, Conda packages, …), by decoupling components and sharing infrastructure, we managed to make OpenDreamKit components available and interoperable in a much wider range of environments. Notable success stories are:
- Porting of the components to Windows, MacOS and Linux;
- Distribution of the components through official package repositories for major Linux distributions (Debian, Ubuntu, Fedora, Arch, Gentoo);
- Distribution of Docker images and Conda packages suited for deployment of our components in cloud environments;
- Porting of SageMath to Python 3;
- Modularization of various OpenDreamKit components, in particular SageMath and GAP.
This was reinforced by tuning computational software to deliver high performance on modern computing platforms ranging from personal laptops to HPC clusters.
We contributed to a tight integration of OpenDreamKit components with Jupyter, and to the Jupyter ecosystem itself, notably around collaboration, validation, interactive web pages, 3D visualisation, interactive mathematics, and collaborative workspaces.
To assess and demonstrate the applicability of our technology well beyond mathematics, we integrated micromagnetism computational engines into Jupyter to deliver a VRE taylored for interactive micromagnetism. We also used ourselves extensively all the elements of the OpenDreamKit for research, teaching, training, authoring interactive textbooks material, and much more, in a variety of fields ranging from biology to computer science, all using Jupyter technology. The outcome was immediately fed back into the project development.
Finally we explored novel knowledge-based approaches for the high-level integration of large stacks of mathematical components. Like an interpreter for human language, it ensures meaning-preserving communication in mathematical languages between computation systems, databases, and humans. Concrete outcomes for bridging the gap between computation and data are already in production for data generation, online data exploration and live data exploration.
In addition, community building, training, and dissemination activities have been at the forefront of OpenDreamKit, with more than one hundred events organized or co-organized by OpenDreamKit, including developer workshops, training conferences, or satellite sessions, adding up to 1800 trainees. We also actively promoted diversity, through dedicated actions aimed at women and residents of developing countries. We also created resources for researchers and teachers to accompany them in their use of the technologies we promote, such as tutorials, explainer comics and short videos. This includes contributions to the open textbook Computational Mathematics with SageMath which is nowadays reaching 500 downloads per day.
- Porting of the components to Windows, MacOS and Linux;
- Distribution of the components through official package repositories for major Linux distributions (Debian, Ubuntu, Fedora, Arch, Gentoo);
- Distribution of Docker images and Conda packages suited for deployment of our components in cloud environments;
- Porting of SageMath to Python 3;
- Modularization of various OpenDreamKit components, in particular SageMath and GAP.
This was reinforced by tuning computational software to deliver high performance on modern computing platforms ranging from personal laptops to HPC clusters.
We contributed to a tight integration of OpenDreamKit components with Jupyter, and to the Jupyter ecosystem itself, notably around collaboration, validation, interactive web pages, 3D visualisation, interactive mathematics, and collaborative workspaces.
To assess and demonstrate the applicability of our technology well beyond mathematics, we integrated micromagnetism computational engines into Jupyter to deliver a VRE taylored for interactive micromagnetism. We also used ourselves extensively all the elements of the OpenDreamKit for research, teaching, training, authoring interactive textbooks material, and much more, in a variety of fields ranging from biology to computer science, all using Jupyter technology. The outcome was immediately fed back into the project development.
Finally we explored novel knowledge-based approaches for the high-level integration of large stacks of mathematical components. Like an interpreter for human language, it ensures meaning-preserving communication in mathematical languages between computation systems, databases, and humans. Concrete outcomes for bridging the gap between computation and data are already in production for data generation, online data exploration and live data exploration.
In addition, community building, training, and dissemination activities have been at the forefront of OpenDreamKit, with more than one hundred events organized or co-organized by OpenDreamKit, including developer workshops, training conferences, or satellite sessions, adding up to 1800 trainees. We also actively promoted diversity, through dedicated actions aimed at women and residents of developing countries. We also created resources for researchers and teachers to accompany them in their use of the technologies we promote, such as tutorials, explainer comics and short videos. This includes contributions to the open textbook Computational Mathematics with SageMath which is nowadays reaching 500 downloads per day.
Researchers, teachers and scientists in general now benefit from a rich uniform web-based user interface to all the OpenDreamKit computational components, which they can use on a wide variety of platforms, from laptops to commercial or non commercial cloud services. We contributed to a number of such deployments, demonstrating the versatility of the toolkit.
By enabling Research Software Engineers to focus full time for months in a row on delicate challenges, we have tackled problems that the communities had been facing for a decade, such as porting SageMath and its dependencies to the Windows platform. This has demonstrated the enormous value that Research Software Engineers (RSE) can bring to research, to support of end-users and collaborative developments by the community, and, generally speaking, Open Science. We have played an important role in the RSE movement, notably in the UK, exploring numerous hot and difficult questions: role, funding patterns, structuring, career paths, etc.
As was hoped, OpenDreamKit delivered a boost to the landscape of open source math software; it strives ever more than five years ago. We are particularly proud to have brought together many computational mathematics communities that, at earlier times, had been evolving independently or competing
The major challenge to come is to secure employment of a number of Research Software Engineers, as much as possible in long term positions, to support hundreds of contributors and tens if not hundreds of thousands of users of open source math software.
As witnessed by the ACM award and by the wide adoption of the Jupyter technology in research, education and industry, the Jupyter ecosystem is there to remain a major player of the field for years to come. OpenDreamKit played a significant accelerator role by providing core Jupyter development. Of particular relevance is its adoption by a number of EU-funded projects working towards the European Open Science Cloud (EOSC) which directly benefit from outcomes of OpenDreamKit.
It is therefore crucial that the EU keeps funding core Jupyter development, as an asset for the EOSC, science and industry, as well as to enable the continuing acceleration and steering of this key open technology.
Finally, beyond the traditional items of Open Science, we have explored a novel item, namely Open EU Project Management, and demonstrated its effectiveness. Indeed, we strongly believe that preparing the proposal and running the project completely in the open with a bottom up approach has played a fundamental role in engaging the communities, developing reciprocal trust, locating the toughest hurdles that users and developers actually face, co-designing solutions and co-implementing them efficiently.
We strongly recommend the experience; long live OpenDreamKit’s spirit!
By enabling Research Software Engineers to focus full time for months in a row on delicate challenges, we have tackled problems that the communities had been facing for a decade, such as porting SageMath and its dependencies to the Windows platform. This has demonstrated the enormous value that Research Software Engineers (RSE) can bring to research, to support of end-users and collaborative developments by the community, and, generally speaking, Open Science. We have played an important role in the RSE movement, notably in the UK, exploring numerous hot and difficult questions: role, funding patterns, structuring, career paths, etc.
As was hoped, OpenDreamKit delivered a boost to the landscape of open source math software; it strives ever more than five years ago. We are particularly proud to have brought together many computational mathematics communities that, at earlier times, had been evolving independently or competing
The major challenge to come is to secure employment of a number of Research Software Engineers, as much as possible in long term positions, to support hundreds of contributors and tens if not hundreds of thousands of users of open source math software.
As witnessed by the ACM award and by the wide adoption of the Jupyter technology in research, education and industry, the Jupyter ecosystem is there to remain a major player of the field for years to come. OpenDreamKit played a significant accelerator role by providing core Jupyter development. Of particular relevance is its adoption by a number of EU-funded projects working towards the European Open Science Cloud (EOSC) which directly benefit from outcomes of OpenDreamKit.
It is therefore crucial that the EU keeps funding core Jupyter development, as an asset for the EOSC, science and industry, as well as to enable the continuing acceleration and steering of this key open technology.
Finally, beyond the traditional items of Open Science, we have explored a novel item, namely Open EU Project Management, and demonstrated its effectiveness. Indeed, we strongly believe that preparing the proposal and running the project completely in the open with a bottom up approach has played a fundamental role in engaging the communities, developing reciprocal trust, locating the toughest hurdles that users and developers actually face, co-designing solutions and co-implementing them efficiently.
We strongly recommend the experience; long live OpenDreamKit’s spirit!