Evaluating the impact and outcomes of European SSH research

Final Report Summary - IMPACT-EV (Evaluating the impact and outcomes of European SSH research)

Executive Summary:
In recent years, the impacts of SSH research have become a highly contested arena. Not only due to inadequate tools to comprehensively assessing them, but also a well-spread perception of relatively poor impacts achieved through the EU funded SSH research and a deep questioning of the need to preserve specific funding for SSH research in Europe. Amidst this context, an increasing number of R&I systems as well as funding organizations all around the world are attempting to measure the impact of SSH research responding to some citizens claims’ vindicating major levels of accountability and an enhanced access to scientific developments.

IMPACT-EV has been dedicated to respond to this scientific and socio-political scenario in many ways. During four years, the consortium first developed a mapping of what is already known about SSH research evaluation, a comparative analysis of SSH assessment in 12 national research systems (9 EU countries and 3 international), an Ex-post evaluation of the impacts and outcomes obtained from previous FP6 (last call) and FP7 SSH projects and the Interim evaluation for H2020 Challenge 6 projects. Then, the consortium explored in depth the following impact dimensions: 1) a bibliometric pilot study for to better measure scientific impact; 2) interviews to policy makers and a Delphi Panel with researchers, policy makers and end-users, focused on developing appropriate indicators for political and social impact of research; 3) an extensive examination of the presence of SSH outcomes and social impacts in social media; and 4) a review of EU research policy, ERA initiatives and framework programmes to look at macro, mezzo and micro level which impacts from SSH are achieved (or expected) on the promotion of ERA. Besides, 5) a total of 23 case studies on SSH research FP projects were conducted: 14 top success stories and 8 which did not achieve the expected success. This has allowed identifying factors and strategies to promote the achievement of all types of impacts at the short, medium and long term basis. Finally, all this work and the collaboration with funding agencies have resulted in the definition of procedures and indicators for ex-ante, in-itinere and ex-post evaluation of SSH research projects regarding social, political and scientific impacts and impact on strengthening the ERA.

The IMPACT-EV consortium launched the Social Impact Open Repository (SIOR), a pioneering data source (open access) at international level in which researchers display, cite and store evidences of the social impact of their research. SIOR is already connected to ORCID and WIKIPEDIA and it is highly instrumental to funders and research institutions.

The consortium has already published 16 articles in JCR journals (i.e. Nature, Research Evaluation, Scientometrics). Beyond scientific impact, the project has already informed assessment procedures of funding agencies, the design of social and policy impact assessment in the Framework Programmes, and initiatives from several universities for promoting social impact evaluation or recognition. Finally, it has highly contributed to a shift towards a much more positive attitude among researchers to social impact assessment, by supporting them in this endeavour by making available tools and strategies to enhance their research impacts, as well as larger changes in raising awareness about the importance of open access initiatives (e.g. ORCID, Wikipedia) and empowering citizens.

Among the multiple dissemination activities with a handful of stakeholders, important to highlight is the conference Social Impact of Sciences (SIS2016) with three Nobel prizes and some of the most important funding agencies worldwide; and the final IMPACT-EV Conference at the European Parliament headquarters in Brussels.

Project Context and Objectives:
An increasing concern for worlwide research funding organizations is the evaulation of the impact, especially beyond the scientific impact. In the case of Social Sciences and Humanities, there is also an important debate on their contributions to society. In fact, the perception of poor impact of EU funded SSH research have been an object of concern for the European Commission, and even the need for specific support to SSH has been questioned.

In this context, the main objective of IMPACT-EV is to develop a permanent system of selection, monitoring, evaluation and comparison of the impact and outcomes from European SSH research, taking into account the latest quantitative and qualitative evaluation tools, identifying new ways of implementing them and exploring new standards and indicators that complement existing impact assessment processes. Specifically, IMPACT-EV addresses the following sub-objectives:
1) To review the existing scientific knowledge concerning research evaluation tools and the changing structure of scientific, policy and social impact of SSH research as well as in relation to the development of the ERA.

2) To compare European SSH national research evaluation systems, as well as other relevant systems developed in leading countries and organizations in order to identify the systems which better contribute to the EU2020 and all EU main targets. This comparative analysis will focus on how these systems measure scientific impact and assess social and political impact.
3) To conduct an impact assessment of FP6 (last call) and FP7 SSH research in the three highlighted impact areas (scientific, policy and social impact) and to develop a map of SSH research impact in the EU, in which Success Stories are identified as well as those that have not achieved the expected success.
4) To analyse strategies that promote the achievement of high quality outcomes and impact in terms of the scientific production accomplished by SSH research as shown in the quality of publications, training of young researchers, forms of interdisciplinarity and transdisciplinarity and the constitution of European scientific excellence networks, among others.
5) To explore in which ways the impact of SSH research on European, national, regional and local policies can be assessed, and therefore upgraded.
6) To analyse the social impact of SSH research and the factors that have contributed to obtain or not this impact in order to create indicators to identify and evaluate the social impact of the SSH research ex-ante and ex-post. Especially relevant for this elaboration are the results obtained from the analysis of the Top Success Stories in social impact from SSH research.
7) To examine a) in which ways SSH research projects have contributed to develop the European Research Area in SSH and b) in which ways the various ERA initiatives at EU level have had an impact on the design and development of SSH research.
8) To integrate the knowledge produced in IMPACT-EV and develop a system of selecting, monitoring and evaluating SSH research which includes comprehensive indicators of scientific impact and the procedures to identify Success Stories on policy and social impact.
9) To explore and propose, based on previous experience, potential indicators of policy and social impact of SSH research projects, which can complement existing qualitative assessment methods. The focus will be on the projects which totally or partially address at least one of the EU main targets.
10) To define appropriate management procedures which secure the scientific and technical dimension of the project’s implementation.

11) To disseminate the project results and outputs among the scientific community, policymakers and stakeholders in order to increase its effectiveness and maximize its impacts.

Project Results:
The results of the IMPACT-EV project correspond to the specific objectives and research tasks that have been undertaken.

1. State of the art in the scientific, policy and social impact of SSH research and its evaluation.

The first project result was a review of the existing scientific literature on research evaluation procedures and the changing structure of scientific, social and political impacts of SSH research. This state of the art was conducted by means of an extensive literature review. Information was retrieved from journals with the highest impact factor and indexed in databases of the various fields and disciplines (i.e. WoS, SCOPUS), the most relevant books and research reports, and other grey literature from relevant evaluation institutions. Furthermore, scientific discussions, contributions, approaches and new tools have been taken into account. The main conclusions for each type of impact are as follows.

Scientific impact. The literature on scientific impact is dominated by interest in bibliometric analyses, indicators and tools focusing on publications and other research outputs (mainly patents). In the mid-80s, changes in scholarly communication emerged, and practices informed by open access principles gained more attention from the epistemic communities and from research institutions, new indicators were developed, based on the web (webindicators, webometrics, cybermetrics, altmetrics), as well as new approaches in scientometrics (e.g. h-Index and g-index). The most recent tools have the potential for measuring the impact and outreach of articles, which allow benchmarking their impact not only with citations, but also tracking the different uses of papers (cut/paste activities, citations in media reports, online newspapers, peer review discussions, blogs, etc.).

Generally speaking, changes in scholarly communication are likely to transform and improve our capability to understand the scientific impact of research outputs, going beyond the simple paper publication. The diffusion of open access practices are supposed to further reinforce the capability to monitor scientific impact.
As for SSH, despite efforts to use WoS data and scientometric techniques, and the transformation of some disciplinary fields, the analysis of the literature found many shortcomings with the methods and solutions proposed, which largely agree that assessments based on bibliometric resources generally underestimate the value of the SSH research outputs. Alternative metrics, methods and data sources have been explored in order to understand their potential for scientific impact assessment; negative consequences for the quality of research due to the extensive use of bibliometrics concerns scholars and countries where qualitative research traditions are prevalent. All in all, bibliometrics is largely considered no more than one resource among many others for scientific impact assessment, which can provide better results when used in combination with other metrics and information sources than when it is used as sole tool. This evidence raises the problem of choosing the most suitable mixed method for impact assessment.

Political impact. The assessment of the political impact of research is an important topic, which gained a momentum in Europe, especially in investigating the relationship between science and policy, and how to improve the impact of the results of research on the policy process. Its specific features relate to the fact that it deals with transformations produced in policy development and the policy process (motivations and rationales, policy design, policy implementation, policy assessment).

Participation and public engagement of researchers and stakeholders in policy making is considered one key element to strengthening the impact of research on decision-making; also the co-production of research between academics and policy makers has been assessed as a promising practice, which is likely to lead to greater political impact.

These results also apply in the case of SSH research; moreover, plenty of evidence emerged in the literature on the contribution and impact of SSH research on policies related to social problems like exclusion, gender discrimination, and other relevant social challenges. To this end, the contribution of the Science and/in Society actions, funding research projects dealing with governance and public engagement, have been of crucial importance.
Social impact. The attention that social impact has received is increasing in recent times. Research efforts have overcome the deterministic linear model that foresee the impact as a natural destination of all research results soon after they are delivered; they also support a more precise conceptual distinction of impact from other activities, like dissemination or knowledge transfer, etc.

Examples of SSH research producing social impact include areas that are key components of the EU2020 strategy (e.g. employment, education, social exclusion, poverty, researchers’ career and mobility, knowledge-based society). There are several ways by which SSH research with impact has been identified. For example, when researchers generate interventions based on research findings and provide evidence on its social improvements, or when researchers identify actions that are having a positive impact on society and analyse their features to create possibilities for transferability.

Strengthening of the ERA. SSH research also contributes to the structure of the ERA, with a view to strengthening the integration of research activities, projects, programmes and government agendas. This contribution is documented by several pieces of literature discussing changes in the modes of knowledge production, and the new relevance of transnational and transdisciplinary approaches in research projects, and collaboration in research outputs (from sole-authored to co-authored research, from disciplinary oriented to interdisciplinary oriented research).
Change has also affected SSH research evaluation systems at the transnational level and the national level, to overcome inefficiencies and limitations in understanding the type of impacts that research outputs actually have achieved.

2. Comparative policy analysis of SSH national Evaluation frameworks

IMPACT-EV compared European SSH national research evaluation systems, as well as other relevant systems developed in leading countries, particularly focusing on how these systems measure scientific impact and assess social and political impact, based on an exhaustive literature review. Particularly, 12 national research evaluation systems were analysed. Nine of them are European and have different profiles in terms of their experience and current approach in research assessment: Finland, The Netherlands, UK, France, Germany, Ireland, Sweden, Hungary, and Spain. The other three are non-European countries with significant practices to be considered: USA, Australia, and Brazil.

In terms of data collection, for every case, desk research was done including relevant legislation, research reports and other documents from evaluation institutions. Moreover, the IMPACT-EV consortium conducted a total of 96 interviews with major actors involved, referring to policy makers, researchers and representatives from the main agencies and research institutions. Data analysis was conducted first per country and second, across country comparison among the different cases was performed. Overall issues and debates, commonalities and main differences were identified.

The analyses revealed that structure (more or less centralized) of the research system do not determine the way research impact assessment is considered. In general, there are multiple connections among the major actors involved at the level of planning, funding and conducting research, thus, not necessarily corresponding to a particular way to approach research assessment. Evaluation is embedded in all these bodies and, in several cases, in the creation of specific agencies in charge of it. Obviously, there is a relationship between % of GDP investment, accountability of research funding agencies, and impact assessment maturity. However, this relationship is not linear and do not apply to all the national scenarios covered here. Research policies and legislations are increasingly aligned with the societal challenges (as Horizon 2020, which has been also informed from national developments) and SSH is concerned to avoid acting as an “ivory tower”, and make decisive contributions to the main goals and aspirations of our societies. The humanities do also claim for their specific engagement with citizens problems.
In general terms, assessment of research projects is more widely spread in ex-ante stages. Ex-post evaluations are less frequent and usually cover programmes or applied to specific scientific disciplines. In regards to the tools that funding agencies and other involved bodies do use, peer review system appears as the main technique (and similar modalities such as expert panels). Despite some controversy about the suitability and existing limitations of peer review for all kinds and scope of evaluations, this procedure is considered as crucial in many cases, being in most cases the only one research assessment procedure used. At the same time, criteria and measures to support the peer review processes, as well as to increase its rigour and quality, are being developed worldwide. Some countries do include end users and other stake holders in this processes, a trend worth to be highlighted here.

Related to this, a contested debate has been growing on recent years about the role of metrics and quantitative measures. Open consultations in some of the countries show the arguments for claiming or rejecting a stronger role of metrics in research assessment. The debate is, however, especially relevant in regards to social impact assessment. As a more novel issue at the international arena, different models are in discussion. We have encountered that the difficulties of showing and demonstrating the social impact do coincide with those highlighted in the literature. For instance, the time needed for achieving actual effects on the society (longer than for having tangible outcomes) or the problem of attribution. The “bottom up” model of Research Excellence Framework (REF) of the United Kingdom, in which the researchers self-report the social impact of their work providing a narrative and supporting evidences, is a reference in many of the debates. While being acknowledged as an innovative system for impact assessment, its narrative approach is seen (in some cases) as difficult and expensive to be extended to all research. Many of the interviewed share their concern for having appropriate measures and criteria to improve the assessment of social and political impact, including measurable indicators.

Finally, this work opened up the opportunity for policy makers, research assessment agencies and other stakeholders involved to consider and rethink their own work from the social impact perspective, and with many of whom new venues for further collaborations have been opened up for the future. In general, it is necessary to remark the positive response received to contribute to the project, and how the IMPACT-EV project is already present in the main ongoing debates on social impact assessment that are taking place all around the world.

3. Impact evaluation of FP6 (last call) and FP7 SSH research projects

An impact assessment of a total of 439 SSH projects funded under FP6 (last call) and FP7 SSH in four domains: scientific, political, social and the promotion of the European Research Area. IMPACT-EV has been particularly interested in exploring the social impact that the research projects have obtained in relation to the EU2020 targets (such as increased employment, reducing school dropout or reducing poverty rates) or the objectives of the Lisbon strategy. Finally, impact on strengthening ERA includes aspects such as mobility, training young researchers, interdisciplinarity, and international collaboration, among other related items. As a result of these analysis, success stories of projects that achieved impact in some of these four domains (scientific, political, social, ERA) and unsuccessful stories of projects that did not achieve these impacts were identified and analysed, and general conclusions were obtained regarding each type of impact. The case studies served to identify strategies and barriers for the achievement of each impact and are outlined in the next sections. The overall conclusions are presented as follows.
Against what is often believed, IMPACT-EV project has identified European funded projects which results have been used to inform policies or actions that have in turn contributed to achieve the politically established societal goals (i.e. Lisbon strategy, EU2020 targets, Agenda for Jobs, Growth, Fairness and Democratic Change). Besides, plenty of particular examples of how the EC funded projects have had scientific, political and social impact.

We observed that FP7 projects are achieving significant scientific impact in different ways and they show a positive progress in this regard. However, there is still some space for improvement an ex-ante evaluation could be improved in order to better select the groups, as it is noticed in the publication records (articles in JCR journals) that there are groups without this scientific impact. Among the reviewed projects, we observe that there is a significant achievement of political impact. There are projects which findings have served as the basis for policy-makers or civil society to develop actions, programs and interventions; thus, generating what has been defined as political impact. Policy briefs and policy recommendations are the most common way to connect scientific findings with political recommendations, however, an increasing diversification of strategies have been unfolded among FP7 project to pursue greater political impact. Based on our analysis, it can be concluded that there is still space for greater improvements. While examples of projects that have directly informed EU legislation and policy, as well as OECD, national, regional and local one have been reported here, there are others who are not achieving political impact. This might be due to the fact that these latter could achieve so later on, as they are still ongoing or have just been finished.

A growing concern about SSH research social impact has been recently raised, being a crucial and ongoing debate within the scientific community. Our analysis here captures this current debate, and in most part of the analysed projects, researchers widely recognize the importance of social impact, and even those who until now had never thought about this issue they are now starting to consider so. There are projects which has a significant social impact. However, there are many other projects which do not have social impact or which researchers do not know whether they have it or not. In some cases, the IMPACT-EV questionnaire has opened up the opportunity for these researchers to consider and rethink their own work from the social impact perspective, and with many of whom we have collaborated with in order to analyse their research social impact. Other main researchers, even in cases where the projects have already finished more than two years ago, are contacting with their consortium in order to explore whether their project has generated social impact or not. In general, it is necessary to remark the positive response received by the scientific community when assessing SSH research social impact. Again, a difficulty also faced for evaluating the social impact is that it takes longer to occur, posing additional barriers for ex-post evaluation to fully capture it. This is the case of the present evaluation in which an important amount of projects are still
ongoing or recently finished, and it might be too soon to identify social impact achieved.

Some FP7 SSH projects have also contributed to the achievement of the European Research Area (ERA). It is important to highlight that FP7 has made possible for all EU member states to participate. FP7 SSH projects have succeeded in involving excellent researchers in their consortiums. Not only in terms of the scientific productivity average but also because of the multiple awards (i.e. Nobel Prize). Participation is showed to be more even from the gender perspective. A slight increase in the female participation is reported in comparison to previous programmes. It is also important to note that sexual orientation starts to be considered, especially among those projects focused on interrelated areas. Funded projects derivate to further research proposals that tend to generate employment opportunities for priory hired researchers, involving an upgrading of their junior researchers careers.

Besides, identifying the different impacts, an important aspect to highlight is the transformation that is taking place within the scientific community. Particularly, in the process of data collection, we have identified many researchers whose projects have had or potentially might have social impact but they did not gather these types of information before. Their contact with the IMPACT-EV projects makes them realize about the relevance and the importance in doing so, and they have committed to start doing so. On the other hand, those researchers whose work has had very limited social impact, we have witnessed many examples of how they started to reconsider the way they approach and design their works.

The analysis of IMPACT-EV has shown that there is not one indicator or quantitative measure to cover impact of various kinds. Characteristics for our approach is an intense, time consuming, but also informative, engaging and to some extent even intervening personal contact to PI´s and main researchers, study of documentation, and usual qualitative methods. The later can help to structure observations, but not replace them. This unique combination of different methods of project analysis (interviews, questionnaires, baseline statistics) give weight to the reported observations.

These analyses showed that, even though the ex-ante evaluation has been progressively enhanced, there are some steps that can be done in this regard. First, a clarification should be made in differentiating between scientific, political and social impact. A suggestion can be made in order to require all the project proposals to clearly state this distinction. Project proposals should nail down the types of impacts they expect to obtain and to include in which way they will be measured, and assessed. Second, in the same way researchers are required to include their record in scientific publications and presentations, they could also be required to include their previous achievements in terms of pursuing political and social impact, as part of the description of their resume. This resonates to different initiatives which are already selected success stories among projects which have achieved significant social impact, for instance, the recent European Commission selection, or the expressed interest of ORCID in including this type of information into their database of researchers’ production by linking with SIOR, the newly created Social Impact Open Repository (IMPACT-EV).

4. Comparative analysis of national SSH funded projects and SSH FP funded projects

Besides European funded research, the different impacts of the projects funded by national R&D calls in Europe were analysed, drawing from the particular results of the analysis of three case studies: Spain, Sweden and the UK. A comparison of the nationally funded research projects and the FP projects found show that the national projects are in fact achieving impact in the social, political and scientific domains. In an overall consideration, it can be said that the results are very similar to those obtained in the analysis of EU funded projects.

However, the information that we obtained in each case study was limited by the unbalance limits the scope of the conclusions that can be drawn from the analysis. In this sense, it is important to point out that there is, with a great difference, more information available for the national case of the UK than that for Spain and, to a lesser extent, Sweden. In the UK, there is also a far longer tradition of impact evaluation, and the information of the impact of research is systematically collected by the researchers themselves, for this reason even many projects from the field of humanities are found to have impact. This showcases that having a research evaluation system that takes into account the impact of projects is relevant in order to achieving impact as well as to commit to the task of collecting data regarding it. This debate is in the line of the conclusions obtained in Report 3 on the transformation occurred among the researchers during the data collection process, which enhanced their ability of identifying the different impacts when they became aware of what they consist of and their importance, and they have the opportunity to show them. With the analysis of nationally funded projects we saw a similar transformation, in this case related to the existence of research evaluation systems that takes
into account these impacts.

The nationally funded projects studied all have smaller budgets, lower number of researchers and in many cases shorter duration than the European projects funded by the Framework Programme. The evidences show that the SSH research groups are achieving significant impacts also when they are funded under national research calls. In this regard, there are national projects that have impacted on national policies or have influenced the political debate. Yet, there is not available evidence that many other projects’ results have reached the political domain. Therefore, similarly to what we found in the analysis of EU funded projects there is still room for improvement. The same can be said as regards social impact. While some projects have achieved clear social impact, in other cases there is no evidence about it. The same as occurred with the EU funded projects, it might be due to projects not having impact but also to researchers not being aware of whether they had or not, and to the fact that social impact may take longer to occur than other impacts.

One of the differences between national and European projects is the scope of the impact. While the political impact of nationally funded projects tends to be focused at the national level, EU funded projects have a broader scope and specially have an impact at the European and international level. A similar difference is observed as regards the social impact; EU funded projects respond more frequently to European social targets (EU2020 or Lisbon 2010) than the nationally funded projects. This is related to the existence or lack of research priorities in the different research programmes. While the EU Framework programmes stablish priorities linked with the EU objectives, in the case of the different countries the situation is diverse. In this sense, EU funded research entails an added value as compared to nationally funded one, as European research reaches impacts that cannot be achieved by national research to the same extent.

In this regard, we have observed that national and European projects are frequently connected. In many of the cases studied researchers have a trajectory in European or international research and/or have links with European or international projects. Some projects are the continuation of European funded projects or have subsequently led to them. Therefore, research at both levels is contributing to building new knowledge that responds to challenges relevant both for Europe and for specific countries generating positive synergies in both directions.

5. Scientific impact of European Founded SSH Research

A system of indicators that can be expected to represent the scientific impact of European funded SSH projects (under the 6th and 7th Framework Programme) was outlined and tested. Adhering to the principle that scholarly impact is primarily expressed through scholarly communication, the proposed system is based on analysing the publication output of FP projects. In order to set up a feasible indicator system, our strategy involved a conceptual exploration of the possible aspects of project-related scientific impact, in order to identify relevant, potentially latent dimensions of the academic influence (or effect) of SSH projects that are (1) quantifiable or can be formalized into well-defined measurements and (2) do not depend on quantities and size of output. This latter consideration was aimed at circumventing the coverage issues of SSH outcomes in most available databases. As a consequence, the proposed measures (mostly) fall into the category called size-independent measures in bibliometrics, known for a better characterization of impact or quality, than size-dependent indicators. Detection of impact-related quantities were, generally speaking, substituted by detection of impact-related patterns via various channels of scholarly communication (extending the use of bibliometrics), that is, diagnostics of the structural effects of projects. Examples of such impact dimensions reveal the capacity for attracting academic actors outside the project or the scope of knowledge diffusion (interdisciplinary impact).

In order to explore the practical value of the system of indicators proposed for SSH project assessment, we have undertaken an empirical study of this toolkit. This task involved the analysis of a large-scale sample of FP6 and FP7 projects. In order to assist the empirical investigation of the indicator system, a large-scale database on FP6 and FP7 SSH projects was delivered, containing the necessary interlinked datasets of projects and their research outcomes. This database was built in a comprised of two main modules: (1) the first module contained project characteristics and project outputs available through formal channels of scientific communication (Web of Science, other publication databases). (2) the second module covered the outcomes of projects detectable through alternative web-based channels, primarily the scholarly social media and social networks. In particular they have identified 419 projects for the further analysis. Of them, 166 were FP6 (SOCIETY) and 253 were FP7 projects (SSH). Finally, all data was placed in the final database that links calls, projects, publications and citations together with pieces of additional information from WoS (institution, address, authors, keywords, subject categories, etc.), when these are available. The second module was built to link projects with outcomes in web content, by i) Crawling for Altmetrics data, ii) Building an Altmetrics database, iii) Identifying and books and book chapters. 419 projects identified in Part I were followed to obtain Altmetrics data for them using various data sources including social media. Based on the project database delivered, indicators were studied and tested in relation to two set of research questions: (RQ1) Given the pool of European funded SSH projects, do these indicators signal different impact dimensions, as it was assumed in our proposal? If so, do empirical results support our taxonomy, i.e. outline similar dimensions? (RQ2) Can the pool of European funded SSH projects can be characterized along these dimensions and indicators? Are there recognizable „impact profiles”, that is, groups of projects sharing a similar composition of indicator values? Do impact profiles correlate with project type, funding scheme or other formal attributes of SSH projects?
The two sets of research questions were addressed by different analytic strategies, including (1) a principal component analysis on indicators and (2) a cluster analysis of projects, respectively. Results and findings confirm the taxonomy and parallel use of proposed indicators as well the existence of various impact profiles among FP6 and FP7 projects.

Besides the elaboration and comprehensive quantitative analysis of the metrics, a qualitative analysis was also conducted among successful and unsuccessful EU funded SSH research projects to reveal the expression and factors of scientific impact. The main goal of the case studies was to analyse in depth the scientific success achieved by the selected projects (3 success projects and 1 non-success project) and the key elements that led them to success or non-success in terms scientific knowledge production through publications, attendance to conferences of national and international scope or through other scientific activities. Scientific impact not only at the European but also at the national level was taken into account. According to the findings of the case studies both in case of successful and non-successful stories the following dimensions proved to be most important and effective in achieving high scientific impact:
• experienced and effective leadership
• committed consortium
• orientation to achieve at high standards
• number of national level top scientists among the participants
• clear concept
• effective internal communication
• effective dissemination activities.

6. How does policy take SSH into account? Analysis of the fieldwork on success stories on Policy Impact

In the analysis of political impact, three types of political impact have been considered. The first one is the policies and the transfer of research outcomes into policies adopted by governments and supranational institutions. Political impact is considered to occur if there are evidences of a relationship between the research project and policies. In the case of Policies at the European level, they include Regulations, Directives, Decisions, Recommendations, Resolutions, Opinions and Conclusions adopted by the European Parliament, the Council of the European Union or the European Commission. Resolutions, declarations, treaties, protocols, global plans of actions, conventions, agreements, conferences and summits, consultative opinions, etc. adopted or promoted by the United Nations, as well as agreements, guidelines, recommendations and declarations adopted by OECD are also included in this category.
Second, programmes implemented by civil society organizations (NGOs, companies, unions, trusts, professional bodies and other stakeholders) that have used the findings of the research are considered. Political impact is considered if there are evidences of the relationship between the research project and the policy. Finally, political forums in which the findings of research have been presented and discussed with policy makers and/or other relevant stakeholders are also taken into consideration.

For the analysis of the political impact of research projects, three activities were conducted: first, 40 interviews with researchers and policy makers; second, a comparison of four success projects and one non-success project, based on case studies for each project, which were prepared with desk-research and interviews conducted with project partners and stakeholders; third, a Delphi Panel with 15 participants including researchers, end-users and policy-makers.
The results of the 40 interviews reveal that the most important factors related to policy impact are certain processes and channels. Inclusion of the right people in the project is a strong factor. There is a need to involve all the necessary stakeholders from the beginning. Another important point is the need for mediators and that people who agree to be mediators should be given incentives in order to enhance the discussion between the scientists and policy makers. However, the interview results also provide evidence to problems related to communication strategies including language barriers between the policy makers and scientists, the lack of interdisciplinary studies in SSH, lack of information flow as well as the inadequate institutional setup of EU. The main issues connected to the usage of social media is that there is no proof yet to state that it is efficient as well as privacy issues and internet laws. Regarding monitoring and evaluation, the interviewees agree that they should be increased if it is done in the correct way which provides useful results and cross analysis. For this, sufficient financial resources should be provided.

The comparison of four success cases and one non-success case represents the common strategies used to achieve political impact in EC funded projects along with the barriers they have encountered. The main strategies used in order to reach political impact are dissemination and promoting the debate in conferences, having partners with strong relationships with the policymakers and intention of achieving political impact from the beginning, conducting focus group before the project initiated and involving stakeholders at different levels and relevance of the project topic for policy-makers together with producing publication targeting policy-makers. Yet, all the project partners indicated that they have had barriers to achieve political impact. Most of their actions involve strategies necessary to achieve political impact and to overcome these foreseen barriers. These obstacles included involvement of the key people such as policy makers, dissemination of the results to the right audience with the right language and harmonization of tools and data bases to ease the mass usage to achieve political impact.

The Delphi panel was aimed at reaching a consensus on the relevant indicators to measure political impact. These indicators are divided to three phases of the project: ex-ante, in-itinere and ex-post. In total, consensus was achieved on eight indicators which half of the indicators are ex-ante and the rest is divided for in-itinere and ex-post evaluation. The ex-ante indicators include previous experience of the main researchers in transferring scientific knowledge into policy, quality of project’s design of communication channels and strategy to reach out to policy makers, involvement of policy-makers and policy stakeholders in the preparation phase of the project and strong focus of the research project on identifying solutions for policy/social challenges. In-itinere indicators are: the level of information flow and interaction between the policy makers and the project and contribution to specific project tasks by policy-makers, stakeholders and end-users. Ex-post indicators involve references to the project in political documents, discourses and other sources made from policy-makers and the use of the results by international/national authorities, NGOs and enterprises. The final indicators of the Delphi Panel also corresponds with the successful strategies of the four success case studies as well as with the 40 interviews with researchers and policy-makers. For sure, there are barriers to achieve these indicators. All the projects do face these barriers however, the success of the projects is defined in their early decisions to foresee these barriers and agree to strategies to overcome these barriers within their consortiums.

7. How social impact of SSH research is made visible

An in-depth analysis of SSH research projects that either succeeded or failed to achieve social impact was conducted in order to identify the strategies and barriers that promote or hinder the achievement of social impact. In addition, we explored the extent to which open access publishing and the use of social media in the dissemination of scientific outcomes may influence the impact of research on the civil society.

Drawing from an ex-post evaluation of the projects of the 7th Framework Programme (which was developed in a previous stage) a total of 10 projects were revised in depth: 5 were identified as successful in achieving social impact, and 5 were considered not successful in achieving this type of impact. The development of these case studies supposed an effort in terms of collecting new evidences and measuring the specific impacts for each case, as well as to understanding in more detail how these projects made in order to make (or not) a relevant contribution to society.

In order to conduct the study IMPACT-EV researchers consulted documents from the researched projects and did qualitative fieldwork with as many agents as possible, including researchers, stakeholders and end-users. In this sense, more than 90 interviews have been carried out.
Among the social impacts that have been identified, we highlight employment creation, which has been promoted in different fields depending on the project –such as offices against corruption, or in the tourist sector. Furthermore, impacts achieved are as diverse as the projects are. For instance, raising academic achievement in the field of education, or increasing citizenship engagement in anti-corruption fight.

On the other hand, our exploration confirms that even in the cases in which researchers have clearly achieved to improve living conditions and social results related to main social challenges, they do not have always the evidences that allow to explain and measure such contributions or found it difficult. In that respect, it is noteworthy the difficulties for accessing stakeholders and specially end-users through the main researchers, both in the top success cases and in those that did not achieve impacts. This difficulty illustrates the distance between SSH researchers and society.

The common identified strategies to achieve social impact include a clear focus on social impact and to define an active strategy for achieving it; a real involvement of stakeholders and end-users throughout the project lifespan; counting with credibility and previous social impacts achieved in order to build new collaborations; improvements for the end-users already during the project; dissemination activities showing useful evidences; and dissemination activities oriented to create spaces of public deliberation with diverse publics.

In turn, the in-depth analysis of the different cases has allowed us to identify some of the main barriers and difficulties that hindered the projects from achieving greater impacts. The most common ones can be summarised as: lack of social impact purpose (emphasis on the scientific impact rather than promoting direct political and/or social effects) and lack of an appropriate strategy for achieving social impact; low involvement of stakeholders and end-users in the project; a misdirected dissemination; lack of appropriate skills for promoting impact among the research team; and the lack of culture of evaluating the social impact of research.
As it has happened in other stages of the IMPACT-EV project, the communicative approach used has produced transformative effects in the own perception of interviewees about their role as researchers for promoting social impact. The retrospective analysis about the focus of the project and the process has led in some cases to identify potential effects of which they were not aware before, and in other cases, to think about alternative strategies for increasing such impact in future
works.

We have emphasized the relationship between dissemination of SSH results beyond the academy and the achievement or not of social results. For doing this we have analyzed in detail the presence of project outcomes of the selected case studies in social media and in Open Access channels. A first conclusion is that some projects (especially the most successful ones) do have a reasonable presence in websites and social media, even though there is still ample room for improvement. In regards to Open Access publications we have found relevant differences among the projects, also between successful projects. In general, there are greater efforts on making reports and other outcomes available (such as policy briefs) rather than publishing in OA journals. This is expected to change in Horizon 2020 due to the explicit requirement of Open Access publishing.

The interviews with relevant figures involved in Open Access movements highlight the opportunities of OA for reaching wider audiences and stakeholders, transparency and democratization of knowledge. At any case, Open Access and dissemination in social media or other alternative channels that reach society beyond the academy, are more a precondition than a guarantee of social impact. Another conclusion in this sense is the need to make the information about the social impact of a research –and not just its scientific impact– both available and discoverable to the citizenship.

The Social Impact Open Repository (SIOR), launched under the IMPACT-EV project, is a pioneer initiative in this direction (http://sior.ub.edu). SIOR is an unprecedented data source (open access) at international level in which researchers from all disciplines display, cite and store the social impact of their research results. SIOR is contributing to make visible the ways research is achieving social impact and therefore is highly instrumental to funders and research institutions.
As we have seen, for a future and better evaluation of social impact, it will be needed to facilitate to researchers to gather and track a set of data which at the moment is not systematically compiled. Specifically, the following type of evidences could allow researchers to track the social impact of the projects: collaborations established with any type of stakeholder (policy-makers; members of nationals or internationals institutions; NGOs; people from the private sector; etc.); data about the use of the projects’ findings by the previously listed stakeholders or by other actors, as well as the effects of using these findings (to what extent a group of actors have benefited from action/ in which area/ etc.); presence of the project outcomes and impacts in the social media. These results have been important for the development of an integrated system of indicators of scientific, political and social impact of research in close collaboration with researchers from different SSH fields, funding agencies, policy makers and end users.

8. Monitoring and evaluation tools in promoting European Research Area

Our study reveals that the European research policy has had a significant effect on how scientific disciplines do research and on shaping research paradigms across disciplines. European research policy defines which areas/topics/societal challenges will be funded, the amount of budget to be allocated to the selected areas/topics/societal challenges, and how research should be conducted in terms of disciplinary approaches and involvement of stakeholders.

Attempts for linking directly individual SSH actions funded under specific programmes, initiatives, funding schemes, and individual Calls for proposals produce fragmented and inconclusive evidence of their impact for a broader picture of the European research landscape. Monitoring and evaluation of ERA can draw on case studies providing that these are integrated in a comprehensive approach.

The contribution of SSH to ERA cannot be realistically measured by numbers in those societal challenges in which the SSH cluster of disciplines has a significant role to play. In SSH triggers of change are not confused with causes of change just for the sake of arguing that particular discipline or cluster of disciplines are or are not relevant for society. The value of small effects of scientific disciplines should not be underestimated.

SSH research findings are conducive to a broad range of policy choices by the very nature of societal challenges they address. The policy-making culture at the national and European level would benefit from including clearer ethics, or at least be clear about the ideology underpinning the selective take up of European SSH research evidence.

Research of the ERA process, ERA programmes, ERA specific instruments, ERA initiatives, European standards for the selection and monitoring of individual actions to be funded and case studies form a comprehensive landscape for the elaboration of indicators and data that can be collected and pooled to set qualitative targets and measure achievements of actions. Minimum common rules and requirements for collection of data for documenting contribution to ERA are necessary.

The current perception of political correctness imposes quest for the simplification of funding rules, regulations, and procedures. It is not popular to be seen as asking to add responsibilities and burden projects, even if the added value for doing it could be significant. The use of taxpayers’ money for public funding of European research must surely come with some requirements that do not impinge academic freedom, but are somewhat broader than the obligation to justify the use of money in terms of time-sheets and paper deliverables. Simplification and responsibility need to go hand in hand.

Impact assessment to justify the use of public funds, the scope and quality of the knowledge-base for informing the policy making process, the potential for contributing to solving some of the identified key societal challenges and the potential for helping deal with unexpected and unforeseeable future reveal assessment phases each needed to be underpinned by robust instruments. Ex ante, in itinere and ex post assessments best serve their purpose when they are interlinked.

In view of the centrality of the research policy quest for interdisciplinary approaches in EC funded and co-funded actions, it would be highly appreciated both by the research funding agencies and researchers if some work would be done at EC level on the development of tools and methods for ex ante evaluation of interdisciplinarity.

The excellence of research portfolio of ERA is well served by the identification of scientific, policy, and social impacts that may be assessed at the level of individual actions and balanced against expected impacts specific to the funding scheme under which an action was supported.

The key indicators that are relevant for ERA in addition to the excellence of research portfolio relate to impacts for research careers, structuring effects for European research institutions, and scope and scale of use of research evidence. A set of data as key performance indicators (KPI) can be collected.

Analysis of the way individual actions proceeded in view of contributing to and reinforcing ERA show little if any specificity for the cluster of SSH disciplines. They include robust research; effective dissemination, communication and outreach; and exploitation of research results.

SSH share with other clusters of disciplines same tools for assessing and monitoring contributions to ERA in terms of research careers in Europe (preservation of research jobs, employability enhancement and job creation in research); and structuring effect for institutions (improving working conditions and broadening and internationalization of collaborations).

What is specific to SSH cluster of disciplines is that the societal impact is strongly mediated by the policy actors and the influence of ideology in the take-up of research evidence. Assessing the scope, scale and ethics of the take-up of European SSH research evidence is a pathway to assessing the European policy-making culture. National and European policy-making culture may be one of the key societal challenges to be addressed in a future European framework programme.

Research and training are ERA and EHEA components that for administrative reasons are splitted in different Directorate Generals and are funded under different specific programmes. There is a clear benefit of further reinforcing the human resource dimension of ERA through the powerful impact-generating programme “Marie Skłodowska-Curie”, enhancing the research mobility dimension of ERA and incorporating staff exchange via cross-border mobility. This could also be the case for research actions that have been selected for funding under the Societal Challenges programme. A common pot could be created in the EC Societal Challenges programme for funding cross-border staff mobility so that funded projects can ask for mobility grants when one or more partners deem short-term mobility useful, without having to go through a burdensome process of project proposal writing under another EC programme. It is also desirable to extend the use of high standards for the recruitment of early stage researchers to all EU funded actions in view of realizing a single labour market for researchers as an ERA goal.

9. System of selection, monitoring and evaluation of SSH research

Procedures for selection, monitoring and evaluation of SSH research have been defined, which are a result of the integrated work and results in IMPACT-EV, including an analysis of evaluation systems of SSH research projects across different countries in Europe and beyond, collaborations with national research funding agencies and universities, a Delphi panel with different stakeholders, including experts in science and policy makers, collaborations with research funding agencies, and different working groups composed of policy-makers, end-users, stakeholders and researchers. In addition to all the accumulated knowledge within the consortium, the coordinating team invited an expert to review all the work and outputs produced and provide an external insight for the definition of indicators.
These procedures have been defined for each stage of the evaluation process: selection (ex-ante evaluation), monitoring during the project lifespan (in-itinere evaluation) and evaluation once the project finished (ex-post evaluation), which can be of special relevance for research evaluation agencies to enhance the various impact of the research they fund at different stages of the research evaluation process, and have already been proposed to the national agencies for their implementation.

The ex-ante evaluation procedures are aimed at the selection of research proposals that are likely to have impact. Criteria for ex-ante evaluation of research impact can be included in the call that is published, in the templates that research teams have to fill in and in the guidelines for evaluators. In addition, the inclusion of such information related to impact in the call documents can make researchers more aware of the potential impacts of their research and encourage them to design more impact-oriented research projects.

The in-itinere evaluation procedures proposed are aimed at helping funding agencies and research teams to assess the progress of an ongoing project in achieving impact. These procedures can be included in the mid-term evaluations and reports and can be used to redress the project limitations in this regard before its finalisation.

Ex-post evaluation of research projects evaluates the impacts achieved once the project finalised. Impact needs time, therefore some of the impacts cannot be observed just after the project ends, and even some of them have not yet been achieved by the project end date. This is especially applicable to social and political impacts. For this reason, ex-post evaluation of research should include a follow-up of the projects for a period after the funding ends.
Besides the procedures defined for each stage, a list of specific impact indicators for each type of impact (scientific, political, social, ERA) and evaluation stage (ex-ante, in-itinere and ex-post evaluation) of research impact have been proposed.

The proposal has put special emphasis on the definition of indicators of social impact in regards with Europe 2020 targets. Different concepts around social impact have been defined to help assess the achievement of social impact:

Social Impact of Research is when the published and disseminated research results, which have been transferred, lead to an improvement in relation to the goals agreed in our societies (through our political representatives). For instance, the EU2020 Strategy (i.e. employment, R+D investment, climate change/energy, poverty/social exclusion and education) and the UN Sustainable Development Goals (SDG). Thus, Social Impact (SI) is the improvement of society and citizens in relation to their own goals (like the United Nations Sustainable Development Goals) and Social Impact of Research (SIR) is the SI achieved by the research. In consequence,

Social Impact Indicator (SII) refers to the set of indicators enabling the quantitative and qualitative evaluation of the SI achieved (the set of Social Impact Indicators includes subsets like the ones on economic impact, societal impact and so on).
Indicators of Social Impact of Research (ISIR) refers to the set of indicators enabling the quantitative and qualitative evaluation of the SI achieved due to the use of the results of the research.

Evidence of Social Impact of Research (ESIR) is the set of quantitative and qualitative evidence which proves the evaluation of the social impact of the research.
Another relevant concept here is Research Enabling Social Impact (RESI): the set of researches that have been necessary to finally obtain a concrete research that has specifically achieved social impact (although some of those researches have not individually obtained SI, they have been essential in order to make another research that has specifically achieved SI).

While scientific impact of research counts on a long tradition of development and wide academic discussion, social and political impacts are newer issues raising and increasing interest not only among the academia but also among the whole citizenship. In this context, IMPACT-EV has done a step beyond the discussion on how should be understood and measured social and political impacts, by placing particular emphasis on identifying the actual impacts achieved by projects. This is an important step in a particularly moment, as most funders and evaluation agencies are looking for solutions on how to capture these impacts.

Potential Impact:

The IMPACT-EV project has contributed to two main larger processes: an increasing generalized concern for the future development of Social Sciences and Humanities’ research, and the move towards assessing different impacts of research, besides scientific one. Amidst this context, IMPACT-EV has gone beyond providing a system and indicators for the analysis of research impact; and particularly, it has contributed to transform the way social sciences and humanities’ research are developed and presented to society.

The IMPACT-EV has already achieved scientific, political and social impacts, and have made relevant advancements that are expected to evolve to further impacts. Here we summarise these impacts and advancements.

Scientific impact

The IMPACT-EV project has produced 16 papers in indexed journals in Journal Citation Reports, as shown below:

JCR Q1 7 Nature; Research Evaluation; Scientometrics

JCR Q2 2 International journal of Environmental Research and Public Health

JCR Q3 1 Evidence & Policy (note)

JCR Q4 6 Orvosi Hetilap

Besides the JCR articles, 4 other papers have been published or accepted in peer-reviewed journals indexed in Web of Science or other databases. At the time of reporting, other articles are in process of review, edition or elaboration. Specifically, there are 7 articles submitted in Qualitative Inquiry (Q1). The consortium is also working on other potential articles with findings of the project.

Researchers of the consortium have presented the IMPACT-EV findings in more than 30 papers in national and international conferences, and they have given 59 keynote lectures in a wide range of international and national organizations.

On the other hand, the project has contributed to the career advancement of more than 10 early stage researchers. Three doctoral thesis have been completed to the research, plus two others in process of elaboration. Six post-doc researchers have been hired (by different teams of the consortium) expanding their possibilities for progressing in the scientific career.

Finally, the project has contributed to the advancement of knowledge with the collaboration with other projects. It has to be noted here the case of “ACCOMPLISSH. ACcelerate CO-creation by setting up a Multi-actor Platform for Impact from Social Sciences and Humanities” and “DANDELION–Promoting EU funded projects of inclusive, innovative and reflective societies”. The coordination team has maintained a continuous collaboration with NET4SOCIETY and has participated in the launch of the European Alliance for the Social Sciences and Humanities.

Political impact

Political impacts of the project can be summarised as (1) Influence in the design of social and policy impact assessment in the Framework Programmes; (2) Influence in assessment procedures of national Funding agencies; (3) Influence in Universities and other institutions.

1) Influence in the social and policy impact assessment of the Framework Programmes.

The IMPACT-EV consortium has been required by the European Commission to undertake two tasks that were not foreseen in the Project proposal. One was the ex post evaluation of projects funded by the 7th Framework Programme. The other one was a contribution to the in itinere evaluation of the Horizon 2020 Programme. The conclusions of these evaluations have been used by the European Commission in their official evaluation reports of the Framework programme (https://ec.europa.eu/research/evaluations/index.cfm?pg=fp7; https://ec.europa.eu/research/evaluations/pdf/archive/h2020_evaluations/swd(2017)220-in-depth-interim_evaluation-h2020.pdf).

The IMPACT-EV main researcher was elected Chair of the EC Expert Group on Evaluation Methodologies for the Interim and Ex-post Evaluations of Horizon 2020. This group was composed by 17 experts (selected by the EC among more than 300 candidates) from all scientific areas of H2020 (i.e. Health, Engineering, Space, Agriculture). The group was meant to develop new methodologies to identify and assess the social impact of the three pillars of Horizon 2020 as well as of past Framework Programmes. In 2018, a reduced group of experts has been settled to work specifically on the definition of Indicators for the assessment of four types of impacts: Scientific impact, Economic/Innovation Impact, Societal Impact and Policy Impact. In this group, Ramon Flecha is the appointed expert responsible for societal impact and policy impact. Based on the results of the IMPACT-EV project, his contributions will be highly relevant for the design of methodologies and ways to measure these impacts in future FPs.

(2) Influence in assessment procedures of national funding agencies

The collaboration with funding agencies has led to new advancements in the integration of impact assessment (especially social and political) at different stages of research evaluation and funding. As a result of the development of national case studies, several stakeholders and agencies expressed their interest in the results of the project and potential collaborations. This has been translated in contacts during the whole length of the project and new initiatives foreseen in 2018. Two particular cases are as follow:

Catalan Agency for Management of University and Research Grants in Catalonia (AGAUR). The Agency is a public funding body within the Secretariat of Universities and Research, Ministry of Enterprise and Knowledge of the Government of Catalonia. It annually manages over 30 grants programmes with an independent evaluation system through external experts. The influences that have been produced are: (1) The Agency incorporates a specific training on the impact of research to the 60 early stage researchers awarded by the Beatriu de Pinós programme (Postdoctoral grants for the professional development of research staff and their incorporation into the Catalan science and technology system). (2) The 2017 Grants to support the activities of the research groups in Catalonia (which is launched every 4 years) included as a clear issue for the assessment the social impact of the research group.

La Caixa Banking Foundation. The Foundation has among its lines of action the support to research and researchers. In particular, it supports a programme in social sciences and humanities called RecerCaixa. In the last calls of this programme, the social impact of projects has had an increasingly central role in the approach and in the assessment of the proposals. Agreements were reached for the piloting of indicators and criteria as follow: For 2016 Call, IMPACT-EV has contributed to the monitoring, in itinere evaluation and technical support for the selected projects. For 2017 Call, IMPACT-EV reviewed the Guidelines for ex-ante evaluation provided to the external evaluators and indicated criteria and indicators to be more specific.

3) Influence in Universities and other institutions

Different universities have launched initiatives related to social impact as a result of the Project.
For instance, University Rovira I Virgili (Tarragona, Spain) has promoted a “Social Impact Award” which has had two open calls up to now (2017 and 2018). The Call for proposals has two modalities: 1) Ex-ante social impact of research (forecast of social impact) and 2) Ex-post social impact of research (evidences on social impact). The Call is open to proposals from three main fields: Social Sciences and Humanities; Experimental sciences, engineering and architecture; Health and life sciences.
On the other hand, there have been also efforts to increase awareness and engagement of Universities towards the impacts of research and benefits of learning mobility in the context of the European Research Area (ERA). The most visible impact has been achieved in Italy where Universities such as LUMSA in Rome, Università di Padova, and Sapienza Università di Roma are particularly engaged in enhancing learning mobility of their Doctoral and post-Doctoral fellows.
A number of universities and other institutions (including evaluation agencies and policy-makers) beyond directly involved countries have also shown their interest and debate on the issues of IMPACT. For instance, PSPC has promoted enhanced engagement in the Western Balkans in deliberations about social and political impact. This transfer of knowledge may contribute to alignment of efforts in this region with practices in the Member States that are forerunners in valorisation and evaluation of benefits of science for society.

In the same vein, a result of the dissemination activities, Sari Hanafi (professor at the American University of Beirut and Vice President of the Int. Sociological Association) has expressed their interested in the Social Impact Open Repository and is taking the lead for launching the initiative in the arabic countries. The initiative is expected to be developed in 2018 with two stages: a first portal to gather data and projects, and a second stage, for piloting a specific-repository software.

Social impact

Social impact of the project can be summarised as (1) Key changes in the researchers’ attitude on social impact assessment, (2) Increased awareness among open access initiatives of the importance of considering social impact, (3) Increased awareness among citizens of the importance of social impact of research.

(1) Key changes in the researchers’ attitude on the social impact assessment

Along the four years of the IMPACT-EV Project, the consortium has been in contact with hundreds of researchers, most of them from SSH fields. The evaluation of 452 finalised projects (from FP6 and FP7) as well as those on-going (Horizon2020), plus the 23 case studies conducted and the dissemination and networking activities have meant the opportunity of not only getting information but also of establishing fruitful dialogues and exchanges.

In several cases, the IMPACT-EV initiatives have opened up the opportunity for researchers to reflect on their own work from the perspective of the social and political impact. We have diverse examples on how this happened: there have been main researchers who contacted their former consortiums even if they had finished their projects years before, asking for the impacts that they achieved after the end of the project. In some cases, they gathered new information and awareness of such impacts. Other researchers have been very interested in having new information about how to improve the impact of their research and how to gather evidences of the impacts achieved.

More than 1000 researchers participated in the scientific events that have been organized (Social Impact of Science conference; Evidence for Society; Open Seminars; Country Seminars, Webinars, etc).

17 research teams have introduced their social impact on the SIOR repository.

More than 80 researchers have participated in tutorials about how to introduce the social impact on SIOR.

More than 50 research teams have contacted IMPACT-EV team to receive more information or advise.

In overall, IMPACT-EV has opened the floor for a public debate on the social impact of research, which has gone beyond SSH. The shift on the researchers’ attitude contributes in turn to the actual enhancement of the social impact of research; as researchers are more aware of the importance to orientate their projects towards the achievement of social improvements (e.g. reducing poverty, etc.) and are more aware of the strategies to enhance the impact of their research, the possibilities of increasing the social impact of their research increases.

(2) Increased awareness of the importance of taking into account social impact among open access initiatives

The launch of the Social Impact Open Repository (SIOR) has been one of the main outcomes of the IMPACT-EV project. Besides of the creation of the repository, IMPACT-EV has established cooperation with other organizations and initiatives related to open access and research transparency, advancing on the awareness of the need for involving appropriate tools for viewing and valuing social impacts. It should be highlighted that ORCID has showed a strong interest in advancing towards the interoperability between both platforms to make visible the impact of researchers’ work. The research team also reached an agreement with representatives of Wikipedia about the use of SIOR in their platform. In brief, SIOR provides evidence to information that is included in Wikipedia, thus verifying the declared social impact in the platform.

(3) Increased awareness of the importance of social impact of research among citizen

IMPACT-EV has opened the debate on the social impact of sciences not only among researchers and research founders but also with the citizenry in general. Several spaces created within the project, including the working groups with policy-makers, end-users, stakeholders, the open seminars with international speakers and integrative panels gathering both experts, researchers and citizens discussing on the impacts of SSH, and the Final IMPACT-EV conference Evidence for Society (see dissemination section) have allowed such debates. In these debates participated entities such as educative and cultural associations and networks, associations serving vulnerable groups, cooperatives, scientific societies, research centers and groups, educational centers and agencies, governmental units and observatories.

Also, the SIOR workshops have been conducted with citizens interested in the topic. Particularly, they are training sessions oriented to show how to use SIOR to identify in which ways research is achieving social impact.

Main dissemination activities

The main dissemination activities have been (1) Conference Social Impact of Sciences. SIS 2016 (Barcelona, 2016); (2) Evidence for Society (European Parliament, Brussels, 2017); (3) Two Open Seminars with international speakers (Barcelona 2015 and Barcelona 2016); (4) Other national seminars and webinars; (5) Scientific publications; (6) Webpage and social media; (7) “How to enhance the impact of SSH research“ Guide for researchers”; (8) Policy briefs and Newsletters.

In what follow, we briefly describe each of these activities.

(1) Conference Social Impact of Sciences. SIS 2016 (Barcelona, 2016)

Due to the interest for the social impact of science all over the world, the coordination team of IMPACT-EV decided to organize an international conference for discussing successful practices of projects with social impact from all the scientific fields. The Conference was held in Barcelona, July 2nd -29th 2016 and brought together researchers and stakeholders from all the world, including:

Three Nobel Prizes: François Englert (Nobel Prize in Physics, 2013), Ada Yonath (Nobel Prize in Chemistry, 2009), and Harald zur Hausen (Nobel Prize in Medicine, 2008) who gave keynote lectures on the relevance of social impact in each of their research, world's leading agencies in Science (Nature, Thomson Reuters, PLOS, ORCID, National Science Foundation, etc.) and around 400 researchers (including college students and PhD students, scholars, full professors) from the five continents.
The SIS2016 was organized around social impact in five scientific fields: Physics & Mathematics, Social Sciences & Humanities, Engineering, Information & Technology, Life Sciences, Health & Pharmacy. All the scientific fields were presented throughout the five days of the conference, but each day granted especial emphasis on one of them. In addition, the sessions, papers and posters presented addressed contributions on 10 Societal Challenges: Economic growth; Eradicate hunger, poverty and exclusion; Gender equality; Peace and social justice; Sustained infrastructure and industrialization; Global partnership for development; Sustainable water management; Environmental sustainability; Combat diseases; Education for all.
Midday special sessions were dedicated to: The role of research agencies, publishers, funding agencies, universities, and scientific societies in the promotion of social impact.
The Conference took a relevant step forward in the analysis of research that are having most social impact (in the different scientific domains), the increase of small scale social impact to further contexts, the planning of social impact for future research proposals, and the improvement of the communication of social impact both to evaluation agencies and to society. As well as to generate new bonds between the different actors involved in this initiative including researchers, scientific agencies, science stakeholders and civil society.

(2) Evidence for Society (European Parliament, Brussels, 2017)

The IMPACT-EV project Final Conference was jointly organized by the IMPACT-EV Consortium and Ms. Soledad Cabezón, Member of the European Parliament, who has coordinated the Report on the assessment of Horizon 2020 implementation in view of its interim evaluation and the Framework Program 9 proposal (2016/2147 (INI)).
The Conference was held in the European Parliament Headquarters, November 8, 2017.
The Evidence for Society Conference, provided evidence of projects financed by the European Union that had achieved social impact. These success stories were presented by different agents, including scientists, policy makers and end users who had benefited from the social impact of the projects funded by the European Commission.
The conference was structured in three major blocs. First, Evidence for enhancing health in society, then a second section on Evidence from Social Sciences and Humanities and finally, a bloc on Scientific literacy, open science and social impact. Finally, a Sciencethon was carried out on social impact in Wikipedia. In this last technical session, attendees had the opportunity to receive advice by several SIOR specialists for including their best research in the Social Impact Open Repository (SIOR) and then in Wikipedia.
The conference was opened by Dr. Ramon Flecha, main researcher of IMPACT-EV, Ms. Soledad Cabezon, Member of the European Parliament, and Maria Kayamanidou, Senior Policy Officer, European Commission.

More than 160 people with diverse profiles attended the event: end-users, policy-makers, members form the EP, professors, PhD students, project assistants, parliament assistants, research fellows, consultants, teachers, project managers, directors and institutions’ presidents, among others. Attendees came from different universities, research centres, schools, government departments and projects, and from countries such as Austria, Belgium, Bosnia and Herzegowina, Colombia, France, Germany, Ireland, Italy, Luxembourg, Macedonia, Netherlands, Nigeria, Norway, Portugal, Spain, Sweden, Switzerland, Turkey, United Kingdom and United States.
The link to the conference: http://impact-ev.eu/conference/

(3) Two Open Seminars with international speakers (Barcelona 2015 and Barcelona 2016)

As established in the Description of Work, the integrative panels have met twice after the 2nd and the 3rd Consortium meetings. In both cases, the panels were conducted as discussions that were open to the public debate. Two Open Seminars were organized with round-tables involving different profiles of participants and issues. In both cases, the Work Package leader in collaboration with KMC coordinator elaborated a proposal of participants and uploaded it in the workspace for discussion and suggestions from the rest of the Consortium.
The first Integrative Panel was held on the 2nd of October of 2015. It included three round-tables about scientific, political and social impact. A diverse and plural audience of more than 40 people attended the seminar. A briefing document was prepared including the main contributions and conclusions of the Integrative Panel and is available on the webpage of the project.
The second Integrative Panel was held on the 15th of December of 2016, the day before of the third consortium meeting (it was before and not the day after due to agenda reasons and in order to facilitate the participation of stakeholders).

In this second Integrative Panel, the 3 roundtables and debates were organised in relation to specific social topics: housing, education and public healthcare. A diverse audience of approximately 60 people attended the seminar. This second integrative panel put more emphasis on the role of social actors and endusers. Attendants included researchers, endusers, stakeholders and other citizens interested in the three social areas discussed. Examples of participants were a center for mental health, a foundation that provides aid to vulnerable children, families and women, a hospital, and an African cultural association. A briefing document was prepared including the main contributions and conclusions of this Integrative Panel and is available on the webpage of the project.

(4) Other national seminars and workshops

During the project lifespan, the consortium organized more than 15 seminars and workshops in different places (e.g. Barcelona, Budapest, Warsaw). Some of these seminars were open not only to scientific community but also to policy makers and/or the wider public.

(6) Webpage and social media

IMPACT-EV’s website was created and publicly available from the beginning of the project (Month 1): http://impact-ev.eu/. Along the project, the websitewas regularly updated and had more than 140.000 visits from more than 25 countries. In parallel, the website of the Social Impact Open Repository (SIOR) was launched as a unique tool resulting from the project. Sior website: http://www.ub.edu/sior/index.php. Finally, both the 1s Conference on Social Impact of Science SIS2016 and the Final conference had their own webpage. IMPACT-ev and SIOR Twitter accounts have been very useful for dissemination purposes.

(7) “How to enhance the impact of SSH research“ Guide for researchers

The Guide for Researchers gathers some of the main contributions of the project IMPACT-EV. Evaluating the impact and outcomes of EU SSH research (FP7, 2014-2017). Focusing on Social Sciences and Humanities research, IMPACT-EV has analysed, measured and helped enhancing the impact of research. Starting from a wide evaluation of more than 500 FP6, FP7 and H2020 projects, plus a set of case studies of successful projects, the team has identified strategies enabling scientific, political or social impact or impact on strengthening the European Research Area. This guide is not oriented to improve researchers’ ability to disseminate their project results but to provide advice and tools of how they can reconsider their research in a way that it can bring about social improvements, and therefore have more impact in the four areas explored (scientific, policy, social and ERA impact). The Researchers guide is available in the IMPACT-EV webpage (http://impact-ev.eu/outputs/)

(8) Policy briefs and Newsletters.

The IMPACT-EV consortium has elaborated two policy briefs, the first one in December 2015 and the second one in December 2017. Both of them are available on the IMPACT-EV webpage. Besides, we elaborated a position paper based on some results of the IMPACT-EV projects and discussions on SSH social and political impacts, which we submitted to the public consultation about H2020 launched by the European Commission.

List of Websites:

www.impact-ev.eu

Final Report Summary - IMPACT-EV (Evaluating the impact and outcomes of European SSH research)

Share this page

Download