Final Report Summary - ESS-DACE (The European Social Survey - Data for a Changing Europe)
Executive Summary:
ESS-DACE (July 2010 – December 2014) aimed to secure the continuation and consolidation of the European Social Survey (ESS) as a key data source for measuring and understanding long-term changes in public attitudes across Europe. By the end of the project, December 2014, this goal was achieved. Links were significantly strengthening between the 7 institutions responsible for the central design and coordination of the ESS infrastructure, as well as between the ESS and its funders, data users, national teams and distinguished network of advisers. ESS was awarded ERIC legal status in November 2013, an achievement underscoring its position as a research infrastructure of pan-European relevance, as a facility serving user communities (scientific, policy, civil society) in Europe and beyond.
The main objectives of the ESS infrastructure included:
- to chart and explain changes in Europe’s social, political and moral climate;
- to achieve and spread improved standards of rigour in cross-national social measurement; simultaneously raising threshold standards of national surveys;
- to introduce and encourage the use of soundly-based attitudinal indicators of national progress, based on citizens’ judgments of key aspects of their societies;
- to undertake and facilitate the training of European researchers in comparative quantitative measurement and analysis;
- to provide enhanced access remotely to the ESS data and meta data;
- to improve the visibility and outreach of quantitative data on social change to academics, policy makers and the wider public;
- to maintain and update a specially-created ‘contextual’ data repository to help isolate the independent effect of background political and social developments on shifts in attitudes and values; and
- to refine the financial, operational and organisational arrangements of the ESS infrastructure to suit its longer term perspectives and commitments.
In sum, ESS DACE aimed to ensure not just the survival of the ESS time series, but also to consolidate its achievements by creating a transparent, accessible and increasingly valued mechanism for unpicking the relationship between Europe’s changing institutions and the shifting values of its people.
By the end of the project all 97 deliverables were completed. A successfully concluded Mid-Term Review confirmed the relevance of the objectives and planned activities. Wide dissemination of the results took place embedding the exploitation of project learning in the infrastructure in its implementation phase as a European Research Infrastructure Consortium. The dissemination and exploitation efforts undertaken to date will continue beyond the lifetime of the project.
Project Context and Objectives:
The overriding aim during the 54 month period during which ESS DACE was implemented was not just to continue replenishing and refining what is probably the dedicated social science data source in Europe that is experiencing the fastest growth in usage, but also to continue raising standards of quantitative social measurement generally.
ESS DACE aimed to support the further development of the research infrastructure and the main objectives included:
- to chart and explain changes in Europe’s social, political and moral climate;
- to achieve and spread improved standards of rigour in cross-national social measurement; simultaneously raising threshold standards of national surveys;
- to introduce and encourage the use of soundly-based attitudinal indicators of national progress, based on citizens’ judgments of key aspects of their societies;
- to undertake and facilitate the training of European researchers in comparative quantitative measurement and analysis;
- to provide enhanced access remotely to the ESS data and meta data;
- to improve the visibility and outreach of quantitative data on social change to academics, policy makers and the wider public;
- to maintain and update a specially-created ‘contextual’ data repository to help isolate the independent effect of background political and social developments on shifts in attitudes and values; and
- to refine the financial, operational and organisational arrangements of the ESS infrastructure to suit its longer term perspectives and commitments.
These objectives were achieved in ESS-DACE in accordance with the requirements of the CP&CSA grant:
Workpackage 1 – this was the home of the management and coordination of the Consortium and of its distributed fieldwork. It involved working with others to improve the survey and its component parts, overseeing the quality of the whole operation, and spreading our data and our message to a wider audience than hitherto. The management work package was led by City, but all partners are involved in these activities.
Workpackages 2 – 7 & 9 were all designed as networking (coordination) activities with the aim of drawing the participating infrastructures together, linking them to other networks and infrastructures, and thereby improving both our overall measurement capability and our ability to deliver it. All partners were involved in either leading or working in at least one of the networking (coordination) activities. WP6 covered exploitation planning.
Workpackage 8 – WP8 involved the archiving, documentation, upkeep and updating of the ESS Data Archive at NSD Bergen which attracts tens of thousands of virtual visitors each year. Unless the experience remains pleasant and fruitful, demand would quickly drop off. Over the past number of years it continued to rise inexorably and the work carried out aimed to ensure that trend continued, thus disseminating our work – both data and protocols - to the widest possible audience.
Workpackages 10 – 13 were the joint research activities (RTD) designed to refine our measurement and access tools for the health of the future time series. They were designed in response to identified or predictable deficiencies in present facilities which needed rectifying.
Project Results:
This section presents the key results for each Work Package (2-13). 94 Deliverables with full details of the results were prepared and submitted during the lifetime of ESS DACE. Full details relating to the results achieved have been covered in the Periodic Reports submitted. In addition, the results from different WPs have been communicated to the scientific, policy and wider user communities through publications, conference presentations and workshop organisation and participation. The data and documentation, the key results of the European Social Survey, are made available to all via its website: www.europeansocialsurvey.org
Ethics
The project received an Ethics screening. Where research was undertaken, for example, in commissioning pre-testing (WP4) and innovation surveys (WP7), the City University London research ethics procedure was followed and ethics approval awarded for any activities involving human participants.
Work Package 2 - Sustainable Cooperation
The objectives of this WP were to continue and further develop sustainable cooperation across the ESS network thus ensuring the continuation of the ESS time series and responsiveness to the social science community; to provide central coordination of ESS specifications and develop the specifications for participating countries. These were achieved through the successful execution of three tasks -
Task 1: engaging the CCT with ESS funders and leading social scientists across Europe.
Task 2: in line with the global objective of the i3 Programme (Work Programme 2010: Research Infrastructures, European Commission, 2008:3), reinforcing the structure of ESS governance, creating systems and protocols better suited to a longstanding and multiple-funded enterprise, with formal agreements, agreed succession arrangements and a stable ‘constitution’.
Task 3: To centrally coordinate the production of ESS specifications .
In relation to Task 1 and 2, the engagement by the ESS Core Scientific Team with key stakeholders took place via meetings of the ESS Scientific Advisory Board, its Methods Advisory Board, of the National Coordinators and of the Funders Forum, a body superseded by the establishment of the ESS ERIC Steering Committee. At the end of ESS DACE, the ESS ERIC was in implementation phase and Round 7 fieldwork was underway in most of the 23 countries participating in ESS ERIC.
Task 3 related to the development of the Specifications for Participating countries. This work was ongoing over the lifetime of the project and included not only the production of Specifications for Round 6 and 7 but also their ongoing clarification in response to issues arising in the field. Their ‘fitness – for –purpose’ was continuously established.
WP3 - Sampling and fieldwork coordination and monitoring
This Work Packaged aimed to maintain and improve the robustness of the ESS in the course of raising standards of social measurement. It achieved this objective through two tasks:
Task 1: Develop and implement viable and equivalent sampling schemas for each participating country (with the aid of a standing panel of experts) in ESS Rounds 5 and 6.
Task 2: Optimise and monitor the quality of data collection (including response rates) across participating countries in ESS Rounds 5 and 6.
Task 1: The ESS aims to be representative of all people in the residential population aged 15 and over in each participating country (with no upper age limit), regardless of their nationality, citizenship or legal status. In keeping with the principle of strict probability sampling, the selection probability of every sample member is known and non-zero. Quota sampling or substitution is not allowed at any stage. This state of the art sampling specification is a central plank of the scientific rigor of the ESS. The objective of WP 3/task 1 is to ensure that courtiers adhere to these standards in terms of their sampling designs. The Sampling Expert Panel met throughout the lifetime of ESS DACE overseeing the sampling and sign off procedures in place.
Task 2: High quality data collection is at the heart of any serious survey. The objective of WP 3/Task 2 is to optimise data collection procedures across participating countries in order to achieve a high and equivalent quality of data in all countries participating in the ESS survey. The ESS Core Scientific Team played an important part in monitoring and support to ensure that data collection in each participating country complies as closely as possible with the highest standards set out in the ESS specifications.
Scientists as GESIS prepared country by country reports, analysing compliance with and deviation from the specifications. These reports utilise information in the Documentation Report produced by NSD and the Response Based Quality Reports produced by KUL.
The learning arising from the implementation in Round 6 was captured and communicated to field directors in Round 7 through a Field Directors’ Meeting, established in WP3.
WP4 - Designing and developing the survey instrument for translation and implementation in 25+ countries
This WP, although largely focused on coordination activities, underpins the generation of significant scientific results.
Tasks 1 and 3 related to the design, pre-test and documentation of the development of an ESS source questionnaire (for Round 6 of the time series) suitable for fielding across a variety of European cultures and socio-political systems.
Task 2 focused on efforts to improve the previous translation methodology and develop, with experts, a revised set of methods and protocols which may become an exemplar of best practice. Its also coordinated and supported participating countries in the translation of the Round 5 and Round 6 questionnaires.
The development of the source questionnaire is a constant activity, notwithstanding the fact that ESS is a time series. Ongoing evaluation is undertaken by scientists across the ESS network and used to amend features of the questionnaire. One exercise undertaken prior to Round 6, and drawing on the bibliographic analyses undertaken as part of Work Package 6, was the removal of a number of questions which had received little/no attention by analysts. Their replacement was the subject of ongoing deliberation between the Scientific Advisory Board and the Core Scientific Team. Significant work is undertaken on the development of the rotating modules – the Question Design Team work with members of the Core Scientific Team to effectively operationalise concepts within the limited amount of space available on the survey. These items are pre-tested and further refined.
Ensuring equivalence requires significant translation expertise. ESS instituted a systems of advance translations of all new items for Round 6. Results include translation protocols which can be used by scientists outside the ESS network. The programme Survey Quality Predictor (SQP), developed and refined continuously by UP is offered to national coordinators to test questions prior to final inclusion in the questionnaire.
WP5 - Quality Management
This WP was focused on implementing five main tasks:
Task 1: ensuring timely and effective communication between the CCT and National Coordinators (NCs) with regard to the planning and execution of fieldwork R5 (year 2), and R6 through a series of meetings.
Task 2: ensuring that ESS methodology remains state of the art through meetings of the Methods Group.
Task 3: enabling dialogue between CCT members and external substantive and methodological experts to ensure the ESS remains at the cutting edge of research developments and remains responsive to its users through a series of Quality Enhancement Meetings.
Task 4: improving the measurement of selected socio demographic variables in the ESS core.
Task 5: producing a set of best practice protocols for the production of data and metadata based on the conclusions and recommendations from task 4.
The work carried out achieved each of these tasks; meetings with National Coordinators were held; the international Methods Group was convened and the Quality Enhancement meetings took place. The Socio-Economic Working Groups have concluded and their recommendations were incorporated in the development of the ESS Specifications and Protocols.
WP6 - Outreach
The aim of this work package was to enlarge and intensify the engagement of a diverse range of user communities with the ESS - in particular, academia, policy analysts and the general public. It achieved this aim through an intense programme of communication and dissemination activities, all designed to achieve three interlocking objectives:
• to build and reinforce virtual research communities by providing a range of information and educational resources through the project’s parallel websites
• to produce and disseminate attractive and accessible publications in hard copy to present and interpret the methods and findings of the European Social Survey time series
• to provide the various user-constituencies of ESS with opportunities to interact directly at a series of events in locations across Europe.
Specific objectives guiding action included:
Task 1: to promote the use of secondary analysis of ESS data by academics and to provide supporting materials for public and policy communities
Task 2: to maintain the ESS Bibliography as a bibliometric record of outputs related to or inspired by the ESS
Task 3: to promote the use of the existing ESS EduNet online series
Task 4: to organise a series of policy seminars as a mechanism for knowledge exchange between academics and policy makers
Task 5: to disseminate and popularise a set of evidence-based indicators of societal well-being that have been developed by the ESS team since 2006.
The results of this WP included papers on indicators of societal well-being; a report on policy seminars; the ongoing development of the EduNet e-learning resource; the further development of the ESS Bibliography and the initiation and development of google analytics relating to ESS citations.
WP7 - Developing and implementing innovative solutions across the ESS network
Task 1: Coordinating across the ESS network, both within and outside the CCT, to establish and, if possible, introduce the protocols and procedures which allow (some) flexibility between countries in their modes of data collection. This is to be achieved without either unduly sacrificing the equivalence of the national datasets or the integrity of the ESS time series
Task 2: Establish in conjunction with the ESS network and wider social science scholarly community an economical multi-country ‘innovation sample’ on which to test methodological and substantive innovations in advance of their implementation. This coordination workpackage aims to achieve the objectives of these two tasks by bringing together key actors from across the ESS network and the European Research Area (CCT, NCs, SAB, Methods Group, substantive experts, data users).
Key results include findings from the Innovation Report which assessed whether societal wellbeing could be measured by asking individuals questions about their perception and evaluation of how society is working and their satisfaction with it. A Feasibility Report in respect of costs, response rates, design effects and quality of alternative data collection was also carried out; findings suggest that a web-CAPI mixed mode design has potential to save field data collection costs without harming the survey response rate, but that it may prove impossible to realise this potential in all countries. In ESS DACE Deliverable 7.6 – Correcting for mode differences, protocols and tools for correcting mode differences in the data were produced . ESS DACE Deliverable 7.7. User protocols for correcting differences due to mode effects, questions arise with regard to the comparability of data collected using different modes. Deliverable 7.7 attempts to explain to the wider community how to implement the necessary research design and how to later use it to disentangle measurement differences from selection effects, using ESS DACE data to illustrate how this can be achieved. ESS DACE Deliverable 7.8. Innovation report (2). The second experiment conducted using the ESS DACE Innovation Sample was designed to further understanding of whether Don’t Know responses should be accepted or whether respondents should be encouraged to select a “substantive” answer. Probing significantly reduced rates of DK responses across all questions and across all three countries, affecting also patterns of response. Differences observed between probed and un-probed responses can be explained—at least in part—by measurement differences. Probed responses were much less strongly associated with the underlying latent constructs. Even though probed responses were found to be significantly associated with the underlying constructs being measured, there was a lack of measurement equivalence between probed and un-probed responses. Based on these results, it was concluded that probing DK responses can provide useful additional information about respondent attitudes but does so at the cost of introducing additional measurement error into the data. ESS DACE Deliverable 7.9. Set of Guidelines for Data Collection Strategies in the ESS. Before the ESS can select specific design options, fieldwork procedures and data collection strategies for mixed mode data collection, a decision needs to be made with respect to whether or not mixed-mode data collection would be acceptable within the ESS context. In order to guide this decision, it was judged necessary to commission and agree on a set of criteria that the collected evidence needs to meet for mixed modes to be a possibility in the future of the ESS. Therefore, rather than setting up guidelines for a hypothetical design, it was deemed crucial to commission this set of criteria against which to later evaluate the complete body of mixed mode research that has been carried out so far under the ESS mixed mode research programme. The methodological consultant of the work package provided a draft set of criteria that were then agreed with the Core Scientific Team, the Scientific Advisory Board and the Methods Advisory Board. During discussion of the criteria (summarised in the deliverable), certain guidelines and specifications were identified and noted.
WP8 - Web based access to the ESS data and metadata repository
The objective of this WP was to provide end users with on-line access to all ESS data, metadata and related reports from all previous and the two next rounds of ESS, and various dissemination tools. User support will also be provided.
The service NSD provided was access to the following output from the completed ESS Rounds 1-4 and the coming ESS Rounds 5 and 6:
• Data from the ESS interviews, as integrated and country-wise files
• ESS methodological data, i.e. Sample data files, Contact form files, Reliability test files and data from
Interviewers’ forms
• All documentation from the ESS fieldworks, for example source questionnaires and questionnaires in all languages fielded
• Data Documentation reports, Quality Assessment reports, User Guides etc
• On-line analysis of ESS data
• ESS Multilevel Data Repository (contextual data)
• ESS on-line bibliography
• ESS EduNet e-learning resource
The Service Access Review Panel reported positively: (D8.3)
“The content provided by the ESS website fully meets all the objectives of the service activity. The content includes data files, comprehensive information about the quality of data and methodological issues, all fieldwork documents, metadata and multilevel data. The data are available without restrictions for non-for-profit purposes and the access is free after a simple registration procedure. Other documents are available without any restrictions to all users.
The available data include the integrated file for all the participating countries and the country-specific files. There is also access to methodological data like the test data (MTMM) data from the supplementary questionnaire, interviewer data, call record data (contact form data) and sample design data files for a subset of countries. The scope of the available information is impressive and significantly exceeds what is typical for other large comparative surveys.
The review panel therefore concluded that the information provided by the website is comprehensive and fully meets the objective of the WP8 service activity.”
WP9 - Programme of Academic Cooperation and Training
One of the key objectives of the ESS itself is to improve the rigour and equivalence of cross-national survey research across Europe. The key target group of WP9 are researchers employed at research institutions and working in research teams all over Europe and wider, involved in cross-national survey research. ESS Train courses and SCC visits were set up for researchers who may be new to cross-national research but already have an understanding of the basics of survey research yet who need to understand more about the challenges of comparative survey research. Both the participants and their home institutions stand to benefit directly from the enhanced knowledge and skills acquired via SCC visits and ESS Train courses.
This WP was implemented around two key tasks:
Task 1: To achieve knowledge transfer and capacity building in survey methodology by offering training courses to researchers at all levels who can both benefit directly and act as multipliers in their own research communities
Task 2: Each of the CCT partners, recognised as having specific competences in the area of survey research, are designated ‘Survey Competence Centres’ (SCCs). The objective of this task is to design and implement a Programme of Cooperation between the SCCs and the wider academic / survey research community, facilitating the mobility of researchers and knowledge across the European Research Area
Task 1: Training courses in survey methods (UL, GESIS)
Building on the success of ten ESS train courses during the ESSi project, a series of three ESSTrain courses, with groups of 25 participants per course is to take place under ESS-DACE. The seminars are designed to enable intense, workshop-type courses, with participants interacting with trainers, receiving individualised feedback and doing practical exercises in small ad-hoc teams. In order to offset the reduced opportunity for direct participation, the participants' recruitment process emphasises the ‘training the trainers’ principle of selection so that in the long run a much higher number of researchers will benefit. Three courses were planned during the DACE project, training 75 participants in total.
Task 2: Survey Competence Centres' Academic Programme
The ESS-DACE academic cooperation strategy was designed to complement the other ESS-DACE dissemination and communication activities (WP6) and the service access arrangements (WP8). The ESS DACE is comprised of seven Survey Competence Centres (the CCT) which individually and collectively offer world-class expertise in survey methodology. This task offered 21 researchers across Europe the opportunity to engage directly with the scientific and research staff in each of the Survey Competence Centres, enabling junior and senior scientists with interest in particular areas to interact with staff in each of the CCT institutions over an extended period of time. The programme essentially allows researchers the opportunity to see the internal operations of the ESS in person and ensured ESS –DACE meets the global i3 goal of maximising the use and development of the best research infrastructures in Europe. Each Survey Competence Centre will offered a specific scientific programme relating to its identified expertise in comparative social measurement, dissemination strategies or the management of large scale cross-national surveys. The application process was via the European Social Survey website (www.europeansocialsurvey.org). 52 applications were received in total and 21 Visitors accepted.
WP10 - Improving the efficiency, usability and functionality of on-line access to ESS data and protocols
This workpackagae aimed to improve the quality and quantity of the services provided by the infrastructure. It had two objectives:
• To enhance both the functionality and user interfaces of on-line access to ESS data and metadata
• To produce research software applications for access to and utilisation of advanced methodological data
Task 1 Produce a tailored interface for retrieval of ESS cumulative data (NSD)
ESS data and meta data are currently organised on a round by round basis, but there are also well-documented cumulative data files which contain the full time series. The work we envisage here is to develop a new interface that enables users to create bespoke data files by sub-setting variables, countries and ESS rounds. In selecting the sub-sets they will be supported by immediate access to tailored meta data along the lines of variable documentation with an overview of the time series, etc. The system will be developed by programmers at NSD, while researchers at NSD will provide input on usability. This task was successfully completed.
Task 2. Software for correcting for measurement error in the analysis of ESS data (UPF)
A database of quality estimates for earlier ESS questions that have been included in MTMM experiments has been completed by the midpoint of the project. This task also aimed to refine this database by extending it to question wording in all ESS language/country versions of Round 1 to round 4. This has been done in the last reporting period even including all the questions of Round 5.
Information about the concepts the questions are supposed to measure was added to the programme and to make all this information available to all ESS data users in a user friendly manner. Besides that a simple procedure should be developed for correction for measurement errors using the SQP information.
An EduNet e-training module was prepared by Anna DeCastellarnau and Willem Saris at UPF : ‘A simple procedure to correct for measurement errors in survey research’, and is available for free on the ESS website.
WP11 - Curation of cumulative ESS data and meta data
Task 1: The curation of data and meta data for the new rounds of the ESS (Rounds 5-6), including preparatory work and quality assurance of complex methodological data. A vital part of the preparatory work is the production of the ESS Data Protocol and the National Technical Summary specifications for the delivery of high quality data and metadata to the ESS. Task 1 forms the base for the quantitative improvement and cumulative growth of data and meta data, and thereby other outcomes and updates of the ESS. Furthermore, it ensures the continuation of swift and free provision of high-quality ESS data and documentation.
Task 2: To plan for and execute long-term curation and preservation of ESS data and metadata, involving the active management of these data over the entire survey lifecycle, including the timely release of new and updated editions. Long-term curation also includes the establishment of appropriate administrative systems and procedures to meet the challenges of the growing volume and complexity of the ESS and incorporates risk prevention. This work will be based on recommendations in ISO 14721 (ISO 2003).
The twin objectives of this WP are designed to improve the quality and quantity of services provided by the infrastructure. ‘Data curation’ is a central concept to this WP and includes the active management and enhancement of data and metadata. These objectives were to a large extent pursued in parallel, in that long term curation solutions are inevitably to be adjusted and amended alongside the curation of new data and metadata.
The long term curation and preservation of the ESS data and metadata includes the upgrade of data and metadata of previous ESS rounds and the production and release of the ESS 1-5 cumulative data set are key results from this activity.
The National Technical Summary form for ESS round 6 was produced as were the methodological data from the contact forms as well as the test section of the supplementary questionnaires. These were published in their second edition in January 2013.
The preparation for ESS round 6 was undertaken in parallel to the curation work on ESS rounds 1-5. Round 6 data were released in October 2013 (first edition) and May 2014 (second edition).
WP12 - Minimising non sampling errors in cross-national data
This WP aims to measure and reduce non-sampling errors, incorporating the relationship with sampling errors, for instance, in the study of non-response. Specific objectives, with matching tasks, are
(i) to assess the quality of ESS response enhancing efforts and fieldwork strategies
(ii) to assess the quality of ESS realised samples and degree of non-response bias
(iii) to calculate improved post-stratification weights to correct for unequal representation for all rounds, for all countries
(iv) to assess and improve data collection methods via interviewer effects
(v) to evaluate concepts in the ESS core questionnaire in order to improve the quality of the questionnaire.
By the end of RP3 the following results have been achieved:
Task 1 (for Rounds 5 and 6):
Task 1: to assess the quality of ESS response enhancing efforts and fieldwork strategies
In order to finalize the research objective of this WP (task 1 & 4, data analysis concerning the assessment of the quality of ESS response-enhancing efforts and fieldwork strategies in Round 5 (total 27 countries), and also the assessment and improvement of data collection methods via interviewer effects in Round 5 is underway.
• assessment response and nonresponse rates in all countries
• assessment of the implementation of fieldwork procedures especially on noncontacts
• assessment of the information on nonresponse units (including type of refusal)
• assessment of the refusal conversion activities and success rate
• assessment of the relationship between response behaviour and characteristics of dwelling and neighbourhood, and
• based on the analyses tailored and country-specific feedback has been provided to accompany the standardised contact files (the technical generation of which is within WP11). Feedback will be provided in collaboration with WP3 to ensure an integrated feedback to the network of ESS NCs
• evaluation of the quality of observable data in Round 5
• a proposal for new specifications on observable data to be used in fieldwork data collection
Task 2: Assessing the quality of the realised samples by using external and internal criteria (GESIS, SCP)
• collection external auxiliary information at the population level in order to measure the quality of the realised samples
• analysis internal criteria of representativeness
• relating external and internal quality criteria with selected indicators of the sampling and data collection process (e.g. type of sample used, length of fieldwork period, response rate, etc.)
• analysis nonresponse bias
• based on the above analysis, producing a report on the quality of the realised samples and the degree of nonresponse bias for Round 5
Task 3: to calculate improved post-stratification weights to correct for unequal representation for all rounds, for all countries (UL, SCP, GESIS)
There was significant work undertaken on the release of post-stratification weights for all rounds of ESS.
• inventory and assessment of variables available and their suitability for PS
• a strategy for dealing with missing data in sample and/or control dataset
• post-stratification eights for all Rounds
• extensive documentation
• feedback with NCs
• sensitivity analysis
Task 4: Assessment and improvement of data collection methods via interviewer effects (KUL, GESIS)
Activities within this task related to the assessment and improvement of data collection methods on interviewer effects. The results from this task have been widely disseminated.
• overview of interviewer variance in ESS participating countries
• assessment of the level of interviewer variance when making cross-country comparisons
• separation of interviewer effect and design effect in complex sample surveys by applying model-basing techniques of variance decomposition to evaluate the separate and common influences of geographical and interviewer clustering
• identification interviewer variance in specific questions and contribute it to interviewer training
Task 5: to evaluate concepts in the ESS core questionnaire in order to improve the quality of the questionnaire.
The ESS is the only large scale survey that has incorporated an evaluation of the quality of the questions into its design. So-called ‘Multitrait-Multimethod’ (MTMM) experiments (Campbell and Fiske 1959) with alternative formulations of questions are included in a supplementary questionnaire at each round of the survey (note they are also included in the two-country pilot tests – see WP4). These experiments provide the ESS with the unique possibility of estimating the quality of single questions. This information is important because lack of question quality will drastically reduce the strength of the relationships between variables, and decrease efficiency of estimation. Unequal measurement quality in different countries will make comparison of relationships between variables across countries impossible. Besides that the quality of the questions is also necessary to evaluate the quality of the composite scores for more complex concepts with multiple indicators and for the evaluation of the equivalence of the measurement instruments across countries. During the first four rounds a number of questions for different concepts have been evaluated. In WP12 several concepts in the ESS core questionnaire will be evaluated.
Results achieved included:
• evaluation of the quality of the questions using MTMM experiments during ESS R5-R6
• evaluation of the equivalence of the indicators with regard to metric and scalar invariance
• calculation of the composite scores for each concept and evaluation of their quality
• evaluation of the external validity of the composite scores by testing for relationships with other variables in the nomologic network of these concepts
• storage of the data for each respondent on the composite score in a file for further use by ESS data users via the ESS data website
• reports about each concept concerning the steps which have been indicated above
• integrated software that allows the use of the derived data, taking into account the measurement error in these variables (WP10, task 2)
In addition, a large number of presentations were given and articles published.
WP13 - Improving the availability of and access to relevant contextual data
This WP aimed to update and further strengthen the inclusion of a wide range of contextual data (at a number of spatial levels) that can supplement the data collected about individuals in each participating country in the ESS. To make the contextual data available in a form that can easily be used alongside the main data files in the archive.
Task 1: Identify and collect relevant new variables and topics for the contextual data repository and to update the repository in time with each new round of the ESS survey.
Task 2: To provide information on events that occur during fieldwork, and to develop a set of procedures and guidelines for the harmonised collection and coding of media reports by participant countries.
The aim of the WP was to update and further strengthen the inclusion of a wide range of contextual data, at a number of spatial levels, which can supplement the data collected about individuals in each participating country in the ESS.
Task 1 included the following - the work on the technical platform and resource for bringing the ESS micro and macro data together, the ESS Multilevel Data (ESS MD), has been funded outside the ESS DACE. The ESS MD was released for public use September 18th 2012 including the updated macro data to match the round 5 ESS micro data. In the final stages of developing the ESS MD, focus was on adding content to the resource. Data from 2010 and 2011 (if available) were included prior to the release of the resource covering the following topics: Demography and Geography, Economy, Education, Health, Crime, and some Composite measures like the Human development index. Since the release in September 2012, the focus has been on providing user support.
Task 2: From the outset of the European Social Survey in 2001, National Coordination Teams have collected information about media-reported events that occurred during fieldwork in each participating country. In the preceding period the event reporting implemented in Round 1 to 5 has been replaced by an alternative, innovative approach funded as part of ESSi (i3, FP6). This new approach, based on the Political Claims making Approach offers great potential for refining the collection of ‘event data’. The media claims approach has been implemented in the ESS R6. Countries were asked to collect claims in a ten week period, starting one week prior to fieldwork. A month before coding the claims, NCs sent their media landscape with an overview of the two selected newspapers which will be used for the collection of media claims. A media claims guidelines R6 and training package were made available for further guidance. The process of coding is monitored by SCP. A total of 22 countries agreed in the collection of the claims and send their media landscape document, five countries have now finished coding and send their SPSS data file. The results of this task have been widely disseminated.
Potential Impact:
"Public attitudes matter in democratic societies. They reflect what citizens believe, want, fear and prefer. They are difficult to measure, are often unexpressed, and cannot be inferred from electoral choices alone. Nor can they be gleaned from media opinion polls which tend to give momentary and incomplete glimpses of attitude formation and change. But long-term changes in the population’s attitude and value positions are as important to governance and social analysis as are shifts in a country’s (or a continent’s) demographic profile, economic outlook, behaviour patterns and cultural norms. Politicians, policy makers and academics all need to keep abreast of and understand such changes if they are to achieve a nuanced picture of their society. This is why the ESS exists. It is specifically designed to meet the exacting demands of academics, policy-makers and civil society alike for rigorous cross national data on social attitudes and behaviour. In its first decade the ESS has thus become a time-series of undisputed authority and utility" (Founder Director, Roger Jowell)
The impact of the European Social Survey Data for a Changing Europe (ESS DACE) is captured by the statement made by Professor Sir Roger Jowell. Public policy making requires data that are defensible in scientific terms and relevant in societal terms. The ESS, in round 7 by the end of the ESS DACE project responds to both of these terms. Its impact
The dissemination activities are varied but may be categorised as follows
- conference presentations (for example, at the ESS International Conference in Cyprus, the results were presented by CST members, as an example, Matsuo, H. & G. Loosveldt (2012) Assessing data quality in interviewer collected observable data: interviewer burden perspective from Round 5. Paper presented at ESS Cyprus, 22-25 November 2012; Stoop, I., A. Koch & H. Matsuo (2012) Not Able and Not Available: the Other Reasons. An exploratory research not, Prepared for the international non-response workshop’, Ottawa, 4-6 September 2012.
- workshop participation for knowledge and best practice exchange (for example, with European Commission Agencies for example, Matsuo, H. (2013) Nonresponse bias detection and adjustment through calibrated models: case study on Dutch reluctant respondents in ESS4. Presentation prepared for Expert meeting Eurofound, Dublin, 21-22 January, 2013
- submission for publication to relevant international journals (for example, Loosveldt, G. & K. Beullens (2013), ‘ How long will it take?’ An analysis of interview length in the fifth round of the European Social Survey, Survey Research Methods, Vol. 7, No 2.)
- The Topline Results Series (see below) - four issues
- The ESS Policy Seminar Series (see below) - six events
Across its 13 Work Packages, the ESS DACE results have been formally captured and recorded in Deliverables. All Work Package were successfully concluded in line with Annex I. The learning and results arising from the implementation of tasks within WPs cumulatively informed the later scheduled Deliverables throughout the lifetime of the project. This learning has been captured and will continue to be shared with the wider survey community (both academic and practitioners) thereby reinforcing the impact of the project beyond its immediate implementation.
Scientists across the ESS network have been active in disseminating the results to the relevant user communities, in particular, for example, at the European Survey Research Association (ESRA) and international conferences on research methodology. In addition, scientists prepared papers for publication in relevant journals; for example, the results of WP10 Improving the Efficiency, Usability and Functionality of on-line access to ESS data and protocols, specifically relating to the estimation of measurement error is included in the revised edition of Saris W and Gallhofer, Design, evaluation and analysis of questionnaires for survey research, Wiley (forthcoming, 2014). The ESS Service Activity allows full access to the growing ESS data archive and on-line and direct training tools support use of the research infrastructure. Results from the ESS DACE project have been presented at national and international conferences, including at the European Survey Research Association conference and at AAPOR and WAPOR.
Four issues of the ESS Topline Results Series were published. The series is targeted at policy makers and the wider community. This has been widely disseminated to the ESS community in hard copy and via the ESS website in PDF. The ESS Topline Results Series was complemented by the ESS Policy Seminar Series, of which six seminars were held over the lifetime of the project.
ESS Policy Seminar Series ESS Topline Results Series
Seminar 1: Trust in Justice: How the ESS can inform policy, M23/May 2012, Centre for European Policy Studies (CEPS) Brussels
Seminar 2: Attitudes to Welfare, M30/November 2012, European Policy Centre (ECP), Brussels
Seminar 3: Work, Family and Wellbeing, M31/January 2013, OECD, Paris
Seminar 4: Europeans’ Understanding and Evaluations of Democracy, M47/May 2014, OECD, Paris
Seminar 5: Europeans’ Understanding and Evaluations of Democracy, M51/September 2014, Italian Parliament, Rome
Seminar 6: Europeans’ Personal and Social wellbeing – Findings from the European Social Survey Rounds 3 and 6, M51/September 2014 Issue 1: Trust in Justice: Topline Results from Round 5 of the European Social Survey, (December 2011)
Issue 2: Attitudes to Welfare: Topline Results from Round 4 of the European Social Survey (November 2012)
Issue 3: Economic Crisis, Quality of Work and Social Integration: Topline Results from Round 2 and Round 5 of the European Social Survey (April 2013)
Issue 4: Europeans’ Understanding and Evaluations of Democracy: Topline Results from Round 6 of the European Social Survey (September 2014)
The ESS DACE project was instrumental in ensuring continuity of the ESS at a time of transition. Led by the host country, the UK, the application for the legal status ‘European Research Infrastructure Consortium’ (ERIC), was successfully realised in November 2013. This award was recognition of the ESS as a research infrastructure of pan-European relevance, serving the European and wider research, policy and other user communities. At the end of the ESS DACE project period, 23 countries had formally agreed to participate in ESS ERIC as it implemented Round 7. The learning arising from the ESS DACE Project and its results in relation to survey coordination and survey methodological rigor and quality are incorporated into the ESS ERIC as it consolidates its position as a research infrastructure of pan European relevance.
List of Websites:
www.europeansocialsurvey.org
ESS-DACE (July 2010 – December 2014) aimed to secure the continuation and consolidation of the European Social Survey (ESS) as a key data source for measuring and understanding long-term changes in public attitudes across Europe. By the end of the project, December 2014, this goal was achieved. Links were significantly strengthening between the 7 institutions responsible for the central design and coordination of the ESS infrastructure, as well as between the ESS and its funders, data users, national teams and distinguished network of advisers. ESS was awarded ERIC legal status in November 2013, an achievement underscoring its position as a research infrastructure of pan-European relevance, as a facility serving user communities (scientific, policy, civil society) in Europe and beyond.
The main objectives of the ESS infrastructure included:
- to chart and explain changes in Europe’s social, political and moral climate;
- to achieve and spread improved standards of rigour in cross-national social measurement; simultaneously raising threshold standards of national surveys;
- to introduce and encourage the use of soundly-based attitudinal indicators of national progress, based on citizens’ judgments of key aspects of their societies;
- to undertake and facilitate the training of European researchers in comparative quantitative measurement and analysis;
- to provide enhanced access remotely to the ESS data and meta data;
- to improve the visibility and outreach of quantitative data on social change to academics, policy makers and the wider public;
- to maintain and update a specially-created ‘contextual’ data repository to help isolate the independent effect of background political and social developments on shifts in attitudes and values; and
- to refine the financial, operational and organisational arrangements of the ESS infrastructure to suit its longer term perspectives and commitments.
In sum, ESS DACE aimed to ensure not just the survival of the ESS time series, but also to consolidate its achievements by creating a transparent, accessible and increasingly valued mechanism for unpicking the relationship between Europe’s changing institutions and the shifting values of its people.
By the end of the project all 97 deliverables were completed. A successfully concluded Mid-Term Review confirmed the relevance of the objectives and planned activities. Wide dissemination of the results took place embedding the exploitation of project learning in the infrastructure in its implementation phase as a European Research Infrastructure Consortium. The dissemination and exploitation efforts undertaken to date will continue beyond the lifetime of the project.
Project Context and Objectives:
The overriding aim during the 54 month period during which ESS DACE was implemented was not just to continue replenishing and refining what is probably the dedicated social science data source in Europe that is experiencing the fastest growth in usage, but also to continue raising standards of quantitative social measurement generally.
ESS DACE aimed to support the further development of the research infrastructure and the main objectives included:
- to chart and explain changes in Europe’s social, political and moral climate;
- to achieve and spread improved standards of rigour in cross-national social measurement; simultaneously raising threshold standards of national surveys;
- to introduce and encourage the use of soundly-based attitudinal indicators of national progress, based on citizens’ judgments of key aspects of their societies;
- to undertake and facilitate the training of European researchers in comparative quantitative measurement and analysis;
- to provide enhanced access remotely to the ESS data and meta data;
- to improve the visibility and outreach of quantitative data on social change to academics, policy makers and the wider public;
- to maintain and update a specially-created ‘contextual’ data repository to help isolate the independent effect of background political and social developments on shifts in attitudes and values; and
- to refine the financial, operational and organisational arrangements of the ESS infrastructure to suit its longer term perspectives and commitments.
These objectives were achieved in ESS-DACE in accordance with the requirements of the CP&CSA grant:
Workpackage 1 – this was the home of the management and coordination of the Consortium and of its distributed fieldwork. It involved working with others to improve the survey and its component parts, overseeing the quality of the whole operation, and spreading our data and our message to a wider audience than hitherto. The management work package was led by City, but all partners are involved in these activities.
Workpackages 2 – 7 & 9 were all designed as networking (coordination) activities with the aim of drawing the participating infrastructures together, linking them to other networks and infrastructures, and thereby improving both our overall measurement capability and our ability to deliver it. All partners were involved in either leading or working in at least one of the networking (coordination) activities. WP6 covered exploitation planning.
Workpackage 8 – WP8 involved the archiving, documentation, upkeep and updating of the ESS Data Archive at NSD Bergen which attracts tens of thousands of virtual visitors each year. Unless the experience remains pleasant and fruitful, demand would quickly drop off. Over the past number of years it continued to rise inexorably and the work carried out aimed to ensure that trend continued, thus disseminating our work – both data and protocols - to the widest possible audience.
Workpackages 10 – 13 were the joint research activities (RTD) designed to refine our measurement and access tools for the health of the future time series. They were designed in response to identified or predictable deficiencies in present facilities which needed rectifying.
Project Results:
This section presents the key results for each Work Package (2-13). 94 Deliverables with full details of the results were prepared and submitted during the lifetime of ESS DACE. Full details relating to the results achieved have been covered in the Periodic Reports submitted. In addition, the results from different WPs have been communicated to the scientific, policy and wider user communities through publications, conference presentations and workshop organisation and participation. The data and documentation, the key results of the European Social Survey, are made available to all via its website: www.europeansocialsurvey.org
Ethics
The project received an Ethics screening. Where research was undertaken, for example, in commissioning pre-testing (WP4) and innovation surveys (WP7), the City University London research ethics procedure was followed and ethics approval awarded for any activities involving human participants.
Work Package 2 - Sustainable Cooperation
The objectives of this WP were to continue and further develop sustainable cooperation across the ESS network thus ensuring the continuation of the ESS time series and responsiveness to the social science community; to provide central coordination of ESS specifications and develop the specifications for participating countries. These were achieved through the successful execution of three tasks -
Task 1: engaging the CCT with ESS funders and leading social scientists across Europe.
Task 2: in line with the global objective of the i3 Programme (Work Programme 2010: Research Infrastructures, European Commission, 2008:3), reinforcing the structure of ESS governance, creating systems and protocols better suited to a longstanding and multiple-funded enterprise, with formal agreements, agreed succession arrangements and a stable ‘constitution’.
Task 3: To centrally coordinate the production of ESS specifications .
In relation to Task 1 and 2, the engagement by the ESS Core Scientific Team with key stakeholders took place via meetings of the ESS Scientific Advisory Board, its Methods Advisory Board, of the National Coordinators and of the Funders Forum, a body superseded by the establishment of the ESS ERIC Steering Committee. At the end of ESS DACE, the ESS ERIC was in implementation phase and Round 7 fieldwork was underway in most of the 23 countries participating in ESS ERIC.
Task 3 related to the development of the Specifications for Participating countries. This work was ongoing over the lifetime of the project and included not only the production of Specifications for Round 6 and 7 but also their ongoing clarification in response to issues arising in the field. Their ‘fitness – for –purpose’ was continuously established.
WP3 - Sampling and fieldwork coordination and monitoring
This Work Packaged aimed to maintain and improve the robustness of the ESS in the course of raising standards of social measurement. It achieved this objective through two tasks:
Task 1: Develop and implement viable and equivalent sampling schemas for each participating country (with the aid of a standing panel of experts) in ESS Rounds 5 and 6.
Task 2: Optimise and monitor the quality of data collection (including response rates) across participating countries in ESS Rounds 5 and 6.
Task 1: The ESS aims to be representative of all people in the residential population aged 15 and over in each participating country (with no upper age limit), regardless of their nationality, citizenship or legal status. In keeping with the principle of strict probability sampling, the selection probability of every sample member is known and non-zero. Quota sampling or substitution is not allowed at any stage. This state of the art sampling specification is a central plank of the scientific rigor of the ESS. The objective of WP 3/task 1 is to ensure that courtiers adhere to these standards in terms of their sampling designs. The Sampling Expert Panel met throughout the lifetime of ESS DACE overseeing the sampling and sign off procedures in place.
Task 2: High quality data collection is at the heart of any serious survey. The objective of WP 3/Task 2 is to optimise data collection procedures across participating countries in order to achieve a high and equivalent quality of data in all countries participating in the ESS survey. The ESS Core Scientific Team played an important part in monitoring and support to ensure that data collection in each participating country complies as closely as possible with the highest standards set out in the ESS specifications.
Scientists as GESIS prepared country by country reports, analysing compliance with and deviation from the specifications. These reports utilise information in the Documentation Report produced by NSD and the Response Based Quality Reports produced by KUL.
The learning arising from the implementation in Round 6 was captured and communicated to field directors in Round 7 through a Field Directors’ Meeting, established in WP3.
WP4 - Designing and developing the survey instrument for translation and implementation in 25+ countries
This WP, although largely focused on coordination activities, underpins the generation of significant scientific results.
Tasks 1 and 3 related to the design, pre-test and documentation of the development of an ESS source questionnaire (for Round 6 of the time series) suitable for fielding across a variety of European cultures and socio-political systems.
Task 2 focused on efforts to improve the previous translation methodology and develop, with experts, a revised set of methods and protocols which may become an exemplar of best practice. Its also coordinated and supported participating countries in the translation of the Round 5 and Round 6 questionnaires.
The development of the source questionnaire is a constant activity, notwithstanding the fact that ESS is a time series. Ongoing evaluation is undertaken by scientists across the ESS network and used to amend features of the questionnaire. One exercise undertaken prior to Round 6, and drawing on the bibliographic analyses undertaken as part of Work Package 6, was the removal of a number of questions which had received little/no attention by analysts. Their replacement was the subject of ongoing deliberation between the Scientific Advisory Board and the Core Scientific Team. Significant work is undertaken on the development of the rotating modules – the Question Design Team work with members of the Core Scientific Team to effectively operationalise concepts within the limited amount of space available on the survey. These items are pre-tested and further refined.
Ensuring equivalence requires significant translation expertise. ESS instituted a systems of advance translations of all new items for Round 6. Results include translation protocols which can be used by scientists outside the ESS network. The programme Survey Quality Predictor (SQP), developed and refined continuously by UP is offered to national coordinators to test questions prior to final inclusion in the questionnaire.
WP5 - Quality Management
This WP was focused on implementing five main tasks:
Task 1: ensuring timely and effective communication between the CCT and National Coordinators (NCs) with regard to the planning and execution of fieldwork R5 (year 2), and R6 through a series of meetings.
Task 2: ensuring that ESS methodology remains state of the art through meetings of the Methods Group.
Task 3: enabling dialogue between CCT members and external substantive and methodological experts to ensure the ESS remains at the cutting edge of research developments and remains responsive to its users through a series of Quality Enhancement Meetings.
Task 4: improving the measurement of selected socio demographic variables in the ESS core.
Task 5: producing a set of best practice protocols for the production of data and metadata based on the conclusions and recommendations from task 4.
The work carried out achieved each of these tasks; meetings with National Coordinators were held; the international Methods Group was convened and the Quality Enhancement meetings took place. The Socio-Economic Working Groups have concluded and their recommendations were incorporated in the development of the ESS Specifications and Protocols.
WP6 - Outreach
The aim of this work package was to enlarge and intensify the engagement of a diverse range of user communities with the ESS - in particular, academia, policy analysts and the general public. It achieved this aim through an intense programme of communication and dissemination activities, all designed to achieve three interlocking objectives:
• to build and reinforce virtual research communities by providing a range of information and educational resources through the project’s parallel websites
• to produce and disseminate attractive and accessible publications in hard copy to present and interpret the methods and findings of the European Social Survey time series
• to provide the various user-constituencies of ESS with opportunities to interact directly at a series of events in locations across Europe.
Specific objectives guiding action included:
Task 1: to promote the use of secondary analysis of ESS data by academics and to provide supporting materials for public and policy communities
Task 2: to maintain the ESS Bibliography as a bibliometric record of outputs related to or inspired by the ESS
Task 3: to promote the use of the existing ESS EduNet online series
Task 4: to organise a series of policy seminars as a mechanism for knowledge exchange between academics and policy makers
Task 5: to disseminate and popularise a set of evidence-based indicators of societal well-being that have been developed by the ESS team since 2006.
The results of this WP included papers on indicators of societal well-being; a report on policy seminars; the ongoing development of the EduNet e-learning resource; the further development of the ESS Bibliography and the initiation and development of google analytics relating to ESS citations.
WP7 - Developing and implementing innovative solutions across the ESS network
Task 1: Coordinating across the ESS network, both within and outside the CCT, to establish and, if possible, introduce the protocols and procedures which allow (some) flexibility between countries in their modes of data collection. This is to be achieved without either unduly sacrificing the equivalence of the national datasets or the integrity of the ESS time series
Task 2: Establish in conjunction with the ESS network and wider social science scholarly community an economical multi-country ‘innovation sample’ on which to test methodological and substantive innovations in advance of their implementation. This coordination workpackage aims to achieve the objectives of these two tasks by bringing together key actors from across the ESS network and the European Research Area (CCT, NCs, SAB, Methods Group, substantive experts, data users).
Key results include findings from the Innovation Report which assessed whether societal wellbeing could be measured by asking individuals questions about their perception and evaluation of how society is working and their satisfaction with it. A Feasibility Report in respect of costs, response rates, design effects and quality of alternative data collection was also carried out; findings suggest that a web-CAPI mixed mode design has potential to save field data collection costs without harming the survey response rate, but that it may prove impossible to realise this potential in all countries. In ESS DACE Deliverable 7.6 – Correcting for mode differences, protocols and tools for correcting mode differences in the data were produced . ESS DACE Deliverable 7.7. User protocols for correcting differences due to mode effects, questions arise with regard to the comparability of data collected using different modes. Deliverable 7.7 attempts to explain to the wider community how to implement the necessary research design and how to later use it to disentangle measurement differences from selection effects, using ESS DACE data to illustrate how this can be achieved. ESS DACE Deliverable 7.8. Innovation report (2). The second experiment conducted using the ESS DACE Innovation Sample was designed to further understanding of whether Don’t Know responses should be accepted or whether respondents should be encouraged to select a “substantive” answer. Probing significantly reduced rates of DK responses across all questions and across all three countries, affecting also patterns of response. Differences observed between probed and un-probed responses can be explained—at least in part—by measurement differences. Probed responses were much less strongly associated with the underlying latent constructs. Even though probed responses were found to be significantly associated with the underlying constructs being measured, there was a lack of measurement equivalence between probed and un-probed responses. Based on these results, it was concluded that probing DK responses can provide useful additional information about respondent attitudes but does so at the cost of introducing additional measurement error into the data. ESS DACE Deliverable 7.9. Set of Guidelines for Data Collection Strategies in the ESS. Before the ESS can select specific design options, fieldwork procedures and data collection strategies for mixed mode data collection, a decision needs to be made with respect to whether or not mixed-mode data collection would be acceptable within the ESS context. In order to guide this decision, it was judged necessary to commission and agree on a set of criteria that the collected evidence needs to meet for mixed modes to be a possibility in the future of the ESS. Therefore, rather than setting up guidelines for a hypothetical design, it was deemed crucial to commission this set of criteria against which to later evaluate the complete body of mixed mode research that has been carried out so far under the ESS mixed mode research programme. The methodological consultant of the work package provided a draft set of criteria that were then agreed with the Core Scientific Team, the Scientific Advisory Board and the Methods Advisory Board. During discussion of the criteria (summarised in the deliverable), certain guidelines and specifications were identified and noted.
WP8 - Web based access to the ESS data and metadata repository
The objective of this WP was to provide end users with on-line access to all ESS data, metadata and related reports from all previous and the two next rounds of ESS, and various dissemination tools. User support will also be provided.
The service NSD provided was access to the following output from the completed ESS Rounds 1-4 and the coming ESS Rounds 5 and 6:
• Data from the ESS interviews, as integrated and country-wise files
• ESS methodological data, i.e. Sample data files, Contact form files, Reliability test files and data from
Interviewers’ forms
• All documentation from the ESS fieldworks, for example source questionnaires and questionnaires in all languages fielded
• Data Documentation reports, Quality Assessment reports, User Guides etc
• On-line analysis of ESS data
• ESS Multilevel Data Repository (contextual data)
• ESS on-line bibliography
• ESS EduNet e-learning resource
The Service Access Review Panel reported positively: (D8.3)
“The content provided by the ESS website fully meets all the objectives of the service activity. The content includes data files, comprehensive information about the quality of data and methodological issues, all fieldwork documents, metadata and multilevel data. The data are available without restrictions for non-for-profit purposes and the access is free after a simple registration procedure. Other documents are available without any restrictions to all users.
The available data include the integrated file for all the participating countries and the country-specific files. There is also access to methodological data like the test data (MTMM) data from the supplementary questionnaire, interviewer data, call record data (contact form data) and sample design data files for a subset of countries. The scope of the available information is impressive and significantly exceeds what is typical for other large comparative surveys.
The review panel therefore concluded that the information provided by the website is comprehensive and fully meets the objective of the WP8 service activity.”
WP9 - Programme of Academic Cooperation and Training
One of the key objectives of the ESS itself is to improve the rigour and equivalence of cross-national survey research across Europe. The key target group of WP9 are researchers employed at research institutions and working in research teams all over Europe and wider, involved in cross-national survey research. ESS Train courses and SCC visits were set up for researchers who may be new to cross-national research but already have an understanding of the basics of survey research yet who need to understand more about the challenges of comparative survey research. Both the participants and their home institutions stand to benefit directly from the enhanced knowledge and skills acquired via SCC visits and ESS Train courses.
This WP was implemented around two key tasks:
Task 1: To achieve knowledge transfer and capacity building in survey methodology by offering training courses to researchers at all levels who can both benefit directly and act as multipliers in their own research communities
Task 2: Each of the CCT partners, recognised as having specific competences in the area of survey research, are designated ‘Survey Competence Centres’ (SCCs). The objective of this task is to design and implement a Programme of Cooperation between the SCCs and the wider academic / survey research community, facilitating the mobility of researchers and knowledge across the European Research Area
Task 1: Training courses in survey methods (UL, GESIS)
Building on the success of ten ESS train courses during the ESSi project, a series of three ESSTrain courses, with groups of 25 participants per course is to take place under ESS-DACE. The seminars are designed to enable intense, workshop-type courses, with participants interacting with trainers, receiving individualised feedback and doing practical exercises in small ad-hoc teams. In order to offset the reduced opportunity for direct participation, the participants' recruitment process emphasises the ‘training the trainers’ principle of selection so that in the long run a much higher number of researchers will benefit. Three courses were planned during the DACE project, training 75 participants in total.
Task 2: Survey Competence Centres' Academic Programme
The ESS-DACE academic cooperation strategy was designed to complement the other ESS-DACE dissemination and communication activities (WP6) and the service access arrangements (WP8). The ESS DACE is comprised of seven Survey Competence Centres (the CCT) which individually and collectively offer world-class expertise in survey methodology. This task offered 21 researchers across Europe the opportunity to engage directly with the scientific and research staff in each of the Survey Competence Centres, enabling junior and senior scientists with interest in particular areas to interact with staff in each of the CCT institutions over an extended period of time. The programme essentially allows researchers the opportunity to see the internal operations of the ESS in person and ensured ESS –DACE meets the global i3 goal of maximising the use and development of the best research infrastructures in Europe. Each Survey Competence Centre will offered a specific scientific programme relating to its identified expertise in comparative social measurement, dissemination strategies or the management of large scale cross-national surveys. The application process was via the European Social Survey website (www.europeansocialsurvey.org). 52 applications were received in total and 21 Visitors accepted.
WP10 - Improving the efficiency, usability and functionality of on-line access to ESS data and protocols
This workpackagae aimed to improve the quality and quantity of the services provided by the infrastructure. It had two objectives:
• To enhance both the functionality and user interfaces of on-line access to ESS data and metadata
• To produce research software applications for access to and utilisation of advanced methodological data
Task 1 Produce a tailored interface for retrieval of ESS cumulative data (NSD)
ESS data and meta data are currently organised on a round by round basis, but there are also well-documented cumulative data files which contain the full time series. The work we envisage here is to develop a new interface that enables users to create bespoke data files by sub-setting variables, countries and ESS rounds. In selecting the sub-sets they will be supported by immediate access to tailored meta data along the lines of variable documentation with an overview of the time series, etc. The system will be developed by programmers at NSD, while researchers at NSD will provide input on usability. This task was successfully completed.
Task 2. Software for correcting for measurement error in the analysis of ESS data (UPF)
A database of quality estimates for earlier ESS questions that have been included in MTMM experiments has been completed by the midpoint of the project. This task also aimed to refine this database by extending it to question wording in all ESS language/country versions of Round 1 to round 4. This has been done in the last reporting period even including all the questions of Round 5.
Information about the concepts the questions are supposed to measure was added to the programme and to make all this information available to all ESS data users in a user friendly manner. Besides that a simple procedure should be developed for correction for measurement errors using the SQP information.
An EduNet e-training module was prepared by Anna DeCastellarnau and Willem Saris at UPF : ‘A simple procedure to correct for measurement errors in survey research’, and is available for free on the ESS website.
WP11 - Curation of cumulative ESS data and meta data
Task 1: The curation of data and meta data for the new rounds of the ESS (Rounds 5-6), including preparatory work and quality assurance of complex methodological data. A vital part of the preparatory work is the production of the ESS Data Protocol and the National Technical Summary specifications for the delivery of high quality data and metadata to the ESS. Task 1 forms the base for the quantitative improvement and cumulative growth of data and meta data, and thereby other outcomes and updates of the ESS. Furthermore, it ensures the continuation of swift and free provision of high-quality ESS data and documentation.
Task 2: To plan for and execute long-term curation and preservation of ESS data and metadata, involving the active management of these data over the entire survey lifecycle, including the timely release of new and updated editions. Long-term curation also includes the establishment of appropriate administrative systems and procedures to meet the challenges of the growing volume and complexity of the ESS and incorporates risk prevention. This work will be based on recommendations in ISO 14721 (ISO 2003).
The twin objectives of this WP are designed to improve the quality and quantity of services provided by the infrastructure. ‘Data curation’ is a central concept to this WP and includes the active management and enhancement of data and metadata. These objectives were to a large extent pursued in parallel, in that long term curation solutions are inevitably to be adjusted and amended alongside the curation of new data and metadata.
The long term curation and preservation of the ESS data and metadata includes the upgrade of data and metadata of previous ESS rounds and the production and release of the ESS 1-5 cumulative data set are key results from this activity.
The National Technical Summary form for ESS round 6 was produced as were the methodological data from the contact forms as well as the test section of the supplementary questionnaires. These were published in their second edition in January 2013.
The preparation for ESS round 6 was undertaken in parallel to the curation work on ESS rounds 1-5. Round 6 data were released in October 2013 (first edition) and May 2014 (second edition).
WP12 - Minimising non sampling errors in cross-national data
This WP aims to measure and reduce non-sampling errors, incorporating the relationship with sampling errors, for instance, in the study of non-response. Specific objectives, with matching tasks, are
(i) to assess the quality of ESS response enhancing efforts and fieldwork strategies
(ii) to assess the quality of ESS realised samples and degree of non-response bias
(iii) to calculate improved post-stratification weights to correct for unequal representation for all rounds, for all countries
(iv) to assess and improve data collection methods via interviewer effects
(v) to evaluate concepts in the ESS core questionnaire in order to improve the quality of the questionnaire.
By the end of RP3 the following results have been achieved:
Task 1 (for Rounds 5 and 6):
Task 1: to assess the quality of ESS response enhancing efforts and fieldwork strategies
In order to finalize the research objective of this WP (task 1 & 4, data analysis concerning the assessment of the quality of ESS response-enhancing efforts and fieldwork strategies in Round 5 (total 27 countries), and also the assessment and improvement of data collection methods via interviewer effects in Round 5 is underway.
• assessment response and nonresponse rates in all countries
• assessment of the implementation of fieldwork procedures especially on noncontacts
• assessment of the information on nonresponse units (including type of refusal)
• assessment of the refusal conversion activities and success rate
• assessment of the relationship between response behaviour and characteristics of dwelling and neighbourhood, and
• based on the analyses tailored and country-specific feedback has been provided to accompany the standardised contact files (the technical generation of which is within WP11). Feedback will be provided in collaboration with WP3 to ensure an integrated feedback to the network of ESS NCs
• evaluation of the quality of observable data in Round 5
• a proposal for new specifications on observable data to be used in fieldwork data collection
Task 2: Assessing the quality of the realised samples by using external and internal criteria (GESIS, SCP)
• collection external auxiliary information at the population level in order to measure the quality of the realised samples
• analysis internal criteria of representativeness
• relating external and internal quality criteria with selected indicators of the sampling and data collection process (e.g. type of sample used, length of fieldwork period, response rate, etc.)
• analysis nonresponse bias
• based on the above analysis, producing a report on the quality of the realised samples and the degree of nonresponse bias for Round 5
Task 3: to calculate improved post-stratification weights to correct for unequal representation for all rounds, for all countries (UL, SCP, GESIS)
There was significant work undertaken on the release of post-stratification weights for all rounds of ESS.
• inventory and assessment of variables available and their suitability for PS
• a strategy for dealing with missing data in sample and/or control dataset
• post-stratification eights for all Rounds
• extensive documentation
• feedback with NCs
• sensitivity analysis
Task 4: Assessment and improvement of data collection methods via interviewer effects (KUL, GESIS)
Activities within this task related to the assessment and improvement of data collection methods on interviewer effects. The results from this task have been widely disseminated.
• overview of interviewer variance in ESS participating countries
• assessment of the level of interviewer variance when making cross-country comparisons
• separation of interviewer effect and design effect in complex sample surveys by applying model-basing techniques of variance decomposition to evaluate the separate and common influences of geographical and interviewer clustering
• identification interviewer variance in specific questions and contribute it to interviewer training
Task 5: to evaluate concepts in the ESS core questionnaire in order to improve the quality of the questionnaire.
The ESS is the only large scale survey that has incorporated an evaluation of the quality of the questions into its design. So-called ‘Multitrait-Multimethod’ (MTMM) experiments (Campbell and Fiske 1959) with alternative formulations of questions are included in a supplementary questionnaire at each round of the survey (note they are also included in the two-country pilot tests – see WP4). These experiments provide the ESS with the unique possibility of estimating the quality of single questions. This information is important because lack of question quality will drastically reduce the strength of the relationships between variables, and decrease efficiency of estimation. Unequal measurement quality in different countries will make comparison of relationships between variables across countries impossible. Besides that the quality of the questions is also necessary to evaluate the quality of the composite scores for more complex concepts with multiple indicators and for the evaluation of the equivalence of the measurement instruments across countries. During the first four rounds a number of questions for different concepts have been evaluated. In WP12 several concepts in the ESS core questionnaire will be evaluated.
Results achieved included:
• evaluation of the quality of the questions using MTMM experiments during ESS R5-R6
• evaluation of the equivalence of the indicators with regard to metric and scalar invariance
• calculation of the composite scores for each concept and evaluation of their quality
• evaluation of the external validity of the composite scores by testing for relationships with other variables in the nomologic network of these concepts
• storage of the data for each respondent on the composite score in a file for further use by ESS data users via the ESS data website
• reports about each concept concerning the steps which have been indicated above
• integrated software that allows the use of the derived data, taking into account the measurement error in these variables (WP10, task 2)
In addition, a large number of presentations were given and articles published.
WP13 - Improving the availability of and access to relevant contextual data
This WP aimed to update and further strengthen the inclusion of a wide range of contextual data (at a number of spatial levels) that can supplement the data collected about individuals in each participating country in the ESS. To make the contextual data available in a form that can easily be used alongside the main data files in the archive.
Task 1: Identify and collect relevant new variables and topics for the contextual data repository and to update the repository in time with each new round of the ESS survey.
Task 2: To provide information on events that occur during fieldwork, and to develop a set of procedures and guidelines for the harmonised collection and coding of media reports by participant countries.
The aim of the WP was to update and further strengthen the inclusion of a wide range of contextual data, at a number of spatial levels, which can supplement the data collected about individuals in each participating country in the ESS.
Task 1 included the following - the work on the technical platform and resource for bringing the ESS micro and macro data together, the ESS Multilevel Data (ESS MD), has been funded outside the ESS DACE. The ESS MD was released for public use September 18th 2012 including the updated macro data to match the round 5 ESS micro data. In the final stages of developing the ESS MD, focus was on adding content to the resource. Data from 2010 and 2011 (if available) were included prior to the release of the resource covering the following topics: Demography and Geography, Economy, Education, Health, Crime, and some Composite measures like the Human development index. Since the release in September 2012, the focus has been on providing user support.
Task 2: From the outset of the European Social Survey in 2001, National Coordination Teams have collected information about media-reported events that occurred during fieldwork in each participating country. In the preceding period the event reporting implemented in Round 1 to 5 has been replaced by an alternative, innovative approach funded as part of ESSi (i3, FP6). This new approach, based on the Political Claims making Approach offers great potential for refining the collection of ‘event data’. The media claims approach has been implemented in the ESS R6. Countries were asked to collect claims in a ten week period, starting one week prior to fieldwork. A month before coding the claims, NCs sent their media landscape with an overview of the two selected newspapers which will be used for the collection of media claims. A media claims guidelines R6 and training package were made available for further guidance. The process of coding is monitored by SCP. A total of 22 countries agreed in the collection of the claims and send their media landscape document, five countries have now finished coding and send their SPSS data file. The results of this task have been widely disseminated.
Potential Impact:
"Public attitudes matter in democratic societies. They reflect what citizens believe, want, fear and prefer. They are difficult to measure, are often unexpressed, and cannot be inferred from electoral choices alone. Nor can they be gleaned from media opinion polls which tend to give momentary and incomplete glimpses of attitude formation and change. But long-term changes in the population’s attitude and value positions are as important to governance and social analysis as are shifts in a country’s (or a continent’s) demographic profile, economic outlook, behaviour patterns and cultural norms. Politicians, policy makers and academics all need to keep abreast of and understand such changes if they are to achieve a nuanced picture of their society. This is why the ESS exists. It is specifically designed to meet the exacting demands of academics, policy-makers and civil society alike for rigorous cross national data on social attitudes and behaviour. In its first decade the ESS has thus become a time-series of undisputed authority and utility" (Founder Director, Roger Jowell)
The impact of the European Social Survey Data for a Changing Europe (ESS DACE) is captured by the statement made by Professor Sir Roger Jowell. Public policy making requires data that are defensible in scientific terms and relevant in societal terms. The ESS, in round 7 by the end of the ESS DACE project responds to both of these terms. Its impact
The dissemination activities are varied but may be categorised as follows
- conference presentations (for example, at the ESS International Conference in Cyprus, the results were presented by CST members, as an example, Matsuo, H. & G. Loosveldt (2012) Assessing data quality in interviewer collected observable data: interviewer burden perspective from Round 5. Paper presented at ESS Cyprus, 22-25 November 2012; Stoop, I., A. Koch & H. Matsuo (2012) Not Able and Not Available: the Other Reasons. An exploratory research not, Prepared for the international non-response workshop’, Ottawa, 4-6 September 2012.
- workshop participation for knowledge and best practice exchange (for example, with European Commission Agencies for example, Matsuo, H. (2013) Nonresponse bias detection and adjustment through calibrated models: case study on Dutch reluctant respondents in ESS4. Presentation prepared for Expert meeting Eurofound, Dublin, 21-22 January, 2013
- submission for publication to relevant international journals (for example, Loosveldt, G. & K. Beullens (2013), ‘ How long will it take?’ An analysis of interview length in the fifth round of the European Social Survey, Survey Research Methods, Vol. 7, No 2.)
- The Topline Results Series (see below) - four issues
- The ESS Policy Seminar Series (see below) - six events
Across its 13 Work Packages, the ESS DACE results have been formally captured and recorded in Deliverables. All Work Package were successfully concluded in line with Annex I. The learning and results arising from the implementation of tasks within WPs cumulatively informed the later scheduled Deliverables throughout the lifetime of the project. This learning has been captured and will continue to be shared with the wider survey community (both academic and practitioners) thereby reinforcing the impact of the project beyond its immediate implementation.
Scientists across the ESS network have been active in disseminating the results to the relevant user communities, in particular, for example, at the European Survey Research Association (ESRA) and international conferences on research methodology. In addition, scientists prepared papers for publication in relevant journals; for example, the results of WP10 Improving the Efficiency, Usability and Functionality of on-line access to ESS data and protocols, specifically relating to the estimation of measurement error is included in the revised edition of Saris W and Gallhofer, Design, evaluation and analysis of questionnaires for survey research, Wiley (forthcoming, 2014). The ESS Service Activity allows full access to the growing ESS data archive and on-line and direct training tools support use of the research infrastructure. Results from the ESS DACE project have been presented at national and international conferences, including at the European Survey Research Association conference and at AAPOR and WAPOR.
Four issues of the ESS Topline Results Series were published. The series is targeted at policy makers and the wider community. This has been widely disseminated to the ESS community in hard copy and via the ESS website in PDF. The ESS Topline Results Series was complemented by the ESS Policy Seminar Series, of which six seminars were held over the lifetime of the project.
ESS Policy Seminar Series ESS Topline Results Series
Seminar 1: Trust in Justice: How the ESS can inform policy, M23/May 2012, Centre for European Policy Studies (CEPS) Brussels
Seminar 2: Attitudes to Welfare, M30/November 2012, European Policy Centre (ECP), Brussels
Seminar 3: Work, Family and Wellbeing, M31/January 2013, OECD, Paris
Seminar 4: Europeans’ Understanding and Evaluations of Democracy, M47/May 2014, OECD, Paris
Seminar 5: Europeans’ Understanding and Evaluations of Democracy, M51/September 2014, Italian Parliament, Rome
Seminar 6: Europeans’ Personal and Social wellbeing – Findings from the European Social Survey Rounds 3 and 6, M51/September 2014 Issue 1: Trust in Justice: Topline Results from Round 5 of the European Social Survey, (December 2011)
Issue 2: Attitudes to Welfare: Topline Results from Round 4 of the European Social Survey (November 2012)
Issue 3: Economic Crisis, Quality of Work and Social Integration: Topline Results from Round 2 and Round 5 of the European Social Survey (April 2013)
Issue 4: Europeans’ Understanding and Evaluations of Democracy: Topline Results from Round 6 of the European Social Survey (September 2014)
The ESS DACE project was instrumental in ensuring continuity of the ESS at a time of transition. Led by the host country, the UK, the application for the legal status ‘European Research Infrastructure Consortium’ (ERIC), was successfully realised in November 2013. This award was recognition of the ESS as a research infrastructure of pan-European relevance, serving the European and wider research, policy and other user communities. At the end of the ESS DACE project period, 23 countries had formally agreed to participate in ESS ERIC as it implemented Round 7. The learning arising from the ESS DACE Project and its results in relation to survey coordination and survey methodological rigor and quality are incorporated into the ESS ERIC as it consolidates its position as a research infrastructure of pan European relevance.
List of Websites:
www.europeansocialsurvey.org