Here is the CMT Uptime check phrase

Achieving Comparability

Have questions or comments for the authors? Click here to post a question or comment about this session. Please specify which author/presentation your post is addressing.


Applying the Total Survey Error Paradigm to Comparative Surveys: Comparison Error Lessons from the International Social Survey Programme (ISSP)
Tom W Smith, NORC at the University of Chicago

Abstract: Cross-national and other comparative surveys need to make their study designs, data collection procedures, and other survey elements as comparable as possible to maximize the possibility that measured differences represent true variation and not measurement differences. Examples are drawn from the ISSP to illustrate cases were success, failure, and to be determined have occurred. The list below shows the different survey components examined and the governing rules in ISSP’s Working Principles.

  1. Acceptable sample designs – “The sample is a national representative probability sample of the adult population without substitution…”
    1. quota samples – never allowed
    1. substitution – rules clarified to exclude use of substitution
  2. Mode – “The questionnaire should be suitable for self-administration, but can be administered in a face-to-face interview as well. Telephone interviews are generally not acceptable.”
  3. Weighting – “Each country needs to supply the Data Archive with a clear description of the weighting procedure, what justification there is for weighting the data, and what the weight variable is designed to accomplish. This description should be in English, and must give enough detail so that non-members can understand the conditions under which weighting should be applied.”
    1. Weight or not weight
    1. Design weights – Number of eligible respondent per household
  4. Validation of interviews and interviewers – No requirements
    1. Duplicates and near duplicates
    1. Training and checking procedures
  5. Other

In Harmony: Comparison of the European Social Survey and the European Values Study Questionnaires
Eva Aizpurua, European Social Survey HQ – City, University of London
Angelica Maineri, Tilburg University
Rory Fitzgerald, ESS ERIC – City, University of London
Vera Lomazzi, GESIS – Leibniz-Institute for the Social Sciences
Ruud Luijkx, Tilburg University

Abstract: The European Social Survey (ESS) and the European Values Study (EVS) are large, cross-national social surveys that collect data in most European countries. As part of the ESS-SUSTAIN-2 project, both teams are exploring the possibility of collecting EVS data as part of the ESS infrastructure. The suggested strategy consists of bridging compatible measures when possible and potentially designing a 30-item module measuring EVS core questions to be occasionally fielded in the ESS. This presentation summarises the first step undertaken to assess the feasibility of this proposal, based on the comparison of item characteristics from the source versions of the ESS and EVS questionnaires. After identifying 75 pairs of items covering similar concepts, the items were compared using 17 attributes adapted from the Survey Quality Predictor (SQP) framework. The attributes were organised in four domains: 1) Question attributes (e.g., reference period, part of a battery), 2) Interviewer role (i.e., clarifications and instructions), 3) Response (e.g., number of categories, scale symmetry), and 4) Showcards (i.e., content and layout). The findings from this comparison revealed a high degree of consistency, especially for the socio-demographic variables, which accounted for 55% of the potentially compatible items. For substantive variables, however, there was larger variation. In addition to systematic design differences (e.g., 10- vs 11-points scales), discrepancies related to the stimuli offered to respondents (showcards) and to the number and range of categories were the most frequent mismatches. In-depth examples are used to illustrate the results of this comparison, which was conducted to inform an empirical comparison of the ESS and EVS most recent data (ESS Round 9: 2018/2019 and EVS Wave 5:2017/2018). The findings of this methodological work will be of interest for researchers interested in survey harmonisation and cross-national surveys.

Examining Measurement Invariance of Depression among Male and Female in the China Health and Retirement Survey
Mengyao Hu, University of Michigan
Edmundo Roberto Melipillán, Universidad del Desarrollo

Abstract: With the increasing popularity of cross-cultural research and comparison studies, survey researchers are facing a difficult problem: responses to the same scales obtained from different population groups may not always be comparable. This paper examines the measurement invariance of the 10-item version of the Center for Epidemiological Studies Depression (CES-D) Scale across male and female. Data are drawn from the baseline wave of the China Health and Retirement Survey (CHARLS), a national survey conducted biennially with a sample of the Chinese population who are 45 years of age or older. The final sample size includes 15,977 respondents; 53.2% of whom are female. The mean age for the sample is 58.3 (SD = 10.2). Given that the question items in the CES-D scale use ordinal indicators, measurement invariance (MI) tests based on Multiple Group Categorical Confirmatory Factor Analyses (MGCCFA) was performed. Results show that full scalar model was not supported, and question items variant and invariant across groups were identified. These results indicate that any comparisons between the means across male and female not accounting for the noninvariance in the identified set of thresholds could be biased, highlighting the importance of performing MI tests before conducting mean comparisons across groups.

What Drives Item Nonresponse in a 3MC Context?
Katharina Meitinger, Utrecht University
Timothy Johnson, University of Illinois at Chicago/NORC

Abstract: A potential source of non-comparability in cross-cultural survey research is the risk of differential rates of item nonresponse, which may be attributed to differences in data collection procedures and also to differences in cultural perceptions of question meaning or sensitivity. A modest body of literature has confirmed variability in item nonresponse propensity across cultures (Owens, Johnson & O’Rourke, 2001; Lee, Li & Hu, 2017). Our previous research revealed that minority status affects the willingness to report sensitive information during survey interviews in the ISSP. However, item nonresponse might also be driven by Basic Human Values. We address this possibility with data from the European Social Survey. Using HLM models, we examine the effects of minority status and Basic Human Values on rates of nonresponse after adjusting for individual, survey and country level characteristics. Implications for the interpretation of survey findings and recommendations for best practices will be discussed.