Credit: Pixabay/CC0 Public Domain
Surveys that ask too most of the identical kind of query tire respondents and return unreliable knowledge, in response to a brand new UC Riverside-led research.
The research discovered that individuals tire from questions that modify solely barely and have a tendency to provide related solutions to all questions because the survey progresses. Marketers, policymakers, and researchers who depend on lengthy surveys to foretell shopper or voter conduct can have extra correct knowledge in the event that they craft surveys designed to elicit dependable, unique solutions, the researchers counsel.
“We wished to know, is gathering extra knowledge in surveys at all times higher, or may asking too many questions result in respondents offering much less helpful responses as they adapt to the survey,” mentioned first creator Ye Li, a UC Riverside assistant professor of administration. “Could this paradoxically result in asking extra questions however getting worse outcomes?”
While it could be tempting to imagine extra knowledge is at all times higher, the authors questioned if the choice processes respondents use to reply a collection of questions may change, particularly when these questions use an analogous, repetitive format.
The analysis addressed quantitative surveys of the kind sometimes utilized in market analysis, economics, or public coverage analysis that search to know folks’s values about sure issues. These surveys typically ask numerous structurally related questions.
Researchers analyzed 4 experiments that requested respondents to reply questions involving selection and desire.
Respondents within the surveys tailored their choice making as they reply extra repetitive, equally structured selection questions, a course of the authors name “adaptation.” This means they processed much less data, realized to weigh sure attributes extra closely, or adopted psychological shortcuts for combining attributes.
In one of many research, respondents had been requested about their preferences for various configurations of laptops. They had been the type of questions entrepreneurs use to find out if clients are prepared to sacrifice a little bit of display screen dimension in return for elevated storage capability, for instance.
“When you are requested questions time and again about laptop computer configurations that modify solely barely, the primary two or 3 times you take a look at them rigorously however after that perhaps you simply take a look at one attribute, equivalent to how lengthy the battery lasts. We use shortcuts. Using shortcuts provides you much less data for those who ask for an excessive amount of data,” mentioned Li.
While people are recognized to adapt to their atmosphere, most strategies in behavioral analysis used to measure preferences have underappreciated this reality.
“In as few as six or eight questions individuals are already answering in such a manner that you simply’re already worse off for those who’re attempting to foretell real-world conduct,” mentioned Li. “In these surveys for those who preserve giving folks the identical kinds of questions time and again, they begin to give the identical sorts of solutions.”
The findings counsel some ways that may improve the validity of knowledge whereas additionally saving money and time. Process-tracing, a analysis methodology that tracks not simply the amount of observations but additionally their high quality, can be utilized to diagnose adaptation, serving to to determine when it’s a risk to validity. Adaptation is also lowered or delayed by repeatedly altering the format of the duty or including filler questions or breaks. Finally, the analysis means that to maximise the validity of desire measurement surveys, researchers may use an ensemble of strategies, ideally utilizing a number of technique of measurement, equivalent to questions that contain selecting between choices out there at totally different instances, matching questions, and quite a lot of contexts.
“The tradeoff is not at all times apparent. More knowledge is not at all times higher. Be cognizant of the tradeoffs,” mentioned Li. “When your objective is to foretell the true world, that is when it issues.”
Getting vaccinated for COVID-19 discovered to enhance psychological well being
More data:
Ye Li et al, EXPRESS: The More You Ask, the Less You Get: When Additional Questions Hurt External Validity, Journal of Marketing Research (2021). DOI: 10.1177/00222437211073581
Provided by
University of California – Riverside
Citation:
Surveys with repetitive questions yield dangerous knowledge, research finds (2022, January 28)
retrieved 28 January 2022
from https://phys.org/information/2022-01-surveys-repetitive-yield-bad.html
This doc is topic to copyright. Apart from any honest dealing for the aim of personal research or analysis, no
half could also be reproduced with out the written permission. The content material is offered for data functions solely.