• search hit 14 of 15
Back to Result List

Filtering Survey Responses from Crowdsourcing Platforms: Current Heuristics and Alternative Approaches

  • Information Systems research continues to rely on survey participants from crowdsourcing platforms (e.g., Amazon MTurk). Satisficing behavior of these survey participants may reduce attention and threaten validity. To address this, the current research paradigm mandates excluding participants through filtering heuristics (e.g., time, instructional manipulation checks). Yet, both the selection of the filter and the filtering threshold are not standardized. This flexibility may lead to suboptimal filtering and potentially “p-hacking”, as researchers can pick the most “successful” filter. This research is the first to tests a comprehensive set of established and new filters against key metrics (validity, reliability, effect size, power). Additionally, we introduce a multivariate machine learning approach to identify inattentive participants. We find that while filtering heuristics require high filter levels (33% or 66% of participants), machine learning filters are often superior, especially at lower filter levels. Their “black box”Information Systems research continues to rely on survey participants from crowdsourcing platforms (e.g., Amazon MTurk). Satisficing behavior of these survey participants may reduce attention and threaten validity. To address this, the current research paradigm mandates excluding participants through filtering heuristics (e.g., time, instructional manipulation checks). Yet, both the selection of the filter and the filtering threshold are not standardized. This flexibility may lead to suboptimal filtering and potentially “p-hacking”, as researchers can pick the most “successful” filter. This research is the first to tests a comprehensive set of established and new filters against key metrics (validity, reliability, effect size, power). Additionally, we introduce a multivariate machine learning approach to identify inattentive participants. We find that while filtering heuristics require high filter levels (33% or 66% of participants), machine learning filters are often superior, especially at lower filter levels. Their “black box” character may also help prevent strategic filtering.show moreshow less

Export metadata

Additional Services

Search Google Scholar
Metadaten
Author: Lennard Schmidt, Florian DostORCiD, Erik MaierORCiD
URL:https://aisel.aisnet.org/icis2019/research_methods/research_methods/8
Title of the source (English):International Conference on Information Systems
Document Type:Conference publication peer-reviewed
Language:English
Year of publication:2019
Tag:Amazon MTurk; Effect Size; Filtering; Reliability; Survey Research; Validity
Article number:2294
Faculty/Chair:Fakultät 5 Wirtschaft, Recht und Gesellschaft / FG ABWL, insbesondere Marketing
Einverstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.