ORIGINAL RESEARCH article
Front. Psychol.
Sec. Quantitative Psychology and Measurement
Volume 16 - 2025 | doi: 10.3389/fpsyg.2025.1588579
This article is part of the Research TopicPromoting Replicability: Empowering Method and Applied Researchers in Driving Reliable ResultsView all 4 articles
Quality and Representativeness of Research Online with Yahoo! Crowdsourcing
Provisionally accepted
Katie Seaborn1*
Satoshi Nakamura2- 1Tokyo Institute of Technology, Meguro City, Japan
- 2Meiji University, Suginami, Tokyo, Japan
Select one of your emails
You have multiple emails registered with Frontiers:
Notify me on publication
Please enter your email address:
If you already have an account, please login
You don't have a Frontiers account ? You can register here
Conducting research online has become common in human participant research and notably in the field of human-computer interaction (HCI). Many researchers have used English-language and Western participant pool and recruitment platforms like Amazon Mechanical Turk and Prolific, with panel quality and representativeness known to vary greatly. Less is known about non-English, non-Western options. We consider Japan, a nation that produces a significant portion of HCI research. We report on an evaluation of the widely-used Yahoo! Crowdsourcing (YCS) recruitment platform. We evaluated 65 data sets comprising N = 60, 681 participants, primarily focusing on the 42 data sets with complete meta data from studies requiring earnest participation (n = 29, 081). We found generally high completion (77.6%) and retention rates (70.1%). Notably, use of multimedia stimuli exhibited higher completion (97.7%) and retention (91.9%) rates. We also found that the "general" participant setting attracted middle-aged men, requiring additional requests and filtering to capture a wider audience. We reveal the nature, power, and limitations of YCS for HCI and other fields conducting human participant research.
Keywords: Online sampling, Sampling quality, participant characteristics, Participant pool, recruitment, Research quality, Japan
Received: 06 Mar 2025; Accepted: 15 Jul 2025.
Copyright: © 2025 Seaborn and Nakamura. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
* Correspondence: Katie Seaborn, Tokyo Institute of Technology, Meguro City, Japan
Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.