Expanding the footnote: how minor methodological decisions have impactful consequences for developmental results

Over the last quarter century, there has been an increased emphasis on taking a systems approach to development, especially considering possible mechanisms and non-obvious, often ignored factors driving children's behavior. Although this work has given us greater insight into the mechanisms behind developmental outcomes, considerably less focus has been given to the methodological choices researchers make along the way (at times, leaving some of those decisions at the level of a footnote). However small, these methodological choices impact the validity of our data and thus, the conclusions we can draw from developmental studies. The current manuscript expands on one such choice—that of stimuli—to highlight how the everyday decisions researchers make about the objects and contexts we use in a study may have larger cascading effects than what is anticipated. We use stimuli used in language development studies as an example case. As part of this argument, we review individual bodies of research exploring how salience, typicality, and presentation context of stimuli may uniquely influence children's behavior and subsequently the results in word learning research. We argue that in order for the field of development science to advance and for us to further a holistic understanding of developmental processes and mechanisms, the next steps must consider these often ignored factors and methodological choices.

1 Introduction

In 2009, Spencer et al. (2009) argued for the importance of taking a systems approach to studying development with a particular emphasis on considering “a broader interpretation of experience and an appreciation for the non-obvious nature of development.” In the decade and a half since their publication, more work in the field of development has explored the non-obvious factors that contribute to developmental outcomes, expanding mechanisms to include neurological, cognitive, social, and behavioral processes. Although these studies have given us greater insight into the processes behind developmental outcomes, considerably less focus has been given to the methodological choices researchers make along the way (at times, leaving some of those decisions at the level of a footnote).

Take, for instance, the body of research on children's early word learning. A long-held belief was that exposure to any speech was sufficient to help children build a vocabulary (e.g., Hart and Risley, 1995; Huttenlocher, 1998). However, though the link between overall language exposure and expressive vocabulary remains strong, more recent research has shown that the impact of the input is more nuanced with multiple features and properties being important. This includes findings that being exposed to child-directed speech facilitates children's processing of linguistic input and leads to long-term vocabulary gains (Weisleder and Fernald, 2013) and lexically diverse speech has been shown to aid early word acquisition (Hills et al., 2010).

Critically, this body of research has also moved beyond the input itself to explore the contextual factors that help young children capitalize on word learning opportunities. For example, (Yu and Smith 2012) used head-mounted eye-trackers to assess where 18-month-old children looked during a free-play session with novel objects. They found that the physical constraints of infants' short arms led to moments where one object was dominant in their visual field; if caregivers provided labels in those hot spot moments, infants were more likely to retain the novel word-referent mapping at a later test. More recently, (Schroer and Yu 2022) demonstrated that these pivotal word learning opportunities are also driven by children holding the object; simply looking at an object while hearing its label was not sufficient to support the learning at test.

Collectively, the previous research has shown, then, that word learning is not influenced by a single, obvious factor. Rather, components such as the properties and features of the input as well as the context in which it is presented matter. Knowing that there are many non-obvious factors that influence the word learning process then leads to the question—what else should researchers consider when making claims about what influences language learning? One possibility that has been given considerably less attention is the role of the stimuli, especially their salience, typicality, and presentation context.

2 The challenge of stimuli in word learning

A key (yet often forgotten) part of methodological discussions is the stimuli (images, sounds, contexts, etc.) used. To assess the extent to which developmentalists are currently transparent about their stimuli choices, we conducted a preliminary review of word learning studies published since 2020 in five developmental journals (Child Development, Developmental Science, Developmental Psychology, Infancy, Frontiers in Psychology). Specifically, naive undergraduate coders answered three questions: 1) are the stimuli described in depth?; 2) are the stimuli shared via OSF or similar framework?; 3) were the stimuli normed or from a normed set? While 76.5% of articles had “yes” for at least one question, very few (7%) answered “yes” for all three questions. Less than half of the articles were able to be coded as “yes” for questions individually − 1) 42.9% described stimuli in depth, 2) 32.7% shared the stimuli, and 3) 48.0% used normed stimuli. Taken together, this suggests that though most researchers share information about their stimuli in some form, a notable 23.5% did not share information at all, and most papers may have not provided enough information for other researchers to truly consider the effect that the stimuli may have had.

In many of these studies, the effect of stimuli are an afterthought—while stimuli may be reported, differences in children's behavior because of those decisions are not. There are of course exceptions; some studies include supplements, appendixes, or footnotes about stimuli decisions—see for instance, as noted in Figure 1, (Perry and Saffran 2017) clarify that the type of “duck” being used was a critical factor in order to accurately assess children's knowledge of features in the main study. However, few of the reviewed studies discuss how such seemingly minor decisions about which “duck” to use may have changed the results (in this example, for the better!). Here, we focus on the area of word learning as an example for how the objects themselves and the contexts in which they are presented have important implications for children's behavior and thus, the conclusions we draw.

Figure 1

Figure 1. Footnote of (Perry and Saffran 2017) specifying subtle stimuli details that changed the proportion of color reported. Reprinted with permission.

2.1 Stimuli themselves: what is presented matters

The hallmark tasks of word learning research ask children to select a target from an array by either looking at it or reaching for it. For instance, a child might be given two items and asked to point to the shoe, or find a zup. Growing work has shown that accuracy on such a task may be pushed around due to the features of the items themselves, including how typical a familiar item is. For instance, when the canonical shape of highly familiar items is altered to either be atypical or more sparse (such as using an atypical margarita glass for a “cup” or a sparse gray cylinder shape-caricature), children are less accurate at selecting and slower to reach for those items (Kucker et al., 2023). This is particularly pronounced for younger children and children with language delays who have difficulty recognizing simple representations of known items that are missing rich realistic features (Jones and Smith, 2005; Smith, 2003). Altering a familiar object's canonical color can also lower accuracy—for instance, if a cow is colored pink or a banana purple, children looked at it for significantly less time than they looked at typically colored objects (Perry and Saffran, 2017). Though children are still largely above chance in these studies, demonstrating they still can recognize these “weird” versions of objects, their significantly lower accuracy rates suggests that altering features of items has subtle, but important effects on children's processing of known words and subsequent behavior in standard word learning tasks.

Altering the familiarity of an item also changes children's behavior; children respond differently to objects when they are novel compared to when they are familiar. This is not a new concept to developmental scientists—the classic habituation paradigms rest on the assumption that children respond differently to something new compared to something familiar after all. However, this means that the level of novelty for a given object during word learning will alter children's attention to it. In a study by (Kucker et al. 2018), 18-month-old children were found to be so attracted to unknown objects they reliably chose the novel foil item even when the task was to choose a well-known object such as a cup and ignore the novel one. The simple fact that an item was unknown leads children to react differently in-the-moment despite the linguistic input they received. Even older 3-year children struggle with the saliency of stimuli. (Pomper and Saffran 2019) found that children were much slower to orient and fixate on a novel target when the familiar foil item was highly salient (e.g., realistic cat) compared to when the familiar foil was more boring (e.g., cardboard box). That is, the novelty or saliency of specific items present (even if they aren't the target) alters behavior and performance in word learning paradigms.

Stimuli effects are not limited to referent selection, but can impact retention and learning of names for novel items. Recent work has found that even in cases when all items are novel, there are differences in children's choices and retention accuracy for specific items (e.g., Lorenz and Kucker, 2025; see Figure 2). In this study, all novel items were treated similarly by children during an initial generalization task, suggesting that all were similarly attractive and salient. However, when children's learning for the novel word-referent pairs were tested, children only succeeded in learning the bap and mip. Order of items, task procedure, and various other design elements were fully counterbalanced and thus, cannot account for why children think these items are cool. Current work cannot answer the question of why children find these items easier to learn, leaving a gap in our field.

Figure 2

Figure 2. Differences in children's generalization and retention based on specific stimuli set. Y-axis represent children's percent choice to the target during the novel noun generalization (NNG) trials (a) and during retention (b); x-axis represents unique stimuli sets. Reprinted with permission from Figure S1 of Lorenz and Kucker (2025).

2.2 The context: how stimuli are presented matters

The impact of stimuli goes beyond individual items—children's behavior is also shaped by how those items are presented, including the setting, order, and modality of presentation. For instance, children's understanding of non-solid substances and their generalization of novel names for these items is improved when they are tested in a highchair (a common context for interacting with non-solid food items) compared to a tabletop (Perry et al., 2014). Given that many lab-based tasks are situated at tables, this means the settings we use for testing knowledge may be ill-equipped for capturing that knowledge in some cases. Even the broader social context and who is presenting the toys influence task performance. (Hilton and Westermann 2017) found that shy children performed more poorly on referent selection tasks when the experimenter was a stranger compared to when a familiar friend (e.g., a parent) was presenting the stimuli.

The order of stimuli presentation also matters. In Research Methods 101, we learn to counterbalance the trials of a study to prevent order effects. While this is standard practice in many studies now, we can still see stimuli make differential impacts on children's behavior because of the order in which they are presented. Take (Samuelson and Horst 2007) for instance. In this study, what type of stimuli children were exposed to during the warm-up trials changed what children did during the subsequent test trials. Here, children presented with a novel item as a whole object during warm-up were more likely to attend to shape in the subsequent trials, whereas children warmed up with items in pieces were biased to look more toward material features. Given they used a classic test of children's shape bias, it is a big deal if a minor change in training fundamentally changes the bias a child presents. On a broader level, this means that the ways in which a child became familiar with the task changed how they approached the task, their resulting choices, and ultimately, the researchers' conclusions. Counterbalancing in this case would offset the training effects and likely result in a null effect, but doing so doesn't actually prevent the stimuli from affecting children's behavior or informing our understanding of how this non-obvious factor impacts learning.

Finally, changing the modality by which stimuli are presented changes both the stimuli themselves as well as the context. This has become particularly prominent in the last decade with the rise in the use of 2D digital learning contexts, both in the home and lab. As technology has advanced, we have shifted from the use of real 3D items or puppets to teach children novel words in the lab to presenting information to children through 2D images or videos. This move allows us to capture fine-grained eye movements or neurological data, but it changes the features of the items and the context in which they are presented. This is potentially troubling as we know from a growing body of work that children cannot learn from 2D digital sources as well as they can from 3D sources, and they certainly struggle to transfer information across modalities (a phenomenon known as the transfer deficit; Barr, 2010). As just one example, (Strouse and Troseth 2014) showed that 24-month old children taught a novel word through a video failed to learn the new word unless a caregiver provided extended scaffolding and compared it with a real life version of the object the children held. Multiple meta-analyses and reviews have also confirmed this finding, especially for younger children (Barr, 2010; Jing and Kirkorian, 2020; Strouse and Samson, 2021; Taylor et al., 2024). This means that decisions about the modality of a study context matter—not only does it change children's behavior in the task itself, but it may not be representative of their real-world learning, threatening the validity of our conclusions.

3 Addressing the challenge

Central to all research is the reliability and validity of the methods used. As apparent by multiple recent publications (e.g., Byers-Heinlein et al., 2022; Kominsky et al., 2022; Kucker and Chmielewski, 2022; Zettersten et al., 2022), this has become a particularly salient discussion within the developmental area, especially as the field continues to shed light on the non-obvious factors that might influence our results. Outside of word learning, classic theories in motor development traditionally posited that children navigated visual cliffs due to a fear of heights. However, work by Karen (Adolph 1995) showed that adjusting the slope of a visual cliff creates different affordances and tactile experiences that interact with the child's motor ability which ultimately influences whether or not they descend a slope. Likewise in visual search paradigms, children's attention and processing of stimuli varies based on if the stimuli presented are faces or salient objects, with object saliency diminishing as children age (Kwon et al., 2016). Thus, paying attention to the stimuli and environments we choose for a study is likely relevant for multiple developmental areas.

So if we want to improve our conclusions, regardless of the topic of study, what should developmentalists consider when designing our methods and specifically our stimuli? One idea is that researchers can continue to increase their transparency—not only can we share the datasets associated with our individual studies, but we can also provide detailed descriptions and samples of the stimuli for use by other labs. Part of this is utilizing public databases of stimuli, such as the NOUN database (Horst and Hout, 2016), OSF submissions (https://osf.io/), and Databrary (https://databrary.org/) and if there isn't a validated database or stimuli that works for your purposes, then norm the stimuli prior to running the full study (much like the first steps used when creating standardized measures and assessment tools). By taking such steps we can assure that our stimuli are not just arbitrary choices that can lead to artifacts, but are intentionally selected and quantified in ways that help us reduce noise and enhance focus on the core variables of interest.

Transparency and validation of our stimuli is important not just at the lab level, but across the field. Cross-lab collaborations, such as that of the ManyBabies consortium (https://manybabies.org/), have begun doing just this. In their seminal study, The ManyBabies Consortium et al. (2020) were able to conclude that babies preferred infant-directed speech; importantly, though, they also found that the effect was stronger “in those children for whom the stimuli matched their native language and dialect.” The effect was also specific to the head-turn preference procedure, demonstrating that children's individual experiences interact with the stimuli and context. Looking at these phenomena beyond a single lab and more cross-culturally, allows us to be more mindful of possible effects of methodological choices before making claims about universal developmental milestones, whether in language studies, those that focus on theory of mind (e.g., Schuwerk et al., 2021), rule learning (e.g., Visser et al., 2021), and more.

Moreover, another option for tackling these challenges is to take advantage of the growth in statistical modeling available to us. For instance, in addition to systematically testing for order effects, researchers can use mixed-effects models to account for item (and even participant) variability (Baayen et al., 2008; Muradoglu et al., 2023). Accounting for that variability in our analyses should enable us to more clearly see the throughline, or answers to, the research questions at the heart of developmental science even when the stimuli and methods may vary.

Many of these are things we have been aware of for years and may seem obvious, but may be ignored or dismissed as not relevant or applicable to our specific study. But, the evidence suggests it might indeed be critical to consider these details about our stimuli and methods. In short, we need to go back to our Methods 101 lessons—remembering that the way we ask our questions, the sequence of those questions, and what we present, all alter behavior and present possible confounds that can cloud our understanding of the true variables of interest. So as we look forward to the next quarter century for our dynamic field, let's move beyond the footnote to collaborate and share methods and stimuli that will allow us to draw stronger conclusions about cognitive development.

Data availability statement

The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author/s.

Author contributions

SK: Writing – original draft, Writing – review & editing. ML: Writing – original draft, Writing – review & editing.

Funding

The author(s) declare that no financial support was received for the research and/or publication of this article.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Gen AI was used in the creation of this manuscript.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Adolph, K. E. (1995). Psychological assessment of toddler's ability to cope with slopes. J. Exp. Psychol. Hum. Percept. Perform. 21, 734–750. doi: 10.1037//0096-1523.21.4.734

PubMed Abstract | Crossref Full Text | Google Scholar

Baayen, R. H., Davidson, D. J., and Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. J. Mem. Lang. 59, 390–412. doi: 10.1016/j.jml.2007.12.005

Crossref Full Text | Google Scholar

Barr, R. (2010). Transfer of learning between 2D and 3D sources during infancy: informing theory and practice. Dev. Rev. 30, 128–154. doi: 10.1016/j.dr.2010.03.001

PubMed Abstract | Crossref Full Text | Google Scholar

Byers-Heinlein, K., Bergmann, C., and Savalei, V. (2022). Six solutions for more reliable infant research. Infant Child Dev. 31:e2296. doi: 10.1002/icd.2296

Crossref Full Text | Google Scholar

Hart, B., and Risley, T. (1995). Meaningful Differences in the Everyday Experience of Young American Children. Baltimore, MD: Brookes.

PubMed Abstract | Google Scholar

Hills, T. T., Maouene, J., Riordan, B., and Smith, L. B. (2010). The associative structure of language: contextual diversity in early word learning. J. Mem. Lang. 63, 259–273. doi: 10.1016/j.jml.2010.06.002

PubMed Abstract | Crossref Full Text | Google Scholar

Hilton, M., and Westermann, G. (2017). The effect of shyness on children's formation and retention of novel word–object mappings. J. Child Lang. 44, 1394–1412. doi: 10.1017/S030500091600057X

PubMed Abstract | Crossref Full Text | Google Scholar

Horst, J. S., and Hout, M. C. (2016). The novel object and unusual name (NOUN) database: a collection of novel images for use in experimental research. Behav. Res. Methods 48, 1393–1409. doi: 10.3758/s13428-015-0647-3

PubMed Abstract | Crossref Full Text | Google Scholar

Huttenlocher, J. (1998). Language input and language growth. Prev. Med. 27, 195–199. doi: 10.1006/pmed.1998.0301

PubMed Abstract | Crossref Full Text | Google Scholar

Jing, M., and Kirkorian, H. L. (2020). “Video deficit in children's early learning,” in The International Encyclopedia of Media Psychology, 1–8.

Google Scholar

Jones, S. S., and Smith, L. B. (2005). Object name learning and object perception: a deficit in late talkers. J. Child Lang. 32, 223–240. doi: 10.1017/S0305000904006646

PubMed Abstract | Crossref Full Text | Google Scholar

Kominsky, J. F., Lucca, K., Thomas, A. J., Frank, M. C., and Hamlin, J. K. (2022). Simplicity and validity in infant research. Cogn. Dev. 63:101213. doi: 10.1016/j.cogdev.2022.101213

Crossref Full Text | Google Scholar

Kucker, S. C., Braun, B. E., and Markham-Anderson, J. F. (2023). Margarita glasses and high heels: how attention to shape, age, and vocabulary impacts children's recognition of typical and atypical exemplars. Child Dev. 94, 603–616. doi: 10.1111/cdev.13883

PubMed Abstract | Crossref Full Text | Google Scholar

Kucker, S. C., and Chmielewski, M. (2022). Yes, but … we need reliability to advance infant work, but there's more to consider too: important nuances of reliability and the need to include validity. Infant Child Dev. 31:e2325. doi: 10.1002/icd.2325

PubMed Abstract | Crossref Full Text | Google Scholar

Kucker, S. C., McMurray, B., and Samuelson, L. K. (2018). Too much of a good thing: how novelty biases and vocabulary influence known and novel referent selection in 18-month-old children and associative learning models. Cogn. Sci. 42, 463–493. doi: 10.1111/cogs.12610

PubMed Abstract | Crossref Full Text | Google Scholar

Kwon, M. K., Setodehnia, M., Baek, J., Luck, S., and Oakes, L. S. (2016). The development of visual search in infancy; attention to faces versus salience. Dev. Psychol. 52, 537–555. doi: 10.1037/dev0000080

PubMed Abstract | Crossref Full Text | Google Scholar

Lorenz, M. G., and Kucker, S. C. (2025). To touch or not to touch: the role of vocabulary and object exploration in children's attention to shape. Infancy 30:e12632. doi: 10.1111/infa.12632

PubMed Abstract | Crossref Full Text | Google Scholar

Muradoglu, M., Cimpian, J. R., and Cimpian, A. (2023). Mixed-effects models for cognitive development researchers. J. Cogn. Dev. 24, 307–340. doi: 10.1080/15248372.2023.2176856

Crossref Full Text | Google Scholar

Perry, L. K., and Saffran, J. R. (2017). Is a pink cow still a cow? Individual differences in toddlers' vocabulary knowledge and lexical representations. Cogn. Sci. 41, 1090–1105. doi: 10.1111/cogs.12370

PubMed Abstract | Crossref Full Text | Google Scholar

Perry, L. K., Samuelson, L. K., and Burdinie, J. B. (2014). Highchair philosophers: the impact of seating context-dependent exploration on children's naming biases. Dev. Sci. 17, 757–765. doi: 10.1111/desc.12147

PubMed Abstract | Crossref Full Text | Google Scholar

Pomper, R., and Saffran, J. R. (2019). Familiar object salience affects novel word learning. Child Dev. 90, e246–e262. doi: 10.1111/cdev.13053

PubMed Abstract | Crossref Full Text | Google Scholar

Samuelson, L. K., and Horst, J. S. (2007). Dynamic noun generalization: Moment-to-moment interactions shape children's naming biases. Infancy 11, 97–110. doi: 10.1207/s15327078in1101_5

Crossref Full Text | Google Scholar

Schroer, S. E., and Yu, C. (2022). Looking is not enough: multimodal attention supports the real-time learning of new words. Dev. Sci. 26:e13290. doi: 10.1111/desc.13290

PubMed Abstract | Crossref Full Text | Google Scholar

Schuwerk, T., Kampis, D., Baillargeon, R., Biro, S., Bohn, M., Byers-Heinlein, K., et al. (2021). Action anticipation based on an agent's epistemic state in toddlers and adults. PsyArXiv. doi: 10.31234/osf.io/x4jbm

Crossref Full Text | Google Scholar

Smith, L. B. (2003). Learning to recognize objects. Psychol. Sci. 14, 244–250. doi: 10.1111/1467-9280.03439

PubMed Abstract | Crossref Full Text | Google Scholar

Spencer, J. P., Blumberg, M. S., McMurray, B., Robinson, S. R., Samuelson, L. K., and Tomblin, J. B. (2009). Short arms and talking eggs: why we should no longer abide the nativist–empiricist debate. Child Dev. Perspect. 3, 79–87. doi: 10.1111/j.1750-8606.2009.00081.x

PubMed Abstract | Crossref Full Text | Google Scholar

Strouse, G. A., and Samson, J. E. (2021). Learning from video: a meta-analysis of the video deficit in children ages 0 to 6 years. Child Dev. 92, e20–e38. doi: 10.1111/cdev.13429

PubMed Abstract | Crossref Full Text | Google Scholar

Strouse, G. A., and Troseth, G. L. (2014). Supporting toddlers' transfer of word learning from video. Cogn. Dev. 30, 47–64. doi: 10.1016/j.cogdev.2014.01.002

PubMed Abstract | Crossref Full Text | Google Scholar

Taylor, G., Sala, G., Kolak, J., Gerhardstein, P., and Lingwood, J. (2024). Does adult-child co-use during digital media use improve children's learning aged 0–6 years? A systematic review with meta-analysis. Educ. Res. Rev. 44:100614.

Google Scholar

The ManyBabies Consortium Frank, M. C. Alcock K. J. Arias-Trejo N. Aschersleben G. Baldwin D. . (2020). Quantifying sources of variability in infancy research using the infant-directed-speech preference. Adv. Methods Pract. Psychol. Sci. 3, 24–52. doi: 10.1177/2515245919900809

Crossref Full Text | Google Scholar

Visser, I., Geambasu, A., Baumgartner, H. A., Bergmann, C., Byers-Heinlein, K., Carstensen, C. A., et al. (2021). Many babies 3 rule learning—Stage 1 Registered Report—In principle acceptance. doi: 10.31234/osf.io/aex7v

Crossref Full Text | Google Scholar

Weisleder, A., and Fernald, A. (2013). Talking to children matters: early language experience strengthens processing and builds vocabulary. Psychol. Sci. 24, 2143–2152. doi: 10.1177/0956797613488145

PubMed Abstract | Crossref Full Text | Google Scholar

Yu, C., and Smith, L. B. (2012). Embodied attention and word learning by toddlers. Cognition 125, 244–262. doi: 10.1016/j.cognition.2012.06.016

PubMed Abstract | Crossref Full Text | Google Scholar

Zettersten, M., Pomper, R., and Saffran, J. (2022). Valid points and looks: reliability and validity go hand-in-hand when improving infant methods. Infant Child Dev. 31:e2326. doi: 10.1002/icd.2326

Crossref Full Text | Google Scholar