This study triangulated measurement of informal science learning using three approaches: objective testing of scientific content knowledge, self-reporting of learning, and open questions which asked for specific examples of learning. A survey was piloted in 2017 and then three surveys were conducted in 2018, administered by the first author, using the same surveying methodology for all. Surveys were created and hosted in SurveyGizmo^TM. The study was approved by the Human Research Ethics Committee at the University of Otago (17/062). The sections of the surveys that provided data analysed in this manuscript are attached as Supplementary Material.

The research instrument comprised five multiple-choice questions focused on light and electromagnetism, key topics showcased in Tūhura, plus a control question that was not included in the exhibits (Table 1). Multiple-choice questionnaires can be used to assess content knowledge (Brady, 2005; Kahan et al., 2012). All the items had one answer that was right, two that were wrong, and an extra “I don’t know”. The questionnaire was created by the authors and was iteratively reviewed by a panel of experts in science communication.

The score of scientific content knowledge in light and electromagnetism was calculated as the sum of right answers (1 for each right answer, 0 for incorrect answers, not including the control question). “Don’t know” options were counted as incorrect, (Salmi et al., 2015).

A short two-item test (plus a control question) was piloted in 2017 at Discovery World. The number of right answers increased significantly (Wilcoxon Signed Rank Test, Z = 5.816, p < 0.001, r = 0.389, 89 discordant pairs of 224) from a median of 0 before the visit to 1 (out of 2) after the visit.

Given the formal nature of this instrument in an informal setting, alienation could be a concern. To minimize alienation, a number of approaches were taken: 1) Questions were selected such that the risk of conflict between the questions and visitor worldviews were minimal ² , 2) The survey was as short as possible, 3) The person administering the survey welcomed visitors and was friendly and respectful when asking for participation, responding to all questions from parents and children, 4) Respondents were given enough space to fill out the survey without feeling observed or pressured, 5) Places to sit were provided), 6) iPads were used to survey (Section 2.2.3), 7) A token was given to respondents on completion, as a sign of appreciation (Section 2.2.4).

There were few signs of bias (e.g. children feeling everything is five stars) or visitor alienation (e.g. skipping questions in the survey), giving confidence to add more questions to a final five-item (plus control) questionnaire that was then conducted in 2018 at Tūhura. The questions from the pilot were included in the final version of the test. The control question in the pilot became an actual question in the final version of the test, as its topic was not covered in an exhibit in Discovery World, but it was in Tūhura. A new control question was added to the final version. This questionnaire (Table 1) was asked in what hereafter is called Survey A. Although all items were related to light and electromagnetism, they cover multiple subtopics and it cannot be expected that someone who learns about one, knows about the others. In other words, the multiple-choice test is not a scale and scientific knowledge is not necessarily mathematically unidimensional, nor a concrete construct. The Kuder-Richardson Formula 20 (KR-20) coefficient (equivalent to Cronbach’s alpha for dichotomous values, such as right/wrong) was 0.506 before and 0.542 after the visit.

The scientific content knowledge test is able to quantify learning, but does not capture non-content learning, or content learning outside the specific questions asked. To provide a measure of whether visitors themselves believe they have learned and how they learned, Environmetrics Pty Ltd. created the Modes of Learning Inventory (MOLI), a 10-item, five-point, Likert-type scale developed by Griffin et al. (2005). MOLI was designed to be conducted only once, after a visit. For the present research, reversed items and those considered complicated for children were dropped. The remaining six items (Table 2) were included in the after-the-visit Survey B. As expected, the subset was still unidimensional ³

Since cognitive changes are highly individual and difficult to assess in a standardized way, outcomes need to be assessed in a variety of ways (National Research Council, 2009). Individuals are capable of understanding and self-reporting their own learning ⁴ (National Research Council, 2009; Falk and Needham, 2013; Colliver and Fleer, 2016). Directly asking a visitor if they learnt something new is one way to assess changed knowledge and understanding (Longnecker et al., 2014). In Survey C, visitors were asked “Do you consider you learnt something at Tūhura’s exhibits that you did not know before? (including any previous visits)” (Yes/No/I haven’t interacted with Tūhura’s exhibits). In total, 276 said Yes and 78 said No ⁵ . Those who said Yes were asked “Can you give an example of something you learnt?”. Examples were given by 196 respondents. In addition, “It was cool learning about … ” was an open question included in Survey A, answered by 394 participants. Qualitative responses from both surveys are provided as examples of learning.

To combat the view of some young people that science is boring (Linder et al., 2010), the first generation of interactive museums started in 1969 with the Exploratorium in San Francisco and the Ontario Science Centre in Canada (Patiño, 2013). Since then, interactivity has been expected to be a key variable in learning science at a science centre, as interactive elements are more attractive to visitors (McKenna-Cress and Kamien, 2013), promote learning (Fenichel and Schweingruber, 2010), and make the experience more memorable (Maxwell and Evans, 2002).

It is important to define what is meant here by interactivity. Hands-on interactives are those where the user interacts with their hands, but interactivity is a much broader concept, as broad as the ways a visitor can influence an exhibit’s functioning. For example, Tūhura showcased an infrared camera. To interact with it, visitors do not need to touch anything. The simple act of standing in front of the camera makes the exhibit change what is displayed on the screen (the temperatures of the visitor’s body). However, interactivity does not occur until the user completes the cycle of interaction; in this example, the cycle is complete when the visitor pays attention to what the screen is displaying.

Learning is a complex process that is influenced by a multitude of factors, such as age and gender (Wehmeyer et al., 2011). However, conclusions about the relationship between these variables and learning vary. For example, Ramey-Gassert (1997) concluded that both children and adults learn science at science centres, but Allen (1997) found a very different result. Allen interviewed visitors who interacted with a “coloured shadows” exhibit ⁶ to see if they provided more correct answers to questions about the nature of those shadows (asked during the interview and later assessed). The success rate in getting the correct answers after an intervention was null for visitors under 12 years old, very small for those between 13 and 15 years old, and only considerable for those 16 and above (Allen, 1997).

Since learning occurs more readily if there is some prior knowledge and the topic resonates with the visitor (Krajcik and Sutherland, 2010; Falk and Dierking, 2016; Mattar, 2018), prior knowledge (operationalized in this research as the score in the pre-knowledge test) was another variable to study. Comparison of results of pre and post answers to survey questions with answers to a control question which asked about information that was not included in the science centre exhibits provides greater confidence that differences observed after the visit were indeed indications of learning. Even if science learning is one of a venue’s primary objectives, it is not necessarily on a visitor’s free-time radar (Burns and Medvecky, 2016). The three surveys also asked why visitors came to Tūhura (pre-visit) and what they actually did during their visit (post-visit). The option “Interact with the exhibits” appeared in the pre- and post-test surveys to measure how many originally disengaged visitors became engaged with the exhibits. Complementing, but not paired between surveys, “Learn some science” was a pre-visit option and “Read some panels” was a post-visit option. These questions were added to potentially explain other results as complementary factors to interactivity.

Visitors of all ages come to Tūhura, but given that survey questions require a certain maturity to be answered correctly, it was decided to limit participants to those over a minimum respondent age. According to the National Research Council (2009), children older than seven years are able to respond to questionnaires, but the age limit for this study was increased to eight years old, as seven to eight is the age when children enter the “concrete operational stage” in Piaget’s theory of cognitive development (Piaget, 1968).

We acknowledge that emotional reactivity and regulation are age-related (Silvers et al., 2012), but as mentioned in Section 2.1.2, questions were designed to minimize alienation. The format was explicitly designed and tested for being easy for younger respondents while not being patronizing for adults. This allows use of age as a variable in statistical comparison of changes in content knowledge. More detail about development of instruments directed at children as well as adults can be found in Solis (2020).

The MOLI instrument and the open questions do not require a comparison between two points of time and were only asked in the corresponding post-survey. The knowledge questionnaire matched participants’ pre-test and post-test responses, allowing true comparison (Friedman, 2008; Hernández et al., 2014) to assess changes in scientific literacy of visitors.

A strategy used to avoid alienation involved the use of iPads to administer the surveys. The pilot assessed the formal test and compared the use of iPads versus paper. Visitors commented that the use of iPads was “cool” and paper surveys were only kept for emergency (e.g., if internet was down) or visitors who might prefer paper requested one. None of these scenarios happened, all data analyzed in this study were collected on iPad.

Although sometimes younger visitors needed to instruct older relatives on iPad use, the appeal to use iPads in this informal setting was independent of age, gender and group composition. Compared to paper, electronic surveying produces equivalent results in terms of missing data, item means, and internal consistencies (Giduthuri et al., 2014; Ravert et al., 2015), response rates (Ravert et al., 2015; Shah et al., 2016), and time spent completing the survey (Shah et al., 2016). Moreover, using iPads instead of pencil and paper has advantages such as saving time in responding to closed questions (Giduthuri et al., 2014), presenting a more attractive and uncluttered questionnaire (Fowler, 2013), and allowing randomized presentation of items, which increases reliability of the instrument (Fowler, 2013). Lastly, using the iPads allowed visit time to be recorded. However, this information was of limited use, as the time spent at the relevant exhibits could not be separated from time spent in the Tropical Forest.

As an incentive for answering a formal questionnaire, a small token was given to respondents as a token of appreciation—a small glow in the dark item or a magnetic butterfly. The token was attached to a piece of paper with a scientific fact and it was given after completing the post-survey.

All Tūhura visitors were asked to participate in the surveys provided they were at least 8 years old (with consent of the carer), there were at least two iPads available, and there were enough caretakers in a group to look after the youngest children while other members of the group filled out the survey.

Survey A was conducted from May to August 2018, Survey B in September and October 2018, and Survey C from July to September 2018. Piloting at Discovery World happened in June and July 2017.

Table 3 shows respondent demographics. For ease of interpretation, age was divided into groups: Children (8–12 years old), Adolescents (13–18), Young Adults (19–40) and Mature Adults (41+). Visitors came mainly in family groups (75%), their ethnicity was mainly European (87%) and most (78%) agreed to participate. Response rate was calculated by dividing the number of groups that accepted by the number of groups that were asked.

To be able to compare respondent demographics to those of the general visitor population, visitors (respondents and non-respondents) demographics were also visually assessed (Table 3). The sampling method affected the group distribution because of exclusion of visitors under seven years old.

Ideally, data should be correct, unambiguous and complete (Kimball and Caserta, 2004), but real world data are often inaccurate and need to be cleaned (pre-processed). For example, data quality can be improved by removing survey responses that exceed an acceptable number of missing attributes (Kimball and Caserta, 2004). A method to detect these invalid responses was devised (for full description, Solis, 2020) and data reported in this paper were cleaned. Respectively, for each survey, the number of drop-outs/invalid responses/valid responses with not enough answers in the instrument/and final number of valid responses with enough answers in the instrument, were as follows. Survey A: 45/26/8/456. Survey B ⁷ : 18/12/51 (7)/198. Survey C ⁸ : 24/13/32/354.

To comply with ethics recommendations by the institutions involved, no questions were forced and respondents were allowed to skip any as they so desired. As a result, sample sizes vary for different questions. Of the 198 valid MOLI responses, 23 included up to two missing values (pre and post counted separately), either blanks or I Don’t Know ⁹ . After determining data were missing at random (MAR), missing values were input with Expectation Maximization in SPSS v25. Cronbach’s alpha before and after imputation changed minimally from 0.788 to 0.784. The multiple-choice questionnaire does not form a scale and therefore it is not imputable. Blanks and I Don’t Know responses were counted as incorrect.

Quotes are shown verbatim, with clarifications signaled in brackets. Respondent gender and age in years are reported in brackets after each quote.

Scientific content knowledge about light and electromagnetism increased significantly (N = 456, t (455) = 11.9, p < 0.001) from a mean score of 1.96 correct answers (out of five) before a visit to the Tūhura science centre, to 2.61 after a single visit. Length of visits varied ¹⁰ from 8 min to 3 h:31 min with an average stay of 1 h:52 min. The control question added confidence to this result as there was no change in proportion of right or wrong answers after the visit.

The effect size (d = 0.560, d_CI = 0.068) ¹¹ falls in what Hattie (2009) catalogues as the “zone of desired effects learning”, i.e., learning surpassed what is expected from formal schooling. Although formal education may produce deeper learning than a one-off visit to a science centre, Hattie’s interpretation of Cohen’s d reinforces that informal education can be a powerful ally to formal education.

From the questions from the Modes of Learning Inventory (MOLI), 86% (n = 170) of visitors reported their visit resulted in high or very high learning ¹² . While only 36% (n = 128, N = 356) of Tūhura visitors specifically said that they came to the science centre to learn some science in the pre-visit survey, 78% (n = 276, N = 354) reported in the post-visit survey that they had learned something they didn’t know before. Those who responded yes were asked to give an example.

“Plasma the fourth form of matter was something I knew but almost forgot previously” (F, 33). “Recalling torque and inertia was leanring (a learning) event—need to go back to my physics texts of 40 years ago!” (M, 58). Remembering something we have forgotten or strengthening existing knowledge can be considered learning (Falk and Dierking, 2016). These quotes are evidence that formal and informal education can work together to help people learn and consolidate their learning.

The following two responses exemplify that learning is an individual process: “That you can balance an object on the tourqe (torque) board if you get the object to have a matched tourqe (torque)” (F, 19). “That if you spin the ball in the opposite direction that the disc is spinning, it stays on there longer” (F, 52). These two visitors both caught what the Torque Table exhibit ¹³ was trying to convey. The response of the former appears more conceptual, and she is using the terminology displayed at the panel. The second visitor’s explanation is practical and direct, and her learning may have occurred primarily by experimentation rather than reading the panel.

Any doubt of whether children can learn science by visiting a science centre should consider the following self-reported example of learning: “1. I have learned how to make still objects move at the animation station 2. Through an experiment I have learned how humans conduct electricity 3. I learned that white has many different colours” (F, 9).

The effect of the science centre does not stop with learning science content, visitors can develop a sense of inquiry, as can be appreciated from the following quote: “How you could create white by using the colours “Red + Blue + Green = White”. I wonder if [I] could make white using paints?” (F, 11) ¹⁴ .

The selected MOLI items suggest that learning is not age dependent, as no clear age pattern was found (Children, n = 59, Mdn = 3.83, IQR = 0.83, CI = 0.33; Adolescents, n = 37, Mdn = 4.17, IQR = 0.75, CI = 0.17; Young Adults, n = 42, Mdn = 4.00, IQR = 0.71, CI = 0.17; Adults, n = 58, Mdn = 4.17, IQR = 0.67, CI = 0.17). However, the multiple-choice test provided an opportunity for an objective test of the possible correlation. In results consistent with MOLI, the test found learning at Tūhura was not age dependent (r = 0.023, p = 0.622, N = 451).

To further consider how age relates to learning content knowledge, a LOESS fit ¹⁵ was done on a scientific content knowledge scatter plot before and after the visit to Tūhura (Figure 1) against the independent variable of age. While a LOESS fit does not produce correlation coefficients, it allows us to see two clear sections with roughly linear relationships between scientific content knowledge and age, but with different slopes. The domain of one of the relationships includes Children and Adolescents, while the domain of the other one includes Young Adults and Mature Adults. The independence of age and learning can be visually appreciated in Figure 1 as shapes from before and after are similar, regardless of the age group, with both shifting upwards after the visit.

In contrast, Allen (1997) found considerable science learning from a science exhibit only in visitors 16 years and older. However, that result may be due to the nature of the exhibit that was studied. In “coloured shadows”, how shadows get their colour is counterintuitive and requires a good deal of abstraction—something that does not start to develop until adolescence (Piaget, 1968). Also, prior knowledge is important for learning abstract concepts (Krajcik and Sutherland, 2010).

Figure 1 shows how Tūhura visitors’ prior scientific content knowledge in the topic of this study depended on their age in the range from eight to 22 years old ¹⁶ (r (237) = 0.440, p < 0.001). From the age of 23 there was no further age-related increase in prior scientific knowledge, (r (214) = 0.005, p = 0.938). This finding agrees with Lindon (1996), in that knowledge is accumulated with age, especially in young people. The ages where knowledge increased rapidly is consistent with the typical age of formal schooling. “From eight to 18 years there is great potential for children and young people to extend their knowledge tremendously (Lindon, 1996). Notwithstanding, the parallel upwards shift of curves from pre to post in Figure 1 also demonstrates that the influence of informal learning can be important, even when compared to that of traditional schooling, as has been suggested by Falk and Needham (2013). The increase in scores from pre to post-test at all ages demonstrates that adults continue to learn when provided opportunities outside of school.

The flatter section (from 23 years old) does not mean adults learn less, but that their priorities may tilt their learning to other subjects (Flynn, 2012), not assessed with this instrument (which only measured the topic of light and electromagnetism). Instead of being generalists, adults tend to develop expertise in specific domains (Fenichel and Schweingruber, 2010).

The prior scientific knowledge of males (M = 2.23, SD = 1.40, CI = 0.20) was significantly higher (t (345) = 3.69, p < 0.001, n_m = 185, n_f = 267, d = 0.359, d_CI = 0.096) than that of females (M = 1.77, SD = 1.15, CI = 0.14). Females scoring lower than males in prior scientific knowledge about physics (Figure 2), is consistent with other reports showing a gender gap in scientific literacy unfavourable to females (e.g. Allen, 1997; Skaalvik and Skaalvik, 2004; Kurtz-Costes et al., 2008). A multitude of reasons have been proposed to explain this gap, including low self-esteem in science (Bamberger, 2014), stereotype related issues (Bian et al., 2017) and lack of opportunities (Aikman and Unterhalter, 2007). We agree with the reasons above and discuss another factor.

In Table 4 it is seen that there is no prior knowledge gap in Children; the gender gap starts from adolescence onwards. This difference does not need to come from some sort of discouragement necessarily. On the one hand, engagement is a cornerstone that supports effective science learning and interest in learning more (Krapp and Prenzel, 2011). On the other hand, career choices are influenced not only by confidence and interest in science, but by relative academic strengths (Stoet and Geary, 2018), and it was found in 2015 PISA that boys had a significantly larger rescaled intra-strength in Science, while girls’ intra-strength was in Reading ¹⁷ (Stoet and Geary, 2018).

STEM careers can be divided into two broad categories, physical STEM careers and life sciences STEM careers (Mohtar et al., 2019). It is well documented that girls tend to have less interest in physical sciences than boys (Krapp and Prenzel, 2011). More specifically, females tend to be more attracted to biology and males to physics (Akarsu and Kariper, 2013). An important factor for women’s underrepresentation in physics may be their own choices that start at a young age (Williams and Ceci, 2012) and that are based on having more areas where they feel they can succeed (Mostafa, 2019). A deeper discussion on the gender gap and the relation between content knowledge and self-concept will be presented elsewhere. Regardless of the gap, it is interesting to note that both genders increased their content knowledge significantly, males going up from M = 2.23 to M = 2.84 (t (184) = 7.13, p < 0.001, d = 0.524, n = 185, d_CI = 0.106) and females from M = 1.77 to M = 2.45 (t (266) = 9.51, p < 0.001, d = 0.582, n = 267, d_CI = 0.088). If we take the pre-post difference in right answers (∆M) as a measure of content knowledge learning, females (n_females = 259, ∆M = 0.68, CI = 0.14) are not significantly different (t (423) = 0.180, p = 0.857) from males (n_males = 175, ∆M = 0.66, CI = 0.17). This agrees with Piraksa et al. (2014), who found that gender did not influence scientific reasoning in students in Thailand.

Self-reports are also interesting in this regard. The MOLI responses for males (n = 78, Mdn = 4.08, IQR = 0.83, CI = 0.08) and females (n = 117, Mdn = 4.00, IQR = 0.83, CI = 0.17) were not statistically different (Mann-Whitney U = 4,342, p = 0.564, r = 0.041), but the percentage of females reporting new learning when asked “Do you consider you learnt something at Tūhura’s Exhibits that you did not know before?” (82%, n = 213) was significantly higher (χ2 (1) = 6.37, p = 0.012) than that of males (72%, n = 138). Due to the small sample size of sub-groups, medians instead of means were used. Table 4 complements Figure 2 by showing the results to testing for statistical differences in these subgroups. While adult female visitors increased their test scores more than adult male visitors, no statistical difference was found in children.

Tūhura visitors who interacted with exhibits changed their answers significantly between the pre and post-test surveys (McNemar-Bowker test χ ² (3,n’ = 1973) = 166, p_asym<0.001, DPRS = 14.0). The non-interacting group did not (χ ² (3,n’ = 127) = 3.628, p_asym = 0.305, DRPS = 0.007). Figure 3 shows this graphically ¹⁸ . The amount of answers that changed ¹⁹ was the same in both groups (33%). However, those who interacted with the exhibits have a large net flow towards the right answer, while the distribution of those who did not interact is more random.

It is important to acknowledge that interactivity is not a factor that works alone. Engagement with the exhibits translates into more time playing with them, and more time at the exhibits means more opportunities for learning (Serrell, 1997). As expected, visitors who interacted with the exhibits stayed (n = 692, t = 67 m 09s, SD = 25 m 02s, CI = 1 m 52 s) significantly longer at Tūhura (t (742) = 3.542, p < 0.001, d = 0.516, F = 0.144) than those who did not interact with the exhibits (n = 52, t = 54 m 26 s, SD = 24 m 14 s, CI = 6 m 34 s). Unfortunately, time spent exclusively at the exhibits could not be isolated from the total which could include time spent in the Tropical Forest.

Another indirect factor that could account for the increased learning by those interacting is the possibility that those interacting also read the panels. But the difference in means of right answers from pre to post in panel readers (ΔM = 0.60) and non-readers (ΔM = 0.60) was not significant (t (425) = 0.544, p = 0.587, n_NR = 115, n_PR = 312, d = 0.061, d_CI = 0.109), meaning that those who did not read the panels were as likely to provide correct answers as those who did. This is predictable to some extent, given the interactive nature of the exhibits, which were designed to be self-explanatory. Another possible factor is that visitors who came with the intention of learning science worked hard towards their aim and their increase in science knowledge was so high that it influenced the results of the entire interacting group. However, the amount learned by those who said they came to learn some science (n = 295, ΔM = 0.64, CI = 1.14) was not statistically different (t (425) = 0.183, p = 0.855, d = 0.03, F = 0.322) from those who stated no intention to learn science in the pre-visit survey (n = 132, ΔM = 0.67, CI = 0.20).