Evaluating citizen science projects: insights from radon research

Abstract

Citizen science projects have garnered attention for their potential to engage the public in scientific research and address societal challenges. However, assessing their impacts has often been overlooked or approached with overly simplistic methods. Aiming to fill this gap, this article draws on existing literature to propose an evaluation framework to critically examine how citizen science initiatives influence science, society and the participants themselves. This framework is tested on four citizen sciences projects in the field of radon research through content analysis of project reports and deductive analysis of 11 semi-structured interviews with citizen scientists and coordinators of the projects. The study demonstrates the feasibility of measuring the impacts of citizen science projects across scientific, participant, societal and researcher dimensions at the outcome level but also process evaluation at the process level. Our findings indicate that the proposed framework provides a comprehensive evaluation tool for citizen science projects, particularly in the field of radon research, and underscore the significant potential for improving participants’ knowledge on radon and risk mitigation strategies, as well as positive shifts in behaviour towards testing and mitigation and influencing public health policies.

1 Introduction

Evaluating citizen science projects is essential for ensuring their effectiveness, enhancing planned outcomes, improving data quality, supporting scientific, educational and policy goals, among others. Citizen science (CS) has been evaluated based on the achievement of different outcomes: “outcomes for research (e.g., scientific findings); outcomes for individual participants (e.g., acquiring new skills or knowledge); and/or outcomes for social–ecological systems (e.g., influencing policies, building community capacity for decision making)” (Shirk et al., 2012: 1). However, comprehensive standardized frameworks which systematically assess the impacts of CS are scarce, either there is a lack of a common framework (Kieslinger et al., 2018) or they are too simplistic (Somerwill and Wehn, 2022) or overlook relevant advancements in the field (Wehn et al., 2021).

In the field of public health risk radon, the evaluation of particular radon CS initiatives has mainly focused only on participants gains such as risk awareness and knowledge (Hahn et al., 2020; Stanifer et al., 2022; Tsapalov et al., 2020). Other aspects, for instance scientific and societal impacts or sustainability, have been overlooked. In a recent study, Martell et al. (2021) revealed that CS projects in the field of radon are few, despite the World Health Organization (2009) indicating that this radioactive indoor air pollutant is a severe public health problem. Martell et al. (2021) evaluated eight citizen science projects identified through a systematic review (covering the period 1984–October 2020) utilizing the ten principles of the European Citizen Science Association (ECSA). These projects were evaluated on the scientific outcome of projects, benefits to scientists and participants, ethical and legal considerations, data accessibility, scientific output, and the level of citizens engagement. However, a comprehensive framework and practical guidance for assessing a variety of CS projects impacts is still lacking.

The research question guiding this study is: what criteria and indicators can help to better evaluate the impact of CS projects, particularly those focusing on radon research? The purpose of this article is twofold: first, to develop a comprehensive evaluation framework based on existing CS literature, and second, to utilize the RadoNorm CS projects in the field of radon to test and refine this framework. This will include improving indicators for evaluating the effectiveness of CS projects and contributing to public health from multiple perspectives. The framework proposed assesses the impact of CS in the field of radon on five dimensions, enabling evaluation of both outcome and process level: 1) scientific, 2) participants, 3) wider societal impact, 4) learning gains to researchers and 5) process evaluation.

2 Theoretical background: citizen science impact evaluation

The literature on CS impact evaluation reveals diverse approaches and frameworks employed to assess the effects of projects on different measures. Some studies have assessed CS impacts broadly on society, the economy, the environment, science, technology, and governance (Wehn et al., 2021; Bonney et al., 2014), whilst others have focused on the gains experienced by individual participants, such as learning outcomes, behavioral changes, scientific skills and personal empowerment (Stanifer et al., 2022; Trumbull et al., 2000; Phillips et al., 2018; Den Broeder et al., 2017; Davis et al., 2023; Kieslinger et al., 2018; Von Gönner et al., 2023). Stanifer et al. (2022) evaluated the impact of a CS approach in home radon testing on participants using indicators such as environmental health literacy (knowledge of radon exposure as a health hazard), health information efficacy (ability to search and process radon information), response efficacy (the belief that health threats can be identified via radon testing) and self-efficacy (respondents confidence in their ability to perform home radon testing, contacting a radon professional and hiring a radon professional).

In terms of scientific impacts, various indicators are used to assess the effectiveness of CS projects, including peer-reviewed publications, considerations of data quality, publication of data and results in specific media outlets and online repositories, and the generation of new research topics or new knowledge resources (Davis et al., 2023; von Gönner et al., 2023; Kieslinger et al., 2018). The European Union’s Horizon 2020 research project Monitoring Impact of Citizen Science (MICS) include metrics such as citation counts for project publications, impact indices or publication impact factors, support for student dissertations or theses, and strategies for disseminating project outcomes (MICS platform online).

Another significant aspect of assessing the impact of CS projects is the socio-ecological and economic impact, which is measured through indicators such as societal impact, ecological impact, and wider innovation potential (Kieslinger et al., 2018). These broader societal impacts extend beyond individual participants and scientific outputs, often providing evidence for policy formulation and decision-making.

The learning gains to participants can be situated within the context of learning theories in education. CS initiatives exemplify free-choice learning, where participants pursue research based on personal interests and motivations. For student participants, where the emphasis is on learning by doing, experiential theory of learning comes into play. According to Kolb and Kolb (2009), experiential learning is characterized by acquiring knowledge through direct experience, with reflection on those experiences being crucial. In the context of radon where participants learn how to use radon devices in their homes, this is applicable also to adult participants.

3 The case of public health risk radon

Radon, a naturally occurring radioactive gas resulting from the decay of uranium in rocks and soil, is odorless, colourless and tasteless (World Health Organization, 2009). Radon gas can enter into buildings, particularly homes, through cracks in the foundation, walls, floors, and other openings. Hence, it can accumulate in high concentrations within indoor environments such as dwellings and workplaces.

Exposure to indoor radon presents a significant health risk, as radon is ubiquitous and is also the main cause of lung cancer after tobacco smoking. Due to the unique characteristics of radon, testing of an indoor radon concentration is the only way to determine associated exposure risks. While some European countries show higher number of measurements of indoor radon and more detailed surveys in high-radon areas (e.g., the Czech Republic, Germany, Switzerland, United Kingdom) in the Digital Atlas of Natural Radiation by the Joint Research Centre (JRC),¹ others have either reduced their anual measurement campaigns or need to update their data (European Commission: Directorate-General for Energy et al., 2023). The voluntary participation of citizens to measure radon in their homes represents a main challenge to achieve significant radon testing rates (Mc Laughlin et al., 2022). Furthermore, mitigation rates remain low (Hevey et al., 2023). A recent empirical study conducted in Slovenia, involving a representative sample of respondents from low, middle and high radon risk areas (N = 2012), suggests that radon interventions should go beyond enhancing awareness, knowledge and risk perception. Instead, they should evoke emotions, share personal stories, highlight successful mitigation cases, include testimonies from affected individuals and incorporate positive social norms to inspire more people to engage in testing and mitigation measures (Perko et al., 2024). CS could be recognised as such intervention.

CS has emerged as one of the promising means to increase radon testing and mitigation rates (Hahn et al., 2020; Martell et al., 2021), thereby safeguarding public health. Through CS, citizens may become empowered to actively participate in research, motivating them to take protective action. To explore this potential, the European funded research project RadoNorm established a “CS Incubator.” At the onset of this initiative, four CS pilot projects were developed across four European countries, France, Hungary, Ireland, and Norway over a period of 6 months to test a CS model for radon testing and mitigation in Europe (Martell et al., 2024).

Having implemented these pilot projects, a fundamental question that arises and which is the core focus of this article is: how effective have these four CS projects been in generating impacts and what criteria and indicators can help to better evaluate the impact of CS projects? Such evaluations are crucial to assist policymakers in improving CS funding schemes, enhancing project management practices, and, most importantly, extracting lessons for future CS initiatives (Kieslinger et al., 2018). This is particularly important in CS projects in the field of radon, as these projects address an important public health issue and comprehensive evaluations are rare (Martell et al., 2021).

3.1 Method: data collection and analysis

For this study, we conducted a non-systematic literature review on CS impact evaluation which is presented in the results section and we utilised the four RadoNorm CS pilot projects to test and improve the indicators of the evaluation framework. Qualitative techniques of semi-structured interviews and content analysis of project reports served to test and improve the evaluation framework.

The interviews included both participants and coordinators (who are also researchers) of the four CS pilot projects and hereafter referred to as researchers. We employed protocols tailored to each specific group of interviewees, that is, researchers, adult participants, and student participants. The interview protocol was pre-tested with an adult and a young adult. Purposive sampling was utilized to select participants for the interviews, with inclusion criteria being all individuals involved in the CS project, regardless of their level of activity. Participation in the interviews was voluntary, and was communicated to potential participants via email. Prior to the interviews, participants completed an informed consent form, which provided details regarding the use of the results, assured participants of their anonymity and informed them of their right to terminate the interview at any point without explanation. A total of nine interviews were conducted with eleven people, in accordance with the minimum requirements for interviews as stipulated by Vasileiou et al. (2018). This ensured representation from both participants and researchers. Two of these interviews were conducted jointly with two researchers and two students respectively. The interviews were conducted in English, French, and Hungarian with interview questions relating to the role of citizen scientists and researchers, their experiences and recommendations for CS. The English and Hungarian interviews were carried out online, whilst the interview in French was conducted via telephone due to the participant’s technological limitations. The interviews in French and English were conducted by some authors of this article. The Hungarian interview was conducted by an external researcher not related to this project but with experience in scientific research. The interviews were conducted between late November 2003 and early February 2024, with durations ranging from 35 min to slightly over an hour. The interview transcripts were then coded deductively in the NViVo software based on the indicators presented in Supplementary Tables S3-S5. Peer debriefing with an experienced qualitative researcher was used for research validity.

4 Results

4.1 Developing the evaluation framework

The evaluation framework was developed based on the systematic review by Martell et al. (2021), and a non-systematic review of CS impact evaluation and was tested and improved using the RadoNorm CS projects. Hence, the framework takes stock of the works of Kieslinger et al. (2018), Martell et al. (2021), Phillips et al. (2018), Von Gönner et al. (2023), Wehn et al. (2021). The framework proposed assesses the impact of CS in the field of radon on five dimensions, enabling evaluation of both outcome and process level: 1) scientific, 2) participants, 3) wider societal impact, 4) learning gains to researchers and 5) process evaluation. With outcome-based evaluation, the goals of activities or projects, the benefits to participants, and recipient of results are assessed whereas in process evaluation, the operational strengths and weaknesses of activities or projects are identified (Kieslinger et al., 2018).

The sustainability of CS projects is not explicitly addressed as a separate dimension or indicator since the assessment questions across the five described dimensions already encompass this type of evaluation. According to the European Commission (2006), “a project is sustainable when it continues to deliver benefits to the project beneficiaries and/or other constituencies for an extended period after the Commission’s financial assistance has been terminated.” The evaluation of sustainability proposed by the European Commission in higher education cooperation projects follows two main criteria: 1) diversity and intensity of either activities/outputs or outputs and outcomes maintained, developed and/or disseminated and 2) intensity or enlargement of the cooperation. Therefore, the sustainability of CS projects can be assessed based on project results and the continuity of the outcomes and the visibility of results beyond the conclusion of initial funding.

For the scientific dimension (Table 1), indicators relate to scientific publications, the science outcome in terms of whether the CS project answers a specific research question, data and data management, generation of new knowledge and other means of sharing the CS project. Additionally, we incorporate the scientific productivity indicator, a composite measure evaluating scientific output over a defined period. Scores are assigned based on weighted criteria, with an overall score ranging from 30, indicating lower productivity to 42, reflecting higher productivity (see MICS platform for details). In terms of disseminating outcomes through alternative channels other than scientific publications, a score of 0 suggests no dissemination while a score of 6 signifies strong dissemination efforts (MICS online tools).

On the participants’ dimension (Table 2), we assess indicators that gauge the learning outcomes and empowerment experienced by participants, including aspects such as behaviour and ownership, knowledge, skills, interest, motivation and engagement, and self-efficacy. With respect to behaviour, we focus on observable actions related to radon testing, mitigation, re-testing and intention to mitigate, as intention serves as a determinant of actual behaviour according to the theory of planned behaviour (Ajzen, 2011). Regarding knowledge gains, we assess citizen scientists’ knowledge and awareness of radon in general, and on mitigation strategies whereas skills relates to scientific enquiry and the use of scientific devices such as radon detectors. Interest in science assesses the degree to which participants assign relevance to the topic of radon, and science in general in terms of their contribution to scientific work and whether they recommend CS to others. We assess motivation as an outcome and not initial motivation to participate in the CS project (see Phillips et al. (2018) for the distinction). Motivation and engagement encompass increased motivation via participation in the CS project, self-esteem and empowerment of citizen scientists. The final indicator on this dimension is self-efficacy, assessed as participants’ confidence to engage in radon testing and mitigation, engage mitigation professionals and general ease of participation in the CS project. It is also crucial to evaluate participants’ learning or experiences regarding non-participants’ attitudes toward scientific topics of relevance to society, such as the topic of radon. Similar to the indicators used to gauge participants’ interest and attitudes towards the scientific topic of study, comparable metrics should be applied to assess participants’ learning about the attitudes of non-participants with whom they discuss the CS project.

At the broader societal impact level (Table 3), indicators assess the collective capacity of participants to achieve shared objectives, the influence of the CS project on policy formulation and decision-making processes, and the use or development of new technologies such as the assembling of low-cost sensors for radon testing. Our research emphasises the importance of including public outreach of CS projects, which can occur at three levels or circles of influence: a) the participants themselves; b) the individuals directly influenced by the participants, including family members (e.g., the number of individuals within households), friends, and colleagues and c) the general public through press coverage and the overall visibility of the project (Van Brussel and Huyse, 2019).

Our research proposes to include a critical assessment of the learning impact on researchers, such as project management skills, communication and interpersonal skills and the development of expertise in CS research (Table 4). Hence, at the outcome level, the learning gains for researchers are assessed based on the benefits to researchers (e.g., increased visibility, improved communication skills) and expertise gained in CS, for example, through literature reviews and CS workshops.

In the process evaluation (Table 5), indicators include the level of engagement of citizen scientists (e.g., involvement of citizens in the different stages of the research process), feedback mechanisms (e.g., feedback to citizen scientists on project outcomes), legal and ethical considerations (e.g., informed consents, ethical approvals), funding programmes (e.g., availability of funding for CS projects), collaboration and synergies (e.g., with schools, experts from other disciplines, corresponding programmes such as energy saving, air quality, cancer or anti-smoking programmes, new European Bauhaus initiative, etc.), and the evaluation of CS projects, whether conducted internally or externally. This dimension of process evaluation can be linked to support mechanisms (as referred to by Von Gönner et al., 2023) essential for ensuring the effective implementation and success of CS projects.

4.2 Impact of the RadoNorm citizen science pilot projects on science

A number of scientific publications were produced from the CS pilot projects, a year after their closure (see Supplementary Table S1). The CS project in Hungary has also supported a PhD thesis in Physics in the area of science education.

Each CS project sought to answer a specific research question in radon risk management, as illustrated in Table 6. All of these questions have been answered to a large extent.

In addition, all four CS projects generated valuable data in the form of guidelines, communication supporting materials, and toolkits aimed at enhancing awareness and knowledge about radon, particularly concerning testing and mitigation measures. These resources and Do-It-Yourself (DIY) toolkits were co-created through various workshops and meetings with citizen scientists. These are adaptable for future CS projects. The data generated from the pilot projects (Table 7) will be accessible online via the RadoNorm toolbox on STOREDB, ensuring open access in accordance with European open science policy and European General Data Protection Regulation. The French data is openly accessible online on STOREDB.² Besides journal publications, various communication channels were utilized to effectively disseminate project outcomes including conferences, meetings, webinar, project designed websites³ or webpages, Facebook, project reports and deliverable. Through the use of a diverse array of communication channels, the projects significantly expanded their outreach and amplified the impact and visibility of their research.

In terms of generating new proposals, the project in Hungary is ongoing and has enhanced the low-cost sensors to improve user-friendliness and to enable real-time data publication via Wi-Fi connection. Additionally, there are plans to expand radon measurements to multiple sites. Furthermore, based on the experiences of the pilot projects, RadoNorm funded and helped to launch six new CS projects on radon risk management consisting a diverse network of citizens in six European countries. In France, the project team has submitted a proposal, in collaboration with partners, addressing radon in response to an open call. The proposed project emphasises a CS approach, facilitated by the achievements of the RadoNorm CS pilot project. Indeed, “we propose a project, we build a consortium with the partner, and the project is underpinned by the citizen science approach and we could not have done that without having achieved the RadoNorm CS pilot project” (CO4). Further initiatives, albeit without concrete timelines include replicating the pilot project in a different region of France and transforming the online guide into a smartphone application with the assistance of computer-literate citizens.

For the final indicator on scientific impact in the framework, we estimated the scientific productivity indicator of the CS projects, as detailed in Supplementary Table S2. The projects achieved a score of 22, comprising 10 points for publications and citations, and 12 points for other communication and dissemination efforts related to the CS project. The score of 12 for the latter indicates successful sharing of the project, extending learning beyond the project’s scope. However, the score of 10 for publications suggests a limited contribution to scientific literature.

4.3 Impact of the RadoNorm citizen science pilot projects on citizen scientists learning and empowerment

The learning and empowerment outcomes of the pilot projects are illustrated showing both self-reported data and researchers’ perspectives from the interviews. Analysis of self-reported evidence reveals that the most frequently reported impact was behavioral action, entailing testing, mitigation, or both, followed by skill acquisition, increased interest in science, motivation and engagement, enhanced radon knowledge, understanding of mitigation techniques, and self-efficacy. However, a more substantial reference to knowledge is apparent when improved radon knowledge and knowledge of mitigation are considered together. The pilot CS project served as their initial exposure to information about radon for some citizen scientists; “I had never heard of radon. It was the first time I’d taken part in this type of activity. I could take part in similar radon projects in the future, why not? I learnt that radon can be dangerous to human life” (PA9).

The first action on behaviour motivated by the CS projects is indoor radon testing. The self-reported evidence in Supplementary Table S3 shows that all participants interviewed revealed action on radon testing. Testing for radon was carried out either for a week or 12 weeks. In Norway, the CS project launched a radon measurement campaign between October 2022 and January 2023, aimed at addressing the relatively low rates of radon testing in a priority area. A total of 97 households were successfully recruited to participate in the radon measurement campaign (Tomkiv and Anjum, 2023), marking a significant increase in testing activities within the targeted municipality. By encouraging homeowners to test their homes for radon in a high priority area, the campaign effectively stimulated a change in behavior towards radon testing. These efforts not only raised awareness about the importance of indoor radon testing but also contributed to a greater understanding of radon exposure risks within the community.

The outcomes reveal that participants in the CS projects engaged in various actions following radon measurements in their homes. While some undertook mitigation measures when concentrations exceeded reference levels (e.g., sump and fan, either through DIY approach or with professional assistance, Dowdall and McKittrick (2023)), others adjusted their ventilation behaviours (Olahne Groma, 2023). Furthermore, a citizen scientist with high radon concentration levels who had not yet taken action expressed intention to do so: “we have talked about it that we want to do something there, for example to have the air outside or something” (PA7). To ensure the effectiveness of mitigation actions in safeguarding themselves and their families, participants undertook re-testing. “We repeated the measurements and analyses and the radon level dropped” (PA9). Another participant explained: “as we were like with a monitoring because I had the monitor there at that stage, but I was really surprised to see that the results were literally immediate that the level kind of dropped down” (PA6). Indeed, this participant continues to monitor indoor radon levels with an active radon monitoring device as of the time of the interview: “But I still use that monitoring device as well. Just out of curiosity, even though it's always given me good news which is low levels and, in the greens, so it's nice to know” (PA6). For citizen scientists interviewed, participation in the CS project induced a feeling of responsibility and ownership. “I think at least the ones that took part, they feel more responsibility, to also telling others about it, and talk about it to the people you have around you” (PA7); Whether it's colleagues or family or friends or something like that as well, just advising them like a little bit you know to get the 3 month monitoring done and kind of go from there. It’s nice to know. Ohh yeah, kind of like an advert or advocate for it now. Just passing on the good news you know” (PA6); “I discussed this project with my friends and convinced them to take action” (PA9); “I was very excited to participate in this project, and talked about it a lot” (PA10).

Participants gained skills related to scientific inquiry, the use of scientific devices, and other practical skills such as using Excel, data analysis and developing communication skills as shown in Supplementary Table S3. In the interviews, scientific inquiry skills were mentioned as frequently as the use of scientific devices. Self-reported evidence of these learning gains included statements such as: “I kind of did a bit of research, then I knew obviously getting the ground sump with the extraction fan on it would be the only way to do with the particular design and year of build of my house” (PA6); “I’ve already like for example made adjustments to the speed levels of the fan, things like that, and then just retest it to see if there would be, you know, maybe an increase if I slow the speed of the fan down too much. But you know, it was fine, and it had obviously a positive impact as it reduced the noise down a little bit more but not risking kind of with increasing radon levels so which was good, it’s more just experimenting you know” (PA6); “Our students started to watch on internet as well to try to find which sensors will be good” (PA8).

On motivation and engagement, participants expressed how their involvement in the CS projects boosted their self-esteem and empowered them to take protective measures against high radon levels. All participants interviewed expressed their willingness to participate in similar future activities, indicating that their involvement in the pilot projects has increased their motivation to engage in such endeavours. Some statements include: “So we understood that it was a bigger project in Europe as well, yeah, and that’s, of course, that’s very good, yeah. That’s very nice to know that, I can participate in that, that I can be a little important” (PA7). “It’s interesting to be when it’s especially as I say, something that’s kind of so new as well umm you kind of feel you are first, first, to get a little bit of extra knowledge” (PA6). “I suppose just Peace of Mind, really. And I think that’s the biggest thing, just to know that it's something that’s been ruled out and I know exactly how it’s done. So if there’s any, ever any replacement parts, for example on the fan or anything like that, I know exactly how everything was done because I was there” (PA6). “I find this experience very useful as it helps me to decide where to go for further studies in which direction. Now I can see if research is for me or not” (PA10). “I am interested in receiving more information on other projects in Europe or other studies of the same style” (PA9).

In terms of self-efficacy, participants across all four projects demonstrated their ability to perform radon testing with radon measurement devices. In Ireland for instance, citizen scientists felt highly confident in explaining mitigation procedures to others following the DIY co-design workshop (Dowdall and McKittrick, 2023). The interviews revealed citizen scientists’ efficacy in mitigation efforts and their ease of participation in the CS projects. However, the references to this indicator were limited in number. Regarding interest in science, it was observed that a predominant interest was expressed towards the scientific topic of radon, among others.

The CS projects implemented a variety of activities that either raised awareness on radon and/or improved radon knowledge among participants. These activities included educational workshops, distribution of informational materials, public meetings, workshops and hands-on training sessions. Through active engagement in these activities, citizen scientists gained a deeper understanding of the risks associated with radon exposure and learned effective mitigation strategies. For example, students in Hungary received comprehensive education on radon. Public information meetings covered topics such as radon exposure risks, the importance of mitigating elevated radon levels, and DIY measures for reducing radon in homes (Dowdall and McKittrick, 2023; Tomkiv and Anjum, 2023). Some meetings also addressed strategies to encourage community involvement in radon mitigation efforts (Tomkiv and Anjum, 2023). Participants also gained knowledge on mitigation, and how to engage with mitigation professionals and firms for assistance. This is supported by self-reported evidence as presented in Supplementary Table S3, indicating significant learning gains among citizen scientists. Some testimonials related to learning gains as reported by citizen scientists include:

“What touched me the most was the fact that I had never heard of radon but it can have an impact in the home and on the health of many people. This impacted me. And that it is the second cause of lung cancer, after tobacco” (PA9).

“Now we know much more about radon and understand better its risk”(PA10).

“I suppose, not really knowing until I found out a little bit more about it, because not knowing, I suppose the effects and that this can have on people as well” (PA6).

“Not a lot as regards health. If I would have known for example from a building point of view….But as regards the effects or anything like that of radon, I wouldn’t have known anything about, like, you know, levels of how harmful It is or isn’t, you know” And so yeah, kind of help both ways. Yeah, learn a little bit more, I say what I heard health wise, but also from a building point of view, what you can do to improve it in the house and to reduce levels” (PA6).

“We also had that meeting in our town as part of the project and we talked about which mitigation firms you can contact and they could say that it was OK, because many people want to remediate and have money, just money, but they do not know which firm to contact” (PA7).

We would open the window, build in ventilation systems, and if the level is very high, we would put thicker base for the houses (PA10).

4.4 Wider societal impact of the RadoNorm citizen science pilot projects

The pilot CS projects have yielded broader societal impacts, including the collective capacity of participants to achieve common goals, provision of tools and data for agenda setting and decision-making, development of simple technologies and generation of economic benefits. For instance, the radon guide in Norway is readily accessible to the public through the website of the intermunicipal company (Tomkiv and Anjum, 2023), enhancing its potential to increase radon awareness, knowledge, and mitigation strategies among visitors. Additionally, the homeowners association in Norway is willing to share the guide with all of their about 200,000 homeowners (ibid). Professional unions, such as contractors union in Norway would like to improve the radon information guide by incorporating perspectives from professional contractors. It is worth noting that the CS project in Norway revealed a lack of trust in actors who offer radon mitigation (Tomkiv and Anjum, 2023) and in response to this, the head of the radon contractors’ union in Norway has expressed an interest in writing an article for distribution to all union members with the aim of addressing issues of public trust concerning contractors and propose strategies for improving trust levels to enhance mitigation rates and stimulate business growth. Thus, the CS data generated in Norway serves as a critical driver for facilitating extensive discourse and informed decision-making within the area of radon mitigation among industry stakeholders.

In Ireland, there has been a notable increase in the sale of DIY toolkits for radon mitigation. This surge in sales can be attributed, at least in part, to the widespread dissemination of the DIY mitigation video on YouTube, which has been viewed over 7,500 times within a year (as of 16th May 2024). Additionally, referrals to the video by sales personnel promoting such toolkits may have further contributed to this trend. The rise in sales of these mitigation toolkits has potential significant economic benefits, as it not only boosts revenue for suppliers but also generates income for those involved in the distribution and retail sectors of such materials. One respondent highlighted the impact of the mitigation video on consumer behavior, stating: “We have come a long way with aids that are helping, and I believe the video has been watched quite a lot, and I think there’s an increase in the sales of the do-it-yourself kits. So the sales of the do-it-yourself kits, I know there’s one contractor when he sells them, he directs people to the video” (CO1).

In Hungary, student citizen scientists created low-cost sensors capable of measuring various air quality components simultaneously. This initiative can be classified as the development of new technology. Through this process, students not only developed, set up, and tested the toolkit, fostering the creation of user-friendly technologies, but also gained valuable hands-on experience, an opportunity they may not have otherwise had due to limited financial opportunities (Olahne Groma, 2023). A first-hand account of this states, “And of course they don't learn anything about the instrumentation, the measurements setup, and how to build and so on”(CO3). Furthermore, through the collaboration with high schools, particularly on the public health issue of radon, the CS project has the potential to reach a broader population, including the families, friends, and relatives of participating students and teachers. This equips them with information to engage in discussions at local, municipal, regional, or national levels on such issues, particularly in a population with low knowledge and awareness about radon that needs to be increased and deepened as highlighted in the National Radon Action Plan (2018) (Martell et al., 2022). Subsequently An example of the potential societal impact is illustrated by this statement: I think for me the easiest to reach citizens is through teachers who work with students, so it’s an easy, easy way for communication and reaching people” (CO3).

The CS project in France facilitated the dissemination of a wealth of information from experts to citizens and vice versa. This enabled citizens to comprehend radon information effectively, thereby contributing to the enhancement of the radon mitigation guide. Consequently, the technical findings yielded were both constructive and comprehensive, offering valuable insights to guide short and medium-term modifications and developments of the radon mitigation guide (Andresz and Schieber, 2023). Additionally, the CS project did prompt a reassessment of the communication strategies of experts, advocating for more adaptable messaging and enriched pedagogical approaches in future initiatives (Andresz and Schieber, 2023). As such, the data generated by the CS project would be instrumental in facilitating the framing of future radon messaging to citizens. Moreover, the CS project improved the guide’s efficacy in fostering post-measurement action against radon exposure, amplifying its visibility and audience reach (Andresz and Schieber, 2023), thus emphasizing its broader societal impact.

4.5 Learning outcomes of the coordinators of the RadoNorm citizen science pilot projects

Across all four CS projects, the coordinators have gained insights from the public perspective regarding radon and mitigation, broadening their understanding of the challenges and beliefs shaping attitudes towards radon. Indeed one respondent said: “getting this like knowledge from other actors, knowledge that we don't think about or finding out about the challenges that others are facing that we can't even imagine because we always look from our own perspective, from our own framing, from academia”(CO2). Another respondent stated: “We expected that the citizens will have this point of view and that point of view and in fact as it is the citizens did not. They had really another point of view” (CO4). By engaging citizens at various stages of the research process, researchers enhance their connection with diverse community groups, thereby potentially expanding their visibility within the academic and public spheres. One respondent mentioned: “we got invited, like we got booked for a conference in November 2023 to come and talk to radon mitigation contractors and I mean others, researchers, people from industry, and representatives of the House Owners Association as keynote speakers” (CO2).

Other learning opportunities and benefits are outlined in Supplementary Table S4. Among them, gaining practical experience on CS had more references, followed by participatory tools for expertise. For three out of the four coordinators, this was their first time coordinating a CS project and they gained knowledge about CS through different means as shown.

4.6 Process evaluation for citizen science in radiation protection

Across the four projects, collaborations were multifaceted, involving partners at local, national, and international levels. While three projects focused primarily on local and national collaborations, the French project engaged with another country (Switzerland). However, regardless of the scope, the outcomes of these collaborations hold significant potential for informing radon mitigation strategies and enhancing communication approaches. Moreover, all partners serve as pivotal facilitators in disseminating the project outcomes (Martell et al., 2022). In addition, collaborations with other programmes, such as energy efficiency, anti-smoking, indoor air quality or cancer programmes was lacking, replicating the results from evaluating national radon action plans in EU Member States which indicate no systematic links with relevant corresponding programmes (European Commission: Directorate-General for Energy et al., 2023).

Ethical advice or approval was diligently obtained for data collection across all four pilot projects, either from the RadoNorm Ethical Committee or external sources like universities or national research centres. Citizen scientists mostly completed forms including expression of interest and consent forms for the research itself and or for the use of citizen scientists’ pictures, ensuring compliance with data protection regulations (Martell et al., 2022). Confidentiality was maintained through the non-publication of internal project documents, and personal data were either not collected or anonymized to safeguard privacy (Martell et al., 2022).

Citizen scientists assumed diverse roles across the pilot projects, engaging in a range of activities. Depending on the objectives of the pilot projects, citizen scientists collaborated in co-creating either measurement toolkits, mitigation toolkits, or strategies and aims of the CS project. In all the CS projects, participation was voluntary and the form of participation was the choice of citizen scientists. A summary of their self-reported level of engagement is provided in Supplementary Table S5. Across all four countries, citizen scientists contributed by collecting radon measurements and promoting awareness of the CS projects to others. In most cases, citizen scientists were also involved in recruiting citizens to conduct radon testing in their homes. For instance, in Norway the first 20–30 participants for the project’s radon measurement campaign were recruited by the citizen scientists (Tomkiv and Anjum, 2023). Only student citizen scientists were involved in data analysis and disseminating project results.

Regarding feedback in the context of radon measurements, citizen scientists received their results and possible action to mitigate in case of high levels (Martell et al., 2022). Furthermore, the results of the CS project in France, including recommendations and suggestions for the guide, along with findings from CS applications, were compiled into slideshows and text documents throughout the project (Andresz and Schieber, 2023). These outcomes were presented to participants during a wrap-up meeting in July for validation and feedback.

On funding, reactions were mixed. While some researchers during the interviews thought it could be easy to access funding for CS projects, others thought otherwise. Overall, funding was considered necessary for sustainability of CS projects. RadoNorm provided funding for the CS projects which was essential for the implementation of the projects.

The evaluation of CS projects varied internally, with assessments conducted either by citizen scientists or researchers themselves, while one project lacked evaluation altogether. For instance, the French pilot administered a feedback questionnaire to citizen scientists, guided by RadoNorm partners and existing literature (Phillips et al., 2018). This questionnaire included a tailored section designed to gather insights from diagnostic experts, a group from which feedback was typically scarce (Andresz and Schieber, 2023). In one CS project, internal evaluation was absent, while in two others, project coordinators conducted self-evaluation, documenting lessons learned and or the extent to which project objectives were met (Supplementary Table S6). In Norway, coordinators justified their reliance on self-evaluation, citing the impracticality of formal internal assessments by participants due to the project’s adoption of an escalator engagement model (Skarlatidou and Haklay, 2021). This model provided citizen scientists with the autonomy to choose activities aligned with their interests and needs (Tomkiv and Anjum, 2023).

Supplementary Table S7 provides the summary of the evaluation results for each country on all dimensions and indicators.

5 Discussion

The study provides evidence of the measurable impacts of CS projects in the field of radon on all three outcome dimensions: scientific, participants and socio-ecological dimensions as outlined in (Kieslinger et al., 2018), It also aligns with the framework of public participation in scientific research by Shirk et al. (2012). The study demonstrates the feasibility of measuring the impacts of CS projects across scientific, participant, and societal dimensions at the outcome level but also process evaluation at the process level. The framework presented herein and the suggested improvements, by incorporating researchers’ learning gains as an additional dimension coupled with other dimensions, offers a structured approach to assess CS projects’ effectiveness, particularly in radon research and radiation protection. Thus, this research contributes valuable insights applicable not only to radon-specific projects but also to similar research within related fields.

The key findings on the scientific dimension reveal that the research questions of the pilot projects predominantly centered on radon mitigation, attempting to address a relevant gap in the current literature whereby most research on radon has focused on testing rather than mitigation (Mc Laughlin et al., 2022). Nevertheless, radon testing was performed by participants as that is the first measurable action to determine concentration levels and assess the subsequent need for mitigation measures. Regarding co-authorship, citizen scientists were acknowledged in project results and publications but were not included as co-authors. This finding has already been critically assessed by Martell et al. (2021), as it seems standard practice. A data management plan was crafted for the pilot project in France. This aspect holds particular importance as data management practices in CS projects are frequently overlooked and remain scarce (Bowser et al., 2020; Hansen et al., 2021). It is worth noting that the RadoNorm CS pilot projects have resulted in the funding of new CS projects focusing on radon, the generation of a new proposal in response to an open call, and the emergence of new research questions. Even though the CS projects have not yet yielded many scientific publications, their scientific publications are significant, considering that the CS projects resulted in two publications within the time frame of a year after the end of the CS projects. In fact, Follett and Strezov (2015) also observe that most CS projects need time to result in scientific publications.

In line with findings from other studies, the CS pilot projects increased participants’ knowledge, awareness and informed behavioural actions such as testing and mitigation. For instance, the French pilot project emphasised its contributions to increasing participants’ understanding of radon risks in homes, building diagnosis, mitigation strategies, as well as fostering scientific inquiry skills and behaviour change (Andresz et al., 2023). Similarly, in their assessment of a CS approach to home radon testing in four rural Kentucky counties, Stanifer et al. (2022) observed an increase in environmental health literacy among citizen scientists. Additionally, Brossard et al. (2005) conducted a study on the Birdhouse network project, demonstrating a significant impact on participants’ understanding of bird biology. In general, the CS projects motivated participants to change their personal behaviour regarding testing, ventilation and other mitigation measures, resulting in increased radon testing and mitigation in priority areas. This demonstrates the potential of the CS approach to significantly contribute to radon risk management. This finding is consistent with Hahn et al. (2020), where radon testing rates were reported to have increased through the engagement of citizen scientists in research.

Furthermore, in Ireland, 6 out of 9 citizen scientists carried out mitigation as compared to a previous finding where 1 out of 5 did so (Dowdall et al., 2016). Citizen scientists were supported financially to mitigate, with no cost to them together with the support of a radon contractor as compared to the former case mentioned (Dowdall et al., 2016) where private householders bore the cost of both testing and mitigation (Martell et al., 2024). This suggests the necessity of further assessing mitigation rates in CS projects with and without funding support to determine whether participation drives mitigation actions or if financial support plays a significant role. Additionally, it underscores the importance of governmental or CS project support for mitigation actions.

In general, the findings regarding participant gains are consistent with the outcomes of the VegeSafe and DustSafe CS programs (Isley et al., 2022), which focus on metal concentrations in homes (specifically soil in gardens and dust from vacuum cleaners) and were evaluated within the framework of Kieslinger et al. (2018). For instance, the CS results align with those of Isley et al. (2022), who demonstrated that involvement in these CS projects led to various benefits for participants, including increased knowledge, changes in personal behaviour, a feeling of responsibility or ownership toward the project, and motivation to participate in similar or future studies. Notably, participants also expressed the benefit of peace of mind from understanding the levels of metal concentrations and measures to mitigate elevated levels, as reflected in the current findings on radon.

Besides these traditional measures of skills gained as commonly measured in CS impact evaluations, citizen scientists also expressed gaining other skills such as using Excel, creating diagrams, analysing data, and communicating effectively with others.

The involvement of students in radon-related research offers a ‘learning by doing’ experience, serving as a pivotal educational process with the capacity to extend its impact to a wider audience through these students. Indeed, schools are viewed as potential multipliers where teachers and educators assume crucial role of intermediary experts by engaging as participants, facilitators and motivators in CS initiatives (Kloetzer et al., 2021). For instance, Tsapalov et al. (2020) assessed a CS approach in a radon survey, RadonTest online system project in Israel that actively engaged students. Their findings indicate that involving students through a CS approach in radon research significantly enhances the dissemination of crucial information regarding radon risks across populations and administrations at various levels. Furthermore, incorporating schools into CS activities holds particular significance, as it not only cultivates scientific literacy among students but also has the potential to motivate or increase interest in STEM (science, technology, engineering, and mathematics) careers (Heggen et al., 2012). The involvement of students and local communities also fosters inclusivity in CS beyond the typical well-educated, affluent individual audience (Ruiz-Mallén et al., 2016).

On the influence of the CS projects on policy making, not much has been achieved yet, similar to concerns raised by Von Gönner et al. (2023), even though the pilots resulted in wider societal impacts as previously mentioned in the results. Particularly in comparison to other CS projects like the Evict Radon CS project which contributed in forming the essential rationale for legislative action, in the form of the first ever Radon Awareness and Testing Act: Bill 209 (Martell et al., 2021) or the CS study on insect biomass trends that led to the adoption of the new German Insect Protection Law, BMUV 2019 (Von Gönner et al., 2023). Nonetheless, not much CS results have been incorporated into political decision making until now (Hecker et al., 2018; Nascimento et al., 2018; Von Gönner et al., 2023). Furthermore, the testimonial regarding the Irish project underscores the perceived impact of the mitigation video on the increased sales of DIY kits, with sales personnel and or contractors actively recommending the video as a supplementary resource. While these qualitative insights offer valuable anecdotal evidence suggesting that the CS project may have contributed to the observed rise in sales of mitigation toolkits thereby, generating economic impacts, hence, contributing to a wider societal impact. It is important to note that other contributing factors may also be at play. Therefore, attributing the hike in sales solely to the CS project may not be supported by rigorous evidence. Further research is necessary to thoroughly investigate the precise influence of the video on consumer behaviour and its overall contribution to the mitigation toolkit market in Ireland. Nevertheless, it is worth noting that the availability of radon information (e.g., radon guide, mitigation toolkit, low-cost sensors toolbox) on the internet via the CS projects increases the accessibility of this information, which is often limited in radon-prone areas, especially at the local level (Perko and Turcanu, 2020).

The analysis of project reports, and interviews with both participants and researchers, indicate a notable discrepancy in the level of engagement among citizen scientists, particularly adult participants. For instance, project reports reveal that the CS approach utilized was either participatory or extreme CS according to the CS typology of Haklay (2013). In both approaches, data analysis and dissemination of project results are key aspects. Nonetheless, adult participants were not involved in disseminating project results, with this task predominantly undertaken by student participants in the case of Hungary. Moreover, students have exhibited greater involvement in analysing radon measurement results compared to their adult counterparts. This observation however aligns with Weinstein (2012) assertion that the involvement of citizen scientists often falls short of full participation in the complete scientific process. The active engagement of high school students in multiple stages of the research process highlights the necessity for providing adequate training to citizen scientists, especially in the context of radon research. The absence of such training not only limits the range of research activities in which participants can effectively engage but also underscores the importance of equipping them with the necessary research skills. In fact, Von Gönner et al. (2023) note that it is common for CS projects to prioritize the development of specific data collection skills, such as species identification, rather than explicitly engaging participants in the principles of scientific research. Indeed, our analysis highlights participants self-efficacy in using radon detectors to collect data on radon concentration levels, be it passive or active detectors. Therefore, to truly engage citizens beyond mere data collection in CS and enhance their scientific skills, concerted efforts should be made to offer comprehensive training on the scientific process to citizens. Such efforts are crucial for effectively involving them in CS and maximizing their contributions to research activities. Furthermore, similar to the findings of Stanifer et al. (2022), citizen scientists exhibited confidence in their ability to contact radon mitigation professionals and or carry out mitigation, either independently or with contractors.

Based on the results of the CS projects, their potential or achieved sustainability becomes evident through the outputs and impacts they generate. These outputs and outcomes can be readily adapted to future CS endeavors or similar projects. For instance, the DIY mitigation video developed in Ireland has been successfully adapted by the new RadoNorm CS project in Slovakia.⁴ Additionally, the dissemination of project outcomes through diverse channels and the ongoing CS initiative in Hungary, despite the conclusion of its initial funding, underscore the sustainability achieved or potential thereof of the CS projects. Indeed, one of the criteria outlined by the European Commission (2006) for assessing project sustainability is the extent to which results are maintained, developed further, or disseminated following the end of funding. Consequently, project results serve as a solid foundation for sustainability and other Horizon 2020 projects, such as PERFORM (Vizzini et al., 2017) and BuildERS (Kajganovic et al., 2022), have applied this criterion in their sustainability plans, demonstrating its significance in ensuring the enduring impact of such initiatives.

Given the variability, subjectivity, and lack of standardization in internal evaluation methods observed across the CS projects within the RadoNorm initiative (Supplementary Table S6), this article’s objective is justified in addressing the inherent challenges posed by inconsistent evaluation methodologies. Through the development of a standardized framework, this study aims to aid CS projects in the evaluation of project impacts and facilitating easy comparison of these impacts across diverse CS initiatives. Notwithstanding, the diverse nature of CS projects and their varying aims of impact assessments (Wehn et al., 2021), the framework presented in this study can be tailored to accommodate different CS projects, particularly in the field of radon.

6 Conclusion

This research proposes a holistic and comprehensive evaluation framework to help improve the outcomes and processes of CS projects. It develops and tests different evaluation dimensions and indicators in the form of questions. Five dimensions, enabling evaluation at both the outcome and process levels, are elaborated: 1) scientific impact, 2) participant engagement, 3) wider societal impact, 4) learning gains for researchers, and 5) process evaluation.

To test the developed indicators for practical application, four CS projects related to public health protection from radon were evaluated. Content analysis of project documents and semi-structured interviews were used to respond to all questions (indicators). The limited number of projects evaluated hinders the generalisation of the results. However, the evaluation results help identify strong and weak points in the projects, providing insights for improving future CS initiatives in radon research.

All four evaluated projects focused on indoor radon, a carcinogenic natural radioactive gas. The evaluation demonstrated that CS projects in this field can improve radon testing and mitigation rates, as well as increase knowledge and awareness about radon. These projects can provide new scientific knowledge, empower participants in decision-making, and have a societal impact on affected communities. However, the evaluation also identified challenges, such as initial motivation of citizens, sustainability constraints, limited scientific impact and the influence of project results on policy-making and legislation. Further research is needed to assess the impact of CS projects in the five defined dimensions, particularly on the behavioural and socio-cultural effects on participating citizen scientists and their decision-making in mitigating efforts. Refining the approaches to evaluating CS initiatives can enhance the design of future CS projects and support policy and research in this field.

The framework presented in this study can be tailored to accommodate various CS projects, particularly those addressing environmental and public health risks.

References

• 1

  AjzenI. (2011). The theory of planned behaviour: reactions and reflections. Psychol. Health26, 1113–1127. 10.1080/08870446.2011.613995

  • CrossRef
  • Google Scholar
  • View reference in article

• 2

  AndreszS.Marchand-MouryA.Goyette-PernotJ.RivièreA.-L.SchieberC. (2023). When citizen science meets radon building diagnosis: synthesis of a French pilot project developed in the framework of the European RadoNorm research project. Open Res. Eur.3, 106. 10.12688/openreseurope.15968.2

  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  AndreszS.SchieberC. (2023). Citizen science pilot project in France. (Deliverable).

  • Google Scholar
  • View reference in article

• 4

  BowserA.CooperC.SherbininA. D.WigginsA.BrentonP.ChuangT.-R.et al (2020). Still in need of norms: the state of the data in citizen science. Citiz. Sci.5, 18. 10.5334/cstp.303

  • CrossRef
  • Google Scholar
  • View reference in article

• 5

  BonneyR.ShirkJ. LPhillipsT. B.WigginsA.BallardH. L.Miller-RushingA. J.et al (2014). Next steps for citizen science. Science343, 1436–1437. 10.1126/science.1251554

  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  BrossardD.LewensteinB.BonneyR. (2005). Scientific knowledge and attitude change: the impact of a citizen science project. Int. J. Sci. Educ.27, 1099–1121. 10.1080/09500690500069483

  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  DavisL. S.ZhuL.FinklerW. (2023). Citizen science: is it good science?Sustainability15, 4577. 10.3390/su15054577

  • CrossRef
  • Google Scholar
  • View reference in article

• 8

  Den BroederL.LemmensL.UysalS.KauwK.WeekenborgJ.SchönenbergerM.et al (2017). Public health citizen science; perceived impacts on citizen scientists: a case study in a low-income neighbourhood in The Netherlands. Sci. Theory Pract.2, 7. 10.5334/cstp.89

  • CrossRef
  • Google Scholar
  • View reference in article

• 9

  DowdallA.FentonD.RaffertyB. (2016). The rate of radon remediation in Ireland 2011–2015: establishing a base line rate for Ireland’s National Radon Control Strategy. J. Environ. Radioact.162–163, 107–112. 10.1016/j.jenvrad.2016.05.001

  • CrossRef
  • Google Scholar
  • View reference in article

• 10

  DowdallA.McKittrickL. (2023). Citizen science pilot project in Ireland. (deliverable).

  • Google Scholar
  • View reference in article

• 11

  European Commission (2006). Directorate-General for Education, Youth, Sport and Culture, Sustainability of international cooperation projects in the field of higher education and vocational training – handbook on sustainability. Luxembourg: Publications Office. ISBN: 92-9157-476-7.

  • Google Scholar
  • View reference in article

• 12

  European Commission: Directorate-General for Energy, PerkoT.MartellM.RovenskáK.FojtíkováI.ParidaensJ.GeysmansR. (2023). Review and evaluation of national radon action plans in EU Member States according to the requirements of Council Directive 2013/59/Euratom. Luxembourg, Publications Office of the European Union. 10.2833/531667

  • CrossRef
  • Google Scholar
  • View reference in article

• 13

  FollettR.StrezovV. (2015). An analysis of citizen science based research: usage and publication patterns. PLOS ONE10, e0143687. 10.1371/journal.pone.0143687

  • CrossRef
  • Google Scholar
  • View reference in article

• 14

  HahnE. J.WilmhoffC.RayensM. K.ConleyN. B.MorrisE.LarckA.et al (2020). High school students as citizen scientists to decrease radon exposure. Int. J. Environ. Res. Public. Health17, 9178. 10.3390/ijerph17249178

  • CrossRef
  • Google Scholar
  • View reference in article

• 15

  HaklayM. (2013). “Citizen science and volunteered geographic information: overview and typology of participation,” in Crowdsourcing geographic knowledge: volunteered geographic information (VGI) in theory and practice. Editors SuiD.ElwoodS.GoodchildM. (Netherlands, Dordrecht: Springer), 105–122. 10.1007/978-94-007-4587-2_7

  • CrossRef
  • Google Scholar
  • View reference in article

• 16

  HansenJ. S.GadegaardS.HansenK. K.LarsenA. V.MøllerS.ThomsenG. S.et al (2021). Research data management challenges in citizen science projects and recommendations for library support services. A scoping review and case study. Data Sci. J.20, 25. 10.5334/dsj-2021-025

  • CrossRef
  • Google Scholar
  • View reference in article

• 17

  HeckerS.BonneyR.HaklayM.HölkerF.HoferH.GoebelC.et al (2018). Innovation in citizen science – perspectives on science-policy advances. Sci. Theory Pract.3, 4. 10.5334/cstp.114

  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  HeggenS.OmokaroO.PaytonJ. (2012). “MAD science: increasing engagement in STEM education through participatory sensing,” in UBICOMM 2012 : The Sixth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies, Barcelona, Spain, September 23-28, 2012.

  • Google Scholar
  • View reference in article

• 19

  HeveyD.PerkoT.MartellM.BradleyG.ApersS.RovenskáK. N. (2023). A psycho-social-environmental lens on radon air pollutant: authorities’, mitigation contractors’, and residents’ perceptions of barriers and facilitators to domestic radon mitigation. Front. Public Health11, 1252804. 10.3389/fpubh.2023.1252804

  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  IsleyC. F.FryK. L.SharpE. L.TaylorM. P. (2022). Bringing citizen science to life: evaluation of a national citizen science program for public benefit. Environ. Sci. Policy134, 23–33. 10.1016/j.envsci.2022.03.015

  • CrossRef
  • Google Scholar
  • View reference in article

• 21

  KajganovicJ.MorsutC.KuranC.HintonJ.RhinardM.NaevestadT.et al (2022). Sustainability report. Deliverable 8.5. BuildERS (Building European Communities’ Resilience and Social Capital) project.

  • Google Scholar
  • View reference in article

• 22

  KieslingerB.SchäferT.HeiglF.DörlerD.RichterA.BonnA. (2018). “Evaluating citizen science towards an open framework,” in Citizen science: innovation in open science, society and policy. Editors HeckerS.HaklayM.BowserA.MakuchZ.VogelJ.BonnA. (London: UCL Press).

  • Google Scholar
  • View reference in article

• 23

  KloetzerL.LorkeJ.RocheJ.GolumbicY.WinterS.JõgevaA. (2021). “Learning in citizen science,” in The science of citizen science. Editors VohlandK.Land-ZandstraA.CeccaroniL.LemmensR.PerellóJ.PontiM.et al (Cham: Springer International Publishing), 283–308. 10.1007/978-3-030-58278-4_15

  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  KolbA. Y.KolbD. A. (2009). “Experiential learning theory: a dynamic, holistic approach to management learning, education and development,” in The SAGE handbook of management learning, education and development (United Kingdom: SAGE Publications Ltd), 42–68. 1 Oliver’s Yard, 55 City Road, London EC1Y 1SP. 10.4135/9780857021038.n3

  • CrossRef
  • Google Scholar
  • View reference in article

• 25

  MartellM.AndreszS.DowdallA.GromaV. O.KenensJ.McKittrickL.et al (2022). Deliverable 6.9: citizen science model for radon priority areas (citizen science incubator).

  • Google Scholar
  • View reference in article

• 26

  MartellM.PerkoT.AndreszS.SchieberC.TomkivY.DowdallA.et al (2024). “Radon and citizen science,” in Communicating public health: risk communication strategies and public perceptions of radon gas. Editor Sixto-GarcíaJ.Pérez-SeijoS.García-OrosaB. (London: Routledge).

  • Google Scholar
  • View reference in article

• 27

  MartellM.PerkoT.TomkivY.LongS.DowdallA.KenensJ. (2021). Evaluation of citizen science contributions to radon research. J. Environ. Radioact.237, 106685. 10.1016/j.jenvrad.2021.106685

  • CrossRef
  • Google Scholar
  • View reference in article

• 28

  Mc LaughlinJ. P.Gutierrez-VillanuevaJ.-L.PerkoT. (2022). Suggestions for improvements in national radon control strategies of member states which were developed as a requirement of EU directive 2013/59 EURATOM. Int. J. Environ. Res. Public. Health19, 3805. 10.3390/ijerph19073805

  • CrossRef
  • Google Scholar
  • View reference in article

• 29

  NascimentoS.IglesiasJ.OwenR.SchadeS.ShanleyL. (2018). “Citizen science for policy formulation and implementation: innovation in open science, society and policy,” in Citizen science, 219–240. 10.2307/j.ctv550cf2.23

  • CrossRef
  • Google Scholar
  • View reference in article

• 30

  Olahne GromaV. (2023). Citizen science pilot project in Hungary. (Deliverable).

  • Google Scholar
  • View reference in article

• 31

  PerkoT.ThijssenP.HeveyD.TurcanuC.MuricM. (2024). Measuring societal attitudes and behaviours towards radon indoors: a case study of Slovenia. J. Environ. Radioact.272, 107355. 10.1016/j.jenvrad.2023.107355

  • CrossRef
  • Google Scholar
  • View reference in article

• 32

  PerkoT.TurcanuC. (2020). Is internet a missed opportunity? Evaluating radon websites from a stakeholder engagement perspective. J. Environ. Radioact.212, 106123. 10.1016/j.jenvrad.2019.106123

  • CrossRef
  • Google Scholar
  • View reference in article

• 33

  PhillipsT.PorticellaN.ConstasM.BonneyR. (2018). A framework for articulating and measuring individual learning outcomes from participation in citizen science. Science3, 3. 10.5334/cstp.126

  • CrossRef
  • Google Scholar
  • View reference in article

• 34

  Ruiz-MallénI.Riboli-SascoL.RibraultC.HerasM.LagunaD.PeriéL. (2016). Citizen science: toward transformative learning. Sci. Commun.38, 523–534. 10.1177/1075547016642241

  • CrossRef
  • Google Scholar
  • View reference in article

• 35

  ShirkJ.BallardH.WildermanC.PhillipsT.WigginsA.JordanR.et al (2012). Public participation in scientific research: a framework for deliberate design. Ecol. Soc.17, art29. 10.5751/ES-04705-170229

  • CrossRef
  • Google Scholar
  • View reference in article

• 36

  SkarlatidouA.HaklayM. (2021). Citizen science impact pathways for a positive contribution to public participation in science. J. Sci. Commun.20, A02. 10.22323/2.20060202

  • CrossRef
  • Google Scholar
  • View reference in article

• 37

  SomerwillL.WehnU. (2022). How to measure the impact of citizen science on environmental attitudes, behaviour and knowledge? A review of state-of-the-art approaches. Environ. Sci. Eur.34, 18. 10.1186/s12302-022-00596-1

  • CrossRef
  • Google Scholar
  • View reference in article

• 38

  StaniferS.HooverA. G.RademacherK.RayensM. K.HanebergW.HahnE. J. (2022). Citizen science approach to home radon testing, environmental health literacy and efficacy. Citiz. Sci. Theory Pract.7, 26. 10.5334/cstp.472

  • CrossRef
  • Google Scholar
  • View reference in article

• 39

  TomkivY.AnjumR. L. (2023). Citizen science pilot project in Norway. (deliverable).

  • Google Scholar
  • View reference in article

• 40

  TrumbullD. J.BonneyR.BascomD.CabralA. (2000). Thinking scientifically during participation in a citizen-science project. Sci. Educ.84, 265–275. 10.1002/(SICI)1098-237X(200003)84:2<265::AID-SCE7>3.0.CO;2-5

  • CrossRef
  • Google Scholar
  • View reference in article

• 41

  TsapalovA.KovlerK.ShpakM.ShafirE.GolumbicY.PeriA.et al (2020). Involving schoolchildren in radon surveys by means of the “RadonTest” online system. J. Environ. Radioact.217, 106215. 10.1016/j.jenvrad.2020.106215

  • CrossRef
  • Google Scholar

• 42

  Van BrusselS.HuyseH. (2019). Citizen science on speed? Realising the triple objective of scientific rigour, policy influence and deep citizen engagement in a large-scale citizen science project on ambient air quality in Antwerp. J. Environ. Plan. Manag.62, 534–551. 10.1080/09640568.2018.1428183

  • CrossRef
  • Google Scholar

• 43

  VasileiouK.BarnettJ.ThorpeS.YoungT. (2018). Characterising and justifying sample size sufficiency in interview-based studies: systematic analysis of qualitative health research over a 15-year period. BMC Med. Res. Methodol.18, 148. 10.1186/s12874-018-0594-7

  • CrossRef
  • Google Scholar
  • View reference in article

• 44

  VizziniC.Da SilvaA.San MartínR. (2017). Sustainability plan. Deliverable 5.1. Paris: PERFORM (Participatory Engagement with Scientific and Technological Research through Performance) project.

  • Google Scholar
  • View reference in article

• 45

  Von GönnerJ.HerrmannT. M.BruckermannT.EichingerM.HeckerS.KlanF.et al (2023). Citizen science’s transformative impact on science, citizen empowerment and socio-political processes. Res.5, 11–33. 10.1007/s42532-022-00136-4

  • CrossRef
  • Google Scholar
  • View reference in article

• 46

  WehnU.GharesifardM.CeccaroniL.JoyceH.AjatesR.WoodsS.et al (2021). Impact assessment of citizen science: state of the art and guiding principles for a consolidated approach. Sustain. Sci.16, 1683–1699. 10.1007/s11625-021-00959-2

  • CrossRef
  • Google Scholar
  • View reference in article

• 47

  WeinsteinM. (2012). Schools/citizen science. A response to “the future of citizen science.”Democr. Educ.20, 6.

  • Google Scholar
  • View reference in article

• 48

  World Health Organization (2009). WHO Handbook on indoor radon: a public health perspective. Geneva: World Health Organisation.

  • Google Scholar
  • View reference in article