Previous studies have shown that surveys are a suitable way to determine the reliability of citizen science data (Rinderer et al., 2012; Artell et al., 2013; Strobl et al., 2020). Therefore, in this study, we used a structured pen-and-paper questionnaire (see section 2.3) and asked 1,268 participants questions about the flow state of the stream that they could see in front of them. Like Strobl et al. (2020), we approached passers-by and asked them to participate in the 5-min questionnaire, i.e., we used a convenience sampling approach. We did this for eight temporary streams in Switzerland and southern Germany on 23 days between April and September 2022.

We selected eight temporary streams for the surveys. The streams were located along hiking paths or near attractions, so that we could reach a sufficient number of participants. Five of the study sites were located in Switzerland (Pizol (P), Sihlwald (S), Thurfälle (Th), Töss (T), Züriberg (Z)) and three in Germany (Donauversickerung (D), Dreisam (Dr), Hinterzarten (H)) (Figure 1). The selected streams varied from small alpine streams (P), to headwater streams (H, S, Z), to larger rivers (D, Dr, T, Th) (Figure 1; Table 1). The stream width and visible stream length were determined in the field and used to group the streams into three classes (small, medium, large; Table 1). Some streams (Z, S, and H) had vegetation in the stream or on the streambank, which made it (more) difficult to see the streambed, while others were open. The climate at the study sites varies from temperate, with no dry season and a warm summer (Köppen-Geiger classification: Cfb (Kottek et al., 2006)), to cold, with no dry season and a warm summer (DfB).

The field days were selected based on the expected flow state to cover a variety of flow states. Not all flow states were observed for each stream and not all flow states were surveyed a similar number of times (Table 1). We did most of the surveys on weekends and during pleasant weather conditions so that people would be more willing to participate. We stayed for 2.5–5 h (average 3.6 h) at each site and aimed to survey at least 40 people per day (cf. Field, 2015; Strobl et al., 2020). The average number of participants was 55 (median: 56). However, on field day S2, only 30 people filled in the questionnaire. We did not count the number of people we asked to participate but estimated that about 80% of the people that we approached were willing to join (excluding bikers).

The questionnaire was developed iteratively over three test runs before deciding on the final version. Between the test runs, we changed the questions and structure of the questionnaire based on participants’ feedback and discussions with colleagues. After these initial tests, we finalized the questionnaire (Supplementary Material S1). We used this final version for all surveys presented in this paper.

The pen and paper questionnaires were available in English and German and contained four parts (see Supplementary Material S1). In the first part, the participants were asked about their knowledge of the survey location and whether they had previously heard of the CrowdWater project. Then, they were asked to determine the flow state of the stream in front of them based on six different icons with either a one- or two-word description (Table 2). The six classes are the same as those used in the CrowdWater app: dry streambed, damp/wet streambed, isolated pools, standing water, trickling water, and flowing (Table 2). These six flow state classes represent unique habitats for aquatic and terrestrial species. At the end of this second part, we asked the participants for an estimate of the length of the stream reach that they considered when they determined the flow state, whether they found it challenging to decide on one flow state, and whether they considered other information than what they could see in front of them.

In the third part of the questionnaire, the participants were given 14 statements that could be answered with either yes or no, such as “There is water along the entire stream length.” and “The stream is completely dry”. These yes-no statements were designed to assess if the participants understood the icons and the one- or two-word descriptions of the flow states, i.e., we used the answers to these statements to determine the consistency with the flow state that they had selected in part 2. In the final part, we asked some demographic questions (see Supplementary Material S2 for results) and provided space for comments. We did not give the participants additional information or help while filling in the questionnaire because we wanted to assess how people intuitively interpret the six flow state classes. This was motivated by the general observation that people rarely study manuals or explanation videos and rather start using an app immediately. However, we were available for questions and discussions afterwards.

Combining all surveys, two-thirds of the participants selected the same class as the mode (A_overall). The A_{overall, weighted} was 0.67, and thus almost identical to A_overall. Only 17% and 4% of all participants selected a flow state that was more than one or two classes away from the mode, respectively (Figure 2). For field days for which most participants selected either the dry streambed class or flowing class, the agreement was highest (A_mode±1: >89%; Figure 3). Thus, for these “extreme” classes, most participants selected the same flow state as the majority of other participants (A_mode) or were just one class off (A_mode±1). When the mode was not the dry streambed or flowing water class, 28% of participants selected a flow state that was more than one class off from the mode. The agreement was lowest when most participants selected the isolated pools or standing water classes (median A_mode±1: 67% and 62%, respectively; Figure 3). A_mode±1 was particularly low (<57%) for surveys H3 and T3 for which the flow state selected by most participants was standing water and dry streambed, respectively (Supplementary Figure S1). Δ_mode was also furthest from zero for the standing water class (Table 3).

The mode and expert opinion agreed for 14 of the 23 field days. For five field days, the opinions differed by one class, and for four field days, they differed by two classes. Participants tended to select a drier flow state than the expert. The one class too dry mainly occurred when the expert’s opinion was damp/wet streambed and most participants selected the dry streambed (P2, Th3), or when the expert selected the flowing class and the participants selected the trickling water class (S1, S2) (Figures 2, 3). The difference of two flow state classes mainly occurred when the expert had selected the isolated pools or standing water class and the participants (mode) selected the dry or wet/damp streambed class (e.g., H2, T2). As a result, the overall agreement between the participants and the expert was lower than for the mode (A_overall: 56% for the expert vs. 66% for the mode).

When assessed by the mode, the sensitivity and precision were highest for the dry streambed and flowing classes (>80%, Figure 4, top row). The sensitivity was much lower for the other classes (S_mode: ∼50%), and lowest for the standing water class (S_mode: 40%). The precision was intermediate for the standing water and isolated pools classes, and lowest for the damp/wet streambed and trickling water classes.

The most striking point of the confusion matrix for the expert’s opinion, is the high number of false negatives for the dry streambed class (n _FN : 230; Supplementary Figure S2). Many participants selected the dry streambed class when the expert chose the wet/damp streambed or isolated pools class (Figure 3). This resulted in a low precision (P_expert: 59%) for the dry streambed class, even though the sensitivity was high (S_expert: 92%) (Figure 4, top row). The precision and sensitivity were lowest for the trickling water class, but this flow state was selected by the expert on only two field days.

The answers to the different yes-no statements of the questionnaire were consistent with the (previously) selected flow state class for 78%–98% of the participants, depending on the statement (Figure 5). Theoretically, participants who selected the dry or damp/wet streambed flow state should not have seen any water in the stream. However, 27% of the participants who selected the dry streambed class answered yes when asked if they could see disconnected pools of water. Similarly, 16% said that they could see water in the stream, 7% could see water along the entire visible stream length, and 49% answered yes when asked if the stream was completely damp/wet (Figure 5). A large proportion of the participants who selected the damp/wet streambed class also said that they could see disconnected pools of water (46%), water in the stream (57%), or water along the entire stream (25%). For the other four flow states, the percentage of inconsistent answers was small (1%–6%).

The deviation of the selected flow state class from the expert was related to the visible stream length and the considered stream length. Δ_expert was closer to zero and Δ_{expert, abs} was smaller when the considered stream length was shorter than 5 m than when it was longer than 50 m (Table 3; Supplementary Figure S3); 28% of the participants considered a distance shorter than 5 m and 61% shorter than 50 m. The participants at the Dreisam (Dr) considered by far the longest stream length when determining the flow state class (median: 500 m). It was more than twice the visible length for 14% of all the responses.

Participants who did not use any additional information tended to select a wetter stream state than the mode (Δ_mode = 0.25), though the difference was not statistically significant (Table 3). The most frequently used additional information by the participants were the flow state on the previous days or familiarity with the stream for several years, the weather in the last days/weeks, newspaper articles about the stream, or information on water management actions (e.g., abstraction, re-naturalization programs). For those who used additional knowledge, Δ_mode was close to zero (−0.04) but the deviation from the mode was bigger than if no additional information was used (Δ_{mode, abs} of 0.90 vs. 0.58) (Table 3). Only 4% of the participants who were at a site for the first time used additional knowledge to decide on the flow state vs. 12% of those who had been there more than five times. The deviation from the mode or expert’s opinion was significantly smaller for participants who visited the site for the first time than those who had been there more than five times. The deviation was also smaller for the 75% of participants who felt confident in their selection of the flow state class than for those who were insecure about their decision (Δ_mode of −0.07 vs. 0.11 and Δ_{mode, abs} of 0.53 vs. 0.73; Table 3).

The level of agreement among the participants (A_mode: 66%) compares well to the agreement reported in other studies that used surveys to determine the quality of citizen science data. These previous studies based on visual assessments (e.g., water level classes (Strobl et al., 2020), soil moisture (Rinderer et al., 2012; Rinderer et al., 2015), and water quality (Artell et al., 2013)) reported overall agreements of 46%–70%. In these studies, the classes selected by the participants were either compared to the experts’ opinions (Strobl et al., 2020), the opinion of the majority of the participants (Rinderer et al., 2012; Rinderer et al., 2015), or measured values (Artell et al., 2013). The number of classes ranged from five (Artell et al., 2013), to seven (Rinderer et al., 2012; Rinderer et al., 2015), to ten (Strobl et al., 2020). Thus, the overall agreement values obtained in our study seem to be good and comparable to these studies. Aceves-Bueno et al. (2017) report that for 80% of citizen science studies in natural sciences, the agreement was higher than 55%; for 60% of the studies, it was higher than 81%.

Interestingly, the agreement among the participants was also on par with studies that used sensors to detect the presence of water or flow. For example, the mean agreement between data from four different low-cost sensors and time-lapse camera images for several streams in a small Swiss headwater catchment varied between 75% and 99.9% (Assendelft and van Meerveld, 2019). Another study suggested that temperature sensors always determined the dry and not-dry flow state correctly (based on visual flow state observations) (Constantz et al., 2001). Comparisons of dry and not-dry flow states between water level logger data and pre-processed time-lapse camera images led to concordant observations if the water level logger indicated a dry stream (true positive rate: 100%), but if the water level logger measurements indicated a not-dry condition, observations diverged by 48%–85% (Herzog et al., 2022). A national hydrography dataset of stream permanence classifications disagreed one-quarter of the time with flow state assessments from topographic maps or aerial imagery and interviews with residents, with the disagreement being higher for smaller streams (Hafen et al., 2020).

The flowing flow state class could be determined best. The agreement and sensitivity were also high for the dry streambed class, but participants tended to select the dry streambed class even when they saw some water in the streambed (Figure 5). This led to a low precision when assessed by the expert’s opinion and caused the mode and the expert’s opinion to differ for half of the field days for which the mode was a dry streambed. This result also makes the observations regarding water and no water in the streambed less reliable than for flow and no flow (Figure 4). The spread in the chosen flow states and the inconsistency between the answers to the different yes-no statements may suggest that the participants did not understand the six flow state classes well without any explanation. However, the differences are in part also related to the variability in flow states along the stream, as several flow state classes can be present along one stream segment. The variation in the opinions (Table 3) and the deviation from the expert’s opinion were smaller when the considered stream lengths were smaller. Thus, the differences between the opinions of the expert and the participants could be caused by the fact that participants considered a different/longer stream length than the expert. For example, on field days Dr2 and Dr3, the stream flow state at the survey location was trickling (according to the expert), but the stream was dry 2 km further downstream. This may explain why 13% and 12% of the respondents selected the dry state for Dr2 and Dr3, respectively, even though the streambed was not dry at the survey location. The selected flow state can also be affected by the news, as several news outlets reported on the dry Dreisam (Dr) earlier in the summer.

The size of the pools, in comparison to the stream width and visibility of the pools, likely also influenced the choice of the flow state. During field day D1, there were big pools (approximately 40 m long) and very few participants said that the streambed was dry (Figure 3). During field days D2, T2, and T3, there were very small pools (approximately 0.5–5 m), or the pools were slightly hidden, and more participants selected the dry streambed class. However, in the yes-no statements, most participants said that they saw isolated pools of water. Thus, the participants selected the dry streambed class despite acknowledging the presence of pools. During field day T2, when small, isolated pools (approximately 0.5 m long) were present, one participant wrote that they could see isolated pools on the streambed but that they knew that these would not be there on the next day and that they, therefore, selected the dry streambed class. For some field days for which it had rained the night before (D2, P2, Th3, D2), some participants similarly questioned if they should consider the water in the stream or the dampness of the streambed. They argued that it was “just” rainwater, and the water was not from the stream itself. Other participants questioned if the stream could be considered to be dry or not flowing only because there was no (flowing) water in the streambed. Based on comments like “The stream is flowing below ground” at the Töss (T), it seems that some participants knew that water was still flowing below the ground in the alluvial aquifer.

Participants also commented on the visibility of the stream, especially when the stream was not well visible due to vegetation (H1-3, Z2). The flow condition on field day Z2 was the same as on Z1, but the survey was carried out further downstream, where lots of vegetation covered the stream. This poor visibility may have resulted in a larger spread in the selected flow state classes between participants for Z2 (A_mode: 49% vs. 96%; Figure 3).

In this study, we used six flow state classes because each class has a unique hydro-ecological function. For the classes in the middle of the dry to flowing spectrum the agreement, precision, and sensitivity were low, which raises the question if it is better to use fewer classes. Indeed, the agreement is higher when there are fewer classes, but one advantage of having more classes is that the effect of being a class off is smaller. Furthermore, only a few participants selected a flow state that was more than one class away from the mode (17% for all surveys, and 28% when the mode was not the dry streambed or flowing water class). Therefore, we still recommend using six classes and later regrouping the data into three or two classes (if needed). However, one should first consider what kind of data are needed for the project’s objectives.

We did not give participants any guidance on how to choose the flow state as we wanted to study how people intuitively interpret the six flow state classes. The results show that the icons and one or two-word descriptions are not immediately clear and/or that flow state observations for temporary streams are not as simple or obvious as one might expect. Thus, additional descriptions and some guidance or training would be useful, especially on which flow state class to select if several flow states are visible. Jaeger et al. (2021) already highlighted the importance of clear definitions and logical workflows for temporary stream observations to be helpful for end users. The effects of such training on data quality and participation, however, remains to be explored.

Similarly, the fact that for some sites, participants considered a very long stream length (longer than what was visible) suggests that clear instructions to determine the flow state locally or, for instance, over twice the river width would be useful. In the CrowdWater app, citizen scientists are asked to take a picture of the stream and to indicate the spot for which they determine the flow state by an arrow or circle. This provides a spatial frame that should help focus citizen science observations to one point in the stream. This spatial frame was not used in the surveys. Thus, it remains to be explored whether it improves the agreement, precision, and sensitivity of actual citizen science data.

The results of the surveys do not allow us to determine how well an individual person is able to observe changes in flow states over time and whether repeated observations of flow states are more reliable if they are made by the same person. One could expect observations from one person to be more reliable because that person probably always looks at the same spot. The results of Etter et al. (2020) for water level class observations also suggest that observations by the same person better reflect actual changes in the stream level than when observations are made by many different people (which potentially each have a different bias).