Incorporating information about risk stratification into a bowel cancer screening information leaflet: a think aloud study and user testing process

Abstract

Background:

Communication of concepts relating to risk and risk assessment can be challenging for lay individuals to understand. As we move toward risk-stratified screening for bowel cancer, it is necessary to identify public information needs and explore understanding of communication about risk stratification and personal cancer risk as this may have implications for screening uptake. We aimed to develop and test comprehension of a screening leaflet relating to risk-stratified fecal immunochemical test (FIT) screening intervals and to explore participant attitudes toward stratified intervals for bowel cancer screening.

Methods:

We adapted an existing NHS England bowel cancer screening leaflet to communicate a bowel cancer screening programme with risk-stratified screening intervals. The leaflet was used in 13 think aloud interviews to elicit areas of misunderstanding and potential alteration. We analyzed the interviews using codebook thematic analysis and made changes based on our findings. We then tested comprehension of the final leaflet with a further 20 participants in a user testing survey. We also analyzed attitudes toward risk-based bowel screening thematically.

Results:

The think aloud interviews identified 42 areas of the adapted leaflet that required improvement and 35 of these were incorporated into the final version. Changes included clarifying terminology, improving layout including greater use of bullet points, and resolving areas of misunderstanding. They also suggested additional information to mitigate cancer worry. At least 90% of user testing participants answered each true or false statement correctly after reading the final version of the leaflet.

Conclusions:

Specific elements of the initial risk-stratified leaflet required improvement; after revision, user testing indicated that a minimum threshold of comprehension had been achieved in the final version. Based on the information provided in this leaflet, a risk-stratified approach to bowel cancer screening was considered acceptable overall. With appropriate care taken to develop materials, the public are, therefore, able to understand information about risk-based bowel screening programmes delivered as part of population-based communications.

1 Background

The current bowel cancer screening programme in England invites those aged 54–74 to participate in fecal immunochemical testing (FIT), to detect blood in the feces (1). Individuals who have a fecal hemoglobin (FHb) concentration above the current threshold of 120 μg/g are referred for further investigation, whilst those below the threshold are invited to complete FIT screening again in 2 years' time. There is an ongoing shift toward a more personalized approach to screening, based on individual risk, both nationally within the NHS and internationally (2). For example, high risk individuals may be offered a reduced screening interval while those at low risk are offered an extended one. This would target constrained healthcare resources toward the people in society who stand to benefit most from screening and simultaneously minimize exposure to the harms of screening for lower risk groups. In the context of screening by FIT, such harms include overdiagnosis (where an adenoma is treated that would never have gone on to cause harm), anxiety and the potential for false positive and false negative results, causing unnecessary concern and false reassurance respectively (3). There is also the potential for unnecessary further investigations such as colonoscopy, which carry additional harms. Such an approach will likely require communication of personal risk, risk-informed screening schedules and the rationale behind risk-stratified screening to the public. Successful communication is a necessary factor for facilitating public acceptability of changes away from traditional age and/or sex-based policies and instilling confidence in the outputs of a risk algorithm (4, 5).

The concept of risk in general is both complex and multidimensional, meaning that conveying personalized estimates in a meaningful way and altering risk schedules in accordance with these remains challenging (6). Studies across multiple diseases have shown that the public typically prefer gist-based information supported by diagrams and comparative illustrations of risk, and that comprehension of mathematical concepts, such as percentages and frequencies, may be prone to misinterpretation (6–12). Communication of personal cancer risk and risk-related concepts potentially incurs an additional challenge in the context of the implementation of risk-based screening as the public often have limited knowledge and understanding of the harms and benefits associated with cancer screening (8, 9, 13).

Previous literature has identified that communication of risk and risk-stratified screening must therefore be careful not to contradict existing public health messaging primarily concerned with promoting uptake (4). This is especially key when communicating reduced screening for low-risk groups, which is generally perceived as less acceptable than increasing screening for those at high risk (4, 5, 14). Additionally, the English Bowel Cancer Screening Programme (BCSP) currently delivers dichotomous results, either positive or negative, omitting the nuance of absolute FHb concentration. As a result, the public are largely unaware that someone with a negative result may have a FHb concentration between 0–120μg/g. This existing style of communication makes it more difficult to successfully communicate a risk-stratified programme where elements of screening are based on absolute concentrations. As we move toward personalized screening for bowel cancer, there is therefore a need to explore the information needs of the public and how these relate to risk communication, screening schedule alterations, and the screening programme as a whole. We therefore aimed to develop and test comprehension of an information leaflet for a bowel cancer screening programme employing risk-stratified screening intervals, based on absolute FHb concentration with additional risk factors. Secondarily, we aimed to explore individual-level views toward this approach to bowel cancer screening amongst those provided with this information.

2 Methods

The “NHS bowel cancer screening: helping you decide” leaflet (15), which members of the public are sent at the first invite to bowel cancer screening, was adapted by the authors according to previous literature and the input of two Patient and Public Involvement (PPI) members, to reflect a bowel cancer screening programme employing risk-stratified screening intervals. We also piloted the adapted leaflet with three members of the Cambridge BRC PPI panel ahead of the think aloud interviews.

The initial leaflet contained information on the following topics: (I) Why we offer bowel cancer screening; (II) Who we invite; (III) How the bowel works; (IV) Bowel cancer; (V) Risks of developing bowel cancer; (VI) How bowel cancer screening works; (VII) Reduce your risk of bowel cancer; (VIII) Using the FIT kit; (IX) Bowel cancer screening results; (X) What happens to samples after testing; (XI) If you need further tests; (XII) Colonoscopy; (XIII) Possible benefits and risks of bowel cancer screening; (XIV) Bowel cancer symptoms; (XV) Treatment for bowel cancer. Eight of these headings (Why we offer bowel cancer screening; Who we invite; How the bowel works; Bowel cancer; Risks of developing bowel cancer; How bowel cancer screening works; Possible benefits and risks of cancer screening; colonoscopy) were retained in the leaflet communicating risk-stratified screening intervals in order to provide adequate background to bowel cancer and FIT screening. These sections were reordered and the text amended slightly. Additional sections on the risk of developing bowel cancer, risk calculation, and risk-stratified screening outcomes were added. The risk-stratified leaflet communicated a screening programme in which individuals with a moderate or high risk of developing bowel cancer (according to their FIT result and individual level risk factors) would receive a 1 year screening interval, whereas those with a low risk would receive a 3 year interval. Individuals determined to be of average risk would continue to receive biennial FIT screening, in line with the current screening programme in England.

The original NHS England leaflet contains public sector information licensed under the Open Government Licence V3.0 (https://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/).

Ethical approval for this study was granted by the Cambridge Psychology Research Ethics Committee (ref: PRE.2023.023).

2.1 Study design

2.1.1 Think aloud interviews

During the interviews, participants were asked to “think aloud” by vocalizing their internal cognitive process whilst reading the leaflet about risk-stratified screening intervals. The information processed whilst completing a task is available within the short term memory, meaning that retrospective expression of the thinking process inadequately reflects the spontaneous thoughts involved in mediation of the task (16). However, such thoughts may be elicited through concurrent verbalization during the task, meaning the think aloud method provides novel insight into the research question by illuminating a participant's internal cognition. This method remains distinct from traditional interview techniques which employ introspection and reflection and has previously been successfully used in both health psychology and healthcare research (16–21).

The think aloud interviews were held using Zoom video conferencing software (Zoom Video Communications). A technical check was conducted prior to each interview to mitigate any technical issues and ensure participants were comfortable with the video conferencing process. Each interview was up to 1 h in duration. Participants were not sent the information leaflet in advance and viewed it on the researcher's shared screen during the interview. The interviews were conducted by a researcher with previous experience of qualitative research and interview technique (LT).

The researcher asked participants to read the risk-stratified information leaflet whilst thinking aloud by verbalizing any internal thoughts prompted by the material. In order to ensure they felt comfortable doing this, she asked them to first practice thinking aloud using a pilot text about an unrelated topic (16). Any participants who failed to vocalize their thoughts during the first half of the pilot text had the think aloud process explained again and were given the opportunity to practice using the second half of the pilot before moving on to the interview itself. Both the pilot text and the leaflet employed a marked protocol design in which red dots are strategically placed in the text and act as prompts for verbalization. These were placed at the end of short paragraphs, sentences or bullet points that were of particular relevance to the research question(s) (11).

The results of the think aloud interviews were used to identify areas of misunderstanding in the initial leaflet which was then refined before being evaluated by a user testing procedure.

We also explored participant attitudes toward risk-based screening for bowel cancer expressed during the interviews. The think aloud participants reacted to the concept of risk stratification based solely on the information provided in the screening leaflet and did not receive supplementary education about the approach. Therefore, this strategy enabled us to capture the views and opinions of members of the public receiving such information.

2.1.2 User testing procedure

The use of true or false statements in user testing has previously been used to assess health-related knowledge and understanding (11, 22). A short survey adapted from a previous study by Smith et al. (11) that aimed to develop and test supplementary information for bowel cancer screening was used to assess understanding of the refined leaflet. Participants read the refined text before answering 14 true or false questions about the leaflet content. The user testing statements were designed to measure objective comprehension, i.e., whether the participants understood the content, rather than subjective comprehension, meaning whether participants thought that they understood the content. In line with the approach published by Usher-Smith et al. (12), a minimum of 80% of participants were required to answer each question correctly to meet the threshold of comprehension. After each round, the information is typically revised in areas where fewer than 80% of participants answered correctly for a maximum of five rounds. Eight of the true or false statements were adapted from Smith et al., including statements about the prevalence of bowel cancer, the logistics of FIT screening, and the associated benefits and harms. The remaining six statements were developed based on the key areas of misunderstanding identified by the think aloud interviews and information specific to the risk-stratified approach. These statements were reviewed and amended by the PPI representatives to maximize understanding among lay individuals.

2.2. Participants and recruitment

2.2.1 Think aloud interviews

We aimed to initially recruit 15 participants, purposely sampled by age, sex, social grade, geographic location in England, and previous bowel cancer screening attendance via iPoint Research Ltd, a market recruitment company. Previous think aloud studies have reported minimum sample sizes of between five and 18 participants (23). Eligible participants were aged 50–74 living in England without medical expertise or a previous bowel cancer diagnosis, and had not participated in previous studies conducted by this research group (5, 24). iPoint Research obtained consent, provided study details and facilitated participant reimbursement at their recommended rate. We also used a short Qualtrics survey distributed via iPoint using individualized links to collect demographic information and the list of personal identifiers was stored on the Cambridge School of Clinical Medicine Secure Data Hosting Server (SDHS).

2.2.2 User Testing Procedure

We planned to recruit participants in rounds of 20 individuals to complete the user testing survey until the threshold of comprehension (80%) was reached by all participants in a round, or until a maximum of 100 participants completed the survey. As described in the results (Section 3.3), the threshold of comprehension was reached in the first round; therefore no further participants were recruited. Participants were recruited online via the prolific platform, a recruitment platform for researchers where participants volunteer to participate in studies and are compensated at a previously agreed rate. Eligible participants were aged 50–74 living in the UK, without a personal history of bowel cancer or medical expertise and were purposely sampled by age and sex.

In line with standard procedure on the prolific platform, participants were able to view a short summary of the study before agreeing to take part. Once they had agreed to do so, they were able to download the participant information sheet and complete written online consent before beginning the survey. All participants were free to withdraw their consent at any time.

2.3 Analysis

2.3.1 Think aloud interviews

Interview data were collected online using Zoom's record function. Recordings were transcribed verbatim via an external transcription company, using a 256 bit SSL encrypted upload process. They were then pseudonymized by a member of the study team (LT). The interview transcripts were analyzed using codebook thematic analysis (25, 26); whereby the leaflet section headings were used to deductively generate a high-level framework. The data within each heading were reviewed and suggested leaflet changes were identified according to the preferences, opinions, and misunderstandings expressed by interview participants. These suggested changes were tabulated according to the coding frame and a traffic light and icon system was used to indicate the feasibility of applying each change to the risk-stratified screening leaflet. The same method of codebook thematic analysis was used to analyze participant attitudes toward the intervention, in which areas of commonality were grouped and both unifying and divergent themes identified. The interviews and analysis were led by a member of the study team with previous experience of qualitative methods and thematic analysis techniques (LT). All members of the study team, who also have previous experience analyzing qualitative research, reviewed and contributed to the analysis, and two PPI members without a personal history of cancer reviewed the findings.

2.3.2 User testing procedure

The user testing survey was hosted via Qualtrics and anonymized demographic information and survey responses were stored on the University of Cambridge shared drive. The quantitative data from the survey were analyzed using descriptive statistics, where the percentage of participants who answered each true or false statement correctly was calculated.

3 Results

3.1 Participant characteristics

The demographic characteristics of the 13 participants who took part in the think aloud interviews are summarized in Table 1. We chose not to recruit further participants as no new themes were identified in the later interviews, indicating that sufficient data had been collected with which to draw conclusions about the research question. Reflecting the demographics of the UK, the sample was balanced between male (53.8%) and female (46.2%) participants and comprised a range of ethnicities. In general, the sample was educated to a greater degree than average, with 15.4% of participants educated to degree level. Additionally, all participants had completed at least A levels or an equivalent qualification. Attendance at cancer screening was high (84.6%), with only one participant having previously declined screening, and the majority of participants reported having a family history of cancer (61.5%).

Twenty participants were recruited to take part in the think aloud survey (Table 1). Again, the sample was evenly split between male (50.0%) and female (50.0%) participants, and the majority had completed further education to degree level (60.0%). Screening attendance was high (80%) and most participants reported a family history of cancer (55.0%), with three individuals reporting a personal cancer history (15.0%). Notably, all 20 participants were of White ethnicity.

3.2 Changes to the information leaflet: think aloud interviews

The think aloud interviews were analyzed using codebook thematic analysis and key findings are summarized in Table 2. Participants suggested a total of 42 potential changes to the screening leaflet; 32 of these were incorporated fully, three were incorporated partially and seven changes were rejected from the refined version of the leaflet. Overall, 25 suggested changes related to sections of the leaflet that were included in the original NHS leaflet and a further 17 changes related to the additional risk-stratified elements of the text. The section of the leaflet entitled “Possible benefits and risks of bowel cancer screening” generated the greatest number of suggested changes, despite being included in the original NHS information. A total of seven changes were suggested within this section, largely relating to minimizing potential areas of confusion, and all of these were included in the refined version.

Table 2

Leaflet section	Suggested change	Outcome	Supporting quote
1. How the bowel works
1.1 Picture of the bowel	More detail/labeling of other organs	Only one individual suggested this change, whereas others were positive about the picture.	“…It doesn't explain the middle section of the small intestine. It doesn't explain what the liver is. It doesn't explain what the stomach is.” (P5) “…I really love the picture…” (P6)
	Use color to highlight which part is the bowel	Only one individual suggested this. Other participants were positive about the original image.	“You might want to change the colour, for example, so make the colon bright orange the rest of it shaded… to emphasise the part you're interested in…” (P5)
2. Bowel cancer	Break up the text	More paragraphs were added to break up the text and improve readability.	“…might be useful to, kind of, maybe incorporate some bullet points or something, maybe break it up a little bit…” (P6)
	Include statistics on incidence/survival	Participants were generally keen for more information. Sentences about bowel cancer incidence and development time were added.	“So yes I think people should know what the developments are, how serious it had become, how long have you got to live if you get it, what's the options, what the treatments are. Is there a big survival rate now…” (P7)
2.1 Language and terminology	Image or diagram of a polyp and/or its progression to cancer to support the text	This is challenging to add without using a medical or scientific diagram that could be confusing. Instead, the repetition about polyps in the first two sections was consolidated to improve clarity.	“A little bit [challenging to understand] because I'd say if you knew what a polyp was then you'd understand it more. But if there's an image of a polyp maybe in the tract that would make you understand. More pictures as it were than words.” (P10)
	Avoid repetition between the “Bowel cancer” section and “Why we offer bowel screening” section	These paragraphs were rearranged to improve clarity. Repeated description of polyps was removed.	“…Instead of just repeating the whole paragraph you could just say generally that this is why we screen, and to remove polyps etc… So, you're talking about something else rather than repeating some information then adding a little bit.” (P10)
	Differentiate between polyps and hemorrhoids	Although the distinction between polyps and hemorrhoids was not specifically added, piles are mentioned earlier in the leaflet as a possible alternative cause of blood in the feces.	“The polyps, that polyp thing is that the same as a hemorrhoid?... Well I thought they were the same thing. I would be pleased to see this is different from a…” (P12)
3. Why we offer bowel screening	Change heading to “Why bowel screening is important”	The heading was amended to “Bowel cancer screening” from “Why we offer bowel screening” to better reflect the breadth of information.	“But it could be bowel cancer and then also a sub-heading of cancer screening. I think we don't need to know why you offer it because, you know why you offer it.” (P10)
	Break up the text	More paragraphs were added to break up the text and improve readability.	“…I mean this is just me, and I think the way people absorb information just now again, like, you know, you can prevent this by early detection bullet point da da da, you know, just, kind of, keep it… maybe make it stand out a little bit more. I mean there is not an awful lot of text there so I think I would read it, but I just think maybe the way it's, kind of, laid out could be a little bit more engaging.” (P6)
4. Who we invite
4.1 Interacting with the GP	Mention the option for assistance from an HCP	A sentence was added to refer individuals who require assistance to complete the FIT kit to the screening programme helpline.	“Because obviously it keeps mentioning kits, I assume these are home kits. Which generally, possibly they're living alone, and they can't do it themselves for whatever reason, it doesn't mention whether you can get help from a GP. From a nurse or something… and I think some people obviously being elderly might not be able to do the test properly themselves” (P10)
5. How bowel cancer screening works	Change heading to better reflect the paragraph content (e.g., “The screening kit”)	The heading was amended as suggested.	“Yeah, yeah. I mean as I say it's not actually explaining it. The title says how bowel cancer screening works but obviously that would be quite scientific in terms of trying to understand. So it's more about what the kit looks like sort of thing.” (P10)
	Explain who would do the colonoscopy	A sentence explaining that colonoscopy is typically performed in hospital by a trained HCP was added.	“My first thought… would be, well who would be doing it. Who would be doing the examination? Would you be called in or would your GP do it?” (P12)
	Clarify that people don't need further tests because nothing concerning is found	A sentence was added as suggested.	“Are we saying that most people do not need any further tests because they don't detect anything abnormal in the sample? … So why not say that? Why not say most people do not need further tests because nothing untoward is found? Or nothing outwardly untoward. Because what you're saying there is, what you could read into, that is most people do not need any further tests but it's not explaining why.” (P5)
	Clarify that FIT is done at home	The leaflet specifies that FIT is a home testing kit.	“You could ask if you could take somebody with you or can somebody be there when you're testing. Just in case at certain ages people like my mum were probably a bit frightened. She'd say, would you come with me? Or, you know, would you be there when I'm doing the kit?” (P8)
6. Bowel cancer symptoms	Add that looser poo means runny	The word “runny” has been added in brackets as suggested.	“I mean that one already says loose poo and I thought that meant it's more runny… But then maybe it should say runny then…But looser, I mean when you think about loose you think about tight and loose you see so it's kind of a funny word to use.” (P10)
	Highlight the importance of speaking to a GP if you have symptoms	A suggestion to visit a GP if symptoms persist for 3 weeks or more was added.	“I would highlight that. I wouldn't just grey it. But I would also further highlight or underline perhaps please speak with your GP… If somebody has got these sorts of symptoms for more than three weeks and they can't get hold of their GP, for whatever reasons, the temptation is just to leave it.” (P5)
7. Possible benefits and risks of bowel cancer screening
7.1 Benefits	Clarify that screening is FIT, not colonoscopy	An explanation that home screening is completed using the FIT kit was added.	“Yes sorry I'm a bit unclear what the screening compared to the... the difference between the screening and the colonoscopy, yes.” (P6)
	Use numbers instead of bullet points	The points were numbered.	“Maybe you need to put it like one, two, three, do you know what I mean?” (P6)
7.1.1 Reduces your risk of dying from bowel cancer by 25%	Remove— sufficient to say it reduces the risk of dying	This statistic was removed.	“I'm going to be brutally honest in my perspective if I was looking at it, you know, I'd be like, 25% wow that's quite low… I'd probably not include it.” (P13) “I think yes once you see 25% that makes it evaluative and then all of a sudden you're thinking oh so 75%, you know, have kind of had it by then anyway, is there much point doing it?... And I don't like that 25% personally, but that's just me.” (P6)
7.2 Risks	Describe complications or just remove as colonoscopy is only relevant to 2% of people	This sentence was amended to explain the possible complications of colonoscopy.	“Okay, that is really interesting because it is telling us there that it may cause complications, for example during or after. It would be nice if we knew what those complications might be.” (P11)
	False positive/negative terminology not understood	The explanations of false positive and false negative were retained but the terms themselves have been removed to avoid adding confusion.	“Detect blood in the stools, it might be a false positive. I don't know if that's particularly necessary. Why do you want to detect the false positive?” (P4)
	Reiterate help seeking from a GP	An additional reminder about help seeking was added.	“If you've had the tests, got a lot of negative out of the way and then you're still having the symptoms I think it's good to get back in touch with your GP and maybe go through it all again. Don't ignore, that's what I say.” (P8)
	Include examples of what could cause false positive, e.g. hemorrhoids	Hemorrhoids, anal fissure and diverticulitis have been added as alternative causes of blood.	“Like if you didn't say hemorrhoid I wouldn't know, you know where the bleeding was coming from, So, it needs more explaining and more reassurance.” (P13)
8. Risk of developing bowel cancer
8.1 Risk factors	“Males are 1.8 times more likely to develop bowel cancer than females” statistic may discourage women	The statistic was retained as only one participant expressed concerns that it may discourage women. Other participants felt it would be valuable in encouraging men to participate.	“I don't think it's necessary. I think it should say more men than females develop bowel cancer. But it's trying to highlight the fact that they both get it because this seems to err on the side that females might not need to get checked.” (P10) “Or that males are more likely to get it than females. Because females are more likely to test than males, because females aren't so finicky, if you know what I mean. I would imagine that most women wouldn't think twice about testing. Where a man might be more delicate.” (P2)
	Separate dietary risk factors based on what increases and decreases risk	This sentence was split into two points as suggested.	“I don't get this point where it says ‘eating a diet high in red meat, red and processed meat, low in fibre, vegetables and fruits.' Does that mean eating too many vegetables and fruits or not eating enough vegetables and fruits?” (P10)
	Add a sentence about changing your lifestyle to lower your risk	A sentence about a healthy lifestyle was added after the list of risk factors.	“Because stopping smoking reduces the risk of you getting it sort of thing. I'd like to see that said a little bit more, in these bullet points. Scattered about here and there. If you do this it will reduce your risk.” (P12)
	Order the factors in terms of contribution to risk	As the bowel cancer risk algorithm is hypothetical at this stage it is not feasible to reflect the weighting of different risk factors. However, they are grouped into broad categories: behavioral characteristics, inflammatory bowel diseases and family history.	“I think that having a history of bowel cancer should probably be the first one in the list. Because that's probably more of a hereditary thing isn't it? If somebody in the family has already had it, you're more likely at risk to get it.” (P10)
	Quantify “too much alcohol” and family history	This is challenging to quantify and the relevance also varies between individuals.	“Yeah, but they come up with these things of, you can only drink X amount of units… If they said to me, right you can only have two pints a night, I'd understand. Or you could have only two whiskies a night. But all this other nonsense of units, I don't know what that means.” (P3) “For example, I know my mother had a bowel cancer… So yeah, it would be nice to know when they say family history of bowel cancer, can you quantify that in some way.” (P5)
8.2 Risk calculator	Remove sliding scale sentence	This sentence was removed as suggested, as participants found it to be confusing.	“A sliding scale. It's a little bit wordy that bit. A sliding scale is used so that the factors that contribute most to your risk level are weighted more highly than factors which only impact your… I don't like that sentence at all. I don't understand what it means.” (P12)
	Remove automatically calculated sentence	This sentence was removed as suggested as participants found it to be unnecessary.	“Again that last line where it says ‘A risk level will automatically be calculated for everyone in a screening programme' you'd expect that anyway. So, if everyone gets a screening you would expect it to be part of that… Again, yeah that whole section there could just be removed.” (P10)
	Reduce repetition of the words “risk/risk factors”	This section was condensed and repetition has been reduced.	“I think this risk, risk, risk it has to be clarified out because you're speaking about risks through it, but it's actually meaning slightly different things in different sections. Or it's just too much risk…” (P6)
9. Bowel cancer screening outcomes
9.1 Referred for colonoscopy	None
9.2 Screening intervals	Again reassure help seeking from a GP if needed	A final reminder not to wait until your next screen to seek help if you have concerning symptoms was added.	“I mean presumably if you were worried yourself you could then request your GP… if you had any of the other symptoms before this time limit you could request that in the meantime… Yeah, because it doesn't say there does it. But if you were worried in the meantime between now and year three… but if you were worried about any of the symptoms, you could still… they're not telling you to wait that long are they, that's just what they'll do as a minimum. But if you were worried in the meantime you could still go and see somebody. I'd like to see something like that, yeah.” (P12)
	Add a statistic to support the safety of intervals (e.g., bowel cancer takes ~10 years to develop)	The time it takes for bowel cancer to develop is covered in the introductory sections.	“Well for me it would justify why there's a three-year wait if I knew what evidence was there.” (P9)
	Include examples of risk profiles	Example risk profiles were not added at this stage as these are more likely to be included in alternative communication about collecting risk information or delivering risk results.	“Yeah. Like this little box you've got here, low risk, you know what I mean? People who don't smoke, people who don't drink, people who are between I don't know 18 to 35, it doesn't run in the family, eats a good, balanced diet, exercises. Then you've got moderate. The man, 35 to 50, I mean slightly overweight, eats processed food, might run in the family, doesn't exercise as much. Just a breakdown of, you know, what box am I?” (P13)
	Explain what is meant by “a tiny amount of blood”	A sentence was added to “The screening kit” section to explain that a tiny amount of blood is not visible to the naked eye and is not a cause for concern. The absolute fecal hemoglobin concentration was not added as it is not easily understood.	“What's the difference between a tiny amount and a lot of blood?... If I was to see that… a tiny amount, I would be horrified. And again, what's above average blood. As I say, I wouldn't know the difference. If any blood… tiny amounts or a lot of blood, you would be straight to the doctor.” (P7)
9.3 Flow chart of possible screening outcomes by risk result	Remove horizontal arrows—considered confusing	These arrows were removed as suggested.	“Yeah, I'm trying to understand it. I mean I know it's what I've said but I'm just trying to think is it from top to bottom?” (P8)
	Simplify it, use images or reduce the text	The text in the flow chart was reduced.	“I'm not sure. I'm looking at it thinking there's an awful lot of information there and what does it actually mean? Does that make sense?” (P5)
	Add a column for “worries or symptoms in between”	This column was added to the flow chart as suggested.	“I don't suppose you could have anything else in this flow thing, saying worries in between screenings? Contact your GP, or something, I don't know.” (P12)
	Specify this refers to screening with FIT, not colonoscopy	The use of the FIT kit is specified.	“That's what I'm confused about. So, invited again in one years' time if I fall between the ages of 60 to 74. So if I'm moderate or higher risk then I have a colonoscopy in a years' time, every year?” (P13)
10. Invited for FIT test in 1–3 years	Bold “being aware of symptoms”	This section is highlighted in bold as suggested.	“Okay, so again I like the way you bolded some of it, the result does not have bowel cancer, or it will never develop in the future, but what's important to me is not that bit, that makes me nervous and my blood pressure goes right up. What's important to me is being aware of the symptoms of bowel cancer is very important.” (P6)
11. Invited for colonoscopy	Use percentages not raw numbers	A percentage value was added as suggested.	“Or you could put down 98% I suppose… Yeah. Yeah, probably. Because we sort of understand percentages better than we do numbers sometimes. If you say 99 pence and you say £1, you understand that 99 percent is cheaper than £1.” (P10)
11.1 Picture of the percentage of people referred to colonoscopy	None
11.2 Other causes of blood in your stool	Mention other causes earlier in the leaflet	Hemorrhoids and diverticulitis are cited earlier in the leaflet.	“Where it says down here other things that may cause blood in your poo – hemorrhoids, that wasn't mentioned in the first, the original part. It mentions polyps and things so that's new information here and obviously you could put the whole line here I suppose… Yeah, I think anything that could concern should be at the beginning of the whole list. You're not getting new information at the end of the information guide. Because somebody might have hemorrhoids and just disregard it because… and go actually, there's a new list at the end as well.” (P10)

Codebook thematic analysis results: summary of suggested changes identified by the think aloud interviews.

FIT, fecal immunochemical test; GP, general practitioner; HCP, healthcare professional.

Green, change was incorporated in full.

Orange, change was partially incorporated.

Red, change was not incorporated.

Changes that were most commonly made to the refined leaflet related to increased clarity and improving readability. Examples of this included the introduction of bullet points, simplification of terminology and minimizing unnecessary details.

The most frequent reason for rejecting a suggested change was where only a single participant had proposed the change and others had expressed positive reflections on the same section of text. For example, one participant suggested labelling other organs in the image of the bowel to help them to interpret it, however, multiple other participants were very positive about the existing image.

3.3 Understanding of the revised information leaflet: user testing survey

The user testing survey results are reported in Table 3. At least 90% of participants (18/20) answered each of the user testing survey statements correctly. The previously established threshold of comprehension was 80%, therefore, the leaflet could be considered comprehensible after a single round of user testing. Seven statements were answered correctly by all 20 participants, including statements about who would be sent a FIT kit, how risk factors are used and whether low risk people may be offered a colonoscopy. The fewest number of participants (90%) answered the following two statements correctly: “Most people who do the FIT will need further tests” and “Information about your lifestyle will be used to determine your personal risk of bowel cancer.”

3.4 Wider reflections on risk stratification: think aloud interviews

Think aloud participants expressed their views on risk assessment and risk-stratified screening for bowel cancer throughout the interviews.

3.4.1 Risk factors and calculating risk estimates

The majority of participants expressed positive views on the inclusion of bowel cancer risk factors in the information leaflet, seeing it as a potential “alert for a lot of people” (P8) who may not be aware of their increased risk status. Some participants considered that awareness of the risk factors may incentivize people to be compliant with screening recommendations:

“Actually yeah, if somebody's doubting about having the screening, doing the screening or participating, on seeing the things that could actually help develop an issue, might make them feel more inclined, ‘Well, I'll go and get screened. I'm not going to change my diet or my alcohol intake but I'll do a screening'.” – P9

For many of the participants, reading the list of potential risk factors resulted in a self-appraisal of their own risk status and lifestyle choices. However, no participants vocalized reflections on having a low-risk lifestyle. Individuals who identified with high-risk lifestyle choices appeared to experience a degree of anxiety:

“So that's me, not being active enough. I'm overweight and I can't stand veg, I will not eat veg. So that's it, I might as well quit now, goodbye (laughs).” – P13

Despite this, the use of risk factors in calculating personal risk estimates was well-received and participants appreciated that “as long as you're being honest about it, it will tell you your risk and then you can cut those risks” (P3). This information was therefore also seen as a potential impetus for behavior change to manage personal risk of bowel cancer.

The leaflet stated that “estimating the risk of bowel cancer is not 100% perfect. People who have a high risk are not guaranteed to develop bowel cancer. Similarly, it is not guaranteed that people with a low risk will not develop bowel cancer.” Some individuals found this statement to be reassuring for people in receipt of a high-risk estimate whilst also ensuring that low-risk cohorts remain cautious. Other participants saw this as inevitability or a fact of life, while a minority questioned the purpose of individualized risk scores if they may get cancer regardless:

“It doesn't guarantee that you'll necessarily get bowel cancer, or that the low-risk people might get it. It's just that's the way the cookie crumbles probably.” – P12

3.4.2 Risk stratification and screening outcomes

In general, participants felt that risk stratification for bowel cancer screening was a sensible and acceptable approach to cancer prevention:

“I mean it's what you'd hope… a low-risk patient doesn't need to waste both of their times [patient and HCP] coming back often, but a medium- and high-risk has got to come more often, dependent on their risks…” – P1

The fact that the risk assessment would have a tangible impact on screening outcomes was viewed positively and participants felt that action ability of risk scores was beneficial:

“...I think that's really good. ‘You'll be invited to the next screening.' I think it's nice that people don't feel that they're left in the lurch. They've filled in a questionnaire and then they never hear anything else. That's nice to think that they've put their time in, so the professionals will then put their time in to analyse what's been said. So I think that's good.” – P12

The intensification of screening for groups of the population deemed to be at higher risk of bowel cancer was met with clear enthusiasm and participants found it to be a source of reassurance that the healthcare service would be “taking it serious” (P9) and would “keep an eye on you” (P1). Conversely, most participants were skeptical about a reduction in screening for those at lower risk. For many, a 3 year screening interval initially felt too long as “anything can happen in three years” (P3) and some even advocated for yearly FIT screening for all members of the population, regardless of individual risk, despite being aware that this was not a feasible option at the population level. In general, supporting evidence was requested to demonstrate the safety of an increased screening interval:

“Yes, I mean, personally I would like to know, I mean what is that based on? What are you basing these decisions on for one year or two years?” – P6

The knowledge that people with a concerning level of FHb would be offered a colonoscopy regardless of their risk score was a source of reassurance that helped to alleviate some concerns over increasing screening intervals:

“The first bit's very good. If they detect blood you'll be offered a colonoscopy, so that's excellent. I'd be very happy with that.” – P2

Throughout the interviews, participants repeatedly advocated for greater reassurance and signposting relating to seeking help directly from primary care if symptoms arise between screening rounds. This addition was considered essential if risk-based intervals were to be accepted and some felt that, without it, symptomatic individuals may wait for their next invite to screening. It was thought that this could increase the rate of interval cancers and cancers diagnosed at a later stage:

“Exactly, because I think you always have to make that clear. That don't just wait for the three years to be up… you might not be here to tell the tale… If anything changes then obviously seek help professionally. I think that's always important for people to know. ‘Oh, I'm alright for three years' – no you're not. Just like a lump in our breasts, we don't wait for the next screening'.” – P8

After being informed that bowel cancer takes 10–15 years to develop, participants vocalized feelings of reassurance and comfort around the potential for an increased screening interval:

“If people knew that is how long it would take to develop, then that is comforting to know that. That this is the screening, you have been invited yearly, two years, three years' time. That is more reassuring, and I know that if something goes wrong within those three years, I will still be in that window of getting help.” – P11

4 Discussion

To the authors' knowledge, this is the first study to explore public understanding of written information relating to risk-adapted screening intervals for bowel cancer screening in England and to explore views on risk-adapted screening amongst those presented with information in a similar format to that likely to be used within future bowel cancer screening programmes. The think aloud interviews identified a total of 42 areas of the information leaflet that could potentially be improved. Participants expressed confusion over the harms of screening, particularly in relation to terminology such as “false positive/negative” results, and consequently suggested removing such terms. A notable finding was that many participants repeatedly vocalized a desire for greater reassurance about contacting a GP if they developed concerns or symptoms throughout the leaflet, fearing that the public may otherwise wait for their next invite to screening and risk a potential interval cancer with a later-stage diagnosis. When asked to consider sections of the leaflet communicating information specific to risk stratification, participants found the notion of a risk calculator to be particularly challenging. Although many participants called for more information around reassurance, in the case of risk calculation and the weighting of risk factors, they felt that less information would be more effective and encouraged simplified language. Somewhat unsurprisingly, participants were enthusiastic about the prospect of intensified screening for people at higher risk of developing bowel cancer and no participants expressed concern over this aspect of a risk-based approach to screening. On the other hand, many participants vocalized concerns or reticence over reducing screening using extended screening intervals for those at low risk. Some speculated that a 3 year interval was too long to wait between successive rounds of FIT screening and the prospect of this induced feelings of anxiety or concern. As such, participants suggested the addition of information about the safety of extended screening intervals, specifically the approximate time taken for bowel cancer to develop.

4.1 Implications

Overall, participants in this study demonstrated sufficient understanding of information about risk-stratified intervals for bowel cancer screening. The majority of user testing participants answered all the survey statements correctly in the first round, demonstrating that the refined leaflet had successfully achieved the minimum level of comprehension. The leaflet we have developed here can also be used as a starting point to develop further communication materials. For example, we have adapted the refined leaflet to reflect hypothetical bowel cancer screening programmes with risk-stratified eligibility criteria and FIT thresholds, used in survey study published elsewhere (27).

The results of the think aloud interviews highlight several key requirements for successful public communication: simple terminology, language, and layout should be used, especially when addressing complex concepts where there is increased potential for misinterpretation. We have also elucidated fundamental principles for communicating information specific to risk-based screening. Firstly, the most important information should be prioritized when describing and explaining the process of risk calculation, to avoid overwhelming people with non-essential details. Secondly, the rationale for and safety of reduced screening for low-risk groups must be evidenced if the public are to feel confident accepting such changes. This information is aimed at reassuring the public that less frequent screening would be both safe and based on scientific evidence and may assist in preventing public backlash about programme changes. Finally, clear and repeated reassurance about contacting a GP if symptoms or concerns arise may help to alleviate public concerns about interval cancers, particularly if risk-stratification resulted in some individuals having longer intervals between screening episodes.

The majority of suggested changes to the leaflet related to information that is provided as part of the original NHS leaflet, indicating that the current information is unlikely to meet the needs of some individuals in the screening population. As such, efforts to improve current bowel cancer screening materials should be pursued. We have demonstrated here that use of a think aloud method alongside a quantitative user testing survey is an effective way of refining patient-facing written materials and would be a suitable approach to improving existing materials.

4.2 Comparison with other literature

Several think aloud studies have utilized NHS cancer screening leaflets and other written materials to develop and test public-facing information. Such studies similarly identified that NHS documentation may not be effectively meeting the needs of all members in the target populous (21, 28, 29). Additionally, public understanding of screening harms and associated terminology around false positive or negative results is generally considered to be sub-optimal, as previously demonstrated by research into public attitudes toward screening leaflets for lung and cervical cancer (28, 30). It was, therefore, not unsurprising that think aloud participants found understanding such concepts to be challenging and a source of significant confusion in the initial version of the leaflet. Nevertheless, the fact that effective communication of screening harms to the public has been a long-standing challenge does not mean we should cease striving to improve public understanding of these concepts, particularly when the future of many screening programmes is likely to involve the incorporation of some level of risk stratification (2, 31). This sustained ineffective communication and hence poor public understanding has the potential to impede informed decision-making, a necessary factor in the UK National Screening Committee (UKNSC) criteria for implementing screening programmes (32).

A specific area that participants in this study had difficulty interpreting was the statistic that taking part in FIT screening reduces the risk of dying by 25%. Although responses to this statistic varied, participant reactions were largely negative as some perceived this to be an underwhelming incentive to complete screening and others conflated it's meaning entirely, leading to the recommendation that it be omitted from the refined leaflet. For example, one participant erroneously interpreted the risk of dying from bowel cancer being 75%. A 2019 study exploring how the public interpreted the NHS screening leaflet for cervical cancer reported comparable misunderstanding of statistics conveying the impact of cervical screening on incidence. In this case, participants mistakenly overestimated the number of women who would go on to be diagnosed with cervical cancer (28). Such findings suggest that particular consideration should be given to expressing statistics associated with cancer risk, incidence and mortality and best practice guidelines should be adhered to (33, 34).

Elicitation of emotional responses when reading cancer screening leaflets, such as the way study participants were prompted to engage in self-appraisal when reading about risk factors, has also been previously documented in related contexts (21, 30, 35), including participants in a think aloud study presented with a list of lung cancer symptoms as part of an NHS targeted lung health check leaflet who also experienced thoughts of self-reflection about their own health (30).

4.3 Strengths and limitations

A key strength of this study is the use of an established think aloud method that has demonstrated previous success when employed in related contexts and facilitated concurrent exploration of cognition and processing that may have been unobservable were participants asked to recall their thoughts in retrospect. A further strength is the use of the NHS leaflet as a starting point, ensuring our adapted leaflet was as realistic as possible. A marked protocol ensured sufficient data collection relating to risk-related sections of the leaflet to enable us to meet the aims of the study. Involvement of PPI in the development phase as well as pilot interviews with PPI representatives served to refine readability and comprehension of the leaflet before it was used in the think aloud interviews, potentially contributing to the need for minimum rounds of user testing. Furthermore, exploring participant reflections on risk stratification generated by reading the screening leaflet is likely to better reflect public reactions as the amount and format of information is more comparable to real-life screening programme communication than other qualitative research where participants have either been more or less informed about the approach. A final strength was the addition of the user testing component to quantitatively assess public understanding of the refined leaflet in an independent sample of participants. This setting more closely represents how participants would read the leaflet in a real world setting than in the think aloud interviews where they were prompted to think in greater detail.

On the other hand, our study sample was well-educated and predominantly of White ethnicity. This is a potential consequence of using a market recruitment agency and the prolific online platform as such an approach is susceptible to self-selection bias and may exclude participants with lower digital literacy. Consequently, the final leaflet may not be applicable for people who do not have English as their first language and/or among adults with low literacy levels. Further research should be conducted to develop and test understanding of risk-adapted cancer screening leaflets among these populations. Additionally, as there was only a single round of user testing, we are unable to comment on comparative comprehension between the initial and refined versions of the risk-stratified leaflet. A further potential limitation is that participants in both the interviews and the user testing process may have read the leaflet in more detail as a consequence of the research setting. Furthermore, user testing participants were aware that they would be asked true or false questions before they read the leaflet. Finally, the majority of the sample had previously been invited to attend screening, meaning they would have been exposed to screening information leaflets.

Additionally, the personal beliefs, prior research and attributes of a researcher can impact the researcher-participant relationship and analytical interpretations of results. The researcher who conducted the interviews (LT) is a White female below the screening age range. This may have narrowed the perceived power imbalance due to her younger age but also positioned her as someone who has no personal experience of bowel cancer screening. Nevertheless, personal experience of attendance at screening programmes for female-specific cancers helped to build rapport with some of the participants and the use of an unrelated pilot leaflet about gardening enabled the finding of common ground early in the interview. It is also possible that the research team's prior research and beliefs about the efficacy of risk-stratified screening impacted their interpretation of the results. However, two PPI members were also involved in interpreting the findings to mitigate this.

5 Conclusion

The high number of suggested changes to the initial leaflet suggested that the current NHS bowel cancer screening leaflet may not currently meet the needs of all members of the screening population. In the case of information relating to risk prediction and risk stratification specifically, participants advocated for clear and simple information to prevent confusion and were generally positive about a risk-based approach. This study demonstrates that an educated sample of the public are capable of understanding written information about risk-adapted intervals for bowel cancer screening when delivered as part of screening programme communication and find such an approach acceptable.

References

• 1.

  NHSEngland. Bowel cancer screening: programme overview (2024). Available online at: https://www.gov.uk/guidance/bowel-cancer-screening-programme-overview (accessed July 8 2024).

  • Google Scholar
  • View reference in article

• 2.

  AutierP. Personalised and risk based cancer screening. BMJ. (2019) 367:l5558. 10.1136/bmj.l5558

  • CrossRef
  • Google Scholar
  • View reference in article

• 3.

  CRUK. Bowel Cancer Screening. (2024). Available online at: https://www.cancerresearchuk.org/about-cancer/bowel-cancer/getting-diagnosed/screening (accessed December 6, 2024).

  • Google Scholar
  • View reference in article

• 4.

  TaylorLCHutchinsonALawKShahVUsher-SmithJADennisonRA. Acceptability of risk stratification within population-based cancer screening from the perspective of the general public: a mixed-methods systematic review. Health Expect. (2023) 26:989–1008. 10.1111/hex.13739

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 5.

  TaylorLCDennisonRAGriffinSJJohnSDLansdorp-VogelaarIThomasCVet al. Implementation of risk stratification within bowel cancer screening: a community jury study exploring public acceptability and communication needs. BMC Public Health. (2023) 23:1798. 10.1186/s12889-023-16704-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 6.

  RiedingerCCampbellJKleinWMPFerrerRAUsher-SmithJA. Analysis of the components of cancer risk perception and links with intention and behaviour: a UK-based study. PLoS ONE. (2022) 17:e0262197. 10.1371/journal.pone.0262197

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 7.

  Kelley-JonesCScottSWallerJ. UK Women's views of the concepts of personalised breast cancer risk assessment and risk-stratified breast screening: a qualitative interview study. Cancers. (2021) 13:5813–5813. 10.3390/cancers13225813

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 8.

  Usher-SmithJAMillsKMRiedingerCSaundersCLHelsingenLMLytvynLet al. The impact of information about different absolute benefits and harms on intention to participate in colorectal cancer screening: a think-aloud study and online randomised experiment. PLoS ONE. (2021) 16:e0246991. 10.1371/journal.pone.0246991

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 9.

  BayneMFaireyMSilarovaBGriffinSJSharpSJKleinWMPet al. Effect of interventions including provision of personalised cancer risk information on accuracy of risk perception and psychological responses: a systematic review and meta-analysis. Patient Educ Couns. (2020) 103:83–95. 10.1016/j.pec.2019.08.010

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 10.

  WeinsteinNDAtwoodKPuleoEFletcherRColditzGEmmonsKM. Colon cancer: risk perceptions and risk communication. J Health Commun. (2004) 9:53–65. 10.1080/10810730490271647

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 11.

  SmithSGWolfMSObichereARaineRWardleJvon WagnerC. The development and testing of a brief (‘gist-based') supplementary colorectal cancer screening information leaflet. Patient Educ Couns. (2013) 93:619–25. 10.1016/j.pec.2013.08.013

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 12.

  Usher-SmithJASilarovaBLophatananonADuschinskyRCampbellJWarcabaJet al. Responses to provision of personalised cancer risk information: a qualitative interview study with members of the public. BMC Public Health. (2017) 17:977. 10.1186/s12889-017-4985-1

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 13.

  Usher-SmithJvon WagnerCGhanouniA. Behavioural challenges associated with risk-adapted cancer screening. Cancer Control. (2022) 29:10732748211060289. 10.1177/10732748211060289

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 14.

  TaylorLCLawKHutchinsonADennisonRAUsher-SmithJA. Acceptability of risk stratification within population-based cancer screening from the perspective of healthcare professionals: A mixed methods systematic review and recommendations to support implementation. PLoS ONE. (2023) 18:e0279201. 10.1371/journal.pone.0279201

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 15.

  NHSEngland. NHS bowel cancer screening: helping you decide (2024). Available online at: https://www.gov.uk/government/publications/bowel-cancer-screening-benefits-and-risks/nhs-bowel-cancer-screening-helping-you-decide (accessed April 25, 2014).

  • Google Scholar
  • View reference in article

• 16.

  EcclesDWArsalG. The think aloud method: what is it and how do I use it?Qual Res Sport Exerc Health. (2017) 9:514–31. 10.1080/2159676X.2017.1331501

  • CrossRef
  • Google Scholar
  • View reference in article

• 17.

  GardnerBTangV. Reflecting on non-reflective action: an exploratory think-aloud study of self-report habit measures. Br J Health Psychol. (2014) 19:258–73. 10.1111/bjhp.12060

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 18.

  DarkerCDFrenchDP. What sense do people make of a theory of planned behaviour questionnaire? A think-aloud study. J Health Psychol. (2009) 14:861–71. 10.1177/1359105309340983

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 19.

  EricssonKAFoxMC. Thinking aloud is not a form of introspection but a qualitatively different methodology: reply to schooler. Psychol Bull. (2011) 137:351–4. 10.1037/a0022388

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 20.

  FoxMCEricssonKABestR. Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychol Bull. (2011) 137:316–44. 10.1037/a0021663

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 21.

  SmithSGVartGWolfMSObichereABakerHJRaineRet al. How do people interpret information about colorectal cancer screening: observations from a think-aloud study. Health Expect. (2015) 18:703–14. 10.1111/hex.12117

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 22.

  JohnsonAKHaiderSNikolajukKKuhnsLMOttEMotleyDet al. An mHealth intervention to improve pre-exposure prophylaxis knowledge among young black women in family planning clinics: development and usability study. JMIR Form Res. (2022) 6:e37738.

  • Pubmed Abstract
  • Google Scholar
  • View reference in article

• 23.

  NoushadBVan GervenPWMde BruinABH. Twelve tips for applying the think-aloud method to capture cognitive processes. Med Teach. (2024) 46:892–7.

  • Pubmed Abstract
  • Google Scholar
  • View reference in article

• 24.

  DennisonRABoscottRAThomasRGriffinSJHarrisonHJohnSDet al. A community jury study exploring the public acceptability of using risk stratification to determine eligibility for cancer screening. Health Expect. (2022) 25:1789–806. 10.1111/hex.13522

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 25.

  BraunVClarkeV. Conceptual and design thinking for thematic analysis. Qual Psychol. (2022) 9:3–26. 10.1037/qup0000196

  • CrossRef
  • Google Scholar
  • View reference in article

• 26.

  RobertsKDowellANieJB. Attempting rigour and replicability in thematic analysis of qualitative research data; a case study of codebook development. BMC Med Res Methodol. (2019) 19:66. 10.1186/s12874-019-0707-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27.

  TaylorLCDennisonRAUsher-SmithJA. Public acceptability and anticipated uptake of risk-stratified bowel cancer screening in the UK: an online survey. Prev Med Rep. (2024) 48:102927. 10.1016/j.pmedr.2024.102927

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 28.

  OkanYPetrovaDSmithSGLesicVBruinede. Bruin W. How do women interpret the NHS information leaflet about cervical cancer screening? Med Decis Mak. (2019) 39:738–54. 10.1177/0272989X19873647

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 29.

  AngelovaNTaylorLMcKeeLFearnsNMitchellT. User testing a patient information resource about potential complications of vaginally inserted synthetic mesh. BMC Womens Health. (2021) 21:35. 10.1186/s12905-020-01166-4

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 30.

  JallowMBlackGvan OsSBaldwinDRBrainKEDonnellyMet al. Acceptability of a standalone written leaflet for the national health service for England targeted lung health check programme: a concurrent, think-aloud study. Health Expect. (2022) 25:1776–88. 10.1111/hex.13520

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 31.

  BarrattAJørgensenKJAutierP. Reform of the national screening mammography program in France. JAMA Intern Med. (2018) 178:177–8. 10.1001/jamainternmed.2017.5836

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 32.

  UK National Screening Committee. Criteria for a population screening programme (2024). Available online at: https://www.gov.uk/government/publications/evidence-review-criteria-national-screening-programmes/criteria-for-appraising-the-viability-effectiveness-and-appropriateness-of-a-screening-programme (accessed October 14, 2024).

  • Google Scholar
  • View reference in article

• 33.

  TrevenaLJZikmund-FisherBJEdwardsAGaissmaierWGalesicMHanPKet al. Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers. BMC Med Inform Decis Mak. (2013) 13(Suppl 2):S7. 10.1186/1472-6947-13-S2-S7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 34.

  FischhoffB. Brewer NT, Downs JS, editors. Communicating Risks and Benefits: An Evidence-Based User's Guide [Internet]. Maryland: Food and Drug Administration (FDA), US Department of Health and Human Services (2011). Available online at: https://books.google.co.uk/books?hl=en&lr=&id=lLA2vrcQN_AC&oi=fnd&pg=PA1&dq=Communicating+Risks+and+Benefits:+An+Evidence-Based+User%E2%80%99s+Guide.&ots=iYfDuQWgpl&sig=SMszPCF8SeMLBOA8O2ah_Z_l8Hw#v=onepage&q=Communicating%20Risks%20and%20Benefits%3A%20An%20Evidence-Based%20User%E2%80%99s%20Guide.&f=false (accessed January 28, 2025).

  • Google Scholar

• 35.

  GormanLSRuaneHWoofVGSouthworthJUlphFEvansDGet al. The co-development of personalised 10-year breast cancer risk communications: a ‘think-aloud' study. BMC Cancer. (2022) 22:1264. 10.1186/s12885-022-10347-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar