The steps that young people and suicide prevention professionals think the social media industry and policymakers should take to improve online safety. A nested cross-sectional study within a Delphi consensus approach

Discussion: Study ﬁ ndings re ﬂ ect the complexity associated with trying to minimise the risks of communicating online about self-harm or suicide whilst capitalising on the bene ﬁ ts. However, a clear message was the need for better collaboration between policymakers and the social media industry and between government and its international counterparts


Introduction
Suicide is the fourth leading cause of death among young people worldwide (1), and the leading cause of death in many countries including Australia (2)(3)(4).Self-harm (defined here as an act of deliberate self-injury or self-poisoning, irrespective of motive or suicidal intent) ( 5) is also prevalent among young people (6,7) and is a key risk factor for future suicide (8).
Although the reasons for self-harm and suicide are complex, there is concern about the role social media plays in introducing or exacerbating psychological distress among young people (9,10).There are also fears regarding the potential for certain types of content, for example, graphic imagery and livestreams of selfharm and suicidal acts, to cause distress and lead to imitative behaviour among others (11).However, social media is popular among young people and our work has identified several potential benefits (12).For example, it allows young people to feel a sense of community, to seek help for themselves and to help others, and to grieve for people who have died by suicide.Additional benefits include its accessibility, non-stigmatizing nature, and capacity to deliver highly personalized content directly to a user's feed (12,13).While this work highlights the potential for social media to be a useful tool for suicide prevention, we need to identify ways to minimize the risks associated with social media without simultaneously diminishing the benefits.
Potential levers for creating and maintaining a safe and healthy online environment for users include educational approaches, whereby users are provided with the information to keep themselves and their peers safe online; policy approaches, whereby governments develop and implement legislation to support online safety; and industry self-regulation, whereby the social media industry takes responsibility for maximising safety on its platforms (14).
Progress is being made in each of these areas.For example, the #chatsafe program, comprising evidence-informed guidelines plus social media content, is an example of an educational approach that has been shown to better equip young people to communicate safely online about suicide (12,15,16).Although important, many would argue that the onus should not be placed on young people alone to keep themselves safe online.Therefore, additional approaches are required.
Governments in several countries have taken steps towards developing policies designed to regulate the social media industry.For example, the United Kingdom (UK) (17), Australia (18), and the United States of America (USA) (19,20) have all introduced legislation designed to regulate the social media industry.Though these are broad-brush approaches, they provide guidance to platforms on operating safely and impose fines and penalties on companies that do not comply.However, the rapid evolution of social media, together with the lengthy process of passing legislation, means we cannot rely on legislation alone.Further, restrictions imposed in one country does not necessarily prevent distressing information spreading among young people elsewhere.
Finally, the social media industry has also taken steps to improve user safety on their platforms.For example, many platforms have appointed safety advisory boards and developed safety functions that help users control the type of content they view and interact with (14).However, little is known about either the acceptability or perceived utility of these measures among young people.To the best of our knowledge, only one study, conducted by The Samaritans in the UK has examined this.This study was a cross sectional survey of 5,294 people living in the UK (87% of whom had a history of self-harm) and found that more than 75% of the sample had viewed self-harm content online before the age of 14 years (11).Of those who encountered self-harm and suicide-related content, 83% reported that they had not intentionally searched for it; many reported that it worsened their mood, and over three quarters reported that they self-harmed in similar, or a more severe way, after exposure to the content.Despite mixed findings related to the efficacy of trigger warnings (21), study participants reported that a specific content warning (as opposed to a generic one) would have been helpful.Overall, participants were largely in favour of having more control over the content that they see, such as the ability to mute certain types of content or block other users.
Each of these approaches are in their infancy, and it is unlikely that one approach alone will be sufficient to create and maintain safe online environments.Further, questions remain as to how helpful young people and suicide prevention professionals consider existing approaches to be, and what additional steps they believe could be taken by the social media industry and policymakers.

Aims
The aim of this study was to examine what young people and suicide prevention professionals believed that the social media industry and policymakers should do to create and maintain safer online environments in relation to communication about

Questionnaire development
Statements extracted from 149 peer-reviewed articles, 52 grey literature sources, and the roundtables formed the basis of the current questionnaire.Participants were informed that these items were designed to identify what actions participants thought that the social media industry and policymakers could be doing to improve online safety.
There were 158 survey non-forced response items in this section of the survey (See Supplementary File 1).Examples include: "Social media companies should provide clear policies on safe and unsafe online behaviour in relation to suicide/selfharm"; "Government should create a rating system of social media companies against a set of safety standards".Participants were asked to rate the extent to which they agreed with items on a 5-point Likert scale, with 1 being strongly disagree and 5 being strongly agree.Participants were also asked to indicate how social media companies should manage graphic or potentially unsafe content.For these 10-items participants responded using one of the following five response options: "remove", "shadow ban", "allow users to view content but disable interactions", "restrict", and "unsure".

Participants and recruitment
Two expert panels were recruited.One comprised suicide prevention professionals identified via the literature search and one comprised young people recruited via social media advertisements.
Professionals were invited to take part via email and were eligible for inclusion if they were: (1) aged at least 18 years; (2) an expert on self-harm or suicide (e.g., research, teach, or treat self-harm or suicide; and had published research on self-harm or suicide and social media, or contributed to guidelines on communication about self-harm or suicide); and (3) were proficient in English.Young people were recruited via Instagram advertisements and were eligible for inclusion if they: (1) were aged between 15 and 25 years inclusive; (2) lived in Australia; (3) were proficient in English; and (4) had seen, communicated about, or wanted to communicate online about self-harm or suicide.

Data analysis
Data were analysed in Microsoft Excel.The proportion of participants "strongly agreeing", "somewhat agreeing", "neither agreeing nor disagreeing", and "somewhat disagreeing" or "strongly disagreeing" for each item were calculated for both professionals and young people separately.We used a similar approach to the Delphi consensus study in the current analysis.Specifically, items that achieved =>80% of agreement from both panels (either "somewhat" or "strongly") were considered to be strongly endorsed and steps that should be taken, items that received an agreement level (either "somewhat" or "strongly") between 70 and 79.99% were considered moderately endorsed and steps worth considering.If an item received agreement (either "somewhat" or "strongly") from less than 70% of either panel, it was considered to have low endorsement and to unnecessary at this time.In cases where there was large discrepancy in endorsement (e.g., 50% of one group and 70% of the other group agreed with the item) the lowest score was taken as the level of endorsement.This approach allowed us to identify key actions that were considered a priority for social media companies and policymakers by both professionals and young people.For the 10 items that asked participants to rate how social media companies should manage graphic or potentially unsafe content, the proportion of agreement for each of the five response options (e.g., "remove", "shadow ban", "allow users to view content but disable interactions", "restrict", and "unsure") was reported and any major discrepancies between panels were noted in the results.

Sample characteristics
Forty-three young people responded to this survey.The mean age of youth participants was 21.30 years (SD = 2.54, range = 17-25).Seventy-two per cent identified as female, 20.9% as transgender or gender diverse, and 7.0% as male.Just under one half identified as LGBTIQA + (48.8%), and almost one-quarter came from a culturally or linguistically diverse background (23.3%).Most had their own lived experience of self-harm or suicide (81.4%), and/or had supported someone who was selfharming or suicidal (65.1%); 16.3% were bereaved by suicide.
Twenty-three professionals responded to the survey.They included PhD students (17.4%), postdoctoral researchers or lecturers (26.1%), professorial staff (43.5%); 8.7% were in roles such as research advisors or funders.One participant did not report their role.Almost one fifth also worked as clinicians (17.4%).They resided in a variety of countries including Australia (26.1%), UK (17.39%),USA (13.0%), and 4.4% each from Austria, Canada, China, Estonia, France, Germany, Ghana, Hong Kong, New Zealand, South Africa, and Spain.

Key findings
The results presented below are broken down as follows: (1) actions that the social media industry should take in terms of enhancing safety policies, responding to self-harm and suiciderelated content, and staffing; and (2) actions that policymakers should take, including better industry regulation, educational programs, and future collaboration and investment.

Actions the social media industry should take policies, responsibility and reporting
Table 1 presents the views of both young people and suicide prevention professionals with regards to the development and implementation of safety policies, where responsibility for online safety lies, and reporting.Both professionals and young people agreed that companies should have clear and accessible policies that cover content promoting or encouraging self-harm or suicide, graphic depictions of self-harm or suicide, and games, pacts and hoaxes.Participants agreed that policies should be developed in collaboration with platform users (both with and without lived experience), suicide prevention and mental health professionals, legal and communications professionals, other industry professionals and policymakers.There was also strong agreement that social media companies hold responsibility for the content posted on their platforms and that they should maximise user agency by enabling users to control the type of content that they see.
In terms of reporting, participants believed companies should promote a culture of reporting content that violates their policies, should review all self-harm or suicide related reports, and if content is removed they should explain to users why this is the case.There was strong agreement that companies should expand reporting categories to cover a broader range of content.
Managing and responding to self-harm and suicide-related content Table 2 presents the views of participants with regards to the ways in which platforms should manage and respond to selfharm and suicide-related content.
There was moderate agreement that companies should use artificial intelligence (AI) to send helpful resources to users at risk, though less agreement on whether companies should use AI to intervene where risk was detected.There was strong agreement that companies should not allow self-harm or suicide content to appear in people's "suggested content".There was also strong agreement that all companies should have a clearly accessible safety centre that contains evidence-based information and links to support services.Professionals and young people agreed that companies should restrict access to content that could be harmful to others and that membership of a platform should be paused if individuals repeatedly breach safety policies.
There was strong agreement that social media companies should provide content warnings for potentially harmful selfharm or suicide-related content and for content with self-harm or suicide-related hashtags.These should include information about why the content warning exists plus links to resources.There was weaker agreement with how livestreams of self-harm or suicidal acts should be managed, although >50% of both groups agreed that the stream should be removed by platforms immediately.There was strong agreement that platforms should send resources and links to support services to both the person posting the livestream and viewers, and the livestream should be reported to emergency services.However, there was low endorsement of platforms contacting law enforcement more broadly if a user appeared to be at risk of suicide.(37% of young people and 18% of professionals).Finally, there was strong agreement that platforms should actively promote helpful content such as messages that encourage help-seeking plus stories of hope and recovery.Table 3 presents participants' views on some of the safety strategies currently employed by social media companies.There was less consensus across the two groups for most of the items listed in Table 3, however for the most part the two groups did agree that access to the types of content listed should be restricted in some way.That said, >50% of both groups agreed that images of self-harm or suicide methods and locations should be removed, >60% agreed that livestreams should be removed and >70% agreed that videos depicting preparations for suicide should be removed.Around 80% of both groups agreed that content relating to suicide hoaxes, games and challenges should be removed.For several types of content, almost a quarter of participants stated they were unsure what social media companies should do.

Staffing and training
There was strong agreement that social media companies should have safety teams and paid content moderators with appropriate qualifications and experience.Moderators (and other staff who encounter self-harm and suicide-related content) should receive training and ongoing support.Companies should also provide training to influencers or content creators, on how to communicate safely about self-harm and suicide.See Table 4.

Regulation and legislation
Just over half of professionals (56.52%) and just under half of young people (48.84%) agreed that social media companies should be independently regulated by government.However, there was recognition that policy development would need to be fast tracked to keep up with the rapidly evolving social media landscape.There was strong agreement regarding the need for a special department, or regulator, to manage social media policies and that systems should be developed to appropriately monitor adherence to them.
There was lower agreement between participants regarding the type of legislation that should be developed.However, young people strongly believed that legislation should be developed prohibiting social media companies from allowing accounts that encourage or promote self-harm or suicide and that fines should be imposed for breaching this policy.They moderately agreed that social media companies should be held legally accountable for content published on their platforms.See Table 5.

Education and awareness
Table 6 shows that there was strong support for governments to require secondary schools to provide education regarding safe online communication about self-harm and suicide.There was also strong support for public education campaigns.Both groups agreed that educational programs and campaigns should be developed in partnership with young people with lived experience, mental health and suicide prevention experts, and educators.

Collaboration and investment
Table 7 presents the views of participants on the ways they believe government should collaborate with others and future investment.Both young people and professionals agreed that governments should collaborate with their international counterparts to coordinate efforts.They agreed that an international body should be established, that an international set of safety standards be developed, and that governments should work in collaboration with the social media industry to improve online safety.

Discussion
This paper reports on a cross sectional survey that examined the views of young people and suicide prevention professionals regarding the steps that both social media companies and policymakers should be taking to improve the safety of online communication about self-harm and suicide.It also sought the views of both groups of participants on some of the actions that social media companies are already taking in response to selfharm and suicide-related content.

Social media companies
There was strong agreement among participants about some of the more basic measures platforms should be taking to promote safety.For example, by developing and implementing robust safety policies covering self-harm and suicide, moderating potentially harmful content, and hosting safety centres.Similarly, there was strong agreement that safety and moderation teams should be led and staffed by people with appropriate training and experience and that all staff who encounter self-harm and suicide-related content receive appropriate training and support.Although most of the major companies already employ many of these strategies there seemed to be the view that policies should be broadened, made more visible, and their implementation could be strengthened.Further, not only are many of the strategies implemented platform-specific, most have not been tested empirically for either acceptability or effectiveness.Future research should consider ways of testing content-moderation functions to ensure that they meet the needs of social media users, and consider developmental and other differences among users.
There was agreement that platforms should provide content warnings for potentially harmful content including content containing self-harm or suicide-related hashtags.As noted above, there is mixed evidence regarding the efficacy of content warnings (11, 12) and in the overarching Delphi study (23)   Frontiers in Child and Adolescent Psychiatry consensus was not reached about their inclusion.However, in the study conducted by the Samaritans (11), and the current study, participants believed these would be useful, particularly if they were specific to the subject matter and contained links to helpful resources.Again, meeting the needs of social media users is important here given that participants in this study felt that trigger warnings would be helpful despite the evidence suggesting that they can at times be harmful (21).There is a need for future research to test different features and delivery of trigger warnings, specific to self-harm and suicide, that help us understand when, for whom, and in what context, these safety features are helpful.
There was less endorsement regarding how platforms should manage livestreams of self-harm or suicide.All participants agreed that livestreams should be reported to emergency services and that both posters and viewers should be sent links to helpful resources.But there was no consensus regarding how long the livestream should be allowed to run and whether or not comments should be permitted.Managing livestreams appropriately is challenging.For example, allowing the livestream to transmit for a period of time and permitting others to comment, provides an opportunity for intervention.However, it also runs the risk of viewers being exposed to a live suicide act, which would be distressing and could potentially increase the risk of others engaging in similar acts (24,25).It also violates most platforms' policies.
There was strong agreement that platforms should use their AI capabilities to promote helpful content such as psychoeducation about self-harm and suicide, plus messaging that encourages help-seeking and stories of hope and recovery.Just as certain types of self-harm and suicide-related content can be harmful to viewers, it is well established that stories of hope and recovery can be helpful (26, 27) and there was clear support for platforms to use their capabilities to promote this.There was moderate agreement that companies should use AI to proactively detect users at risk and send them helpful information and resources.However, risk can fluctuate rapidly and there is debate in the literature regarding the accuracy of risk prediction tools in general (28,29).There is also concern regarding some of the ways the platforms use their algorithms to direct certain types of (potentially harmful) content to (often vulnerable) users (30-32) and respondents in the current study agreed that AI should not be used for this purpose.Therefore, using AI to detect and respond to people who may be at risk will likely present ethical challenges for companies.However, studies have demonstrated that risk can be detected, with some accuracy, using content posted on platforms such as Twitter and Reddit (33,34) and participants in the current study appeared to be relatively comfortable with the idea of companies using this type of technology if it helps keeps young people safe.Therefore, perhaps using their AI in this way could be a next step for platforms providing no harm is done in the process.
There was moderate agreement that companies provide training to influencers on how to communicate safely about selfharm and suicide to their followers.Influencers have grown in number and popularity over recent years, with many having Frontiers in Child and Adolescent Psychiatry significant numbers of (often young) followers.While some companies do provide broad mental health training to some of their content creators (35), it is important that this extends to self-harm and suicide.The new #chatsafe guidelines provide some guidance for influencers (22), and whilst this is a step in the right direction, uptake by influencers is likely to be limited.As a result, it is also important that the companies themselves support the influencers on their platforms to communicate safely online about self-harm and suicide.There was disagreement between the groups of participants on some items.A notable example was reporting users at risk of suicide to law enforcement.Indeed, some companies have policies that state that if a user is clearly at risk of suicide, law enforcement will be contacted (36).The fact that participants did not support this (except for livestream events as discussed above) may reflect the fact that in some countries suicide remains illegal (37) and that in others, first responders may be unlikely to respond in either a timely or compassionate manner (38).It may also be seen as somewhat heavy handed and a possible breach of privacy.However, it does mean that social media companies need to think carefully about how they respond to users who are expressing acute risk on their platforms and that tailored, country-specific responses are needed.

Policymakers
There was strong agreement for some measures such as the need for specific departments to develop policies relating to social media and to monitor their implementation by the companies.Arguably, in Australia, we are leading the way in this regard with the creation of the e-Safety Commission which plays an important role in bridging the gap between government and the social media industry, in providing public education, and in developing and monitoring adherence to safety standards.That said, the Commission's brief is far broader than self-harm and suicide and perhaps more could be done to strengthen work in this area.
There was strong support for international collaboration and moderate support for an international body to help support online safety efforts.International collaboration on this issue makes good sense.Rates of self-harm and suicide in young people are increasing in many parts of the world (39,40) and social media companies are multi-national, therefore, more coordinated efforts including the development of international safety standards and cross-sector collaboration is a logical next step.It's possible that international bodies such as the World Health Organisation or the International Association for Suicide Prevention could play a role here.
In terms of other steps policymakers could be taking, there was strong agreement that government should support public health campaigns promoting safe online communication.There is some evidence regarding the effectiveness of public health campaigns for suicide prevention generally (41), but to the best of our knowledge, few campaigns exist specifically on this topic or that target young people.One exception may be the #chatsafe social media campaign that was tested in two separate studies and appeared to improve young people's perceived online safety when communicating about suicide and their willingness to intervene against suicide online (12, 42).However, these were relatively small pilot studies and more work is needed to robustly assess the effectiveness of such campaigns.
There was moderate agreement that governments should require secondary school curricula to include education about safe online communication about self-harm and suicide and mixed views regarding primary schools.School curricula are already crowded, but schools are an obvious place to provide education to young people about online safety and an acceptable setting for suicide prevention activities (43).As such, it stands to reason that at least some degree of education regarding online safety when communicating online about self-harm and suicide could be a useful addition to school curricula.It may also be useful for this type of education to extend beyond students and to include both educators and parents/carers.Somewhat surprisingly, there were mixed views regarding blanket regulation of the platforms by government, and legislation making certain types of content posted by users illegal.That said, there was some support (particularly from young people) for legislation prohibiting companies from allowing accounts that promote or encourage self-harm or suicide and that would hold companies accountable for content posted on their platforms.

Limitations and strengths
This was by no means a large-scale representative survey; nor was it intended to be.Rather, the survey items were nested within a larger Delphi study, the main purpose of which was to inform the development of new #chatsafe guidelines.As such, the study findings cannot be generalised beyond the study population and a larger, representative survey is warranted.However, the fact that the survey was nested in the Delphi study is also a strength, as the survey was based on a robust review of the literature plus consultations with key stakeholders, including young people, policymakers, and representatives from social media companies.A further limitation was the uneven distribution of the two panels with more than twice as many young people as professionals, although the level of engagement from young people may also be considered a strength.Additionally, the youth panel comprised only young people from Australia.This was a deliberate decision to facilitate safety management during the study, but it further limits generalisability.

Conclusions
This study examined the views of young people and suicide prevention professionals about the steps that social media companies and policymakers should take to improve online safety when it comes to self-harm and suicide.Although many of the strategies identified are already being implemented, at least to a certain extent, it is clear that more could be done.
Our findings reflect the complexity associated with trying to achieve a balance that minimises the risks of communicating online about self-harm or suicide (i.e., exposure to harmful content) whilst capitalising on some of the benefits (i.e., opportunities for intervention).Indeed, much of our work, and that of others has demonstrated that online communication about self-harm and suicide is both complex and nuanced, and content, or decisions, that may be helpful for some may be harmful for others (13,44).With such a large user base (Meta's platforms currently have around 3.9 billion (45) users and TikTok has 755 million (46)), even small changes to their policies and practices can have a significant impact on suicide prevention efforts worldwide.
A clear message was the need for better collaboration between policymakers and the social media industry and between government and its international counterparts.To the best of our knowledge, no national or international suicide prevention policies include recommendations relating to online safety and to date there is no international body coordinating efforts in this area.In our view addressing these gaps would help to create safer online environments and would help make inroads into reducing self-harm and suicide among young people.

TABLE 1
Current strategies employed by platforms.
Frontiers in Child and Adolescent Psychiatry 06 frontiersin.orgTABLE 2 Policies, responsibility, and reporting.Social media companies should work collaboratively with the following groups to develop policies and educational campaigns and resourcesUsers regardless of living or lived experience of suicide/self-harm (i.e., young people)

TABLE 3
Responding to self-harm and suicide-related content.
If a social media company has been alerted that a user is engaging in or at risk of suicide/self-harm (e.g., via a report or AI), it should send information, resources, and links to support to users via

TABLE 3 Continued
All social media users should be legally mandated to report suicide/self-harm content that breaches policies/laws/guidelines/terms and conditions of use to