Huxley's dystopian novel Brave New World (1) describes a futuristic World State with immense scientific advances, but also psychological manipulation and classical conditioning. Today's “free-to-use” ChatGPT, which is taking the world by storm, should lead to discussions about the disruptive impact that artificial intelligence (AI) could have on our future, how to shape it, and how to avoid dystopian developments. Indeed, AI is an important frontier in public health that requires ethical discussion (2).

ChatGPT is a large language model (LLM) trained with very large amounts of textual data to generate new texts in response to text prompts from humans. Its responses resemble human answers to human questions. The progress and success of this deep learning model has also puzzled its developers at OpenAI (San Francisco, CA) (3). Both concerns and possibilities (4–9) have been raised about the impact and developments of ChatGPT or similar AI.

Like a tsunami, the emergence of chatbots sweeps us into terra incognita. More waves of AI will follow. In such a climate, our opinion targets chatbots to contribute to medical advice from ethical points of view. By medical advice [MA] we mean integrated, private, confidential, dependable, and trustworthy health and medical information for citizens. Numerous articles from 2023 deal with ChatGPT and medicine and MA, but few from the perspectives of ethics (2, 5–7, 10–14).

ChatGPT's ability to provide on-demand and specific answers to questions could surpass the use of “Dr Google” when searching for medical and health-related information (15–17). While “Dr Google” returns torrents of information that citizens must wade through, “Dr ChatGPT” offers users more focused distillations, although the results may still not be accurate.

Considering that chatbots mimic conversational interaction, we ask: What could come next? Where can AI take us, possibly faster than most expect? What can we do? And what should we do? In the following sections, we outline current warnings of chatbots like ChatGPT from developers and calls for ethical discourse. In regards to MA, we sketch potential developments of chatbots and associated risks, hallucinations, and “bullshit.” From an ethics perspective, we address the critical confidentiality of information and data, which serve as key drivers of advancing AI, and close with imperative questions and guardrails to benefit from chatbots and avoid dystopian developments.

A powerful call for ethical discourse came on May 30, 2023, in a one-sentence statement signed by more than 350 AI executives, researchers, and engineers: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war” (18). On May 16, 2023 (19), the OpenAI CEO Altman urged for regulation of AI in a US Senate panel hearing: “We think that regulatory intervention by governments will be critical to mitigate the risks of increasingly powerful models.” Moreover, “‘The Godfather of AI' leaves Google and warns of danger ahead” wrote the NYT on May 1, 2023 (20): “For half a century, Geoffrey Hinton nurtured the technology at the heart of chatbots like ChatGPT. Now he worries it will cause serious harm.”

Clearly, even developers foresee massive potential for disruption by AI technology. Thus, the question arises as to how we can “prioritise responsible and beneficial applications that serve the best interests of society” (13), including utmost reliability, privacy, confidentiality, data protection, and disclosure of AI interests? We think that we should have ethical debate and develop safe-guards and red lines to allow the good and disallow the bad—and given the stakes, to initially err on the side of caution.

One evolutionary step for chatbots like ChatGPT is that “chatting,” which today consists of typing and reading, will become talking and listening. That the voice serves as the AI control and response device, would be in line with the HAL 9000 computer in the science fiction classic “2001: A Space Odyssey.” In that film, HAL (Heuristically programmed ALgorithmic computer) is an advanced AI, programmed to obey and not harm its creators, and should respond to voice instructions of the human crew when controlling their spaceship Discovery One.

Such technically advanced advice service seems neither far-fetched nor distant. Alexa, Siri, and Cortana, produced by Amazon, Apple, and Microsoft, respectively, are already voice-activated and voice-responding internet-connected devices as part of the “Internet of Things (IoTs).” Combined with spoken language systems, citizens will talk and listen to chatbots that combine extensive (personal and general) information with massive computing power. Unlike ChatGPT, which has been closed off from new information since September 2021 (7), advanced AI for MA would have access to real-time information in order to provide up-to-date MA.

What about potential risks to citizens—both patients and doctors—who become “information providers and consumers”? What about potential mind-manipulation—be that intentional or unintentional—of citizens through convincingly worded and reasoned individualized advice? With Huxley's Brave New World in mind, is it possible that the boundaries between human and machine will become so blurred that citizens will no longer be able to distinguish MA provided by chatbots from that given by humans? Or might they not recognize who they give their personal information to? Could weakened encryption of human-machine exchanges reduce individuals' control over data (21) and open doors to ethically unethical surveillance?

While made-up “facts” or hallucinations in AI (7) limit ChatGPT's results in relation to science, representatives of medicine are beginning to weigh its potential utility and benefits for numerous areas and applications. But: In view of the broad interest in ChatGPT, please bear a key point in mind: despite extensive media coverage stating the contrary, ChatGPT is not capable of human levels of thought. It is a sophisticated chatbot that is trained on vast quantities of data to offer persuasively sounding responses. Sometimes these responses are accurate; sometimes they are not. Sometimes its rhetoric is so persuasive that gaps in logic and facts are obscured. In effect, ChatGPT includes the generation of “bullshit” (22, 23) i.e., speech intended to persuade but without regard for truth, and such “bullshit” can be right some of the time. The question is whether citizens should seek MA from such a fallible information source.

The current lack of information about how personal data is “used” makes AI boxes opaque: Are citizens aware of this non-transparent use and what control is in place so that personal data is not shared and disseminated for uses beyond MA? A key driver is that the more information citizens provide to AI, the more personalized (and potentially better) MA can become. However, this could lead patients and doctors to provide ever more information at the expense of privacy and confidentiality, making citizens and their data unduly transparent; thereby, potentially opening the door to other uses of their data.

In a nutshell, the modus operandi of current chatbot success is that “Artificial intelligence could never have been so successful in recent years … if these corporations had not collected masses of data. This information enabled them to train their AI models in the first place. This—in addition to an increase in computing power—is the driver of the current AI boom” (24).

That society will not abandon the potential of LLMs is a realistic prospect. Under this assumption, the following questions (Box) should be urgently discussed in order to consider the mindful use of AI for MA. The answers to these questions seem open, but they must be found, and quickly.

To exemplify the complexities above, let us briefly look at liability. When and how doctors who use medical AI could be held liable under current law is explored step-by-step elsewhere (25). Because AI is new to medical practice in general and to medical advice in particular, and with the lack of case law on liability when using AI, physicians would be entering terra incognita. To offer orientation, Price et al. (25) took the following approach: with a view to more general principles of tort law, examples of likely or potential legal consequences of the use of AI in clinical practice were developed. Importantly, the current legal basis for liability for medical AI, in which MA can play a central role, is unlikely to remain unchanged. As a rule of thumb, whenever AI is used to replace human (clinician) judgment, this may pose safety risks to patients and may render clinicians legally liable (26).

The ever-evolving chatbots have the potential to benefit us in personalized ways, but they also have the potential to manipulate and condition us through effective words and language. As for the power of information and data, they are the fuel for the performance and ultimately the competence of chatbots. As one step toward remedying conceivable misuse of information, the publication practice that authors disclose all possible conflicts of interest should also apply to AI and the companies that develop such products. But: Shouldn't we collect, store, connect, and share information about citizens as little as possible—and if at all, then anonymized and encrypted?

Overall, we have outlined current warnings from AI developers, sketched potential developments and associated risks of using chatbots regarding MA, and provided imperative ethical questions. As humans are unlikely to forego the use of AI, significant ethical challenges need to be addressed. Echoing the cautionary tale in the introduction, we need to guard against bias, protect trust, equality and privacy, and establish a “Code of Conduct for AI in Health Care” (11) and guidelines for MA.

Of course, we should do all we ethically can to benefit from chatbot advice, provided it is medically sound. It is equally clear that we must avoid the danger of Orwellian transparency (27) and of conditioning (Huxley's “mind-manipulation”) to believe in non-sensical information about our bodies and health and in non-sensical MA. The latter would be a recipe for not having to be brave in the new AI world that lies ahead.

TE: Conceptualization, Writing—original draft, Writing—review and editing. PL: Writing—review and editing. DS: Writing—review and editing.