Follow ZDNET: Add us as a preferred source on Google.
How personal do you get with your chatbot?
Does it interpret your lab results? Help you sort out your finances? Offer advice at 2 a.m. when your worries are particularly existential?
Without thinking about it too deeply, you might be revealing a whole trove of personal information about yourself, and that could be a problem.
At a time when people are increasingly integrating chatbots into their everyday lives, researchers are trying to work out the implications of feeding AI personal information.
Also: 43% of workers say they’ve shared sensitive info with AI – including financial and client data
By now, you’ve likely heard stories of people forging romantic relationships with chatbots or using them as life coaches and therapists. In fact, just over half of US adults use large language models, according to a 2025 study from Elon University. What’s more, chatbots are designed to be friendly and keep people chatting — and talking about themselves.
“The ultimate problem is that you just can’t control where the information goes, and it could leak out in ways that you just don’t anticipate,” said Jennifer King, privacy and data policy fellow at Stanford Institute for Human-Centered Artificial Intelligence.
As abstract as that theory may sound, researchers like King say it’s worth considering exactly what you’re telling chatbots, and what repercussions that info might have in the future.
Here are six things you should know about getting too personal with a chatbot.
1. Memorization, prediction, surveillance
So, what’s the harm in giving a chatbot sensitive information about yourself?
No one is sure, exactly, and that’s the issue. One question researchers have is whether models memorize information and, if so, whether that information can be coaxed back out verbatim or near-verbatim. Memorization is actually one of the core complaints in The New York Times‘ lawsuit against OpenAI. (OpenAI, in a statement from 2024, said “regurgitation is a rare bug” it’s trying to eliminate.)
(Disclosure: Ziff Davis, ZDNET’s parent company, filed an April 2025 lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)
“We’re very dependent on the companies doing the right thing and trying to put guardrails that prevent memorized data from coming out,” King said.
On the internet, people have all kinds of personal information floating around, including in public records, that might end up as training data. Or someone might have uploaded a document, such as a radiology report or medical billing statement, without redacting sensitive information.
A concern is that all of this data might be used for surveillance, King said.
Also: Worried about AI privacy? This new tool from Signal’s founder adds end-to-end encryption to your chats
If that fear sounds alarmist, King called back to Anthropic’s tussle with the Department of Defense in the last few weeks, where the company objected to its product being used for mass domestic surveillance.
“One of the most important things that came out of that was the kind of tacit admission that these things can be used for mass public surveillance,” she said. “This is exactly the type of thing that we would be worried about, that you can use these models to look across so many different data points.”
And even if models don’t have specific data, they might still be able to make predictions about people.
In a piece for Stanford about her team’s research, King gave the example of a request for heart-healthy dinner ideas getting filtered through a developer’s ecosystem, classifying you as a “health-vulnerable” person, and that info ending up in the hands of an insurance company.
King’s research findings showed that it’s not always clear what companies are doing to address these issues. Some organizations take steps to de-identify data before using it for training, such as blurring faces in uploaded photos, which could prevent these pictures from being used for facial recognition in the future. Other companies might not be doing anything at all.
2. Your settings might be too lax
Though platform settings can often be labyrinthine, it’s worth taking the time to understand your options. Some chatbots, like Claude and ChatGPT, offer private chats. If you use Claude’s incognito chat, your conversation will not be saved to your chat history or used for training. Those chats, though, are not fixed settings. The same applies to ChatGPT’s Temporary Chats.
There may be other options in the platforms to delete chat histories or opt out of having your chat used in model training data altogether.
Also: 5 easy Gemini settings tweaks to protect your privacy from AI
King also said it’s good to remember, for example, if you’re using your own account or a work account.
“People either don’t know [or] they lose track of what they’ve been conversing with,” she said. “This is your work context, your work AI, and you’ve been telling it you’re feeling really depressed. There’s no employee expectation of privacy there.”
3. Emotions reveal extra context
Most people are likely used to a certain amount of disclosure when they’re on the internet. Even a Google search can contain sensitive information about a person’s life.
A conversation with a chatbot, though, adds even more information and context.
“A search query is much less revealing, especially about your emotional state, than a whole chat transcript,” King said, comparing a search for something like a suicide prevention hotline to a 1,000-line transcript detailing a person’s innermost thoughts and feelings.
4. Humans might be reading
AI is, quite famously, not human. For some people, that concept might make them more comfortable sharing sensitive information. But just because there’s no human typing back doesn’t mean one might not be able to read your messages.
Also: Can Meta workers see through your Ray-Ban smart glasses? What a security expert says
King noted that some platforms use humans for reinforcement learning, where systems are trained, in part, based on human inputs. For example, if you flag a chatbot response, a worker somewhere in the world might check it in an effort to improve the model. As King said, it’s not always clear when something you type might end up being reviewed by a human.
5. Policy is lagging
What makes any of these points especially tricky is the lack of regulation around how AI companies store sensitive data.
The California Consumer Privacy Act, for example, has certain requirements around how data like medical records need to be treated differently from other forms of data. But regulation in the US may differ from state to state, and at the federal level — well, there is no regulation.
“If we had the law that protected us, it wouldn’t be so much of a risk,” King said.
What to do if you’ve said too much…
If you find yourself cringing because you may have already disclosed too much to a chatbot, you may have a few options. King recommended deleting old conversations and personalizations you might have made for the future.
Whether those steps remove your info from the training data, King said, researchers just don’t know.
Each platform has its own policies and methods for handling personal data, which may require some digging into. Here are links to resources from some of the major players.
- OpenAI, ChatGPT
- Anthropic, Claude
- Google, Gemini
- Microsoft, Copilot
