Julie Carpenter doesn’t blame human users at all for anthropomorphizing chatbots. The longtime human-AI interaction expert and author of The Naked Android sees generative AI tools, like ChatGPT, as intentionally designed to elicit these reactions from users through highly personalized memories and interactions that replicate social experiences.
These one-sided connections may become even more complicated, and potentially disastrous, when AI-generated erotica is added as an option. Carpenter is apprehensive about OpenAI’s upcoming plans to allow adults to sext with ChatGPT. “I’m for people exploring their sexuality, but I want it in a way that’s safe for them as well as fun,” she says. “People have to be very aware that there’s a surveillance aspect to the data.”
OpenAI first hinted at allowing adult users to generate erotica using ChatGPT two years ago in an official document describing how its models are designed to behave. While the company still seems to be planning its eventual release, the timeline is murky. In recent months, OpenAI’s advisory council of outside experts expressed concerns about the risks of this imminent release, including the possibility of a “sexy suicide coach,” according to a report from The Wall Street Journal. OpenAI declined to comment when reached by WIRED.
People talking dirty to machines, like ChatGPT, is nothing new. “They want to monetize something they see that people are going to try and do anyway,” says Kate Devlin, a professor of AI and society at King’s College London, whose research involves digital sex.
What’s different here is OpenAI’s potential embrace of smut on a mainstream platform. Erotic chatbots are a revenue-driving business, but they are often relegated to niche sites with unremarkable names or Elon Musk’s Grok, known for nonconsensual deepfake images and going into a “MechaHitler” mode.
When the adult mode is released, it’ll be interesting to see what exactly ChatGPT will remember about the sexual proclivities of users. Over the past couple of years, OpenAI has continued to improve the chatbot’s memory feature, which logs user preferences and responds with personalized outputs. Are you a vegan? ChatGPT won’t recommend a steakhouse when asked for restaurant ideas. Want to go on a hike? It knows you’re in San Francisco from past chats, and will suggest reasonably close trails, even if that location context is not included in the prompt.
But, what happens when this type of data logging and hyper-specific personalization meets the horny sexting habits of millions of adult ChatGPT users? Erotic interactions with ChatGPT will further heighten the stakes for user privacy.
Rather than remembering your dietary preferences or what type of movies you enjoy the most, ChatGPT could recall how much you fantasized about furry porn, threesomes at the beach, and piss play, as well as tailor its daily responses with these sexual details in mind. It remains unclear how OpenAI plans to handle these mature conversations.
People who choose to interact with the smut version of ChatGPT at launch may find solace in the “temporary chats” feature. When enabled, these conversations are not shown in a user’s log history or used to improve OpenAI’s models.
But these chats are not as ephemeral as they may feel for users. “For safety purposes, we may still keep a copy for up to 30 days,” reads OpenAI’s website. There’s also a disclaimer included on the site page claiming that “data retention for certain services may be affected by recent legal developments.”
“You’re sharing your most intimate sexual thoughts because you’re lost in the moment.”
Julie Carpenter, author of The Naked Android
Smutty AI chats enable a kind of intimate surveillance that goes beyond someone’s porn-watching history or who they’re subscribed to on OnlyFans. The details are more specific. The interactions can escalate over months to more and more extreme fantasies. The kind of fantasies you never initially considered or would ever express to a human partner.
While the interactions may feel private, the data is still a major risk. If an account is hacked, a security breach occurs, or a government official demands access to your chat logs, troves of highly sensitive and sexual conversations could be exposed in ways the user never imagined.
Information about users’ ChatGPT interactions has previously leaked or been unintentionally made public. OpenAI briefly took ChatGPT down in 2023 when a bug exposed the titles of some users’ chat history. “It’s also possible the first message of a newly-created conversation was visible in someone else’s chat history if both users were active around the same time,” read OpenAI’s statement at the time. Last year, multiple ChatGPT interactions were unintentionally made public by users who didn’t realize their sharing settings would index the chats on Google Search for anyone to discover.
“You’re sharing your most intimate sexual thoughts because you’re lost in the moment,” Carpenter says. “You’re vulnerable in that way because you’re under the impression that you are in this cool, transformative, almost co-constructive creative space that seems like a great way to practice your sexual fantasies.”
The post ChatGPT’s ‘Adult Mode’ Could Spark a New Era of Intimate Surveillance appeared first on Wired.
