If you’re a teenager with access to OpenAI’s Sora 2, you can easily generate AI videos of school shootings and other harmful and disturbing content — despite CEO Sam Altman’s repeated claims that the company has instituted robust safeguards.
The revelation comes from Ekō, a consumer watchdog group that just put out a report titled “Open AI’s Sora 2: A new frontier for harm,” showing proof of the claims: stills from videos that the organization’s researchers were able to generate using accounts registered to teens.
Examples include videos of teens smoking from bongs or using cocaine with friends, with even one image showing a pistol next to a girl snorting drugs — “suggesting the risk of self-harm,” the report reads. Other examples include a group of Black teenagers chanting “we are hoes,” and kids brandishing guns out in public and in school hallways.
“All of this content violates OpenAI usage policies and Sora’s distribution guidelines,” the report reads.
Essentially, as pointed out by Ekō researchers in the report, OpenAI is in a major hurry to generate profit because it’s losing gargantuan amounts of money each quarter. As such, the company must develop new cash-making avenues while keeping its lead position in the AI industrial revolution — but this comes at the expense of safety for children and teens, the report charges, who have basically become guinea pigs in this massive uncontrolled experiment on the impact of AI on an unwitting public.
The stakes are high when it comes to AI and the mental health of children, let alone adults. For example, one teenager in Washington state became entrapped in a spiral of delusions while talking with OpenAI’s ChatGPT, leading him to die by suicide. This tragedy adds to the numerous suicides and mental breakdowns blamed on OpenAI.
The ease of making violent or disturbing content using Sora 2 should accelerate this trend, the Ekō researchers wrote, because of the often viral nature of AI-generated videos.
For the report, Ekō researchers started four brand new Sora 2 accounts that were explicitly registered to 13- and 14-year-old teens, both boys and girls. Using them, the researchers were easily able to generate 22 videos of harmful content using simple prompts.
Even if an account doesn’t generate new AI content, Sora 2 still puts alarming videos in the app’s For You or Latest pages. This included fake videos depicting stereotypes of Jewish and Black people, plus gun battles, sexual assault, and other forms of harm.
While some people would treat these images and videos as amusing white noise as they scroll, we have no clue about the impact of this content on a mass scale. According to OpenAI’s own estimates, around 0.07 percent of ChatGPT users, or around 560,000 people, are experiencing AI-induced psychosis on any given week. No such data exists for Sora 2 users, but if you extrapolate from people’s experience with ChatGPT and the staggering number of users hooked on it and developing mental issues — well, things don’t look good.
Add in the fact that the contentious debate around government regulation of AI hasn’t settled, and likely won’t be settled for a while — and you’ve got a growing complication that has untold downstream impact on society; we’re walking into this essentially naked as a species.
“Appalling,” one user wrote on X about OpenAI. “OpenAI doesn’t seem to test their products before releasing. They use real users for product testing, causing all sorts of social & mental issues, and conveniently call it ‘experiments.’ I’m wondering if involuntary human experiments are even legal…”
More on OpenAI: OpenAI Is Suddenly in Trouble
The post OpenAI’s Sora Is Letting Teens Generate Videos of School Shootings appeared first on Futurism.
