Open-weight AI models have skyrocketed in popularity. They allow users to host their own AI systems locally and evade safety guardrails enforced by commercial companies like OpenAI or Anthropic.
Unfortunately, those qualities also make open-weight models popular among people who use AI to generate child sex abuse material (CSAM). As Bloomberg lays out in a shocking new feature, AI models that are partially or fully open-source have led to a major surge in disturbing CSAM material online.
Tracking down perpetrators can be difficult since many of these AI models can be trained on existing CSAM and used to generate many new derivatives without ever connecting to the internet, where they could be identified by experts and reported to law enforcement.
The alarming trend has ballooned into entire communities on the dark web that can attract hundreds of thousands of members, according to Bloomberg.
The rapid rise of open source models, particularly from China, has sparked a major debate, with CEOs of leading AI firms arguing they should be reined in, citing concerns over national security.
However, their use to generate CSAM, a process that’s trivial once safety guardrails are bypassed by “jailbreaking” the models, is growing into a far more immediate and glaring problem.
“There is almost no way to have oversight once you have downloaded a model locally and you’re running it on your private system,” AI child safety expert Rebecca Portnoff told Bloomberg. “That gives offenders a perceived sense of safety.”
According to Portnoff, there are more than 500 open source AI models being used for CSAM, a massive increase over less than a dozen two years ago. Meanwhile, the number of instances of AI-generated CSAM reported to the National Center for Missing and Exploited Children spiked over the 1.5 million mark last year, up from just 67,000 in 2024.
And while open source models continue to be exploited, even AI companies are struggling to meaningfully address the problem with their closed source alternatives. Despite XAI-turned-SpaceXAI-turned-SpaceXSI founder Elon Musk promising for years to put an end to the trend, CSAM generated with the company’s AI models has continued to run rampant. A proposed class action lawsuit against the company accused a man of using Musk’s Grok chatbot to generate some 7,000 sexually explicit images of his stepdaughter.
Experts have found that Musk’s X social media platform continues to be flooded with sexualized images of children.
And as the AI race continues to heat up, child safety advocates are growing concerned that the move-fast-and-break-things approach Silicon Valley is applying could end up making matters even worse.
More on CSAM: Meta Caught Running Ads for Child Abuse
The post A Core Demographic of People Who Love AI: Pedophiles appeared first on Futurism.
