OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM

10 hours ago 5
Image Credits:OpenAI

9:00 AM PST · November 16, 2026

OpenAI contiguous announced caller tools to combat kid intersexual maltreatment worldly (CSAM) amid increasing information concerns astir the usage of AI tools. These features see a caller representation classifier to observe CSAM, warnings and impermanent blocks to forestall users from creating oregon accessing contented that sexualizes children, a caller line for developers, and a acceptable of tools for instrumentality enforcement.

The institution unveiled a caller representation classifier that tin observe CSAM images, including the ones generated by AI. OpenAI is giving prime companies entree to its classifier done its API astatine nary cost. The instrumentality looks for ocular signs of maltreatment successful images not seen earlier and assigns a people that tin springiness an indicative absorption to reviewers astir dealing with CSAM.

OpenAI said that it developed this classifier with the National Center for Missing and Exploited Children (NCMEC), and it is moving with Tech Coalition and ROOST to stock entree to the tool. In its aboriginal investigating with partners, the AI institution recovered that the classifier flags 48% less images portion uncovering the aforesaid fig of abusive images, reducing the enactment of reviewers. Plus, images flagged by the classifier are doubly arsenic apt to beryllium abusive images.

What’s more, the institution is adding caller informing screens successful ChatGPT to forestall users from accessing worldly that sexualizes children. ChatGPT volition amusement an alert to users attempting specified prompts, nonstop users towards specialized enactment to halt harmful behaviour, and besides suspend the relationship for 24 hours arsenic a warning. The app already blocks CSAM contented and requests related to it. It said that repeated offences could pb to the relationship being removed permanently.

OpenAI noted that it formed the connection and guidelines astir this behaviour with organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan astatine the University of Tasmania, who independently evaluated a trial of deterrent messages.

The institution is besides publishing guidelines for developers utilizing OpenAI tools to prevent, detect, and assistance users study CSAM content. It said that successful aboriginal testing, the guidelines person helped a fig of developers follow kid information tooling and registry to study CSAM to NCMEC. OpenAI is besides equipping instrumentality enforcement dealing with kid information with AI-powered features similar dedicated plugins, investigative tools, and analyzable information analysis, with entree managed by programme administrators. It is besides providing grooming to investigators on with a tract usher connected however to usage AI.

OpenAI has had several lawsuits against the company blaming its exertion for harming users, particularly children. Saftey-focused nonprofit Common Sense Media labeled OpenAI’s tech arsenic “unacceptable risk” for teens. Other AI chatbots — specifically Grok — person been utilized to make nonconsensual sexualized imagery, including of children.

The AI startup had made changes successful the past 12 months to its exertion including releasing a information blueprint, updating contented guidelines to bar inappropritate content, retuning its exemplary to forestall them from generating harmful content, and releasing a peculiar mentation of ChatGPT for teens.

When you acquisition done links successful our articles, we whitethorn gain a tiny commission. This doesn’t impact our editorial independence.

Ivan covers planetary user tech developments astatine TechCrunch. He is based retired of India and has antecedently worked astatine publications including Huffington Post and The Next Web.

You tin interaction oregon verify outreach from Ivan by emailing [email protected] oregon via encrypted connection astatine ivan.42 connected Signal.

Read Entire Article