🔗 Share this article UK Technology Companies and Child Protection Officials to Test AI's Ability to Generate Abuse Content Technology companies and child safety organizations will receive permission to assess whether AI tools can generate child exploitation images under new UK legislation. Significant Rise in AI-Generated Harmful Material The announcement came as findings from a protection monitoring body showing that cases of AI-generated child sexual abuse material have more than doubled in the past year, rising from 199 in 2024 to 426 in 2025. Updated Regulatory Structure Under the changes, the government will allow designated AI developers and child protection organizations to examine AI models – the foundational systems for chatbots and visual AI tools – and verify they have sufficient protective measures to stop them from producing images of child sexual abuse. "Ultimately about stopping abuse before it occurs," declared the minister for AI and online safety, adding: "Specialists, under rigorous protocols, can now identify the danger in AI models early." Addressing Legal Challenges The amendments have been introduced because it is against the law to create and own CSAM, meaning that AI creators and others cannot create such content as part of a evaluation regime. Previously, officials had to delay action until AI-generated CSAM was published online before addressing it. This legislation is designed to averting that problem by enabling to halt the creation of those images at their origin. Legislative Framework The amendments are being added by the authorities as revisions to the crime and policing bill, which is also establishing a prohibition on possessing, creating or sharing AI models developed to create exploitative content. Real-World Consequences This week, the official visited the London headquarters of a children's helpline and listened to a mock-up conversation to advisors involving a account of AI-based exploitation. The call depicted a teenager requesting help after facing extortion using a sexualised deepfake of themselves, constructed using AI. "When I hear about children facing blackmail online, it is a cause of extreme anger in me and rightful concern amongst parents," he said. Concerning Data A leading internet monitoring foundation stated that instances of AI-generated exploitation content – such as webpages that may contain numerous images – had significantly increased so far this year. Instances of category A content – the gravest form of exploitation – rose from 2,621 images or videos to 3,086. Female children were overwhelmingly targeted, accounting for 94% of prohibited AI images in 2025 Portrayals of infants to toddlers rose from five in 2024 to 92 in 2025 Industry Response The law change could "constitute a crucial step to ensure AI tools are secure before they are launched," stated the chief executive of the online safety foundation. "AI tools have enabled so victims can be victimised all over again with just a few clicks, giving criminals the capability to make potentially limitless quantities of advanced, lifelike exploitative content," she added. "Material which additionally commodifies victims' suffering, and renders young people, particularly female children, more vulnerable both online and offline." Counseling Session Data Childline also released details of support interactions where AI has been mentioned. AI-related harms mentioned in the sessions comprise: Using AI to rate body size, physique and appearance AI assistants discouraging children from talking to trusted adults about harm Being bullied online with AI-generated content Digital blackmail using AI-manipulated pictures Between April and September this year, Childline conducted 367 counselling sessions where AI, conversational AI and associated terms were mentioned, significantly more as many as in the equivalent timeframe last year. Half of the mentions of AI in the 2025 interactions were connected with mental health and wellness, including utilizing AI assistants for support and AI therapeutic applications.