The campaign directed users to roughly 50 apps across Google Play and Apple’s App Store. Researchers found that approximately 300 of the 350 ads promoted AI image- or video-generation tools, many of them connected to developers based in China. By the time WIRED began investigating, several of the apps had already been removed.
Eight identical apps on Apple’s App Store generated explicit images of adults, and several were associated with Chinese developer names. After opening one of the apps, users were shown explicit AI-generated content and a menu of sexual scenarios, including a violent scenario labeled “blackmail knife.” WIRED identified seven additional apps with the same design, explicit material, and developer contact information.
“The App Store is designed to be a safe and trusted place for our users, and we have no tolerance for developers who intentionally try to circumvent our review processes,” Apple spokesperson Adam Dema said. Apple said the developers initially submitted apps that complied with its app review guidelines but later introduced “nudify” functionality that violated its rules. Of the 20 apps investigated, 17 had already been removed, while Apple said it had taken action against the remaining three and seven additional apps identified by WIRED.
Google spokesperson Dan Jackson said the company prohibits sexual content and apps that facilitate it. Google’s policy on preventing non-consensual intimate content outlines the company’s approach. “We continue to take proactive steps to detect and remove apps with harmful content,” Jackson said, adding that Google had removed all of the apps included in the report. The company said it had suspended hundreds of violating apps and restricted related search terms, including “nudity,” from Google Play.
“The proportion of Chinese developers creating nudify apps is increasing every year,” said David Pham, a former Meta employee who investigated non-consensual intimate images and abusive behavior on the platform. Pham said he encountered similar advertisements while working at Meta and noted that advertisers promoting these tools often attempt to bypass platform safety systems.
“Every time a platform or an investigator built a new detection model or a new way to do some kind of enforcement, they immediately built a new way to avoid detection, within a day, sometimes within five days,” Pham said. Advertisers evade detection by changing domains, concealing who posted the material, and blurring or subtly modifying images. Pham, who left Meta last year, said the company has the financial and technical resources to address the problem. “I want to live in a world where CSAM and NCII material isn’t sent directly to your phone,” he said.
Meta’s systems for detecting advertisements containing child sexual abuse material (CSAM) and non-consensual intimate images (NCII) appear to require further improvement. A recent Indicator Media report found that more than 11,000 AI nudifier advertisements had appeared on Meta’s platforms since February, with many reportedly linked to repeat offenders. WIRED’s analysis also identified hundreds of ads containing explicit material similar to the content found by researchers from the Tech Transparency Project. Meta said malicious advertisers continually use adversarial tactics to evade detection and that it is incorporating the language identified by WIRED into its moderation systems.
According to the Tech Transparency Project’s investigation, one advertisement involving a teenager whose real-world identity was identified by researchers was reported to Meta after reaching four accounts in Europe. However, by the time Meta removed the ad several days later, more than 330 people had viewed it. “This isn’t something that some small nonprofit doing research stumbles across and has to investigate and report on,” said TTP’s Paul. “We’re talking about one of the most technologically and financially well-resourced companies on the planet.”
Source: www.wired.com