The Hugging Face artificial intelligence repository recently uncovered a serious security breach. Attackers exploited an autonomous AI agent system to gain unauthorized access to internal datasets and credentials within their production infrastructure.
Hugging Face is a prominent open-source AI and machine learning platform that hosts access to over 45,000 models from leading AI providers, actively utilized by more than 50,000 organizations worldwide.
The company is currently investigating the potential impact on partner and customer data and pledges to reach out directly to any affected parties. So far, Hugging Face has reported no evidence of tampering with public models, datasets, or spaces, asserting that its software supply chain remains “confirmed to be clean.”
The security incident originated from Hugging Face’s data processing pipeline. Attackers leveraged a malicious dataset to exploit two code execution vulnerabilities, which enabled them to execute unauthorized code on processing workers. Consequently, they were able to steal cloud and cluster credentials and move laterally across various internal clusters.
According to Hugging Face, “The campaign was carried out using an autonomous agent framework (reportedly based on the Agent Security Research Harness, although the specific LLM remains unknown). The attackers utilized self-migrating command and control mechanisms in a public service to execute thousands of actions across a fleet of ephemeral sandboxes.” This method aligns with predictions about the rising threat of “agent attackers” in the industry. More details are elaborated in the incident disclosure published last Thursday.
In response to the breach, Hugging Face has closed vulnerable code execution paths (including template injection in dataset configurations and remote code dataset loaders), expelled the intruder, rebuilt the compromised node, and revoked all affected credentials while implementing a rotation strategy.
Furthermore, Hugging Face has enhanced its malicious activity detection systems, reported the incident to law enforcement, and is collaborating with external forensic experts to evaluate the breach’s impact.
While the specific model powering the attacker’s agents—be it the jailbroken host model or an unrestricted promiscuous weight model—remains unidentified, Hugging Face has mentioned that the attacker is subject to usage policies. The company’s forensic efforts were hampered by the host model’s guardrails.
“The key takeaway for defenders is to prepare and vet capable models that can operate within their own infrastructure prior to an incident, thereby preventing guardrail lockouts that could allow attacker data and credentials to escape the environment,” Hugging Face added.
Users are urged to rotate their access tokens and review recent account activity for any suspicious behavior. Hugging Face also committed to continuing its updates on defending against AI-related attacks.
This marks the first security breach impacting a platform associated with AI agents, although Hugging Face has a history of prior incidents. The company previously recommended revoking authentication secrets for certain members and transitioning to fine-grained access tokens following breaches of its Spaces platform two years ago.
In recent years, malicious actors have exploited the Hugging Face platform to disseminate harmful AI/ML models, deploy information-stealing malware, and circulate thousands of Android malware variants.
Security teams document 54% of successful attacks but only issue warnings for 14%. The rest typically goes undetected. Picus’ whitepaper elaborates on how to test your SIEM and EDR rules during breach and attack simulations to enhance threat detection.
Source: www.bleepingcomputer.com