AI-generated code could introduce hidden security risks into U.S. Department of Defense systems, even when the code appears to work correctly on one network, experts warn. Potential threats include malicious behavior on sensitive networks, distributed denial-of-service attacks, operational delays and declining user confidence. Although there is no indication that hackers penetrated sensitive or top-secret systems, the incident highlights weaknesses in automated code generation and conventional security testing.
AI-generated code can behave differently across networks
“It’s possible for your code to behave perfectly fine in NIPR and do malicious, private behavior in a SIPR network. It’s 100% possible,” Williams said.
Extracting data from these networks can be difficult, but malicious code could still be used to attack systems from within, he added. “I think the most likely scenario is some kind of distributed denial of service attack,” he said.
Even if the underlying issue is resolved, Triner said, subtler problems such as intelligence or operational delays could cause damage and undermine users’ confidence in the technology.
Vibecoding makes code harder to scrutinize
As “vibecoding”—using an AI assistant to write code for a product described by an engineer—becomes more common, reviewing that code is becoming increasingly difficult. AI systems can generate far more code than a human developer, making it time-consuming and challenging to examine every part of it.
“When you’re using an agent, you don’t know the supply chain or even the inference chain of what the code was ultimately generated for,” Williams said. “Many of the codes that these LLMs were trained on have vulnerabilities of their own and can feed back into the same types of architecture and architectural issues that we see in other code.”
Security testing may miss malicious backdoors
The government uses Static Application Security Testing, or SAST, to identify flawed code within NIPRNet. SAST scans software for bugs and defects before it is allowed to run.
However, Williams said SAST may not detect deliberately planted backdoors. “Backdoors inserted by hackers are not vulnerabilities, even if we didn’t want them to be,” he said. Such code could therefore evade SAST’s detection. The testing method also typically does not identify structural problems in the logical chain through which code operates.
No evidence of a breach of sensitive systems
There is no indication that the hackers penetrated sensitive or top-secret systems. Still, Williams said the incident should prompt serious concern.
“Architecturally, we believe that any information about NIPR that cannot be verified through open source intelligence will be held in close custody by the Department of Defense, whether or not it touches classified networks,” he said.
Earlier AI-related security concerns
This is not the first time AI-related issues have left the Department of Defense vulnerable. Last year, ProPublica reported that Microsoft uses “digital escorts” on projects with the federal government. U.S. employees with security clearances oversee code written by outsourced workers in China.
Experts who spoke to ProPublica said the arrangement creates particular security risks because U.S. officials are often unable to vet the code themselves. That could leave room for malicious code to be inserted into government systems without their knowledge.
Source: www.wired.com