As Large Language Models (LLMs) evolve in complexity and are utilized for diverse tasks—especially as agents that interact with computer files, websites, third-party code, and other agents—it’s increasingly challenging for human teams alone to mitigate all potential attacks. “It increases the danger surface and expands the explosion radius,” stated Nikhil Kampal, research scientist at OpenAI and co-creator of GPT-Red.
OpenAI has developed GPT-Red to enhance the safety testing process against future threats. “As more advanced models are introduced, we will already have systems in place capable of identifying new attack vectors,” commented Dylan Hung, research scientist and co-inventor of GPT-Red. Researchers report that they have identified a new type of attack never encountered before.
OpenAI’s primary focus has been on a specific type of attack called prompt injection. This occurs when a hacker exposes LLM instructions to perform unintended actions, such as extracting sensitive information, tampering with a company’s codebase, or generating damaging outputs. In theory, such malicious instructions could be concealed within any text format (including code or websites) that an LLM might encounter.
Training Dojo
To create GPT-Red, OpenAI researchers implemented a self-play loop using an LLM that had not undergone hacker training. In this setup, it initiated attacks on several models designed to defend themselves. Through numerous iterations, GPT-Red significantly improved its attacking capabilities, while the other LLMs enhanced their defensive techniques.
Source: www.technologyreview.com