The new SecureForge method aims to reduce security vulnerabilities in code generated by large language models. Researchers from Stanford University have shown that automatic optimization of system prompts can significantly reduce the proportion of vulnerable Python code without compromising its functionality.
Large language models are increasingly used in programming, but their outputs are not always secure. Even functional code may contain vulnerabilities that could be exploited by attackers.
A team of researchers led by Houjun Liu, Lisa Einstein, and John Jang has therefore developed SecureForge. The method automatically modifies system prompts, which are the basic instructions that the model follows when generating code.
The research showed that simply instructing the model to "write secure code" is not enough. The model may only partially respect security instructions and still generate code that is vulnerable to SQL injection or other common types of attacks.
SecureForge therefore uses iterative testing. The model first receives a realistic programming task and generates code. This code is then checked by a static analyzer called Semgrep. If a security issue is found, another language model suggests modifications to the system prompt, and the entire process repeats.
The researchers used data from the MITRE Common Weakness Enumeration database, which contains known types of software vulnerabilities. They created realistic and commonly worded programming tasks for Python based on 25 significant categories of security errors using GPT-5.4.
One example was a task to create a function that searches for users by name in an SQLite database. Such a task can lead to a SQL injection error if not implemented correctly, even though the task itself does not explicitly mention any vulnerabilities.
The authors initially collected 500 tasks that led to the generation of vulnerable code. They kept half of these for final evaluation and gradually expanded the other half to approximately 80,000 variations.
They used a genetic algorithm called GEPA to optimize the prompts. This algorithm repeatedly tested different versions of system instructions, evaluated the errors found, and suggested modifications.
SecureForge was tested on models such as CodeLlama 7B, Qwen2.5-Coder, Qwen3, Kimi K2, Claude Sonnet 4.6, and several GPT series models.
The results showed significant improvements. When using SecureForge, an average of 11.8 percent of the tested programs contained a security vulnerability. With a standard instruction to write secure code, this proportion was 20.1 percent.
For GPT-5.4, the error rate decreased from 15.8 to 10.1 percent. For GPT-5.4 Mini, it decreased from 21.4 to 12.4 percent, and for GPT-5.4 Nano, it decreased from 22.1 to 15.6 percent.
However, the method also has limitations. It was primarily optimized against known classes of vulnerabilities. Therefore, it is not yet clear how effective it will be in protecting against new or previously unknown security issues.
SecureForge nevertheless demonstrates one possible path towards increasing the security of programming assistants that use artificial intelligence. The same models that can detect vulnerabilities can also create new errors if not properly configured.