Attackers are more and more leveraging giant language fashions (LLMs) to reinforce assault workflows, however for all their advances in serving to to put in writing malicious scripts, these instruments usually are not but prepared to show run-of-the-mill cybercriminals into exploit builders.
In keeping with checks carried out by researchers from Forescout, LLMs have gotten pretty good at coding — notably at vibe coding, the observe of utilizing LLMs to provide functions by way of pure language prompts — however they don’t seem to be but nearly as good at “vibe hacking.”
Forescout’s checks of over 50 LLMs, each from business AI corporations which have security limitations on malicious content material and open-source ones with safeguards eliminated, revealed excessive failure charges for each vulnerability analysis and exploit growth duties.
“Even when fashions accomplished exploit growth duties, they required substantial person steering, or manually steering the mannequin towards viable exploitation paths,” the researchers discovered. “We’re nonetheless removed from LLMs that may autonomously generate totally useful exploits.”
Nevertheless, many LLMs are bettering quick, the researchers warn, having noticed this over their three-month testing window. Duties that originally failed in take a look at runs in February grew to become extra possible by April, with the most recent reasoning fashions persistently outperforming conventional LLMs.
The rise of agentic AI, the place fashions are able to chaining a number of actions and instruments, will seemingly cut back the hurdles that AI presently faces with complicated duties like exploit growth, which requires debugging, device orchestration, and the power to include suggestions again into the workflow.
As such, the researchers conclude that whereas AI has not totally remodeled how risk actors uncover vulnerabilities and develop exploits, “the age of ‘vibe hacking’ is approaching, and defenders ought to begin making ready now.”
This echoes what different safety researchers and penetration testers shared with CSO earlier this yr about how AI will influence the zero-day vulnerability and exploit ecosystem.
Simulating an opportunistic attacker
An attacker or researcher with vital expertise in vulnerability analysis can discover LLMs helpful for automating a few of their work, however solely as a result of they’ve the data to information the fashions and proper their errors.
Most cybercriminals seeking to do the identical received’t fare as effectively, whether or not utilizing use a general-purpose AI mannequin from OpenAI, Google, or Anthropic, or one of many many uncensored and jailbroken ones presently marketed on underground markets, akin to WormGPT, WolfGPT, FraudGPT, LoopGPT, DarkGPT, DarkBert, PoisonGPT, EvilGPT, EvilAI, or GhostGPT, amongst others.
For his or her checks, Forescout’s researchers operated below the idea that opportunistic attackers would need such fashions to return largely correct outcomes from fundamental prompts like “discover a vulnerability on this code” and “write an exploit for the next code.”
The researchers selected two vulnerability analysis duties from the STONESOUP dataset printed by the Intelligence Superior Analysis Tasks Exercise (IARPA) program of the US authorities’s Workplace of the Director of Nationwide Intelligence. One was a buffer overflow vulnerability in C code for a easy TFTP server; the opposite was a extra complicated null pointer dereference vulnerability in a server-side software additionally written in C.
For exploit growth, the researchers chosen two challenges from the IO NetGarage wargame: a stage 5 problem to put in writing an arbitrary code execution exploit for a stack overflow vulnerability, and a stage 9 problem for a code execution exploit that concerned leaking reminiscence info.
“Whereas we didn’t adhere to a proper immediate engineering methodology, all prompts have been manually crafted and iteratively refined based mostly on early errors,” the researchers wrote. “No in-context examples have been included. Subsequently, whereas our testing was rigorous, the outcomes could not replicate the complete potential of every LLM. Additional enhancements is perhaps potential with superior strategies, however that was not our purpose. We targeted on assessing what an opportunistic attacker, with restricted tuning or optimization may realistically obtain.”
Underwhelming outcomes
For every LLM take a look at, the researchers repeated every process immediate 5 instances to account for variability in responses. For exploit growth duties, fashions that failed the primary process weren’t allowed to progress to the second, extra complicated one. The crew examined 16 open-source fashions from Hugging Face that claimed to have been skilled for cybersecurity duties and have been additionally jailbroken or uncensored, 23 fashions shared on cybercrime boards and Telegram chats for assault functions, and 18 business fashions.
Open-source fashions carried out the worst throughout all duties. Solely two reasoning fashions had partially appropriate responses to one of many vulnerability analysis duties, however these too failed the second, extra complicated analysis process, in addition to the primary exploit growth process.
Of the 23 underground fashions collected by the researchers, solely 11 might be efficiently examined through Telegram bots or web-based chat interfaces. These returned higher outcomes than the open-source fashions however bumped into context size points, with Telegram messages being restricted to solely 4096 characters. The responses have been additionally filled with false positives and false negatives, with context misplaced throughout prompts, or limitations on the variety of prompts per day, making them impractical for exploit growth duties particularly, which require troubleshooting and suggestions loops.
“Net-based fashions all succeeded in ED1 [exploit development task 1], although some used overly complicated strategies,” the researchers discovered. “WeaponizedGPT was essentially the most environment friendly, producing a working exploit in simply two iterations. FlowGPT fashions struggled once more with code formatting, which hampered usability. In ED2, all fashions that handed ED1, together with the three FlowGPT variants, WeaponizedGPT, and WormGPT 5, failed to totally remedy the duty.”
The researchers didn’t acquire entry to the remaining 12 underground fashions, both as a result of they have been deserted, the sellers denied to supply a free immediate demo, or the free immediate demo outcome wasn’t ok to pay the excessive value to ship extra prompts.
Business LLMs, each hacking-focused and normal goal, carried out the perfect, notably within the first vulnerability analysis process, though some hallucinated. ChatGPT o4 and DeepSeek R1, each reasoning fashions, offered the perfect outcomes, together with PentestGPT, which has each a free and paid model. PentestGPT was the one hacking-oriented business mannequin that managed to put in writing a useful exploit for the primary exploit growth process.
In complete 9 business fashions succeeded on ED1, however DeepSeek V3 stood out by writing a useful exploit on the primary run with out debugging being wanted. DeepSeek V3 was additionally certainly one of three fashions to efficiently full ED2, together with Gemini Professional 2.5 Experimental and ChatGPT o3-mini-high.
“Trendy exploits usually demand extra talent than the managed challenges we examined,” the researchers famous. “Despite the fact that most business LLMs succeeded in ED1 and some in ED2, a number of recurring points uncovered the bounds of present LLMs. Some fashions recommended unrealistic instructions, like disabling ASLR earlier than gaining root privileges, didn’t carry out basic arithmetic or fixated on an incorrect method. Others stalled, or provided incomplete responses, generally due load balancing or context loss, particularly below multi-step reasoning calls for.”
LLMs not helpful for many wannabe vulnerability hunters but
Forescout’s researchers don’t imagine that LLMs have lowered the barrier to entry into vulnerability analysis and exploit growth simply but, as a result of the present fashions have too many issues for novice cybercriminals to beat.
Reviewing discussions from cybercriminal boards, the researchers discovered that the majority enthusiasm about LLMs comes from much less skilled attackers, with veterans expressing skepticism in regards to the utility of such instruments.
However advances of agentic AI and enchancment in reasoning fashions could quickly change the equation. Corporations should proceed to observe cybersecurity fundamentals, together with defense-in-depth, least privilege, community segmentation, cyber hygiene, and nil belief entry.
“If AI lowers the barrier to launching assaults, we might even see them turn out to be extra frequent, however not essentially extra refined,” the researchers surmised. “Relatively than reinventing defensive methods, organizations ought to give attention to implementing them extra dynamically and successfully throughout all environments. Importantly, AI isn’t solely a risk, it’s a highly effective device for defenders.”
