An analysis published this week of vulnerabilities discovered in open-source code using three different artificial intelligence (AI) models found that all three identified the same issue in only one instance out of 121 distinct findings.
Conducted by Echo, a provider of hardened software artifacts and an AI service for discovering vulnerabilities, the report compares the vulnerability discoveries using Claude Mythos from Anthropic, GPT-5.6 Sol from OpenAI and GLM-5.3 from Z.ai.
Of the 446 potential findings first raised, only eight were flagged by all three models, and just one of those survived filtering. Of the 121 distinct findings left, 100 (83%) were found by a single model, 20 by two, and just one by all three. No single model found everything.
Mythos had the most findings pass validation, but GPT-5.6 Sol led on one project, while GLM-5.3 found validated issues that neither of the others spotted. The models also could not agree on severity. One issue in an npm package was rated Critical by GLM-5.3, High by Mythos, and dismissed as documented behavior by GPT-5.6 Sol. Actual validation rated it Low.
Nor did a more expensive AI model guarantee better coverage. GLM-5.3 was the least reliable of the three at judging what could actually be exploited, but at about $22 to run, compared with $303 for Mythos and $30 for GPT-5.6 Sol, it still found validated issues Mythos missed.
Most troubling, many AI-generated severity assessments didn’t hold up. The models rated 45% of their 121 distinct findings High or Critical. After testing, only 26% held up at that severity.
Yuval Saggie, security researcher at Echo, said the report makes it clear that any organization looking to uncover vulnerabilities in their code base will need to invoke multiple AI models. Otherwise, they are only likely to discover a limited number of the vulnerabilities that actually exist, he added.
Of course, that will also increase the total cost of vulnerability discovery, but DevSecOps teams should assume that malicious actors will be making use of multiple AI models to discover as many vulnerabilities as possible. The issue is that once they discover those vulnerabilities, adversaries will use the same AI tools to reverse engineer an exploit in a matter of minutes.
More troubling still, some of those attacks will be targeted to compromise a specific software supply chain, noted Saggie. Adversaries will also become more adept at chaining together low-level vulnerabilities to create more lethal exploits, he added. As such, severity rankings while still relevant are not the only metric DevSecOps teams should be using to prioritize remediation efforts, noted Saggie.
Not every vulnerability is reachable, so each DevSecOps team will need to carefully evaluate what actually needs to be fixed when. In many cases, applying additional security controls in a more timely manner will mitigate potential threats. However, the best approach will be to remediate as many vulnerabilities as possible at the code level.
In the meantime, however, DevSecOps teams should expect that senior IT and business leaders, for better or worse, are going to be asking a lot more questions about the state of application security in the AI era.

