The disbelief was palpable when Mozilla’s CTO final month declared that AI-assisted vulnerability detection meant “zero-days are numbered” and “defenders lastly have an opportunity to win, decisively.” In spite of everything, it regarded like a part of an all-too-familiar sample: Cherry-pick a handful of spectacular AI-achieved outcomes, miss any of the nice print that may paint a extra nuanced image, and let the hype prepare roll on.
Aware of the skepticism, Mozilla on Thursday offered a behind-the-scenes look into its use of Anthropic Mythos—an AI mannequin for figuring out software program vulnerabilities—to ferret out 271 Firefox safety flaws over two months. In a publish, Mozilla engineers stated the lastly ready-for-prime-time breakthrough they achieved was primarily the results of two issues: (1) enchancment within the fashions themselves and (2) Mozilla’s growth of a customized “harness” that supported Mythos because it analyzed Firefox supply code.
“Nearly no false positives”
The engineers stated their earlier brushes with AI-assisted vulnerability detection have been fraught with “undesirable slop.” Sometimes, somebody would immediate a mannequin to investigate a block of code. The mannequin would then produce plausible-reading bug stories, and sometimes at unprecedented scales. Invariably, nonetheless, when human builders additional investigated, they’d discover a big share of the small print had been hallucinated. The people would then want to take a position vital work dealing with the vulnerability stories the old school method.
