Summary
Pointed at a codebase with no guidance, an agent produces a long list of plausible-sounding findings and very few real ones. Given a narrow scope, the build, a sanitizer and a rule that every claim needs a reproducing test, the same model found two genuine memory-safety bugs in a protocol parser in an afternoon. Most of the gain came from the setup around the model, not from the model.
Background
Vulnerability research is mostly reading code with a suspicious mind, which looks like the kind of work an agent should be good at. The failure mode is confidence: agents describe a bug convincingly whether or not one exists, so the cost shifts from finding candidates to verifying them.
Method
I write a short brief for each session and keep it in the repository so it can be reused and diffed. The brief names the target, the files in scope, the tools the agent may call, and the rules for reporting.
task: audit the packet parser for memory-safety bugs
scope: src/mqtt/parse*.c, include/mqtt/*.h
tools: [ripgrep, clang-tidy, build_asan, run_fuzz_corpus]
rules:
- cite file and line for every claim
- reproduce with a failing test before reporting
- stop and ask if a fix needs a design change
Findings
Requiring a reproducing test removed almost every false positive, because an agent that cannot make an address sanitizer complain has to say so. Giving it the fuzzing corpus as a tool turned vague suspicions about length fields into concrete crashing inputs.
Takeaways
Treat the agent as a fast, tireless junior who must show its work. Invest in scope, tools and verification, and keep a human in the loop for anything that changes behaviour.