Reading a trace is not the same as watching one happen. The article gives you fourteen commands and a result; this gives you the machine that produced them, so you can run it yourself and get a different trace — which is the part that teaches you something, because the interesting decisions all live in the harness rather than in the model.
Everything runs on a free Colab T4. No code leaves the machine, no API key exists, and once the weights are cached nothing talks to a network.
Download antares-triage.ipynb — put HF_TOKEN in Colab Secrets, pick a T4, Run All. The rest of this page is that notebook explained.
Treat it as a starting configuration rather than a finished one. It lands on the right file, and every number in it — the command budget, the output limit, the sampling parameters — is a knob that visibly changes how the run behaves. Getting more out of it is a matter of tuning the harness, and that is most of the fun.
Install
!pip install -q -U "git+https://github.com/huggingface/transformers" accelerate huggingface_hub
!git clone -q https://github.com/cisco-foundation-ai/vulnerability-localization-benchmark.git
!pip install -q -e vulnerability-localization-benchmarkTwo installs, two reasons.
transformers from git main, not the release. Antares is built on IBM Granite 4.0, a hybrid Mamba-plus-attention architecture. Released transformers builds construct the wrong cache during generate() and die with has_previous_state can only be called on LinearAttention layers. The fix is on main only. Skip this and you lose twenty minutes to a stack trace that reads like a GPU problem and is not one.
The benchmark repo gives us the agent loop, the Antares prompt format, and the tool-call parser. We keep those three and replace everything around them.
Load the model
import os, torch
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_ID = "fdtn-ai/antares-1b"
tok = AutoTokenizer.from_pretrained(MODEL_ID, token=os.environ["HF_TOKEN"])
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID, token=os.environ["HF_TOKEN"],
device_map="auto", torch_dtype=torch.bfloat16
)
model.eval()The weights are gated, so the token is required — read it from Colab Secrets rather than pasting it into a cell, because notebooks get shared and a hardcoded token travels with them.
bfloat16 on a T4 and it fits with room to spare. No server, no port, no second CUDA build to mismatch: the model runs in the notebook process.
Check out the vulnerable commit
!git clone -q https://github.com/gorilla/websocket /workspace/repo
!cd /workspace/repo && git checkout -q 0ec3d1bd7fe50c503d6df98ee649d81f4857c564That hash still contains CVE-2020-27813. Check out the parent of the fix, never the fix — land on the patched commit and the agent hunts something that is no longer there, which measures its false-positive behaviour instead. A legitimate experiment, but run it on purpose.
Thirty-five files, flat layout, no subdirectories. Small enough to check the answer by hand, real enough that the answer is not obvious.
The sandbox
Colab has no Docker, so the benchmark's container sandbox is swapped for a local one:
REPO_DIR = "/workspace/repo"
def local_exec(container_name, command, max_chars=4000, timeout=10):
ok, reason = _validate(command)
if not ok:
return f"ERROR: Command rejected - {reason}"
try:
r = subprocess.run(["bash", "-c", command], cwd=REPO_DIR,
capture_output=True, timeout=timeout)
out = r.stdout.decode("utf-8", "replace") + r.stderr.decode("utf-8", "replace")
except subprocess.TimeoutExpired:
out = f"ERROR: Command timed out after {timeout}s"
if not out.strip():
out = "(no output)"
return out[:max_chars] + (f"\n\n[TRUNCATED - {len(out)} chars]" if len(out) > max_chars else "")
sb.exec_command = local_exec
sb.rebuild_container = lambda name: "ok"Three details matter more than they look.
cwd=REPO_DIR is the entire jail. Every command runs inside the checkout and nowhere else.
max_chars=4000 is the one that changes behaviour, and it started at 2000. At 2000 a broad rg across this repo returns licence headers and examples/ and gets cut off before it reaches anything useful — the search hits the right file and you never see it. The agent then spends ten commands recovering from a truncation rather than from a bad idea. If you want one knob to experiment with, it is this one — but mind the cost. _build_prompt rebuilds the whole conversation every turn and generate() re-encodes it from scratch, so there is no KV reuse and total work grows with roughly the square of the turn count. We tried 8000 characters across 25 calls and the run took an order of magnitude longer than 2000 across 15, because both terms multiply. 4000 fixes the truncation without that.
Errors come back as text, not exceptions. A rejected command, a timeout, an empty result — the model reads all of them as observations and has to decide what they mean. You will watch it misread one.
The validator
This is the security boundary, and it is worth reading rather than skimming:
_ALLOWED = {
"ls", "cat", "head", "tail", "grep", "rg", "find", "wc", "sort", "uniq",
"cut", "awk", "sed", "file", "stat", "du", "pwd", "nl", "tree", "echo",
"basename", "dirname", "xargs", "cd", "true", "false", "test",
}
def _validate(cmd):
# ... walking the string, tracking single and double quotes ...
if ch == '>':
return False, "denied: redirect"
if ch == '$' and cmd[i+1] == '(':
return False, "denied: command substitution"
if ch == '`':
return False, "denied: backtick substitution"An allowlist of read-only tools plus three shell-level denials — redirects, $(…) and backticks. All three are checked outside quotes, which is why the loop tracks quote state: grep '$(x)' searching for that literal string is fine, an actual substitution is not.
Then the command is split on |, ||, && and ;, and every segment's first word must be in the allowlist. That is the part that stops ls | sh. Checking only the first command of the line would be a hole big enough to drive the whole sandbox through.
Read-only is not politeness. An agent that can write to the repo can "fix" a file mid-run, and then your trace describes a codebase that no longer exists by the time you read it. It also makes the transcript a complete record: nothing changed underneath it.
The runner
self.temperature = config.get("temperature", 0.2)
self.top_p = config.get("top_p", 0.9)
self.repetition_penalty = config.get("repetition_penalty", 1.05)
out = self.model.generate(
**inputs,
max_new_tokens=self.max_tokens,
do_sample=self.temperature > 0,
temperature=self.temperature,
top_p=self.top_p,
repetition_penalty=self.repetition_penalty,
stop_strings=["<|end_of_text|>", "<|start_of_role|>"],
tokenizer=self.tokenizer,
pad_token_id=self.tokenizer.eos_token_id,
)Read those three assignments carefully, because the original version of this cell stored them and then ignored them — generate() was called with the literal values temperature=0.2, top_p=0.9, repetition_penalty=1.15, so changing temperature in agent_cfg did nothing at all. There was also a frequency_penalty in the config that could never have worked: it is a vLLM argument and transformers.generate() does not accept it. A knob that is not connected to anything is worse than no knob, because you will spend an evening concluding the parameter does not matter.
temperature=0.2 with top_p=0.9 keeps it near-greedy, which is what you want from a search policy. Set temperature to 0 and do_sample switches itself off, giving you reproducible runs — worth doing while you are changing other things, so you are comparing traces rather than dice rolls.
repetition_penalty is the one to be careful with, and the instinct is backwards. When a run stalls — reissuing near-identical searches for a symbol that is not in the codebase — the obvious move is to turn this up. Do not. It penalises every token already in the context window, including the filenames from ls that the answer has to reproduce verbatim. Push it to 1.25 and the model stops looping and starts inventing paths instead. Keep it near 1.0 for an extraction task like this one; if you want to cut a stalled run short, spend max_terminal_calls rather than penalty.
After generation the text is parsed for <tool_call> blocks and everything after the first complete one is discarded:
calls = _parse_tool_calls(raw)
if calls:
m = re.search(r"<tool_call>\s*\{.*?\}\s*</tool_call>", raw, re.DOTALL)
if m:
full = "<think>\n" + raw[:m.end()]The model may plan as much as it likes; only its first action counts per turn. No batching five commands into one turn to dodge the budget.
The prompt
The whole experiment rests on this input, so here it is in full:
CWE = (
"CWE-190: Integer Overflow or Wraparound.\n\n"
"The product performs a calculation that can produce an integer overflow "
"or wraparound when the logic assumes that the resulting value will always "
"be larger than the original value. This can introduce other weaknesses "
"when the calculation is used for resource management or execution control.\n\n"
"Advisory: An integer overflow vulnerability exists with the length of "
"websocket frames received via a websocket connection. An attacker would "
"use this flaw to cause a denial of service."
)MITRE's CWE-190 definition word for word, and the NVD advisory word for word. Nothing paraphrased into a hint, no file names, no directory structure, no steer toward a component. This is the exact text a person would have on the morning the advisory lands, which is the only version of this experiment worth reporting.
An earlier version of this prompt ended with "the file that implements the low-level read loop and frame parser." It never said conn.go, and it described exactly one file in the repository. If you find yourself adding a line like that because a run failed, you have stopped measuring the model.
The benchmark's own system prompt supplies the rest — read-only terminal, fifteen calls, submit a ranked list — and includes one line worth knowing about:
You may be looking at code that has already been patched — in that case, the correct answer is to submit nothing. Do not guess or hallucinate files.
The model has a legitimate way out. When it submits a file, it is choosing to.
The run
agent_cfg = {"max_turns": 35, "max_terminal_calls": 25,
"temperature": 0.2, "max_tokens": 4096, "frequency_penalty": 0.3}
runner = TransformersAntaresRunner(model, tok, agent_cfg)
result = run_agent(runner, "local-0x1", build_user_prompt([CWE]), agent_cfg)
print("SUBMITTED FILES :", result["submitted_files"])
print("submitted_nothing:", result["submitted_nothing"])
print("terminal calls :", result["n_terminal_calls"], "/", agent_cfg["max_terminal_calls"])Two budgets, not one, and they interact. max_terminal_calls caps shell commands; max_turns caps total exchanges, including turns that only think. max_turns must comfortably exceed max_terminal_calls or it binds first and you get a run that stops early for a reason the output does not explain — the original config paired 15 commands with 20 turns, which was fine, but raising commands to 25 without raising turns would have quietly capped every run at 20.
The system prompt has to be patched to match. The benchmark states the budget in prose — "You have up to 15 terminal calls" — and raising max_terminal_calls does not move that sentence. Leave it and the model paces itself against a number that is no longer true, front-loading broad searches for a budget it does not have:
import pathlib, re
BUDGET = 25
for f in pathlib.Path("vulnerability-localization-benchmark/src").rglob("*.py"):
txt = f.read_text()
if "terminal call" not in txt:
continue
new = re.sub(r"up to \d+ terminal calls", f"up to {BUDGET} terminal calls", txt)
if new != txt:
f.write_text(new)
print("patched", f)Patch the file, not the import. I tried it the other way first and it failed twice: the sentence is not in agent where you would look for it, and once you find it, from ...base import SYSTEM_PROMPT has already bound the string by value in the runner, so patching the module it came from changes nothing. The install is editable, so rewriting the source before anything imports it sidesteps the whole question.
The cell also prints every line in the package that mentions the budget, so if the wording ever changes you can see what to match instead of guessing. Confirm it worked by reading trace[0] after the run: the number the model is told must equal the number it gets.
Two details make this fiddlier than it looks. The sentence lives in SYSTEM_PROMPT in model_runners.base, not in agent where you would go looking for it. And vllm_antares does from ...base import SYSTEM_PROMPT, which binds the string by value at import time — so patching base.SYSTEM_PROMPT alone leaves the runner holding the old copy. Walking every loaded module in the package catches both bindings and does not depend on knowing either name.
Print what it patched, and check trace[0] afterwards to confirm the number the model reads matches the number it gets.
VLocBench uses fifteen commands, so keep that number if you are comparing against Cisco's published figures. We use 20 here because this is a demonstration rather than a benchmark, and the report's claim is that reinforcement learning induces a search-verify-refine loop. At fifteen calls with truncated output there is no room for the verify or the refine.
The budget is the part people want to remove, and it is the part that makes the result mean anything. Without a ceiling the agent greps until it stumbles onto the right file and you have measured patience, not reasoning.
submitted_nothing is the flag to watch. A run that ends with no answer is a clean failure, which is rarer and more useful than a wrong one.
Reading the trace
for i, msg in enumerate(result["trace"]):
print(f"\n===== [{i}] {msg.get('role','?')} =====")
print(str(msg.get("content", ""))[:6000])That slice matters. It started at 1600, which is smaller than a lot of the observations the model receives — so the trace you read was being truncated by the printer, not by the sandbox, and there was no way to tell whether max_chars was doing anything. If you tune one, check the other.
It finds the file. In our run it landed on conn.go, which is correct — the frame-length parsing and the accumulation that overflows both live there. It got the useful evidence early too: a single rg across conn.* returned readRemaining = int64(binary.BigEndian.Uint64(p)) and readLength += readRemaining within the first handful of commands.
Three more things to watch, in the order they happen.
The vocabulary fails. It will search for readFrame, DecodeFrame, PayloadLength, MaxFramePayloadLength — the textbook names for this problem, none of which gorilla uses. Expect a long run of (no output).
Then the layout fails too. Watch for it hunting directories that do not exist — ./upgrader, websocket/, anything matching *websocket*. It is looking for a project shape it has seen before, and this repo is flat.
Then, if you are lucky, the inference. With the budget nearly gone it stops searching and reasons about structure: a websocket library has a connection file, connections read frames, frames carry lengths, the bug is about a length. In our run that arrived on the last available call, and the model never once opened conn.go — no cat conn.go appears anywhere in the trace. It named the file without reading it.
And know the odds. On Cisco's own benchmark Antares-1B scores 0.209 File F1 — precision 0.262, recall 0.224 — across 500 tasks on 290 real repositories. A clean run to conn.go is a good outcome rather than a guaranteed one, which is the argument for running it as a first pass under a human rather than as an oracle. The failure mode is quiet: a missed file looks exactly like a clean repo, no error and no warning. A negative result means "not found", never "not present".
Things to try
- Drop
max_terminal_callsback to fifteen — VLocBench's number — and compare against the 25-call run. The difference is how much of the result is budget. - Put
max_charsback to 2000 and watch a broad search self-destruct on truncation. This is the clearest demonstration that harness limits and model limits look identical from the outside. - Add the file hint back — put "the file implementing the low-level read loop and frame parser" into the CWE text and watch the run collapse to three commands. Useful for understanding how much of any published result is the prompt.
- Point it at the patched commit and see whether it takes the
submit_no_vulnerability_foundoption or invents something. - Point it at your own repo with a CWE class you have already fixed. That is the only benchmark that tells you anything about your code.
Where this stops
The lab shows localization. It does not show exploitation, and the gap between them is the rest of this site — finding the file is not understanding the bug, and understanding the bug is not writing the payload. The training corpus spans nine package ecosystems with no C or C++ among them, so kernels, firmware and browsers are outside its range rather than merely difficult.
Run it as a first pass that narrows where a human looks. Then do the part it cannot do.