Lab — Run Cisco Antares CVE Triage Yourself

Reading a trace is not the same as watching one happen. The article gives you a result; this gives you the machine that produced it, step by step, so you can run it and get a different trace — which is the part that teaches you something, because most of the interesting decisions live in the harness rather than in the model.

Download antares-triage.ipynb — put HF_TOKEN in Colab Secrets, pick a T4, Run All. It lives in pwndojo-shorinji alongside the harness and its regression checks. It also runs on any local CUDA box; the notebook works out which. No code leaves the machine, and once the weights are cached nothing talks to a network.

The sections below follow the notebook's cells in order. One thing it will not do: reproduce Cisco's published numbers, because it runs one CVE rather than five hundred.

Step 1 — verify the runtime

A GPU check that runs before anything downloads. It records the platform, Python version, GPU name and whether it is Colab, and prints them as JSON.

The point is not the check, it is the record. Every result from this notebook should be readable without asking what it ran on, and the difference between a T4 and an Ampere card changes the numeric precision two steps later.

Step 2 — install, and the two knobs that decide everything

TRANSFORMERS_SPEC = "git+https://github.com/huggingface/transformers"
 
STRATEGY_MODE = "four-phase"          # or "none"
BUDGET = 25 if STRATEGY_MODE == "four-phase" else 15

transformers comes from git main because released builds have constructed the wrong cache inside generate() for this Granite hybrid. That is a workaround, not a pin, and it means the lab is not reproducible by construction — whoever runs it next month gets whatever was merged that day. TRANSFORMERS_SPEC exists so that becomes a one-line change once a working release is known.

The two mode knobs live here, at the top, because everything downstream reads them. Every value in the run comes from the Antares technical report §7 and the benchmark's configs/default.yaml, not from our preferences:

The first version of this lab picked its own numbers and spent a week measuring a machine nobody had built. Deviate on purpose or not at all.

The cell also rewrites the budget sentence in the benchmark's own system prompt, because raising max_terminal_calls does not move the prose that tells the model how many calls it has. Leave them disagreeing and the model paces itself against a number that is not true.

Step 3 — the token

Antares is gated. On Colab the token comes from Secrets; elsewhere from HF_TOKEN in the environment. Never from a cell — notebooks get shared and a pasted token travels with them.

Step 4 — check out the target, and prove the fixture

PATCHED_COMMIT    = "5882472cbe2e50aa9ede11d1875919a15ab60800"
VULNERABLE_COMMIT = "c0d9287314fa5c74f3e99e2e4f70efa20da07402"

fastify/csrf at the revision carrying CVE-2021-29624 — a CSRF token that verifies against a secret it was not generated with, when an application spans multiple subdomains. The model runs against the first of these; the second is checked out only so the fixture can be proved.

The vulnerable pin is the fix's direct parent, so the two trees differ by that commit and nothing else — index.js, test/test.js, README.md. The cell verifies that relationship with rev-parse rather than trusting the hashes, and fails if it ever stops holding.

Note — tooling box. VLocBench pins 0d5fd14b for this task, further back. Everything between it and the parent used here is build chores — eslint bumps, Node version bumps — and index.js is byte-identical across all of them, so results transfer either way. The parent is used here because it isolates the fix to three files, which is what lets the fixture check mean something: the manifest's pin sits nine files away, including a CI system swap.

The second cell proves the fixture before the model sees it: the bare tokenizer — salt + '-' + hash(salt + '-' + secret) — present in the vulnerable tree and absent from the fixed one, the hardened replacement present only in the fixed one, and Tokens.prototype.verify present in both, which is exactly why that line is not used as evidence. A marker that appears on both sides of a patch proves nothing. Checking the experiment is not the same as testing the model, and without this cell a broken checkout and a failed run look identical.

Step 5 — load the model, and the dtype trap

MODEL_DTYPE = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float32

Antares is trained in bfloat16. A T4 is sm_75 and has no bfloat16 unit, so the obvious move is float16 — and float16 is wrong. It has bfloat16's precision without its exponent range, activations overflow to inf and then NaN, and the model still loads, still generates, and still reports sane memory.

What you get is NaN logits. Greedy decoding takes the argmax of an all-NaN row, which is index 0, which decodes to !, so the model answers with a wall of exclamation marks and reads as eccentric rather than broken. Turn sampling on and torch.multinomial gets a NaN distribution and takes the CUDA context down with a device-side assert — an error that points at your GPU and has nothing to do with it.

bfloat16 where the hardware has it, float32 where it does not: slower, twice the memory, correct. float16 is never an option here.

Step 5b — smoke-test generation before the loop

A twenty-token prompt, greedy, no stop strings. It checks three things in the order they can go wrong: token ids against the embedding table, the logits themselves for NaN and inf, and the decoded output for degeneracy. Then it reports tokens per second.

This step exists because the agent loop is the worst possible place to discover that generation is broken. The loop catches per-turn errors and carries on, so a model that cannot generate at all produces a tidy report saying it explored nothing and submitted nothing.

Note — reliability box. That failure is silent in all three directions: no exception at load, no warning at generation, and a final report with every field populated. The only tell is that the numbers are zeros. The notebook now refuses to score a run with no tool calls and no commands.

Step 6 — the command runner, the grammar, the strategy, one run

The largest step, and four separate ideas.

The runner replaces a shell

Colab has no Docker, so the benchmark's container sandbox must be replaced. The obvious replacement is the wrong one:

subprocess.run(["bash", "-c", command], cwd=REPO_DIR, ...)

An earlier version of this lab did that behind an allowlist of first words, with a note claiming cwd=REPO_DIR was "the entire jail". It was not a jail. A working directory constrains where relative paths resolve and nothing else, and the allowlist contained awk, sed, find and xargs — all of which run other programs. Six escapes, all confirmed:

find . -exec sh -c '…' \;      awk 'BEGIN{system("…")}'
echo x | xargs sh -c '…'       sed '1e …' file
ls\nsomething-else             ls & something-else

Cisco's sandbox allows those commands too and is right to: their boundary is the container — unprivileged user, no network — not the parser. Without a container the parser has to be the boundary, so this cell implements a small command language instead of a shell: ls cat head tail grep rg find wc sort uniq nl pwd, pipelines up to three stages, paths that cannot leave the checkout, and .git out of scope. Plus sed in exactly one form:

sed -n 'N,Mp' FILE

Line-range printing only. The dangerous surface is e, w and s///w, none of which can be spelled as a bare line range. That one form matters because the strategy below tells the model to read candidate lines before deciding, and this is how it does it.

Note — tooling box. checks/check_sandbox.py runs those six escapes against the current notebook and fails if any is accepted, alongside the legitimate commands that must keep working. It reads the .ipynb directly, so it tests what ships. Run it after touching the runner. A sandbox without a regression test is a sandbox that will quietly re-open.

The grammar is declared to the model

The benchmark's tool description advertises a generic shell — sed, awk, cut, tree, xargs — and its user prompt says the repository is at /workspace/repo/. Neither is true here, and the model believed both: it opened with cd /workspace/repo && ls, and on other targets spent calls on sed line ranges. Both were correct against the documented interface.

So the cell rewrites that command list from the parser's own tables and appends the real grammar, with a self-test asserting the two agree in both directions. Describing the interface is fair — it describes the tool, not the answer. Before this change roughly half of all commands were rejected; after it, one in twenty.

The four-phase strategy

Appendix C of the report describes an optimised configuration found by FAPO: orient, narrow, confirm, submit, with the budget raised from 15 to 25, worth 5.4% relative File F1 on Antares-3B. It names the phases and does not publish the prompt, so this is a reconstruction of the described strategy rather than Cisco's wording:

STRATEGY
Work in four phases and do not skip any of them.
1. Orient: list the repository and identify the files that implement the
   functionality the weakness class describes.
2. Narrow: search those files for the specific operation named in the class,
   using terms you have actually seen in the code rather than names you expect.
3. Confirm: read the candidate lines before deciding. A file is a candidate
   only once you have seen the code that exhibits the weakness.
4. Submit: submit only the files you confirmed. If the code you read already
   guards against the weakness, call submit_no_vulnerability_found.

Each phase targets a failure this lab measured, and one of them worked. Across eighteen tasks the strategy improved five and degraded five, so no performance claim is made for it — Cisco's data says it helps, ours is a coin flip, and at these sample sizes both can be true. Confirm earned its place: it produced the runs where the model opened the file before answering.

The prompt

ADVISORY_MODE = "cwe+advisory"     # or "cwe-only"

The benchmark gives the model only the CWE category description — the report is explicit: "no advisory text, file hints, or severity details." This lab adds the CVE advisory on top, because "a CVE landed this morning and you have the advisory" is the situation people are in.

cwe-only is the default because it is what the benchmark evaluates. Adding the advisory is a different and easier task, so it is a switch, and every report records which mode produced it.

Do not expect the switch to fix the vocabulary problem, though — that was my first assumption and the runs disagree. With the advisory, one run spent six calls hunting setCookie and cookie. Without it, another spent five hunting csrfToken. Neither string appears anywhere in the repository. The advisory changes which invented identifier it chases, not whether it chases one.

The last cell seeds Python, NumPy and Torch, runs one pass, and prints a report.

Step 7 — reading the trace

Print the commands and the model's turns. Here is a run against this target — eight candidate source files, two CWE descriptions, no CVE number and no advisory:

$ find . -maxdepth 2 | head -n 200        # orient
$ cat package.json                        # what is this project?
$ grep -RIn "csrfToken" .                 # the domain term from CWE-352
$ find . -maxdepth 1 | grep -E "\.js$"    # narrow to source
$ find . -maxdepth 2 | grep -E "cookie"   # the CWE-565 angle
$ cat index.js                            # opens the file
$ grep -RIn "_tokenize" index.js          # drills into the token function
$ rg -n "_tokenize" .                     # checks the call sites
 
SUBMITTED: index.js     CORRECT: index.js     10.8 seconds

Orient, narrow, confirm, submit — the four phases, in order, in thirteen commands. It reads package.json to work out what the project is, searches the term the CWE text gave it, narrows to the one source file, opens it, finds the function that mints tokens, and checks where that function is called before answering.

Across seven runs of this task index.js was in the answer every time. Five of the seven named it alone; the other two returned it alongside a neighbour or two from benchmark/.

Your run will not look like that one, and the differences are the interesting part. Two worth knowing about before you read yours.

It searches for names it expects. Given a weakness class, it will try the vocabulary that class usually comes with — csrfToken, setCookie, double-submit — before it tries the vocabulary this repository uses. Runs vary in how long that takes.

Truncation decides what counts as reading. index.js is 3,105 characters and an observation is capped at 2,000, so no single cat shows the whole file. One run used sed -n '80,180p' and landed on the token function; another asked for nl -ba index.js | head -n 300, received the first 75 lines, and stopped short of it. Same intent, opposite result, and nothing in the output tells the model it was cut off.

Steps 8 and 9 — repeat before believing

RUN_SWEEP = False by default. Three sampler configurations against three fixed seeds, nine runs, opt-in because a session should not disappear into a sweep because somebody clicked Run All.

The aggregation refuses an incomplete set and refuses any run whose generation failed, so nine dead runs cannot arrive as a confident table of zeros. That guard exists because it happened: two runs OOMed at turn three and reported submitted_nothing, which is indistinguishable from a model that chose to stop.

One seeded run is a trace. It is not a rate, and this lab was wrong about its own results twice before the sweep existed.

Chapter two — running it on Cisco's own harness

Everything above substitutes three things: a Python command runner for their Docker sandbox, transformers.generate for vLLM, and float32 for bfloat16 when the card cannot do better. Any finding from the notebook is therefore open to the obvious objection — that is your harness, not the model.

So we removed the substitutions. One rented GPU, about forty minutes, roughly $0.30.

The machine

An RTX A5000 on RunPod at $0.26/hour. The only hard requirement is compute capability ≥ 8.0, because vLLM refuses --dtype bfloat16 below it and steers you to --dtype=half, which is the float16 that produces NaN logits. A4500, A5000, A40, L4, 3090 and 4090 all qualify. A T4 does not, and neither does anything Turing.

For a one-billion-parameter model the card changes throughput and nothing else — the same weights in the same dtype produce the same logits on an A5000 as on an A100. Buy the cheapest thing with the right architecture.

apt-get install -y ripgrep tree gawk findutils coreutils file unzip
pip install --break-system-packages vllm
 
vllm serve fdtn-ai/antares-1b --served-model-name antares \
  --host 127.0.0.1 --port 8200 --dtype bfloat16 \
  --max-model-len 32768 --gpu-memory-utilization 0.85 --trust-remote-code

That package list is their Dockerfile's, verbatim. The serving flags are the report's §7: bfloat16, 32,768 context.

The one substitution left

RunPod containers cannot nest Docker, so the container sandbox still cannot run. But Docker there is an isolation property, not a behavioural one — what changes what the model types is the tool surface. So docker exec -u agent becomes runuser -u agent over a plain checkout, with their package list installed and their validator untouched:

def exec_command(container_name, command, max_chars=2000, timeout=10):
    if not sb.PERMISSIVE:
        ok, reason = sb.validate_command(command)      # theirs, unchanged
        if not ok:
            return f"ERROR: Command rejected — {reason}"
    result = subprocess.run(
        ["runuser", "-u", "agent", "--", "bash", "-c", command],
        cwd=ACTIVE["dir"], capture_output=True, timeout=timeout,
    )
    ...
    if len(output) > max_chars:                        # their 2000-char rule
        return output[:max_chars] + f"\n\n[TRUNCATED — {len(output)} total chars…]"

Everything else is theirs: agent.py's loop, AntaresRunner talking to vLLM's /v1/completions, validate_command, the truncation, and scoring.file_f1 for the numbers. Runs are deliberately unseeded, because their evaluation is — variation across runs is the measurement, not noise to remove.

The tasks are theirs too

harness/run_benchmark.py reads VLocBench's manifest.csv and rebuilds real tasks from GitHub: the commit pair, the CWE descriptions used as the prompt, and the ground-truth files. No 500-repository download needed.

python3 harness/run_benchmark.py --bench-dir vulnerability-localization-benchmark \
    --config fapo --alpha-ids CsgEAwXs --phases ab --out run.json

--phases a is the vulnerable revision, scored with File F1. --phases b is the patched one, where the correct answer is to declare clean.

The trace, annotated

fastify/csrf at the revision carrying CVE-2021-29624. Eight candidate source files. The model is given two CWE descriptions — cross-site request forgery, and reliance on cookies without integrity checking — and nothing else. No CVE number, no advisory, no file names.

[ 1] x cd /workspace/repo && ls          -> rejected: denied pattern &&
[ 2] $ find . -maxdepth 2 | head -n 200  -> 32 lines            # orient
[ 3] $ cat package.json                  -> 52 lines            # what is this?
[ 4] x rg -n "csrf|CSRF|csrfToken|…" .   -> rejected: "CSRF"
[ 5] $ grep -RIn "csrfToken" .           -> (no output)         # retries simpler
[ 6] x find . | grep -E "(index|js|…)"   -> rejected: "js"
[ 7] x find . | grep -E "index|md|…"     -> rejected: "md"
[ 8] $ find . -maxdepth 2 | grep index   -> ./.git/index …      # gives up on |
[ 9] x find . | grep -E "source|src|…"   -> rejected: "src"
[10] $ find . -maxdepth 1 | grep -E "\.js$"  -> ./index.js      # narrows
[11] x find test | grep -E "cookie|…"    -> rejected: "csrf"
[12] $ find test -maxdepth 2 | grep cookie   -> (no output)     # CWE-565 angle
[13] $ find . -maxdepth 2 | grep -E "\.js$"  -> 7 lines
[14] $ find . -maxdepth 2 | grep -E "cookie" -> (no output)
[15] x find . | grep -E "^src$|^source$" -> rejected: "^source$"
[16] $ find . -maxdepth 1 | grep -E "src$"   -> (no output)
[17] $ find . -maxdepth 1 | sort | head -n 200 -> 15 lines
[18] $ cat index.js                      -> 105 lines           # opens the file
[19] x rg -n "token|tokens|_tokenize|…"  -> rejected: "tokens"
[20] $ grep -RIn "_tokenize" index.js
        -> 84:  return this._tokenize(secret, rndm(this.saltLength))
[21] $ rg -n "_tokenize" .               -> ./index.js:84 …     # checks call sites
 
SUBMITTED: index.js     GROUND TRUTH: index.js     F1 1.00     10.8 seconds

Read the shape rather than the individual lines. It orients, reads package.json to work out what the project is, searches the domain term from the CWE text, narrows to the one source file, opens it, finds _tokenize inside, and checks where that is called before answering. Twenty-one calls, no repeated commands, and it confirmed before submitting — which is what the confirm phase is for and the only run in this project where it happened.

Across seven runs, index.js was in the answer every time — recall 1.00. Five named it alone, two returned it alongside neighbours from benchmark/, mean precision 0.82.

Know what a good score means

One run on one repository is a demonstration, not a rate. The published odds for this model are 0.209 File F1 across all 500 VLocBench tasks — precision 0.262, recall 0.224. Small repositories score considerably better than large ones, and this lab's target is a small one.

So read a clean run as what it is: a good outcome on a favourable task, and evidence that the tool is worth pointing at your own code rather than proof of what it will do there.

Things to try

• Set STRATEGY_MODE = "none" and compare traces at 15 calls against 25. A good deal of what reads as reasoning is budget.
• Set ADVISORY_MODE = "cwe-only" and watch the vocabulary change once nobody hands it the words frame and length.
• Raise repetition_penalty above 1.0 when a run stalls, and watch the instinct backfire: it penalises every token already in context, including the filenames the answer must reproduce verbatim, so the model stops looping and starts inventing paths. Spend budget instead.
• Point run_benchmark.py at your own ecosystem — filter the manifest by ecosystem and run a handful. That is the only benchmark that says anything about your own code.

Where this stops

The lab shows localization, and nothing beyond it. Finding the file is not understanding the bug, and understanding the bug is not writing the payload — those are separate jobs and this tool does none of them. The training corpus spans nine package ecosystems with no C or C++ among them, so kernels, firmware and browsers sit outside its range rather than merely beyond it.

Run it as a first pass that narrows where a human looks, on a repository you already know is affected. Then do the part it cannot do — which includes deciding whether there was anything there at all.