Repository navigation
docs(swe-router): say plainly that the shipped router data is illustrative - #39
Merged
shekharprateek merged 6 commits intoOct 20, 2026
Conversation
…dry-run api_key_env was accepted by the validator and ignored by the harness. A user following the pattern the docs recommend -- gateway token in an environment variable, kept out of the committed config -- got a run that authenticated with the literal string "local", because every agent path read config.api_key directly. runner_config even validated that the named variable was set, so the run looked correctly configured right up to the 401. The one exception was agent=strands, wired in PR #32, which is how the gap surfaced: one path honoured api_key_env and five did not. Now resolved_api_key() on every path: claude (both the subprocess env and the inline settings JSON, including apiKeyHelper), codex, pi and omp. Two related credential-handling fixes, since the same token now genuinely reaches these files: - benchmarks/scripts writes the endpoint apiKey into pi's models.json and omp's models.yml. Both were created at the default umask, so a real gateway token landed in a group- and world-readable file. Both are now chmod 0600. - --dry-run printed the claude command verbatim, and that command carries its settings as an inline JSON argument containing ANTHROPIC_API_KEY. Dry-run output routinely gets redirected to a file or pasted into a ticket. _redact_api_key replaces a real key with ***REDACTED***; it deliberately leaves the "local" placeholder alone, because that is not a secret and blanking the string would also mangle paths like ~/.local/bin. Also adds the --api-key-env flag to the harness. runner_config already read args.api_key_env, but run-swe-headless never defined it, so passing it was an argparse error and the YAML config was the only way in. Nine new tests: that each of the five agent paths reads the environment variable, that both config files land 0600, and the three redaction cases. Verified by hand that `--api-key-env TOK --dry-run` is accepted and that the secret appears zero times in the output. 614 tests pass, no new failures. ruff, mypy and bandit (0 issues at every severity) clean.
Replaces the scrub-the-output redactor with one that never puts the secret in the output to begin with. CodeQL flagged the previous version (py/clear-text-logging-sensitive-data, high) and it was right to. Two things worth recording. First, the alert only appeared BECAUSE of this branch. Before it, config.api_key was an inert literal and the dry-run printed it with no complaint. resolved_api_key() reads os.environ, which CodeQL treats as a credential source, so the value is now genuinely tainted all the way to the print. The scanner started objecting at the moment the field started carrying a real secret. Second, scrubbing was the weaker design regardless of the scanner. It has to find every rendering of the secret in an already-built string, and a key that one of the command builders splits, escapes or re-encodes survives a plain str.replace. _display_config instead hands the builders a copy whose api_key is the "local" placeholder, so there is nothing to find: the printed command never contained the credential at any point. Clearing api_key_env on the copy is load-bearing. Leaving it set would send resolved_api_key() back to the environment and return the real token, defeating the whole thing. Bedrock and kiro runs have no endpoint credential, so the copy is skipped and the config is returned unchanged. Tests follow the behaviour rather than the helper: one asserts the real config does resolve the secret and that it is still absent from the built command, one pins that api_key_env is cleared and not just the literal, one pins the Bedrock no-op. Verified by hand that the dry-run output contains the secret zero times and prints ANTHROPIC_API_KEY": "local". 614 tests pass, ruff clean.
fix(harness): consume api_key_env, restrict credential files, redact dry-run output
…ative Masking the vended model names makes the skill a demonstration rather than something a developer can act on, and neither README said so. Both still framed Step 2 as install-and-use. Two concrete consequences, verified against the masked files: - The model it names out of the box is something like `closed-mid-gen6`, which nobody can switch to. - model-aliases.json no longer contains any real vendor spelling, so a real id an assistant offers does not match the shipped list. `--available claude-opus-5-5`, `us.anthropic.claude-sonnet-5-5` and `qwen3.8-27b` each return `no_candidates` rather than a recommendation. Both are fixed the same way, and the fix is already the documented path: run Step 1, regenerate models.json with build_vended_models.py -- which still emits REAL ids, it is only the committed copy that is masked -- and set allowed-models.txt to what the team can reach. The rubric, the thresholds and the routing rules are real and need no change; it is the model table underneath them that has to become the reader's own. Says that in the two places a reader meets the skill: an IMPORTANT callout under the root README's install commands, and one in vend/swe-router/README.md between the install block and the file table. No code change. Claims checked before writing them: the generator contains zero placeholder strings and still references real ids, the three --available probes above were run against the masked copy, and every link and anchor added here resolves.
…ed-router docs(swe-router): mask measured model names with indicative placeholders
…outer-as-illustrative
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #37 (base is that branch, not
main) so the two land together andmainnever carries a README promising something the skill cannot do.Why
Masking the vended model names turns the skill into a demonstration rather than something a developer can act on, and neither README said so — both still framed Step 2 as install-and-use.
Two consequences, both verified against the masked files on #37:
closed-mid-gen6, which nobody can switch to.model-aliases.jsonno longer contains any real vendor spelling, so a real id an assistant offers does not match the shipped list:So the one path that matters — "here's what I can run, what should I use?" — returns nothing.
What this adds
An
IMPORTANTcallout in the two places a reader meets the skill: under the root README's install commands, and invend/swe-router/README.mdbetween the install block and the file table.Both say the same thing: install it to see the mechanism, then point it at your own measurements before acting on it. The fix is already the documented path — run Step 1, regenerate
models.jsonwithbuild_vended_models.py, setallowed-models.txtto what your team can reach.The distinction worth keeping visible: the generator still emits real ids. Only the committed copy is masked. And the rubric, thresholds and routing rules are all real and need no change — it's the model table underneath them that has to become the reader's own.
Claims checked before writing them
build_vended_models.pycontains zero placeholder strings and still references real ids — so "the generator emits real ids" is accurate, not assumed.--availableprobes above were run against the masked copy extracted from docs(swe-router): mask measured model names with indicative placeholders #37.#step-1--benchmark-and-get-a-frontier.No code change. 605 tests pass (unchanged from #37),
ruffclean.One thing #37 does not achieve, for the record
Not addressed here because it's a policy call, not a doc fix: the mask is reversible. Every score and cost is preserved by design, and upstream's
models.jsonis public with the same numbers and the real names — so a join on(score, cost_per_task_usd)recovers all 23 of 23. The masking keeps vendor names out of an AWS-published artifact; it does not conceal which model scored what. Making it non-recoverable would mean banding or perturbing the figures, which would change the routing thresholds and so needs separate thought.