Repository navigation
Conversation
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
ruff EXE001: the file has a shebang but shipped as mode 100644. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q7W9hWwuZU8Mkwo3ZCrR7m Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
- Layout lists the real paths (src/assetops_harbor, template/, overlays/) and marks datasets/ as generated. - Run steps work from the repo root: correct Dockerfile path, adapter defaults instead of a --template pointing at a generated task, no PYTHONPATH=agent, no 'push'. - Drop the pin to main@81265cb; the branch targets aafeedback_changes. - Replace 'Still unverified' with the Docker-host results reported in #558. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
generate_tasks.py gains three options for corpora that do not ship in the repo, such as AssetOpsBenchScenarioGeneration/scenarios_data: - --runtime-image: the image each task builds FROM. - --data-dir: where the corpus lives inside that image. The per-task layer copies the scenario there and the healthcheck runs init_data.py with SCENARIOS_DATA_DIR pointed at it; the agent keeps the repo copy, as scenario_suite_runner does. - --skip-missing: warn and skip profile scenarios absent from the corpus (all.yaml lists wosr-62, which the corpus lacks). corpus-image/Dockerfile bakes the corpus (2.1 GB, mostly shared/iot) into one layer over the runtime image, so tasks share it instead of each carrying it in their build context. Also stop copying scenario 1's description and the wosr keyword into every generated task. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Runs a scenario corpus through Harbor with the same profile, the same
stirrup-agent and the same Docker code sandbox as benchmarks/run.sh, but
with scenarios in parallel, each trial on its own CouchDB.
- Builds assetopsbench/runtime:corpus from -s, and the code sandbox tar
for the per-trial Docker-in-Docker daemon.
- Regenerates the dataset from scratch (the generator never removes stale
task folders), skipping profile entries the corpus lacks.
- One Harbor job per model under <leaderboard>/harbor-jobs; re-running
resumes it, the equivalent of --skip-existing.
- Credentials reach the Harbor process through uv run --env-file only.
Uses [[ -z "${arr[*]+set}" ]] and ${arr[@]+...} so empty arrays work
under set -u in macOS's bash 3.2.
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
corpus-image/ -> suite-image/, assetopsbench/runtime:corpus -> assetopsbench/runtime:suite, /opt/corpus/scenarios_data -> /opt/suite/scenarios_data, and the generated dataset assetopsbench-corpus -> assetopsbench-suite, with matching wording in run.sh, the generator's help and the README. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Completes 533bc5a, which only moved corpus-image/ to suite-image/: assetopsbench/runtime:corpus -> assetopsbench/runtime:suite, /opt/corpus/scenarios_data -> /opt/suite/scenarios_data, the dataset assetopsbench-corpus -> assetopsbench-suite, and the wording in run.sh, the generator's help and the README. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
A dropped VPN made the LiteLLM proxy unreachable mid-session. Every trial still built its containers, loaded its data, retried the model for four minutes and exited 1, and the job recorded each as done, so a resume would not have re-run them. - Probe LITELLM_BASE_URL / TOKENROUTER_BASE_URL (per model prefix) before starting or resuming a job, and skip the model with a clear message when the router does not answer. - Resume with --filter-error-type NonZeroAgentExitCodeError, so trials whose agent crashed run again while scored trials are kept. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Add the generate_tasks.py flags a non-open profile needs, and record that those tasks load empty CouchDB collections: manifests reference shared/ files that exist only in the full corpus, not the runtime image, and the loader only warns when a data file is missing. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
StirrupAgent now loads the nearest .env above the working directory before checking router credentials, so tokenrouter/ and litellm_proxy/ models run without exporting their variables. override=False keeps exported variables and --ae ahead of the file, and only CREDENTIAL_ENV_VARS reach the container. Tests chdir into tmp_path so a developer's .env cannot leak in, and the credential fixture now restores variables that load_dotenv writes. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
The earlier note said the three FMEA mini scenarios load cleanly. They do not: their manifest lists files as an array, which the check that produced the note skipped. All 35 mini scenarios reference corpus-only files. Also record that the referenced files are 1.5 MB (mini) to 58 MB (all), not the 2.1 GB of the corpus shared/ directory, and how the failure shows up. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
feat(harbor): load .env for StirrupAgent credentials
Resolve the README conflict by keeping both new sections. The base's hand-generation section now uses this branch's --runtime-image, --data-dir and --skip-missing flags, and its known-gap note becomes what happens without the suite image. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Add overlays/private-data.yaml, which bind-mounts the directory in AOB_PRIVATE_DIR read-only into main at /opt/suite/scenarios_data, the path the suite image uses. Tasks generated with --data-dir then load their manifests' shared/ files from the mount, with no suite image rebuild when the data changes. The variable is required with :? so an unset value fails naming it rather than as an invalid volume spec. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
DhavalRepo18
force-pushed
the
aafeedback_changes
branch
from
October 5, 2026 13:39
b4d8e41 to
7c4b27d
Compare
Standarize the System Promot and Coed Exec tool for AA Study
This PR is based on v1 run
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Updated car list with additional values.
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Add tsfm and car
Bring TSFM and IoT server updates
run.sh gains -k (repeat each task n times, the basis for pass@k and pass^k), -t (sampling temperature, default 0.6) and -u (agent turn budget, default 100), with validation on each. passk.py computes pass@k and pass^k from a leaderboard directory using the existing "passed" reward key. Taken from #618 without the to_reward.py change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PcbP743uYAdkJiV9UbiTjy Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
Add pass@k/pass^k, temperature and turn-budget controls to harbor runs
Signed-off-by: Dhaval Patel <pateldha@us.ibm.com>
A CAR answer now passes when it is a single-key object whose key matches the gold mode (response / clarification / abstain). The value under the key no longer affects pass or score; required-term coverage stays in the mode_* fields as a diagnostic. Replaces the CAR tests that targeted the removed groundtruth_eval.json term scoring and updates docs/static-json-evaluation.md to match. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
ShuxinLin
force-pushed
the
aafeedback_changes
branch
from
October 8, 2026 13:21
25ed13f to
8c59cd9
Compare
fix(evaluation): score CAR scenarios on the mode key only
Reverts the scorer change from f8805be. A CAR answer again passes only when it is a single-key object whose key matches the gold mode and whose value contains every required term extracted from the gold value; the score is the F1 over the mode key plus those terms. This is the scorer every aa_v1 job before 9584e28 was graded with. Re-scoring their saved CAR answers with it reproduces the recorded f1 exactly, so restoring it puts the 9584e28+ runs back on the same scale. The pre-f8805be CAR tests and docs describe the groundtruth_eval.json metadata scoring added in 79d6356 (car_score = 0.6 mode + 0.4 coverage, optional terms, pass on any one required term). 8f1e538 took that out of the scorer two days later but left the tests and docs, so six tests fail against this code. They are replaced with tests of the behaviour the code has, and the docs are corrected to match. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
The suite's CAR groundtruth_eval.json files were removed on 2026-10-07, and static_json never read them. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
fix(evaluation): restore term-based CAR scoring
When a CAR scenario ships groundtruth_eval.json with required_terms, the static_json scorer now uses it. The answer earns 0 unless it is a single-key object whose key is the groundtruth.txt mode. Otherwise its credit is the fraction of required concepts covered, a concept being covered when any one of its any_of forms appears, on word boundaries, in the value under the key. The scenario passes at credit >= 0.5 (CAR_PASS_THRESHOLD). Precision, recall, f1 and the new car_score field all report the credit: with one matching key the model side holds exactly the gold entries, so an F1 reduces to coverage. CAR scenarios without required_terms keep the extracted-term F1 and the strict pass rule. Term matching also read structured values through json.dumps, which kept upper-case letters out of the a-z0-9 filter and turned a newline into "\n", gluing an "n" onto the next word. It now joins the structure's keys and scalars and lowercases the result. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
A groundtruth_eval.json form now also matches its regular plural: an added s, es after s/x/z/ch/sh, or ies for a consonant + y. Before, "Which assets?" missed the form "asset". Plurals apply to the last word, so "failure type" matches "failure types". Terms extracted from the gold value, used when a scenario has no groundtruth_eval.json, still match exactly. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
A CAR answer scored on groundtruth_eval.json concepts now reports recall as its score and pass criterion (recall >= 0.5). Precision and f1 are None: a free-text value gives no count of what the model asserted, so precision could only repeat recall and the f1 built from them was recall under another name. car_score duplicated recall and is removed, along with its report aggregate. Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
fix(evaluation): score CAR on groundtruth_eval.json concept recall
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
placeholder for now