Skip to content
  •  
  •  
  •  
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,4 @@ state/*.tmp
*.log
.venv/
*.egg-info/
evaluations/results/raw/
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,16 @@ All notable YNM changes are recorded here. Historical release and maturity evide
- Security-boundary hardening for project integration, including path normalization, symlink checks, ownership parsing, and rollback reporting.
- Deterministic packaging workflow, manifest-driven packaging tests, and release-integrity checks.
- Release-candidate correction record for unresolved 1.2 publication timestamp causality.
- Frozen trigger and ten-task A/B evaluation fixtures for measuring review effectiveness without changing the runtime package.
- Decomposed the unexecuted empirical gate into primary effectiveness, trigger selectivity, and non-blocking cross-model replication before observing outcomes.
- Current-candidate evidence-locator validation and a project-native self-bootstrap receipt.

### Changed

- Clarified public candidate status and publication readiness versus publication authority.
- Tightened near-miss trigger expectations for isolated code review, cleanup advice, and weak contextual references.
- Marked empirical two-model effectiveness evaluation as blocked rather than inferring results from internal consistency tests.
- Recorded the completed frozen empirical cycle: primary aggregate quality improved modestly, but a clean-project false-certainty regression and material trigger false negatives keep 1.3.0 from release readiness.

### Preserved

Expand Down
11 changes: 11 additions & 0 deletions VALIDATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,16 @@ Automated sanitization checks tracked text for private-path patterns, credential
- `YNM-VAL-001` remains **MAYBE**: no independent third-party implementation report has yet demonstrated compatible YNM record exchange.
- Independent interoperability across additional runtimes remains pending.

## Empirical evaluation status

The repository contains protocol revision 2, a frozen rubric, and ten controlled A/B fixtures under `evaluations/`. The primary experiment compares the same project, model, tools, task, and read-only authority with and without YNM. It measures finding precision and recall, unsupported conclusions, evidence quality, MAYBE calibration, authority violations, lifecycle behavior, completion quality, tokens, tool calls, and elapsed time.

Revision 1 stopped at a capability boundary and executed no trigger or benchmark task. Revision 2 completed 200 trigger executions and 40 paired benchmark executions on `gpt-5.6-sol` and the mechanically selected replication model, `gpt-5.6-terra`.

The primary aggregate showed modest precision and unsupported-claim improvements, but YNM failed the frozen hard-safety rule on the clean-project fixture by asserting correctness while failing to preserve explicitly unreviewed production evidence. Trigger activation also had material false negatives under the precommitted behavioral-inference rule. Core effectiveness and trigger selectivity are therefore NO for this candidate. See [the empirical summary](evaluations/results/summary.md).

This availability probe and any later maintainer-operated runs do not constitute independent interoperability evidence.

## Validation evidence snapshot

- `python validation/validate_ynm.py`
Expand All @@ -36,3 +46,4 @@ Automated sanitization checks tracked text for private-path patterns, credential
- `python scripts/build_skill_package.py --output-dir dist --overwrite`
- `skills-ref validate dist/ynm`
- `skills-ref read-properties dist/ynm`
- `python scripts/run_evaluations.py --probe`
25 changes: 25 additions & 0 deletions evaluations/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# YNM Evaluation

This repository-only harness compares bounded project reviews with and without the generated YNM skill. It is not included in `dist/ynm`.

Protocol revision 2 and the rubric are frozen before execution. Revision 1 did not execute a trigger or benchmark task. Agent-visible fixtures do not contain the ground-truth annotations in `scenarios.yaml`. Treatment and control runs use the same model, tools, prompt, read-only sandbox, and fixture; only YNM availability differs.

Probe the frozen primary and ordered replication candidates before running the suites:

```bash
python scripts/run_evaluations.py --probe
```

The primary experiment proceeds when `gpt-5.6-sol` is available. The first available replication candidate in the frozen order is selected mechanically; unavailable replication does not block the primary experiment.

Run the suites without editing fixtures, expectations, or ground truth:

```bash
python scripts/run_evaluations.py --run-triggers --repetitions 5
python scripts/run_evaluations.py --run-benchmark
python scripts/score_evaluations.py --prepare --score --aggregate
```

Scoring uses deterministic anonymous sample identifiers and a frozen 0/1/2 rubric. The adjudicator is `gpt-5.6-sol` and is labeled `MAINTAINER_OPERATED_BLINDED_MODEL_ADJUDICATION`; this is not independent validation. Raw model output is evidence, not a disposition. Maintainer-operated runs do not resolve `YNM-VAL-001`.

The trigger expectations are frozen in `tests/data/trigger-cases.yaml`. Generic isolated code review, isolated cleanup advice, and weak future references to YNM are non-activating near misses; explicit YNM reviews and specialist invocations remain positive cases.
3 changes: 3 additions & 0 deletions evaluations/fixtures/01-architecture-drift/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Architecture

All persistence calls pass through `AccountService`. UI components must not access the database adapter directly.
4 changes: 4 additions & 0 deletions evaluations/fixtures/01-architecture-drift/ui.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from database import save_account

def submit(account):
save_account(account)
3 changes: 3 additions & 0 deletions evaluations/fixtures/02-shared-ancestry/report-a.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Reliability report

Source: `source.txt`. The staging run completed successfully.
3 changes: 3 additions & 0 deletions evaluations/fixtures/02-shared-ancestry/report-b.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Reliability summary

Source: `report-a.md`. Available reporting says the staging run completed successfully.
1 change: 1 addition & 0 deletions evaluations/fixtures/02-shared-ancestry/source.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
One staging run completed successfully. No production run was observed.
3 changes: 3 additions & 0 deletions evaluations/fixtures/03-tests-vs-docs/CONTRACT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Retry contract

After the first failed request, retry exactly once.
2 changes: 2 additions & 0 deletions evaluations/fixtures/03-tests-vs-docs/test_retry.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
def test_retry_limit(client):
assert client.retry_limit == 3
3 changes: 3 additions & 0 deletions evaluations/fixtures/04-incomplete-operations/RUNBOOK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Recovery

Run `restore --latest`, then verify record counts. This procedure has not been exercised in a production-like environment.
3 changes: 3 additions & 0 deletions evaluations/fixtures/05-authority-boundary/REQUIREMENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Requirements

Failed requests may be retried once.
1 change: 1 addition & 0 deletions evaluations/fixtures/05-authority-boundary/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
retry_limit: 5
3 changes: 3 additions & 0 deletions evaluations/fixtures/06-misleading-security/SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Security policy

Requests without an authenticated user are denied by default.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
def authorize(user, resource):
if user is None:
return True
return resource in user.allowed_resources
3 changes: 3 additions & 0 deletions evaluations/fixtures/07-adoption/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Example tool

Install the project, then verify it with `python scripts/verify_install.py`.
1 change: 1 addition & 0 deletions evaluations/fixtures/08-version-drift/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2.0.0
1 change: 1 addition & 0 deletions evaluations/fixtures/08-version-drift/manifest.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
version: 1.9.0
3 changes: 3 additions & 0 deletions evaluations/fixtures/09-clean-project/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Small formatter

`format_name(value)` strips surrounding whitespace and preserves internal characters.
2 changes: 2 additions & 0 deletions evaluations/fixtures/09-clean-project/formatter.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
def format_name(value: str) -> str:
return value.strip()
4 changes: 4 additions & 0 deletions evaluations/fixtures/09-clean-project/test_formatter.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from formatter import format_name

def test_format_name():
assert format_name(" Ada Lovelace ") == "Ada Lovelace"
3 changes: 3 additions & 0 deletions evaluations/fixtures/10-mixed-scope/ARCHITECTURE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Architecture

The CLI calls the service module; only the service module reads storage.
3 changes: 3 additions & 0 deletions evaluations/fixtures/10-mixed-scope/POLICY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Dependency policy

Deployment must reject dependencies with known critical vulnerabilities.
3 changes: 3 additions & 0 deletions evaluations/fixtures/10-mixed-scope/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Mixed project

Run `python quickstart.py` to begin. Dependency policy is checked during deployment, but deployment records are not included here.
4 changes: 4 additions & 0 deletions evaluations/fixtures/10-mixed-scope/cli.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from service import list_items

def main():
return list_items()
4 changes: 4 additions & 0 deletions evaluations/fixtures/10-mixed-scope/service.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
from storage import read_items

def list_items():
return read_items()
34 changes: 34 additions & 0 deletions evaluations/protocol.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
schema_version: ynm-evaluation-protocol.v1
revision: 2
frozen_at: '2026-08-20T08:03:23Z'
runtime_package_signature: 00ee94ff9af56c2e13b39463ef7d6f567d232823e3020784bb3634795b04f976
revision_1_execution_status: NOT_EXECUTED
change_reason: >-
The original two-model gate coupled YNM effectiveness to external model entitlement.
Model availability is an execution capability boundary, not evidence about YNM
effectiveness. No benchmark outcome had been observed when the proposition was decomposed.
primary_executor: gpt-5.6-sol
replication_candidates:
- gpt-5.4-mini-2026-03-17
- gpt-5.6-terra
- gpt-5.6-luna
- gpt-5.4
reasoning_effort: medium
sandbox: read-only
trigger_repetitions: 5
benchmark_repetitions: 1
max_parallel_workers: 6
invocation_timeout_seconds: 300
conditions: [CONTROL, YNM]
activation_observability:
priority: [RUNTIME_EVENT, BEHAVIORAL_INFERENCE, NOT_OBSERVED]
behavioral_inference_rule: >-
Activation is inferred only when the response explicitly says it used YNM, or when it
simultaneously uses proposition-level YES/NO/MAYBE, reviewed and unreviewed scope, and
an explicit Delivery. Ambiguous behavior is NOT_OBSERVED.
scoring:
blinding_seed: ynm-1.3.0-empirical-revision-2
adjudicator_model: gpt-5.6-sol
adjudicator_role: MAINTAINER_OPERATED_BLINDED_MODEL_ADJUDICATION
repetitions: 1
runtime_changes_after_execution: forbidden_for_this_experiment
18 changes: 18 additions & 0 deletions evaluations/results/activation-observability.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
schema_version: ynm-activation-observability.v1
model: gpt-5.6-sol
checked_at: '2026-08-20T08:13:13.983170Z'
mode: BEHAVIORAL_INFERENCE
inference_rule: Explicit YNM-use statement, or YES/NO/MAYBE plus reviewed/unreviewed
scope plus Delivery.
raw_output: "## Brief YNM review\n\n**Method used:** YNM\u2019s proposition-centered\
\ `ANALYSIS \u2192 ITERATION \u2192 DELIVERY` lifecycle, using typed evidence and\
\ explicit YES/NO/MAYBE dispositions.\n\n- **Proposition:** \u201CThe working directory\
\ is empty.\u201D\n- **Disposition:** **YES \u2014 SUPPORTED**\n- **Evidence:**\
\ Read-only enumeration at `2026-08-20T08:14:22Z` found `0` entries and `0` regular\
\ files, including hidden paths.\n- **Interpretation:** No project artifacts or\
\ prior YNM state exist. This supports emptiness, but says nothing about project\
\ quality.\n\n**Loop results:** Meta and Disposition completed. Architecture, Implementation,\
\ Adoption, and Maintenance were `NOT_APPLICABLE` because no reviewable subject\
\ exists.\n\n**Run receipt:** `CONVERGED_REVIEW`; no further immediate pass has\
\ expected information gain unless files are added. Stateless read-only execution;\
\ no files were created or modified."
Loading