diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index af98a15..2fb0fe0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1148,3 +1148,32 @@ jobs: KIT=$(python -c "import json;print(json.load(open('generator.lock'))['kit_version'])") echo "pyproject=$PKG __version__=$INIT generator.lock=$KIT" test "$PKG" = "$INIT" && test "$PKG" = "$KIT" || { echo "::error::version sources disagree"; exit 1; } + + - name: Project generator pin is found, validated and bounded (regression, #169) + run: | + set -uo pipefail + CLI=/tmp/ci/bin/specfuse-authoring + fail() { echo "::error::$1" >&2; exit 1; } + B=$(mktemp -d) || fail "mktemp failed" + P=$B/proj && mkdir -p "$P/.specfuse" "$P/sub" && git -C "$P" init -q || fail "scratch repo setup failed" + + # An incomplete project pin is refused, from a subdirectory too. + echo '{"generator":{"version":"0.19.0"}}' > "$P/.specfuse/generator.lock" + out=$(cd "$P/sub" && "$CLI" generate --version 2>&1) && fail "incomplete project pin was accepted" + grep -q "must set generator.version and generator.sha256" <<<"$out" || fail "unexpected message: $out" + + # Malformed JSON is named, not a traceback. + echo '{' > "$P/.specfuse/generator.lock" + out=$(cd "$P/sub" && "$CLI" generate --version 2>&1) && fail "malformed project pin was accepted" + grep -q "is not valid JSON" <<<"$out" || fail "unexpected message: $out" + + # A pin above the repo root never applies: the search stops at .git. + rm "$P/.specfuse/generator.lock" + mkdir -p "$B/.specfuse" && echo '{' > "$B/.specfuse/generator.lock" + out=$(cd "$P/sub" && "$CLI" generate --version 2>&1) || true # falls back to the kit pin, whose download may fail; only the message matters + grep -q "is not valid JSON" <<<"$out" && fail "a pin outside the repo was read" + + # A missing local jar fails before any download. + out=$(cd "$P" && SPECFUSE_GENERATOR_JAR=/nope.jar "$CLI" generate --version 2>&1) && fail "missing SPECFUSE_GENERATOR_JAR was accepted" + grep -q "SPECFUSE_GENERATOR_JAR=/nope.jar is not a file" <<<"$out" || fail "unexpected message: $out" + echo "==> project pin lookup and validation behave." diff --git a/docs/generator-access.md b/docs/generator-access.md index 0f1d3c2..fb4010c 100644 --- a/docs/generator-access.md +++ b/docs/generator-access.md @@ -93,7 +93,8 @@ In CI, store it as a secret and export it for the generate step only. ### What happens on first run -1. The CLI reads the generator pinned for your kit version +1. The CLI reads the generator pin: the project's own `.specfuse/generator.lock` + if it has one (see below), otherwise the one pinned for your kit version ([`generator.lock`](../generator.lock)). 2. Downloads `specfuse-generator-.jar` from the `Specfuse/generator-dist` release tagged `v` (via `gh` or `SPECFUSE_TOKEN`). @@ -101,6 +102,37 @@ In CI, store it as a secret and export it for the generate step only. 4. Caches it under `~/.specfuse/jars/` (override the root with `$SPECFUSE_HOME`) and runs `java -jar`. Later runs reuse the cache. +### Pinning a different generator for one project + +By default every project uses the generator its kit version was tested with. To +move one project to a different release (adopting a fix early, or holding back +while you absorb a breaking change) without changing the kit, commit a project +pin at `.specfuse/generator.lock`: + +```json +{ "generator": { "version": "0.19.0", "sha256": "" } } +``` + +- `version` and `sha256` are required. `asset`, `release_tag` and `release_repo` + default to the dist repo's conventions (`specfuse-generator-.jar`, + `v`, `Specfuse/generator-dist`); `min_java` defaults to the kit's. + Take the SHA-256 from the generator release notes, or from + `shasum -a 256 ` on a jar you have verified. +- The CLI searches for the file upward from the working directory and stops at + the repo root, so it applies to every run inside the project. +- The jar is downloaded, verified and cached exactly like the kit's pin. A wrong + checksum fails closed. +- Each run says which pin it used. When the project pin differs from the kit's, + it also warns: a generator outside the kit's tested pair (see + [`compatibility.md`](../compatibility.md)) can accept or emit vocabulary the + handbooks and Spectral rules don't describe. Delete the file to return to the + kit's pin. +- `specfuse authoring upgrade` never writes or removes it. It is project content. + +**Local jar, for generator development only.** `SPECFUSE_GENERATOR_JAR=/path/to.jar` +runs that jar as-is: no pin and no checksum. Every run warns that it is +unpinned. Don't set it in CI or commit it to a script; use a project pin instead. + ### Troubleshooting | Symptom | Cause | Fix | @@ -109,6 +141,8 @@ In CI, store it as a secret and export it for the generate step only. | `404` / `release not found` on download | no Read access, or wrong account | confirm the maintainer granted access to the authenticated user | | `no generator is pinned for this kit version yet` | kit release predates a published generator | `pipx upgrade specfuse` once a generator-bearing release ships | | `checksum mismatch on downloaded generator jar` | corrupted download or a tampered asset | delete `~/.specfuse/jars/` and retry; if it persists, report it — do not use the jar | +| `.specfuse/generator.lock must set generator.version and generator.sha256` | incomplete project pin | add both fields, or delete the file to use the kit's pin | +| `.specfuse/generator.lock pins generator X; this kit (...) is tested with Y` | warning, not an error: the project pins its own generator | intended? carry on. Otherwise delete the project pin | | `'java' not found` / wrong version | no JRE 21+ (every generator jar since `0.1.0` is class-file 65) | install a JRE that meets `min_java` in `generator.lock` | --- @@ -118,7 +152,10 @@ In CI, store it as a secret and export it for the generate step only. - **Least privilege:** grant only `Read` on only `Specfuse/generator-dist`. The generator needs nothing else. - **Checksum pinning:** the kit verifies every jar against the SHA-256 committed - in its public `generator.lock`. A swapped or corrupted asset fails closed. + in its public `generator.lock`, or in the project's own + `.specfuse/generator.lock`. A swapped or corrupted asset fails closed. + `SPECFUSE_GENERATOR_JAR` is the one path that skips verification. Keep it + out of CI. - **Token hygiene:** prefer `gh` for humans; reserve `SPECFUSE_TOKEN` for CI, store it as a secret, scope it fine-grained, and rotate it on a schedule. - **Revocation is clean:** removing access stops generation without affecting the diff --git a/docs/getting-started.md b/docs/getting-started.md index f7ad5b0..28ea3bd 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -112,7 +112,7 @@ specfuse authoring generate generate --group "" -project.js Everything after `specfuse authoring generate` goes to the generator unchanged, so name the generator's own command (`generate`, `validate`, `baseline`, …) first. Its `--group` takes one group per run. -On first run the CLI resolves the generator pinned for your kit version (see [`generator.lock`](../generator.lock)), downloads it, **verifies its SHA-256**, caches it under `~/.specfuse/jars/`, and runs it. Subsequent runs use the cache. +On first run the CLI resolves the generator pinned for your kit version (see [`generator.lock`](../generator.lock)), or the project's own `.specfuse/generator.lock` when it has one ([generator access](generator-access.md)), downloads it, **verifies its SHA-256**, caches it under `~/.specfuse/jars/`, and runs it. Subsequent runs use the cache. Generated code lands in the component repos named by each group's `destination`. See [Setting up component repos](component-repos.md) for the layout, commit policy, and regen flow. diff --git a/plugins/specfuse-authoring/skills/generate/SKILL.md b/plugins/specfuse-authoring/skills/generate/SKILL.md index cfdda36..713da45 100644 --- a/plugins/specfuse-authoring/skills/generate/SKILL.md +++ b/plugins/specfuse-authoring/skills/generate/SKILL.md @@ -14,7 +14,7 @@ Generate code and documentation artifacts from the API specifications, one artif ## Prerequisites — verify before running -1. `specfuse` is on PATH (`pipx install specfuse`, or `uv tool install specfuse`). The generator itself is not a file in the project — the CLI resolves and checksum-verifies the version pinned in `generator.lock` on demand. On a standalone kit install the deprecated flat `specfuse-authoring` command still works; drop the `specfuse ` prefix from the commands below. +1. `specfuse` is on PATH (`pipx install specfuse`, or `uv tool install specfuse`). The generator itself is not a file in the project — the CLI resolves and checksum-verifies the version pinned in `generator.lock` (or the project's `.specfuse/generator.lock`, which takes precedence) on demand. On a standalone kit install the deprecated flat `specfuse-authoring` command still works; drop the `specfuse ` prefix from the commands below. 2. The project config file (typically `-project.json`) exists in the project root 3. Specifications are valid — if not recently validated, run `/validate` first diff --git a/specfuse/authoring/component.py b/specfuse/authoring/component.py index d49f83e..945ff2f 100644 --- a/specfuse/authoring/component.py +++ b/specfuse/authoring/component.py @@ -32,6 +32,7 @@ from pathlib import Path from . import kit_root +from .generator import find_project_lock LANGUAGES = ("csharp", "python") @@ -138,9 +139,10 @@ def _find_project_file(spec_dir: Path, explicit: Path | None) -> Path: def _generator_version(spec_dir: Path) -> tuple[str, str | None]: """(kit-pinned version, project-pinned version or None).""" kit = json.loads((kit_root() / "generator.lock").read_text())["generator"]["version"] - project_lock = spec_dir / ".specfuse" / "generator.lock" + # Same lookup `specfuse authoring generate` uses, so the warning names the pin it will run. + project_lock = find_project_lock(spec_dir) project = None - if project_lock.is_file(): + if project_lock is not None: try: project = json.loads(project_lock.read_text()).get("generator", {}).get("version") except json.JSONDecodeError: diff --git a/specfuse/authoring/generator.py b/specfuse/authoring/generator.py index df1f754..f980c4e 100644 --- a/specfuse/authoring/generator.py +++ b/specfuse/authoring/generator.py @@ -4,12 +4,18 @@ Release asset. The public kit never contains the jar; it contains only the pin (generator.lock) and the expected SHA-256. On `generate`, the launcher: - 1. reads the pin for the installed kit version, + 1. reads the pin: the project's `.specfuse/generator.lock` when one exists + (searched upward from the working directory, stopping at the repo root), + otherwise the one shipped with the installed kit version, 2. uses the cached jar if its checksum matches, 3. otherwise downloads the pinned release asset (gh CLI, or HTTPS + token), 4. verifies SHA-256 against the pin (aborts on mismatch), 5. runs `java -jar `. +`SPECFUSE_GENERATOR_JAR=/path/to.jar` skips all of the above and runs that jar +as-is: no pin, no checksum. It exists for generator development and is +reported as unpinned on every run. + Access is gated entirely by the client's ability to pull the private release (a GitHub token). Revoke the token -> revoke the generator; the kit is unaffected. @@ -30,21 +36,81 @@ CACHE_DIR = Path(os.environ.get("SPECFUSE_HOME", Path.home() / ".specfuse")) / "jars" TOKEN_ENV = "SPECFUSE_TOKEN" # GitHub token with read access to the dist repo +JAR_ENV = "SPECFUSE_GENERATOR_JAR" # unpinned local jar; same name the project scripts honour +PROJECT_LOCK = Path(".specfuse") / "generator.lock" -def _load_pin() -> dict: +def _load_kit_pin() -> dict: lock = kit_root() / "generator.lock" if not lock.is_file(): sys.exit(f"Error: generator.lock not found at {lock}") - pin = json.loads(lock.read_text()) - gen = pin.get("generator", {}) + return json.loads(lock.read_text()) + + +def find_project_lock(start: Path | None = None) -> Path | None: + """The nearest `.specfuse/generator.lock` at or above `start`. + + The search stops at the first directory holding `.git`, so a project never + picks up a pin from an unrelated checkout that happens to enclose it. + """ + here = (start or Path.cwd()).resolve() + for d in (here, *here.parents): + if (d / PROJECT_LOCK).is_file(): + return d / PROJECT_LOCK + if (d / ".git").exists(): + return None + return None + + +def _project_pin(lock: Path, kit_pin: dict) -> dict: + """A project pin, with unset fields defaulted from the generator's release conventions. + + Only `version` and `sha256` are required. `asset`, `release_tag` and + `release_repo` follow the dist repo's naming, so a project bumping its pin + writes two values, not five. + """ + try: + pin = json.loads(lock.read_text()) + except json.JSONDecodeError as e: + sys.exit(f"Error: {lock} is not valid JSON: {e}") + gen = pin.get("generator") + if not isinstance(gen, dict) or not gen.get("version") or not gen.get("sha256"): + sys.exit( + f"Error: {lock} must set generator.version and generator.sha256, e.g.\n" + ' {"generator": {"version": "0.19.0", "sha256": "<64 hex chars>"}}' + ) + version = gen["version"] + kit_gen = kit_pin.get("generator", {}) + gen.setdefault("asset", f"specfuse-generator-{version}.jar") + gen.setdefault("release_tag", f"v{version}") + gen.setdefault("release_repo", kit_gen.get("release_repo", "Specfuse/generator-dist")) + pin.setdefault("min_java", kit_pin.get("min_java", 21)) + return pin + + +def _load_pin() -> tuple[dict, str]: + """The effective pin and a short label naming where it came from.""" + kit_pin = _load_kit_pin() + lock = find_project_lock() + if lock is not None: + pin = _project_pin(lock, kit_pin) + kit_version = kit_pin.get("generator", {}).get("version") + if pin["generator"]["version"] != kit_version: + print( + f"Warning: {lock} pins generator {pin['generator']['version']}; this kit " + f"({kit_pin.get('kit_version', '?')}) is tested with {kit_version}. " + "See compatibility.md.", + file=sys.stderr, + ) + return pin, f"project pin {lock}" + gen = kit_pin.get("generator", {}) if gen.get("sha256") in (None, "", "PENDING"): sys.exit( "Error: no generator is pinned for this kit version yet.\n" " This kit release predates a published generator jar.\n" " Upgrade the kit once a generator-bearing release is available." ) - return pin + return kit_pin, "kit pin" def _sha256(path: Path) -> str: @@ -118,14 +184,25 @@ def _download(pin: dict, dest: Path) -> None: shutil.copyfileobj(resp, out) -def resolve_jar() -> tuple[Path, dict]: - """Return a path to a verified, cached generator jar (downloading if needed).""" - pin = _load_pin() +def resolve_jar() -> tuple[Path, dict, str]: + """Return a verified, cached generator jar (downloading if needed), its pin, and the pin's source.""" + override = os.environ.get(JAR_ENV) + if override: + jar = Path(override).expanduser() + if not jar.is_file(): + sys.exit(f"Error: {JAR_ENV}={override} is not a file.") + print( + f"Warning: {JAR_ENV} is set; running {jar} UNPINNED (no checksum verified).", + file=sys.stderr, + ) + return jar, {"min_java": _load_kit_pin().get("min_java", 21)}, f"{JAR_ENV}, unpinned" + + pin, source = _load_pin() gen = pin["generator"] jar = CACHE_DIR / gen["asset"] if jar.is_file() and _sha256(jar) == gen["sha256"]: - return jar, pin + return jar, pin, source _download(pin, jar) actual = _sha256(jar) @@ -133,13 +210,14 @@ def resolve_jar() -> tuple[Path, dict]: jar.unlink(missing_ok=True) sys.exit( "Error: checksum mismatch on downloaded generator jar.\n" - f" expected {gen['sha256']}\n got {actual}" + f" expected {gen['sha256']}\n got {actual}\n" + f" (pin: {source})" ) - return jar, pin + return jar, pin, source def generate(passthrough_args: list[str]) -> int: - jar, pin = resolve_jar() + jar, pin, source = resolve_jar() _check_java(int(pin.get("min_java", 21))) - print(f"Running generator: {jar.name}") + print(f"Running generator: {jar.name} ({source})", flush=True) return subprocess.run(["java", "-jar", str(jar), *passthrough_args]).returncode diff --git a/templates/project-init/scripts/specfuse/README.md b/templates/project-init/scripts/specfuse/README.md index 89c5f33..a8584e8 100644 --- a/templates/project-init/scripts/specfuse/README.md +++ b/templates/project-init/scripts/specfuse/README.md @@ -107,7 +107,9 @@ yet exits 0. `generate-scenario-docs.sh` drives the generator through `specfuse authoring generate`, which resolves and checksum-verifies the version -pinned in the kit's `generator.lock`. There is no jar to keep in this directory. +pinned in the kit's `generator.lock`, or in the project's own +`.specfuse/generator.lock` when it has one. There is no jar to keep in this +directory. (The script falls back to the deprecated flat `specfuse-authoring` command when the suite CLI is not installed, and `SPECFUSE_AUTHORING` overrides both.) diff --git a/templates/project-init/scripts/specfuse/check-extension-vocabulary.py b/templates/project-init/scripts/specfuse/check-extension-vocabulary.py index 257d0fe..b37d4c4 100644 --- a/templates/project-init/scripts/specfuse/check-extension-vocabulary.py +++ b/templates/project-init/scripts/specfuse/check-extension-vocabulary.py @@ -133,14 +133,28 @@ def find_jar() -> Path | None: `specfuse authoring generate` verifies each jar's SHA-256 against generator.lock before caching it under $SPECFUSE_HOME/jars, so a cached jar - is a jar this project's pinned kit vouched for. SPECFUSE_GENERATOR_JAR - overrides for CI, where the cache may be primed from a different path. + is a jar this project's pinned kit vouched for. When the project pins its + own generator in `.specfuse/generator.lock`, that version's jar is the one + examined, never a newer one cached for another project. + SPECFUSE_GENERATOR_JAR overrides for CI, where the cache may be primed from + a different path. """ override = os.environ.get("SPECFUSE_GENERATOR_JAR") if override: p = Path(override).expanduser() return p if p.is_file() else None home = Path(os.environ.get("SPECFUSE_HOME", Path.home() / ".specfuse")) + project_lock = ROOT / ".specfuse" / "generator.lock" + if project_lock.is_file(): + try: + gen = json.loads(project_lock.read_text()).get("generator", {}) + except json.JSONDecodeError: + return None + version = gen.get("version") + if not version: + return None + p = home / "jars" / gen.get("asset", f"specfuse-generator-{version}.jar") + return p if p.is_file() else None jars = sorted((home / "jars").glob("specfuse-generator-*.jar")) if not jars: return None