From e4e601849379a9e5bb17ae87894d054b195e2e5a Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Wed, 15 Jul 2026 09:11:15 +0200 Subject: [PATCH 1/3] Codegen: typed clients, restructured CLI modes, detached durable runs (#62) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: generate the typed clients from the bundles via pipelex codegen Delete the hand-written output models: pipelex codegen projects each method's concepts into piper/generated//models.py (stamped, locked by a sibling codegen.lock); the examples keep only the bundle path, pipe code, and the parse() narrower. generate-image now parses into the generated built-in Image model. New make codegen (regen, write-if-changed) + make codegen-check (offline drift check) targets — the pipelex CLI is not a dependency, point the PIPELEX make variable at an install that ships codegen; CI wiring is release-gated on a published pipelex. Adds inputs.template.json scaffolds beside each bundle, offline smoke tests for the generated clients, and docs/codegen.md. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EaRwjPbPsan4rCeyWirKpp * fix(devx): Checkpoint C review — ship codegen.lock in wheel + lockstep comments Cold-review triage: package each codegen.lock as package data so the offline drift check also works against an installed copy, and cross-reference the three places the method list is enumerated (piper/methods/*, pyproject packages, the Makefile codegen targets). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EaRwjPbPsan4rCeyWirKpp * chore(codegen): regenerate typed clients — pinned native definitions + flattened Image Regenerated with the codegen engine after the native-materialization change: Image now emits width/height integer fields instead of the interim dict-with-imprecision size, and native field descriptions come from the pinned normative definitions (storage-neutral url wording, described Text). Crate fingerprints and stamps updated accordingly; codegen-check green. Co-Authored-By: Claude Fable 5 * docs: changelog — call out the generated Image shape change (width/height) Co-Authored-By: Claude Fable 5 * chore(Makefile): update PIPELEX_RUN to streamline codegen invocations * simplify * feat(cli): update execution modes to use a single --mode option; remove --detach flag * feat: remove legacy file input and runner modules; introduce new test suite for document input handling * feat: introduce detached CLI mode for durable runs - Added `piper.detached` package with CLI commands for starting and managing durable runs. - Implemented commands: `extract-entities`, `summarize-pdf`, `generate-image`, `wait`, `status`, and `result`. - Created shared input handling functions in `piper.inputs` for text and document inputs. - Enhanced error presentation in `piper.errors` to provide mode-specific hints. - Updated tests to cover the new detached mode and ensure symmetry across CLI modes. - Refactored existing tests to align with the new structure and added new unit tests for input handling. * feat: enhance CLI demos to run with zero arguments using built-in samples - Updated all demo commands to utilize bundled sample inputs when no arguments are provided, ensuring a working result on the first command execution. - Introduced a `sample` parameter in `read_text_input()` to facilitate this feature. - Enhanced error handling to surface detailed API error messages instead of generic httpx errors, improving user feedback. - Refactored execution modes into self-contained units, removing the previous shared dispatch structure for clarity and maintainability. - Updated documentation to reflect changes in execution modes and sample usage. - Added tests to verify that demos correctly fall back to sample inputs when no user input is provided. --------- Co-authored-by: Louis Choquel <8851983+lchoquel@users.noreply.github.com> Co-authored-by: Claude Fable 5 --- .claude/skills/bootstrap/scripts/bootstrap.py | 27 ++- CHANGELOG.md | 16 ++ CLAUDE.md | 15 +- Makefile | 39 ++- README.md | 166 ++++++++----- TODOS.md | 100 ++++++++ docs/cli-architecture.md | 73 ++++++ docs/codegen.md | 45 ++++ piper/{examples => attended}/__init__.py | 0 piper/attended/cli.py | 150 ++++++++++++ piper/blocking/__init__.py | 0 piper/blocking/cli.py | 128 ++++++++++ piper/cli.py | 218 ++--------------- piper/detached/__init__.py | 0 piper/detached/cli.py | 200 +++++++++++++++ piper/errors.py | 60 ++++- piper/examples/extract_entities.py | 35 --- piper/examples/generate_image.py | 41 ---- piper/examples/summarize_pdf.py | 36 --- piper/file_input.py | 34 --- piper/generated/__init__.py | 5 + piper/generated/extract_entities/__init__.py | 0 piper/generated/extract_entities/codegen.lock | 8 + piper/generated/extract_entities/models.py | 41 ++++ piper/generated/generate_image/__init__.py | 0 piper/generated/generate_image/codegen.lock | 8 + piper/generated/generate_image/models.py | 47 ++++ piper/generated/summarize_pdf/__init__.py | 0 piper/generated/summarize_pdf/codegen.lock | 8 + piper/generated/summarize_pdf/models.py | 46 ++++ piper/inputs.py | 82 +++++++ .../extract-entities/inputs.template.json | 3 + .../generate-image/inputs.template.json | 3 + .../summarize-pdf/inputs.template.json | 3 + piper/runner.py | 105 -------- pyproject.toml | 20 +- tests/e2e/test_extract_entities.py | 20 +- tests/e2e/test_generate_image.py | 18 +- tests/e2e/test_summarize_pdf.py | 16 +- tests/integration/test_fundamentals.py | 14 +- tests/unit/test_attended_cli.py | 94 ++++++++ tests/unit/test_blocking_cli.py | 94 ++++++++ tests/unit/test_bootstrap_script.py | 24 +- tests/unit/test_cli.py | 106 -------- tests/unit/test_detached_cli.py | 140 +++++++++++ tests/unit/test_errors.py | 37 ++- tests/unit/test_extract_entities.py | 21 -- tests/unit/test_file_input.py | 30 --- tests/unit/test_generate_image.py | 33 --- tests/unit/test_generated_clients.py | 58 +++++ tests/unit/test_inputs.py | 54 +++++ tests/unit/test_mode_symmetry.py | 50 ++++ tests/unit/test_summarize_pdf.py | 25 -- wip/mode-split-clis.md | 227 ++++++++++++++++++ 54 files changed, 2025 insertions(+), 798 deletions(-) create mode 100644 TODOS.md create mode 100644 docs/cli-architecture.md create mode 100644 docs/codegen.md rename piper/{examples => attended}/__init__.py (100%) create mode 100644 piper/attended/cli.py create mode 100644 piper/blocking/__init__.py create mode 100644 piper/blocking/cli.py create mode 100644 piper/detached/__init__.py create mode 100644 piper/detached/cli.py delete mode 100644 piper/examples/extract_entities.py delete mode 100644 piper/examples/generate_image.py delete mode 100644 piper/examples/summarize_pdf.py delete mode 100644 piper/file_input.py create mode 100644 piper/generated/__init__.py create mode 100644 piper/generated/extract_entities/__init__.py create mode 100644 piper/generated/extract_entities/codegen.lock create mode 100644 piper/generated/extract_entities/models.py create mode 100644 piper/generated/generate_image/__init__.py create mode 100644 piper/generated/generate_image/codegen.lock create mode 100644 piper/generated/generate_image/models.py create mode 100644 piper/generated/summarize_pdf/__init__.py create mode 100644 piper/generated/summarize_pdf/codegen.lock create mode 100644 piper/generated/summarize_pdf/models.py create mode 100644 piper/inputs.py create mode 100644 piper/methods/extract-entities/inputs.template.json create mode 100644 piper/methods/generate-image/inputs.template.json create mode 100644 piper/methods/summarize-pdf/inputs.template.json delete mode 100644 piper/runner.py create mode 100644 tests/unit/test_attended_cli.py create mode 100644 tests/unit/test_blocking_cli.py delete mode 100644 tests/unit/test_cli.py create mode 100644 tests/unit/test_detached_cli.py delete mode 100644 tests/unit/test_extract_entities.py delete mode 100644 tests/unit/test_file_input.py delete mode 100644 tests/unit/test_generate_image.py create mode 100644 tests/unit/test_generated_clients.py create mode 100644 tests/unit/test_inputs.py create mode 100644 tests/unit/test_mode_symmetry.py delete mode 100644 tests/unit/test_summarize_pdf.py create mode 100644 wip/mode-split-clis.md diff --git a/.claude/skills/bootstrap/scripts/bootstrap.py b/.claude/skills/bootstrap/scripts/bootstrap.py index 5bad2a2..c7401cf 100644 --- a/.claude/skills/bootstrap/scripts/bootstrap.py +++ b/.claude/skills/bootstrap/scripts/bootstrap.py @@ -239,27 +239,30 @@ def transform_pyproject(text: str, names: Names, opts: "Options") -> str: else: text = text.replace(f"yourusername/{TEMPLATE_NAME}", f"yourusername/{names.dist}") - # Name tokens: targeted per-key edits rather than the character-after pass, - # because a package name sitting bare in a TOML array (`packages = ["piper"]`) - # or as a bare table key looks like a command position to the generic - # heuristic. Each occurrence is edited by its key; the post-run assertion in - # run() is the backstop that catches any key not handled here. + # Name tokens: targeted edits rather than the character-after pass, because + # a package name sitting bare in a TOML array (`packages = ["piper"]`) or as + # a bare table key looks like a command position to the generic heuristic. + # The name-bearing keys are edited by key first; the post-run assertion in + # run() is the backstop that catches any occurrence not handled here. key_edits = { # [project].name -> distribution name f'name = "{TEMPLATE_NAME}"': f"name = {toml_str(names.dist)}", # [project.scripts]: = ".cli:app" f'{TEMPLATE_NAME} = "{TEMPLATE_NAME}.cli:app"': f'{names.dist} = "{names.package}.cli:app"', - # [tool.setuptools] packages (import names) - f'packages = ["{TEMPLATE_NAME}", "{TEMPLATE_NAME}.examples"]': f'packages = ["{names.package}", "{names.package}.examples"]', - # [tool.setuptools.package-data] table key (import name) + # [tool.setuptools.package-data] bare table key (import name) f'{TEMPLATE_NAME} = ["py.typed"': f'{names.package} = ["py.typed"', - # [tool.mypy] packages (import name) - f'packages = ["{TEMPLATE_NAME}"]': f'packages = ["{names.package}"]', - # [tool.pyright] include (import name) - f'include = ["{TEMPLATE_NAME}", "tests"]': f'include = ["{names.package}", "tests"]', } for old_text, new_text in key_edits.items(): text = text.replace(old_text, new_text) + # Everything left is the package (import/path) form: quoted-exact array + # entries ("piper" in [tool.setuptools] packages / [tool.mypy] / [tool.pyright]), + # and dotted/path positions (the "piper.generated.*" package + package-data + # entries, the "piper/generated" ruff exclude, prose comments about + # piper/methods/*). The name-bearing keys above have already been rewritten, + # so the quoted-exact replace plus the package-position rule cover the rest — + # and neither goes stale when a package is added to or removed from an array. + text = text.replace(f'"{TEMPLATE_NAME}"', f'"{names.package}"') + text = PIPER_PACKAGE_RE.sub(names.package, text) return text diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ae67bb..f4a8d7d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## [Unreleased] + +- **Every demo now runs with zero arguments, using a bundled sample.** `piper blocking extract-entities` (or any demo in any mode) no longer errors out when you give it no input — it falls back to a built-in sample so a fresh clone shows a working result on the first command, then a one-line notice on stderr tells you what it used and how to pass your own (`extract-entities`/`generate-image` take a sample text/prompt; `summarize-pdf` defaults to `samples/sample-invoice.pdf`). The samples live in `piper/inputs.py` (they are also the values the README documents); the notice goes to stderr, so stdout stays pipeable. `read_text_input()` gains a `sample` parameter and returns a `TextInput(text, is_sample)`. +- **Protocol-route HTTP errors now surface the API's problem+json, not httpx's stringification.** A non-2xx from `execute`/`start`/`runs/*` used to print `Client error '400 Bad Request' for url …` plus an MDN link — the RFC 7807 body (its `title`, `detail`, and machine `error_type`) was thrown away. `piper/errors.py` now reads that body and shows the server's `detail`; and it branches on the structured `error_type`, so a `/start` against a synchronous-only runner (`StartRequiresAsyncOrchestration`) is presented with a hint pointing you at `piper blocking`, the mode that works there. +- **Breaking (CLI): the execution mode is now the command group, not an option.** `--mode`, `--detach`, and the `PIPELEX_EXECUTION_MODE` env var are all gone; each mode is its own command group — `piper blocking `, `piper attended `, `piper detached ` — and every demo exists in all three. `piper runs status|result|wait ` moved into the mode that produces run ids: `piper detached status|result|wait `. There is no default mode anymore; the mode is explicit in every invocation, which is itself the lesson. The middle mode is named `attended`, not `durable`, because detached runs are durable too — the axis the names describe is who waits. +- **Each execution mode is now a self-contained copy-paste unit.** A mode used to be an `ExecutionMode` value threaded through a dispatch chain (CLI command → `_dispatch()` → `match` → `piper/runner.py` → the SDK), so answering "how do I call the API?" meant hopping through four layers, and the `match` existed only to showcase all three lifecycles behind one option — nothing anyone would copy. Now each mode is one file (`piper/blocking/cli.py`, `piper/attended/cli.py`, `piper/detached/cli.py`) holding its commands, its SDK lifecycle, and its progress rendering, with one public lifecycle helper that *is* the mode (`execute_pipe` / `start_and_wait` / `start_pipe`). `piper/runner.py`, `ExecutionMode`, `_dispatch()`, and the `RunResults` adaptation that existed only to give the dispatch one return type are all deleted; `piper/cli.py` shrinks to a `load_dotenv` callback plus three `add_typer` calls. New: `docs/cli-architecture.md`. +- **Only mode-orthogonal code is shared: `piper/inputs.py` (new) and `piper/errors.py`.** `piper/file_input.py` is merged into `piper/inputs.py`, which now also owns the text-or-file input helper (`read_text_input`). Lifecycle code is never shared — the demo commands are deliberately near-duplicated across the three mode files (diff two of them and only the lifecycle helper differs), and `tests/unit/test_mode_symmetry.py` guards that duplication against drift. The hints in `piper/errors.py` now name the mode groups (a blocking run that hits the ~30s cap points at `piper attended`; an interrupted attended run points at `piper detached wait `, which is exactly what it has become). +- **The typed output models are now generated from the bundles, not hand-written.** `pipelex codegen` projects each method's concepts into `piper/generated//models.py` (a stamped module plus a `codegen.lock` recording the artifact set). `generate-image` parses into the generated built-in `Image` model (natives are materialized into the generated client), replacing the hand-written `GeneratedImage`. +- **Removed the `piper/examples/` layer.** Each demo is now one self-contained command in `piper/cli.py`: bundle path (via a single `METHODS_DIR` constant), pipe code, and inline narrowing into the generated model (`Model.model_validate(results.main_stuff)`) — no more per-demo wrapper module hiding that line behind a `parse()` indirection. The `parse()` unit tests went with it (they only exercised pydantic's `model_validate`); the e2e tests narrow inline the same way. +- **Fixed the `/bootstrap` pyproject transform.** Its per-key edits had gone stale against the current `pyproject.toml` (multi-line `packages` array with the `piper.generated.*` subpackages, quoted package-data keys, the `piper/generated` ruff exclude), so bootstrapping aborted on the survivor check. Package-name tokens in pyproject are now rewritten generically (quoted-exact + dotted/path positions), which no longer goes stale when the package list changes; the bootstrap test fixture now mirrors the real pyproject shapes. +- **New make targets:** `make codegen` regenerates the typed clients and the runnable `inputs.template.json` scaffolds beside each bundle; `make codegen-check` verifies offline (pure hashing, no network or API key) that the generated clients are current. The `pipelex` CLI is not a starter dependency — point the `PIPELEX` make variable at a pipelex install that ships `codegen`. CI wiring of `codegen-check` lands once a released pipelex ships the command. +- **New offline smoke tests** (`tests/unit/test_generated_clients.py`): the generated modules import, carry the stamp + lock, round-trip their serialization, and each committed input template names exactly the inputs the CLI dispatches. +- `piper/generated` is excluded from ruff (reformatting generated files would trip the drift check) and remains fully type-checked; documented the whole flow in `docs/codegen.md`. +- Each `codegen.lock` is shipped as package data, so `pipelex codegen check` also works against an installed (wheel) copy, not just a git checkout. +- **Breaking (generated `Image` shape):** the generated built-in `Image` model now carries typed `width` / `height` optional integer fields instead of the untyped `size` dict, and native field descriptions come from the standard's pinned definitions. Regenerated all committed clients (stamps, locks, and crate fingerprints updated). + ## [v0.13.0] - 2026-07-07 - **Breaking:** renamed the starter's placeholder project from `my-project` / `my_project` / `My Project` to `piper` / `Piper`. The console command is now `uv run piper ...`, the template package is `piper/`, `pyproject.toml` points at `piper.cli:app`, and the e2e extract-entities test file no longer carries the project placeholder in its name. diff --git a/CLAUDE.md b/CLAUDE.md index 118908d..0a67b5a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,6 +25,8 @@ Run specific tests (local only): `make tp TEST=test_function_name` - `make li` - Lock + install - `make cleanderived` - Remove caches/compiled files (useful when linters get confused) - `make validate` / `make v` - Lint/validate the `.mthds` bundle with plxt (offline) +- `make codegen` - Regenerate the typed clients + input templates from the `.mthds` methods (needs the `pipelex` CLI — see below) +- `make codegen-check` - Verify generated clients are current (offline, pure hashing against each `codegen.lock`) - `make tb` - Quick boot test (constructs the API client, no network) - `make fui` - Fix unused imports only - `make plxt-format` - Format `.mthds`/`.toml` files with plxt @@ -35,14 +37,17 @@ Run specific tests (local only): `make tp TEST=test_function_name` This starter calls the **hosted Pipelex API** via the `pipelex-sdk` package (`PipelexAPIClient`) — it does **not** run Pipelex as a local library. The `.mthds` bundle is read from disk and sent to the API as content (`mthds_contents`); the API runs the method and returns the output. - Credentials/endpoint come from `PIPELEX_BASE_URL` / `PIPELEX_API_KEY` (see `.env.example`). `python-dotenv` loads `.env` when running the CLI or tests. -- The `piper` CLI (`piper/cli.py`) is a Typer app; `piper/runner.py` dispatches each run by execution mode — `blocking` (`client.execute`), durable attended (`client.start` + `client.wait_for_result`), and durable detached (`client.start` only, resumed via `piper runs status|result|wait `). It branches on mode explicitly rather than using the SDK's `start_and_wait` self-healing one-liner, because teaching the mode difference is the point. -- The SDK resolves the main output on both modes: `client.execute` returns a `PipelexExecuteResult` and the durable path a `RunResults`, both exposing a resolved `.main_stuff` (a completed run with no main stuff raises `MissingMainStuffError`). Per-example narrowing lives in `piper/examples/`, one "copy me" module per demo (bundle path, output model, `parse()`) — `extract_entities.parse()` validates `results.main_stuff` into a typed `ExtractedEntities` model. SDK errors are mapped to CLI-facing messages + hints in `piper/errors.py`. -- Three demo commands share that dispatch: `extract-entities` (text in), `summarize-pdf` (a *file* in — `piper/file_input.build_document_input()` encodes a local file as a base64 `data:` URL wrapped in a `{"concept": "Document", "content": …}` envelope), and `generate-image` (prompt in). `generate-image` is the deliberate slow case that overruns the ~30s blocking cap, so it's how the starter demonstrates the durable-vs-blocking difference concretely. `samples/sample-invoice.pdf` is shipped for `summarize-pdf`. +- **The execution mode is the command group, not an option.** There are exactly three, each a self-contained Typer sub-package: `piper blocking …` (`client.execute` — one call, dies at the hosted ~30s cap), `piper attended …` (`client.start` + `client.wait_for_result` — durable, you wait), `piper detached …` (`client.start` only — durable, you collect it later with `piper detached status|result|wait `). Attended and detached start the *same* durable run; the axis they name is who waits. There is no default mode and no `--mode` option: `piper/cli.py` is a thin assembler (`load_dotenv` callback + three `add_typer` calls in reading order) and nothing else. +- **Each mode file is a copy-paste unit; lifecycle code is never shared.** `piper//cli.py` holds that mode's whole story: its `typer.Typer`, its consoles (results → stdout, progress → stderr), its one public lifecycle helper (`execute_pipe` / `start_and_wait` / `start_pipe`, plus `attend_run` + the fetchers in detached), its demo commands, and a private `_run()` that wraps `asyncio.run` and catches SDK errors once. The **only** shared modules are the two that are orthogonal to execution: `piper/inputs.py` (text-or-file input with a built-in **sample fallback** so every demo runs with zero arguments — `read_text_input` returns `TextInput(text, is_sample)` and the demo prints a stderr notice when the sample was used; plus file → `{"concept": "Document", "content": …}` envelope with the file base64-encoded into a `data:` URL, and the `SAMPLE_*` constants) and `piper/errors.py` (SDK error → message + hint, hints naming the mode groups; it reads the RFC 7807 **problem+json** body off raw protocol-route `httpx.HTTPStatusError`s and branches on the structured `error_type`, e.g. `StartRequiresAsyncOrchestration` → "use `piper blocking`"). Do not introduce a shared runner — the dispatch indirection is exactly what this layout removed. See `docs/cli-architecture.md`. +- **Full demo matrix, guarded.** All three demos exist in all three modes: `extract-entities` (text in), `summarize-pdf` (a *file* in), `generate-image` (prompt in). `generate-image` is the deliberate slow case that overruns the ~30s blocking cap — `piper blocking generate-image` is *expected to fail*, and that is the teaching moment for the durable modes. The near-duplication across mode files is the pedagogy (diff two mode files and only the lifecycle helper differs); `tests/unit/test_mode_symmetry.py` keeps it from drifting. `samples/sample-invoice.pdf` is shipped for `summarize-pdf`. +- The SDK resolves the main output on every path (`client.execute` returns a `PipelexExecuteResult`, the durable path a `RunResults`, both exposing a resolved `.main_stuff`, typed `Any`; a completed run with no main stuff raises `MissingMainStuffError`). So every lifecycle helper returns `main_stuff`, and each demo command narrows it inline — e.g. `ExtractedEntities.model_validate(main_stuff)` — into the generated model. There is no per-example wrapper layer. +- The modes spell out lifecycles the SDK could hide: `client.start_and_wait()` is a self-healing one-liner that picks the path for you (the production shortcut). The starter writes them out because teaching the difference is the point. +- **The typed models are generated, never hand-written.** `pipelex codegen` projects each bundle's concepts into `piper/generated//models.py` (stamped, locked by a sibling `codegen.lock`); the mode CLIs and the e2e tests import from there. Do NOT edit generated files — edit the bundle, then `make codegen` (regenerates models + `inputs.template.json` scaffolds) and `make codegen-check` (offline drift check). The `pipelex` CLI is not a dependency of this starter; point the `PIPELEX` make variable at a pipelex install that ships `codegen`. `piper/generated` is excluded from ruff (reformatting would trip the drift check) but fully type-checked. See `docs/codegen.md`. ## Project Structure -- Package: `piper/` (Python 3.11+, target 3.11) -- Tests: `tests/` (unit = offline CLI/example/error-mapping tests; integration = offline boot/bundle checks + API `validate`; e2e = full run via the API) +- Package: `piper/` (Python 3.11+, target 3.11) — root `cli.py` + one sub-package per execution mode (`blocking/`, `attended/`, `detached/`) + the two shared modules (`inputs.py`, `errors.py`). New mode sub-packages must be added to `[tool.setuptools] packages` in `pyproject.toml`. +- Tests: `tests/` (unit = offline per-mode CLI tests patching each mode's public lifecycle helper, plus mode-symmetry / error-mapping / generated-client tests; integration = offline boot/bundle checks + API `validate`; e2e = full run via the API, one execution mode per demo so all three get end-to-end coverage) - Dependency manager: uv (>=0.7.2) - Pipelex dependency: `pipelex-sdk` package from PyPI (the API client — see pyproject.toml). The `pipelex` runtime is **not** a dependency. - `.mthds` files: Pipelex method definition files in `piper/methods//main.mthds` diff --git a/Makefile b/Makefile index 8fd14f0..7cb5d38 100644 --- a/Makefile +++ b/Makefile @@ -21,6 +21,15 @@ UV_MIN_VERSION = $(shell grep -m1 'required-version' pyproject.toml | sed -E 's/ USUAL_PYTEST_MARKERS := "(dry_runnable or not inference) and not (needs_output or pipelex_api)" +# The pipelex CLI that runs codegen. It is NOT a dependency of this starter (the starter talks to +# the hosted API through pipelex-sdk) — point PIPELEX at a pipelex install that ships `codegen`, +# e.g. `PIPELEX=/path/to/pipelex/.venv/bin/pipelex make codegen`. +PIPELEX ?= pipelex + +# Every programmatic invocation goes through this: --no-logo keeps the banner out of CI logs +# and agent context. Override PIPELEX, not this. +PIPELEX_RUN = $(PIPELEX) --no-logo + define PRINT_TITLE $(eval PROJECT_PART := [$(PROJECT_NAME)]) $(eval TARGET_PART := ($@)) @@ -50,6 +59,8 @@ make export-requirements-dev - Export requirements-dev.txt (all dependencies in make er - Shorthand -> export-requirements make erd - Shorthand -> export-requirements-dev make validate - Lint/validate the .mthds bundle with plxt +make codegen - Regenerate the typed clients + input templates from the .mthds methods +make codegen-check - Verify the generated clients are current (offline, pure hashing) make format - Format all (ruff-format + plxt-format) make lint - Lint all (ruff-lint + plxt-lint) @@ -104,7 +115,7 @@ export HELP test t test-quiet tq test-with-prints tp test-inference ti \ codex-tests gha-tests \ run-all-tests run-manual-trigger-gha-tests run-gha_disabled-tests \ - validate v check c cc agent-check agent-test \ + validate v check c cc agent-check agent-test codegen codegen-check \ merge-check-ruff-lint merge-check-ruff-format merge-check-plxt-format merge-check-plxt-lint merge-check-mypy merge-check-pyright \ li check-unused-imports fix-unused-imports check-uv check-TODOs @@ -191,6 +202,32 @@ validate: env $(call PRINT_TITLE,"Validating the .mthds bundle with plxt") $(VENV_PLXT) lint +########################################################################################## +### CODEGEN +########################################################################################## + +# Regenerate the typed clients (stamped models + codegen.lock) and the runnable input +# templates from the .mthds methods. Run after editing any main.mthds, then commit the result. +# The method list here must stay in lockstep with piper/methods/* and the +# packages/package-data lists in pyproject.toml. +codegen: + $(call PRINT_TITLE,"Regenerating typed clients from the .mthds methods") + @$(PIPELEX_RUN) codegen types --target python-pydantic --output piper/generated/extract_entities piper/methods/extract-entities && \ + $(PIPELEX_RUN) codegen types --target python-pydantic --output piper/generated/summarize_pdf piper/methods/summarize-pdf && \ + $(PIPELEX_RUN) codegen types --target python-pydantic --output piper/generated/generate_image piper/methods/generate-image && \ + $(PIPELEX_RUN) codegen inputs --output piper/methods/extract-entities/inputs.template.json piper/methods/extract-entities && \ + $(PIPELEX_RUN) codegen inputs --output piper/methods/summarize-pdf/inputs.template.json piper/methods/summarize-pdf && \ + $(PIPELEX_RUN) codegen inputs --output piper/methods/generate-image/inputs.template.json piper/methods/generate-image && \ + echo "Regenerated typed clients and input templates" + +# Offline drift check: pure hashing against each codegen.lock — no engine boot, no network, +# no API key. Exit 0 = current, 1 = drift (stale/hand-edited), 2 = no lock. +codegen-check: + $(call PRINT_TITLE,"Checking generated clients are current - offline") + @$(PIPELEX_RUN) codegen check piper/generated/extract_entities && \ + $(PIPELEX_RUN) codegen check piper/generated/summarize_pdf && \ + $(PIPELEX_RUN) codegen check piper/generated/generate_image + ############################################################################################## ############################ Cleaning ############################ ############################################################################################## diff --git a/README.md b/README.md index e6bb999..8b27750 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ It ships a handful of demo methods, each exposed as a `piper` CLI command: - **`extract-entities`** — given a piece of text, pull out the people, organizations, and dates it mentions. - **`summarize-pdf`** — given a document (PDF), produce a title, document type, and key points. Shows how to feed a *file* to a pipe. -- **`generate-image`** — given a text prompt, generate an image. Being slow, it's the example that best shows the durable-vs-blocking split (image generation routinely outlives the hosted ~30s blocking cap). +- **`generate-image`** — given a text prompt, generate an image. Being slow, it's the example that best shows the split between the execution modes (image generation routinely outlives the hosted ~30s blocking cap). Each prints its result as JSON. @@ -44,7 +44,7 @@ Install the dependencies, then run your first method. `uv run` executes a comman ```bash make install # create the venv and install deps with uv -uv run piper extract-entities "Alice from Acme met Bob on May 3rd, 2026." +uv run piper blocking extract-entities "Alice from Acme met Bob on May 3rd, 2026." ``` You get the extracted entities as JSON: @@ -57,17 +57,25 @@ You get the extracted entities as JSON: } ``` -A `Run started: run_…` line shows up first (on stderr) — durable mode (the default) prints the run id before polling, so a long run is never lost. Prefer a bare `piper …`? Activate the venv once with `source .venv/bin/activate` and drop the `uv run` prefix. +`blocking` is the execution mode — one call, one response. It is the first of three, and every demo runs in all three: see **Execution modes** below. Prefer a bare `piper …`? Activate the venv once with `source .venv/bin/activate` and drop the `uv run` prefix. ## Try the demos -Each demo is one `piper` command backed by one "copy me" module in `piper/examples/` — a bundle path, an output model, and a `parse()` narrower. Run them straight from the template. +Each demo is one self-contained `piper` command — a bundle path, a pipe code, and a typed narrowing of the result into its *generated* model. Every demo exists in every execution mode; the commands below use `blocking`, the simplest one. + +**Every demo runs with no arguments.** Give it nothing and it uses a bundled sample (and tells you so on stderr), so you can see a working result before you have any input of your own — then pass your own text, prompt, or file to replace it: + +```bash +uv run piper blocking extract-entities # uses the sample text +uv run piper blocking summarize-pdf # uses samples/sample-invoice.pdf +uv run piper blocking generate-image # uses the sample prompt +``` **Extract entities** — text in, structured entities out. ```bash -uv run piper extract-entities "Alice from Acme met Bob on May 3rd, 2026." -uv run piper extract-entities --file notes.txt # or read the text from a file +uv run piper blocking extract-entities "Alice from Acme met Bob on May 3rd, 2026." +uv run piper blocking extract-entities --file notes.txt # or read the text from a file ``` ```json @@ -77,7 +85,7 @@ uv run piper extract-entities --file notes.txt # or read the text from **Summarize a PDF** — a *file* goes in; `piper` base64-encodes it into a `Document` envelope for you, so you never host the file yourself. ```bash -uv run piper summarize-pdf samples/sample-invoice.pdf +uv run piper blocking summarize-pdf samples/sample-invoice.pdf ``` ```json @@ -92,11 +100,11 @@ uv run piper summarize-pdf samples/sample-invoice.pdf } ``` -**Generate an image** — the slow one, and the reason durable mode exists. Image generation routinely outlives the hosted ~30s blocking cap. +**Generate an image** — the slow one, and the reason the durable modes exist. Image generation routinely outlives the hosted ~30s blocking cap. ```bash -uv run piper generate-image "a fox reading under a tree" # durable (default): waits it out -uv run piper generate-image "a fox reading under a tree" --mode blocking # watch it hit the ~30s cap +uv run piper blocking generate-image "a fox reading under a tree" # watch it hit the ~30s cap +uv run piper attended generate-image "a fox reading under a tree" # durable: waits it out, no cap ``` ```json @@ -108,16 +116,16 @@ uv run piper generate-image "a fox reading under a tree" --mode blocking # wa } ``` -Open `public_url` in a browser to see the image. Run it with `--mode blocking` and you'll get a `PipelineExecuteTimeoutError` with a hint pointing you back to durable mode — that contrast is what the **Execution modes** section below is about. +Open `public_url` in a browser to see the image. Run it under `blocking` and you'll get a `PipelineExecuteTimeoutError` with a hint pointing you at `piper attended` — that contrast is what the **Execution modes** section below is about. ## How it works -`piper extract-entities ""` runs entirely through the SDK — nothing about the method lives on the server: +`piper blocking extract-entities ""` runs entirely through the SDK — nothing about the method lives on the server: ```mermaid flowchart TD subgraph Local["Local starter"] - CLI(["piper extract-entities '…'"]):::operation + CLI(["piper blocking extract-entities '…'"]):::operation Bundle[/" .mthds bundle
file on disk "/]:::data Read["read bundle contents"]:::operation Client["create PipelexAPIClient
from env credentials"]:::operation @@ -125,11 +133,11 @@ flowchart TD subgraph Hosted["Hosted Pipelex API"] Run["run method from submitted bundle"]:::service - MainStuff[/" results.main_stuff "/]:::data + MainStuff[/" main_stuff "/]:::data end - subgraph Output["Example output"] - Parse["parse() validates
typed model"]:::operation + subgraph Output["Typed output"] + Parse["validate into the
generated model"]:::operation JSON[/" JSON on stdout "/]:::terminal end @@ -149,49 +157,66 @@ flowchart TD 1. **Read the bundle.** `piper` reads `methods/extract-entities/main.mthds` from disk and constructs a `PipelexAPIClient`, which picks up `PIPELEX_BASE_URL` / `PIPELEX_API_KEY` from the environment. 2. **Run it on the API.** The bundle is sent as *content* (`mthds_contents`), so nothing method-specific needs to live in the runtime — edit the `.mthds` file and re-run, no redeploy. -3. **Narrow the result.** The SDK resolves `results.main_stuff`; the example's `parse()` validates it into a typed `ExtractedEntities` model, printed as JSON. +3. **Narrow the result.** The SDK resolves the run's `main_stuff`; the command validates it into the generated `ExtractedEntities` model (`ExtractedEntities.model_validate(main_stuff)`), printed as JSON. -The other demos run through the exact same path — they differ only in their inputs and output shapes. `summarize-pdf` sends a `Document` envelope (`file_input.build_document_input()` base64-encodes the file into a `data:` URL); `generate-image` returns the built-in `Image` content. +The typed models are **not hand-written**: they are generated from the `.mthds` bundles by `pipelex codegen` into `piper/generated/` (stamped, with a `codegen.lock` per method). Edit a bundle → `make codegen` regenerates the models and input templates → `make codegen-check` verifies offline that nothing is stale or hand-edited. See [docs/codegen.md](docs/codegen.md). -## Execution modes: durable vs blocking +The other demos run through the exact same path — they differ only in their inputs and output shapes. `summarize-pdf` sends a `Document` envelope (`inputs.build_document_input()` base64-encodes the file into a `data:` URL); `generate-image` returns the built-in `Image` content. -Every command takes `--mode` (env var `PIPELEX_EXECUTION_MODE`). The default is **durable**, and `generate-image` is the demo that shows why: +## Execution modes: blocking, attended, detached -### Blocking: finishes under the cap +**The mode is the command group, not an option.** There are three, and every demo runs in all three: -Blocking mode is a single `client.execute()` call. Use it when you expect the run to finish quickly: +```bash +uv run piper blocking extract-entities | summarize-pdf | generate-image +uv run piper attended extract-entities | summarize-pdf | generate-image +uv run piper detached extract-entities | summarize-pdf | generate-image +uv run piper detached wait | status | result +``` + +Read them in that order — each one is a single self-contained file (`piper//cli.py`) you can copy straight into your own project. + +### Act 1 — `blocking`: one call, one response + +A single `client.execute()` call. Nothing to learn, nothing to manage: you get the result in the response. ```mermaid sequenceDiagram - participant U as You (piper CLI) + participant U as You (piper blocking) participant API as Hosted Pipelex API - Note over U,API: blocking succeeds: one request, one response + Note over U,API: one request, one response U->>API: execute(pipe, bundle, inputs) API-->>U: result under ~30s ``` -### Blocking: times out +### Act 2 — the ~30s cap, and why the other two modes exist -The same blocking call fails when the hosted gateway cap is reached: +Behind the hosted gateway, a run longer than ~30s is cut off. `generate-image` is here to show you: + +```bash +uv run piper blocking generate-image "a fox reading under a tree" # expected to fail +``` ```mermaid sequenceDiagram - participant U as You (piper CLI) + participant U as You (piper blocking) participant API as Hosted Pipelex API - Note over U,API: blocking fails: run crosses the ~30s cap + Note over U,API: the run crosses the ~30s cap U->>API: execute(pipe, bundle, inputs) - API--xU: PipelineExecuteTimeoutError + durable-mode hint + API--xU: PipelineExecuteTimeoutError + "rerun with piper attended" ``` -### Durable attended: wait here +Long runs need a *durable* run — one that lives server-side, behind an id, and outlives your terminal. That is what the next two modes give you. They start the very same durable run; they differ only in **who waits**. + +### Act 3a — `attended`: start it, wait here -Durable attended mode starts a server-side run, prints the run id first, then keeps this terminal polling until the result is ready: +`client.start()` gives you a run id, then `client.wait_for_result()` polls it to completion from this terminal. No cap. ```mermaid sequenceDiagram - participant U as You (piper CLI) + participant U as You (piper attended) participant API as Hosted Pipelex API - Note over U,API: durable attended: survives the cap + Note over U,API: durable run, you wait for it U->>API: start(pipe, bundle, inputs) API-->>U: run id (printed first, so Ctrl-C is safe) loop poll every few seconds @@ -200,59 +225,84 @@ sequenceDiagram API-->>U: result ``` -### Durable detached: collect later +The id is printed *before* polling starts, so nothing is ever lost: Ctrl-C leaves the run executing server-side, and you pick it back up with `piper detached wait ` — an interrupted attended run has literally become a detached one. + +### Act 3b — `detached`: start it, collect it later -Detached mode starts the same durable server-side run, then exits immediately so you can resume from any terminal: +Same durable run, but `piper` exits as soon as it has the id — on stdout, so it pipes: + +```bash +RUN_ID=$(uv run piper detached generate-image "a fox reading under a tree") +uv run piper detached status $RUN_ID # where is it now? (no waiting) +uv run piper detached result $RUN_ID # its result, if it is done (no waiting) +uv run piper detached wait $RUN_ID # block until it is done, then print the result +``` ```mermaid sequenceDiagram - participant U as You (piper CLI) + participant U as You (piper detached) participant API as Hosted Pipelex API - Note over U,API: durable detached: start now, collect later + Note over U,API: durable run, nobody waits U->>API: start(pipe, bundle, inputs) API-->>U: run id, then exit - Note over U,API: later, any terminal - U->>API: piper runs wait + Note over U,API: later, any terminal, any machine + U->>API: piper detached wait API-->>U: result ``` -- **blocking** — one `client.execute()` call. Simplest, but behind the hosted gateway a run over ~30s is cut off with a `PipelineExecuteTimeoutError` that points you at durable mode. -- **durable attended** (default) — `client.start()` then poll to completion (`client.wait_for_result`). Survives the cap. The run id is printed *before* polling, so a Ctrl-C leaves the run executing server-side and you can resume it with `piper runs wait `. -- **durable detached** (`--detach`) — `client.start()` and return immediately. Pick the run back up later — even from another terminal — with `piper runs status|result|wait `. +### How the code is laid out + +Each mode is a **self-contained copy-paste unit**: `piper//cli.py` holds that mode's whole story — its commands, its SDK lifecycle, its progress rendering — with no dispatch layer in between. Reading path from command to API call is two hops, both in the same file: + +| Mode | The one function that *is* the mode | SDK calls | +| --- | --- | --- | +| `blocking` | `execute_pipe()` | `client.execute` | +| `attended` | `start_and_wait()` | `client.start` + `client.wait_for_result` | +| `detached` | `start_pipe()` (+ `attend_run()` for `wait`) | `client.start` (then `client.wait_for_result` later) | -`piper/runner.py` branches on the mode explicitly instead of calling the SDK's `start_and_wait()` self-healing one-liner (the production shortcut when you don't care which path runs) — teaching the difference is the point of this starter. +The only things shared across modes are the two concerns that have nothing to do with execution: `piper/inputs.py` (encode the inputs) and `piper/errors.py` (present an SDK error). **Lifecycle code is never shared** — that is what keeps each mode file copy-pasteable, and it is why the demo commands are near-duplicated across the three: diff `blocking/cli.py` against `attended/cli.py` and the only difference is the lifecycle helper. (`tests/unit/test_mode_symmetry.py` guards that duplication against drift.) + +The modes also spell out lifecycles the SDK could hide: `client.start_and_wait()` is a self-healing one-liner that picks the right path by itself — the production shortcut when you don't care which one runs. This starter branches explicitly because teaching the difference is the point. ## Project structure ``` piper/ - cli.py # the `piper` Typer CLI (console-script entry point) - runner.py # execution-mode dispatch: blocking / durable attended / detached - errors.py # maps SDK errors to CLI messages + hints - file_input.py # encode a local file into a Pipelex Document input envelope - examples/ # one "copy me" unit per demo: bundle path, output model, parse() - extract_entities.py # text → { people, orgs, dates } - summarize_pdf.py # document → { title, doc_type, key_points } - generate_image.py # prompt → image + cli.py # the `piper` console script: loads .env, mounts the three mode groups + blocking/cli.py # the whole blocking mode in one file (execute_pipe) + attended/cli.py # the whole attended mode in one file (start_and_wait) + detached/cli.py # the whole detached mode in one file (start_pipe + wait/status/result) + inputs.py # SHARED: text-or-file input, and file → Document envelope + errors.py # SHARED: maps SDK errors to CLI messages + hints + generated/ # typed clients generated from the bundles (`make codegen`) — do not edit + extract_entities/ # models.py (stamped) + codegen.lock + summarize_pdf/ + generate_image/ methods/ # the method bundles (sent to the API as content) - extract-entities/main.mthds - summarize-pdf/main.mthds - generate-image/main.mthds + extract-entities/ # main.mthds + inputs.template.json (generated runnable template) + summarize-pdf/ + generate-image/ samples/ sample-invoice.pdf # a document to try `summarize-pdf` on tests/ - unit/ # offline CLI / example / error-mapping tests + unit/ # offline CLI / error-mapping / generated-client tests integration/ # offline boot/bundle checks + API validate (pipelex_api) - e2e/ # full run against the API (inference) + e2e/ # full run against the API (inference) — one mode per demo .env.example # PIPELEX_BASE_URL + PIPELEX_API_KEY ``` +Only two modules are shared across the modes, and neither knows anything about execution: see [docs/cli-architecture.md](docs/cli-architecture.md) for the sharing rule and the anatomy of a mode file. + ## Useful commands ```bash -uv run piper extract-entities "…" --detach # start a durable run, print its id, return -uv run piper runs wait # resume a detached run (also: runs status | runs result) +uv run piper blocking extract-entities "…" # one call, one response +uv run piper attended generate-image "…" # durable run, wait here for it +uv run piper detached generate-image "…" # durable run, print its id, return +uv run piper detached wait # collect it later (also: detached status | detached result) make validate # lint/validate the .mthds bundles with plxt (offline) +make codegen # regenerate the typed clients + input templates from the bundles +make codegen-check # verify the generated clients are current (offline, pure hashing) make agent-check # fix-imports + format + lint + pyright + mypy make agent-test # offline test suite (silent on success) make test-inference # tests that hit the API (needs a key) diff --git a/TODOS.md b/TODOS.md new file mode 100644 index 0000000..134d28d --- /dev/null +++ b/TODOS.md @@ -0,0 +1,100 @@ +# TODO: split the execution modes into three self-contained CLIs + +Implementation plan for the design in **[`wip/mode-split-clis.md`](wip/mode-split-clis.md)** — read that first; it is the canonical design (target surface, package anatomy, sample code, rejected alternatives). This file tracks execution and carries just enough context to cold-start a fresh session. + +## Cold-start context + +**Goal.** Kill the `--mode` option and its dispatch chain (`cli.py` → `_dispatch` → `match ExecutionMode` → `runner.py` → SDK). Each execution mode becomes a self-contained Typer sub-package whose `cli.py` is a copy-paste unit: `piper blocking …`, `piper attended …`, `piper detached …`. + +**Decisions already taken (do not re-litigate):** +- One `piper` binary, three command groups mounted in reading order: `blocking`, `attended`, `detached`. +- Naming is `blocking` / `attended` / `detached` (not `durable` — detached runs are durable too; attended/detached names the who-waits axis). +- Full demo matrix: every demo command (`extract-entities`, `summarize-pdf`, `generate-image`) exists in every mode group. +- Run-lifecycle commands live inside detached: `piper detached wait|status|result `. No top-level `runs` group. +- Sharing rule: share only what's orthogonal to modes — `piper/errors.py` (presentation) and a new `piper/inputs.py` (input encoding). **Never share lifecycle code**; no shared runner module, ever. +- No default mode, no `PIPELEX_EXECUTION_MODE` env var. Breaking change, no compat shims (workspace policy). + +**Target layout and fixed names** (each mode `cli.py`: docstring with copy-paste contract → `app` + `output_console`/`progress_console` (stderr) → public lifecycle helper → demo commands + private `_run()` asyncio/error wrapper; `METHODS_DIR = Path(__file__).parent.parent / "methods"` in each mode file): + +``` +piper/cli.py root app: load_dotenv callback + add_typer × 3 — nothing else +piper/inputs.py read_text_input(*, text, file) + build_document_input(file) [merges file_input.py] +piper/errors.py kept shared; hints reworded to group paths (see Phase 3) +piper/blocking/cli.py lifecycle helper: execute_pipe(*, pipe_code, bundle, inputs) → client.execute +piper/attended/cli.py lifecycle helper: start_and_wait(*, pipe_code, bundle, inputs) → client.start + wait_for_result (prints run id first; Ctrl-C hint → `piper detached wait `) +piper/detached/cli.py start_pipe(*, …) → client.start; attend_run(run_id) backs `wait`; fetchers back `status`/`result` +``` + +**What gets deleted:** `piper/runner.py`, `piper/file_input.py`, `ExecutionMode`, `--mode`/`ModeOption`/`MODE_HELP`, `_dispatch`, the `RunResults` adaptation in `run_blocking` (each mode returns its natural SDK type / resolved `main_stuff`), the top-level `runs` sub-app. + +**Facts a cold session needs (verified 2026-07-13):** +- e2e tests call the runner helpers **directly** (e.g. `run_durable_attended(...)`), not through CliRunner — they swap to the new public lifecycle helpers, no CLI plumbing needed. +- `tests/integration/test_fundamentals.py` and the e2e tests import `METHODS_DIR` from `piper.cli`. The thin root app won't export it; tests compute it themselves (`Path(piper.__file__).parent / "methods"` in the test module or shared via `tests/conftest.py`). +- Unit CLI tests patch runner functions via `piper.cli` today (`mocker.patch("piper.cli.run_durable_attended", …)`); new tests patch each mode's public lifecycle helper (`piper.blocking.cli.execute_pipe`, etc.). +- The bootstrap rename script (`.claude/skills/bootstrap/scripts/bootstrap.py`) already handles bare and dotted package positions in pyproject (`piper`, `piper.generated.*`) — new `piper.blocking|attended|detached` entries ride the same rewrite; verify, don't rewrite. +- `piper/generated/` and `piper/methods/` are untouched by this work; `make codegen-check` must stay green with no regeneration. +- Workspace test rules: pytest-mock only (never unittest.mock), exactly one TestClass per test module, no `__init__.py` in test dirs. +- Checks: `make agent-check` (format/lint/pyright/mypy) and `make agent-test` (offline suite) after every phase; `pipelex_api`/`inference`-marked tests don't run offline — e2e edits are verified by review (or `make test-inference` if a key is at hand). + +--- + +## Phase 1 — Shared input module ✅ + +- [x] Create `piper/inputs.py`: move `build_document_input()` from `piper/file_input.py` verbatim; move `_read_text_input()` out of `piper/cli.py` as public `read_text_input(*, text: str | None, file: Path | None) -> str` (keeps raising `typer.BadParameter` — this is a CLI starter, typer in a shared CLI-input module is fine). +- [x] Point `piper/cli.py` at the new module; delete `piper/file_input.py`. +- [x] Rename `tests/unit/test_file_input.py` → `tests/unit/test_inputs.py`; add direct `read_text_input` cases (text only / file only / both → error / neither → error). (Class renamed `TestBuildDocumentInput` → `TestInputs`, one TestClass for the module.) +- [x] `make agent-check` and `make agent-test` green. + +## Phase 2 — Build the three mode packages (old surface still intact) ✅ + +Build the new world alongside the old one so the repo stays green mid-flight. Follow the anatomy and sample code in `wip/mode-split-clis.md` closely — the blocking file is spelled out there nearly in full. + +- [x] `piper/blocking/__init__.py` + `cli.py`: `execute_pipe()` (one `client.execute` inside a Rich status), the demo commands, `_run()`. `generate-image`'s help text says it is expected to hit the hosted ~30s cap — that failure is the teaching moment. +- [x] `piper/attended/__init__.py` + `cli.py`: `start_and_wait()` — start, print `Run started: ` on stderr *before* polling, `wait_for_result` with `on_poll` status updates, `asyncio.CancelledError` → "resume with `piper detached wait `" hint then re-raise. Docstring keeps the note that the SDK's `start_and_wait()` one-liner is the production shortcut; the starter spells it out on purpose. +- [x] `piper/detached/__init__.py` + `cli.py`: demo commands print the run id on **stdout** (pipeable: `RUN_ID=$(piper detached …)`) and the resume hint on stderr; `wait` (via `attend_run()`, prints raw `main_stuff` JSON — generic, no per-demo narrowing), `status` (coarse status + degraded caveat), `result` (no-wait `RunResultState` match: running → hint, completed → same JSON rendering as `wait`, failed → red + exit 1). +- [x] Mount the three groups in root `piper/cli.py` with `add_typer` in reading order, one-line help each ("Start here." / "…wait here for the result." / "…collect it later."). Keep the existing flat commands and `runs` group for now. +- [x] Add `piper.blocking`, `piper.attended`, `piper.detached` to `[tool.setuptools] packages` in `pyproject.toml`. +- [x] Resolve the design's open questions while writing the helpers: (a) how `pipelex-sdk` types `main_stuff` — helpers return it as the SDK types it, the generated-model `model_validate` in each command restores type safety; (b) confirm Typer lists groups in `add_typer` order. +- [x] New unit tests (one TestClass per module): `tests/unit/test_blocking_cli.py`, `test_attended_cli.py`, `test_detached_cli.py` — invoke through the **root** app (`["blocking", "extract-entities", …]`), patch the mode's public lifecycle helper, port the input-handling assertions from `test_cli.py` (text/file exclusivity, document envelope shape, missing-input errors) into their new homes. +- [x] `tests/unit/test_mode_symmetry.py`: assert the three groups expose identical demo command names (detached additionally exposes its lifecycle commands) — the drift guard for the full matrix. +- [x] `make agent-check` and `make agent-test` green. + +**CHECKPOINT — end of Phase 2 (reached).** Both surfaces coexist and everything is green (`make agent-check`, `make agent-test`). + +**Open questions — resolved:** +- **(a) `main_stuff` typing.** The SDK types it `Any` on both paths, deliberately (`RunResults.main_stuff: Any` — polymorphic content, and a falsy value like `[]` / `0` is valid; `PipelexExecuteResult.main_stuff` is an `Any` property that digs it out of the working memory). So all three lifecycle helpers return `Any` and each demo command's `Model.model_validate(main_stuff)` is what restores type safety, exactly as the design anticipated. Note this made the helper signatures *uniform*: all three return the resolved main output rather than an SDK envelope — `attend_run(run_id) -> Any` too, so `wait` and the e2e tests read the same as the other modes. +- **(b) Group ordering.** Typer lists groups in `add_typer` order (verified with `uv run piper --help`): blocking → attended → detached. No alphabetization. Locked in by `test_mode_symmetry.py::test_the_root_app_mounts_the_modes_in_reading_order`. + +**Deviations from the design doc:** none of substance. Two notes: +- `_run()` is byte-identical across the three mode files (SDK errors → presented exit 1; `KeyboardInterrupt` → exit 130). The design's blocking sample omitted the `KeyboardInterrupt` arm; keeping it in all three preserves the "only the lifecycle helper differs" diff contract *and* the current exit-130-on-Ctrl-C behavior. +- `test_mode_symmetry.py` guards more than command names: it also compares each demo's **parameter list** across the three modes, so an argument added to one mode and forgotten in another fails too. + +## Phase 3 — Remove the old surface ✅ + +- [x] Shrink root `piper/cli.py` to the thin assembler (module docstring, `load_dotenv` callback, three `add_typer` calls). Delete the flat demo commands, `runs_app`, `_dispatch`, `_run_cli`, `_render_result_state`, `_print_raw_results`, `ModeOption`, `MODE_HELP`, `METHODS_DIR`. +- [x] Delete `piper/runner.py`. +- [x] Reword the hints in `piper/errors.py` to group paths: timeout → "Rerun the same command with `piper attended …`"; lifecycle-unavailable → "use `piper blocking …`"; run-failed → "Inspect it with `piper detached status `"; run-timeout → "Resume waiting with `piper detached wait `". Update the module docstring (it names the per-command catch site). +- [x] Delete `tests/unit/test_cli.py` (its assertions were ported in Phase 2); update `tests/unit/test_errors.py` hint assertions. +- [x] Fix the `METHODS_DIR` imports in `tests/integration/test_fundamentals.py` and `tests/e2e/*` (compute locally or share via `tests/conftest.py`). +- [x] Re-point the e2e tests, one lifecycle per demo so every mode gets end-to-end coverage: `test_extract_entities` → `piper.blocking.cli.execute_pipe`; `test_summarize_pdf` → `piper.attended.cli.start_and_wait`; `test_generate_image` → `piper.detached.cli.start_pipe` then `attend_run` (also covers the run-id lifecycle). These are direct async helper calls, same style as today. +- [x] Repo-wide sweep for stale references (`runner`, `ExecutionMode`, `--mode`, `PIPELEX_EXECUTION_MODE`, `piper runs`, `file_input`) across `piper/`, `tests/`, `Makefile`, `.env.example`. +- [x] `make agent-check`, `make agent-test`, and `make tb` green. + +**CHECKPOINT — end of Phase 3 (reached).** The new surface is the only surface; `make agent-check`, `make agent-test`, `make tb` all green, and `piper --help` lists exactly blocking → attended → detached. + +**Beyond the plan, the sweep caught / decided:** +- **`METHODS_DIR` in tests, resolved two ways.** The integration test (mode-agnostic — it only checks the bundles parse and validate) computes it from the package: `Path(piper.__file__).parent / "methods"`. Each e2e test imports it from *the mode it exercises* (`from piper.blocking.cli import METHODS_DIR, execute_pipe`), which reads honestly and doubles as proof the mode file is self-sufficient. No `tests/conftest.py` plumbing was needed. +- **`tests/unit/test_generated_clients.py`** spoke the dead "cli dispatch" vocabulary (comment + `test_input_template_matches_cli_dispatch`). Reworded to name the mode CLIs; the test is now `test_input_template_matches_the_cli_inputs`. +- Remaining `runner` hits across the repo are all legitimate and unrelated: the *bare runner* concept in error hints/`.env.example`, `CliRunner` in tests, GitHub Actions runners. + +## Phase 4 — Docs, changelog, packaging polish ✅ + +- [x] README: quick start becomes `uv run piper blocking extract-entities "…"` (result JSON only — no run-id chatter); "Execution modes" becomes a three-act tour in reading order (blocking → the generate-image timeout → attended → detached), reusing the existing mermaid sequence diagrams one per group; update the project-structure block and "Useful commands". +- [x] Starter `CLAUDE.md`: rewrite the architecture bullets — three mode sub-packages, the sharing rule, public lifecycle helper names, `runs` → `detached`, no `runner.py`/`ExecutionMode`. +- [x] `docs/`: sweep `docs/codegen.md` for stale CLI invocations; add a short `docs/cli-architecture.md` capturing the sharing rule + mode-file anatomy (the durable version of what `wip/mode-split-clis.md` designed — wip docs are not release-facing). +- [x] CHANGELOG entry (breaking): `--mode` and `PIPELEX_EXECUTION_MODE` are gone — the mode is the command group (`piper blocking|attended|detached `); `piper runs status|result|wait` is now `piper detached status|result|wait`. The previous `[Unreleased]` entry ("`--detach` is gone; detached is now a third `--mode` value") was **folded into** the new one rather than kept: it gave migration advice (`use --mode detached`) that this change invalidates, and neither `--detach` nor `--mode` survives into the next release, so a reader coming from v0.13.0 needs exactly one entry. +- [x] Verify the `/bootstrap` skill against the new layout: run its unit test (`tests/unit/test_bootstrap_script.py`), confirm the pyproject fixture/rename handles the added `piper.*` package entries; extend the fixture with a mode sub-package entry if it isn't representative. → The pyproject transform is generic (quoted-exact `"piper"` + `piper` followed by `.`/`_`/`/`), so `piper.blocking|attended|detached` ride the same rewrite unchanged. The fixture **was** unrepresentative (only `piper.generated.*` dotted entries, no nested source dir); it now carries a `piper.blocking` package entry and a nested `piper/blocking/cli.py` importing a sibling module, so the survivor check actually covers the new shapes. +- [x] Final gate: `make agent-check`, `make agent-test`, `make codegen-check`, `make tb` — all green. +- [~] `make test-inference` — **blocked by an environment outage, not by this change.** `PIPELEX_BASE_URL` points at `api-dev.pipelex.com`, whose authenticated routes are all returning a raw nginx `503 Service Temporarily Unavailable` (an HTML page from the reverse proxy — nothing reaches the app). `/v1/execute`, `/v1/start`, **and** `/v1/validate` all 503, including the pre-existing `test_validate_bundle` integration test whose request-building code this refactor never touched; only the gateway-served `/v1/version` answers (200). Re-run `make test-inference` once the dev runner is back up. Until then the re-pointed e2e paths are verified by review + collection (they import and collect cleanly): each is a direct async call to its mode's public lifecycle helper, the same style as before, and the helpers make the same SDK calls the deleted `runner.py` made. + +**WRAP-UP.** ✅ Done — the mode split is implemented, the old surface is gone, and the suite is green offline. The lasting architectural notes now live in [`docs/cli-architecture.md`](docs/cli-architecture.md) (the sharing rule, the mode-file anatomy, why `attended`/`detached`) and in the starter `CLAUDE.md`; `wip/mode-split-clis.md` stays as the design record. The one open item is the live-API e2e run above, waiting on the dev environment. diff --git a/docs/cli-architecture.md b/docs/cli-architecture.md new file mode 100644 index 0000000..9bcc120 --- /dev/null +++ b/docs/cli-architecture.md @@ -0,0 +1,73 @@ +# CLI architecture: three modes, three self-contained files + +This starter has two jobs: show how easy it is to call the Pipelex API, and be code you would actually copy into your own project. Both push in the same direction — **the shortest possible reading path from a command to the SDK call it makes**. + +So the execution mode is not an option you pass, it is the command group you type: + +``` +piper blocking extract-entities | summarize-pdf | generate-image +piper attended extract-entities | summarize-pdf | generate-image +piper detached extract-entities | summarize-pdf | generate-image +piper detached wait | status | result +``` + +## The layout + +``` +piper/ + cli.py # the assembler: load .env, mount the three groups in reading order. Nothing else. + inputs.py # SHARED — encode the inputs + errors.py # SHARED — present an SDK error + blocking/cli.py # the whole blocking mode + attended/cli.py # the whole attended mode + detached/cli.py # the whole detached mode + the run-id lifecycle commands +``` + +Each mode file is a **copy-paste unit**: that file plus `inputs.py` plus `errors.py` is everything you need to lift the mode into your own project. Nothing else in `piper/` is load-bearing for it. + +## The sharing rule + +> Share what is orthogonal to execution. **Never share lifecycle code.** + +Input encoding and error presentation don't care how a run is executed, so they are shared. The lifecycle — what you call, in what order, and what you do while waiting — *is* the mode, so it lives in the mode's own file, in full, even when that means near-duplicating a demo command across three files. + +That duplication is deliberate, and it is the pedagogy: diff `blocking/cli.py` against `attended/cli.py` and the only thing that differs is the lifecycle helper. The moment two modes reach for a shared `runner` helper, a dispatch layer is back between the command and the SDK, and the reading path grows a hop. (The earlier version of this starter had exactly that: a `--mode` option → `_dispatch()` → `match ExecutionMode` → `runner.py` → the SDK. Four layers to answer "how do I call the API?".) + +The duplication can drift, so `tests/unit/test_mode_symmetry.py` holds it in place: every demo exists in every mode, with the same parameters. A demo added to one mode and forgotten in another fails CI. + +## Anatomy of a mode file + +All three have the same four-part shape, so they diff cleanly: + +1. **Module docstring** — the mode's contract in a paragraph, plus its copy-paste contract. +2. **App + consoles** — its own `typer.Typer`, its own `Console()` (stdout, for results — pipeable) and `Console(stderr=True)` (stderr, for progress chatter). +3. **The lifecycle helper** — one public async function that *is* the mode. It gets a public name because it is the featured code, and it is what the unit tests patch and the e2e tests call directly. +4. **The demo commands + a private `_run()`** — each command reads its input, reads its bundle, awaits the lifecycle helper through `_run()` (`asyncio.run` + the single `except (PipelineRequestError, httpx.HTTPStatusError)` that presents via `piper/errors.py`), narrows the result into its *generated* model, prints JSON. + +| Mode | Lifecycle helper | SDK calls | What the demo prints | +| --- | --- | --- | --- | +| `blocking` | `execute_pipe()` | `client.execute` | the result, as JSON | +| `attended` | `start_and_wait()` | `client.start` + `client.wait_for_result` | the run id (stderr), then the result as JSON | +| `detached` | `start_pipe()` | `client.start` | the run id, bare, on **stdout** | + +`detached` additionally owns the run-id lifecycle: `attend_run()` backs `wait`, and thin fetchers back `status` and `result`. It is the only mode with more than the demos, because "start now, collect later" is only a complete story if you can come back for the result. + +All three lifecycle helpers return the run's resolved `main_stuff` (the SDK types it `Any` — the content is polymorphic), and each command's `Model.model_validate(main_stuff)` is what restores type safety. The models are generated from the bundles; see [codegen.md](codegen.md). + +## Two conventions worth copying + +**stdout is the result; stderr is everything else.** Progress spinners, run ids in attended mode, error messages, and hints all go to stderr, so stdout stays pipeable. In detached mode the run id *is* the result, so it goes to stdout bare (`print`, not Rich) — `RUN_ID=$(piper detached generate-image "…")` just works. + +**Errors are caught once, at the root of the command.** `_run()` catches `PipelineRequestError` (the base of every error the SDK client raises) and the raw `httpx.HTTPStatusError` its protocol routes surface, maps it to a `(message, hint)` pair via `piper/errors.py`, and exits non-zero. Nothing else is caught anywhere: an unexpected exception crashes loudly with its traceback, which is what you want while you are building. + +The protocol routes (`execute`/`start`/`runs/*`) surface a non-2xx as a raw `httpx.HTTPStatusError`, whose default string is useless (`Client error '400 Bad Request' for url …` + an MDN link). `piper/errors.py` instead reads the API's RFC 7807 **problem+json** body and shows the server's own `detail`, branching on the structured `error_type` (never the transport status) for the cases worth a hint — a `/start` against a synchronous-only runner (`StartRequiresAsyncOrchestration`) is presented with a hint pointing at `piper blocking`. + +The hints name the mode *groups*, because the fix for a failed run is usually another group: a blocking run that hit the ~30s cap tells you to rerun it with `piper attended`; a run that timed out while you waited tells you to resume it with `piper detached wait `; a durable run against a runner that can't do them tells you to use `piper blocking`. + +**Every demo runs with zero arguments.** When you give neither an argument nor `--file`, the input helper returns a bundled sample (`piper/inputs.py`'s `SAMPLE_*` constants), and the command prints a one-line notice on stderr saying so. A fresh clone shows a working result on its very first command; stdout stays the clean, pipeable result because the notice is on stderr. Sample data is orthogonal to execution, so like input encoding it is shared, not duplicated per mode. + +## Why `attended` and `detached`, not `durable` + +Both start the *same* durable run — one that lives server-side behind an id and outlives your terminal. The only difference is who waits: `attended` polls from your terminal, `detached` exits and lets you collect the result later. Naming the middle one "durable" would suggest detached is not, which is exactly backwards. The mode names the axis that actually differs. + +A nice consequence: Ctrl-C during an attended run doesn't lose anything — it has just turned into a detached run, and the hint you get says so (`piper detached wait `). diff --git a/docs/codegen.md b/docs/codegen.md new file mode 100644 index 0000000..488e75d --- /dev/null +++ b/docs/codegen.md @@ -0,0 +1,45 @@ +# Generated typed clients + +The typed models this starter parses run results into are **generated from the `.mthds` bundles**, not hand-written. The method definition is the single source of truth: its output concepts are projected into Pydantic models, so editing a bundle and regenerating is all it takes to keep the Python side in sync. + +## What is generated, and where + +| Artifact | Path | Committed | Edited by hand | +| --- | --- | --- | --- | +| Typed models (one module per method) | `piper/generated//models.py` | Yes | **Never** — regenerate instead | +| Artifact-set lock | `piper/generated//codegen.lock` | Yes | Never | +| Runnable input template | `piper/methods//inputs.template.json` | Yes | Yes — it is a scaffold: copy it, fill in real values | + +Each `models.py` starts with a `pipelex-codegen-stamp` header recording the source crate fingerprint, engine version, projection, and a content hash. The sibling `codegen.lock` records the generated artifact set. Together they make drift detectable offline. + +The demo commands in each mode CLI (`piper/blocking/cli.py`, `piper/attended/cli.py`) import the generated models and only add the bundle path, the pipe code, and the narrowing line (`Model.model_validate(main_stuff)`) — nothing method-shaped is hand-written. + +## Workflow + +Edit a bundle (`piper/methods//main.mthds`), then: + +```bash +make codegen # regenerate models + input templates (writes only what changed) +make codegen-check # offline drift check: exit 0 current · 1 drift · 2 no lock +``` + +`codegen-check` is pure hashing against each `codegen.lock` — no engine boot, no network, no API key. It reports drift by category: missing (a locked artifact was deleted), modified (an artifact no longer matches the locked hash), hand-edited (a generated file was touched below its stamp), and orphan (a stale stamped file the lock no longer lists). + +One thing the offline check cannot see is a bundle edit that was never regenerated — detecting that requires resolving the bundle, which is the engine's job. The guard for it is `make codegen` itself: regeneration is write-if-changed, so running it and checking `git diff` is clean proves the committed clients match the bundles. + +The generated files are excluded from ruff in `pyproject.toml`: reformatting them would change their content hash and trip the drift check. They still go through pyright and mypy like any other code. + +## The `pipelex` CLI is not a dependency of this starter + +This starter talks to the **hosted Pipelex API** through `pipelex-sdk`; the `pipelex` runtime that ships the `codegen` command is not installed here. Point the `PIPELEX` make variable at a pipelex install: + +```bash +PIPELEX=/path/to/pipelex/.venv/bin/pipelex make codegen +PIPELEX=/path/to/pipelex/.venv/bin/pipelex make codegen-check +``` + +Once a released `pipelex` ships `codegen`, `codegen-check` is meant to run in CI so a bundle edit can never land without its regenerated client. + +## Offline test floor + +`tests/unit/test_generated_clients.py` runs everywhere (no pipelex CLI, no API key): it checks the generated modules import, carry the stamp + lock, round-trip their own serialization, and that each committed input template names exactly the inputs the CLI passes — so a regenerated template that no longer matches what `piper//cli.py` sends fails in CI even before `codegen-check` is wired in. diff --git a/piper/examples/__init__.py b/piper/attended/__init__.py similarity index 100% rename from piper/examples/__init__.py rename to piper/attended/__init__.py diff --git a/piper/attended/cli.py b/piper/attended/cli.py new file mode 100644 index 0000000..73d3b65 --- /dev/null +++ b/piper/attended/cli.py @@ -0,0 +1,150 @@ +"""The attended CLI — start a durable run, then wait here for its result. + +`client.start(...)` submits the run server-side (it survives anything, including +the ~30s cap that kills `piper blocking`), and `client.wait_for_result(...)` polls +it to completion from this terminal. The run id is printed *before* polling starts, +so a Ctrl-C never loses the run: it keeps executing server-side and you resume it +with `piper detached wait `. + +The SDK also offers `start_and_wait()`, a self-healing one-liner that picks the +right path by itself — that is the production shortcut. This starter spells the +lifecycle out because teaching the difference is the point. + +Copy-paste unit: this file + `piper/inputs.py` + `piper/errors.py`. The three mode +packages never share lifecycle code, so the demo commands below are deliberate +mirrors of their `blocking` / `detached` twins: only `start_and_wait()` differs. +""" + +import asyncio +from pathlib import Path +from typing import Annotated, Any, Coroutine, TypeVar + +import httpx +import typer +from mthds.protocol.exceptions import PipelineRequestError +from pipelex_sdk.client import PipelexAPIClient +from pipelex_sdk.runs import PollInfo, WaitForResultOptions +from rich.console import Console + +from piper.errors import present_error +from piper.generated.extract_entities.models import ExtractedEntities +from piper.generated.generate_image.models import Image +from piper.generated.summarize_pdf.models import DocumentSummary +from piper.inputs import SAMPLE_ENTITIES_TEXT, SAMPLE_IMAGE_PROMPT, SAMPLE_INVOICE, build_document_input, read_text_input + +ResultT = TypeVar("ResultT") + +METHODS_DIR = Path(__file__).parent.parent / "methods" + +app = typer.Typer(no_args_is_help=True, help="Run a demo as a durable run and wait here for the result.") + +# Results go to stdout (pipeable); progress chatter goes to stderr. +output_console = Console() +progress_console = Console(stderr=True) + + +async def start_and_wait(*, pipe_code: str, bundle: str, inputs: dict[str, Any]) -> Any: + """The whole attended lifecycle: start a durable run, print its id, wait here for the result. + + Credentials come from `PIPELEX_API_KEY` / `PIPELEX_BASE_URL`. The SDK resolves the + method's main output for you: `.main_stuff` is the content the pipe named as its + result (a completed run that names none raises `MissingMainStuffError`). + """ + async with PipelexAPIClient() as client: + start_result = await client.start(pipe_code=pipe_code, mthds_contents=[bundle], inputs=inputs) + run_id = start_result.pipeline_run_id + # Printed before the first poll, so Ctrl-C leaves you with a usable run id. + progress_console.print(f"Run started: [bold]{run_id}[/bold]") + short_id = run_id[:8] + with progress_console.status(f"Run {short_id}… in progress") as status: + + def on_poll(info: PollInfo) -> None: + status.update(f"Run {short_id}… in progress — {info.elapsed_seconds:.0f}s, poll #{info.attempt}") + + try: + results = await client.wait_for_result(run_id, options=WaitForResultOptions(on_poll=on_poll)) + except asyncio.CancelledError: + # Ctrl-C: the run keeps executing server-side — it has just become a detached run. + progress_console.print(f"\nInterrupted — the run is still executing. Resume with: [bold]piper detached wait {run_id}[/bold]") + raise + return results.main_stuff + + +@app.command(name="extract-entities") +def extract_entities( + text: Annotated[str | None, typer.Argument(help="The text to extract entities from.")] = None, + file: Annotated[Path | None, typer.Option("--file", help="Read the input text from a file instead of the argument.")] = None, +) -> None: + """Extract people, organizations, and dates from a piece of text.""" + resolved = read_text_input(text=text, file=file, sample=SAMPLE_ENTITIES_TEXT) + if resolved.is_sample: + progress_console.print(f"[dim]No text given — using the sample: {resolved.text!r}. Pass your own as an argument or via --file.[/dim]") + bundle = (METHODS_DIR / "extract-entities" / "main.mthds").read_text() + main_stuff = _run(start_and_wait(pipe_code="extract_entities", bundle=bundle, inputs={"text": resolved.text})) + # Narrow into the generated typed model (validates the concept's shape), then print it as JSON. + entities = ExtractedEntities.model_validate(main_stuff) + output_console.print_json(data=entities.model_dump()) + + +@app.command(name="summarize-pdf") +def summarize_pdf( + file: Annotated[Path | None, typer.Argument(help="Path to the PDF (or other document) to summarize.")] = None, +) -> None: + """Summarize a document into a title, type, and key points.""" + document = file or SAMPLE_INVOICE + if not document.is_file(): + msg = f"No such file: {document}" + raise typer.BadParameter(msg) + if file is None: + progress_console.print(f"[dim]No file given — using the sample: {document.name}. Pass a path to summarize your own document.[/dim]") + bundle = (METHODS_DIR / "summarize-pdf" / "main.mthds").read_text() + inputs = {"document": build_document_input(document)} + main_stuff = _run(start_and_wait(pipe_code="summarize_pdf", bundle=bundle, inputs=inputs)) + summary = DocumentSummary.model_validate(main_stuff) + output_console.print_json(data=summary.model_dump()) + + +@app.command(name="generate-image") +def generate_image( + prompt: Annotated[str | None, typer.Argument(help="The text prompt describing the image to generate.")] = None, + file: Annotated[Path | None, typer.Option("--file", help="Read the prompt from a file instead of the argument.")] = None, +) -> None: + """Generate an image from a text prompt. + + The same method `piper blocking generate-image` cannot finish: image generation + routinely outlives the hosted ~30s cap. Run durably, it just takes as long as it + takes. + """ + resolved = read_text_input(text=prompt, file=file, sample=SAMPLE_IMAGE_PROMPT) + if resolved.is_sample: + progress_console.print(f"[dim]No prompt given — using the sample: {resolved.text!r}. Pass your own as an argument or via --file.[/dim]") + bundle = (METHODS_DIR / "generate-image" / "main.mthds").read_text() + main_stuff = _run(start_and_wait(pipe_code="generate_image", bundle=bundle, inputs={"image_prompt": resolved.text})) + # On the hosted path the runtime returns a storage `url` (`pipelex-storage://…`) + # *and* a web-renderable `public_url` (a signed URL); the model keeps both. + image = Image.model_validate(main_stuff) + output_console.print_json(data=image.model_dump()) + + +def _run(coro: Coroutine[Any, Any, ResultT]) -> ResultT: + """Await the lifecycle, presenting SDK errors and Ctrl-C as clean exits. + + Every error the SDK client raises descends from `PipelineRequestError`, except the + raw `httpx.HTTPStatusError` its protocol routes surface. Nothing else is caught: + an unexpected exception crashes loudly with its traceback. + """ + try: + return asyncio.run(coro) + except (PipelineRequestError, httpx.HTTPStatusError) as exc: + presentation = present_error(exc) + progress_console.print(f"[red]Error:[/red] {presentation.message}") + if presentation.hint: + progress_console.print(f"\n[yellow]Hint:[/yellow] {presentation.hint}") + raise typer.Exit(1) from exc + except KeyboardInterrupt as exc: + # The resume hint was already printed by `start_and_wait`; the run keeps executing server-side. + raise typer.Exit(130) from exc + + +if __name__ == "__main__": + app() diff --git a/piper/blocking/__init__.py b/piper/blocking/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/piper/blocking/cli.py b/piper/blocking/cli.py new file mode 100644 index 0000000..7ffe85f --- /dev/null +++ b/piper/blocking/cli.py @@ -0,0 +1,128 @@ +"""The blocking CLI — one request, one response, done. + +The simplest way to run a method: `client.execute(...)` and you have the result. +Behind the hosted gateway a run longer than ~30s is cut off (run `generate-image` +to see it happen) — that is what `piper attended` and `piper detached` are for. + +Copy-paste unit: this file + `piper/inputs.py` + `piper/errors.py`. The three mode +packages never share lifecycle code, so the demo commands below are deliberate +mirrors of their `attended` / `detached` twins: only `execute_pipe()` differs. +""" + +import asyncio +from pathlib import Path +from typing import Annotated, Any, Coroutine, TypeVar + +import httpx +import typer +from mthds.protocol.exceptions import PipelineRequestError +from pipelex_sdk.client import PipelexAPIClient +from rich.console import Console + +from piper.errors import present_error +from piper.generated.extract_entities.models import ExtractedEntities +from piper.generated.generate_image.models import Image +from piper.generated.summarize_pdf.models import DocumentSummary +from piper.inputs import SAMPLE_ENTITIES_TEXT, SAMPLE_IMAGE_PROMPT, SAMPLE_INVOICE, build_document_input, read_text_input + +ResultT = TypeVar("ResultT") + +METHODS_DIR = Path(__file__).parent.parent / "methods" + +app = typer.Typer(no_args_is_help=True, help="Run a demo with a single blocking call (~30s cap on hosted).") + +# Results go to stdout (pipeable); progress chatter goes to stderr. +output_console = Console() +progress_console = Console(stderr=True) + + +async def execute_pipe(*, pipe_code: str, bundle: str, inputs: dict[str, Any]) -> Any: + """The whole blocking lifecycle: one call, and the result comes back in the response. + + Credentials come from `PIPELEX_API_KEY` / `PIPELEX_BASE_URL`. The SDK resolves the + method's main output for you: `.main_stuff` is the content the pipe named as its + result (a completed run that names none raises `MissingMainStuffError`). + """ + async with PipelexAPIClient() as client: + with progress_console.status("Running…"): + result = await client.execute(pipe_code=pipe_code, mthds_contents=[bundle], inputs=inputs) + return result.main_stuff + + +@app.command(name="extract-entities") +def extract_entities( + text: Annotated[str | None, typer.Argument(help="The text to extract entities from.")] = None, + file: Annotated[Path | None, typer.Option("--file", help="Read the input text from a file instead of the argument.")] = None, +) -> None: + """Extract people, organizations, and dates from a piece of text.""" + resolved = read_text_input(text=text, file=file, sample=SAMPLE_ENTITIES_TEXT) + if resolved.is_sample: + progress_console.print(f"[dim]No text given — using the sample: {resolved.text!r}. Pass your own as an argument or via --file.[/dim]") + bundle = (METHODS_DIR / "extract-entities" / "main.mthds").read_text() + main_stuff = _run(execute_pipe(pipe_code="extract_entities", bundle=bundle, inputs={"text": resolved.text})) + # Narrow into the generated typed model (validates the concept's shape), then print it as JSON. + entities = ExtractedEntities.model_validate(main_stuff) + output_console.print_json(data=entities.model_dump()) + + +@app.command(name="summarize-pdf") +def summarize_pdf( + file: Annotated[Path | None, typer.Argument(help="Path to the PDF (or other document) to summarize.")] = None, +) -> None: + """Summarize a document into a title, type, and key points.""" + document = file or SAMPLE_INVOICE + if not document.is_file(): + msg = f"No such file: {document}" + raise typer.BadParameter(msg) + if file is None: + progress_console.print(f"[dim]No file given — using the sample: {document.name}. Pass a path to summarize your own document.[/dim]") + bundle = (METHODS_DIR / "summarize-pdf" / "main.mthds").read_text() + inputs = {"document": build_document_input(document)} + main_stuff = _run(execute_pipe(pipe_code="summarize_pdf", bundle=bundle, inputs=inputs)) + summary = DocumentSummary.model_validate(main_stuff) + output_console.print_json(data=summary.model_dump()) + + +@app.command(name="generate-image") +def generate_image( + prompt: Annotated[str | None, typer.Argument(help="The text prompt describing the image to generate.")] = None, + file: Annotated[Path | None, typer.Option("--file", help="Read the prompt from a file instead of the argument.")] = None, +) -> None: + """Generate an image from a text prompt — expected to hit the hosted ~30s cap. + + Image generation routinely outlives the blocking cap, so this command is here to + fail: it is the teaching moment for `piper attended` / `piper detached`, which run + the very same method durably. + """ + resolved = read_text_input(text=prompt, file=file, sample=SAMPLE_IMAGE_PROMPT) + if resolved.is_sample: + progress_console.print(f"[dim]No prompt given — using the sample: {resolved.text!r}. Pass your own as an argument or via --file.[/dim]") + bundle = (METHODS_DIR / "generate-image" / "main.mthds").read_text() + main_stuff = _run(execute_pipe(pipe_code="generate_image", bundle=bundle, inputs={"image_prompt": resolved.text})) + # On the hosted path the runtime returns a storage `url` (`pipelex-storage://…`) + # *and* a web-renderable `public_url` (a signed URL); the model keeps both. + image = Image.model_validate(main_stuff) + output_console.print_json(data=image.model_dump()) + + +def _run(coro: Coroutine[Any, Any, ResultT]) -> ResultT: + """Await the lifecycle, presenting SDK errors and Ctrl-C as clean exits. + + Every error the SDK client raises descends from `PipelineRequestError`, except the + raw `httpx.HTTPStatusError` its protocol routes surface. Nothing else is caught: + an unexpected exception crashes loudly with its traceback. + """ + try: + return asyncio.run(coro) + except (PipelineRequestError, httpx.HTTPStatusError) as exc: + presentation = present_error(exc) + progress_console.print(f"[red]Error:[/red] {presentation.message}") + if presentation.hint: + progress_console.print(f"\n[yellow]Hint:[/yellow] {presentation.hint}") + raise typer.Exit(1) from exc + except KeyboardInterrupt as exc: + raise typer.Exit(130) from exc + + +if __name__ == "__main__": + app() diff --git a/piper/cli.py b/piper/cli.py index 42f8a4f..796c0b4 100644 --- a/piper/cli.py +++ b/piper/cli.py @@ -1,49 +1,27 @@ -"""The `piper` CLI — demo Pipelex methods behind a Typer app. +"""The `piper` CLI — one binary, three execution modes, one command group each. -Commands stay thin: parse arguments, dispatch on execution mode via -`piper.runner`, narrow + render via the matching `piper.examples` -module. SDK errors are caught once per command (in `_run_cli`) and presented by -`piper.errors`; anything unexpected crashes loudly. -""" +The mode is the command group, not an option: `piper blocking …`, `piper attended …`, +`piper detached …`. Every demo method exists in all three, so picking a mode is picking +a group, and reading a mode means reading exactly one file — `piper//cli.py`, a +self-contained copy-paste unit with no dispatch layer between the command and the SDK +call it makes. -import asyncio -from pathlib import Path -from typing import Annotated, Any, Coroutine, TypeVar +This module is the assembler and nothing else: it loads `.env` and mounts the three +groups in reading order (blocking, then attended, then detached — how you'd learn them). +""" -import httpx import typer from dotenv import load_dotenv -from mthds.protocol.exceptions import PipelineRequestError -from pipelex_sdk.runs import RunResultCompleted, RunResultFailed, RunResultRunning, RunResults, RunResultState -from rich.console import Console -from piper.errors import present_error -from piper.examples import extract_entities as extract_entities_example -from piper.examples import generate_image as generate_image_example -from piper.examples import summarize_pdf as summarize_pdf_example -from piper.file_input import build_document_input -from piper.runner import ( - ExecutionMode, - fetch_run_result, - fetch_run_status, - progress_console, - run_blocking, - run_durable_attended, - start_detached, - wait_for_run, -) - -ResultT = TypeVar("ResultT") +from piper.attended.cli import app as attended_app +from piper.blocking.cli import app as blocking_app +from piper.detached.cli import app as detached_app app = typer.Typer(no_args_is_help=True, help="Run the demo Pipelex methods through the Pipelex API.") -runs_app = typer.Typer(no_args_is_help=True, help="Inspect and resume durable runs by id.") -app.add_typer(runs_app, name="runs") - -# Results go to stdout (pipeable); progress/status chatter goes to stderr (see runner.py). -output_console = Console() -MODE_HELP = "How to execute the run: `durable` (start + poll, survives anything) or `blocking` (single call, ~30s cap on hosted)." -DETACH_HELP = "Start the run and exit immediately; fetch it later with `piper runs ...` (durable mode only)." +app.add_typer(blocking_app, name="blocking", help="One call, one response. Start here — but a run over ~30s is cut off on hosted.") +app.add_typer(attended_app, name="attended", help="Start a durable run and wait here for the result. No cap.") +app.add_typer(detached_app, name="detached", help="Start a durable run and exit; collect it later by run id.") @app.callback() @@ -52,171 +30,5 @@ def main() -> None: load_dotenv() -@app.command(name="extract-entities") -def extract_entities( - text: Annotated[str | None, typer.Argument(help="The text to extract entities from.")] = None, - file: Annotated[Path | None, typer.Option("--file", help="Read the input text from a file instead of the argument.")] = None, - mode: Annotated[ExecutionMode, typer.Option(envvar="PIPELEX_EXECUTION_MODE", help=MODE_HELP)] = ExecutionMode.DURABLE, - detach: Annotated[bool, typer.Option("--detach", help=DETACH_HELP)] = False, -) -> None: - """Extract people, organizations, and dates from a piece of text.""" - input_text = _read_text_input(text=text, file=file) - bundle = extract_entities_example.BUNDLE_PATH.read_text() - results = _dispatch( - pipe_code=extract_entities_example.PIPE_CODE, - bundle=bundle, - inputs={"text": input_text}, - mode=mode, - detach=detach, - ) - if results is None: - return - # Narrow into the typed model (validates the concept's shape), then print it - # as JSON — the same rendering `runs result` / `runs wait` give. - entities = extract_entities_example.parse(results) - output_console.print_json(data=entities.model_dump()) - - -@app.command(name="summarize-pdf") -def summarize_pdf( - file: Annotated[Path, typer.Argument(help="Path to the PDF (or other document) to summarize.")], - mode: Annotated[ExecutionMode, typer.Option(envvar="PIPELEX_EXECUTION_MODE", help=MODE_HELP)] = ExecutionMode.DURABLE, - detach: Annotated[bool, typer.Option("--detach", help=DETACH_HELP)] = False, -) -> None: - """Summarize a document into a title, type, and key points.""" - if not file.is_file(): - msg = f"No such file: {file}" - raise typer.BadParameter(msg) - bundle = summarize_pdf_example.BUNDLE_PATH.read_text() - results = _dispatch( - pipe_code=summarize_pdf_example.PIPE_CODE, - bundle=bundle, - inputs={"document": build_document_input(file)}, - mode=mode, - detach=detach, - ) - if results is None: - return - summary = summarize_pdf_example.parse(results) - output_console.print_json(data=summary.model_dump()) - - -@app.command(name="generate-image") -def generate_image( - prompt: Annotated[str | None, typer.Argument(help="The text prompt describing the image to generate.")] = None, - file: Annotated[Path | None, typer.Option("--file", help="Read the prompt from a file instead of the argument.")] = None, - mode: Annotated[ExecutionMode, typer.Option(envvar="PIPELEX_EXECUTION_MODE", help=MODE_HELP)] = ExecutionMode.DURABLE, - detach: Annotated[bool, typer.Option("--detach", help=DETACH_HELP)] = False, -) -> None: - """Generate an image from a text prompt. - - Image generation routinely outlives the hosted ~30s blocking cap, so this is - the example to run with `--mode blocking` to see a timeout — then `--mode - durable` (the default) to actually get an image. - """ - image_prompt = _read_text_input(text=prompt, file=file) - bundle = generate_image_example.BUNDLE_PATH.read_text() - results = _dispatch( - pipe_code=generate_image_example.PIPE_CODE, - bundle=bundle, - inputs={"image_prompt": image_prompt}, - mode=mode, - detach=detach, - ) - if results is None: - return - image = generate_image_example.parse(results) - output_console.print_json(data=image.model_dump()) - - -@runs_app.command(name="status") -def runs_status(run_id: Annotated[str, typer.Argument(help="The pipeline run id printed when the run started.")]) -> None: - """Show a run's coarse status without waiting.""" - run = _run_cli(fetch_run_status(run_id)) - pipe_part = f" (pipe: {run.pipe_code})" if run.pipe_code else "" - output_console.print(f"{run.pipeline_run_id}: [bold]{run.status}[/bold]{pipe_part}") - if run.degraded: - output_console.print("[yellow]Status is degraded — last-known value, the status backend was unreachable; retry shortly.[/yellow]") - - -@runs_app.command(name="result") -def runs_result(run_id: Annotated[str, typer.Argument(help="The pipeline run id printed when the run started.")]) -> None: - """Fetch a run's result if it is finished (no waiting).""" - state = _run_cli(fetch_run_result(run_id)) - _render_result_state(state) - - -@runs_app.command(name="wait") -def runs_wait(run_id: Annotated[str, typer.Argument(help="The pipeline run id printed when the run started.")]) -> None: - """Poll a run to completion, then print its raw result.""" - results = _run_cli(wait_for_run(run_id)) - _print_raw_results(results) - - -def _read_text_input(*, text: str | None, file: Path | None) -> str: - if text is not None and file is not None: - msg = "Give the text either as an argument or via --file, not both." - raise typer.BadParameter(msg) - if file is not None: - return file.read_text() - if text is not None: - return text - msg = "Give the text to process as an argument, or point --file at a text file." - raise typer.BadParameter(msg) - - -def _dispatch(*, pipe_code: str, bundle: str, inputs: dict[str, Any], mode: ExecutionMode, detach: bool) -> RunResults | None: - """Run the pipe in the requested mode; returns None when detached (id already printed).""" - if detach: - match mode: - case ExecutionMode.BLOCKING: - msg = "--detach starts a durable run; it cannot be combined with --mode blocking." - raise typer.BadParameter(msg) - case ExecutionMode.DURABLE: - pass - run_id = _run_cli(start_detached(pipe_code=pipe_code, bundle=bundle, inputs=inputs)) - print(run_id) - progress_console.print(f"Run started — fetch it later with: [bold]piper runs wait {run_id}[/bold]") - return None - match mode: - case ExecutionMode.BLOCKING: - return _run_cli(run_blocking(pipe_code=pipe_code, bundle=bundle, inputs=inputs)) - case ExecutionMode.DURABLE: - return _run_cli(run_durable_attended(pipe_code=pipe_code, bundle=bundle, inputs=inputs)) - - -def _run_cli(coro: Coroutine[Any, Any, ResultT]) -> ResultT: - """Await a runner coroutine, presenting SDK errors and Ctrl-C as clean exits.""" - try: - return asyncio.run(coro) - except (PipelineRequestError, httpx.HTTPStatusError) as exc: - presentation = present_error(exc) - progress_console.print(f"[red]Error:[/red] {presentation.message}") - if presentation.hint: - progress_console.print(f"[yellow]Hint:[/yellow] {presentation.hint}") - raise typer.Exit(1) from exc - except KeyboardInterrupt as exc: - # The resume hint was already printed by the runner; the run keeps executing server-side. - raise typer.Exit(130) from exc - - -def _render_result_state(state: RunResultState) -> None: - match state: - case RunResultRunning(): - progress_console.print( - f"Run {state.pipeline_run_id} is still running — wait for it with: [bold]piper runs wait {state.pipeline_run_id}[/bold]" - ) - case RunResultCompleted(): - _print_raw_results(state.result) - case RunResultFailed(): - progress_console.print(f"[red]Run {state.pipeline_run_id} ended with status {state.status}: {state.message}[/red]") - raise typer.Exit(1) - - -def _print_raw_results(results: RunResults) -> None: - """Print the run's main content as JSON — generic, no per-example narrowing.""" - output_console.print_json(data=results.main_stuff) - - if __name__ == "__main__": app() diff --git a/piper/detached/__init__.py b/piper/detached/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/piper/detached/cli.py b/piper/detached/cli.py new file mode 100644 index 0000000..78d08a9 --- /dev/null +++ b/piper/detached/cli.py @@ -0,0 +1,200 @@ +"""The detached CLI — start a durable run and walk away; collect it later. + +`client.start(...)` submits the run server-side and this command exits with the run +id on stdout, so it pipes: `RUN_ID=$(piper detached generate-image "a fox")`. All the +run's state lives behind that id, so you pick it back up whenever you like — even +from another terminal, another machine, another day: + +- `piper detached wait ` — poll it to completion and print its result. +- `piper detached status ` — where is it right now, without waiting. +- `piper detached result ` — its result if it is done, without waiting. + +Same durable run as `piper attended`; the only difference is who waits. + +Copy-paste unit: this file + `piper/inputs.py` + `piper/errors.py`. The three mode +packages never share lifecycle code, so the demo commands below are deliberate +mirrors of their `blocking` / `attended` twins: only `start_pipe()` differs — plus +the run-lifecycle commands, which exist only here. +""" + +import asyncio +from pathlib import Path +from typing import Annotated, Any, Coroutine, TypeVar + +import httpx +import typer +from mthds.protocol.exceptions import PipelineRequestError +from pipelex_sdk.client import PipelexAPIClient +from pipelex_sdk.runs import PollInfo, RunRead, RunResultCompleted, RunResultFailed, RunResultRunning, RunResultState, WaitForResultOptions +from rich.console import Console + +from piper.errors import present_error +from piper.inputs import SAMPLE_ENTITIES_TEXT, SAMPLE_IMAGE_PROMPT, SAMPLE_INVOICE, build_document_input, read_text_input + +ResultT = TypeVar("ResultT") + +METHODS_DIR = Path(__file__).parent.parent / "methods" + +app = typer.Typer(no_args_is_help=True, help="Start a demo as a durable run and exit; collect the result later.") + +# Results go to stdout (pipeable); progress chatter goes to stderr. +output_console = Console() +progress_console = Console(stderr=True) + + +async def start_pipe(*, pipe_code: str, bundle: str, inputs: dict[str, Any]) -> str: + """The whole detached lifecycle: start a durable run, return its id, don't wait. + + Credentials come from `PIPELEX_API_KEY` / `PIPELEX_BASE_URL`. The run keeps + executing server-side after this process exits. + """ + async with PipelexAPIClient() as client: + start_result = await client.start(pipe_code=pipe_code, mthds_contents=[bundle], inputs=inputs) + return start_result.pipeline_run_id + + +async def attend_run(run_id: str) -> Any: + """Poll an already-started run to completion and resolve its main output.""" + async with PipelexAPIClient() as client: + short_id = run_id[:8] + with progress_console.status(f"Run {short_id}… in progress") as status: + + def on_poll(info: PollInfo) -> None: + status.update(f"Run {short_id}… in progress — {info.elapsed_seconds:.0f}s, poll #{info.attempt}") + + try: + results = await client.wait_for_result(run_id, options=WaitForResultOptions(on_poll=on_poll)) + except asyncio.CancelledError: + # Ctrl-C: the run keeps executing server-side — nothing is lost, just wait on it again. + progress_console.print(f"\nInterrupted — the run is still executing. Resume with: [bold]piper detached wait {run_id}[/bold]") + raise + return results.main_stuff + + +async def fetch_run_status(run_id: str) -> RunRead: + """Fetch a run's coarse status (`GET /v1/runs/{id}/status`) — no polling.""" + async with PipelexAPIClient() as client: + return await client.get_run_status(run_id) + + +async def fetch_run_result(run_id: str) -> RunResultState: + """Fetch a run's result state (`GET /v1/runs/{id}/results`) — no polling.""" + async with PipelexAPIClient() as client: + return await client.get_run_result(run_id) + + +@app.command(name="extract-entities") +def extract_entities( + text: Annotated[str | None, typer.Argument(help="The text to extract entities from.")] = None, + file: Annotated[Path | None, typer.Option("--file", help="Read the input text from a file instead of the argument.")] = None, +) -> None: + """Start extracting people, organizations, and dates from a piece of text.""" + resolved = read_text_input(text=text, file=file, sample=SAMPLE_ENTITIES_TEXT) + if resolved.is_sample: + progress_console.print(f"[dim]No text given — using the sample: {resolved.text!r}. Pass your own as an argument or via --file.[/dim]") + bundle = (METHODS_DIR / "extract-entities" / "main.mthds").read_text() + run_id = _run(start_pipe(pipe_code="extract_entities", bundle=bundle, inputs={"text": resolved.text})) + _print_run_id(run_id) + + +@app.command(name="summarize-pdf") +def summarize_pdf( + file: Annotated[Path | None, typer.Argument(help="Path to the PDF (or other document) to summarize.")] = None, +) -> None: + """Start summarizing a document into a title, type, and key points.""" + document = file or SAMPLE_INVOICE + if not document.is_file(): + msg = f"No such file: {document}" + raise typer.BadParameter(msg) + if file is None: + progress_console.print(f"[dim]No file given — using the sample: {document.name}. Pass a path to summarize your own document.[/dim]") + bundle = (METHODS_DIR / "summarize-pdf" / "main.mthds").read_text() + inputs = {"document": build_document_input(document)} + run_id = _run(start_pipe(pipe_code="summarize_pdf", bundle=bundle, inputs=inputs)) + _print_run_id(run_id) + + +@app.command(name="generate-image") +def generate_image( + prompt: Annotated[str | None, typer.Argument(help="The text prompt describing the image to generate.")] = None, + file: Annotated[Path | None, typer.Option("--file", help="Read the prompt from a file instead of the argument.")] = None, +) -> None: + """Start generating an image from a text prompt. + + The demo that makes detached mode obvious: image generation is slow enough that + you would rather not hold a terminal open for it. + """ + resolved = read_text_input(text=prompt, file=file, sample=SAMPLE_IMAGE_PROMPT) + if resolved.is_sample: + progress_console.print(f"[dim]No prompt given — using the sample: {resolved.text!r}. Pass your own as an argument or via --file.[/dim]") + bundle = (METHODS_DIR / "generate-image" / "main.mthds").read_text() + run_id = _run(start_pipe(pipe_code="generate_image", bundle=bundle, inputs={"image_prompt": resolved.text})) + _print_run_id(run_id) + + +@app.command(name="wait") +def wait(run_id: Annotated[str, typer.Argument(help="The pipeline run id printed when the run started.")]) -> None: + """Poll a run to completion, then print its result.""" + main_stuff = _run(attend_run(run_id)) + _print_main_stuff(main_stuff) + + +@app.command(name="status") +def status(run_id: Annotated[str, typer.Argument(help="The pipeline run id printed when the run started.")]) -> None: + """Show a run's coarse status without waiting.""" + run = _run(fetch_run_status(run_id)) + pipe_part = f" (pipe: {run.pipe_code})" if run.pipe_code else "" + output_console.print(f"{run.pipeline_run_id}: [bold]{run.status}[/bold]{pipe_part}") + if run.degraded: + output_console.print("[yellow]Status is degraded — last-known value, the status backend was unreachable; retry shortly.[/yellow]") + + +@app.command(name="result") +def result(run_id: Annotated[str, typer.Argument(help="The pipeline run id printed when the run started.")]) -> None: + """Fetch a run's result if it is finished (no waiting).""" + state = _run(fetch_run_result(run_id)) + match state: + case RunResultRunning(): + progress_console.print( + f"Run {state.pipeline_run_id} is still running — wait for it with: [bold]piper detached wait {state.pipeline_run_id}[/bold]" + ) + case RunResultCompleted(): + _print_main_stuff(state.result.main_stuff) + case RunResultFailed(): + progress_console.print(f"[red]Run {state.pipeline_run_id} ended with status {state.status}: {state.message}[/red]") + raise typer.Exit(1) + + +def _print_run_id(run_id: str) -> None: + """The id on stdout (bare, so `$(piper detached …)` captures it), the hint on stderr.""" + print(run_id) + progress_console.print(f"Run started — fetch it later with: [bold]piper detached wait {run_id}[/bold]") + + +def _print_main_stuff(main_stuff: Any) -> None: + """Print a run's main output as raw JSON — generic, since any run id can land here.""" + output_console.print_json(data=main_stuff) + + +def _run(coro: Coroutine[Any, Any, ResultT]) -> ResultT: + """Await the lifecycle, presenting SDK errors and Ctrl-C as clean exits. + + Every error the SDK client raises descends from `PipelineRequestError`, except the + raw `httpx.HTTPStatusError` its protocol routes surface. Nothing else is caught: + an unexpected exception crashes loudly with its traceback. + """ + try: + return asyncio.run(coro) + except (PipelineRequestError, httpx.HTTPStatusError) as exc: + presentation = present_error(exc) + progress_console.print(f"[red]Error:[/red] {presentation.message}") + if presentation.hint: + progress_console.print(f"\n[yellow]Hint:[/yellow] {presentation.hint}") + raise typer.Exit(1) from exc + except KeyboardInterrupt as exc: + # The resume hint was already printed by `attend_run`; the run keeps executing server-side. + raise typer.Exit(130) from exc + + +if __name__ == "__main__": + app() diff --git a/piper/errors.py b/piper/errors.py index 3306ca9..4952888 100644 --- a/piper/errors.py +++ b/piper/errors.py @@ -1,13 +1,19 @@ """Present SDK errors as actionable CLI messages. -This module defines no exception classes — it is a presentation mapper. Each -CLI command catches `PipelineRequestError` (the base of every error the -`pipelex-sdk` client raises) exactly once at its root, turns it into a -`(message, hint)` pair here, and exits non-zero. Unexpected exceptions are -deliberately NOT caught anywhere: they crash loudly with a full traceback. +This module defines no exception classes — it is a presentation mapper. Each mode +package's `_run()` wrapper (`piper/blocking/cli.py`, `piper/attended/cli.py`, +`piper/detached/cli.py`) catches `PipelineRequestError` (the base of every error the +`pipelex-sdk` client raises) exactly once, turns it into a `(message, hint)` pair here, +and exits non-zero. Unexpected exceptions are deliberately NOT caught anywhere: they +crash loudly with a full traceback. + +Error presentation is orthogonal to execution mode, so it is shared — but the hints +name the mode *groups*, since the fix for a timed-out blocking run is to rerun it under +another group. """ -from typing import NamedTuple +import json +from typing import Any, NamedTuple, cast import httpx from mthds.protocol.exceptions import PipelineRequestError @@ -41,12 +47,12 @@ def present_error(exc: PipelineRequestError | httpx.HTTPStatusError) -> ErrorPre if isinstance(exc, PipelineExecuteTimeoutError): return ErrorPresentation( message=f"The blocking run exceeded the hosted gateway's ~30s synchronous cap ({exc.elapsed_seconds:.0f}s elapsed).", - hint="Retry with `--mode durable` — the durable path survives long runs.", + hint="Rerun the same command with `piper attended ...` — the durable path survives long runs.", ) if isinstance(exc, RunLifecycleUnavailableError): return ErrorPresentation( message=f"The server at {exc.api_url} has no run store (durable run lifecycle unavailable).", - hint="You are talking to a bare runner — retry with `--mode blocking`.", + hint="You are talking to a bare runner — use `piper blocking ...`.", ) if isinstance(exc, ApiResponseError): if exc.status in (401, 403): @@ -66,21 +72,53 @@ def present_error(exc: PipelineRequestError | httpx.HTTPStatusError) -> ErrorPre if isinstance(exc, RunFailedError): return ErrorPresentation( message=f"Run {exc.run_id} ended with status {exc.status}: {exc}", - hint=f"Inspect it with `piper runs status {exc.run_id}`.", + hint=f"Inspect it with `piper detached status {exc.run_id}`.", ) if isinstance(exc, RunTimeoutError): return ErrorPresentation( message=f"Gave up waiting for run {exc.run_id} after {exc.timeout_seconds:.0f}s — the run is still executing server-side.", - hint=f"Resume waiting with `piper runs wait {exc.run_id}`.", + hint=f"Resume waiting with `piper detached wait {exc.run_id}`.", ) return ErrorPresentation(message=str(exc), hint=None) def _present_http_status_error(exc: httpx.HTTPStatusError) -> ErrorPresentation: + """Present a raw protocol-route HTTP error, reading its RFC 7807 problem+json body. + + The protocol routes (`execute`, `start`, `runs/*`) return errors as + `application/problem+json`: a human `detail`/`title` and a machine `error_type`. + httpx's own stringification throws all of that away (`Client error '400 Bad + Request' for url …` plus an MDN link), so we read the body and surface what the + server actually said — and branch on the structured `error_type`, never the + transport status, for the cases worth a hint. + """ status_code = exc.response.status_code + problem = _read_problem_json(exc.response) if status_code in (401, 403): return ErrorPresentation( message=f"The API rejected the request ({status_code} {exc.response.reason_phrase}).", hint="Set PIPELEX_API_KEY in your environment or .env file — get a key at https://app.pipelex.com", ) - return ErrorPresentation(message=str(exc), hint=None) + # A durable-run endpoint (`/start`) this deployment can't serve: it runs a + # synchronous-only orchestration, so only `piper blocking` (`/execute`) works here. + if problem.get("error_type") == "StartRequiresAsyncOrchestration": + return ErrorPresentation( + message=problem.get("detail") or "This deployment cannot start durable runs — it has no async orchestration.", + hint="This runner only does synchronous runs — use `piper blocking ...` instead.", + ) + detail = problem.get("detail") or problem.get("title") + if detail: + return ErrorPresentation(message=f"The API answered {status_code} {exc.response.reason_phrase}: {detail}", hint=None) + return ErrorPresentation(message=f"The API answered {status_code} {exc.response.reason_phrase}.", hint=None) + + +def _read_problem_json(response: httpx.Response) -> dict[str, Any]: + """Best-effort parse of an RFC 7807 problem+json body; `{}` when it isn't JSON.""" + try: + body: Any = response.json() + except json.JSONDecodeError: + return {} + if isinstance(body, dict): + # JSON object keys are always strings. + return cast("dict[str, Any]", body) + return {} diff --git a/piper/examples/extract_entities.py b/piper/examples/extract_entities.py deleted file mode 100644 index 3dee53b..0000000 --- a/piper/examples/extract_entities.py +++ /dev/null @@ -1,35 +0,0 @@ -"""Example: extract people, organizations, and dates from a piece of text. - -This module is the "copy me" unit for swapping in your own pipeline: a bundle -path, a Pydantic model mirroring the output concept, and a `parse` narrower -that turns the loose run result into that model. The CLI prints the parsed -model as JSON — same rendering as `runs result` / `runs wait`. -""" - -from pathlib import Path - -from pipelex_sdk.runs import RunResults -from pydantic import BaseModel - -BUNDLE_PATH = Path(__file__).parent.parent / "methods" / "extract-entities" / "main.mthds" -PIPE_CODE = "extract_entities" - - -class ExtractedEntities(BaseModel): - """Mirror of the bundle's `ExtractedEntities` output concept.""" - - people: list[str] - orgs: list[str] - dates: list[str] - - -def parse(results: RunResults) -> ExtractedEntities: - """Narrow a run result into a typed `ExtractedEntities`. - - The SDK guarantees a resolved `results.main_stuff` for a completed run (it raises - `MissingMainStuffError` upstream otherwise), so this only validates the shape. - - Raises: - pydantic.ValidationError: The content doesn't match the concept's shape. - """ - return ExtractedEntities.model_validate(results.main_stuff) diff --git a/piper/examples/generate_image.py b/piper/examples/generate_image.py deleted file mode 100644 index 83f6800..0000000 --- a/piper/examples/generate_image.py +++ /dev/null @@ -1,41 +0,0 @@ -"""Example: generate an image from a text prompt. - -The third "copy me" unit — and the one that best demonstrates the durable vs -blocking split. Image generation routinely outlives the hosted gateway's ~30s -synchronous cap, so running it with `--mode blocking` is how you actually see a -`PipelineExecuteTimeoutError`; `--mode durable` (the default) survives it. - -The output concept is the built-in `Image`, so `main_stuff` is an image content -dict — a `url` plus optional `public_url` / `mime_type` / `caption`. On the -hosted path the runtime returns a storage `url` (`pipelex-storage://…`) *and* a -web-renderable `public_url` (a signed URL); `parse` keeps both. -""" - -from pathlib import Path - -from pipelex_sdk.runs import RunResults -from pydantic import BaseModel - -BUNDLE_PATH = Path(__file__).parent.parent / "methods" / "generate-image" / "main.mthds" -PIPE_CODE = "generate_image" - - -class GeneratedImage(BaseModel): - """Mirror of the bundle's built-in `Image` output content.""" - - url: str - public_url: str | None = None - mime_type: str | None = None - caption: str | None = None - - -def parse(results: RunResults) -> GeneratedImage: - """Narrow a run result into a typed `GeneratedImage`. - - The SDK guarantees a resolved `results.main_stuff` for a completed run (it raises - `MissingMainStuffError` upstream otherwise), so this only validates the shape. - - Raises: - pydantic.ValidationError: The content carries no image `url`. - """ - return GeneratedImage.model_validate(results.main_stuff) diff --git a/piper/examples/summarize_pdf.py b/piper/examples/summarize_pdf.py deleted file mode 100644 index 6e60526..0000000 --- a/piper/examples/summarize_pdf.py +++ /dev/null @@ -1,36 +0,0 @@ -"""Example: summarize a PDF document into a title, type, and key points. - -Like `extract_entities`, this is a "copy me" unit: a bundle path, a Pydantic -model mirroring the output concept, and a `parse` narrower. What it adds is a -*file* input — the CLI passes a `Document` envelope built by -`piper.file_input.build_document_input`, so this is the example that shows -how to feed a PDF (or any document) to a pipe. -""" - -from pathlib import Path - -from pipelex_sdk.runs import RunResults -from pydantic import BaseModel - -BUNDLE_PATH = Path(__file__).parent.parent / "methods" / "summarize-pdf" / "main.mthds" -PIPE_CODE = "summarize_pdf" - - -class DocumentSummary(BaseModel): - """Mirror of the bundle's `DocumentSummary` output concept.""" - - title: str - doc_type: str - key_points: list[str] - - -def parse(results: RunResults) -> DocumentSummary: - """Narrow a run result into a typed `DocumentSummary`. - - The SDK guarantees a resolved `results.main_stuff` for a completed run (it raises - `MissingMainStuffError` upstream otherwise), so this only validates the shape. - - Raises: - pydantic.ValidationError: The content doesn't match the concept's shape. - """ - return DocumentSummary.model_validate(results.main_stuff) diff --git a/piper/file_input.py b/piper/file_input.py deleted file mode 100644 index 3504ae4..0000000 --- a/piper/file_input.py +++ /dev/null @@ -1,34 +0,0 @@ -"""Turn a local file into a Pipelex `Document` input envelope. - -The hosted API accepts a file input as `{"concept": "Document", "content": -{"url": ..., "filename": ..., "mime_type": ...}}`, where `url` may be a base64 -`data:` URL. The API decodes it server-side and uploads it to storage, so the -CLI never has to host the file itself. This mirrors `buildDocumentInput` in the -JS starter's `fileEncoding.ts`. -""" - -import base64 -import mimetypes -from pathlib import Path -from typing import Any - -DEFAULT_MIME_TYPE = "application/octet-stream" - - -def build_document_input(path: Path) -> dict[str, Any]: - """Read a file from disk and build its `Document` input envelope. - - The bytes are base64-encoded into a `data:` URL; the MIME type is guessed - from the extension (falling back to `application/octet-stream`). - - Raises: - FileNotFoundError: `path` does not point at a readable file. - """ - data = path.read_bytes() - mime_type = mimetypes.guess_type(path.name)[0] or DEFAULT_MIME_TYPE - encoded = base64.b64encode(data).decode("ascii") - data_url = f"data:{mime_type};base64,{encoded}" - return { - "concept": "Document", - "content": {"url": data_url, "filename": path.name, "mime_type": mime_type}, - } diff --git a/piper/generated/__init__.py b/piper/generated/__init__.py new file mode 100644 index 0000000..1cd31dd --- /dev/null +++ b/piper/generated/__init__.py @@ -0,0 +1,5 @@ +"""Typed clients generated from the `.mthds` methods by `pipelex codegen` — one subpackage per method. + +Do not edit the `models.py` modules by hand: regenerate with `make codegen`, verify with +`make codegen-check` (offline). Each subpackage's `codegen.lock` records the generated artifact set. +""" diff --git a/piper/generated/extract_entities/__init__.py b/piper/generated/extract_entities/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/piper/generated/extract_entities/codegen.lock b/piper/generated/extract_entities/codegen.lock new file mode 100644 index 0000000..dbd6e39 --- /dev/null +++ b/piper/generated/extract_entities/codegen.lock @@ -0,0 +1,8 @@ +# codegen.lock — generated artifact set (Pipelex codegen). Do not edit by hand. + +crate_fingerprint = "0849d91f3818dd7b5602fb9fd0722fe96b384fcf19633d7c9827f07289cab606" +engine_version = "0.38.0" + +[[artifacts]] +path = "models.py" +content_hash = "fff0fe9784be8eb27057f450568d1ecf5869c83fea55e0302780ca44d04a3f86" diff --git a/piper/generated/extract_entities/models.py b/piper/generated/extract_entities/models.py new file mode 100644 index 0000000..7e9e106 --- /dev/null +++ b/piper/generated/extract_entities/models.py @@ -0,0 +1,41 @@ +# >>> pipelex-codegen-stamp >>> +# crate_fingerprint: 0849d91f3818dd7b5602fb9fd0722fe96b384fcf19633d7c9827f07289cab606 +# engine_version: 0.38.0 +# projection: types / python-pydantic +# options: {} +# content_hash: fff0fe9784be8eb27057f450568d1ecf5869c83fea55e0302780ca44d04a3f86 +# <<< pipelex-codegen-stamp <<< +# --------------------------------------------------------------------------- +# AUTOGENERATED by Pipelex codegen — DO NOT EDIT. +# +# This file is a projection of a normalized MTHDS library crate. It is +# regenerated from the method; any hand edit here is overwritten. +# +# To customize a generated type, do NOT edit this file. Create a sibling +# module and subclass — subclasses survive regeneration: +# +# # my_types_ext.py +# from .structures import Report +# +# class MyReport(Report): +# ... +# +# projection: types / python-pydantic +# --------------------------------------------------------------------------- +from __future__ import annotations + +from pydantic import BaseModel, Field + + +class ExtractedEntities(BaseModel): + """Named entities extracted from a piece of text.""" + + people: list[str] = Field(..., description="Names of people mentioned in the text") + orgs: list[str] = Field(..., description="Names of organizations mentioned in the text") + dates: list[str] = Field(..., description="Dates or time references mentioned in the text") + + +class Text(BaseModel): + """A text""" + + text: str = Field(..., description="The text") diff --git a/piper/generated/generate_image/__init__.py b/piper/generated/generate_image/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/piper/generated/generate_image/codegen.lock b/piper/generated/generate_image/codegen.lock new file mode 100644 index 0000000..b38d14c --- /dev/null +++ b/piper/generated/generate_image/codegen.lock @@ -0,0 +1,8 @@ +# codegen.lock — generated artifact set (Pipelex codegen). Do not edit by hand. + +crate_fingerprint = "0c688d0aae772f9b0d2e1d923da0b3e26ef53d7eff93672aa4f519a6f2dce6fd" +engine_version = "0.38.0" + +[[artifacts]] +path = "models.py" +content_hash = "4e4806071d622b2a017a0149c6d09c9e2c6e94f6ab380d2247e3c6d65702c102" diff --git a/piper/generated/generate_image/models.py b/piper/generated/generate_image/models.py new file mode 100644 index 0000000..9f57dcb --- /dev/null +++ b/piper/generated/generate_image/models.py @@ -0,0 +1,47 @@ +# >>> pipelex-codegen-stamp >>> +# crate_fingerprint: 0c688d0aae772f9b0d2e1d923da0b3e26ef53d7eff93672aa4f519a6f2dce6fd +# engine_version: 0.38.0 +# projection: types / python-pydantic +# options: {} +# content_hash: 4e4806071d622b2a017a0149c6d09c9e2c6e94f6ab380d2247e3c6d65702c102 +# <<< pipelex-codegen-stamp <<< +# --------------------------------------------------------------------------- +# AUTOGENERATED by Pipelex codegen — DO NOT EDIT. +# +# This file is a projection of a normalized MTHDS library crate. It is +# regenerated from the method; any hand edit here is overwritten. +# +# To customize a generated type, do NOT edit this file. Create a sibling +# module and subclass — subclasses survive regeneration: +# +# # my_types_ext.py +# from .structures import Report +# +# class MyReport(Report): +# ... +# +# projection: types / python-pydantic +# --------------------------------------------------------------------------- +from __future__ import annotations + +from pydantic import BaseModel, Field + + +class Image(BaseModel): + """An image""" + + url: str = Field(..., description="The image URL: a storage URI, an HTTP(S) URL, or a base64 data URL") + public_url: str | None = Field(default=None, description="The public URL of the image") + source_prompt: str | None = Field(default=None, description="The source prompt of the image") + source_negative_prompt: str | None = Field(default=None, description="The source negative prompt of the image") + caption: str | None = Field(default=None, description="The caption of the image") + mime_type: str | None = Field(default=None, description="The MIME type of the image") + width: int | None = Field(default=None, description="The width of the image, in pixels") + height: int | None = Field(default=None, description="The height of the image, in pixels") + filename: str | None = Field(default=None, description="The original filename of the image") + + +class Text(BaseModel): + """A text""" + + text: str = Field(..., description="The text") diff --git a/piper/generated/summarize_pdf/__init__.py b/piper/generated/summarize_pdf/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/piper/generated/summarize_pdf/codegen.lock b/piper/generated/summarize_pdf/codegen.lock new file mode 100644 index 0000000..8d27e43 --- /dev/null +++ b/piper/generated/summarize_pdf/codegen.lock @@ -0,0 +1,8 @@ +# codegen.lock — generated artifact set (Pipelex codegen). Do not edit by hand. + +crate_fingerprint = "44005fc6f043ba3d1037bb3707002235c18ae281a8e669634317a77535eb3c5b" +engine_version = "0.38.0" + +[[artifacts]] +path = "models.py" +content_hash = "994b227bd2cf44d75f672130902603014afa74e86c519e923bf9a3cc510fabc1" diff --git a/piper/generated/summarize_pdf/models.py b/piper/generated/summarize_pdf/models.py new file mode 100644 index 0000000..7fe5579 --- /dev/null +++ b/piper/generated/summarize_pdf/models.py @@ -0,0 +1,46 @@ +# >>> pipelex-codegen-stamp >>> +# crate_fingerprint: 44005fc6f043ba3d1037bb3707002235c18ae281a8e669634317a77535eb3c5b +# engine_version: 0.38.0 +# projection: types / python-pydantic +# options: {} +# content_hash: 994b227bd2cf44d75f672130902603014afa74e86c519e923bf9a3cc510fabc1 +# <<< pipelex-codegen-stamp <<< +# --------------------------------------------------------------------------- +# AUTOGENERATED by Pipelex codegen — DO NOT EDIT. +# +# This file is a projection of a normalized MTHDS library crate. It is +# regenerated from the method; any hand edit here is overwritten. +# +# To customize a generated type, do NOT edit this file. Create a sibling +# module and subclass — subclasses survive regeneration: +# +# # my_types_ext.py +# from .structures import Report +# +# class MyReport(Report): +# ... +# +# projection: types / python-pydantic +# --------------------------------------------------------------------------- +from __future__ import annotations + +from pydantic import BaseModel, Field + + +class Document(BaseModel): + """A document""" + + url: str = Field(..., description="The document URL: a storage URI, an HTTP(S) URL, or a base64 data URL") + public_url: str | None = Field(default=None, description="The public HTTPS URL of the document") + mime_type: str | None = Field(default=None, description="The MIME type of the document") + filename: str | None = Field(default=None, description="The original filename of the document") + title: str | None = Field(default=None, description="The title of the document or source") + snippet: str | None = Field(default=None, description="A text snippet or excerpt from the document") + + +class DocumentSummary(BaseModel): + """A structured summary of a document""" + + title: str = Field(..., description="A concise title for the document") + doc_type: str = Field(..., description="The kind of document, such as invoice, report, article, or contract") + key_points: list[str] = Field(..., description="The main takeaways from the document, one per item") diff --git a/piper/inputs.py b/piper/inputs.py new file mode 100644 index 0000000..9e0c1bf --- /dev/null +++ b/piper/inputs.py @@ -0,0 +1,82 @@ +"""Build the inputs a pipe run is given — shared by every execution mode. + +Input encoding is orthogonal to *how* a run is executed, so it lives here rather +than in one of the mode packages (`piper/blocking`, `piper/attended`, +`piper/detached`), which never share lifecycle code with each other. + +Two shapes cover the demos: + +- `read_text_input()` — a text argument, a `--file` pointing at one, or (when you + give neither) a built-in sample, so every demo runs with zero arguments. +- `build_document_input()` — a local file, base64-encoded into a `data:` URL and + wrapped in the `Document` envelope the API expects: `{"concept": "Document", + "content": {"url": ..., "filename": ..., "mime_type": ...}}`. The API decodes + it server-side and uploads it to storage, so the CLI never has to host the file + itself. This mirrors `buildDocumentInput` in the JS starter's `fileEncoding.ts`. + +The built-in samples let a fresh clone show a working result before you have any +input of your own; they are also the values the README documents. +""" + +import base64 +import mimetypes +from pathlib import Path +from typing import Any, NamedTuple + +import typer + +DEFAULT_MIME_TYPE = "application/octet-stream" + +# Sample inputs — the zero-argument fallback for each demo (also the README's examples). +SAMPLE_ENTITIES_TEXT = "Alice from Acme met Bob on May 3rd, 2026." +SAMPLE_IMAGE_PROMPT = "a fox reading under a tree" +SAMPLE_INVOICE = Path(__file__).parent.parent / "samples" / "sample-invoice.pdf" + + +class TextInput(NamedTuple): + """A resolved text input, plus whether it came from the built-in sample. + + The `is_sample` flag lets a command tell the user it filled in the input for + them (so the result isn't mysterious) without the input helper owning a console. + """ + + text: str + is_sample: bool + + +def read_text_input(*, text: str | None, file: Path | None, sample: str) -> TextInput: + """Resolve the text to process: an argument, a `--file`, or the built-in `sample`. + + An explicit argument or `--file` wins; with neither, `sample` is used so the + command runs with no arguments at all. + + Raises: + typer.BadParameter: both an argument and `--file` were given. + """ + if text is not None and file is not None: + msg = "Give the text either as an argument or via --file, not both." + raise typer.BadParameter(msg) + if file is not None: + return TextInput(text=file.read_text(), is_sample=False) + if text is not None: + return TextInput(text=text, is_sample=False) + return TextInput(text=sample, is_sample=True) + + +def build_document_input(path: Path) -> dict[str, Any]: + """Read a file from disk and build its `Document` input envelope. + + The bytes are base64-encoded into a `data:` URL; the MIME type is guessed + from the extension (falling back to `application/octet-stream`). + + Raises: + FileNotFoundError: `path` does not point at a readable file. + """ + data = path.read_bytes() + mime_type = mimetypes.guess_type(path.name)[0] or DEFAULT_MIME_TYPE + encoded = base64.b64encode(data).decode("ascii") + data_url = f"data:{mime_type};base64,{encoded}" + return { + "concept": "Document", + "content": {"url": data_url, "filename": path.name, "mime_type": mime_type}, + } diff --git a/piper/methods/extract-entities/inputs.template.json b/piper/methods/extract-entities/inputs.template.json new file mode 100644 index 0000000..dccb791 --- /dev/null +++ b/piper/methods/extract-entities/inputs.template.json @@ -0,0 +1,3 @@ +{ + "text": "text_value" +} \ No newline at end of file diff --git a/piper/methods/generate-image/inputs.template.json b/piper/methods/generate-image/inputs.template.json new file mode 100644 index 0000000..985e885 --- /dev/null +++ b/piper/methods/generate-image/inputs.template.json @@ -0,0 +1,3 @@ +{ + "image_prompt": "text_value" +} \ No newline at end of file diff --git a/piper/methods/summarize-pdf/inputs.template.json b/piper/methods/summarize-pdf/inputs.template.json new file mode 100644 index 0000000..1c4e558 --- /dev/null +++ b/piper/methods/summarize-pdf/inputs.template.json @@ -0,0 +1,3 @@ +{ + "document": "https://mock.invalid/url" +} \ No newline at end of file diff --git a/piper/runner.py b/piper/runner.py deleted file mode 100644 index d55d3eb..0000000 --- a/piper/runner.py +++ /dev/null @@ -1,105 +0,0 @@ -"""Execution-mode dispatch for the CLI: blocking, durable attended, durable detached. - -Each function opens its own `PipelexAPIClient` (credentials come from -`PIPELEX_API_KEY` / `PIPELEX_BASE_URL`), demonstrates exactly one SDK lifecycle -call, and renders progress with Rich on stderr — stdout stays clean for results. - -- blocking -> `client.execute` (one call, dies at the hosted ~30s cap) -- durable attended -> `client.start` + `client.wait_for_result` (survives anything) -- durable detached -> `client.start` only (come back later with `runs ...`) - -The SDK also offers `start_and_wait`, a self-healing one-liner that picks the -right path by itself — this starter branches explicitly because teaching the -difference is the point. -""" - -import asyncio -from enum import Enum -from typing import Any - -from pipelex_sdk.client import PipelexAPIClient -from pipelex_sdk.runs import PollInfo, RunRead, RunResults, RunResultState, WaitForResultOptions -from rich.console import Console - -# Progress and lifecycle chatter go to stderr so stdout stays pipeable. -progress_console = Console(stderr=True) - - -class ExecutionMode(str, Enum): - """How a run is executed against the API — see the module docstring.""" - - BLOCKING = "blocking" - DURABLE = "durable" - - -async def run_blocking(*, pipe_code: str, bundle: str, inputs: dict[str, Any] | None = None) -> RunResults: - """Execute synchronously (`POST /v1/execute`) and wait for the response. - - Simple, but behind the hosted gateway a run longer than ~30s raises - `PipelineExecuteTimeoutError` — the CLI turns that into a "use durable" hint. - """ - async with PipelexAPIClient() as client: - with progress_console.status("Running (blocking)…"): - result = await client.execute(pipe_code=pipe_code, mthds_contents=[bundle], inputs=inputs) - # Adapt the blocking `execute` result onto `RunResults` so both modes return one type; - # `.main_stuff` is already resolved by the SDK (it raises `MissingMainStuffError` if a - # completed run named no main stuff). - return RunResults(pipeline_run_id=result.pipeline_run_id, main_stuff=result.main_stuff) - - -async def run_durable_attended(*, pipe_code: str, bundle: str, inputs: dict[str, Any] | None = None) -> RunResults: - """Start a durable run, print its id immediately, then poll it to completion. - - The id is printed before polling so the run is never lost: Ctrl-C leaves it - executing server-side and you can resume with `piper runs wait `. - """ - async with PipelexAPIClient() as client: - start_result = await client.start(pipe_code=pipe_code, mthds_contents=[bundle], inputs=inputs) - run_id = start_result.pipeline_run_id - progress_console.print(f"Run started: [bold]{run_id}[/bold]") - return await _attend(client=client, run_id=run_id) - - -async def start_detached(*, pipe_code: str, bundle: str, inputs: dict[str, Any] | None = None) -> str: - """Start a durable run and return its id without waiting. - - The run keeps executing server-side; fetch it later with - `piper runs status|result|wait ` — even from another terminal. - """ - async with PipelexAPIClient() as client: - start_result = await client.start(pipe_code=pipe_code, mthds_contents=[bundle], inputs=inputs) - return start_result.pipeline_run_id - - -async def wait_for_run(run_id: str) -> RunResults: - """Poll an already-started run to completion with a live status line.""" - async with PipelexAPIClient() as client: - return await _attend(client=client, run_id=run_id) - - -async def fetch_run_status(run_id: str) -> RunRead: - """Fetch a run's coarse status (`GET /v1/runs/{id}/status`).""" - async with PipelexAPIClient() as client: - return await client.get_run_status(run_id) - - -async def fetch_run_result(run_id: str) -> RunResultState: - """Fetch a run's result state (`GET /v1/runs/{id}/results`) without polling.""" - async with PipelexAPIClient() as client: - return await client.get_run_result(run_id) - - -async def _attend(*, client: PipelexAPIClient, run_id: str) -> RunResults: - """Poll a run to its terminal state, driving a Rich status line per poll.""" - short_id = run_id[:8] - with progress_console.status(f"Run {short_id}… in progress") as status: - - def on_poll(info: PollInfo) -> None: - status.update(f"Run {short_id}… in progress — {info.elapsed_seconds:.0f}s, poll #{info.attempt}") - - try: - return await client.wait_for_result(run_id, options=WaitForResultOptions(on_poll=on_poll)) - except asyncio.CancelledError: - # Ctrl-C: the run keeps executing server-side — tell the user how to pick it back up. - progress_console.print(f"\nInterrupted — the run is still executing. Resume with: [bold]piper runs wait {run_id}[/bold]") - raise diff --git a/pyproject.toml b/pyproject.toml index 2eecccc..2d8c9a6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -26,11 +26,26 @@ dependencies = [ piper = "piper.cli:app" [tool.setuptools] -packages = ["piper", "piper.examples"] +# The method list is enumerated per method in three places that must stay in lockstep: +# piper/methods/*, this packages/package-data block, and the Makefile codegen targets. +packages = [ + "piper", + "piper.attended", + "piper.blocking", + "piper.detached", + "piper.generated", + "piper.generated.extract_entities", + "piper.generated.generate_image", + "piper.generated.summarize_pdf", +] include-package-data = true [tool.setuptools.package-data] -piper = ["py.typed", "methods/*/main.mthds"] +piper = ["py.typed", "methods/*/main.mthds", "methods/*/inputs.template.json"] +# Ship each codegen.lock alongside its models.py so `pipelex codegen check` works on an installed copy. +"piper.generated.extract_entities" = ["codegen.lock"] +"piper.generated.generate_image" = ["codegen.lock"] +"piper.generated.summarize_pdf" = ["codegen.lock"] [project.optional-dependencies] dev = [ @@ -211,6 +226,7 @@ exclude = [ ".ruff_cache", ".venv", ".vscode", + "piper/generated", # generated by `pipelex codegen` — reformatting would trip the codegen.lock drift check "trigger_pipeline", ] line-length = 150 diff --git a/tests/e2e/test_extract_entities.py b/tests/e2e/test_extract_entities.py index 059e851..8bb017a 100644 --- a/tests/e2e/test_extract_entities.py +++ b/tests/e2e/test_extract_entities.py @@ -1,7 +1,9 @@ import pytest -from piper.examples.extract_entities import BUNDLE_PATH, PIPE_CODE, parse -from piper.runner import run_blocking, run_durable_attended +from piper.blocking.cli import METHODS_DIR, execute_pipe +from piper.generated.extract_entities.models import ExtractedEntities + +BUNDLE_PATH = METHODS_DIR / "extract-entities" / "main.mthds" SAMPLE_TEXT = ( "Marie Curie joined the University of Paris in 1906, two years after Pierre Curie won recognition from the Royal Swedish Academy of Sciences." @@ -12,16 +14,10 @@ @pytest.mark.llm @pytest.mark.pipelex_api class TestExtractEntities: - async def test_durable(self): - # Full durable lifecycle through the hosted API: start + poll + narrow. - bundle = BUNDLE_PATH.read_text() - results = await run_durable_attended(pipe_code=PIPE_CODE, bundle=bundle, inputs={"text": SAMPLE_TEXT}) - entities = parse(results) - assert any("Curie" in person for person in entities.people) - async def test_blocking(self): - # Blocking `execute` path (extraction is fast enough for the ~30s cap). + # The blocking lifecycle end to end: one `execute` call, then narrow. + # Extraction finishes well under the hosted ~30s cap, so it is the demo blocking mode owns. bundle = BUNDLE_PATH.read_text() - results = await run_blocking(pipe_code=PIPE_CODE, bundle=bundle, inputs={"text": SAMPLE_TEXT}) - entities = parse(results) + main_stuff = await execute_pipe(pipe_code="extract_entities", bundle=bundle, inputs={"text": SAMPLE_TEXT}) + entities = ExtractedEntities.model_validate(main_stuff) assert any("Curie" in person for person in entities.people) diff --git a/tests/e2e/test_generate_image.py b/tests/e2e/test_generate_image.py index b6c03db..74489ff 100644 --- a/tests/e2e/test_generate_image.py +++ b/tests/e2e/test_generate_image.py @@ -1,7 +1,9 @@ import pytest -from piper.examples.generate_image import BUNDLE_PATH, PIPE_CODE, parse -from piper.runner import run_durable_attended +from piper.detached.cli import METHODS_DIR, attend_run, start_pipe +from piper.generated.generate_image.models import Image + +BUNDLE_PATH = METHODS_DIR / "generate-image" / "main.mthds" SAMPLE_PROMPT = "A watercolor painting of a fox reading a book under a tree." @@ -10,9 +12,13 @@ @pytest.mark.img_gen @pytest.mark.pipelex_api class TestGenerateImage: - async def test_durable(self): - # Image generation outlives the ~30s blocking cap, so only the durable path is exercised. + async def test_detached(self): + # The detached lifecycle end to end, and with it the whole run-id story: + # start, get an id back, then pick the run up again through that id alone. + # Image generation outlives the ~30s blocking cap, so it is the demo detached mode owns. bundle = BUNDLE_PATH.read_text() - results = await run_durable_attended(pipe_code=PIPE_CODE, bundle=bundle, inputs={"image_prompt": SAMPLE_PROMPT}) - image = parse(results) + run_id = await start_pipe(pipe_code="generate_image", bundle=bundle, inputs={"image_prompt": SAMPLE_PROMPT}) + assert run_id + main_stuff = await attend_run(run_id) + image = Image.model_validate(main_stuff) assert image.url diff --git a/tests/e2e/test_summarize_pdf.py b/tests/e2e/test_summarize_pdf.py index 3b7d355..0c82efb 100644 --- a/tests/e2e/test_summarize_pdf.py +++ b/tests/e2e/test_summarize_pdf.py @@ -2,9 +2,11 @@ import pytest -from piper.examples.summarize_pdf import BUNDLE_PATH, PIPE_CODE, parse -from piper.file_input import build_document_input -from piper.runner import run_durable_attended +from piper.attended.cli import METHODS_DIR, start_and_wait +from piper.generated.summarize_pdf.models import DocumentSummary +from piper.inputs import build_document_input + +BUNDLE_PATH = METHODS_DIR / "summarize-pdf" / "main.mthds" SAMPLE_PDF = Path(__file__).parents[2] / "samples" / "sample-invoice.pdf" @@ -13,12 +15,12 @@ @pytest.mark.llm @pytest.mark.pipelex_api class TestSummarizePdf: - async def test_durable(self): - # Full durable lifecycle through the hosted API: encode the PDF, start + poll + narrow. + async def test_attended(self): + # The attended lifecycle end to end: encode the PDF, start a durable run, poll it, narrow. bundle = BUNDLE_PATH.read_text() inputs = {"document": build_document_input(SAMPLE_PDF)} - results = await run_durable_attended(pipe_code=PIPE_CODE, bundle=bundle, inputs=inputs) - summary = parse(results) + main_stuff = await start_and_wait(pipe_code="summarize_pdf", bundle=bundle, inputs=inputs) + summary = DocumentSummary.model_validate(main_stuff) assert summary.title assert summary.doc_type assert summary.key_points diff --git a/tests/integration/test_fundamentals.py b/tests/integration/test_fundamentals.py index 01c32f8..cdc2bcf 100644 --- a/tests/integration/test_fundamentals.py +++ b/tests/integration/test_fundamentals.py @@ -3,11 +3,17 @@ import pytest from pipelex_sdk.client import PipelexAPIClient -from piper.examples.extract_entities import BUNDLE_PATH as EXTRACT_ENTITIES_BUNDLE -from piper.examples.generate_image import BUNDLE_PATH as GENERATE_IMAGE_BUNDLE -from piper.examples.summarize_pdf import BUNDLE_PATH as SUMMARIZE_PDF_BUNDLE +import piper -BUNDLE_PATHS = [EXTRACT_ENTITIES_BUNDLE, SUMMARIZE_PDF_BUNDLE, GENERATE_IMAGE_BUNDLE] +# The bundles ship with the package. Each mode CLI resolves them the same way +# (`METHODS_DIR` in `piper//cli.py`), so this check stays mode-agnostic. +METHODS_DIR = Path(piper.__file__).parent / "methods" + +BUNDLE_PATHS = [ + METHODS_DIR / "extract-entities" / "main.mthds", + METHODS_DIR / "summarize-pdf" / "main.mthds", + METHODS_DIR / "generate-image" / "main.mthds", +] class TestFundamentals: diff --git a/tests/unit/test_attended_cli.py b/tests/unit/test_attended_cli.py new file mode 100644 index 0000000..591385d --- /dev/null +++ b/tests/unit/test_attended_cli.py @@ -0,0 +1,94 @@ +from pathlib import Path + +from pytest_mock import MockerFixture +from typer.testing import CliRunner + +from piper.cli import app +from piper.inputs import SAMPLE_ENTITIES_TEXT, SAMPLE_IMAGE_PROMPT + +ENTITIES_CONTENT = {"people": ["Marie Curie"], "orgs": ["University of Paris"], "dates": ["1906"]} +SUMMARY_CONTENT = {"title": "Q3 Report", "doc_type": "report", "key_points": ["Revenue up 12%"]} +IMAGE_CONTENT = {"url": "https://example.com/cat.png", "public_url": "https://cdn.example.com/cat.png"} + +runner = CliRunner() + + +class TestAttendedCli: + def test_help_lists_the_demos(self): + result = runner.invoke(app, ["attended", "--help"]) + assert result.exit_code == 0 + for command in ("extract-entities", "summarize-pdf", "generate-image"): + assert command in result.output + + def test_extract_entities_starts_waits_and_prints_result(self, mocker: MockerFixture): + attended_mock = mocker.patch("piper.attended.cli.start_and_wait", return_value=ENTITIES_CONTENT) + result = runner.invoke(app, ["attended", "extract-entities", "some text"]) + assert result.exit_code == 0 + attended_mock.assert_awaited_once() + assert attended_mock.await_args is not None + assert attended_mock.await_args.kwargs["pipe_code"] == "extract_entities" + assert attended_mock.await_args.kwargs["inputs"] == {"text": "some text"} + assert "Marie Curie" in result.output + + def test_extract_entities_falls_back_to_the_sample(self, mocker: MockerFixture): + attended_mock = mocker.patch("piper.attended.cli.start_and_wait", return_value=ENTITIES_CONTENT) + result = runner.invoke(app, ["attended", "extract-entities"]) + assert result.exit_code == 0 + assert attended_mock.await_args is not None + assert attended_mock.await_args.kwargs["inputs"] == {"text": SAMPLE_ENTITIES_TEXT} + + def test_extract_entities_rejects_both_text_and_file(self, tmp_path: Path): + input_file = tmp_path / "input.txt" + input_file.write_text("from a file") + result = runner.invoke(app, ["attended", "extract-entities", "inline text", "--file", str(input_file)]) + assert result.exit_code != 0 + + def test_extract_entities_reads_the_file_input(self, mocker: MockerFixture, tmp_path: Path): + attended_mock = mocker.patch("piper.attended.cli.start_and_wait", return_value=ENTITIES_CONTENT) + input_file = tmp_path / "input.txt" + input_file.write_text("text from a file") + result = runner.invoke(app, ["attended", "extract-entities", "--file", str(input_file)]) + assert result.exit_code == 0 + assert attended_mock.await_args is not None + assert attended_mock.await_args.kwargs["inputs"] == {"text": "text from a file"} + + def test_summarize_pdf_falls_back_to_the_sample_invoice(self, mocker: MockerFixture): + attended_mock = mocker.patch("piper.attended.cli.start_and_wait", return_value=SUMMARY_CONTENT) + result = runner.invoke(app, ["attended", "summarize-pdf"]) + assert result.exit_code == 0 + assert attended_mock.await_args is not None + document_input = attended_mock.await_args.kwargs["inputs"]["document"] + assert document_input["concept"] == "Document" + assert document_input["content"]["filename"] == "sample-invoice.pdf" + + def test_summarize_pdf_rejects_a_missing_file(self, tmp_path: Path): + result = runner.invoke(app, ["attended", "summarize-pdf", str(tmp_path / "nope.pdf")]) + assert result.exit_code != 0 + + def test_summarize_pdf_sends_the_document_envelope(self, mocker: MockerFixture, tmp_path: Path): + attended_mock = mocker.patch("piper.attended.cli.start_and_wait", return_value=SUMMARY_CONTENT) + pdf = tmp_path / "doc.pdf" + pdf.write_bytes(b"%PDF-1.4 fake") + result = runner.invoke(app, ["attended", "summarize-pdf", str(pdf)]) + assert result.exit_code == 0 + assert "Q3 Report" in result.output + assert attended_mock.await_args is not None + document_input = attended_mock.await_args.kwargs["inputs"]["document"] + assert document_input["concept"] == "Document" + assert document_input["content"]["mime_type"] == "application/pdf" + assert document_input["content"]["url"].startswith("data:application/pdf;base64,") + + def test_generate_image_falls_back_to_the_sample(self, mocker: MockerFixture): + attended_mock = mocker.patch("piper.attended.cli.start_and_wait", return_value=IMAGE_CONTENT) + result = runner.invoke(app, ["attended", "generate-image"]) + assert result.exit_code == 0 + assert attended_mock.await_args is not None + assert attended_mock.await_args.kwargs["inputs"] == {"image_prompt": SAMPLE_IMAGE_PROMPT} + + def test_generate_image_sends_the_prompt(self, mocker: MockerFixture): + attended_mock = mocker.patch("piper.attended.cli.start_and_wait", return_value=IMAGE_CONTENT) + result = runner.invoke(app, ["attended", "generate-image", "a cat wearing a hat"]) + assert result.exit_code == 0 + assert "example.com/cat.png" in result.output + assert attended_mock.await_args is not None + assert attended_mock.await_args.kwargs["inputs"] == {"image_prompt": "a cat wearing a hat"} diff --git a/tests/unit/test_blocking_cli.py b/tests/unit/test_blocking_cli.py new file mode 100644 index 0000000..366758b --- /dev/null +++ b/tests/unit/test_blocking_cli.py @@ -0,0 +1,94 @@ +from pathlib import Path + +from pytest_mock import MockerFixture +from typer.testing import CliRunner + +from piper.cli import app +from piper.inputs import SAMPLE_ENTITIES_TEXT, SAMPLE_IMAGE_PROMPT + +ENTITIES_CONTENT = {"people": ["Marie Curie"], "orgs": ["University of Paris"], "dates": ["1906"]} +SUMMARY_CONTENT = {"title": "Q3 Report", "doc_type": "report", "key_points": ["Revenue up 12%"]} +IMAGE_CONTENT = {"url": "https://example.com/cat.png", "public_url": "https://cdn.example.com/cat.png"} + +runner = CliRunner() + + +class TestBlockingCli: + def test_help_lists_the_demos(self): + result = runner.invoke(app, ["blocking", "--help"]) + assert result.exit_code == 0 + for command in ("extract-entities", "summarize-pdf", "generate-image"): + assert command in result.output + + def test_extract_entities_executes_and_prints_result(self, mocker: MockerFixture): + execute_mock = mocker.patch("piper.blocking.cli.execute_pipe", return_value=ENTITIES_CONTENT) + result = runner.invoke(app, ["blocking", "extract-entities", "some text"]) + assert result.exit_code == 0 + execute_mock.assert_awaited_once() + assert execute_mock.await_args is not None + assert execute_mock.await_args.kwargs["pipe_code"] == "extract_entities" + assert execute_mock.await_args.kwargs["inputs"] == {"text": "some text"} + assert "Marie Curie" in result.output + + def test_extract_entities_falls_back_to_the_sample(self, mocker: MockerFixture): + execute_mock = mocker.patch("piper.blocking.cli.execute_pipe", return_value=ENTITIES_CONTENT) + result = runner.invoke(app, ["blocking", "extract-entities"]) + assert result.exit_code == 0 + assert execute_mock.await_args is not None + assert execute_mock.await_args.kwargs["inputs"] == {"text": SAMPLE_ENTITIES_TEXT} + + def test_extract_entities_rejects_both_text_and_file(self, tmp_path: Path): + input_file = tmp_path / "input.txt" + input_file.write_text("from a file") + result = runner.invoke(app, ["blocking", "extract-entities", "inline text", "--file", str(input_file)]) + assert result.exit_code != 0 + + def test_extract_entities_reads_the_file_input(self, mocker: MockerFixture, tmp_path: Path): + execute_mock = mocker.patch("piper.blocking.cli.execute_pipe", return_value=ENTITIES_CONTENT) + input_file = tmp_path / "input.txt" + input_file.write_text("text from a file") + result = runner.invoke(app, ["blocking", "extract-entities", "--file", str(input_file)]) + assert result.exit_code == 0 + assert execute_mock.await_args is not None + assert execute_mock.await_args.kwargs["inputs"] == {"text": "text from a file"} + + def test_summarize_pdf_falls_back_to_the_sample_invoice(self, mocker: MockerFixture): + execute_mock = mocker.patch("piper.blocking.cli.execute_pipe", return_value=SUMMARY_CONTENT) + result = runner.invoke(app, ["blocking", "summarize-pdf"]) + assert result.exit_code == 0 + assert execute_mock.await_args is not None + document_input = execute_mock.await_args.kwargs["inputs"]["document"] + assert document_input["concept"] == "Document" + assert document_input["content"]["filename"] == "sample-invoice.pdf" + + def test_summarize_pdf_rejects_a_missing_file(self, tmp_path: Path): + result = runner.invoke(app, ["blocking", "summarize-pdf", str(tmp_path / "nope.pdf")]) + assert result.exit_code != 0 + + def test_summarize_pdf_sends_the_document_envelope(self, mocker: MockerFixture, tmp_path: Path): + execute_mock = mocker.patch("piper.blocking.cli.execute_pipe", return_value=SUMMARY_CONTENT) + pdf = tmp_path / "doc.pdf" + pdf.write_bytes(b"%PDF-1.4 fake") + result = runner.invoke(app, ["blocking", "summarize-pdf", str(pdf)]) + assert result.exit_code == 0 + assert "Q3 Report" in result.output + assert execute_mock.await_args is not None + document_input = execute_mock.await_args.kwargs["inputs"]["document"] + assert document_input["concept"] == "Document" + assert document_input["content"]["mime_type"] == "application/pdf" + assert document_input["content"]["url"].startswith("data:application/pdf;base64,") + + def test_generate_image_falls_back_to_the_sample(self, mocker: MockerFixture): + execute_mock = mocker.patch("piper.blocking.cli.execute_pipe", return_value=IMAGE_CONTENT) + result = runner.invoke(app, ["blocking", "generate-image"]) + assert result.exit_code == 0 + assert execute_mock.await_args is not None + assert execute_mock.await_args.kwargs["inputs"] == {"image_prompt": SAMPLE_IMAGE_PROMPT} + + def test_generate_image_sends_the_prompt(self, mocker: MockerFixture): + execute_mock = mocker.patch("piper.blocking.cli.execute_pipe", return_value=IMAGE_CONTENT) + result = runner.invoke(app, ["blocking", "generate-image", "a cat wearing a hat"]) + assert result.exit_code == 0 + assert "example.com/cat.png" in result.output + assert execute_mock.await_args is not None + assert execute_mock.await_args.kwargs["inputs"] == {"image_prompt": "a cat wearing a hat"} diff --git a/tests/unit/test_bootstrap_script.py b/tests/unit/test_bootstrap_script.py index 307ff3f..67d632f 100644 --- a/tests/unit/test_bootstrap_script.py +++ b/tests/unit/test_bootstrap_script.py @@ -25,6 +25,7 @@ def load_bootstrap() -> Any: def write_template(root: Path, *, extra_pyproject: str = "") -> None: (root / "piper").mkdir() + (root / "piper" / "blocking").mkdir() (root / "tests").mkdir() (root / "pyproject.toml").write_text( f"""[project] @@ -37,10 +38,22 @@ def write_template(root: Path, *, extra_pyproject: str = "") -> None: piper = "piper.cli:app" [tool.setuptools] -packages = ["piper", "piper.examples"] +# piper/methods/*, this packages/package-data block, and the Makefile codegen targets. +packages = [ + "piper", + "piper.blocking", + "piper.generated", + "piper.generated.extract_entities", +] [tool.setuptools.package-data] piper = ["py.typed", "methods/*/main.mthds"] +"piper.generated.extract_entities" = ["codegen.lock"] + +[tool.ruff] +extend-exclude = [ + "piper/generated", +] [project.urls] Repository = "https://github.com/yourusername/piper" # Replace with your repository URL @@ -57,6 +70,8 @@ def write_template(root: Path, *, extra_pyproject: str = "") -> None: (root / "CLAUDE.md").write_text("The `piper` CLI lives in `piper/cli.py`.\n", encoding="utf-8") (root / "LICENSE").write_text("MIT License\n\nCopyright (c) 2025 Example\n", encoding="utf-8") (root / "piper" / "cli.py").write_text('"""The piper CLI."""\n', encoding="utf-8") + # A mode sub-package: nested dir, and an import of a sibling module to rewrite. + (root / "piper" / "blocking" / "cli.py").write_text("from piper.inputs import read_text_input\n", encoding="utf-8") (root / "tests" / "test_cli.py").write_text("from piper.cli import app\n", encoding="utf-8") @@ -81,7 +96,10 @@ def test_survivor_check_allows_requested_values_containing_piper(tmp_path: Path) def test_survivor_check_still_rejects_unhandled_template_tokens(tmp_path: Path) -> None: bootstrap = load_bootstrap() - write_template(tmp_path, extra_pyproject='custom = "piper"\n') + # A bare `piper` word inside prose (not quoted-exact, not in package position) + # is a shape the pyproject transform refuses to guess at — it must survive + # the substitution pass and abort the bootstrap. + write_template(tmp_path, extra_pyproject='custom = "run piper somewhere"\n') names = bootstrap.Names(dist="invoice-extractor", package="invoice_extractor", title="Invoice Extractor") opts = bootstrap.Options( @@ -98,4 +116,4 @@ def test_survivor_check_still_rejects_unhandled_template_tokens(tmp_path: Path) with pytest.raises(SystemExit) as exc_info: bootstrap.run(tmp_path, names, opts) - assert 'custom = "piper"' in str(exc_info.value) + assert 'custom = "run piper somewhere"' in str(exc_info.value) diff --git a/tests/unit/test_cli.py b/tests/unit/test_cli.py deleted file mode 100644 index 6aa4589..0000000 --- a/tests/unit/test_cli.py +++ /dev/null @@ -1,106 +0,0 @@ -from pathlib import Path - -from pipelex_sdk.runs import RunResults -from pytest_mock import MockerFixture -from typer.testing import CliRunner - -from piper.cli import app - -ENTITIES_CONTENT = {"people": ["Marie Curie"], "orgs": ["University of Paris"], "dates": ["1906"]} -SUMMARY_CONTENT = {"title": "Q3 Report", "doc_type": "report", "key_points": ["Revenue up 12%"]} -IMAGE_CONTENT = {"url": "https://example.com/cat.png", "public_url": "https://cdn.example.com/cat.png"} - -runner = CliRunner() - - -class TestCli: - def test_help_lists_commands(self): - result = runner.invoke(app, ["--help"]) - assert result.exit_code == 0 - for command in ("extract-entities", "summarize-pdf", "generate-image", "runs"): - assert command in result.output - - def test_runs_help_lists_subcommands(self): - result = runner.invoke(app, ["runs", "--help"]) - assert result.exit_code == 0 - for subcommand in ("status", "result", "wait"): - assert subcommand in result.output - - def test_extract_entities_requires_input(self): - result = runner.invoke(app, ["extract-entities"]) - assert result.exit_code != 0 - - def test_extract_entities_rejects_both_text_and_file(self, tmp_path: Path): - input_file = tmp_path / "input.txt" - input_file.write_text("from a file") - result = runner.invoke(app, ["extract-entities", "inline text", "--file", str(input_file)]) - assert result.exit_code != 0 - - def test_extract_entities_rejects_bad_mode(self): - result = runner.invoke(app, ["extract-entities", "some text", "--mode", "bogus"]) - assert result.exit_code != 0 - - def test_extract_entities_rejects_blocking_detach(self): - result = runner.invoke(app, ["extract-entities", "some text", "--mode", "blocking", "--detach"]) - assert result.exit_code != 0 - - def test_default_mode_is_durable(self, mocker: MockerFixture): - durable_mock = mocker.patch("piper.cli.run_durable_attended", return_value=RunResults(pipeline_run_id="run-1", main_stuff=ENTITIES_CONTENT)) - result = runner.invoke(app, ["extract-entities", "some text"]) - assert result.exit_code == 0 - durable_mock.assert_awaited_once() - assert "Marie Curie" in result.output - - def test_env_var_selects_blocking_mode(self, mocker: MockerFixture): - blocking_mock = mocker.patch("piper.cli.run_blocking", return_value=RunResults(pipeline_run_id="run-2", main_stuff=ENTITIES_CONTENT)) - result = runner.invoke(app, ["extract-entities", "some text"], env={"PIPELEX_EXECUTION_MODE": "blocking"}) - assert result.exit_code == 0 - blocking_mock.assert_awaited_once() - - def test_file_input_is_read(self, mocker: MockerFixture, tmp_path: Path): - durable_mock = mocker.patch("piper.cli.run_durable_attended", return_value=RunResults(pipeline_run_id="run-3", main_stuff=ENTITIES_CONTENT)) - input_file = tmp_path / "input.txt" - input_file.write_text("text from a file") - result = runner.invoke(app, ["extract-entities", "--file", str(input_file)]) - assert result.exit_code == 0 - assert durable_mock.await_args is not None - assert durable_mock.await_args.kwargs["inputs"] == {"text": "text from a file"} - - def test_detach_prints_run_id(self, mocker: MockerFixture): - mocker.patch("piper.cli.start_detached", return_value="run-abc123") - result = runner.invoke(app, ["extract-entities", "some text", "--detach"]) - assert result.exit_code == 0 - assert "run-abc123" in result.output - - def test_summarize_pdf_requires_file(self): - result = runner.invoke(app, ["summarize-pdf"]) - assert result.exit_code != 0 - - def test_summarize_pdf_rejects_missing_file(self, tmp_path: Path): - result = runner.invoke(app, ["summarize-pdf", str(tmp_path / "nope.pdf")]) - assert result.exit_code != 0 - - def test_summarize_pdf_sends_document_input(self, mocker: MockerFixture, tmp_path: Path): - durable_mock = mocker.patch("piper.cli.run_durable_attended", return_value=RunResults(pipeline_run_id="run-pdf", main_stuff=SUMMARY_CONTENT)) - pdf = tmp_path / "doc.pdf" - pdf.write_bytes(b"%PDF-1.4 fake") - result = runner.invoke(app, ["summarize-pdf", str(pdf)]) - assert result.exit_code == 0 - assert "Q3 Report" in result.output - assert durable_mock.await_args is not None - document_input = durable_mock.await_args.kwargs["inputs"]["document"] - assert document_input["concept"] == "Document" - assert document_input["content"]["mime_type"] == "application/pdf" - assert document_input["content"]["url"].startswith("data:application/pdf;base64,") - - def test_generate_image_requires_input(self): - result = runner.invoke(app, ["generate-image"]) - assert result.exit_code != 0 - - def test_generate_image_sends_prompt(self, mocker: MockerFixture): - durable_mock = mocker.patch("piper.cli.run_durable_attended", return_value=RunResults(pipeline_run_id="run-img", main_stuff=IMAGE_CONTENT)) - result = runner.invoke(app, ["generate-image", "a cat wearing a hat"]) - assert result.exit_code == 0 - assert "example.com/cat.png" in result.output - assert durable_mock.await_args is not None - assert durable_mock.await_args.kwargs["inputs"] == {"image_prompt": "a cat wearing a hat"} diff --git a/tests/unit/test_detached_cli.py b/tests/unit/test_detached_cli.py new file mode 100644 index 0000000..d6c540d --- /dev/null +++ b/tests/unit/test_detached_cli.py @@ -0,0 +1,140 @@ +from pathlib import Path + +from pipelex_sdk.runs import RunRead, RunResultCompleted, RunResultFailed, RunResultRunning, RunResults, RunStatus +from pytest_mock import MockerFixture +from typer.testing import CliRunner + +from piper.cli import app +from piper.inputs import SAMPLE_ENTITIES_TEXT, SAMPLE_IMAGE_PROMPT + +ENTITIES_CONTENT = {"people": ["Marie Curie"], "orgs": ["University of Paris"], "dates": ["1906"]} +RUN_ID = "run-abc123" + +runner = CliRunner() + + +class TestDetachedCli: + def test_help_lists_the_demos_and_the_lifecycle_commands(self): + result = runner.invoke(app, ["detached", "--help"]) + assert result.exit_code == 0 + for command in ("extract-entities", "summarize-pdf", "generate-image", "wait", "status", "result"): + assert command in result.output + + def test_extract_entities_starts_the_run_and_prints_its_id(self, mocker: MockerFixture): + start_mock = mocker.patch("piper.detached.cli.start_pipe", return_value=RUN_ID) + result = runner.invoke(app, ["detached", "extract-entities", "some text"]) + assert result.exit_code == 0 + start_mock.assert_awaited_once() + assert start_mock.await_args is not None + assert start_mock.await_args.kwargs["pipe_code"] == "extract_entities" + assert start_mock.await_args.kwargs["inputs"] == {"text": "some text"} + # The bare id on stdout is the contract: RUN_ID=$(piper detached extract-entities "…") + assert result.stdout.strip() == RUN_ID + + def test_extract_entities_falls_back_to_the_sample(self, mocker: MockerFixture): + start_mock = mocker.patch("piper.detached.cli.start_pipe", return_value=RUN_ID) + result = runner.invoke(app, ["detached", "extract-entities"]) + assert result.exit_code == 0 + assert start_mock.await_args is not None + assert start_mock.await_args.kwargs["inputs"] == {"text": SAMPLE_ENTITIES_TEXT} + # The sample notice goes to stderr, so stdout stays the bare run id. + assert result.stdout.strip() == RUN_ID + + def test_extract_entities_rejects_both_text_and_file(self, tmp_path: Path): + input_file = tmp_path / "input.txt" + input_file.write_text("from a file") + result = runner.invoke(app, ["detached", "extract-entities", "inline text", "--file", str(input_file)]) + assert result.exit_code != 0 + + def test_extract_entities_reads_the_file_input(self, mocker: MockerFixture, tmp_path: Path): + start_mock = mocker.patch("piper.detached.cli.start_pipe", return_value=RUN_ID) + input_file = tmp_path / "input.txt" + input_file.write_text("text from a file") + result = runner.invoke(app, ["detached", "extract-entities", "--file", str(input_file)]) + assert result.exit_code == 0 + assert start_mock.await_args is not None + assert start_mock.await_args.kwargs["inputs"] == {"text": "text from a file"} + + def test_summarize_pdf_falls_back_to_the_sample_invoice(self, mocker: MockerFixture): + start_mock = mocker.patch("piper.detached.cli.start_pipe", return_value=RUN_ID) + result = runner.invoke(app, ["detached", "summarize-pdf"]) + assert result.exit_code == 0 + assert start_mock.await_args is not None + document_input = start_mock.await_args.kwargs["inputs"]["document"] + assert document_input["concept"] == "Document" + assert document_input["content"]["filename"] == "sample-invoice.pdf" + + def test_summarize_pdf_rejects_a_missing_file(self, tmp_path: Path): + result = runner.invoke(app, ["detached", "summarize-pdf", str(tmp_path / "nope.pdf")]) + assert result.exit_code != 0 + + def test_summarize_pdf_sends_the_document_envelope(self, mocker: MockerFixture, tmp_path: Path): + start_mock = mocker.patch("piper.detached.cli.start_pipe", return_value=RUN_ID) + pdf = tmp_path / "doc.pdf" + pdf.write_bytes(b"%PDF-1.4 fake") + result = runner.invoke(app, ["detached", "summarize-pdf", str(pdf)]) + assert result.exit_code == 0 + assert start_mock.await_args is not None + document_input = start_mock.await_args.kwargs["inputs"]["document"] + assert document_input["concept"] == "Document" + assert document_input["content"]["mime_type"] == "application/pdf" + assert document_input["content"]["url"].startswith("data:application/pdf;base64,") + + def test_generate_image_falls_back_to_the_sample(self, mocker: MockerFixture): + start_mock = mocker.patch("piper.detached.cli.start_pipe", return_value=RUN_ID) + result = runner.invoke(app, ["detached", "generate-image"]) + assert result.exit_code == 0 + assert start_mock.await_args is not None + assert start_mock.await_args.kwargs["inputs"] == {"image_prompt": SAMPLE_IMAGE_PROMPT} + assert result.stdout.strip() == RUN_ID + + def test_generate_image_sends_the_prompt(self, mocker: MockerFixture): + start_mock = mocker.patch("piper.detached.cli.start_pipe", return_value=RUN_ID) + result = runner.invoke(app, ["detached", "generate-image", "a cat wearing a hat"]) + assert result.exit_code == 0 + assert start_mock.await_args is not None + assert start_mock.await_args.kwargs["inputs"] == {"image_prompt": "a cat wearing a hat"} + assert result.stdout.strip() == RUN_ID + + def test_wait_prints_the_raw_main_stuff(self, mocker: MockerFixture): + attend_mock = mocker.patch("piper.detached.cli.attend_run", return_value=ENTITIES_CONTENT) + result = runner.invoke(app, ["detached", "wait", RUN_ID]) + assert result.exit_code == 0 + attend_mock.assert_awaited_once_with(RUN_ID) + assert "Marie Curie" in result.output + + def test_status_reports_the_run_status(self, mocker: MockerFixture): + run = RunRead(pipeline_run_id=RUN_ID, pipe_code="extract_entities", status=RunStatus.RUNNING, created_at="2026-07-13T10:00:00Z") + mocker.patch("piper.detached.cli.fetch_run_status", return_value=run) + result = runner.invoke(app, ["detached", "status", RUN_ID]) + assert result.exit_code == 0 + assert RUN_ID in result.output + assert "RUNNING" in result.output + assert "extract_entities" in result.output + + def test_status_flags_a_degraded_reading(self, mocker: MockerFixture): + run = RunRead(pipeline_run_id=RUN_ID, status=RunStatus.RUNNING, created_at="2026-07-13T10:00:00Z", degraded=True) + mocker.patch("piper.detached.cli.fetch_run_status", return_value=run) + result = runner.invoke(app, ["detached", "status", RUN_ID]) + assert result.exit_code == 0 + assert "degraded" in result.output + + def test_result_hints_at_wait_while_the_run_is_running(self, mocker: MockerFixture): + mocker.patch("piper.detached.cli.fetch_run_result", return_value=RunResultRunning(pipeline_run_id=RUN_ID)) + result = runner.invoke(app, ["detached", "result", RUN_ID]) + assert result.exit_code == 0 + assert "piper detached wait" in result.output + + def test_result_prints_the_raw_main_stuff_when_completed(self, mocker: MockerFixture): + completed = RunResultCompleted(pipeline_run_id=RUN_ID, result=RunResults(pipeline_run_id=RUN_ID, main_stuff=ENTITIES_CONTENT)) + mocker.patch("piper.detached.cli.fetch_run_result", return_value=completed) + result = runner.invoke(app, ["detached", "result", RUN_ID]) + assert result.exit_code == 0 + assert "Marie Curie" in result.output + + def test_result_exits_non_zero_when_the_run_failed(self, mocker: MockerFixture): + failed = RunResultFailed(pipeline_run_id=RUN_ID, status=RunStatus.FAILED, message="the pipe blew up") + mocker.patch("piper.detached.cli.fetch_run_result", return_value=failed) + result = runner.invoke(app, ["detached", "result", RUN_ID]) + assert result.exit_code == 1 + assert "the pipe blew up" in result.output diff --git a/tests/unit/test_errors.py b/tests/unit/test_errors.py index b53e9ee..f469cb0 100644 --- a/tests/unit/test_errors.py +++ b/tests/unit/test_errors.py @@ -1,3 +1,5 @@ +from collections.abc import Mapping + import httpx from pipelex_sdk.errors import ( ApiUnreachableError, @@ -11,24 +13,24 @@ from piper.errors import present_error -def _http_status_error(status_code: int) -> httpx.HTTPStatusError: - request = httpx.Request("POST", "https://api.pipelex.com/v1/execute") - response = httpx.Response(status_code, request=request) +def _http_status_error(status_code: int, *, problem: Mapping[str, object] | None = None) -> httpx.HTTPStatusError: + request = httpx.Request("POST", "https://api.pipelex.com/v1/start") + response = httpx.Response(status_code, request=request, json=problem) if problem is not None else httpx.Response(status_code, request=request) return httpx.HTTPStatusError("boom", request=request, response=response) class TestPresentError: - def test_execute_timeout_hints_durable(self): + def test_execute_timeout_hints_attended(self): presentation = present_error(PipelineExecuteTimeoutError("timed out", elapsed_seconds=31.2)) assert "~30s" in presentation.message assert presentation.hint is not None - assert "--mode durable" in presentation.hint + assert "piper attended" in presentation.hint def test_lifecycle_unavailable_hints_blocking(self): presentation = present_error(RunLifecycleUnavailableError("no run store", api_url="http://localhost:8000")) assert "http://localhost:8000" in presentation.message assert presentation.hint is not None - assert "--mode blocking" in presentation.hint + assert "piper blocking" in presentation.hint def test_http_auth_error_hints_api_key(self): # The protocol routes (execute/start/runs) raise raw httpx.HTTPStatusError, @@ -43,6 +45,25 @@ def test_http_server_error_has_no_hint(self): presentation = present_error(_http_status_error(500)) assert presentation.hint is None + def test_start_without_async_orchestration_hints_blocking(self): + # A synchronous-only runner rejects /start with this RFC 7807 error_type; + # the fix is to run the same demo under `piper blocking`. + problem = { + "error_type": "StartRequiresAsyncOrchestration", + "detail": "Orchestration mode 'direct' cannot honor fire-and-forget delivery. Use /execute instead.", + "status": 400, + } + presentation = present_error(_http_status_error(400, problem=problem)) + assert "Orchestration mode 'direct'" in presentation.message + assert presentation.hint is not None + assert "piper blocking" in presentation.hint + + def test_http_error_surfaces_the_problem_detail(self): + problem = {"title": "Bad input", "detail": "Missing required input 'text'.", "status": 400} + presentation = present_error(_http_status_error(400, problem=problem)) + assert "Missing required input 'text'." in presentation.message + assert presentation.hint is None + def test_unreachable_hints_base_url(self): presentation = present_error(ApiUnreachableError("connect failed", api_url="http://nowhere.invalid")) assert "http://nowhere.invalid" in presentation.message @@ -53,9 +74,9 @@ def test_run_failed_names_run_id(self): presentation = present_error(RunFailedError("run failed", run_id="run-9", status=RunStatus.FAILED)) assert "run-9" in presentation.message assert presentation.hint is not None - assert "runs status run-9" in presentation.hint + assert "piper detached status run-9" in presentation.hint def test_run_timeout_hints_wait(self): presentation = present_error(RunTimeoutError("too slow", run_id="run-9", timeout_seconds=1200.0)) assert presentation.hint is not None - assert "runs wait run-9" in presentation.hint + assert "piper detached wait run-9" in presentation.hint diff --git a/tests/unit/test_extract_entities.py b/tests/unit/test_extract_entities.py deleted file mode 100644 index 74f0be0..0000000 --- a/tests/unit/test_extract_entities.py +++ /dev/null @@ -1,21 +0,0 @@ -import pytest -from pipelex_sdk.runs import RunResults -from pydantic import ValidationError - -from piper.examples.extract_entities import parse - -ENTITIES_CONTENT = {"people": ["Marie Curie", "Pierre Curie"], "orgs": ["University of Paris"], "dates": ["1906"]} - - -class TestExtractEntitiesParse: - def test_parse_main_stuff(self): - results = RunResults(pipeline_run_id="run-1", main_stuff=ENTITIES_CONTENT) - entities = parse(results) - assert entities.people == ["Marie Curie", "Pierre Curie"] - assert entities.orgs == ["University of Paris"] - assert entities.dates == ["1906"] - - def test_parse_shape_mismatch_raises(self): - results = RunResults(pipeline_run_id="run-3", main_stuff={"people": "not-a-list", "orgs": [], "dates": []}) - with pytest.raises(ValidationError): - parse(results) diff --git a/tests/unit/test_file_input.py b/tests/unit/test_file_input.py deleted file mode 100644 index ed5d865..0000000 --- a/tests/unit/test_file_input.py +++ /dev/null @@ -1,30 +0,0 @@ -import base64 -from pathlib import Path - -import pytest - -from piper.file_input import build_document_input - - -class TestBuildDocumentInput: - def test_pdf_envelope(self, tmp_path: Path): - pdf = tmp_path / "invoice.pdf" - payload = b"%PDF-1.4 hello" - pdf.write_bytes(payload) - envelope = build_document_input(pdf) - assert envelope["concept"] == "Document" - content = envelope["content"] - assert content["filename"] == "invoice.pdf" - assert content["mime_type"] == "application/pdf" - expected = base64.b64encode(payload).decode("ascii") - assert content["url"] == f"data:application/pdf;base64,{expected}" - - def test_unknown_extension_falls_back_to_octet_stream(self, tmp_path: Path): - blob = tmp_path / "data.unknownext" - blob.write_bytes(b"\x00\x01\x02") - envelope = build_document_input(blob) - assert envelope["content"]["mime_type"] == "application/octet-stream" - - def test_missing_file_raises(self, tmp_path: Path): - with pytest.raises(FileNotFoundError): - build_document_input(tmp_path / "nope.pdf") diff --git a/tests/unit/test_generate_image.py b/tests/unit/test_generate_image.py deleted file mode 100644 index ac4929a..0000000 --- a/tests/unit/test_generate_image.py +++ /dev/null @@ -1,33 +0,0 @@ -import pytest -from pipelex_sdk.runs import RunResults -from pydantic import ValidationError - -from piper.examples.generate_image import parse - -IMAGE_CONTENT = { - "url": "pipelex-storage://runs/abc/image.png", - "public_url": "https://cdn.example.com/image.png", - "mime_type": "image/png", - "caption": "A watercolor fox", -} - - -class TestGenerateImageParse: - def test_parse_main_stuff(self): - results = RunResults(pipeline_run_id="run-1", main_stuff=IMAGE_CONTENT) - image = parse(results) - assert image.url == "pipelex-storage://runs/abc/image.png" - assert image.public_url == "https://cdn.example.com/image.png" - assert image.mime_type == "image/png" - assert image.caption == "A watercolor fox" - - def test_parse_url_only(self): - results = RunResults(pipeline_run_id="run-2", main_stuff={"url": "https://example.com/i.png"}) - image = parse(results) - assert image.url == "https://example.com/i.png" - assert image.public_url is None - - def test_parse_missing_url_raises(self): - results = RunResults(pipeline_run_id="run-3", main_stuff={"caption": "no url here"}) - with pytest.raises(ValidationError): - parse(results) diff --git a/tests/unit/test_generated_clients.py b/tests/unit/test_generated_clients.py new file mode 100644 index 0000000..2168ba2 --- /dev/null +++ b/tests/unit/test_generated_clients.py @@ -0,0 +1,58 @@ +"""Smoke tests for the generated typed clients (`piper/generated/`). + +Everything here runs offline: the generated models must import, carry the codegen stamp + lock, +round-trip their own serialization, and stay aligned with the input templates the CLI ships. +The cryptographic drift check is `make codegen-check` (offline, against each `codegen.lock`); +these tests are the CI-runnable floor that needs no pipelex CLI at all. +""" + +import json +from pathlib import Path + +import pytest + +from piper.generated.extract_entities.models import ExtractedEntities +from piper.generated.generate_image.models import Image +from piper.generated.summarize_pdf.models import DocumentSummary + +PIPER_DIR = Path(__file__).parent.parent.parent / "piper" + +# method dir -> (generated package dir, the input names every mode CLI passes for that method — +# the three modes are symmetric on inputs, which `test_mode_symmetry.py` guards) +METHODS = { + "extract-entities": ("extract_entities", {"text"}), + "summarize-pdf": ("summarize_pdf", {"document"}), + "generate-image": ("generate_image", {"image_prompt"}), +} + + +class TestGeneratedClients: + @pytest.mark.parametrize("method_dir", list(METHODS)) + def test_generated_artifacts_stamped_and_locked(self, method_dir: str): + """Each generated client carries the codegen stamp header and a sibling codegen.lock.""" + package_dir, _ = METHODS[method_dir] + generated_dir = PIPER_DIR / "generated" / package_dir + models_text = (generated_dir / "models.py").read_text() + assert models_text.startswith("# >>> pipelex-codegen-stamp >>>") + assert "crate_fingerprint:" in models_text + lock_text = (generated_dir / "codegen.lock").read_text() + assert 'path = "models.py"' in lock_text + + @pytest.mark.parametrize("method_dir", list(METHODS)) + def test_input_template_matches_the_cli_inputs(self, method_dir: str): + """The committed inputs template names exactly the inputs the CLI passes for the method — + a regenerated template that drifted from what `piper//cli.py` sends fails here, offline. + """ + expected_inputs = METHODS[method_dir][1] + template_path = PIPER_DIR / "methods" / method_dir / "inputs.template.json" + template = json.loads(template_path.read_text()) + assert set(template.keys()) == expected_inputs + + def test_generated_models_round_trip(self): + """Each generated model validates a wire-shaped payload and round-trips its own dump.""" + entities = ExtractedEntities(people=["Marie Curie"], orgs=["University of Paris"], dates=["1906"]) + assert ExtractedEntities.model_validate(entities.model_dump()) == entities + summary = DocumentSummary(title="Q3 Report", doc_type="report", key_points=["Revenue grew"]) + assert DocumentSummary.model_validate(summary.model_dump()) == summary + image = Image(url="pipelex-storage://runs/abc/image.png", public_url="https://cdn.example.com/image.png") + assert Image.model_validate(image.model_dump()) == image diff --git a/tests/unit/test_inputs.py b/tests/unit/test_inputs.py new file mode 100644 index 0000000..d87fe8e --- /dev/null +++ b/tests/unit/test_inputs.py @@ -0,0 +1,54 @@ +import base64 +from pathlib import Path + +import pytest +import typer + +from piper.inputs import build_document_input, read_text_input + + +class TestInputs: + def test_read_text_input_from_argument(self): + resolved = read_text_input(text="inline text", file=None, sample="the sample") + assert resolved.text == "inline text" + assert resolved.is_sample is False + + def test_read_text_input_from_file(self, tmp_path: Path): + input_file = tmp_path / "input.txt" + input_file.write_text("text from a file") + resolved = read_text_input(text=None, file=input_file, sample="the sample") + assert resolved.text == "text from a file" + assert resolved.is_sample is False + + def test_read_text_input_rejects_both(self, tmp_path: Path): + input_file = tmp_path / "input.txt" + input_file.write_text("text from a file") + with pytest.raises(typer.BadParameter): + read_text_input(text="inline text", file=input_file, sample="the sample") + + def test_read_text_input_falls_back_to_the_sample(self): + resolved = read_text_input(text=None, file=None, sample="the sample") + assert resolved.text == "the sample" + assert resolved.is_sample is True + + def test_pdf_envelope(self, tmp_path: Path): + pdf = tmp_path / "invoice.pdf" + payload = b"%PDF-1.4 hello" + pdf.write_bytes(payload) + envelope = build_document_input(pdf) + assert envelope["concept"] == "Document" + content = envelope["content"] + assert content["filename"] == "invoice.pdf" + assert content["mime_type"] == "application/pdf" + expected = base64.b64encode(payload).decode("ascii") + assert content["url"] == f"data:application/pdf;base64,{expected}" + + def test_unknown_extension_falls_back_to_octet_stream(self, tmp_path: Path): + blob = tmp_path / "data.unknownext" + blob.write_bytes(b"\x00\x01\x02") + envelope = build_document_input(blob) + assert envelope["content"]["mime_type"] == "application/octet-stream" + + def test_missing_file_raises(self, tmp_path: Path): + with pytest.raises(FileNotFoundError): + build_document_input(tmp_path / "nope.pdf") diff --git a/tests/unit/test_mode_symmetry.py b/tests/unit/test_mode_symmetry.py new file mode 100644 index 0000000..141469d --- /dev/null +++ b/tests/unit/test_mode_symmetry.py @@ -0,0 +1,50 @@ +"""The drift guard for the full demo matrix. + +Every demo exists in every mode group, with the same arguments — that symmetry is +the pedagogy (diff two mode files and only the lifecycle helper differs), and the +duplication it implies is exactly what drifts. A demo added to one mode and +forgotten in another fails here. +""" + +import inspect + +import typer + +from piper.attended.cli import app as attended_app +from piper.blocking.cli import app as blocking_app +from piper.cli import app as root_app +from piper.detached.cli import app as detached_app + +MODE_APPS = {"blocking": blocking_app, "attended": attended_app, "detached": detached_app} +DEMO_COMMANDS = {"extract-entities", "summarize-pdf", "generate-image"} +LIFECYCLE_COMMANDS = {"wait", "status", "result"} + + +def _command_names(mode_app: typer.Typer) -> set[str]: + return {command.name for command in mode_app.registered_commands if command.name is not None} + + +def _demo_signatures(mode_app: typer.Typer) -> dict[str, list[str]]: + signatures: dict[str, list[str]] = {} + for command in mode_app.registered_commands: + if command.name in DEMO_COMMANDS and command.callback is not None: + signatures[command.name] = list(inspect.signature(command.callback).parameters) + return signatures + + +class TestModeSymmetry: + def test_every_mode_exposes_every_demo(self): + assert _command_names(blocking_app) == DEMO_COMMANDS + assert _command_names(attended_app) == DEMO_COMMANDS + # Detached owns the run-lifecycle commands on top of the demos — nobody else has them. + assert _command_names(detached_app) == DEMO_COMMANDS | LIFECYCLE_COMMANDS + + def test_the_demos_take_the_same_arguments_in_every_mode(self): + blocking_signatures = _demo_signatures(blocking_app) + assert set(blocking_signatures) == DEMO_COMMANDS + assert _demo_signatures(attended_app) == blocking_signatures + assert _demo_signatures(detached_app) == blocking_signatures + + def test_the_root_app_mounts_the_modes_in_reading_order(self): + mounted = [group.name for group in root_app.registered_groups if group.name in MODE_APPS] + assert mounted == ["blocking", "attended", "detached"] diff --git a/tests/unit/test_summarize_pdf.py b/tests/unit/test_summarize_pdf.py deleted file mode 100644 index a9593c2..0000000 --- a/tests/unit/test_summarize_pdf.py +++ /dev/null @@ -1,25 +0,0 @@ -import pytest -from pipelex_sdk.runs import RunResults -from pydantic import ValidationError - -from piper.examples.summarize_pdf import parse - -SUMMARY_CONTENT = { - "title": "Q3 Financial Report", - "doc_type": "report", - "key_points": ["Revenue grew 12%", "Costs held flat", "Cash runway extended to 2028"], -} - - -class TestSummarizePdfParse: - def test_parse_main_stuff(self): - results = RunResults(pipeline_run_id="run-1", main_stuff=SUMMARY_CONTENT) - summary = parse(results) - assert summary.title == "Q3 Financial Report" - assert summary.doc_type == "report" - assert summary.key_points == ["Revenue grew 12%", "Costs held flat", "Cash runway extended to 2028"] - - def test_parse_shape_mismatch_raises(self): - results = RunResults(pipeline_run_id="run-2", main_stuff={"title": "x", "doc_type": "y", "key_points": "not-a-list"}) - with pytest.raises(ValidationError): - parse(results) diff --git a/wip/mode-split-clis.md b/wip/mode-split-clis.md new file mode 100644 index 0000000..c70b707 --- /dev/null +++ b/wip/mode-split-clis.md @@ -0,0 +1,227 @@ +# Design: split the execution modes into three self-contained CLIs + +- **Status:** design agreed — implementation plan not written yet +- **Date:** 2026-07-13 +- **Scope:** `pipelex-starter-python` only + +## The problem + +The starter has two goals: (1) show how easy it is to call the Pipelex API to execute your methods, and (2) provide best-practice code you'd actually copy into your own project (or start from via the template). + +The current architecture serves neither well. Every demo command takes a `--mode` option, which forces a dispatch chain: CLI command → `_dispatch()` helper → `match mode` → `piper/runner.py` helper → the actual SDK call. That `match/case` over `ExecutionMode` exists purely to showcase all three SDK lifecycles behind one option — no real project needs a runtime mode switch, so nobody would ever copy it. And a newcomer tracing "how do I call the API?" has to hop through four layers before reaching `client.execute(...)`, which is the opposite of demonstrating simplicity. + +## The idea + +Make each execution mode a completely separate CLI sub-package, discovered in reading order: + +1. **`blocking`** — the front door. One call, one response. Trivially simple. +2. **`attended`** — start a durable run, wait here for the result. Still simple. +3. **`detached`** — start a durable run and exit; collect it later. Slightly more going on (a run-id lifecycle), but by the time a reader gets here they understand the model — and it's still copy-pasteable. + +Each sub-package's `cli.py` is a **self-contained copy-paste unit**: the whole mode's story — commands, the SDK lifecycle, progress rendering — in one file, with no dispatch layer and no mode enum. The `--mode` option, `PIPELEX_EXECUTION_MODE` env var, `ExecutionMode`, `_dispatch()`, and `piper/runner.py` all disappear. + +## Decisions taken (2026-07-13, with Louis) + +| Question | Decision | +|---|---| +| CLI surface | **Command groups under one binary**: `piper blocking …`, `piper attended …`, `piper detached …`. One console script, one `--help` discovery page, bootstrap/rename stays trivial. (Rejected: three console scripts — heavier pyproject/bootstrap, no unified help.) | +| Naming | **`blocking` / `attended` / `detached`**. Attended and detached read as a natural pair — both run durably; the difference is who waits. (Rejected: `durable` for the middle one — detached is durable too, so the pair was misleading.) | +| Demo spread | **Full matrix**: every demo command exists in every mode group. Symmetric and diffable — compare two mode files and *only* the execution lifecycle differs. | +| Run lifecycle commands | **Inside `detached`**: `piper detached wait\|status\|result `. The detached package is the complete "start now, collect later" story. (Rejected: top-level `piper runs` — scatters the detached story across two places.) | + +## Target CLI surface + +``` +piper --help # lists the three modes, in reading order +piper blocking extract-entities | summarize-pdf | generate-image +piper attended extract-entities | summarize-pdf | generate-image +piper detached extract-entities | summarize-pdf | generate-image +piper detached wait | status | result +``` + +- The demo commands keep their existing arguments (`text`/`--file`, PDF path, prompt/`--file`). Only the mode selection moves from an option into the command path. +- There is no default mode anymore — the mode is explicit in every invocation, which is itself the lesson. The quick start becomes `piper blocking extract-entities "…"`, the cleanest possible first contact (no run-id chatter on stderr, just the JSON result). +- `piper blocking generate-image` is *expected to fail* on hosted (~30s cap) — that's deliberate. It's the teaching moment for why attended/detached exist, and its help text says so. The full matrix keeps this demo alive without special-casing. + +## Package layout + +``` +piper/ + cli.py # root Typer app: loads .env, add_typer × 3 in reading order — nothing else + errors.py # SHARED: SDK error → (message, hint) presentation — mode-agnostic + inputs.py # SHARED: read_text_input (text-or-file) + build_document_input (file → Document envelope) + blocking/ + __init__.py + cli.py # the whole blocking mode in one file + attended/ + __init__.py + cli.py # the whole attended mode in one file + detached/ + __init__.py + cli.py # start commands + wait/status/result lifecycle in one file + generated/ # unchanged + methods/ # unchanged +``` + +**Sharing rule:** share what is orthogonal to execution modes — input encoding (`inputs.py`) and error presentation (`errors.py`). **Never share execution lifecycle code.** The moment two mode packages share a "runner" helper, the dispatch indirection is back. The copy-paste contract, stated in each mode file's docstring: *this file + `piper/inputs.py` + `piper/errors.py`*. + +`piper/inputs.py` merges today's `file_input.py` (the `Document` envelope builder) with the `_read_text_input` text-or-file helper currently private to `cli.py`. One shared input module instead of two keeps the copy unit small. (`file_input.py` is deleted; its tests move accordingly.) + +## Anatomy of a mode package + +Every mode `cli.py` has the same four-part shape, so the three files diff cleanly: + +1. **Module docstring** — the mode's contract in one paragraph, plus the copy-paste contract. +2. **App + consoles** — its own `typer.Typer` and its own `Console()` / `Console(stderr=True)` (two lines; not worth sharing — results on stdout stay pipeable, progress goes to stderr, same convention as today). +3. **The lifecycle helper** — one public async function that *is* the mode. This is the featured code, so it gets a public name (also what unit tests patch): + - `blocking/cli.py` → `execute_pipe(...)`: `client.execute(...)`, done. + - `attended/cli.py` → `start_and_wait(...)`: `client.start(...)`, print the run id (so Ctrl-C never loses the run), then `client.wait_for_result(...)` with a Rich status line. + - `detached/cli.py` → `start_pipe(...)`: `client.start(...)`, return the id. Plus `attend_run(run_id)` backing the `wait` command, and thin fetchers backing `status` / `result`. +4. **The demo commands + a `_run()` wrapper** — each command parses its input, reads its bundle, awaits the lifecycle helper through `_run()` (asyncio.run + the one `except (PipelineRequestError, httpx.HTTPStatusError)` presenting via `piper.errors`), narrows into its generated model, prints JSON. + +### `piper/blocking/cli.py` — the proof of goal 1 + +The file a Pipelex newcomer reads first, in full (imports elided): + +```python +"""The blocking CLI — one request, one response, done. + +Simplest way to run a method: `client.execute(...)` and you have the result. +On the hosted API a run longer than ~30s is cut off (try `generate-image` to +see it) — that's what `piper attended` and `piper detached` are for. + +Copy-paste unit: this file + piper/inputs.py + piper/errors.py. +""" + +METHODS_DIR = Path(__file__).parent.parent / "methods" + +app = typer.Typer(no_args_is_help=True, help="Run a demo with a single blocking call (~30s cap on hosted).") +output_console = Console() +progress_console = Console(stderr=True) + + +async def execute_pipe(*, pipe_code: str, bundle: str, inputs: dict[str, Any]) -> Any: + """The whole blocking lifecycle: one call, the result comes back in the response.""" + async with PipelexAPIClient() as client: + with progress_console.status("Running…"): + result = await client.execute(pipe_code=pipe_code, mthds_contents=[bundle], inputs=inputs) + return result.main_stuff + + +@app.command(name="extract-entities") +def extract_entities( + text: Annotated[str | None, typer.Argument(help="The text to extract entities from.")] = None, + file: Annotated[Path | None, typer.Option("--file", help="Read the input text from a file.")] = None, +) -> None: + """Extract people, organizations, and dates from a piece of text.""" + input_text = read_text_input(text=text, file=file) + bundle = (METHODS_DIR / "extract-entities" / "main.mthds").read_text() + main_stuff = _run(execute_pipe(pipe_code="extract_entities", bundle=bundle, inputs={"text": input_text})) + entities = ExtractedEntities.model_validate(main_stuff) + output_console.print_json(data=entities.model_dump()) + +# … summarize-pdf and generate-image follow the same shape … + +def _run(coro: Coroutine[Any, Any, ResultT]) -> ResultT: + """Await the lifecycle, presenting SDK errors as clean exits.""" + try: + return asyncio.run(coro) + except (PipelineRequestError, httpx.HTTPStatusError) as exc: + presentation = present_error(exc) + progress_console.print(f"[red]Error:[/red] {presentation.message}") + if presentation.hint: + progress_console.print(f"[yellow]Hint:[/yellow] {presentation.hint}") + raise typer.Exit(1) from exc +``` + +Reading path from command to API call: `extract_entities` → `execute_pipe` → `client.execute`. Two hops, both in the same file, no enum, no dispatch. A nice simplification falls out for free: today `run_blocking` adapts its result onto `RunResults` purely so `_dispatch` could return one type across modes — with no dispatch, each mode uses its natural SDK type and the adaptation is deleted. + +### `piper/attended/cli.py` + +Same shape; only the lifecycle helper differs: + +```python +async def start_and_wait(*, pipe_code: str, bundle: str, inputs: dict[str, Any]) -> Any: + """The attended lifecycle: start a durable run, print its id, wait here for the result.""" + async with PipelexAPIClient() as client: + start_result = await client.start(pipe_code=pipe_code, mthds_contents=[bundle], inputs=inputs) + run_id = start_result.pipeline_run_id + progress_console.print(f"Run started: [bold]{run_id}[/bold]") + with progress_console.status(f"Run {run_id[:8]}… in progress") as status: + def on_poll(info: PollInfo) -> None: + status.update(f"Run {run_id[:8]}… in progress — {info.elapsed_seconds:.0f}s, poll #{info.attempt}") + try: + results = await client.wait_for_result(run_id, options=WaitForResultOptions(on_poll=on_poll)) + except asyncio.CancelledError: + progress_console.print(f"\nInterrupted — the run is still executing. Resume with: [bold]piper detached wait {run_id}[/bold]") + raise + return results.main_stuff +``` + +The docstring keeps the current teaching note: the SDK's `start_and_wait()` one-liner is the production shortcut when you don't care which path runs — this starter spells the lifecycle out because teaching it is the point. + +Note the Ctrl-C hint now points at `piper detached wait ` — and reads *better* than before: an interrupted attended run has literally become a detached run, so you resume it with the detached tooling. This cross-mode reference is a string, not an import; the packages stay decoupled. + +### `piper/detached/cli.py` + +The demo commands call `start_pipe(...)`, print the run id on stdout (pipeable — `RUN_ID=$(piper detached generate-image "…")` works) and the resume hint on stderr. The lifecycle commands complete the story: + +- `wait ` — `attend_run()`: poll to completion with the same Rich status line, print the raw `main_stuff` as JSON (generic — no per-demo narrowing, since any run id can be waited on). +- `status ` — coarse status via `client.get_run_status`, including the degraded-status caveat. +- `result ` — no-wait fetch via `client.get_run_result`, rendering the running/completed/failed states (the `RunResultState` match stays, unchanged from today's `runs result`). + +This is the "third act" file: more commands than the others, but each one is still a straight line to a single SDK call. + +## What gets deleted + +- `piper/runner.py` — dissolved into the three mode files. +- `ExecutionMode`, the `--mode` option, `ModeOption`, `MODE_HELP`, the `PIPELEX_EXECUTION_MODE` env var. +- `_dispatch()` and the `RunResults` adaptation in `run_blocking`. +- The top-level `runs` sub-app (absorbed by `detached`). +- `piper/file_input.py` (merged into `piper/inputs.py`). + +`piper/cli.py` shrinks to the root app: the `load_dotenv()` callback and three `add_typer` calls, in reading order, with one-line group help texts that teach the progression ("Start here." / "…wait here for the result." / "…collect it later."). + +## Error hints + +`piper/errors.py` stays shared and mode-agnostic, but its hints currently speak `--mode` and `piper runs`; they become group-path commands: + +- `PipelineExecuteTimeoutError` → "Rerun the same command with `piper attended …` — the durable path survives long runs." +- `RunLifecycleUnavailableError` → "You are talking to a bare runner — use `piper blocking …`." +- `RunFailedError` → "Inspect it with `piper detached status `." +- `RunTimeoutError` → "Resume waiting with `piper detached wait `." + +## Duplication policy and the drift guard + +The full matrix means the demo commands are deliberately near-duplicated across the three mode files. That duplication *is* the pedagogy — diff `blocking/cli.py` against `attended/cli.py` and the only difference is the lifecycle helper — but it can drift. Two guards: + +1. Each mode file's docstring states the mirror contract: demo commands identical across modes, only the lifecycle differs. +2. A symmetry unit test (e.g. `tests/unit/test_mode_symmetry.py`) asserts the three groups expose the same demo command names (detached additionally exposing its lifecycle commands), so a demo added to one mode and forgotten in another fails CI. + +## Tests + +- `tests/unit/test_cli.py` splits into one module per mode (one TestClass per module, per workspace rules): `test_blocking_cli.py`, `test_attended_cli.py`, `test_detached_cli.py`, plus `test_mode_symmetry.py`. Mocking gets simpler and more honest: each test patches its mode's *public* lifecycle helper (`piper.blocking.cli.execute_pipe`, `piper.attended.cli.start_and_wait`, `piper.detached.cli.start_pipe` / `attend_run`) instead of today's runner functions — no more "which runner function does this mode hit" indirection in the tests either. +- Mode-specific assertions that were previously option-flavored become group-flavored: the detached demo prints the run id on stdout; the bad-mode test (`--mode bogus`) becomes a bad-group test (unknown command → non-zero, courtesy of Typer). +- E2E suggestion: assign one lifecycle per demo so all three modes get end-to-end coverage without tripling the suite — `extract-entities` via blocking, `summarize-pdf` via attended, `generate-image` via detached (`start` then `wait`). The generate-image assignment doubles as e2e coverage of the run-id lifecycle. +- `tests/unit/test_errors.py` hint assertions update to the new command paths; `test_file_input.py` follows the module merge into `inputs.py`. + +## Packaging, bootstrap, docs + +- `pyproject.toml`: add `piper.blocking`, `piper.attended`, `piper.detached` to `[tool.setuptools] packages`. `[project.scripts]` unchanged — still the single `piper = "piper.cli:app"` entry, which is what keeps the `/bootstrap` rename trivial (verify the bootstrap script's package rename handles the nested `piper.*` imports; it should, since it rewrites the import prefix globally). +- README: the quick start becomes `piper blocking extract-entities "…"`; the "Execution modes" section becomes a three-act tour matching the reading order (blocking → the timeout → attended → detached), reusing the existing mermaid sequence diagrams one-per-group; the project-structure block and "Useful commands" update to the new paths. +- `CLAUDE.md` (starter repo): the architecture bullet describing `runner.py` / `ExecutionMode` / `--mode` is rewritten around the three sub-packages and the sharing rule. +- CHANGELOG (at implementation time): breaking — `--mode` and `PIPELEX_EXECUTION_MODE` are gone, the mode is now the command group (`piper blocking|attended|detached …`); `piper runs …` is now `piper detached wait|status|result`. + +## Rejected alternatives (summary) + +- **Three console scripts** (`piper-blocking`, …) — real separation, but three pyproject entries to bootstrap-rename and no single `--help` telling the story. +- **`durable` as the middle group's name** — detached runs are durable too; `attended`/`detached` names the actual axis (who waits). +- **Curated demo spread** (each mode carrying only the demos that motivate it) — shorter files, but breaks the diffability of the full matrix and forbids legitimate combinations (e.g. summarize-pdf detached). +- **Top-level `piper runs` group** — "execute vs inspect" separation is principled, but it splits the detached story across two places, which is exactly the scatter this redesign removes. +- **A shared runner module called by all three packages** — recreates the indirection under a new name; lifecycle code is never shared, period. + +## Open questions + +- **`main_stuff` typing in the lifecycle helpers:** the helpers return the SDK-resolved `main_stuff` (typed as the SDK types it). If the SDK's typing is loose (`Any`), the generated-model `model_validate` narrowing in each command is what restores type safety — acceptable, but worth a look at the SDK types during implementation. +- **Group help ordering:** Typer lists groups in `add_typer` order with default settings — confirm at implementation time (if it alphabetizes, blocking/attended/detached conveniently still reads okay, but verify). +- **`piper detached result` vs `wait` output parity:** both print raw `main_stuff` JSON; keep the rendering identical so scripts can use either interchangeably. From f13e6193119f29221d24707cc9200ed1f931a0ce Mon Sep 17 00:00:00 2001 From: Louis Choquel Date: Wed, 15 Jul 2026 11:11:35 +0200 Subject: [PATCH 2/3] fix: address PR #62 review feedback for zero-argument CLI demos (#63) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: generate the typed clients from the bundles via pipelex codegen Delete the hand-written output models: pipelex codegen projects each method's concepts into piper/generated//models.py (stamped, locked by a sibling codegen.lock); the examples keep only the bundle path, pipe code, and the parse() narrower. generate-image now parses into the generated built-in Image model. New make codegen (regen, write-if-changed) + make codegen-check (offline drift check) targets — the pipelex CLI is not a dependency, point the PIPELEX make variable at an install that ships codegen; CI wiring is release-gated on a published pipelex. Adds inputs.template.json scaffolds beside each bundle, offline smoke tests for the generated clients, and docs/codegen.md. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EaRwjPbPsan4rCeyWirKpp * fix(devx): Checkpoint C review — ship codegen.lock in wheel + lockstep comments Cold-review triage: package each codegen.lock as package data so the offline drift check also works against an installed copy, and cross-reference the three places the method list is enumerated (piper/methods/*, pyproject packages, the Makefile codegen targets). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EaRwjPbPsan4rCeyWirKpp * chore(codegen): regenerate typed clients — pinned native definitions + flattened Image Regenerated with the codegen engine after the native-materialization change: Image now emits width/height integer fields instead of the interim dict-with-imprecision size, and native field descriptions come from the pinned normative definitions (storage-neutral url wording, described Text). Crate fingerprints and stamps updated accordingly; codegen-check green. Co-Authored-By: Claude Fable 5 * docs: changelog — call out the generated Image shape change (width/height) Co-Authored-By: Claude Fable 5 * chore(Makefile): update PIPELEX_RUN to streamline codegen invocations * simplify * feat(cli): update execution modes to use a single --mode option; remove --detach flag * feat: remove legacy file input and runner modules; introduce new test suite for document input handling * feat: introduce detached CLI mode for durable runs - Added `piper.detached` package with CLI commands for starting and managing durable runs. - Implemented commands: `extract-entities`, `summarize-pdf`, `generate-image`, `wait`, `status`, and `result`. - Created shared input handling functions in `piper.inputs` for text and document inputs. - Enhanced error presentation in `piper.errors` to provide mode-specific hints. - Updated tests to cover the new detached mode and ensure symmetry across CLI modes. - Refactored existing tests to align with the new structure and added new unit tests for input handling. * feat: enhance CLI demos to run with zero arguments using built-in samples - Updated all demo commands to utilize bundled sample inputs when no arguments are provided, ensuring a working result on the first command execution. - Introduced a `sample` parameter in `read_text_input()` to facilitate this feature. - Enhanced error handling to surface detailed API error messages instead of generic httpx errors, improving user feedback. - Refactored execution modes into self-contained units, removing the previous shared dispatch structure for clarity and maintainability. - Updated documentation to reflect changes in execution modes and sample usage. - Added tests to verify that demos correctly fall back to sample inputs when no user input is provided. * fix: address PR #62 review feedback for zero-argument CLI demos Guards bootstrap against package names that collide with the piper placeholder, and refines sample-input/error handling in the attended CLI mode with corresponding test coverage updates. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01AA5fmhTMLRLTJgVmCsW6da * fix: bootstrap rewrites Makefile codegen paths and docs The bootstrap script's rewrite set never included the Makefile, so the codegen targets kept pointing at piper/generated and piper/methods after a rename, breaking make codegen / make codegen-check on bootstrapped projects. Same gap for the docs under docs/, which went stale. Both are now in the rewrite set (the generic context-aware token pass handles their path/command forms), SKILL.md lists them, and a non-dry-run test asserts the rewritten paths. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01AA5fmhTMLRLTJgVmCsW6da --------- Co-authored-by: Louis Choquel <8851983+lchoquel@users.noreply.github.com> Co-authored-by: Claude Fable 5 --- .claude/skills/bootstrap/SKILL.md | 2 +- .claude/skills/bootstrap/scripts/bootstrap.py | 10 +++ CHANGELOG.md | 2 +- CLAUDE.md | 2 +- README.md | 8 +-- docs/cli-architecture.md | 18 +++--- docs/codegen.md | 4 +- piper/attended/cli.py | 5 +- piper/errors.py | 4 +- piper/inputs.py | 6 +- tests/e2e/test_extract_entities.py | 2 +- tests/unit/test_bootstrap_script.py | 62 ++++++++++++++++++- tests/unit/test_errors.py | 10 +++ tests/unit/test_inputs.py | 6 ++ wip/pr-62-review-notes.md | 18 ++++++ 15 files changed, 135 insertions(+), 24 deletions(-) create mode 100644 wip/pr-62-review-notes.md diff --git a/.claude/skills/bootstrap/SKILL.md b/.claude/skills/bootstrap/SKILL.md index e49cb33..ed78a1c 100644 --- a/.claude/skills/bootstrap/SKILL.md +++ b/.claude/skills/bootstrap/SKILL.md @@ -71,7 +71,7 @@ The dry run prints the package-directory rename and the list of files that would Re-run the exact same command **without** `--dry-run`. The script: - renames `piper/` → `/` (via `git mv`, so history follows). The e2e test file is named after its demo (`tests/e2e/test_extract_entities.py`), not the project, so nothing in `tests/` is renamed — only its contents are edited. -- substitutes the name across `pyproject.toml`, `README.md`, `CLAUDE.md`, the package's `.py`/`.mthds` files, and the tests — using context-aware rules: the dash dist form in CLI-command / distribution positions, the underscore package form in imports and paths, the title in prose. `pyproject.toml` is handled with targeted per-key edits (its package-list arrays need the package form in a bare-string position). It then asserts no placeholder token survived, and aborts with the offending locations if one did. +- substitutes the name across `pyproject.toml`, `README.md`, `CLAUDE.md`, the `Makefile` (the codegen targets point at `/generated` and `/methods` paths), the package's `.py`/`.mthds` files, the tests, and the docs under `docs/` — using context-aware rules: the dash dist form in CLI-command / distribution positions, the underscore package form in imports and paths, the title in prose. `pyproject.toml` is handled with targeted per-key edits (its package-list arrays need the package form in a bare-string position). It then asserts no placeholder token survived, and aborts with the offending locations if one did. - fills in description, and (if given) author, repo URL - applies the license choice in all three places: the `LICENSE` body, `license = "..."` in `pyproject.toml`, and the README license line - strips the README template block diff --git a/.claude/skills/bootstrap/scripts/bootstrap.py b/.claude/skills/bootstrap/scripts/bootstrap.py index c7401cf..05d0ff9 100644 --- a/.claude/skills/bootstrap/scripts/bootstrap.py +++ b/.claude/skills/bootstrap/scripts/bootstrap.py @@ -81,6 +81,14 @@ def validate_package(package: str) -> None: ) if keyword.iskeyword(package): sys.exit(f"Invalid package name {package!r}: it is a Python keyword.") + # A name like `piper_tools` re-matches the placeholder regex after its own + # insertion (the pyproject transform would corrupt it into `piper_tools_tools`), + # so refuse it up front. Names merely embedding the token (e.g. `sandpiper`) + # have no word boundary before `piper` and stay allowed. + if PIPER_PACKAGE_RE.search(package): + sys.exit( + f"Invalid package name {package!r}: it collides with the template's 'piper' placeholder — pick a name that doesn't start with 'piper_'." + ) def validate_dist(dist: str) -> None: @@ -441,12 +449,14 @@ def gather_target_files(root: Path, names: Names) -> list[Path]: root / "pyproject.toml", root / "README.md", root / "CLAUDE.md", + root / "Makefile", root / "LICENSE", ] pkg_dir = root / names.package candidates += sorted(pkg_dir.rglob("*.py")) candidates += sorted(pkg_dir.rglob("*.mthds")) candidates += sorted((root / "tests").rglob("*.py")) + candidates += sorted((root / "docs").rglob("*.md")) seen: set[Path] = set() files: list[Path] = [] for path in candidates: diff --git a/CHANGELOG.md b/CHANGELOG.md index f4a8d7d..8850b53 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ - **Each execution mode is now a self-contained copy-paste unit.** A mode used to be an `ExecutionMode` value threaded through a dispatch chain (CLI command → `_dispatch()` → `match` → `piper/runner.py` → the SDK), so answering "how do I call the API?" meant hopping through four layers, and the `match` existed only to showcase all three lifecycles behind one option — nothing anyone would copy. Now each mode is one file (`piper/blocking/cli.py`, `piper/attended/cli.py`, `piper/detached/cli.py`) holding its commands, its SDK lifecycle, and its progress rendering, with one public lifecycle helper that *is* the mode (`execute_pipe` / `start_and_wait` / `start_pipe`). `piper/runner.py`, `ExecutionMode`, `_dispatch()`, and the `RunResults` adaptation that existed only to give the dispatch one return type are all deleted; `piper/cli.py` shrinks to a `load_dotenv` callback plus three `add_typer` calls. New: `docs/cli-architecture.md`. - **Only mode-orthogonal code is shared: `piper/inputs.py` (new) and `piper/errors.py`.** `piper/file_input.py` is merged into `piper/inputs.py`, which now also owns the text-or-file input helper (`read_text_input`). Lifecycle code is never shared — the demo commands are deliberately near-duplicated across the three mode files (diff two of them and only the lifecycle helper differs), and `tests/unit/test_mode_symmetry.py` guards that duplication against drift. The hints in `piper/errors.py` now name the mode groups (a blocking run that hits the ~30s cap points at `piper attended`; an interrupted attended run points at `piper detached wait `, which is exactly what it has become). - **The typed output models are now generated from the bundles, not hand-written.** `pipelex codegen` projects each method's concepts into `piper/generated//models.py` (a stamped module plus a `codegen.lock` recording the artifact set). `generate-image` parses into the generated built-in `Image` model (natives are materialized into the generated client), replacing the hand-written `GeneratedImage`. -- **Removed the `piper/examples/` layer.** Each demo is now one self-contained command in `piper/cli.py`: bundle path (via a single `METHODS_DIR` constant), pipe code, and inline narrowing into the generated model (`Model.model_validate(results.main_stuff)`) — no more per-demo wrapper module hiding that line behind a `parse()` indirection. The `parse()` unit tests went with it (they only exercised pydantic's `model_validate`); the e2e tests narrow inline the same way. +- **Removed the `piper/examples/` layer.** Each demo is now one self-contained command in its mode file (`piper//cli.py`): bundle path (each mode file owns its `METHODS_DIR` constant), pipe code, and inline narrowing into the generated model (`Model.model_validate(main_stuff)`) — no more per-demo wrapper module hiding that line behind a `parse()` indirection. The `parse()` unit tests went with it (they only exercised pydantic's `model_validate`); the e2e tests narrow inline the same way. - **Fixed the `/bootstrap` pyproject transform.** Its per-key edits had gone stale against the current `pyproject.toml` (multi-line `packages` array with the `piper.generated.*` subpackages, quoted package-data keys, the `piper/generated` ruff exclude), so bootstrapping aborted on the survivor check. Package-name tokens in pyproject are now rewritten generically (quoted-exact + dotted/path positions), which no longer goes stale when the package list changes; the bootstrap test fixture now mirrors the real pyproject shapes. - **New make targets:** `make codegen` regenerates the typed clients and the runnable `inputs.template.json` scaffolds beside each bundle; `make codegen-check` verifies offline (pure hashing, no network or API key) that the generated clients are current. The `pipelex` CLI is not a starter dependency — point the `PIPELEX` make variable at a pipelex install that ships `codegen`. CI wiring of `codegen-check` lands once a released pipelex ships the command. - **New offline smoke tests** (`tests/unit/test_generated_clients.py`): the generated modules import, carry the stamp + lock, round-trip their serialization, and each committed input template names exactly the inputs the CLI dispatches. diff --git a/CLAUDE.md b/CLAUDE.md index 0a67b5a..75d2f51 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -40,7 +40,7 @@ This starter calls the **hosted Pipelex API** via the `pipelex-sdk` package (`Pi - **The execution mode is the command group, not an option.** There are exactly three, each a self-contained Typer sub-package: `piper blocking …` (`client.execute` — one call, dies at the hosted ~30s cap), `piper attended …` (`client.start` + `client.wait_for_result` — durable, you wait), `piper detached …` (`client.start` only — durable, you collect it later with `piper detached status|result|wait `). Attended and detached start the *same* durable run; the axis they name is who waits. There is no default mode and no `--mode` option: `piper/cli.py` is a thin assembler (`load_dotenv` callback + three `add_typer` calls in reading order) and nothing else. - **Each mode file is a copy-paste unit; lifecycle code is never shared.** `piper//cli.py` holds that mode's whole story: its `typer.Typer`, its consoles (results → stdout, progress → stderr), its one public lifecycle helper (`execute_pipe` / `start_and_wait` / `start_pipe`, plus `attend_run` + the fetchers in detached), its demo commands, and a private `_run()` that wraps `asyncio.run` and catches SDK errors once. The **only** shared modules are the two that are orthogonal to execution: `piper/inputs.py` (text-or-file input with a built-in **sample fallback** so every demo runs with zero arguments — `read_text_input` returns `TextInput(text, is_sample)` and the demo prints a stderr notice when the sample was used; plus file → `{"concept": "Document", "content": …}` envelope with the file base64-encoded into a `data:` URL, and the `SAMPLE_*` constants) and `piper/errors.py` (SDK error → message + hint, hints naming the mode groups; it reads the RFC 7807 **problem+json** body off raw protocol-route `httpx.HTTPStatusError`s and branches on the structured `error_type`, e.g. `StartRequiresAsyncOrchestration` → "use `piper blocking`"). Do not introduce a shared runner — the dispatch indirection is exactly what this layout removed. See `docs/cli-architecture.md`. - **Full demo matrix, guarded.** All three demos exist in all three modes: `extract-entities` (text in), `summarize-pdf` (a *file* in), `generate-image` (prompt in). `generate-image` is the deliberate slow case that overruns the ~30s blocking cap — `piper blocking generate-image` is *expected to fail*, and that is the teaching moment for the durable modes. The near-duplication across mode files is the pedagogy (diff two mode files and only the lifecycle helper differs); `tests/unit/test_mode_symmetry.py` keeps it from drifting. `samples/sample-invoice.pdf` is shipped for `summarize-pdf`. -- The SDK resolves the main output on every path (`client.execute` returns a `PipelexExecuteResult`, the durable path a `RunResults`, both exposing a resolved `.main_stuff`, typed `Any`; a completed run with no main stuff raises `MissingMainStuffError`). So every lifecycle helper returns `main_stuff`, and each demo command narrows it inline — e.g. `ExtractedEntities.model_validate(main_stuff)` — into the generated model. There is no per-example wrapper layer. +- The SDK resolves the main output on every result-producing path (`client.execute` returns a `PipelexExecuteResult`, the durable path a `RunResults`, both exposing a resolved `.main_stuff`, typed `Any`; a completed run with no main stuff raises `MissingMainStuffError`). So the result-producing lifecycle helpers (`execute_pipe`, `start_and_wait`, detached's `attend_run`) return `main_stuff`, and the blocking/attended demo commands narrow it inline — e.g. `ExtractedEntities.model_validate(main_stuff)` — into the generated model. Detached is the exception by design: `start_pipe` returns only the run id (the demos print it bare), and the run-id commands (`wait`/`result`) print the output generically — no model narrowing, since at collection time the command doesn't know which method the run executed. There is no per-example wrapper layer. - The modes spell out lifecycles the SDK could hide: `client.start_and_wait()` is a self-healing one-liner that picks the path for you (the production shortcut). The starter writes them out because teaching the difference is the point. - **The typed models are generated, never hand-written.** `pipelex codegen` projects each bundle's concepts into `piper/generated//models.py` (stamped, locked by a sibling `codegen.lock`); the mode CLIs and the e2e tests import from there. Do NOT edit generated files — edit the bundle, then `make codegen` (regenerates models + `inputs.template.json` scaffolds) and `make codegen-check` (offline drift check). The `pipelex` CLI is not a dependency of this starter; point the `PIPELEX` make variable at a pipelex install that ships `codegen`. `piper/generated` is excluded from ruff (reformatting would trip the drift check) but fully type-checked. See `docs/codegen.md`. diff --git a/README.md b/README.md index 8b27750..f374394 100644 --- a/README.md +++ b/README.md @@ -168,10 +168,10 @@ The other demos run through the exact same path — they differ only in their in **The mode is the command group, not an option.** There are three, and every demo runs in all three: ```bash -uv run piper blocking extract-entities | summarize-pdf | generate-image -uv run piper attended extract-entities | summarize-pdf | generate-image -uv run piper detached extract-entities | summarize-pdf | generate-image -uv run piper detached wait | status | result +uv run piper blocking extract-entities # or summarize-pdf, generate-image +uv run piper attended extract-entities # or summarize-pdf, generate-image +uv run piper detached extract-entities # or summarize-pdf, generate-image +uv run piper detached wait # or status, result ``` Read them in that order — each one is a single self-contained file (`piper//cli.py`) you can copy straight into your own project. diff --git a/docs/cli-architecture.md b/docs/cli-architecture.md index 9bcc120..051432c 100644 --- a/docs/cli-architecture.md +++ b/docs/cli-architecture.md @@ -5,10 +5,10 @@ This starter has two jobs: show how easy it is to call the Pipelex API, and be c So the execution mode is not an option you pass, it is the command group you type: ``` -piper blocking extract-entities | summarize-pdf | generate-image -piper attended extract-entities | summarize-pdf | generate-image -piper detached extract-entities | summarize-pdf | generate-image -piper detached wait | status | result +piper blocking extract-entities # or summarize-pdf, generate-image +piper attended extract-entities # or summarize-pdf, generate-image +piper detached extract-entities # or summarize-pdf, generate-image +piper detached wait # or status, result ``` ## The layout @@ -23,7 +23,7 @@ piper/ detached/cli.py # the whole detached mode + the run-id lifecycle commands ``` -Each mode file is a **copy-paste unit**: that file plus `inputs.py` plus `errors.py` is everything you need to lift the mode into your own project. Nothing else in `piper/` is load-bearing for it. +Each mode file is a **copy-paste unit**: that file plus `inputs.py` plus `errors.py` is all the *lifecycle* code you need to lift the mode into your own project. A runnable copy also carries the method-specific artifacts the demos reference — the bundles (`piper/methods/`), the generated models (`piper/generated/`), and the sample document — but those are exactly what you replace with your own method anyway. No other code in `piper/` is load-bearing for a mode. ## The sharing rule @@ -41,7 +41,7 @@ All three have the same four-part shape, so they diff cleanly: 1. **Module docstring** — the mode's contract in a paragraph, plus its copy-paste contract. 2. **App + consoles** — its own `typer.Typer`, its own `Console()` (stdout, for results — pipeable) and `Console(stderr=True)` (stderr, for progress chatter). -3. **The lifecycle helper** — one public async function that *is* the mode. It gets a public name because it is the featured code, and it is what the unit tests patch and the e2e tests call directly. +3. **The lifecycle helper** — one public async function that *is* the mode (`detached` adds the run-id lifecycle helpers described below). It gets a public name because it is the featured code, and it is what the unit tests patch and the e2e tests call directly. 4. **The demo commands + a private `_run()`** — each command reads its input, reads its bundle, awaits the lifecycle helper through `_run()` (`asyncio.run` + the single `except (PipelineRequestError, httpx.HTTPStatusError)` that presents via `piper/errors.py`), narrows the result into its *generated* model, prints JSON. | Mode | Lifecycle helper | SDK calls | What the demo prints | @@ -52,19 +52,19 @@ All three have the same four-part shape, so they diff cleanly: `detached` additionally owns the run-id lifecycle: `attend_run()` backs `wait`, and thin fetchers back `status` and `result`. It is the only mode with more than the demos, because "start now, collect later" is only a complete story if you can come back for the result. -All three lifecycle helpers return the run's resolved `main_stuff` (the SDK types it `Any` — the content is polymorphic), and each command's `Model.model_validate(main_stuff)` is what restores type safety. The models are generated from the bundles; see [codegen.md](codegen.md). +The result-producing helpers — `execute_pipe()`, `start_and_wait()`, and detached's `attend_run()` — return the run's resolved `main_stuff` (the SDK types it `Any` — the content is polymorphic). In blocking and attended mode each demo command narrows it with `Model.model_validate(main_stuff)`, which is what restores type safety; the models are generated from the bundles (see [codegen.md](codegen.md)). Detached is different by design: `start_pipe()` returns only the run id, and the run-id commands (`wait`, `result`) print the output generically — at collection time the command doesn't know which method the run executed, so there is no model to narrow into. ## Two conventions worth copying **stdout is the result; stderr is everything else.** Progress spinners, run ids in attended mode, error messages, and hints all go to stderr, so stdout stays pipeable. In detached mode the run id *is* the result, so it goes to stdout bare (`print`, not Rich) — `RUN_ID=$(piper detached generate-image "…")` just works. -**Errors are caught once, at the root of the command.** `_run()` catches `PipelineRequestError` (the base of every error the SDK client raises) and the raw `httpx.HTTPStatusError` its protocol routes surface, maps it to a `(message, hint)` pair via `piper/errors.py`, and exits non-zero. Nothing else is caught anywhere: an unexpected exception crashes loudly with its traceback, which is what you want while you are building. +**SDK errors are presented once, at the root of the command.** `_run()` catches `PipelineRequestError` (the base of every error the SDK client raises) and the raw `httpx.HTTPStatusError` its protocol routes surface, maps it to a `(message, hint)` pair via `piper/errors.py`, and exits non-zero. Ctrl-C is handled separately: the durable lifecycle helpers catch the cancellation just long enough to print the resume hint before re-raising, and `_run()` maps the resulting `KeyboardInterrupt` to exit 130. Beyond those two, nothing is caught: an unexpected exception crashes loudly with its traceback, which is what you want while you are building. The protocol routes (`execute`/`start`/`runs/*`) surface a non-2xx as a raw `httpx.HTTPStatusError`, whose default string is useless (`Client error '400 Bad Request' for url …` + an MDN link). `piper/errors.py` instead reads the API's RFC 7807 **problem+json** body and shows the server's own `detail`, branching on the structured `error_type` (never the transport status) for the cases worth a hint — a `/start` against a synchronous-only runner (`StartRequiresAsyncOrchestration`) is presented with a hint pointing at `piper blocking`. The hints name the mode *groups*, because the fix for a failed run is usually another group: a blocking run that hit the ~30s cap tells you to rerun it with `piper attended`; a run that timed out while you waited tells you to resume it with `piper detached wait `; a durable run against a runner that can't do them tells you to use `piper blocking`. -**Every demo runs with zero arguments.** When you give neither an argument nor `--file`, the input helper returns a bundled sample (`piper/inputs.py`'s `SAMPLE_*` constants), and the command prints a one-line notice on stderr saying so. A fresh clone shows a working result on its very first command; stdout stays the clean, pipeable result because the notice is on stderr. Sample data is orthogonal to execution, so like input encoding it is shared, not duplicated per mode. +**Every demo runs with zero arguments.** When you give neither an argument nor `--file`, the input helper returns a bundled sample (`piper/inputs.py`'s `SAMPLE_*` constants), and the command prints a one-line notice on stderr saying so. A fresh clone shows a working result on its very first command once your API key is set; stdout stays the clean, pipeable result because the notice is on stderr. Sample data is orthogonal to execution, so like input encoding it is shared, not duplicated per mode. ## Why `attended` and `detached`, not `durable` diff --git a/docs/codegen.md b/docs/codegen.md index 488e75d..92165e3 100644 --- a/docs/codegen.md +++ b/docs/codegen.md @@ -38,8 +38,8 @@ PIPELEX=/path/to/pipelex/.venv/bin/pipelex make codegen PIPELEX=/path/to/pipelex/.venv/bin/pipelex make codegen-check ``` -Once a released `pipelex` ships `codegen`, `codegen-check` is meant to run in CI so a bundle edit can never land without its regenerated client. +Once a released `pipelex` ships `codegen`, `codegen-check` is meant to run in CI to catch stale, hand-edited, or orphaned generated artifacts. It does not resolve bundles, so a bundle edit that was never regenerated still needs the other guard: run `make codegen` and fail on a dirty `git diff` (see above). ## Offline test floor -`tests/unit/test_generated_clients.py` runs everywhere (no pipelex CLI, no API key): it checks the generated modules import, carry the stamp + lock, round-trip their own serialization, and that each committed input template names exactly the inputs the CLI passes — so a regenerated template that no longer matches what `piper//cli.py` sends fails in CI even before `codegen-check` is wired in. +`tests/unit/test_generated_clients.py` runs everywhere (no pipelex CLI, no API key): it checks the generated modules import, carry the stamp + lock, round-trip their own serialization, and that each committed input template names exactly the inputs of the test's checked-in contract (its `METHODS` map, which mirrors the inputs each `piper//cli.py` sends) — so a regenerated template that drifts from that contract fails in CI even before `codegen-check` is wired in. diff --git a/piper/attended/cli.py b/piper/attended/cli.py index 73d3b65..32cde60 100644 --- a/piper/attended/cli.py +++ b/piper/attended/cli.py @@ -3,8 +3,9 @@ `client.start(...)` submits the run server-side (it survives anything, including the ~30s cap that kills `piper blocking`), and `client.wait_for_result(...)` polls it to completion from this terminal. The run id is printed *before* polling starts, -so a Ctrl-C never loses the run: it keeps executing server-side and you resume it -with `piper detached wait `. +so once you see it, a Ctrl-C doesn't lose the run: it keeps executing server-side and +you resume it with `piper detached wait `. (A Ctrl-C while the start request is +still in flight is the one window with no id to resume from.) The SDK also offers `start_and_wait()`, a self-healing one-liner that picks the right path by itself — that is the production shortcut. This starter spells the diff --git a/piper/errors.py b/piper/errors.py index 4952888..6128925 100644 --- a/piper/errors.py +++ b/piper/errors.py @@ -116,7 +116,9 @@ def _read_problem_json(response: httpx.Response) -> dict[str, Any]: """Best-effort parse of an RFC 7807 problem+json body; `{}` when it isn't JSON.""" try: body: Any = response.json() - except json.JSONDecodeError: + except (json.JSONDecodeError, UnicodeDecodeError): + # UnicodeDecodeError: `response.json()` is `json.loads(response.content)`, + # which raises it (not JSONDecodeError) on a non-UTF-8 body. return {} if isinstance(body, dict): # JSON object keys are always strings. diff --git a/piper/inputs.py b/piper/inputs.py index 9e0c1bf..d753139 100644 --- a/piper/inputs.py +++ b/piper/inputs.py @@ -51,12 +51,16 @@ def read_text_input(*, text: str | None, file: Path | None, sample: str) -> Text command runs with no arguments at all. Raises: - typer.BadParameter: both an argument and `--file` were given. + typer.BadParameter: both an argument and `--file` were given, or `--file` + does not point at a readable file. """ if text is not None and file is not None: msg = "Give the text either as an argument or via --file, not both." raise typer.BadParameter(msg) if file is not None: + if not file.is_file(): + msg = f"No such file: {file}" + raise typer.BadParameter(msg) return TextInput(text=file.read_text(), is_sample=False) if text is not None: return TextInput(text=text, is_sample=False) diff --git a/tests/e2e/test_extract_entities.py b/tests/e2e/test_extract_entities.py index 8bb017a..24712bb 100644 --- a/tests/e2e/test_extract_entities.py +++ b/tests/e2e/test_extract_entities.py @@ -16,7 +16,7 @@ class TestExtractEntities: async def test_blocking(self): # The blocking lifecycle end to end: one `execute` call, then narrow. - # Extraction finishes well under the hosted ~30s cap, so it is the demo blocking mode owns. + # Extraction finishes well under the hosted ~30s cap, so the blocking mode owns this demo. bundle = BUNDLE_PATH.read_text() main_stuff = await execute_pipe(pipe_code="extract_entities", bundle=bundle, inputs={"text": SAMPLE_TEXT}) entities = ExtractedEntities.model_validate(main_stuff) diff --git a/tests/unit/test_bootstrap_script.py b/tests/unit/test_bootstrap_script.py index 67d632f..3f9cd5c 100644 --- a/tests/unit/test_bootstrap_script.py +++ b/tests/unit/test_bootstrap_script.py @@ -67,6 +67,16 @@ def write_template(root: Path, *, extra_pyproject: str = "") -> None: encoding="utf-8", ) (root / "README.md").write_text("# Piper\n\nRun `piper extract-entities`.\n", encoding="utf-8") + (root / "Makefile").write_text( + "codegen:\n" + "\t@$(PIPELEX_RUN) codegen types --target python-pydantic --output piper/generated/extract_entities piper/methods/extract-entities\n" + "\n" + "codegen-check:\n" + "\t@$(PIPELEX_RUN) codegen check piper/generated/extract_entities\n", + encoding="utf-8", + ) + (root / "docs").mkdir() + (root / "docs" / "codegen.md").write_text("Generated models live in `piper/generated//models.py`.\n", encoding="utf-8") (root / "CLAUDE.md").write_text("The `piper` CLI lives in `piper/cli.py`.\n", encoding="utf-8") (root / "LICENSE").write_text("MIT License\n\nCopyright (c) 2025 Example\n", encoding="utf-8") (root / "piper" / "cli.py").write_text('"""The piper CLI."""\n', encoding="utf-8") @@ -75,11 +85,30 @@ def write_template(root: Path, *, extra_pyproject: str = "") -> None: (root / "tests" / "test_cli.py").write_text("from piper.cli import app\n", encoding="utf-8") +def test_validate_package_rejects_placeholder_colliding_names() -> None: + # A package like `piper_tools` re-matches the placeholder regex after its own + # insertion (piper_tools -> piper_tools_tools in pyproject), so it is refused + # up front instead of producing a corrupted, non-building project. + bootstrap = load_bootstrap() + + with pytest.raises(SystemExit) as exc_info: + bootstrap.validate_package("piper_tools") + + assert "piper" in str(exc_info.value) + + +def test_validate_package_allows_names_embedding_piper() -> None: + # No word boundary before "piper" here, so it never collides with the placeholder. + bootstrap = load_bootstrap() + + bootstrap.validate_package("sandpiper_tools") + + def test_survivor_check_allows_requested_values_containing_piper(tmp_path: Path) -> None: bootstrap = load_bootstrap() write_template(tmp_path) - names = bootstrap.Names(dist="piper-tools", package="piper_tools", title="Piper Tools") + names = bootstrap.Names(dist="sandpiper-tools", package="sandpiper_tools", title="Sandpiper Tools") opts = bootstrap.Options( description="Build Piper workflows", author_name="Piper Team", @@ -94,6 +123,37 @@ def test_survivor_check_allows_requested_values_containing_piper(tmp_path: Path) bootstrap.run(tmp_path, names, opts) +def test_run_rewrites_makefile_and_docs_paths(tmp_path: Path) -> None: + # The Makefile codegen targets and the docs reference piper/... paths; a + # bootstrap that skips them leaves `make codegen` pointing at a directory + # that no longer exists on the renamed project. + bootstrap = load_bootstrap() + write_template(tmp_path) + + names = bootstrap.Names(dist="invoice-extractor", package="invoice_extractor", title="Invoice Extractor") + opts = bootstrap.Options( + description="Extract invoice fields", + author_name=None, + author_email=None, + repo_url=None, + lic=bootstrap.License(kind="mit", spdx="MIT", holder=None, year=2026), + clean=False, + dry_run=False, + use_git=False, + ) + + bootstrap.run(tmp_path, names, opts) + + makefile = (tmp_path / "Makefile").read_text(encoding="utf-8") + assert "--output invoice_extractor/generated/extract_entities invoice_extractor/methods/extract-entities" in makefile + assert "codegen check invoice_extractor/generated/extract_entities" in makefile + assert "piper" not in makefile + + docs = (tmp_path / "docs" / "codegen.md").read_text(encoding="utf-8") + assert "invoice_extractor/generated//models.py" in docs + assert "piper" not in docs + + def test_survivor_check_still_rejects_unhandled_template_tokens(tmp_path: Path) -> None: bootstrap = load_bootstrap() # A bare `piper` word inside prose (not quoted-exact, not in package position) diff --git a/tests/unit/test_errors.py b/tests/unit/test_errors.py index f469cb0..2bfc167 100644 --- a/tests/unit/test_errors.py +++ b/tests/unit/test_errors.py @@ -58,6 +58,16 @@ def test_start_without_async_orchestration_hints_blocking(self): assert presentation.hint is not None assert "piper blocking" in presentation.hint + def test_http_error_with_undecodable_body_falls_back_to_status(self): + # A non-UTF-8 body makes `response.json()` raise UnicodeDecodeError (not + # JSONDecodeError); the best-effort problem+json parse must still fall + # back to the status-only message instead of crashing mid-presentation. + request = httpx.Request("POST", "https://api.pipelex.com/v1/start") + response = httpx.Response(400, request=request, headers={"content-type": "application/problem+json"}, content=b"\xffnot-json") + presentation = present_error(httpx.HTTPStatusError("boom", request=request, response=response)) + assert presentation.message == "The API answered 400 Bad Request." + assert presentation.hint is None + def test_http_error_surfaces_the_problem_detail(self): problem = {"title": "Bad input", "detail": "Missing required input 'text'.", "status": 400} presentation = present_error(_http_status_error(400, problem=problem)) diff --git a/tests/unit/test_inputs.py b/tests/unit/test_inputs.py index d87fe8e..0e11583 100644 --- a/tests/unit/test_inputs.py +++ b/tests/unit/test_inputs.py @@ -26,6 +26,12 @@ def test_read_text_input_rejects_both(self, tmp_path: Path): with pytest.raises(typer.BadParameter): read_text_input(text="inline text", file=input_file, sample="the sample") + def test_read_text_input_rejects_a_missing_file(self, tmp_path: Path): + # Same clean CLI error as the `summarize-pdf` document path — a mistyped + # --file must not surface as a raw FileNotFoundError traceback. + with pytest.raises(typer.BadParameter): + read_text_input(text=None, file=tmp_path / "nope.txt", sample="the sample") + def test_read_text_input_falls_back_to_the_sample(self): resolved = read_text_input(text=None, file=None, sample="the sample") assert resolved.text == "the sample" diff --git a/wip/pr-62-review-notes.md b/wip/pr-62-review-notes.md new file mode 100644 index 0000000..0bca037 --- /dev/null +++ b/wip/pr-62-review-notes.md @@ -0,0 +1,18 @@ +# PR #62 review — deferred items + +Review-agent findings on [PR #62](https://github.com/Pipelex/pipelex-starter-python/pull/62) that were verified as real but deliberately **not** fixed in the PR. Each entry records why it was deferred and what a future fix would look like. + +## Sample PDF is not packaged into a wheel + +- **Reporters:** greptile-apps (`piper/inputs.py:33`), cubic-dev-ai (`piper/attended/cli.py:94`) +- **Finding (verified):** `SAMPLE_INVOICE` resolves to the repo-root `samples/sample-invoice.pdf` via `Path(__file__).parent.parent`, and `pyproject.toml` only ships package data under `piper.*`. On a true wheel install, the zero-argument `summarize-pdf` demo would fail with `FileNotFoundError`. +- **Why deferred:** this repo is a template — README says "Use this template", install is `make install` (uv, editable), and wheel distribution is not a supported or advertised path. The repo-root `samples/` location is deliberately user-facing (README documents `samples/sample-invoice.pdf` as a path to inspect and replace). Fixing this would move the sample under `piper/` (importlib.resources + package-data) and regress that documented UX for a scenario nobody uses. +- **Revisit if:** the template ever starts being consumed as an installed wheel. + +## Ctrl-C during the initial `client.start` can lose the run id + +- **Reporter:** cubic-dev-ai (`piper/attended/cli.py:54`) +- **Finding (verified):** a Ctrl-C landing while the `client.start` request is in flight — after the server committed the run but before the client printed the id — leaves a durable run with no id to resume from. The window is real in both attended and detached modes. +- **Why deferred:** unsolvable client-side. `PipelexAPIClient.start` has no idempotency-key parameter and no accepted-id recovery; closing the window needs server/SDK support (idempotency keys on `/v1/start`). Building persistence around a not-yet-received id in a teaching starter would be overengineering. The attended module docstring was softened in the PR to scope the "Ctrl-C doesn't lose the run" guarantee to the post-start window. +- **Where the real fix lives:** `pipelex-sdk-python` / the platform API (idempotent submission), not this repo. + From a5406f367decd65eb1bd37dab63ac595386ccb03 Mon Sep 17 00:00:00 2001 From: Louis Choquel <8851983+lchoquel@users.noreply.github.com> Date: Wed, 15 Jul 2026 11:24:24 +0200 Subject: [PATCH 3/3] Release v0.14.0: zero-arg demo fallbacks, generated typed clients, restructured CLI modes Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01AA5fmhTMLRLTJgVmCsW6da --- CHANGELOG.md | 43 ++++++++++++++++++++++++++++--------------- pyproject.toml | 2 +- uv.lock | 2 +- 3 files changed, 30 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8850b53..583fefb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,20 +1,33 @@ # Changelog -## [Unreleased] - -- **Every demo now runs with zero arguments, using a bundled sample.** `piper blocking extract-entities` (or any demo in any mode) no longer errors out when you give it no input — it falls back to a built-in sample so a fresh clone shows a working result on the first command, then a one-line notice on stderr tells you what it used and how to pass your own (`extract-entities`/`generate-image` take a sample text/prompt; `summarize-pdf` defaults to `samples/sample-invoice.pdf`). The samples live in `piper/inputs.py` (they are also the values the README documents); the notice goes to stderr, so stdout stays pipeable. `read_text_input()` gains a `sample` parameter and returns a `TextInput(text, is_sample)`. -- **Protocol-route HTTP errors now surface the API's problem+json, not httpx's stringification.** A non-2xx from `execute`/`start`/`runs/*` used to print `Client error '400 Bad Request' for url …` plus an MDN link — the RFC 7807 body (its `title`, `detail`, and machine `error_type`) was thrown away. `piper/errors.py` now reads that body and shows the server's `detail`; and it branches on the structured `error_type`, so a `/start` against a synchronous-only runner (`StartRequiresAsyncOrchestration`) is presented with a hint pointing you at `piper blocking`, the mode that works there. -- **Breaking (CLI): the execution mode is now the command group, not an option.** `--mode`, `--detach`, and the `PIPELEX_EXECUTION_MODE` env var are all gone; each mode is its own command group — `piper blocking `, `piper attended `, `piper detached ` — and every demo exists in all three. `piper runs status|result|wait ` moved into the mode that produces run ids: `piper detached status|result|wait `. There is no default mode anymore; the mode is explicit in every invocation, which is itself the lesson. The middle mode is named `attended`, not `durable`, because detached runs are durable too — the axis the names describe is who waits. -- **Each execution mode is now a self-contained copy-paste unit.** A mode used to be an `ExecutionMode` value threaded through a dispatch chain (CLI command → `_dispatch()` → `match` → `piper/runner.py` → the SDK), so answering "how do I call the API?" meant hopping through four layers, and the `match` existed only to showcase all three lifecycles behind one option — nothing anyone would copy. Now each mode is one file (`piper/blocking/cli.py`, `piper/attended/cli.py`, `piper/detached/cli.py`) holding its commands, its SDK lifecycle, and its progress rendering, with one public lifecycle helper that *is* the mode (`execute_pipe` / `start_and_wait` / `start_pipe`). `piper/runner.py`, `ExecutionMode`, `_dispatch()`, and the `RunResults` adaptation that existed only to give the dispatch one return type are all deleted; `piper/cli.py` shrinks to a `load_dotenv` callback plus three `add_typer` calls. New: `docs/cli-architecture.md`. -- **Only mode-orthogonal code is shared: `piper/inputs.py` (new) and `piper/errors.py`.** `piper/file_input.py` is merged into `piper/inputs.py`, which now also owns the text-or-file input helper (`read_text_input`). Lifecycle code is never shared — the demo commands are deliberately near-duplicated across the three mode files (diff two of them and only the lifecycle helper differs), and `tests/unit/test_mode_symmetry.py` guards that duplication against drift. The hints in `piper/errors.py` now name the mode groups (a blocking run that hits the ~30s cap points at `piper attended`; an interrupted attended run points at `piper detached wait `, which is exactly what it has become). -- **The typed output models are now generated from the bundles, not hand-written.** `pipelex codegen` projects each method's concepts into `piper/generated//models.py` (a stamped module plus a `codegen.lock` recording the artifact set). `generate-image` parses into the generated built-in `Image` model (natives are materialized into the generated client), replacing the hand-written `GeneratedImage`. -- **Removed the `piper/examples/` layer.** Each demo is now one self-contained command in its mode file (`piper//cli.py`): bundle path (each mode file owns its `METHODS_DIR` constant), pipe code, and inline narrowing into the generated model (`Model.model_validate(main_stuff)`) — no more per-demo wrapper module hiding that line behind a `parse()` indirection. The `parse()` unit tests went with it (they only exercised pydantic's `model_validate`); the e2e tests narrow inline the same way. -- **Fixed the `/bootstrap` pyproject transform.** Its per-key edits had gone stale against the current `pyproject.toml` (multi-line `packages` array with the `piper.generated.*` subpackages, quoted package-data keys, the `piper/generated` ruff exclude), so bootstrapping aborted on the survivor check. Package-name tokens in pyproject are now rewritten generically (quoted-exact + dotted/path positions), which no longer goes stale when the package list changes; the bootstrap test fixture now mirrors the real pyproject shapes. -- **New make targets:** `make codegen` regenerates the typed clients and the runnable `inputs.template.json` scaffolds beside each bundle; `make codegen-check` verifies offline (pure hashing, no network or API key) that the generated clients are current. The `pipelex` CLI is not a starter dependency — point the `PIPELEX` make variable at a pipelex install that ships `codegen`. CI wiring of `codegen-check` lands once a released pipelex ships the command. -- **New offline smoke tests** (`tests/unit/test_generated_clients.py`): the generated modules import, carry the stamp + lock, round-trip their serialization, and each committed input template names exactly the inputs the CLI dispatches. -- `piper/generated` is excluded from ruff (reformatting generated files would trip the drift check) and remains fully type-checked; documented the whole flow in `docs/codegen.md`. -- Each `codegen.lock` is shipped as package data, so `pipelex codegen check` also works against an installed (wheel) copy, not just a git checkout. -- **Breaking (generated `Image` shape):** the generated built-in `Image` model now carries typed `width` / `height` optional integer fields instead of the untyped `size` dict, and native field descriptions come from the standard's pinned definitions. Regenerated all committed clients (stamps, locks, and crate fingerprints updated). +## [v0.14.0] - 2026-07-15 + +### Added + +- **Zero-argument demo fallbacks:** every demo now runs with zero arguments, falling back to a built-in sample (e.g. `samples/sample-invoice.pdf` for `summarize-pdf`) and printing a notice to stderr, so stdout stays pipeable and a fresh clone shows a working result on the first command. `read_text_input()` gains a `sample` parameter and returns a `TextInput(text, is_sample)`. +- **Generated typed clients:** output models are now generated from `.mthds` bundles into `piper/generated/` via `pipelex codegen`, replacing hand-written models (`generate-image` now parses into the generated `Image` model, replacing the hand-written `GeneratedImage`). Added `make codegen` to regenerate clients and templates, and `make codegen-check` to verify offline (via hashing against each `codegen.lock`) that generated clients are up to date. Each `codegen.lock` ships as package data, so the check also works against an installed (wheel) copy, not just a git checkout. +- **New documentation:** added `docs/cli-architecture.md`, describing the copy-paste CLI layout, and `docs/codegen.md`, explaining the generated-models workflow. +- **Offline smoke tests:** added `tests/unit/test_generated_clients.py` to verify generated modules import correctly, carry the stamp + lock, round-trip their serialization, and match the committed input templates to the CLI's inputs; and `tests/unit/test_mode_symmetry.py` to guard against drift between CLI modes. +- **Bootstrap validation:** `/bootstrap` now rejects package names that collide with the template's `piper` placeholder (e.g. `piper_tools`, which would corrupt into `piper_tools_tools` under the pyproject transform). + +### Changed + +- **Execution mode is now the command group, not an option (Breaking):** `--mode`, `--detach`, and the `PIPELEX_EXECUTION_MODE` env var are gone — invoke the mode explicitly: `piper blocking `, `piper attended `, or `piper detached `. The top-level `runs` command moves under detached mode (`piper detached status|result|wait `). There is no default mode anymore; the mode is explicit in every invocation, which is itself the lesson. The middle mode is named `attended`, not `durable`, because detached runs are durable too — the axis the names describe is who waits. +- **Typed `Image` dimensions (Breaking):** the generated `Image` model now uses optional integer `width` and `height` fields instead of an untyped `size` dict, with native field descriptions sourced from the standard's pinned definitions. Regenerated all committed clients (stamps, locks, and fingerprints updated). +- **CLI architecture:** each execution mode is now a self-contained, copy-paste unit (`piper//cli.py`) with its own commands, SDK lifecycle helper (`execute_pipe` / `start_and_wait` / `start_pipe`), and progress rendering; lifecycle code is no longer shared. Only mode-orthogonal code remains shared: `piper/inputs.py` (text/file inputs and document envelopes — `piper/file_input.py` is merged into it) and `piper/errors.py`, whose hints now name the mode groups (a blocking run that hits the ~30s cap points at `piper attended`; an interrupted attended run points at `piper detached wait `). +- **Structured HTTP errors:** protocol-route HTTP errors now parse and surface the API's RFC 7807 `problem+json` body (`title`, `detail`, machine `error_type`) instead of httpx's stringification, with hints pointing to the correct CLI mode based on error type (e.g. `StartRequiresAsyncOrchestration` points at `piper blocking`). +- **E2E tests:** end-to-end tests now call the mode lifecycle helpers directly, with one execution mode per demo (`extract-entities` → blocking, `summarize-pdf` → attended, `generate-image` → detached) for full matrix coverage. +- `piper/generated` is excluded from ruff (reformatting generated files would trip the drift check) but remains fully type-checked. + +### Fixed + +- **Bootstrap `pyproject.toml` transform:** `/bootstrap` now rewrites `pyproject.toml` using generic, context-aware rules (quoted-exact and dotted/path positions) instead of per-key edits, preventing staleness as the package list changes, and correctly handles the multi-line `packages` array with the `piper.generated.*` subpackages, quoted package-data keys, and the `piper/generated` ruff exclude. + +### Removed + +- **Dispatch layer:** removed `piper/runner.py`, the central `_dispatch()` chain, and the `ExecutionMode` enum, in favor of the self-contained mode CLIs; `piper/cli.py` shrinks to a `load_dotenv` callback plus three `add_typer` calls. +- **`piper/examples/` layer:** removed the per-demo wrapper modules; demo logic and inline model narrowing (`Model.model_validate(main_stuff)`) now live directly in the mode CLI files. The `parse()` unit tests went with it (they only exercised pydantic's `model_validate`). +- **`piper/file_input.py`:** merged into the new shared `piper/inputs.py` module. ## [v0.13.0] - 2026-07-07 diff --git a/pyproject.toml b/pyproject.toml index 2d8c9a6..83d20c5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "piper" -version = "0.13.0" +version = "0.14.0" description = "Replace this with your project description" # authors = [{ name = "Your Name", email = "your.email@example.com" }] license = "MIT" diff --git a/uv.lock b/uv.lock index 3b6253a..6405de4 100644 --- a/uv.lock +++ b/uv.lock @@ -313,7 +313,7 @@ wheels = [ [[package]] name = "piper" -version = "0.13.0" +version = "0.14.0" source = { editable = "." } dependencies = [ { name = "httpx" },