From ae33af4aab7d0036046d38bccbe61e4eacf7f06f Mon Sep 17 00:00:00 2001 From: simontreanor Date: Mon, 28 Sep 2026 02:59:58 +0100 Subject: [PATCH] docs: INTERNALS for tonight's native lowering, spawn handler and packages The crate layout gains manifest.rs, packages.rs and native_build.rs; new sections map the native lowering (ladders, annotations, build), the spawn handler, and facade resolution to the functions that implement them. --- INTERNALS.md | 56 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 53 insertions(+), 3 deletions(-) diff --git a/INTERNALS.md b/INTERNALS.md index a03b479..00fc0cf 100644 --- a/INTERNALS.md +++ b/INTERNALS.md @@ -26,18 +26,22 @@ src/ python_emitter/ Python-AST IR → readable source (py311.rs = the --target 3.11 rewrite) diagnostics/ rustc-style errors: codes (E001…), levels, spans, notes project/ module graph: acyclic import resolution, cross-module checking + manifest.rs pyfun.toml reader/editor, project root and vendored façades (DESIGN §6.2) lsp/ front-end-first language server (stdio JSON-RPC) json.rs hand-rolled, dependency-free JSON value + parser + serializer resolve.rs AST resolver: symbol_at / find_references / definitions repl.rs read-eval-print loop over one long-lived Python worker kernel.rs the REPL session inverted, for the Jupyter kernel - main.rs the CLI: check/compile/run/parse/lsp/repl/kernel-engine + packages.rs `pyfun add` / `install` / `remove`: façade packages (DESIGN §6.2) + native_build.rs `pyfun build --native`: compile, then mypyc (DESIGN §5.6) + main.rs the CLI: check/compile/run/build/add/install/remove/parse/lsp/repl/kernel-engine editors/vscode/ minimal VS Code client that launches `pyfun lsp` tests/ parser tests, compile tests, .pyfun fixtures (favor snapshot/golden tests) ``` -The shared Python runtime support (the `Option`/`Result` classes) is not a directory: it is -generated by `lowering::runtime_module()` into a `_pyfun_rt.py` beside the emitted output. +The shared Python runtime support (the `Option`/`Result` classes, and the `_pf_scope` context +variable `Task.scope` publishes its TaskGroup through) is not a directory: it is generated by +`lowering::runtime_module()` into a `_pyfun_rt.py` beside the emitted output. There are no dependencies — `Cargo.toml`'s `[dependencies]` is empty, so the CLI is hand-rolled argument parsing rather than clap. @@ -390,6 +394,30 @@ old signatures by delegating and dropping the notes. Regression tests live in `t "self tail calls become loops", including two end-to-end: 50k/100k-deep recursion that only completes as a loop, and the closure case whose per-frame values the capture check exists to preserve. +### Native lowering (`--native`) — implements DESIGN §5.6 + +A `Lowerer.native` flag, set by `lowering::lower_native_typed` (single file, called from +`pyfun::compile_with`) or `lower_in_project` (from `project::compile_with`). Three pieces: + +- **Match ladders.** `try_lower_native_match` runs before the `Option`/`Result` ladder at both + match call sites (return and value position). `native_pattern_ok` decides whether every arm can be + compiled to plain tests; `native_pattern` compiles one pattern against a subject expression into + `conds` (and-ed) and `binds` (assignments run once the arm is chosen). A top-level or-pattern is + expanded into one piece per alternative (the body lowered per alternative); a nested one is joined + with `or` when no alternative binds a name. The assembly (guards falling through in return + position, the last unguarded arm as a plain `else`) mirrors `try_lower_ladder_match`. +- **Annotations.** `lower_module` post-processes the module: `any_for_object_fields` turns `object` + field annotations into `typing.Any` (mypyc cannot build an `object` field), and + `Lowerer::annotate_functions` adds a union alias per non-generic user sum type after its classes, + spells data class fields that name a user type (`"Expr"`), and rewrites each top-level + `PyStmt::FuncDef` whose binding type is known into a `PyStmt::TypedFuncDef` using + `py_type_annotation` over `Lowerer.binding_types` (span of the binding target → `types::Ty`, from + `TypeSpan::raw`). An unspellable type leaves its position unannotated. +- **Build.** `src/native_build.rs` writes the native output to a directory, runs `python -m mypyc + --allow-redefinition-new --local-partial-types` over every module but `_pyfun_rt.py`, removes the + sources and writes `__main__.py`. `bench/mypyc_run.py` and `.github/workflows/native-bench.yml` + measure it. + ### The `--target 3.11` emission pass Emitted code targets Python 3.12+ for exactly one construct: an f-string hole may carry a nested @@ -555,6 +583,18 @@ becomes `import math as _pf_math` and the reference is re-rooted, leaving the us dotted import binds only its first segment (`import os.path` binds `os`), so the alias replaces as many segments as the module spec covers. +## The `spawn` handler (`Task.scope`) — implements DESIGN §4, §8 + +`EffLabel::Spawn` is performed by `Task.start` (its scheme, `seed_task_prelude`). The discharge +is a rule in `Infer::infer_apply`, not in unification: for a saturated application whose head is +the prelude's `Task.scope` (`qualified_name`, and no user binding named `Task`), the argument is +inferred in a fresh effect context and `spawn` is removed from what it performed before the rest +joins the enclosing effect (`infer_apply_plain` is the ordinary path). `types::run` collects each +top-level evaluation's performed effect (value bindings, expressions, module value members) in +`top_effects` and reports any that still contain `spawn` once inference is done. Lowering routes +`Task.scope`/`Task.start` to `_pf_task_scope`/`_pf_task_start` (`async_prelude`) and +`note_task_scope_var` defines or imports `_pf_scope`. + ## Solving effect equations (`Infer::unify_eff`) — implements DESIGN §4 An `Effect` is a set of concrete labels plus a set of effect variables, so unification is over an @@ -626,6 +666,16 @@ relying on a theme's rendering of `invalid` keeps the hue consistent across them auto-switching, and avoids the semantic lie that these valid keywords are errors. Users can override the colour in their own `editor.tokenColorCustomizations`. +## Façade packages — implements DESIGN §6.2 + +`project::locate_module(dir, name)` is the one resolution rule, used by `build_from_path` (the CLI +graph) and `resolve_exports` (the LSP): the sibling `name.pyfun`, else the same file in each +directory under `/.pyfun/facades/` for the nearest `pyfun.toml` (`manifest::find_root`, +`manifest::facade_dirs`). `src/packages.rs` implements the commands: the installer is `uv pip` +when `uv` is on PATH, else `python -m pip`; the façade's `.pyfun` files are found by asking the +installed distribution for its file list (`importlib.metadata`), and `manifest::set`/`remove` edit +`pyfun.toml` line by line. + ## File-based modules — implements DESIGN §6.1 The module system's semantics (one module per file, qualified use, an acyclic import graph, cross-module ADTs/records/externs/measures, all-public) are in `DESIGN.md` §6.1. The driver, cross-module checking, and project emit are built as follows.