diff --git a/.gitignore b/.gitignore index 2da8ae2..a4dfd36 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,4 @@ __pycache__/ build/ dist/ +.DS_Store diff --git a/README.md b/README.md index c236a38..9baed88 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ language subset to deterministic PyIR. CFG construction, SSA, data-flow, and sec are intentionally deferred. ```text -Python source -> SourceManager -> parser -> PyHIR -> semantic imports -> PyIR +Python source -> SourceManager -> frontend -> PyHIR -> semantic imports and scopes -> PyIR ``` The long-term engine architecture and incremental migration plan are recorded in diff --git a/docs/architecture.md b/docs/architecture.md index ddecefa..91d8d2c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,93 +1,1763 @@ -# CoreTrace Python Analysis Engine Architecture +# Architecture of the Extensible Python Analysis Engine -## Product direction +## 1. Objective -CoreTrace Python Analyzer is evolving from a security-specific source scanner into an -extensible static-analysis engine. Security detectors will be plugins built on shared, -language-level analysis infrastructure. +The objective is to build an **extensible Python static analysis framework** in which the engine exposes every stage of its pipeline and allows plugins to hook into the appropriate layer. -The target data path is: +The engine must not be limited to security. Security is a collection of plugins built on top of a generic analysis infrastructure. + +The system should eventually support: + +- vulnerability detection; +- syntactic rules; +- code quality analysis; +- bug detection; +- dependency analysis; +- secret detection; +- framework analysis; +- configuration analysis; +- false-positive reduction; +- custom enterprise rules; +- CLI, JSON, and SARIF report generation. + +Core principles: + +1. Intermediate representations are produced once and shared. +2. A plugin declares the analyses it depends on. +3. The engine computes analyses lazily whenever possible. +4. Analyses are organized as a **dependency DAG**, not as a simple rigid chain. +5. Plugins consume stable analysis results instead of accessing internal structures directly. +6. Expensive analyses, especially taint, call graph, and interprocedural analysis, are shared. +7. Plugins do not duplicate CFGs, IRs, or global results. +8. IR transformations and read-only analyses are kept separate. +9. The engine manages caching, dependencies, and analysis invalidation. +10. Framework models, detection engines, and proof engines are kept separate. + +--- + +# 2. Overview + +```text + Python Project + │ + ▼ + Source Discovery + │ + ▼ + Tree-sitter + │ + ▼ + PyHIR + │ + ┌─────────┼─────────┐ + ▼ ▼ ▼ + Scopes Imports Symbols + └─────────┼─────────┘ + ▼ + CFG + │ + ▼ + SSA / PyIR + │ + ┌────────────┼────────────┐ + ▼ ▼ ▼ + Data-flow Call Graph Abstract Value + │ │ │ + └────────────┼────────────┘ + ▼ + Function Summaries + │ + ▼ + Taint Engine + │ + ┌────────────┼────────────┐ + ▼ ▼ ▼ + Sources Sanitizers Sinks + └────────────┼────────────┘ + ▼ + Findings + │ + ▼ + Proof / Refutation + │ + ┌───────────┼───────────┐ + ▼ ▼ ▼ + Vulnerability Hotspot Refuted + │ │ + └──────┬────┘ + ▼ + CLI / JSON / SARIF +``` + +This view is intentionally simplified. In practice, the engine must operate as an **analysis dependency graph**. + +--- + +# 3. Layered Architecture + +## 3.1 Frontend + +Responsibilities: + +- discovering Python files; +- parsing; +- preserving source locations; +- collecting syntax errors; +- supporting incremental analysis in the future. + +Pipeline: + +```text +Python source + ↓ +tree-sitter-python + ↓ +Tree-sitter CST +``` + +Tree-sitter must not be exposed directly to advanced analyses. + +Syntactic plugins may use it when appropriate, but semantic plugins must depend on PyHIR or PyIR. + +--- + +## 3.2 PyHIR + +PyHIR is a high-level representation independent of Tree-sitter. + +Example: + +```python +cmd = request.args["cmd"] +``` + +can become: + +```text +Assign + ├── target: Name("cmd") + └── value: + GetItem + ├── GetAttr(Name("request"), "args") + └── Constant("cmd") +``` + +Main instructions or nodes: + +```text +Module +Function +Class +Assign +AugAssign +Name +Constant +Call +Attribute +Subscript +If +While +For +Return +Import +ImportFrom +BinaryOp +UnaryOp +Compare +Await +Yield +Raise +With +Try +Lambda +Comprehension +``` + +Objective: + +> Decouple the Tree-sitter grammar from all subsequent analyses. + +--- + +# 4. Semantic Analysis + +Starting from PyHIR, the engine builds several analyses. + +```text + PyHIR + │ + ┌─────────────┼─────────────┐ + ▼ ▼ ▼ + Scopes Imports Symbols +``` + +## 4.1 ScopeAnalysis + +Responsibilities: + +- local variables; +- global variables; +- `nonlocal` declarations; +- parameters; +- closures; +- function scopes; +- class scopes; +- comprehension scopes. + +Example: + +```python +x = 1 + +def foo(): + x = 2 + return x +``` + +The engine must distinguish between the two `x` symbols. + +--- + +## 4.2 ImportAnalysis + +Example: + +```python +import os as operating_system +from subprocess import run as execute +``` + +Result: + +```text +operating_system -> python.os +execute -> python.subprocess.run +``` + +--- + +## 4.3 SymbolAnalysis + +The engine should prefer canonical identifiers. + +Examples: + +```text +python.builtins.eval +python.builtins.exec +python.os.system +python.subprocess.run +python.requests.get +flask.request.args +django.http.HttpRequest.GET +``` + +This ensures that security rules do not depend on the textual name visible in the file. + +--- + +# 5. CFG + +Each function is transformed into a Control Flow Graph. + +```text +Function + │ + ▼ +Entry + │ + ▼ +BasicBlock + │ + ├── instructions + └── terminator +``` + +Terminators: + +```text +Branch +Jump +Return +Raise +Switch-like constructs if needed +``` + +The CFG must support the computation of: + +- predecessors; +- successors; +- dominance; +- post-dominance; +- reachability; +- loops; +- back edges; +- exception edges; +- control dependence. + +Example: + +```python +if safe: + x = sanitize(x) + +sink(x) +``` + +becomes approximately: + +```text +entry: + branch %safe, bb_safe, bb_merge + +bb_safe: + ... + jump bb_merge + +bb_merge: + ... + call sink(...) +``` + +--- + +# 6. PyIR + +PyIR is the low-level analysis IR, but it must remain semantically faithful to Python. + +Example instructions: + +```text +Const +Copy +LoadSymbol +StoreSymbol +GetAttr +SetAttr +GetItem +SetItem +Call +CallMethod +BinaryOp +UnaryOp +Compare +BuildList +BuildTuple +BuildDict +Branch +Jump +Return +Phi +Import +Await +Yield +Raise +WithEnter +WithExit +``` + +Avoid artificially reproducing LLVM's memory model. + +Example: + +```python +obj.foo +``` + +must remain: + +```text +%1 = get_attr %obj, "foo" +``` + +and must not be transformed into the Python equivalent of a `getelementptr`. + +--- + +# 7. SSA + +Local values are converted to Static Single Assignment form. + +Example: + +```python +x = a + +if cond: + x = b + +use(x) +``` + +becomes: + +```text +%x.0 = %a + +branch %cond, bb_then, bb_merge + +bb_then: + %x.1 = %b + jump bb_merge + +bb_merge: + %x.2 = phi [%x.0, entry], + [%x.1, bb_then] + +call use(%x.2) +``` + +Benefits: + +- explicit def-use chains; +- constant propagation; +- taint propagation; +- abstract value analysis; +- straightforward handling of merges; +- dependency analysis between values. + +--- + +# 8. Analysis Manager + +The core of the system must be an `AnalysisManager`. + +Responsibilities: + +- registering available analyses; +- registering their dependencies; +- computing them only when needed; +- caching their results; +- invalidating results after transformations; +- exposing a uniform API to plugins. + +Conceptual example: + +```python +symbols = ctx.get(SymbolAnalysis) +cfg = ctx.get(CFGAnalysis, function) +taint = ctx.get(TaintAnalysis, function) +``` + +The plugin must not need to know how these results are built. + +--- + +# 9. Analysis DAG + +The actual pipeline is a graph. + +```text + Parse + │ + ▼ + HIR + ┌────────┼────────┐ + ▼ ▼ ▼ + Scopes Imports Symbols + │ │ │ + └────────┼────────┘ + ▼ + CFG + │ + ▼ + PyIR + ┌───────┼─────────┐ + ▼ ▼ ▼ + Dominators DataFlow AbstractValue + │ + ▼ + CallGraph + │ + ▼ + Summaries + │ + ▼ + Taint +``` + +Dependencies must be declarative. + +Example: + +```text +TaintAnalysis requires: + PyIR + CFG + SymbolAnalysis + CallGraph + FunctionSummaryAnalysis +``` + +--- + +# 10. Lazy Evaluation + +The engine must not compute every analysis by default. + +Example: + +```text +Plugin A -> Tree-sitter +Plugin B -> PyHIR +Plugin C -> Symbols +Plugin D -> Taint +``` + +The engine computes only what is required. + +An analysis may also be computed at function granularity. + +Example: + +```text +ctx.get(CFGAnalysis, foo) +``` + +must not necessarily force construction of the CFGs for all 10,000 functions in the project. + +--- + +# 11. Cache + +Every analysis result must be cacheable. + +Possible keys: + +```text +AnalysisType ++ +ModuleId / FunctionId ++ +IR fingerprint ++ +Analysis version ++ +Plugin/API version +``` + +Example: + +```text +(FunctionId, CFGAnalysis, IRHash) + ↓ +CFGResult +``` + +A persistent cache can later enable incremental analysis. + +--- + +# 12. Invalidation + +Read-only analyses do not modify the IR. + +Transformations may invalidate other analyses. + +Two families: + +```text +AnalysisPass +TransformationPass +``` + +Example: + +```text +SimplifyCFGPass +``` + +may preserve: + +```text +Symbols +Imports +Scopes +``` + +but invalidate: + +```text +Dominators +DataFlow +SSA +Taint +AbstractValue +``` + +The engine must manage this invalidation automatically. + +--- + +# 13. Plugin API + +Avoid: + +```python +plugin.on_stage("cfg", arbitrary_object) +``` + +Prefer typed interfaces. + +Example: + +```python +class SecurityPlugin: + requires = { + SymbolAnalysis, + TaintAnalysis, + } + + def analyze(self, ctx: AnalysisContext) -> list[Finding]: + ... +``` + +Other plugin types: + +```text +FrontendPlugin +HIRPlugin +IRPlugin +AnalysisProvider +FrameworkModelPlugin +SecurityDetector +RefutationPlugin +ReporterPlugin +DependencyPlugin +SecretPlugin +ConfigPlugin +``` + +--- + +# 14. Plugin Manifests + +Example: + +```yaml +name: sql-injection +version: 1.0.0 +plugin_api: 1 + +requires: + - semantic.symbols + - analysis.taint + +provides: + - vulnerability.sql-injection + +entrypoint: + module: plugins.sql_injection + class: SQLInjectionPlugin +``` + +Flask plugin: + +```yaml +name: flask-model +version: 1.0.0 +plugin_api: 1 + +requires: + - semantic.symbols + +provides: + - model.http-sources + - model.flask-routes +``` + +--- + +# 15. Analysis Providers and Detectors + +This distinction is important. + +## Analysis Provider + +Produces facts. + +Example: + +```text +FlaskPlugin + ↓ +route(handler) +request.args -> HTTP_SOURCE +``` + +The Flask plugin does not need to know about SQL injection. + +--- + +## Detector + +Consumes generic facts. + +Example: + +```text +HTTP_SOURCE + ↓ +Taint Engine + ↓ +SQL_SINK + ↓ +SQLInjectionDetector +``` + +This avoids: + +```text +FlaskSQLInjection +DjangoSQLInjection +FastAPISQLInjection +``` + +in favor of: + +```text +FlaskModel +DjangoModel +FastAPIModel + ↓ +Common Security Model + ↓ +SQLInjectionDetector +``` + +--- + +# 16. Security Model Registry + +All plugins must be able to register models. + +Example: + +```text +SourceModel +SinkModel +SanitizerModel +PropagatorModel +FrameworkModel +``` + +Example: + +```text +flask.request.args + -> Source(HTTP_INPUT) + +python.os.system + -> Sink(COMMAND) + +html.escape + -> Sanitizer(HTML) +``` + +--- + +# 17. Global Taint Engine + +The taint engine must be shared by all plugins. + +Avoid: + +```text +SQL plugin -> own taint engine +Command plugin -> own taint engine +XSS plugin -> own taint engine +``` + +Prefer: + +```text + Global Taint Engine + │ + ┌────────────┼────────────┐ + ▼ ▼ ▼ + SQL COMMAND HTML +``` + +Taint kinds can be represented using a bitset: + +```text +SQL = bit 0 +COMMAND = bit 1 +HTML = bit 2 +PATH = bit 3 +SSRF = bit 4 +... +``` + +The join operation generally becomes: + +```text +OUT = A | B +``` + +Plugins register their models and consume the same taint result. + +--- + +# 18. Abstract Value Engine + +An SSA value may have an abstract state: + +```text +AbstractValue { + types + constants + taints + numeric_range + string_constraints + truthiness + nullability +} +``` + +Example: + +```text +%x: + type = str + taint = HTTP_INPUT + string_constraint = numeric_only +``` + +This supports: + +- constant propagation; +- type approximation; +- guard validation; +- proof/refutation; +- finding prioritization. + +--- + +# 19. Function Summaries + +Interprocedural analyses must use summaries. + +Example: + +```python +def identity(x): + return x +``` + +Summary: + +```text +RETURN depends_on PARAM[0] +``` + +Example: + +```python +def execute(cmd): + os.system(cmd) +``` + +Summary: + +```text +PARAM[0] -> COMMAND_SINK +``` + +Possible structure: + +```text +FunctionSummary { + return_dependencies + return_taints + parameter_sinks + sanitizers + mutations + side_effects + calls +} +``` + +--- + +# 20. Call Graph + +The call graph is built from: + +- resolved symbols; +- imports; +- direct calls; +- type inference; +- framework models; +- potentially an approximate resolution of dynamic dispatch. + +```text +A +├── B +└── C + └── D +``` + +Because Python is dynamic, the call graph must support: + +```text +KnownTarget +PossibleTargets +UnknownTarget +``` + +--- + +# 21. Multi-File Analysis + +The engine builds: + +```text +ModuleGraph ++ +CallGraph ++ +FunctionSummaryIndex +``` + +Example: + +```text +routes.py + ↓ +services.py + ↓ +repository.py + ↓ +database.py +``` + +Summaries must enable whole-project analysis without retaining all PyIR in memory. + +--- + +# 22. Heap and Aliasing + +SSA is not sufficient for: + +```python +a = [] +b = a +b.append(user_input) +sink(a[0]) +``` + +Gradually introduce: + +```text +AllocationSite +AbstractObject +HeapLocation +AliasSet +``` + +Example: + +```text +List@foo.py:42.elements + -> HTTP_INPUT +``` + +The engine can start with a coarse abstraction and refine it later. + +--- + +# 23. Findings + +A finding must reference lightweight IDs. + +Example: + +```text +Finding { + rule_id + severity + confidence + function_id + source_instruction_id + sink_instruction_id + message + metadata +} +``` + +Avoid storing a complete copy of the CFG or taint path. + +The path can be reconstructed during reporting. + +--- + +# 24. Proof / Refutation Engine + +After initial detection: + +```text +Potential Finding + ↓ +Proof / Refutation +``` + +Sources of evidence: + +```text +Dominators +Post-dominators +Range analysis +String constraints +Type constraints +Path constraints +Sanitizers +Constant propagation +Authorization guards +``` + +Example: + +```python +value = request.args["id"] + +if not value.isdigit(): + return + +sink(value) +``` + +The engine can determine that, along the path to the sink: + +```text +value = numeric_only +``` + +The final result may become: + +```text +Vulnerability +Hotspot +Refuted +``` + +--- + +# 25. Plugin Types + +Possible organization: + +```text +plugins/ +├── syntax/ +│ ├── dangerous_eval/ +│ └── weak_crypto/ +│ +├── frameworks/ +│ ├── flask/ +│ ├── django/ +│ └── fastapi/ +│ +├── security/ +│ ├── sql_injection/ +│ ├── command_injection/ +│ ├── path_traversal/ +│ ├── ssrf/ +│ └── xss/ +│ +├── dependency/ +│ ├── osv/ +│ ├── lockfiles/ +│ └── reachability/ +│ +├── secrets/ +│ ├── generic_entropy/ +│ └── providers/ +│ +├── proof/ +│ ├── sanitizer/ +│ ├── ranges/ +│ └── path_constraints/ +│ +└── reporters/ + ├── cli/ + ├── json/ + └── sarif/ +``` + +--- + +# 26. Dependency Plugins + +The plugin system must not be limited to the PyIR pipeline. + +Example: + +```text +requirements.txt +pyproject.toml +poetry.lock +uv.lock + │ + ▼ +Dependency Resolver + │ + ▼ +Dependency Graph + │ + ├── OSV Plugin + ├── GHSA Plugin + ├── Policy Plugin + └── Reachability Plugin +``` + +The Reachability Plugin can then depend on the Python engine's call graph. + +Example: + +```text +Vulnerable Package + + +Affected API + + +CallGraph + ↓ +Reachable vulnerability +``` + +--- + +# 27. Correlation Engine + +A global engine must be able to correlate multiple results. + +Example: + +```text +SCA: + vulnerable dependency + +SAST: + affected API imported + +CallGraph: + vulnerable function reachable + +Taint: + attacker-controlled value reaches call + + ↓ + +High-confidence exploitable finding +``` + +This is a major differentiator. + +--- + +# 28. Reporter API + +Reporters consume only normalized findings. + +```text +Findings + │ + ├── CLIReporter + ├── JSONReporter + ├── SARIFReporter + └── future UI/API +``` + +A reporter must not run an analysis itself. + +--- + +# 29. Performance Model + +Let: + +```text +N = program size +B = number of basic blocks +E = number of CFG edges +V = number of SSA values +C = number of call graph edges +P = number of plugins +``` + +The expected cost should be: + +```text +Ttotal = + Tcore + + Tunique_analyses + + Σ Tplugin_i +``` + +and not: + +```text +Ttotal = + P × Tcore +``` + +Shared analyses must be computed only once. + +--- + +# 30. Memory Complexity + +The target model is: + +```text +Mtotal = + Mshared_IR + + Mshared_analysis_results + + Σ Mplugin_state +``` + +and not: + +```text +Mtotal = + P × Mproject +``` + +Whenever possible, plugins should retain only: + +```text +FunctionId +BlockId +InstructionId +ValueId +SymbolId +FindingId +``` + +--- + +# 31. Example with 30 Plugins + +Suppose: + +```text +30 plugins +├── 8 syntax checks +├── 5 framework models +├── 10 security detectors +├── 3 proof plugins +├── 2 dependency plugins +└── 2 reporters +``` + +The engine must perform the following only once: + +```text +Tree-sitter +PyHIR +Symbol resolution +CFG +SSA +CallGraph +Taint +AbstractValue +``` + +All 30 plugins share these results. + +--- + +# 32. Stable API + +Plugins must not access internal classes directly. + +Avoid: + +```python +engine._cfg_map +engine._symbol_table +engine._ir_storage +``` + +Prefer: + +```python +ctx.functions() +ctx.cfg(function_id) +ctx.symbol(value_id) +ctx.calls(function_id) +ctx.taint(value_id) +ctx.abstract_value(value_id) +ctx.source_location(instruction_id) +``` + +This API constitutes the Plugin API contract. + +--- + +# 33. Versioning + +Plan for the following from the outset: + +```text +Plugin API v1 +PyHIR schema version +PyIR schema version +Summary schema version +Finding schema version +``` + +A plugin manifest must be able to declare compatibility: + +```yaml +plugin_api: ">=1,<2" +``` + +--- + +# 34. Plugin Isolation + +Several levels may eventually be supported. + +## In-Process + +Fast and simple. + +```text +Core Engine + ↓ +Python plugin module +``` + +Advantages: + +- low overhead; +- direct access to APIs; +- easy development. + +Disadvantage: + +- a plugin can crash the engine. + +## Out-of-Process + +Possible for untrusted plugins. + +```text +Core + ↓ +RPC / IPC + ↓ +Plugin Process +``` + +Safer but more expensive. + +The engine can begin with in-process plugins while providing an abstraction compatible with future isolation. + +--- + +# 35. Example of Resolving a Plugin's Dependencies + +Plugin: + +```text +SQLInjectionPlugin +``` + +Declares: + +```text +requires: + TaintAnalysis + SymbolAnalysis +``` + +The engine resolves: + +```text +SQLInjectionPlugin + │ + ├──── TaintAnalysis + │ │ + │ ├── PyIR + │ ├── CFG + │ ├── Symbols + │ └── Summaries + │ + └──── SymbolAnalysis + │ + ├── PyHIR + ├── Imports + └── Scopes +``` + +The actual DAG becomes: + +```text +Parse + ↓ +PyHIR + ↓ +Scopes + Imports + ↓ +Symbols + ↓ +CFG + ↓ +PyIR + ↓ +Summaries + ↓ +Taint + ↓ +SQLInjectionPlugin +``` + +--- + +# 36. Example of Plugin Composition + +```text +FlaskModelPlugin + ↓ +register HTTP sources + +SQLAlchemyModelPlugin + ↓ +register SQL sinks + +Taint Engine + ↓ +connect source → sink + +SQLInjectionPlugin + ↓ +Finding + +PathConstraintRefuter + ↓ +Finding validated/refuted + +SARIFReporter +``` + +Each component remains independent. + +--- + +# 37. Proposed Repository Structure + +```text +engine/ +├── frontend/ +│ ├── parser/ +│ └── source_manager/ +│ +├── hir/ +│ ├── nodes/ +│ ├── builder/ +│ └── visitors/ +│ +├── semantic/ +│ ├── scopes/ +│ ├── imports/ +│ └── symbols/ +│ +├── cfg/ +│ ├── graph/ +│ ├── builder/ +│ └── dominance/ +│ +├── ir/ +│ ├── values/ +│ ├── instructions/ +│ ├── blocks/ +│ ├── functions/ +│ └── ssa/ +│ +├── analysis/ +│ ├── manager/ +│ ├── registry/ +│ ├── dependencies/ +│ ├── cache/ +│ └── invalidation/ +│ +├── dataflow/ +│ ├── lattice/ +│ ├── worklist/ +│ └── solver/ +│ +├── abstract/ +│ ├── values/ +│ ├── ranges/ +│ ├── strings/ +│ └── types/ +│ +├── interprocedural/ +│ ├── callgraph/ +│ ├── summaries/ +│ └── modulegraph/ +│ +├── taint/ +│ ├── engine/ +│ ├── models/ +│ ├── sources/ +│ ├── sinks/ +│ └── sanitizers/ +│ +├── findings/ +│ ├── model/ +│ ├── correlation/ +│ └── refutation/ +│ +├── plugins/ +│ ├── api/ +│ ├── loader/ +│ ├── manifest/ +│ └── registry/ +│ +├── reporters/ +│ ├── cli/ +│ ├── json/ +│ └── sarif/ +│ +└── tests/ + ├── unit/ + ├── integration/ + ├── fixtures/ + └── plugins/ +``` + +--- + +# 38. Recommended Development Order + +## Phase 1 — Infrastructure + +```text +SourceManager +Tree-sitter +PyHIR +Plugin API +Plugin Loader +Analysis Manager +``` + +## Phase 2 — Semantics + +```text +Scopes +Imports +Symbols +Canonical Symbol IDs +``` + +## Phase 3 — IR + +```text +CFG +PyIR +SSA +Def-use +Dominators +``` + +## Phase 4 — Data Flow + +```text +Worklist solver +Lattice API +AbstractValue +Constant propagation +``` + +## Phase 5 — Security Core + +```text +Source/Sink/Sanitizer Registry +Global Taint Engine +Basic detectors +``` + +## Phase 6 — Interprocedural Analysis + +```text +CallGraph +FunctionSummary +ModuleGraph +Fixpoint +``` + +## Phase 7 — Frameworks + +```text +Flask +Django +FastAPI +SQLAlchemy +Requests/httpx +``` + +## Phase 8 — False-Positive Reduction + +```text +Ranges +Path constraints +Dominant guards +Proof/refutation +``` + +## Phase 9 — Ecosystem + +```text +SCA +Secrets +Dependency reachability +Correlation +SBOM +``` + +## Phase 10 — Performance ```text -Python project - -> source discovery and SourceManager - -> parser (Tree-sitter target; stdlib ast adapter initially) - -> parser-independent PyHIR - -> scopes, imports, and canonical symbols - -> control-flow graph - -> SSA PyIR - -> shared analysis DAG - -> models, detectors, and proof/refutation plugins - -> normalized findings - -> CLI, JSON, and SARIF reporters +Persistent cache +Incremental parsing +Incremental summaries +Parallel analysis +Memory eviction ``` -## Architectural boundaries +--- + +# 39. Architectural Rules to Preserve + +### Rule 1 + +> A plugin requests an analysis; it does not recompute it. -1. Source text and locations belong to `source`; all later representations reference stable - source IDs and spans. -2. Parser-specific objects stop at `frontend`. Semantic analysis and lowering consume PyHIR, - never `ast.*` or Tree-sitter nodes. -3. `semantic` owns scopes, imports, and symbols. PyIR lowering consumes semantic results rather - than resolving names itself. -4. CFG is built before SSA. SSA construction is a transformation over non-SSA control flow. -5. Analyses are immutable providers managed through a dependency DAG. Results are lazy, cached, - and invalidated explicitly after transformations. -6. Plugins consume versioned, typed result APIs. They do not access engine internals or rebuild - shared analyses. -7. Framework plugins provide models and facts; detectors consume generic facts. The taint engine - is global and multi-kind. -8. Findings store lightweight IDs. Reporters consume normalized findings and never initiate - analysis. +### Rule 2 -## Migration strategy +> Analysis results are shared and cached. -The migration is incremental so every merged branch leaves the CLI usable. +### Rule 3 -### Phase 1: frontend foundation +> IRs are immutable for standard analyses. -- Introduce `SourceManager`, `SourceId`, and `SourceSpan`. -- Introduce parser-independent PyHIR. -- Adapt Python's standard-library AST into PyHIR. -- Route existing PyIR lowering through PyHIR without changing emitted output. +### Rule 4 -Tree-sitter is intentionally behind the parser boundary. It can replace the initial AST adapter -once PyHIR is stable, without affecting semantic analysis or PyIR. +> Transformations explicitly declare what they invalidate. -### Phase 2: semantic layer +### Rule 5 -- Move import binding and canonical symbol resolution into `semantic`. -- Add lexical scopes, local/global/nonlocal classification, closures, and class scopes. -- Publish immutable semantic results keyed by stable IDs. +> Frameworks enrich the model; detectors detect vulnerabilities. -### Phase 3: control flow and SSA +### Rule 6 -- Build per-function CFGs with explicit terminators and graph integrity checks. -- Add dominators and dominance frontiers. -- Transform local reads and writes into SSA with phi nodes and def-use chains. +> The taint engine is global and supports multiple taint kinds. -### Phase 4: shared analysis infrastructure +### Rule 7 -- Add typed analysis providers and dependency declarations. -- Add lazy evaluation, caching, and transformation invalidation. -- Validate the framework with constant propagation and abstract values. +> Interprocedural results use Function Summaries. -### Later phases +### Rule 8 -- Plugin manifests, registry, and versioned API. -- Global multi-kind taint and security model registry. -- Call graph, function summaries, module graph, and interprocedural fixpoints. -- Framework models, detectors, proof/refutation, correlation, and reporters. -- Persistent caching, incremental parsing, parallelism, and memory eviction. +> Plugins depend on a stable API, never on internal structures. -## Current milestone acceptance criteria +### Rule 9 -The frontend-foundation milestone is complete when: +> Analyses are lazy and, whenever possible, granular at the function level. -- the CLI loads files through `SourceManager`; -- source locations use `SourceSpan` with stable `SourceId` values; -- supported Python syntax is represented in PyHIR; -- PyIR lowering no longer imports or accepts `ast` nodes; -- existing `--emit-ir` output remains deterministic; -- existing tests pass alongside focused source and PyHIR tests. +### Rule 10 + +> Findings use lightweight identifiers instead of graph copies. + +--- + +# 40. Target Architecture + +```text + ┌──────────────────────┐ + │ Plugin Manager │ + │ discovery / manifest │ + │ versions / loading │ + └──────────┬───────────┘ + │ + ▼ + ┌──────────────────────┐ + │ Analysis Manager │ + │ │ + │ dependency DAG │ + │ lazy evaluation │ + │ caching │ + │ invalidation │ + └──────────┬───────────┘ + │ + ┌───────────────────────────┼──────────────────────────┐ + │ │ │ + ▼ ▼ ▼ + Frontend Semantic PyIR + Tree-sitter / PyHIR Scopes / Imports / Symbols CFG / SSA + │ │ │ + └───────────────────────────┼──────────────────────────┘ + ▼ + Generic Analyses + ┌─────────────────┼─────────────────┐ + ▼ ▼ ▼ + DataFlow CallGraph AbstractValue + │ │ │ + └─────────────────┼─────────────────┘ + ▼ + Function Summaries + │ + ▼ + Taint Engine + │ + ┌──────────────────────┼──────────────────────┐ + ▼ ▼ ▼ + Framework Models Detectors Proofs + │ │ │ + └──────────────────────┼──────────────────────┘ + ▼ + Findings + │ + ▼ + Correlation Engine + │ + ┌────────────┼────────────┐ + ▼ ▼ ▼ + Vulnerability Hotspot Refuted + │ │ + └──────┬─────┘ + ▼ + CLI / JSON / SARIF +``` + +--- + +# 41. Final Vision + +The project must not be designed solely as a Python vulnerability scanner. + +It must be designed as: + +> **A programmable and extensible Python static analysis framework that provides a frontend, an HIR, an SSA IR, shared program analyses, and a plugin API for building specialized detectors.** + +Security then becomes a collection of plugins built on top of this engine. + +This separation eventually allows the framework to host: + +```text +Security +Bug Detection +Code Quality +Performance +Compliance +Framework-specific analysis +Dependency Analysis +Supply-chain analysis +Custom enterprise policies +Research analyses +``` +without duplicating the fundamental infrastructure. diff --git a/src/coretrace_python/cli.py b/src/coretrace_python/cli.py index fc03e2d..2947daf 100644 --- a/src/coretrace_python/cli.py +++ b/src/coretrace_python/cli.py @@ -4,11 +4,11 @@ import sys from pathlib import Path -from coretrace_python.frontend.ast_adapter import HIRBuildError, build_module -from coretrace_python.frontend.imports import ImportResolutionError -from coretrace_python.frontend.lowering import LoweringError, lower_module -from coretrace_python.frontend.parser import ParseError, parse_source_file +from coretrace_python.frontend import HIRBuildError, ParseError, build_hir +from coretrace_python.ir.lowering import LoweringError, lower_module from coretrace_python.ir.printer import format_module +from coretrace_python.semantic.imports import ImportResolutionError +from coretrace_python.semantic.scopes import ScopeError from coretrace_python.source import SourceManager @@ -34,15 +34,14 @@ def main(argv: list[str] | None = None) -> int: try: source = SourceManager().load_file(args.path) - syntax_tree = parse_source_file(source) - hir_module = build_module(source, syntax_tree) - module = lower_module(hir_module) + module = lower_module(build_hir(source)) except ( OSError, UnicodeError, ParseError, HIRBuildError, ImportResolutionError, + ScopeError, LoweringError, ) as error: print(f"error: {error}", file=sys.stderr) diff --git a/src/coretrace_python/frontend/__init__.py b/src/coretrace_python/frontend/__init__.py index 6a288e5..68f946a 100644 --- a/src/coretrace_python/frontend/__init__.py +++ b/src/coretrace_python/frontend/__init__.py @@ -1,2 +1,18 @@ -"""Parsing and AST-to-PyIR lowering.""" +"""Parsing and adaptation of Python source into parser-independent PyHIR. +Parser-specific objects never leave this package: callers receive a PyHIR module. +""" + +from coretrace_python.frontend.ast_adapter import HIRBuildError, build_module +from coretrace_python.frontend.parser import ParseError, parse_source_file +from coretrace_python.hir import nodes +from coretrace_python.source import SourceFile + + +def build_hir(source: SourceFile) -> nodes.Module: + """Parse ``source`` and return its PyHIR module.""" + + return build_module(source, parse_source_file(source)) + + +__all__ = ["HIRBuildError", "ParseError", "build_hir"] diff --git a/src/coretrace_python/frontend/ast_adapter.py b/src/coretrace_python/frontend/ast_adapter.py index ebd860f..fddcbd1 100644 --- a/src/coretrace_python/frontend/ast_adapter.py +++ b/src/coretrace_python/frontend/ast_adapter.py @@ -29,6 +29,7 @@ class HIRBuildError(Exception): ast.MatMult: "matmul", } _UNARY_OPERATORS = {ast.Invert: "invert", ast.Not: "not", ast.UAdd: "pos", ast.USub: "neg"} +_COMPREHENSIONS = {ast.ListComp: "list", ast.SetComp: "set", ast.GeneratorExp: "generator"} _COMPARE_OPERATORS = { ast.Eq: "eq", ast.NotEq: "not_eq", @@ -104,8 +105,31 @@ def expression(self, node: ast.expr) -> nodes.Expression: for keyword in node.keywords ) return nodes.Call(self.expression(node.func), arguments, keywords, span) + if isinstance(node, (ast.ListComp, ast.SetComp, ast.GeneratorExp)): + generators = tuple(self.generator(generator) for generator in node.generators) + return nodes.Comprehension( + _COMPREHENSIONS[type(node)], self.expression(node.elt), generators, span + ) self.fail(node) + def generator(self, node: ast.comprehension) -> nodes.ComprehensionGenerator: + if node.is_async: + self.fail(node.target, "async comprehensions are not supported yet") + if not isinstance(node.target, ast.Name): + self.fail(node.target, "only a single name is supported as a comprehension target") + target = nodes.Name(node.target.id, self.span(node.target)) + conditions = tuple(self.expression(condition) for condition in node.ifs) + # ``ast.comprehension`` carries no location; span it from the target to the last clause. + last = self.span(node.ifs[-1] if node.ifs else node.iter) + span = SourceSpan( + self._source.source_id, + target.span.start_line, + target.span.start_column, + last.end_line, + last.end_column, + ) + return nodes.ComprehensionGenerator(target, self.expression(node.iter), conditions, span) + def statement(self, node: ast.stmt) -> nodes.Statement: span = self.span(node) if isinstance(node, ast.Assign): @@ -130,10 +154,25 @@ def statement(self, node: ast.stmt) -> nodes.Statement: nodes.ImportAlias(alias.name, alias.asname, self.span(alias)) for alias in node.names ) return nodes.ImportFrom(node.module, aliases, node.level, span) + if isinstance(node, ast.Global): + return nodes.Global(tuple(node.names), span) + if isinstance(node, ast.Nonlocal): + return nodes.Nonlocal(tuple(node.names), span) if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): return self.function(node) + if isinstance(node, ast.ClassDef): + return self.class_definition(node) self.fail(node) + def class_definition(self, node: ast.ClassDef) -> nodes.Class: + if node.decorator_list: + self.fail(node, "decorated classes are not supported yet") + if node.keywords: + self.fail(node, "class keyword arguments are not supported yet") + bases = tuple(self.expression(base) for base in node.bases) + body = tuple(self.statement(statement) for statement in node.body) + return nodes.Class(node.name, bases, body, self.span(node)) + def function(self, node: ast.FunctionDef | ast.AsyncFunctionDef) -> nodes.Function: if node.decorator_list: self.fail(node, "decorated functions are not supported yet") diff --git a/src/coretrace_python/frontend/parser.py b/src/coretrace_python/frontend/parser.py index 13c6c4e..370ecc2 100644 --- a/src/coretrace_python/frontend/parser.py +++ b/src/coretrace_python/frontend/parser.py @@ -1,9 +1,8 @@ from __future__ import annotations import ast -from pathlib import Path -from coretrace_python.source import SourceFile, SourceManager +from coretrace_python.source import SourceFile class ParseError(Exception): @@ -20,10 +19,5 @@ def parse_source(source: str, filename: str = "") -> ast.Module: raise ParseError(f"{filename}:{line}:{column}: {message}") from error -def parse_file(path: str | Path) -> ast.Module: - source = SourceManager().load_file(path) - return parse_source_file(source) - - def parse_source_file(source: SourceFile) -> ast.Module: return parse_source(source.text, filename=str(source.source_id)) diff --git a/src/coretrace_python/frontend/symbols.py b/src/coretrace_python/frontend/symbols.py deleted file mode 100644 index 9bc20b1..0000000 --- a/src/coretrace_python/frontend/symbols.py +++ /dev/null @@ -1,5 +0,0 @@ -"""Frontend support for canonical Python symbols.""" - -from coretrace_python.ir.symbol import SymbolId - -__all__ = ["SymbolId"] diff --git a/src/coretrace_python/hir/nodes.py b/src/coretrace_python/hir/nodes.py index b90c89a..064a723 100644 --- a/src/coretrace_python/hir/nodes.py +++ b/src/coretrace_python/hir/nodes.py @@ -78,7 +78,37 @@ class Call: span: SourceSpan -Expression: TypeAlias = Name | Constant | BinaryOp | UnaryOp | Compare | Attribute | Subscript | Call +@dataclass(frozen=True, slots=True) +class ComprehensionGenerator: + """One ``for target in iterable if condition...`` clause of a comprehension.""" + + target: Name + iterable: Expression + conditions: tuple[Expression, ...] + span: SourceSpan + + +@dataclass(frozen=True, slots=True) +class Comprehension: + """A list, set or generator comprehension; ``kind`` names which one.""" + + kind: str + element: Expression + generators: tuple[ComprehensionGenerator, ...] + span: SourceSpan + + +Expression: TypeAlias = ( + Name + | Constant + | BinaryOp + | UnaryOp + | Compare + | Attribute + | Subscript + | Call + | Comprehension +) @dataclass(frozen=True, slots=True) @@ -126,6 +156,18 @@ class ImportFrom: span: SourceSpan +@dataclass(frozen=True, slots=True) +class Global: + names: tuple[str, ...] + span: SourceSpan + + +@dataclass(frozen=True, slots=True) +class Nonlocal: + names: tuple[str, ...] + span: SourceSpan + + @dataclass(frozen=True, slots=True) class Function: name: str @@ -135,7 +177,26 @@ class Function: span: SourceSpan -Statement: TypeAlias = Assign | Return | ExpressionStatement | Pass | Import | ImportFrom | Function +@dataclass(frozen=True, slots=True) +class Class: + name: str + bases: tuple[Expression, ...] + body: tuple[Statement, ...] + span: SourceSpan + + +Statement: TypeAlias = ( + Assign + | Return + | ExpressionStatement + | Pass + | Import + | ImportFrom + | Global + | Nonlocal + | Function + | Class +) @dataclass(frozen=True, slots=True) diff --git a/src/coretrace_python/ir/__init__.py b/src/coretrace_python/ir/__init__.py index 9b5f406..73a6b58 100644 --- a/src/coretrace_python/ir/__init__.py +++ b/src/coretrace_python/ir/__init__.py @@ -1,6 +1,5 @@ """Python-aware intermediate representation.""" from coretrace_python.ir.model import BasicBlock, FunctionIR, ModuleIR, Value -from coretrace_python.source import SourceSpan -__all__ = ["BasicBlock", "FunctionIR", "ModuleIR", "SourceSpan", "Value"] +__all__ = ["BasicBlock", "FunctionIR", "ModuleIR", "Value"] diff --git a/src/coretrace_python/frontend/lowering.py b/src/coretrace_python/ir/lowering.py similarity index 78% rename from src/coretrace_python/frontend/lowering.py rename to src/coretrace_python/ir/lowering.py index ca5e6cc..a292e6a 100644 --- a/src/coretrace_python/frontend/lowering.py +++ b/src/coretrace_python/ir/lowering.py @@ -5,7 +5,6 @@ from dataclasses import dataclass, field from typing import NoReturn -from coretrace_python.frontend.imports import ImportBindings, collect_imports from coretrace_python.hir import nodes from coretrace_python.ir.model import ( BasicBlock, @@ -26,7 +25,15 @@ Value, ValueInstruction, ) -from coretrace_python.ir.symbol import SymbolId +from coretrace_python.semantic.imports import ImportBindings, collect_imports +from coretrace_python.semantic.scopes import ( + Resolution, + ResolutionKind, + Scope, + ScopeAnalysis, + analyze_scopes, +) +from coretrace_python.semantic.symbols import SymbolId class LoweringError(Exception): @@ -36,8 +43,10 @@ class LoweringError(Exception): @dataclass class _FunctionLowerer: imports: ImportBindings + scopes: ScopeAnalysis + scope: Scope next_value_id: int = 0 - locals: set[str] = field(default_factory=set) + assigned: set[str] = field(default_factory=set) parameters: dict[str, Value] = field(default_factory=dict) block: BasicBlock = field(default_factory=lambda: BasicBlock("entry")) @@ -61,9 +70,12 @@ def emit(self, instruction: ValueInstruction) -> Value: def emit_effect(self, instruction: StoreLocal | Return) -> None: self.block.instructions.append(instruction) + def resolve(self, name: str) -> Resolution: + return self.scopes.resolve(self.scope.id, name) + def imported_symbol(self, node: nodes.Expression) -> SymbolId | None: if isinstance(node, nodes.Name): - if node.identifier in self.locals or node.identifier in self.parameters: + if self.resolve(node.identifier).kind is not ResolutionKind.GLOBAL: return None return self.imports.resolve(node.identifier) if isinstance(node, nodes.Attribute): @@ -76,11 +88,14 @@ def expression(self, node: nodes.Expression) -> Value: if imported_symbol is not None: return self.emit(Symbol(self.new_value(), node.span, imported_symbol)) if isinstance(node, nodes.Name): - if node.identifier in self.locals: - return self.emit(LoadLocal(self.new_value(), node.span, node.identifier)) - if node.identifier in self.parameters: + resolution = self.resolve(node.identifier) + if resolution.kind is ResolutionKind.FREE: + self.fail(node, "closures are not supported yet") + if resolution.kind is not ResolutionKind.LOCAL: + return self.emit(Global(self.new_value(), node.span, node.identifier)) + if node.identifier in self.parameters and node.identifier not in self.assigned: return self.parameters[node.identifier] - return self.emit(Global(self.new_value(), node.span, node.identifier)) + return self.emit(LoadLocal(self.new_value(), node.span, node.identifier)) if isinstance(node, nodes.Constant): return self.emit(Constant(self.new_value(), node.span, node.value)) if isinstance(node, nodes.BinaryOp): @@ -111,9 +126,11 @@ def expression(self, node: nodes.Expression) -> Value: def statement(self, node: nodes.Statement) -> None: if isinstance(node, nodes.Assign): + if self.resolve(node.target.identifier).kind is not ResolutionKind.LOCAL: + self.fail(node, "assignment to a global or nonlocal name is not supported yet") value = self.expression(node.value) self.emit_effect(StoreLocal(None, node.span, node.target.identifier, value)) - self.locals.add(node.target.identifier) + self.assigned.add(node.target.identifier) return if isinstance(node, nodes.Return): return_value = self.expression(node.value) if node.value is not None else None @@ -122,7 +139,8 @@ def statement(self, node: nodes.Statement) -> None: if isinstance(node, nodes.ExpressionStatement): self.expression(node.expression) return - if isinstance(node, nodes.Pass): + if isinstance(node, (nodes.Pass, nodes.Global)): + # ``global`` is a declaration already applied by scope analysis. return self.fail(node) @@ -137,10 +155,12 @@ def function(self, node: nodes.Function) -> FunctionIR: def lower_module(module: nodes.Module) -> ModuleIR: imports = collect_imports(module) + scopes = analyze_scopes(module) functions: list[FunctionIR] = [] for statement in module.body: if isinstance(statement, nodes.Function): - functions.append(_FunctionLowerer(imports).function(statement)) + lowerer = _FunctionLowerer(imports, scopes, scopes.scope_for(statement)) + functions.append(lowerer.function(statement)) elif isinstance(statement, (nodes.Import, nodes.ImportFrom)): continue elif ( diff --git a/src/coretrace_python/ir/model.py b/src/coretrace_python/ir/model.py index dca96ed..9ef738a 100644 --- a/src/coretrace_python/ir/model.py +++ b/src/coretrace_python/ir/model.py @@ -3,7 +3,7 @@ from dataclasses import dataclass from typing import TypeAlias -from coretrace_python.ir.symbol import SymbolId +from coretrace_python.semantic.symbols import SymbolId from coretrace_python.source import SourceSpan diff --git a/src/coretrace_python/semantic/__init__.py b/src/coretrace_python/semantic/__init__.py new file mode 100644 index 0000000..356673c --- /dev/null +++ b/src/coretrace_python/semantic/__init__.py @@ -0,0 +1 @@ +"""Semantic analyses computed from PyHIR: imports, symbols and lexical scopes.""" diff --git a/src/coretrace_python/frontend/imports.py b/src/coretrace_python/semantic/imports.py similarity index 97% rename from src/coretrace_python/frontend/imports.py rename to src/coretrace_python/semantic/imports.py index 889319a..8b2944b 100644 --- a/src/coretrace_python/frontend/imports.py +++ b/src/coretrace_python/semantic/imports.py @@ -6,7 +6,7 @@ from dataclasses import dataclass from coretrace_python.hir import nodes -from coretrace_python.ir.symbol import SymbolId +from coretrace_python.semantic.symbols import SymbolId class ImportResolutionError(Exception): diff --git a/src/coretrace_python/semantic/scopes.py b/src/coretrace_python/semantic/scopes.py new file mode 100644 index 0000000..509e231 --- /dev/null +++ b/src/coretrace_python/semantic/scopes.py @@ -0,0 +1,328 @@ +"""Lexical scope analysis over PyHIR (architecture §4.1). + +Python decides where a name lives per scope, not per statement: any binding anywhere in +a function makes the name local to the whole function, ``global`` and ``nonlocal`` +redirect a name to another scope, class bodies are invisible to the functions they +contain, and comprehensions get a scope of their own. The analysis therefore runs in +two passes: collect every binding of every scope, then resolve names against the +finished tree. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +from enum import Enum +from types import MappingProxyType + +from coretrace_python.hir import nodes +from coretrace_python.source import SourceSpan + + +class ScopeKind(Enum): + MODULE = "module" + FUNCTION = "function" + CLASS = "class" + COMPREHENSION = "comprehension" + + +class BindingKind(Enum): + PARAMETER = "parameter" + LOCAL = "local" + IMPORT = "import" + FUNCTION = "function" + CLASS = "class" + GLOBAL = "global" + NONLOCAL = "nonlocal" + + +class ResolutionKind(Enum): + LOCAL = "local" + FREE = "free" + GLOBAL = "global" + UNBOUND = "unbound" + + +_REDIRECTIONS = frozenset({BindingKind.GLOBAL, BindingKind.NONLOCAL}) + + +class ScopeError(Exception): + """A source-located scoping rule violation, such as an invalid ``nonlocal``.""" + + +@dataclass(frozen=True, order=True) +class ScopeId: + """Stable identity of a scope: source ID, then ``name#ordinal`` per nesting level.""" + + value: str + + def child(self, name: str, ordinal: int) -> ScopeId: + return ScopeId(f"{self.value}::{name}#{ordinal}") + + def __str__(self) -> str: + return self.value + + +@dataclass(frozen=True) +class Binding: + name: str + kind: BindingKind + span: SourceSpan + + +@dataclass(frozen=True) +class Scope: + id: ScopeId + kind: ScopeKind + name: str + parent: ScopeId | None + span: SourceSpan + bindings: Mapping[str, Binding] + + +@dataclass(frozen=True) +class Resolution: + kind: ResolutionKind + scope: ScopeId | None + + +class ScopeAnalysis: + """Immutable scope tree with Python name resolution.""" + + def __init__(self, scopes: tuple[Scope, ...], spans: Mapping[SourceSpan, ScopeId]) -> None: + self._scopes: Mapping[ScopeId, Scope] = MappingProxyType({s.id: s for s in scopes}) + children: dict[ScopeId, list[Scope]] = {scope.id: [] for scope in scopes} + for scope in scopes: + if scope.parent is not None: + children[scope.parent].append(scope) + self._children: Mapping[ScopeId, tuple[Scope, ...]] = MappingProxyType( + {scope_id: tuple(found) for scope_id, found in children.items()} + ) + self._spans = MappingProxyType(dict(spans)) + self._module = scopes[0] + + @property + def module_scope(self) -> Scope: + return self._module + + def scope(self, scope_id: ScopeId) -> Scope: + return self._scopes[scope_id] + + def children(self, scope_id: ScopeId) -> tuple[Scope, ...]: + return self._children[scope_id] + + def scope_for(self, node: nodes.Function | nodes.Class | nodes.Comprehension) -> Scope: + """Return the scope introduced by a definition or comprehension node.""" + + return self._scopes[self._spans[node.span]] + + def resolve(self, scope_id: ScopeId, name: str) -> Resolution: + scope = self.scope(scope_id) + binding = scope.bindings.get(name) + if binding is not None: + if binding.kind is BindingKind.GLOBAL: + return self._global(name) + if binding.kind is BindingKind.NONLOCAL: + target = self._enclosing_function_binding(scope, name) + assert target is not None, "validated at construction" + return Resolution(ResolutionKind.FREE, target) + if scope.kind is ScopeKind.MODULE: + return Resolution(ResolutionKind.GLOBAL, scope.id) + return Resolution(ResolutionKind.LOCAL, scope.id) + + current = self._parent(scope) + while current is not None: + if current.kind is ScopeKind.MODULE: + return self._global(name) + if current.kind is not ScopeKind.CLASS: + enclosing = current.bindings.get(name) + if enclosing is not None and enclosing.kind is BindingKind.GLOBAL: + return self._global(name) + if enclosing is not None and enclosing.kind is not BindingKind.NONLOCAL: + return Resolution(ResolutionKind.FREE, current.id) + current = self._parent(current) + return Resolution(ResolutionKind.UNBOUND, None) + + def _global(self, name: str) -> Resolution: + if name in self._module.bindings: + return Resolution(ResolutionKind.GLOBAL, self._module.id) + return Resolution(ResolutionKind.UNBOUND, None) + + def _parent(self, scope: Scope) -> Scope | None: + return None if scope.parent is None else self._scopes[scope.parent] + + def _enclosing_function_binding(self, scope: Scope, name: str) -> ScopeId | None: + current = self._parent(scope) + while current is not None and current.kind is not ScopeKind.MODULE: + if current.kind is ScopeKind.FUNCTION: + binding = current.bindings.get(name) + if binding is not None and binding.kind not in _REDIRECTIONS: + return current.id + current = self._parent(current) + return None + + +# --------------------------------------------------------------------------- construction + + +class _ScopeBuilder: + def __init__( + self, + scope_id: ScopeId, + kind: ScopeKind, + name: str, + parent: _ScopeBuilder | None, + span: SourceSpan, + ) -> None: + self.id = scope_id + self.kind = kind + self.name = name + self.parent = parent + self.span = span + self.bindings: dict[str, Binding] = {} + self.child_count = 0 + self.declarations: list[tuple[nodes.Nonlocal, str]] = [] + + def bind(self, name: str, kind: BindingKind, span: SourceSpan) -> None: + existing = self.bindings.get(name) + if existing is None: + self.bindings[name] = Binding(name, kind, span) + elif kind in (BindingKind.GLOBAL, BindingKind.NONLOCAL): + # Declarations win over ordinary bindings regardless of statement order. + self.bindings[name] = Binding(name, kind, span) + + def child(self, kind: ScopeKind, name: str, span: SourceSpan) -> _ScopeBuilder: + builder = _ScopeBuilder(self.id.child(name, self.child_count), kind, name, self, span) + self.child_count += 1 + return builder + + def freeze(self) -> Scope: + return Scope( + id=self.id, + kind=self.kind, + name=self.name, + parent=None if self.parent is None else self.parent.id, + span=self.span, + bindings=MappingProxyType(dict(self.bindings)), + ) + + +_COMPREHENSION_NAMES = {"list": "", "set": "", "generator": ""} + + +class _Collector: + def __init__(self, module: nodes.Module) -> None: + self.module = _ScopeBuilder( + ScopeId(str(module.span.source_id)), ScopeKind.MODULE, "", None, module.span + ) + self.builders: list[_ScopeBuilder] = [self.module] + self.spans: dict[SourceSpan, ScopeId] = {} + self.body(module.body, self.module) + + def body(self, statements: tuple[nodes.Statement, ...], scope: _ScopeBuilder) -> None: + for statement in statements: + self.statement(statement, scope) + + def statement(self, node: nodes.Statement, scope: _ScopeBuilder) -> None: + if isinstance(node, nodes.Assign): + self.expression(node.value, scope) + scope.bind(node.target.identifier, BindingKind.LOCAL, node.target.span) + elif isinstance(node, nodes.Return): + if node.value is not None: + self.expression(node.value, scope) + elif isinstance(node, nodes.ExpressionStatement): + self.expression(node.expression, scope) + elif isinstance(node, nodes.Import): + for alias in node.names: + local_name = alias.as_name or alias.name.partition(".")[0] + scope.bind(local_name, BindingKind.IMPORT, alias.span) + elif isinstance(node, nodes.ImportFrom): + for alias in node.names: + if alias.name != "*": + scope.bind(alias.as_name or alias.name, BindingKind.IMPORT, alias.span) + elif isinstance(node, nodes.Global): + if scope.kind is not ScopeKind.MODULE: + for name in node.names: + scope.bind(name, BindingKind.GLOBAL, node.span) + self.module.bind(name, BindingKind.LOCAL, node.span) + elif isinstance(node, nodes.Nonlocal): + if scope.kind is ScopeKind.MODULE: + raise ScopeError(f"{node.span.display()}: nonlocal declaration at module level") + for name in node.names: + scope.bind(name, BindingKind.NONLOCAL, node.span) + scope.declarations.append((node, name)) + elif isinstance(node, nodes.Function): + scope.bind(node.name, BindingKind.FUNCTION, node.span) + function = self.open(scope, ScopeKind.FUNCTION, node.name, node.span) + for parameter in node.parameters: + function.bind(parameter.name, BindingKind.PARAMETER, parameter.span) + self.body(node.body, function) + elif isinstance(node, nodes.Class): + for base in node.bases: + self.expression(base, scope) + scope.bind(node.name, BindingKind.CLASS, node.span) + self.body(node.body, self.open(scope, ScopeKind.CLASS, node.name, node.span)) + elif not isinstance(node, nodes.Pass): + raise TypeError(f"unknown statement: {node!r}") + + def expression(self, node: nodes.Expression, scope: _ScopeBuilder) -> None: + if isinstance(node, (nodes.Name, nodes.Constant)): + return + if isinstance(node, (nodes.BinaryOp, nodes.Compare)): + self.expression(node.left, scope) + self.expression(node.right, scope) + elif isinstance(node, nodes.UnaryOp): + self.expression(node.operand, scope) + elif isinstance(node, nodes.Attribute): + self.expression(node.value, scope) + elif isinstance(node, nodes.Subscript): + self.expression(node.value, scope) + self.expression(node.key, scope) + elif isinstance(node, nodes.Call): + self.expression(node.callee, scope) + for argument in node.arguments: + self.expression(argument, scope) + for keyword in node.keywords: + self.expression(keyword.value, scope) + elif isinstance(node, nodes.Comprehension): + self.comprehension(node, scope) + else: + raise TypeError(f"unknown expression: {node!r}") + + def comprehension(self, node: nodes.Comprehension, scope: _ScopeBuilder) -> None: + # The first iterable is evaluated in the enclosing scope; everything else runs + # inside the comprehension's own scope. + self.expression(node.generators[0].iterable, scope) + inner = self.open(scope, ScopeKind.COMPREHENSION, _COMPREHENSION_NAMES[node.kind], node.span) + for index, generator in enumerate(node.generators): + if index: + self.expression(generator.iterable, inner) + inner.bind(generator.target.identifier, BindingKind.LOCAL, generator.target.span) + for condition in generator.conditions: + self.expression(condition, inner) + self.expression(node.element, inner) + + def open( + self, parent: _ScopeBuilder, kind: ScopeKind, name: str, span: SourceSpan + ) -> _ScopeBuilder: + builder = parent.child(kind, name, span) + self.builders.append(builder) + self.spans[span] = builder.id + return builder + + +def analyze_scopes(module: nodes.Module) -> ScopeAnalysis: + """Build the scope tree of ``module`` and validate its declarations.""" + + collector = _Collector(module) + analysis = ScopeAnalysis( + tuple(builder.freeze() for builder in collector.builders), collector.spans + ) + for builder in collector.builders: + for declaration, name in builder.declarations: + scope = analysis.scope(builder.id) + if analysis._enclosing_function_binding(scope, name) is None: + raise ScopeError( + f"{declaration.span.display()}: no binding for nonlocal {name!r}" + ) + return analysis diff --git a/src/coretrace_python/ir/symbol.py b/src/coretrace_python/semantic/symbols.py similarity index 100% rename from src/coretrace_python/ir/symbol.py rename to src/coretrace_python/semantic/symbols.py diff --git a/tests/test_architecture.py b/tests/test_architecture.py new file mode 100644 index 0000000..8000d1d --- /dev/null +++ b/tests/test_architecture.py @@ -0,0 +1,139 @@ +"""Architectural boundary tests derived from ``docs/architecture.md``. + +They encode §3.1 (parser objects stop at the frontend), §32 (stable API instead of +internals), §37 (repository structure) and §39 (rules to preserve). + +These tests describe the "Align repository layout with the target structure" roadmap +item. They are expected to remain red until HIR-to-PyIR lowering and import binding +leave ``frontend`` and the CLI stops reaching into frontend submodules. +""" + +from __future__ import annotations + +import ast +from collections.abc import Iterator +from pathlib import Path + +PACKAGE = "coretrace_python" +PACKAGE_ROOT = Path(__file__).resolve().parent.parent / "src" / PACKAGE + +# Only the frontend may touch a concrete parser (§3.1). +PARSER_MODULES = {"ast", "tree_sitter", "tree_sitter_python"} + +# Dependency direction of the pipeline (§2, §37). A package may import only packages +# with a strictly lower layer number, or itself. +LAYERS = { + "source": 0, + "hir": 1, + "frontend": 2, + "semantic": 2, + "cfg": 3, + "ir": 4, + "analysis": 5, + "dataflow": 6, + "abstract": 6, + "interprocedural": 6, + "taint": 7, + "findings": 8, + "plugins": 9, + "reporters": 10, +} + +# Top-level modules that wire the pipeline together and may import any layer. +ROOT_MODULES = {"__init__", "__main__", "cli"} + + +def package_files() -> Iterator[tuple[Path, str, str]]: + """Yield ``(path, layer, dotted module name)`` for every module in the package.""" + + for path in sorted(PACKAGE_ROOT.rglob("*.py")): + parts = path.relative_to(PACKAGE_ROOT).with_suffix("").parts + layer = parts[0] + module = ".".join((PACKAGE, *parts)) + yield path, layer, module + + +def imported_modules(path: Path, module: str) -> Iterator[tuple[int, str]]: + """Yield ``(line, absolute module name)`` for every import in ``path``.""" + + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + if module.endswith(".__init__"): + package = module[: -len(".__init__")] + else: + package = module.rsplit(".", 1)[0] + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + yield node.lineno, alias.name + elif isinstance(node, ast.ImportFrom): + if node.level: + base = package.rsplit(".", node.level - 1)[0] if node.level > 1 else package + target = f"{base}.{node.module}" if node.module else base + else: + target = node.module or "" + yield node.lineno, target + + +def violations() -> dict[str, list[str]]: + found: dict[str, list[str]] = { + "parser": [], + "layers": [], + "frontend_privacy": [], + "undeclared": [], + } + for path, layer, module in package_files(): + location = path.relative_to(PACKAGE_ROOT.parent) + is_root_module = layer in ROOT_MODULES + if not is_root_module and layer not in LAYERS: + found["undeclared"].append(f"{location}: package {layer!r} is not declared in LAYERS") + continue + for line, target in imported_modules(path, module): + top = target.split(".")[0] + if top in PARSER_MODULES and layer != "frontend": + found["parser"].append(f"{location}:{line}: imports parser module {target!r}") + if top != PACKAGE or "." not in target: + continue + target_layer = target.split(".")[1] + if target.startswith(f"{PACKAGE}.frontend.") and layer != "frontend": + found["frontend_privacy"].append( + f"{location}:{line}: imports frontend submodule {target!r};" + f" only `{PACKAGE}.frontend` is public" + ) + if is_root_module or target_layer == layer: + continue + if target_layer not in LAYERS: + found["undeclared"].append( + f"{location}:{line}: imports undeclared layer {target_layer!r}" + ) + elif LAYERS[target_layer] >= LAYERS[layer]: + found["layers"].append( + f"{location}:{line}: {layer!r} (layer {LAYERS[layer]}) must not import" + f" {target_layer!r} (layer {LAYERS[target_layer]})" + ) + return found + + +def test_every_package_declares_its_layer() -> None: + assert violations()["undeclared"] == [] + + +def test_only_the_frontend_imports_a_concrete_parser() -> None: + assert violations()["parser"] == [] + + +def test_dependencies_follow_the_pipeline_direction() -> None: + assert violations()["layers"] == [] + + +def test_frontend_submodules_are_private() -> None: + assert violations()["frontend_privacy"] == [] + + +def test_lowering_no_longer_lives_in_the_frontend() -> None: + assert not (PACKAGE_ROOT / "frontend" / "lowering.py").exists() + assert not (PACKAGE_ROOT / "frontend" / "imports.py").exists() + assert not (PACKAGE_ROOT / "frontend" / "symbols.py").exists() + + +def test_semantic_package_exists() -> None: + assert (PACKAGE_ROOT / "semantic" / "__init__.py").exists() diff --git a/tests/test_hir_builder.py b/tests/test_hir_builder.py index 9b58d56..e9fb106 100644 --- a/tests/test_hir_builder.py +++ b/tests/test_hir_builder.py @@ -1,15 +1,15 @@ from __future__ import annotations -import ast +import pytest -from coretrace_python.frontend.ast_adapter import build_module +from coretrace_python.frontend import ParseError, build_hir from coretrace_python.hir import nodes from coretrace_python.source import SourceManager def build(source_text: str) -> nodes.Module: source = SourceManager().add_source("example.py", source_text) - return build_module(source, ast.parse(source.text, filename=str(source.source_id))) + return build_hir(source) def test_builds_parser_independent_function_and_expression_nodes() -> None: @@ -54,3 +54,88 @@ def test_represents_calls_attributes_and_subscripts() -> None: assert isinstance(returned, nodes.Return) assert isinstance(returned.value, nodes.Subscript) assert isinstance(returned.value.value, nodes.Attribute) + + +# --------------------------------------------------------------------------- next milestone +# The tests below describe the frontend entry point and the PyHIR nodes required by +# Phase 2 scope analysis (docs/architecture.md §3.2, §4.1). They are expected to +# remain red until ``coretrace_python.frontend.build_hir`` and the ``Global``, +# ``Nonlocal``, ``Class`` and ``Comprehension`` nodes exist. + + +def test_frontend_exposes_one_entry_point_returning_pyhir() -> None: + module = build("def answer():\n return 42\n") + + assert isinstance(module, nodes.Module) + assert isinstance(module.body[0], nodes.Function) + + +def test_frontend_entry_point_reports_syntax_errors_with_location() -> None: + with pytest.raises(ParseError, match=r"example.py:1:12"): + build("def broken(:\n") + + +def test_represents_global_and_nonlocal_declarations() -> None: + module = build( + "def outer():\n" + " global total\n" + " def inner():\n" + " nonlocal total, other\n" + ) + outer = module.body[0] + assert isinstance(outer, nodes.Function) + declaration = outer.body[0] + assert isinstance(declaration, nodes.Global) + assert declaration.names == ("total",) + inner = outer.body[1] + assert isinstance(inner, nodes.Function) + nested = inner.body[0] + assert isinstance(nested, nodes.Nonlocal) + assert nested.names == ("total", "other") + assert nested.span.start_line == 4 + assert nested.span.start_column == 9 + + +def test_represents_nested_functions() -> None: + module = build("def outer():\n def inner():\n pass\n return inner\n") + outer = module.body[0] + assert isinstance(outer, nodes.Function) + inner = outer.body[0] + assert isinstance(inner, nodes.Function) + assert inner.name == "inner" + + +def test_represents_classes() -> None: + module = build( + "class Config(Base):\n" + " debug = True\n" + " def read(self):\n" + " return self.debug\n" + ) + config = module.body[0] + assert isinstance(config, nodes.Class) + assert config.name == "Config" + assert len(config.bases) == 1 + assert isinstance(config.bases[0], nodes.Name) + assert config.bases[0].identifier == "Base" + assert isinstance(config.body[0], nodes.Assign) + assert isinstance(config.body[1], nodes.Function) + + +def test_represents_list_comprehensions() -> None: + module = build("def squares(items):\n return [y * y for y in items if y > 0]\n") + function = module.body[0] + assert isinstance(function, nodes.Function) + returned = function.body[0] + assert isinstance(returned, nodes.Return) + comprehension = returned.value + assert isinstance(comprehension, nodes.Comprehension) + assert comprehension.kind == "list" + assert isinstance(comprehension.element, nodes.BinaryOp) + assert len(comprehension.generators) == 1 + generator = comprehension.generators[0] + assert isinstance(generator.target, nodes.Name) + assert generator.target.identifier == "y" + assert isinstance(generator.iterable, nodes.Name) + assert len(generator.conditions) == 1 + assert isinstance(generator.conditions[0], nodes.Compare) diff --git a/tests/test_imports.py b/tests/test_imports.py index ff91741..ab84d67 100644 --- a/tests/test_imports.py +++ b/tests/test_imports.py @@ -2,21 +2,19 @@ import pytest -from coretrace_python.frontend.ast_adapter import build_module -from coretrace_python.frontend.imports import ( +from coretrace_python.frontend import build_hir +from coretrace_python.semantic.imports import ( ImportBindings, ImportResolutionError, collect_imports, ) -from coretrace_python.frontend.parser import parse_source -from coretrace_python.ir.symbol import SymbolId +from coretrace_python.semantic.symbols import SymbolId from coretrace_python.source import SourceManager def bindings_for(source: str) -> ImportBindings: source_file = SourceManager().add_source("imports.py", source) - tree = parse_source(source_file.text, str(source_file.source_id)) - return collect_imports(build_module(source_file, tree)) + return collect_imports(build_hir(source_file)) def test_collects_plain_and_aliased_imports() -> None: diff --git a/tests/test_ir.py b/tests/test_ir.py index 177fd30..7fd6319 100644 --- a/tests/test_ir.py +++ b/tests/test_ir.py @@ -1,14 +1,12 @@ -from coretrace_python.frontend.ast_adapter import build_module -from coretrace_python.frontend.lowering import lower_module -from coretrace_python.frontend.parser import parse_source +from coretrace_python.frontend import build_hir +from coretrace_python.ir.lowering import lower_module from coretrace_python.ir.model import BinaryOp, Return from coretrace_python.source import SourceManager def test_lowering_preserves_locations() -> None: source = SourceManager().add_source("add.py", "def add(a, b):\n return a + b\n") - tree = parse_source(source.text, str(source.source_id)) - module = lower_module(build_module(source, tree)) + module = lower_module(build_hir(source)) instructions = module.functions[0].blocks[0].instructions assert isinstance(instructions[0], BinaryOp) assert instructions[0].location.start_line == 2 diff --git a/tests/test_scopes.py b/tests/test_scopes.py new file mode 100644 index 0000000..2d15873 --- /dev/null +++ b/tests/test_scopes.py @@ -0,0 +1,370 @@ +"""Acceptance tests for lexical scope analysis (``docs/architecture.md`` §4.1). + +This module describes the public behavior of the Phase 2 ``ScopeAnalysis``. It is +expected to remain red until ``coretrace_python.semantic.scopes`` exists. + +Contract under test: + +- ``analyze_scopes(module: hir.Module) -> ScopeAnalysis`` is a pure function over PyHIR. +- ``ScopeAnalysis`` exposes ``module_scope``, ``scope(id)``, ``children(id)`` and + ``resolve(id, name)``. +- ``Scope`` and ``Binding`` are immutable and keyed by a stable ``ScopeId``. +- Resolution follows Python rules: an assignment anywhere in a function makes the name + local for the whole function, class bodies are skipped by nested functions, and + comprehensions have their own scope. +""" + +from __future__ import annotations + +import dataclasses + +import pytest + +from coretrace_python.source import SourceManager + +try: + from coretrace_python.frontend import build_hir + from coretrace_python.semantic.scopes import ( + BindingKind, + ResolutionKind, + Scope, + ScopeAnalysis, + ScopeError, + ScopeKind, + analyze_scopes, + ) +except ImportError as error: # pragma: no cover - red until the semantic layer lands + MISSING = error +else: + MISSING = None + + +@pytest.fixture(autouse=True) +def require_semantic_layer() -> None: + if MISSING is not None: + pytest.fail(f"semantic scope analysis is not implemented yet: {MISSING}") + + +def analyze(source_text: str) -> ScopeAnalysis: + source = SourceManager().add_source("scopes.py", source_text) + return analyze_scopes(build_hir(source)) + + +def child(analysis: ScopeAnalysis, parent: Scope, name: str) -> Scope: + matches = [scope for scope in analysis.children(parent.id) if scope.name == name] + assert len(matches) == 1, f"expected one child scope named {name!r}, found {matches}" + return matches[0] + + +# --------------------------------------------------------------------------- module scope + + +def test_module_scope_is_the_root() -> None: + analysis = analyze("import os\nvalue = 1\n") + module = analysis.module_scope + + assert module.kind is ScopeKind.MODULE + assert module.parent is None + assert module.bindings["os"].kind is BindingKind.IMPORT + assert module.bindings["value"].kind is BindingKind.LOCAL + assert analysis.scope(module.id) is module + + +def test_definitions_bind_their_name_in_the_enclosing_scope() -> None: + analysis = analyze("def foo():\n pass\n\nclass Bar:\n pass\n") + module = analysis.module_scope + + assert module.bindings["foo"].kind is BindingKind.FUNCTION + assert module.bindings["Bar"].kind is BindingKind.CLASS + assert child(analysis, module, "foo").kind is ScopeKind.FUNCTION + assert child(analysis, module, "Bar").kind is ScopeKind.CLASS + + +def test_bindings_record_their_first_binding_site() -> None: + analysis = analyze("x = 1\nx = 2\n") + binding = analysis.module_scope.bindings["x"] + + assert binding.span.start_line == 1 + assert binding.span.start_column == 1 + + +# --------------------------------------------------------------------------- functions + + +def test_same_name_in_module_and_function_are_distinct_bindings() -> None: + # The example from §4.1: the engine must distinguish between the two ``x`` symbols. + analysis = analyze("x = 1\n\ndef foo():\n x = 2\n return x\n") + module = analysis.module_scope + foo = child(analysis, module, "foo") + + assert module.bindings["x"] != foo.bindings["x"] + assert analysis.resolve(foo.id, "x").kind is ResolutionKind.LOCAL + assert analysis.resolve(foo.id, "x").scope == foo.id + assert analysis.resolve(module.id, "x").kind is ResolutionKind.GLOBAL + assert analysis.resolve(module.id, "x").scope == module.id + + +def test_parameters_are_local_bindings() -> None: + analysis = analyze("def add(a, b):\n return a + b\n") + add = child(analysis, analysis.module_scope, "add") + + assert add.bindings["a"].kind is BindingKind.PARAMETER + assert add.bindings["b"].kind is BindingKind.PARAMETER + assert analysis.resolve(add.id, "a").kind is ResolutionKind.LOCAL + + +def test_assignment_anywhere_makes_the_name_local_for_the_whole_function() -> None: + # Python decides locality per function, not per statement order. Reading ``run`` + # before assigning it must not resolve to the module-level import. + analysis = analyze( + "from os import system as run\n\n" + "def execute(callback, command):\n" + " run(command)\n" + " run = callback\n" + ) + execute = child(analysis, analysis.module_scope, "execute") + + assert execute.bindings["run"].kind is BindingKind.LOCAL + assert analysis.resolve(execute.id, "run").kind is ResolutionKind.LOCAL + + +def test_unassigned_name_in_function_resolves_to_module_binding() -> None: + analysis = analyze("import os\n\ndef execute(command):\n os.system(command)\n") + execute = child(analysis, analysis.module_scope, "execute") + + assert "os" not in execute.bindings + resolution = analysis.resolve(execute.id, "os") + assert resolution.kind is ResolutionKind.GLOBAL + assert resolution.scope == analysis.module_scope.id + + +def test_name_bound_nowhere_is_unbound() -> None: + analysis = analyze("def greet(name):\n print(name)\n") + greet = child(analysis, analysis.module_scope, "greet") + + resolution = analysis.resolve(greet.id, "print") + assert resolution.kind is ResolutionKind.UNBOUND + assert resolution.scope is None + assert analysis.resolve(analysis.module_scope.id, "print").kind is ResolutionKind.UNBOUND + + +def test_function_level_import_binds_a_local_name() -> None: + analysis = analyze("def execute(command):\n import os\n os.system(command)\n") + execute = child(analysis, analysis.module_scope, "execute") + + assert execute.bindings["os"].kind is BindingKind.IMPORT + assert analysis.resolve(execute.id, "os").kind is ResolutionKind.LOCAL + + +# --------------------------------------------------------------------------- closures + + +def test_nested_function_reads_enclosing_local_as_free_variable() -> None: + analysis = analyze( + "def outer():\n" + " secret = 1\n" + " def inner():\n" + " return secret\n" + " return inner\n" + ) + outer = child(analysis, analysis.module_scope, "outer") + inner = child(analysis, outer, "inner") + + assert inner.parent == outer.id + assert "secret" not in inner.bindings + resolution = analysis.resolve(inner.id, "secret") + assert resolution.kind is ResolutionKind.FREE + assert resolution.scope == outer.id + + +def test_nested_assignment_shadows_instead_of_capturing() -> None: + analysis = analyze( + "def outer():\n" + " value = 1\n" + " def inner():\n" + " value = 2\n" + " return value\n" + " return inner\n" + ) + outer = child(analysis, analysis.module_scope, "outer") + inner = child(analysis, outer, "inner") + + assert inner.bindings["value"].kind is BindingKind.LOCAL + assert analysis.resolve(inner.id, "value").scope == inner.id + assert analysis.resolve(outer.id, "value").scope == outer.id + + +def test_nonlocal_declaration_rebinds_the_enclosing_variable() -> None: + analysis = analyze( + "def counter():\n" + " count = 0\n" + " def increment():\n" + " nonlocal count\n" + " count = count + 1\n" + " return increment\n" + ) + counter = child(analysis, analysis.module_scope, "counter") + increment = child(analysis, counter, "increment") + + assert increment.bindings["count"].kind is BindingKind.NONLOCAL + resolution = analysis.resolve(increment.id, "count") + assert resolution.kind is ResolutionKind.FREE + assert resolution.scope == counter.id + + +def test_global_declaration_binds_in_the_module_scope() -> None: + analysis = analyze("def configure():\n global total\n total = 1\n") + module = analysis.module_scope + configure = child(analysis, module, "configure") + + assert configure.bindings["total"].kind is BindingKind.GLOBAL + assert module.bindings["total"].kind is BindingKind.LOCAL + resolution = analysis.resolve(configure.id, "total") + assert resolution.kind is ResolutionKind.GLOBAL + assert resolution.scope == module.id + + +def test_free_variable_resolves_through_several_levels() -> None: + analysis = analyze( + "def a():\n" + " x = 1\n" + " def b():\n" + " def c():\n" + " return x\n" + " return c\n" + " return b\n" + ) + a = child(analysis, analysis.module_scope, "a") + b = child(analysis, a, "b") + c = child(analysis, b, "c") + + resolution = analysis.resolve(c.id, "x") + assert resolution.kind is ResolutionKind.FREE + assert resolution.scope == a.id + + +# --------------------------------------------------------------------------- classes + + +def test_class_body_names_are_local_to_the_class_scope() -> None: + analysis = analyze("class Config:\n debug = True\n") + config = child(analysis, analysis.module_scope, "Config") + + assert config.kind is ScopeKind.CLASS + assert config.bindings["debug"].kind is BindingKind.LOCAL + assert analysis.resolve(config.id, "debug").kind is ResolutionKind.LOCAL + + +def test_methods_skip_the_class_scope_when_resolving_names() -> None: + analysis = analyze( + "x = 1\n\n" + "class Holder:\n" + " x = 2\n" + " def read(self):\n" + " return x\n" + ) + module = analysis.module_scope + holder = child(analysis, module, "Holder") + read = child(analysis, holder, "read") + + assert read.parent == holder.id + assert holder.bindings["read"].kind is BindingKind.FUNCTION + resolution = analysis.resolve(read.id, "x") + assert resolution.kind is ResolutionKind.GLOBAL + assert resolution.scope == module.id + + +def test_class_body_reads_enclosing_function_locals() -> None: + analysis = analyze( + "def factory():\n" + " base = 1\n" + " class Product:\n" + " value = base\n" + " return Product\n" + ) + factory = child(analysis, analysis.module_scope, "factory") + product = child(analysis, factory, "Product") + + resolution = analysis.resolve(product.id, "base") + assert resolution.kind is ResolutionKind.FREE + assert resolution.scope == factory.id + + +# --------------------------------------------------------------------------- comprehensions + + +def test_comprehension_has_its_own_scope() -> None: + analysis = analyze( + "def squares(items, threshold):\n" + " return [y * y for y in items if y > threshold]\n" + ) + squares = child(analysis, analysis.module_scope, "squares") + comprehension = child(analysis, squares, "") + + assert comprehension.kind is ScopeKind.COMPREHENSION + assert comprehension.parent == squares.id + assert comprehension.bindings["y"].kind is BindingKind.LOCAL + assert "y" not in squares.bindings + assert analysis.resolve(comprehension.id, "y").kind is ResolutionKind.LOCAL + threshold = analysis.resolve(comprehension.id, "threshold") + assert threshold.kind is ResolutionKind.FREE + assert threshold.scope == squares.id + + +# --------------------------------------------------------------------------- errors + + +@pytest.mark.parametrize( + "source_text, location", + [ + ( + "def outer():\n def inner():\n nonlocal count\n count = 1\n", + "scopes.py:3:9", + ), + ("nonlocal value\n", "scopes.py:1:1"), + ], + ids=["no-enclosing-binding", "module-level"], +) +def test_invalid_nonlocal_is_a_source_located_error(source_text: str, location: str) -> None: + with pytest.raises(ScopeError, match=rf"{location}: .*nonlocal"): + analyze(source_text) + + +# --------------------------------------------------------------------------- result shape + + +def test_results_are_immutable() -> None: + analysis = analyze("x = 1\n") + module = analysis.module_scope + + with pytest.raises(dataclasses.FrozenInstanceError): + module.name = "other" # type: ignore[misc] + with pytest.raises(TypeError): + module.bindings["y"] = module.bindings["x"] # type: ignore[index] + + +def test_results_are_deterministic_across_runs() -> None: + source_text = "x = 1\n\ndef foo(a):\n def bar():\n return a\n return bar\n" + first = analyze(source_text) + second = analyze(source_text) + + assert first.module_scope == second.module_scope + assert first.children(first.module_scope.id) == second.children(second.module_scope.id) + foo = child(first, first.module_scope, "foo") + assert first.children(foo.id) == second.children(child(second, second.module_scope, "foo").id) + + +def test_children_are_listed_in_source_order() -> None: + analysis = analyze("def b():\n pass\n\ndef a():\n pass\n\nclass C:\n pass\n") + names = [scope.name for scope in analysis.children(analysis.module_scope.id)] + + assert names == ["b", "a", "C"] + + +def test_unknown_scope_id_is_rejected() -> None: + analysis = analyze("x = 1\n") + other = analyze("def foo():\n pass\n") + foreign = child(other, other.module_scope, "foo").id + + with pytest.raises(KeyError): + analysis.scope(foreign) diff --git a/tests/test_symbol_resolution.py b/tests/test_symbol_resolution.py index ec04881..e8e7983 100644 --- a/tests/test_symbol_resolution.py +++ b/tests/test_symbol_resolution.py @@ -119,3 +119,40 @@ def test_wildcard_import_fails_with_a_source_location(tmp_path, capsys) -> None: assert output == "" assert "symbols.py:1:1" in error assert "wildcard imports are not supported" in error + + +# --------------------------------------------------------------------------- next milestone +# Lowering must consume the Phase 2 ScopeAnalysis instead of tracking locals itself +# (docs/architecture.md §4.1 and boundary "PyIR lowering consumes semantic results"). +# Expected to remain red until lowering is routed through scope analysis. + + +def test_read_before_assignment_is_local_not_an_import(tmp_path, capsys) -> None: + exit_code, output, error = emit_ir( + "from os import system as run\n\n" + "def execute(callback, command):\n" + " run(command)\n" + " run = callback\n", + tmp_path, + capsys, + ) + + assert exit_code == 0, error + assert "symbol @python.os.system" not in output + assert 'load_local "run"' in output + assert 'store_local "run", %0' in output + + +def test_global_declaration_is_not_a_local(tmp_path, capsys) -> None: + exit_code, output, error = emit_ir( + "import os\n\n" + "def execute(command):\n" + " global os\n" + " os.system(command)\n", + tmp_path, + capsys, + ) + + assert exit_code == 0, error + assert "symbol @python.os.system" in output + assert 'load_local "os"' not in output diff --git a/tests/test_symbols.py b/tests/test_symbols.py index 59bf899..9bff137 100644 --- a/tests/test_symbols.py +++ b/tests/test_symbols.py @@ -1,6 +1,6 @@ import pytest -from coretrace_python.frontend.symbols import SymbolId +from coretrace_python.semantic.symbols import SymbolId def test_symbol_id_adds_the_python_namespace() -> None: