[SizzleUnrlsd]

Summary

Closes #264.

A module in a directory without __init__.py inside a regular package was named from that directory. Its relative imports were rejected as relative import beyond top-level package, and the file was not analysed. Python, however, takes such a directory for a portion of the package (PEP 420): pkg/sub/app.py is pkg.sub.app.

A module now has one identity, its name after its packages, and may be imported by more than one name, each from its own root:

• Module name: it runs from the parent of the outermost directory holding an __init__.py. That directory is searched up through directories without one as far as the project root; above the root, it climbs only while each directory holds an __init__.py. This name is the module's node in the graph, the symbol of its functions, and the base of its relative imports.
• Script name: the name a script run from the module's own directory gives it, sys.path[0] being that directory, which is the name the engine gave it until now. It is kept as a second import name, rooted at that directory.
• Resolution: an import is resolved by any of these names. When several files share a name, the importer's roots are tried in order: its script root first, then its package's root, as a script finds what lies beside it before what the import path holds. A script name answers only an importer with that directory among its roots, however few files have it.
• Summaries: a module's summaries are computed once. Each module reads the others in an index of its own, a member of an import cycle too: a module it imports under the name it resolved it by, a module reached through another under its module name only.

The name a module had depends on nothing in its content.

Commit	Change
test(modules): cover the names and imports that must survive #264	what must be kept: names in and outside packages, the project root as a bound, a script's sibling, a library named like a sibling, two helpers of one short name
refactor(source): name a file's module and its import root in one place	SourceFile.import_names; the two climbs become one; behaviour unchanged
refactor(modulegraph): resolve an import by any name of a module, from the importer's roots in order	several names per module; the resolution of each import is recorded; behaviour unchanged
refactor(engine): register a module's summaries under every name it is reached by	summaries, members and worker batches with several names; behaviour unchanged
fix(cache): key a module by the names it is imported by	names in the module key; CACHE_FORMAT 19
test(engine): cover the note on an import no root decides	that note is kept
fix(source): name a module in a directory without __init__.py after its package	the module name; the script name kept; ambiguity noted on shared module names only
test(modules): a module reached by both its names keeps one identity	one node, covered once, both importers
test(modules): the result does not depend on the cache or the processes	without a cache, cold, warm, two processes
test(modules): a script's helper comes first, under the name it resolved	root order; a member's own resolution
docs: state how a module in a directory without __init__.py is named	docs/usage.md
docs(changelog): announce the names of modules in directories without __init__.py	CHANGELOG.md
test(regression): record pygoat with its playground modules analysed	the pygoat snapshot, below
fix(modulegraph): answer a module's short name only from its own directory	review: a short name hid the library
fix(engine): register an imported module only under the names the graph resolved	review: an unresolved short name was registered
docs: state that a module's short name answers only its own directory	docs/usage.md, CHANGELOG.md
fix(engine): register a module reached through another under its module name only	review: a name resolved in between was registered
test(modules): a name one member of a cycle resolved does not change another	review: each member of a cycle keeps its own names
refactor(engine): decide in one place the names a component reads each module by	members and imports named by one function; behaviour unchanged
fix(engine): give each member of a cycle the names it resolved	review: a short name was seen across a cycle
test(modules): each member of a cycle reads the helper it resolved	two members, two helpers
test(modules): follow a member of a cycle reached through its package by its module name	pkg.helpers inside a cycle
test(modules): take a module name two reached files share for neither	src and lib, each with a pkg.helpers
perf(engine): share one summary index among the members of a cycle that read alike	Django's cost back to main's

Results

The requirements

Requirement	Test
script-sibling and script-shadow keep their alerts	test_a_script_reaches_the_module_beside_it_by_its_short_name, test_a_module_beside_a_script_named_like_a_library_leaves_the_library_modelled
two roots, each with a helper.py, each use their own	test_each_importer_reaches_the_helper_of_its_own_directory, with the sink in either; test_a_script_reaches_the_helper_beside_it_before_one_at_the_project_root; test_a_script_keeps_the_helper_it_resolved_beside_another_of_that_short_name
a file reached by both its names keeps one identity	test_a_module_reached_by_both_its_names_keeps_one_identity: one module pkg.tools.helper, one coverage entry, both importers report the flow
the same results with a cache, without one and in parallel	test_the_result_does_not_depend_on_the_cache_or_the_processes: no cache, cold cache, warm cache, jobs=2, and jobs=2 with a cache

Projects

• flask 3.1.0 (ab814966, without tests/): sansio/app.py, blueprints.py and scaffold.py are analysed. The three syntax-error notes are gone, files analysed go from 37 to 40 and functions from 413 to 531, and no other finding changes.
• pygoat (regression corpus): A9/api.py and A9/archive.py are analysed and reach A9/main.py. The two main.py files have distinct module names, so neither carries an ambiguous-module note.
• The other 18 regression projects: their snapshots are unchanged.

The pygoat snapshot

   "coverage": {
     "files": 80,
-    "files_analysed": 74,
-    "functions": 295,
-    "functions_analysed": 295
+    "files_analysed": 78,
+    "functions": 303,
+    "functions_analysed": 303
   },

Four findings are removed and none added:

• ambiguous-module on dockerized_labs/insec_des_lab/main.py and on introduction/playground/A9/main.py;
• syntax-error on introduction/playground/A9/api.py and on introduction/playground/A9/archive.py.

No vulnerability finding changes.

Review

Four false negatives, each reported by main and lost with or without a cache and in two processes: three on the first thirteen commits, one on the fixes of the first three.

A short name hid the library. The project had:

• pkg/__init__.py;
• pkg/config.py, defining VALUE;
• pkg/sub/os.py, which imports VALUE from ..config and defines system(command), returning None;
• main.py, at the project root, which imports os and calls os.system(input()).

os is the short name of pkg/sub/os.py from pkg/sub only. Unique, it still answered main.py's import os, and the summary of system hid the model of os.system, whereas Python runs the library. A short name now answers only an importer with its directory among its roots, so main.py reports command-injection. The same file without its relative import was analysed on main too, as a top-level module of pkg/sub, and took the import there already; it no longer does.

An unresolved short name was registered. The project had pkg/a/helper.py, pkg/b/helper.py and a main.py that imports helper, imports noop from pkg.a.helper, and calls os.system(helper.execute(input())). Reached by its full name, pkg.a.helper was also registered under helper, which the graph had not resolved for main.py, so main.py's helper.execute read pkg/a's, which returns a constant.

A name resolved in between was registered. Beside pkg/sub/os.py, a pkg/sub/loader.py imports os, and main.py imports os and noop from pkg.sub.loader. From pkg/sub, loader.py's os is pkg/sub/os.py, which the component of main.py then took for main.py's os too.

A short name was seen across a cycle. main.py imports os and noop from pkg.sub.os, and pkg/sub/os.py imports main. The two files are one component, whose members shared one index: pkg/sub/os.py's functions went under all its names, os among them, and main.py's os.system read the summary of its system. Python runs the library: main.os is the standard os. A name one member resolved from its own directory changed the others' the same way, as pkg/sub/loader.py's os did from such a cycle.

The rule now. Each module reads the others in an index of its own, a member of a cycle too:

• itself under its own names, each of which names it from its own roots;
• a module it imports under the name it resolved it by;
• a module reached through another under its module name only, the name the whole project knows it by, as pkg.helpers is reached through import pkg when its __init__.py imports it;
• a module name two reached files share under neither, unless the module imports one of them directly;
• a name it left unresolved under none.

What a callee, or another member of a cycle, resolved never changes what a module's names mean. Two members that resolve one short name to two files now each read their own, where the shared index put it under neither.

What changes for the ambiguity note. From the project root, helper is now no module of the project, since only other directories' short names match it. Python agrees: run from the root, import helper raises ModuleNotFoundError. That import is neither followed nor noted as ambiguous-module any more, and the test of that note pins this instead. Files that share a module name are still noted, as is an import of such a name that no root decides.

What the tests cover. The new file tests/test_namespace_module_visibility.py has thirteen tests, each run without a cache, with a cold and a warm one, and in two processes:

• the library left to a script elsewhere, with and without a relative import;
• the unresolved short name;
• the short name resolved in between;
• a name a member leaves unresolved, not taken from a module it imports;
• a name one member of a cycle resolved, not taken for another;
• a member's own resolution kept over one an import made;
• pkg.helpers followed through import pkg, and through it inside a cycle;
• the short name seen across a cycle, and the one resolved in between from a cycle;
• a module that imports itself by its short name, which reads its own functions;
• two members of a cycle, each reading its own helper;
• a module name two reached files share, taken for neither.

On the commit before each fix, its tests of the new behaviour fail in all four runs, and the guards pass.

Each of these mutations makes at least one of them fail:

• short names visible from anywhere;
• modules reached through another put under every name they have;
• no module reached through another, or no member of the cycle reached through another;
• names a module left unresolved taken from its imports, or from another member of its cycle;
• one index for a whole cycle, or every member under all its names for every other;
• a module not under its own names;
• a module name two reached files share taken for either;
• members that read differently sharing an index.

Cost. Django 5.1.4 (879 files, a cycle of 144 modules) takes 176 s on the commit before the fix. With an index for each member, it took 185 s; members that read alike now share one, and it takes 175 s, with the same 42 findings. The regression corpus, whose largest cycle has 14 modules, keeps its times.

Compatibility

• Cache: CACHE_FORMAT goes to 19, so caches miss once.
• Project symbols: the symbols of a renamed module's functions change with its name, python.pkg.sub.app.view instead of python.app.view. A model declared on the former symbol must follow.
• Baseline keys: unchanged, since they hold the file, the rule, the function's qualified name and a digest, never a module name. A finding of a renamed module recorded in a baseline before is still recognised by it.
• Ambiguity notes: ambiguous-module now notes the files that share a module name, and an import of such a name no root decides. Files that share only their script name are told apart by the importer's directory, and an import that only other directories' script names match names no module of the project.

Not in this change

• Single-file checks: a lone file is still named from its first directory without an __init__.py, since a single-file check has no project root.
• Scripts in a regular package: a script inside a regular package does not get its directory as a root, which it would have if run directly. That was not the case before either, and it would be new behaviour.
• The remote battery: its 38 projects could not be reached; flask and the regression corpus were measured instead.

Tests

Three new files, tests/test_namespace_module_names.py, tests/test_namespace_module_imports.py and tests/test_namespace_module_visibility.py, with 69 tests: 5, 12 and 52, the last file's thirteen tests each run four ways. Apart from the pygoat snapshot, no test file existing on main changes.

• Before the fix: on the commit before it, the 7 tests of the new behaviour fail, and the 10 others pass and pin what is kept.
• Python's view: test_python_imports_every_module_by_its_name imports every module by its name in a subprocess, with the project's src on the import path.

Each of these mutations makes at least one test fail:

• the package's root tried before the script root;
• the members' own resolutions ignored;
• ambiguous-module noting names shared only as script names;
• the search not bounded by the project root;
• worker processes naming modules without the project root.

Verification

Python 3.12, on main at f7afd78:

• mypy and ruff check .: clean;
• unit suite: 2,574 passed, and every commit of the review green;
• regression: 19 of 19 with the pygoat snapshot, 18 of 19 without it;
• Enterprise plugins at b1c4ed3: 978 passed.