lightspeed: trace child Python processes in the coverage pass

[ArjunS07]

Purpose

The coverage pass records which project files each benchmark executes. It only measured the benchmark's own process. Some benchmarks run project code in a child Python process. An example is scikit-image's benchmark_import_time.ImportSuite.time_import, which runs subprocess.run(f"{sys.executable} -c 'import {pkg}'", shell=True). For these benchmarks the dependency DB had no Python files of the project, so a patch that changes exactly what they measure did not select them.

This change measures child Python processes too, during the coverage pass only.

How it works

flowchart LR
    A[survey_one] --> B[temp dir: coveragerc + sitecustomize.py]
    B --> C[set COVERAGE_PROCESS_START and PYTHONPATH]
    C --> D[run benchmark under coverage]
    D --> E[child python starts coverage, writes .coverage.* file]
    E --> F[read child lines, add to parent lines]
    F --> G[restore environment, delete temp dir]
    G --> H[fingerprints and deps DB as before]

Loading

1. For each benchmark, survey_one makes a temporary directory with a coverage config file (parallel = True, data file in that directory, same source and omit as the parent) and a sitecustomize.py.
2. While the benchmark function runs, COVERAGE_PROCESS_START points to that config and the directory is put first on PYTHONPATH. A child Python interpreter then imports the sitecustomize.py, which calls coverage.process_startup(). After that it runs the environment's own sitecustomize, if there is one, because ours hides it.
3. After the benchmark function returns, the executed lines from the child data files are added to the lines from the parent. The rest of the pass (fingerprints, C extension scan, DB rows) does not change.
4. The environment variables are restored and the directory is deleted before the function returns. The timing pass starts after this, so it does not see any of it.

Changes

• asv/contrib/lightspeed/survey.py: new _child_coverage context manager and _child_lines reader.
  • Effect: files that only a child Python process executes are now dependencies of the benchmark.
  • Before: only the benchmark's own process was measured. A benchmark that ran project code only in a child got no Python file rows. For scikit-image#7035, time_import got only Cython and C rows from the extension scan.
• survey_one: the parent coverage object also disables the no-data-collected warning.
  • Effect: no warning when the parent executes no project code and the child does.
  • Before: coverage printed this warning (or raised it, under -W error) before the child data could be added.
• test/test_lightspeed_survey.py: new test test_child_process_files_are_deps.
  • Effect: a benchmark that runs python -c 'import pkg.child' in a shell must list child.py as a dependency, and os.environ must be the same after the pass.
  • Before: there was no test for child processes. The fixture now also removes the benchmarks package from sys.modules, so two tests can each load their own temp suite.

Verification

• pytest test/test_lightspeed_survey.py test/test_lightspeed_project_imports.py: 7 passed (Python 3.12, coverage 7.16.2).
• pytest test/test_lightspeed_survey.py on Python 3.8 with coverage 7.5.4 and with coverage 5.5: 2 passed each. These versions do not install coverage's own a1_coverage.pth, so this checks the sitecustomize.py path.
• Without the change, the new test fails (no_source_root_coverage, or the no-data-collected warning under the test suite's -W error).
• End to end: asv initialize_diffcheck on a small project with one benchmark that imports pkg.child in a child process. The deps DB has pkg/child.py for that benchmark, the baseline timing is recorded, and no temporary directories remain.
• A sitecustomize.py that is already on PYTHONPATH still runs in the child.

Notes

• Coverage 7.10 and later install a .pth file that already calls process_startup() when COVERAGE_PROCESS_START is set. Our sitecustomize.py then calls it a second time, and that call does nothing. With older coverage, our file is the one that starts it.
• These children are not measured: children started with python -I or -E, children given their own env= without PYTHONPATH and COVERAGE_PROCESS_START, interpreters without coverage installed, and children still running after the benchmark function returns.
• The C extension scan still looks only at modules loaded in the parent process.
• Child lines are read directly from the data files and not merged with CoverageData.update. With coverage 5.5, an update into the in-memory parent data lost all lines.