Skip to content

task template: paired timing builds both sides the same way - #47

Merged
ArjunS07 merged 1 commit into
survey/phase1-template-shadow-conftestfrom
fix/paired-rebuild-both-sides
Oct 8, 2026
Merged

ArjunS07 merged 1 commit into
survey/phase1-template-shadow-conftestfrom
fix/paired-rebuild-both-sides

Conversation

@ArjunS07

@ArjunS07 ArjunS07 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Purpose

Make paired timing compare the patch, not two different builds. When a patch touches compiled sources, only the patched tree is rebuilt, and the base copy keeps the image's build.

How it works

test.sh rebuilds the patched tree (FC_REBUILD_CMD, rebuild.sh) before timing. The base copy /workspace/.fc_base was copied before that, so it still has the image's original build. If the trial's rebuild uses different flags or a different path than the image build (for example, setup.py build_ext --inplace after both pip install -e attempts fail), the timed difference includes the build difference.

lsv_measure.py now runs FC_REBUILD_CMD on the base copy at the repo path (with base_in_place(), as for the timing rounds), then on the patched tree again, before paired timing. Both sides come from the same command at the same path. A failed rebuild ends the measure with an error instead of timing two different builds. The log is rebuild_both.log.

Changes

  • template/tests/lsv_measure.py: rebuild_both_sides(), called before measure_paired when FC_REBUILD_CMD is set.
    • Effect: base and patched trees are built the same way.
    • Before: numpy#21464 (in-place fallback build) showed the same shift in all 3 no-change runs as in the oracle runs. About 520 of 3,481 benchmarks were above log 0.3 and about 50 below -0.3, while the oracle patch is 14 lines in ufunc_object.c. In the survey's approved records with rebuilds, the share of no-change benchmarks with |log speedup| > 0.1 is 0.29 (numpy#21464), 0.42 (bottleneck#329) and 0.16 (bottleneck#327), against at most 0.07 for the other 20.
  • tests/docker/test_lsv_paired_measure.py: the rebuild runs base first and then patched, both at the repo path, and a failed base rebuild leaves the patched tree in place.

Verification

  • The 2 new tests pass. They fail without the change, because the function is missing.
  • tests/docker: the 7 failures present without the change are unchanged.
  • To do: re-measure numpy#21464, bottleneck#329 and bottleneck#327 with this template. The no-change share above should drop to the level of the other tasks.

Notes

  • Each compiled task's trial now rebuilds twice more: once for the base and once for the patched tree again.

@ArjunS07
ArjunS07 merged commit c11b26e into survey/phase1-template-shadow-conftest Oct 8, 2026
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant