Regions: Stop the address lookups from assuming an order the list may not have

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

An address lookup can walk past a region that does cover the address and report
there is none. On binaries/tests/armel/btrfs.ko, .text spans
0x400000-0x4b9b58 and no address in it can be attributed to a section:

.text spans 0x400000-0x4b9b58
19 other sections start at 0x400000 and end inside it, e.g. .rel.gnu.linkonce.this_module (ends 0x400008), .rel.ARM.exidx.exit.text (ends 0x400010), .note.gnu.build-id (ends 0x400024)
find_section_containing(0x400024) -> None
find_section_containing(0x400100) -> None
find_section_containing(0x4b9b00) -> None

find_section_containing is what tells a consumer which bytes are code, so on
this object every block in .text looks like it belongs to no section at all.

Root cause

Regions keeps its members sorted by start address and then bisects their end
addresses:

    We assume none of the regions overlap with others.

That assumption makes the two orders the same order. It does not hold here: the
19 non-allocated sections of a kernel module all state sh_addr 0 and land on
.text's start, so the end-address sequence is 0x400008, 0x400010, 0x400024, … , 0x4b9b58 interleaved with .text last. The bisection lands on the first
region whose end exceeds the address, finds a region that starts at 0x400000
but ends before the address it was asked about, and returns None -- one
short-ended neighbour is enough to hide every region behind it.

Fix

The lookups now carry a running maximum of the end addresses beside the sorted
list: _max_end[i] is the highest end address among the first i + 1 regions.
That sequence is nondecreasing whether or not the regions overlap, so the
bisection over it means something in either case, and where the regions are
disjoint it is each region's own end address and both reduce to exactly the
bisection they were. .tbss is also kept out of the ELF section list, since its
address is where a thread's own copy of the template begins rather than a range of
the image, and the linker places the following section over the top of it.

find_section_containing(0x400024) -> <.text | offset 0x58, vaddr 0x400000, size 0xb9b58>
find_section_containing(0x400100) -> <.text | offset 0x58, vaddr 0x400000, size 0xb9b58>
find_section_containing(0x4b9b00) -> <.text | offset 0x58, vaddr 0x400000, size 0xb9b58>

Testing

tests/test_regions.py::TestOverlappingRegions::test_lookups_survive_an_overlap
puts a small region inside a larger one and asserts the four lookups; on the
merge base find_region_containing(0x3500) returns None.
test_tbss_does_not_answer_for_what_is_over_it checks the exclusion on
binaries/tests/x86_64/libc.so.6, where .tbss covers .init_array and __libc_subfreeres, which hold
the bytes at those addresses. btrfs.ko above is the reproducer in the wild, not
a fixture the tests load.

Fixes #742. Validation: #760 (comment)

session: sharpen

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head 9aa4acf4151ddfade8f40338fc495357f73aa395 against baseline 46a37333f4f59b0facf8774ee743ebc4cc074e9b.

• Regression: pytest tests/test_regions.py::TestOverlappingRegions — both cases fail on the baseline; passes on head
• Focused: pytest tests/test_regions.py tests/test_overlap.py tests/test_blob.py tests/test_namedregion.py — 7 passed
• Full suite: cle 229 passed, 9 skipped; angr 2479 passed, 46 skipped, 2 xfailed, 260 subtests; angr Rust 35 passed
• Lint/type: run-ci-diff-checks.py --repository cle — pylint and pyright unchanged or better on every changed file
• Workspace gate: cle and angr adopted in one feature instance; archinfo, pypcode, pyvex, claripy and angr-management skipped as unadopted and untouched — pass

Reproducer on a public fixture: cle.Loader("binaries/tests/armel/btrfs.ko").main_object.find_section_containing(0x400024) returns None on the baseline although .text spans 0x400000-0x4b9b58; on head it returns .text.

Both lookups were evaluated against the same already-loaded object, so the comparison contains nothing but the lookup. Over the 821 loadable files in angr/binaries (817 scored, 108,301 probed addresses):

	baseline	head
Objects whose sorted region list overlaps	106	51
find_region_containing answers differing	—	404 over 71 objects
find_region_next_to answers differing	—	404 over 71 objects
max_addr differing	—	49 objects

• Deterministic deltas, find_region_containing: 308 addresses where the baseline answered .tbss, which holds none of the bytes there, and now answer the section that does; 86 where the baseline answered None although a region covers the address; 10 inside .tbss and nothing else, which now answer None because no section holds those bytes.
• Deterministic deltas, find_region_next_to: 318 .tbss answers replaced, 71 None answers replaced, and 15 on btrfs.ko, armhf/libc.so.6 and s390x/libstdc++.so.6 where the baseline bisection skipped past the first region whose end is beyond the address.
• max_addr is higher on all 49, 48 of them relocatable objects; nothing in cle, angr or angr-management reads Regions.max_addr.
• Breadth: a further 453 corpus objects sampled 30 per backend, 253 scored across Universal2, PE, XBE, UEFI, TE, COFF, CGC, ELF and Mach-O. Only two ELF objects differ at all: one .tbss overlap, and one where the baseline lost four addresses inside .text.
• Timing: 200,000 random lookups on binaries/tests/x86_64/libc.so.6 take 0.171 s on the baseline and 0.033 s on head; on binaries/tests/armel/btrfs.ko, 0.139 s and 0.131 s.

Caveats: corpus objects are referred to by architecture, container and digest because the dataset is not public. The remaining 51 overlapping objects are relocatable ELFs placing several sections at one address, which #739 addresses separately.

Re-keyed 2026-08-28. The figures above were measured at b759dea9561d01614b16a1068febf7191e494b34 on baseline 45c6509c753d07f740099035cd41f7f473dc6f31, which is the head the opening line named until now; the branch is at 9aa4acf4151ddfade8f40338fc495357f73aa395 on 46a37333f4f59b0facf8774ee743ebc4cc074e9b. git range-diff 45c6509c753d07f740099035cd41f7f473dc6f31..b759dea9561d01614b16a1068febf7191e494b34 46a37333f4f59b0facf8774ee743ebc4cc074e9b..9aa4acf4151ddfade8f40338fc495357f73aa395 reports every commit unchanged and git diff b759dea9561d01614b16a1068febf7191e494b34 9aa4acf4151ddfade8f40338fc495357f73aa395 differs only by master's own advance (12 files changed, 353 insertions(+), 44 deletions(-)). Master touched none of the files this change touches between the two baselines, so every figure above still describes this head.

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_760

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Section lookups over an ELF whose non-allocated sections all land on the start of .text.

the reproducer

import logging, os
logging.getLogger("cle").setLevel(logging.CRITICAL)
import cle

BIN = os.environ["BINARIES"]   # a checkout of angr/binaries
T = lambda *p: os.path.join(BIN, "tests", *p)

path = T("armel", "btrfs.ko")
obj = cle.Loader(path, auto_load_libs=False).main_object
text = next(s for s in obj.sections if s.name == ".text")
print(f".text spans {text.vaddr:#x}-{text.vaddr + text.memsize:#x}")
sharing = [s for s in obj.sections if s.vaddr == text.vaddr and s is not text and s.memsize]
print(f"{len(sharing)} other sections start at {text.vaddr:#x} and end inside it, e.g. "
      + ", ".join(f"{s.name} (ends {s.vaddr + s.memsize:#x})" for s in sorted(sharing, key=lambda s: s.memsize)[:3]))
for addr in (0x400024, 0x400100, 0x4B9B00):
    print(f"find_section_containing({addr:#x}) -> {obj.find_section_containing(addr)}")

Before — the bisection over end addresses stops at a short-ended neighbour and reports no section covers the address:

cle master at 46a37333f4f59b0facf8774ee743ebc4cc074e9b

.text spans 0x400000-0x4b9b58
19 other sections start at 0x400000 and end inside it, e.g. .rel.gnu.linkonce.this_module (ends 0x400008), .rel.ARM.exidx.exit.text (ends 0x400010), .note.gnu.build-id (ends 0x400024)
find_section_containing(0x400024) -> None
find_section_containing(0x400100) -> None
find_section_containing(0x4b9b00) -> None

After — the running maximum of the end addresses finds .text:

with this change, at 9aa4acf4151ddfade8f40338fc495357f73aa395

.text spans 0x400000-0x4b9b58
19 other sections start at 0x400000 and end inside it, e.g. .rel.gnu.linkonce.this_module (ends 0x400008), .rel.ARM.exidx.exit.text (ends 0x400010), .note.gnu.build-id (ends 0x400024)
find_section_containing(0x400024) -> <.text | offset 0x58, vaddr 0x400000, size 0xb9b58>
find_section_containing(0x400100) -> <.text | offset 0x58, vaddr 0x400000, size 0xb9b58>
find_section_containing(0x4b9b00) -> <.text | offset 0x58, vaddr 0x400000, size 0xb9b58>