Load a bare DEX

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Problem

There is no backend for a bare .dex, so one cannot be loaded at all. On
classes.dex extracted from tests/java/android1.apk, detection finds nothing
and naming the backend does not help either:

detected automatically     -> CLECompatibilityError: Unable to find a loader backend
                              for /tmp/.../classes.dex.  Perhaps try the 'blob' loader?
backend named explicitly   -> CLEError: Invalid backend: dex

The bytecode inside an APK is already loadable through Apk; the same bytes on
their own are not.

Root cause

The gap is registration, not lifting: Soot reads a bare dex unchanged, and cle
has no Backend subclass claiming the dex\n magic and no "dex" entry in
the backend registry, so Loader never reaches the lifter.

Fix

Dex subclasses Soot the way Apk does, recognises the dex\n magic so
detection finds it unaided, and forwards a new android_api_version load
option to the lifter.

Requiring that option rather than defaulting it is the decision worth
disagreeing with: a bare dex has no manifest, and the header's version is the
format version, not the API level, so deriving one silently resolves against
the wrong class library. Defaulting to the highest platform in the SDK
directory is the alternative; say so and I will change it.

Testing

tests/test_dex.py uses classes.dex out of the existing android1.apk
fixture. Both of its tests assert what the loader now demands —
test_dex_needs_an_android_sdk and test_dex_needs_an_api_version — and both
fail on master at backend selection, before either option is looked at. Neither
test lifts the file: that needs an Android SDK on the machine, which the runner
here does not have, so the end-to-end load is not covered by this branch. Its
lifter dependency, pysoot 56, has merged; wiring the dex path into cle CI and
the ecosystem CI is the follow-up asked for in review and is not in this
branch.

Validation: #783 (comment)

session: mega-corpus

[angr-bot]

Corpus decompilation diffs can be found at angr/dec-snapshots@master...angr/cle_783

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Validation record for head e5d9d25e58305a1adf9cb55e3af540e1a4d8026b against baseline d2ecea068794d20b1f14d90eecc1bc4bc4cfa431, cle master, with pysoot at angr/pysoot#56 head 2da6b740f5427b3ba2b2cbd772272ef730a53d05 (its own baseline 1b3bfa70888c7a100bf13dcb49b8e3da0dc66d99).

• Regression: python -m pytest -q tests/test_dex.py — on cle master carrying this head's test file, 2 failed, both raising CLECompatibilityError: Unable to find a loader backend for .../classes.dex. Perhaps try the 'blob' loader?; on this head, 2 passed
• Focused, and no more than focused. The two tests are test_dex_needs_an_android_sdk and test_dex_needs_an_api_version, and both are error-path tests: they assert CLEError for a missing load option and never reach the lifter, so they need neither pysoot#56 nor a JVM to pass. The suite therefore does not exercise a bare dex actually loading, which is the thing this change is for. That gap is real and is the first thing to fix on this branch
• Load, measured by hand rather than by the suite: classes.dex extracted from the existing android1.apk fixture (2858452 bytes) loads through cle.Loader with backend Dex and reports 1703 classes and 13718 methods, at android_api_version 25 and at 37 alike. Loading the whole APK through the Apk backend reports the same 1703 classes on cle master and on this head
• Dependency, and a merge-ordering constraint. Without Lift a bare DEX pysoot#56 this head does not raise ParameterError: format needs to be in ['jar', 'apk'], which is what an earlier version of this description claimed; correcting that, it raises TypeError: Lifter.__init__() got an unexpected keyword argument 'android_api_version' at cle/backends/java/soot.py:56, from the unconditional Lifter(...) call. So against released pysoot a plain APK stops loading as well. The same APK loads on cle master with pysoot master, and on this head with pysoot#56. This must not merge before pysoot#56 does
• Environment: the host had no JVM and no Android platform jars, so both were provisioned — OpenJDK 17.0.20 with JAVA_HOME pointing at its lib/openjdk, and the android-15, android-25 and android-37 platforms from the Sable android-platforms repository
• Side observation, in pysoot rather than cle: a bare-dex load at android_api_version=15 fails reproducibly with java.util.ConcurrentModificationException in _convert_class, pysoot/soot_manager.py:182. API 25 and 37 are clean. It matters here only because this change makes the API level user-supplied, so a caller can now pick a value that hits it

Caveats: these are focused runs. The full cle suite and pysoot#56's own suite were not rerun against this head.

[twizmwazin]

I have merged the pysoot PR, make sure that the dex loader is tested in the cle CI and in the ecosystem CI, which might mean making a PR to angr/ci-settings

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

cle.Loader on the classes.dex member of tests/java/android1.apk, before
and after this change. The command is the same on both sides:

import cle, zipfile
zipfile.ZipFile("binaries/tests/java/android1.apk").extract("classes.dex", path=d)
for opts in ({}, {"backend": "dex"}, {"backend": "dex", "android_sdk": "/nonexistent-sdk"}):
    cle.Loader(f"{d}/classes.dex", auto_load_libs=False, main_opts=opts)

Before — no backend claims the file, and naming one does not help:

angr/cle master

detected automatically     -> CLECompatibilityError: Unable to find a loader backend for /tmp/tmp.AMFU2khy2a/classes.dex.  Perhaps try the 'blob' loader?
backend named explicitly   -> CLEError: Invalid backend: dex
with android_sdk           -> CLEError: Invalid backend: dex

After — detection reaches the backend, and the loader now fails on the load
options Soot needs rather than on backend selection:

with this change

detected automatically     -> CLEError:
Path to Android SDK must be specified explicitly, e.g.
    loading_opts = { "android_sdk" : "/home/angr/android/platforms",
                     "android_api_version" : 35 }
    proj = angr.Project("/path/to/classes.dex", main_opts=loading_opts)
backend named explicitly   -> CLEError:
Path to Android SDK must be specified explicitly, e.g.
    loading_opts = { "android_sdk" : "/home/angr/android/platforms",
                     "android_api_version" : 35 }
    proj = angr.Project("/path/to/classes.dex", main_opts=loading_opts)
with android_sdk           -> CLEError:
A bare DEX file has no manifest declaring an Android API level, so android_api_version must be specified explicitly, e.g.
    loading_opts = { "android_sdk" : "/home/angr/android/platforms",
                     "android_api_version" : 35 }
    proj = angr.Project("/path/to/classes.dex", main_opts=loading_opts)

The third line is as far as this machine can take it: lifting the file needs an
Android SDK, which is not installed here, so what is shown is the loader
reaching Soot's requirements rather than a completed load.

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Coming back to this because the CI half of your comment was never answered, and the answer is that neither CI can currently reach the loader. Measured on cle master and ci-settings master today:

• tests/test_dex.py has two tests, test_dex_needs_an_android_sdk and test_dex_needs_an_api_version. Both assert pytest.raises(cle.CLEError). Neither one loads a DEX, so the backend added here is never exercised.
• pyproject.toml names pysoot nowhere — not in dependencies, not in optional-dependencies, not in any dependency-group. cle's own CI runs uv sync --directory cle --group testing, so pysoot is absent, and cle/backends/java/soot.py imports it under a try/except ImportError that leaves it None.
• In the ecosystem image, ci-image/conf/repo-list.txt reads angr/cle -> archinfo, pyvex, cle. pysoot appears only on the angr/angr line, so the image gives cle's job the same environment its own CI does.

So the work is three pieces, and the third is the one that actually costs something: add pysoot to cle's line in ci-settings/ci-image/conf/repo-list.txt; add it to cle's testing dependency group so a local run matches; and give CI an Android platform jar, because Dex.__init__ raises unless both android_sdk and android_api_version are supplied and pysoot.lifter.Lifter resolves the class library against that SDK. Until that third piece exists, any test that actually loads classes.dex can only skip.

I am picking this up. If you would rather the SDK not go into the shared image, say so and I will look at a fixture-based alternative instead.

session: sharpen

[zardus]

THIS MESSAGE WAS GENERATED BY AN AUTOMATED PROCESS

Correcting my own comment above. I said the third piece was "give CI an Android platform jar". That is wrong for most of cle's matrix, and it made the work sound bigger than it is.

angr/pysoot#56 already solved this and it is on pysoot master. tests/test_pysoot.py has _find_android_platforms(), which reads ANDROID_HOME and ANDROID_SDK_ROOT before falling back to ~/Android/Sdk — GitHub's hosted runners ship an SDK and set those. That change took pysoot's android tests from skipping to running on ubuntu-22.04, macos-14 and windows-2022 alike, 10 passed, 3 skipped becoming 12 passed, 1 skipped. It also picks the newest installed platform rather than the API 15 the APK declares, since no current SDK ships that one.

cle's test matrix is windows-2022 and macos-15, both hosted runners, so the SDK is already there. What is left is smaller than I said:

• add pysoot to cle's testing dependency group, so the backend is importable at all;
• add a test that loads classes.dex using the same ANDROID_HOME discovery pysoot uses, so it runs on that matrix instead of skipping;
• add pysoot to cle's line in ci-settings/ci-image/conf/repo-list.txt, which currently reads angr/cle -> archinfo, pyvex, cle.

The one genuine gap is the containerised Linux job: ci-image/Dockerfile installs no Android SDK, so that job skips whatever the other two run. Whether that is worth adding to the shared image is your call, and the honest answer is that the Windows and macOS legs would already be covering the loader.

session: sharpen