[JanZachmann]

Summary

• write_partition compares the partition file with the image in 4 KiB blocks, only where one of them holds data, and writes only the blocks that differ, instead of running fallocate -d. It also writes a block that is data in the partition file but a hole in the image, e.g. a zero block of a file that e2cp or mcopy wrote into free space. fallocate, dd and sync are removed from the Docker image.
• The image is copied with copy_image, and a partition is read out of the image with the same sparse copy instead of dd conv=sparse: it finds the data with SEEK_DATA/SEEK_HOLE, keeps the holes and copies each data range in a loop. Decompressing an image keeps zero blocks as holes, reads a file made of several streams (e.g. from pbzip2), and fails on a truncated or corrupt stream. A compressed image is decompressed straight into the temp dir, without a copy first. Images named .gz and .bz2 are decompressed too; before, the output file had the name of the source, so creating it emptied the source. Compressing fails when the end of the stream cannot be written.
• omnect-cli writes the bmap of -b itself: it maps the ranges of the input bmap, or the whole image without one, plus every block the image file stores as data, with a sha256 per range. No zeros are written into the image, so it stays sparse, and bmaptool is no longer needed at runtime: -b now works in the Docker image, and bmap-tools is removed from the Debian dependencies and the Docker image. The bmap is copied next to the image after the image was written back.
• The input bmap is the file given with the new --input-bmap argument (requires -b), else <image>.bmap or the bmap of the uncompressed image next to it (e.g. x.wic.bmap for x.wic.xz, as the Yocto build creates it). It is only read with -b. The command fails if the input bmap is for an image of another size, if a range does not match its sha256 checksum, or if the bmap has no range checksums. The new --no-input-bmap (requires -b) skips it, and the bmap then maps the whole image. The input bmap of an uncompressed image is checked before the image is copied.
• The created bmap is named <output image>.bmap and written next to the output image, e.g. x.wic.bmap for x.wic and x.wic.xz.bmap for x.wic.xz, where bmaptool finds it. So with -p it no longer replaces the bmap of the input image, which stays unchanged.
• The output image is the input image, for a compressed image without the compression extension, plus the -p extension, as before. The new -o/--output-image writes it to another path, and the input image stays unchanged. The -o name must have the extension of the -p compression, or no compression extension without -p. A command now fails instead of overwriting an existing file that is not the input image (a path given with -o is overwritten), replacing the bmap of an image that stays (e.g. x.wic.bmap for -i x.wic.xz without -p, also with --input-bmap or --no-input-bmap), or writing an image without -b while <output image>.bmap is next to it, or for a compressed output image the bmap of the uncompressed image, which bmaptool also uses. That bmap would no longer match. So for several commands on one image, use -b on each.
• The output image and the bmap are written to a new file in the same directory and renamed over the old one, so a failed write, e.g. on a full disk, no longer destroys an image that is changed in place. A symlink is followed, and the old file keeps its permissions. The directory needs write access and space for one more image, and the new file belongs to the user who runs omnect-cli.
• copy-from-image no longer writes the image back. It only reads, so an error can no longer truncate the source, and for x.wic.xz it no longer leaves x.wic next to it. It reads an uncompressed image directly, without a temp copy.
• The partition files are written to their own temp dir. Before, they were named <num>.img next to the image, so an image named e.g. 1.img was overwritten by its partition 1. The temp dirs follow TMPDIR.
• Partitions of a GPT with 4096-byte sectors are read at the right offset; before, the offsets were always counted in 512-byte sectors.
• Known limitations, tracked as follow-up tasks: the created bmap maps every block that the image file stores as data, so for an image that is not sparse it maps the whole image, also with an input bmap. For a partition that does not start on a 4 KiB boundary, a block at the edge of each data run is written back, so the image is less sparse. In both cases the image content is correct.
• Breaking: --generate-bmap-file is renamed to --output-bmap, so the two options read as a pair. The short option -b stays. The name conflicts above fail where they overwrote files before. -p bzip2 writes <image>.bz2 instead of <image>.bzip2, which bmaptool does not read.
• roxmltree is added to read the bmap, and rustix for SEEK_DATA/SEEK_HOLE; libfs is removed. The test bmap testfiles/image.wic.bmap is now in git (*.bmap is in .gitignore), with checksums that match the test images. The version is bumped to 0.29.0, and ctutils to 0.4.5 for RUSTSEC-2026-0385.

Reason

A sparse image, e.g. one that xz decompressed into a file, or one after fallocate -d, has holes wherever the image is zero, and a bmap skips them. Some of these zeros must still be written: a bmap-based flash otherwise keeps what an earlier installation left there. An old ext4 journal can be replayed after a power cut and apply an old transaction to the new filesystem; old GPT entries break the entry array CRC; old FAT sectors and directory entries show up as used clusters or stale files. On an rpi4-64 image, mapping only the ext4 journals still drops 188 blocks of the Yocto bmap: GPT primary and backup entries, 69 blocks of boot, and 16–24 blocks on every ext4 partition.

The Yocto bmap maps all of these ranges, because the build image holds real data there. Keeping its ranges mapped covers every format without knowing about file systems. From a sparse image alone, omnect-cli cannot tell such zero blocks from free space, so without an input bmap it maps the whole image. bmaptool copy reads a mapped range with plain reads, so a hole in such a range is written as zeros: the zeros only have to be in the bmap, not allocated in the image file. This replaces the image-side approach discussed in omnect/meta-omnect#701.

With default-features = false, libfs made byte copies, so every image came back fully allocated, and the decompress wrote zero blocks as data. fallocate -d hid both. The sparse copy uses SEEK_DATA/SEEK_HOLE directly: libfs::next_sparse_segments seeks both files to the same offset, so it cannot read a partition out of the image, and libfs::copy_file copies each data segment with a single copy_file_range call, which moves at most 2 GiB, and ignores a short copy.

All omnect partitions have a fixed size, so builds of one machine have the same image size, and the bmap of another build (e.g. of an older download next to a new image) passes a size check. The sha256 of each range tells the builds apart; bmaptool copy runs the same check on flash.

omnect-cli derives every output name from the input name and finds the input bmap by name, so names can collide. omnect-cli runs in CI scripts, where nobody reads a warning, so each collision fails with an error that names the option to resolve it. A counter in the output name would not work: a script and the later bmaptool copy need a fixed name.

Writing only changed blocks also fixes a gap of the old dd conv=notrunc,sparse: a block that changed to all zeros was skipped and kept its old content; now it is written because it differs. In the same way, dd conv=sparse turned every zero block of a partition into a hole, so a zero block of a copied file that landed in free space stayed a hole in the image, and a bmap flash kept the old content there.

[mlilien]

A few more points on 3da5a4d. The inline comments are the ones I would look at. The rest are optional nits:

• README, Output image: /tmp also needs space for the uncompressed image plus partition copies, as it did before this PR. That is worth a sentence, since /tmp is often a small tmpfs and an xz image can expand to several GB.
• lib.rs:86: for an uncompressed input with an input bmap, the mapped ranges are read once by verify_checksums and again by the copy to TMPDIR. Checking the checksums during the copy would save one pass. Low priority, because the pass in write_bmap is needed anyway (it hashes the modified image).
• bmap.rs:118 / compression.rs:121: bmap_path and compressed_path both append .ext to the OsString. A shared append_extension(path, ext) helper would do.
• Bmap::read: the root element, version and BmapFileChecksum are not checked. The range sha256 check still rejects a wrong bmap, so this is mainly about refusing a future major version (3.x), as bmaptool does.

[JanZachmann]

On the nits in the review of 3da5a4d:

• README: 19bb6be describes the space TMPDIR needs: a copy of each partition the command reads or changes, and the uncompressed image for a command that changes an image or reads a compressed one.
• append_extension: 9126d55 uses std Path::with_added_extension in bmap_path and compressed_path.
• Bmap::read: 8f08735 checks the root element and refuses a missing version or a major version above 2, as bmaptool does.
• lib.rs:86, checking the checksums during the copy: not changed. f0ead21 checks the input bmap on the source before the copy, so a wrong bmap fails before a multi-GB copy. Checking during the copy would move that failure after it. Also, the copy uses copy_file_range, which does not read the data through user space. Hashing during the copy would need that.

[mlilien]

Thanks for the fixes. The three threads from the last round are resolved; I checked each against 6e9c9e1. Clippy (-D warnings) is clean, and unit and integration tests pass locally.

A few new points inline. One is outside the diff, so here:

src/lib.rs create_image_cert (around line 217): it writes the device certificate and private key under fixed names (device_cert_path.pem, device_key_path.key.pem, the edge_ca_* names) next to the input image and never deletes them. This is older than this PR, but -o makes "one base image, many device outputs" the documented workflow, so it matters more now:

• Two jobs running identity set-device-certificate -i base.wic.xz -o devA.wic and ... -o devB.wic in the same workspace at the same time: B overwrites the files while A is still decompressing, so devA.wic can get B's certificate and key.
• With a read-only input directory (-i /ro/base.wic.xz -o /out/dev.wic), the command fails at fs::write.
• A device private key is left next to the base image, even when check_conflicts rejects the run.

Writing the cert/key into a TempDir would fix all three. Fine as a follow-up if you prefer.

[mlilien]

copy_into_dest truncates the destination before copying. If sparse::copy_to_empty_file fails partway (I/O error, or no space because the host is btrfs with snapshots and truncating frees nothing), the image on the host is left truncated, and the finished image in TMPDIR is removed when Staged drops. Both old and new image are gone.

I know the copy isn't atomic, same as on main. But keeping the staged file on error (TempDir::keep() and logging its path) would be cheap and saves the result.

[mlilien]

OTHER_IMAGE_BMAP has no version attribute and no ChecksumType. Since 8f08735, check_input_bmap_of_another_image and check_no_input_bmap fail on Bmap::read: no valid version and never reach the image-size check they are meant to test. With version="2.0" added it fails on no sha256 range checksums instead.

The tests still pass because they only match cannot use input bmap / --no-input-bmap. If check_layout broke, nothing would notice. Could the fixture be a valid bmap (version, ChecksumType, range checksums) with only the image size differing?

[mlilien]

The staged image now lives in a hidden .omnect-cli-* dir next to the output image, and TempDir only removes it on drop. On Ctrl-C or SIGTERM (e.g. a cancelled CI job) during the long compress/copy/fsync, no destructors run, and a multi-GB .omnect-cli-XXXX/image.wic.xz stays in the user's image directory. Before this PR, leftovers ended up in /tmp, which gets cleaned on reboot or is tmpfs.

Maybe a signal handler that removes the staging dir, or at least a note in the README.

[mlilien]

This asserts that / is a mount root, which depends on the environment. In a plain chroot (pbuilder/sbuild style), / isn't a mount root, and kernels before 5.8 don't report STATX_ATTR_MOUNT_ROOT. Then is_mount_root returns false and the test fails, although the code works (it falls back to the EBUSY/EXDEV path). Maybe skip the assertion when STATX_ATTR_MOUNT_ROOT isn't in stx_attributes_mask.

[mlilien]

-p gzip now writes <image>.gz instead of <image>.gzip (the old extension() returned "gzip"). The Breaking section only mentions the bzip2 rename. A pipeline that uploads or flashes x.wic.gzip after -p gzip won't find the file after updating. Please add it to the breaking notes.

[mlilien]

Staged::write always calls sync_all on the staged file, also when the destination is a mount point and the file is only a TMPDIR copy that gets copied into the destination and deleted right after. With disk-backed TMPDIR, an in-place command on a bind-mounted 8 GiB image forces a full writeback of a throwaway file, and copy_into_dest then syncs the real destination anyway. The sync could be skipped on the copy path.

[mlilien]

copy_into_dest repeats copy_image from lib.rs (open source, get length, sparse::copy_to_empty_file), and the uncompressed branch of temp_image (lib.rs:154) repeats sparse::copy_to_new_file. That makes three copies of the same sequence. Moving copy_image into file::sparse and calling it from here and from temp_image would keep later fixes (error context, permissions) in one place.

[mlilien]

bmap_path_of duplicates the public omnect_cli::file::bmap::bmap_path, and four tests (lines 964, 988, 1035, 1092) still build the path inline with format!("{}.bmap", ...). If the naming rule changes, the tests keep the old one. Using bmap_path everywhere avoids that.

[mlilien]

The module list adds bmap.rs and sparse.rs but not the new src/file/output.rs (output naming, conflict checks, staging, commit).