Skip to content

auto-updates fail with opendir(boot): Operation not permitted when /boot is a systemd automount that has idled out #2402

Open
#2411
Open
auto-updates fail with opendir(boot): Operation not permitted when /boot is a systemd automount that has idled out#2402
#2411
Labels
triagedThis issue appears to be valid

Description

@stavros-k
stavros-k
opened on Aug 22, 2026

⚠️ DISCLAIMER. Report is generated by Claude Code while it was troubleshooting one of my bench systems.
I've read the report and indeed seems to be true.
If you don't want to read an LLM report, please stop here. Feel free to close the report.

Summary

bootc-fetch-apply-updates.service does not declare RequiresMountsFor=/boot. On systems where /boot is a systemd-gpt-auto-generator automount with TimeoutIdleSec=120, /boot unmounts ~2 minutes after boot and the update timer then fires against an unmounted /boot, failing with:

error: Initializing storage: opendir(boot): Operation not permitted

The same failure hits bootc status and bootc upgrade run manually.

ostree hardened its own units against this exact hazard in ostreedev/ostree#2544; bootc's updater unit never got the equivalent.

Environment

  • bootc 1.16.7-1.fc44
  • ostree 2026.3-1.fc44
  • systemd 259.8-1.fc44
  • Base image quay.io/fedora/fedora-bootc:44
  • BIOS/GRUB boot: sda1 BIOS boot, sda2 vfat ESP (unused), sda3 xfs root
  • / is a composefs overlay; no /etc/fstab

Symptom

Aug 21 18:40:55 systemd[1]: Starting bootc-fetch-apply-updates.service - Apply bootc updates...
Aug 21 18:40:56 bootc[2194]: error: Initializing storage: opendir(boot): Operation not permitted
Aug 21 18:40:56 systemd[1]: bootc-fetch-apply-updates.service: Failed with result 'exit-code'.

Repeats on every timer firing. opendir(...) is libostree's glnx_throw_errno_prefix style, i.e. EPERM from openat(sysroot_fd, "boot").

Reproducer

Deterministic on an affected system:

$ sudo umount /boot          # simulate the automount idling out
$ sudo bootc status
error: Status: opendir(boot): Operation not permitted

Note bootc's access does not trigger the automount — /boot remains unmounted afterwards. (I suspect this is because bootc re-execs under unshare -m and autofs won't service the request from that private mount namespace, but I haven't confirmed that.)

Touch /boot by any other means and bootc works again:

$ ls /boot >/dev/null && sudo bootc status   # succeeds

Root cause

Two generators both claim /boot:

  • ostree-system-generator -> /run/systemd/generator/boot.mount (What=/sysroot/boot, Options=bind) — the real mount
  • systemd-gpt-auto-generator -> /run/systemd/generator.late/boot.automount (Where=/boot, TimeoutIdleSec=120, described as "EFI System Partition Automount") — targeting the otherwise-unused ESP

When the autofs idle timer fires it tears down ostree's /boot bind mount. The ostree units survive this because they declare the dependency:

$ systemctl show boot.mount -p RequiredBy
RequiredBy=ostree-finalize-staged.service ostree-boot-complete.service \
           ostree-finalize-staged-hold.service rpm-ostree-fix-shadow-mode.service local-fs.target

$ systemctl show bootc-fetch-apply-updates.service -p RequiresMountsFor
RequiresMountsFor=

Why this is newly visible

This system ran fine for two weeks on ostree 2026.2 and broke on upgrade to 2026.3, with bootc, systemd and the kernel at identical versions across both deployments. The behaviour change is ostreedev/ostree@62109dca ("prepare-root: create /run/systemd/volatile-root for composefs"), whose commit message states:

With composefs enabled both are overlayfs mounts with an anonymous st_dev … As a result systemd-gpt-auto-generator silently skips all partition discovery: the ESP is never automounted on /boot

Before 2026.3, gpt-auto could not resolve the composefs root's block device and skipped partition discovery entirely, so boot.automount was never generated. Journal evidence — Set up automount boot.automount appears only in deployments running 2026.3:

boot -3: ostree 2026.2  automount=0
boot -2: ostree 2026.3  automount=1   <- /boot unmounted 2m41s after boot
boot -1: ostree 2026.2  automount=0
boot  0: ostree 2026.3  automount=1

So ostree 2026.3 makes gpt-auto work as intended, which in turn exposes the missing dependency in bootc. I'd expect this to affect any composefs-based bootc system with a discoverable ESP as 2026.3 rolls out.

Proposed fix

Mirror what ostree does — add to bootc-fetch-apply-updates.service:

[Unit]
RequiresMountsFor=/boot

Verified as effective here via a drop-in: systemd then pulls in boot.mount and orders the service after it.

Two secondary suggestions:

  1. opendir(boot): Operation not permitted is opaque for what is really "/boot is not mounted". A dedicated check with a clearer message would save a lot of digging.
  2. Any other bootc command touching /boot (bootc status, bootc upgrade, bootc switch) has the same exposure when run from an idle shell, and can't be fixed by unit dependencies. Triggering the automount explicitly, or reporting a clear error, would help.

References

Activity

  1. ldobbelsteen commented on Aug 23, 2026

    @ldobbelsteen
    ldobbelsteen
    on Aug 23, 2026

    Just chipping in to say I'm running into the exact same issue under the same environment.

  2. cgwalters commented on Aug 24, 2026

    @cgwalters
    cgwalters
    on Aug 24, 2026
    Collaborator

    The behaviour change is ostreedev/ostree@62109dca ("prepare-root: create /run/systemd/volatile-root for composefs"), whose commit message states:

    This is at least the second regression from this change. The blast radius there is much larger than I realized; apologies.

    Two generators both claim /boot:

    ostree-system-generator -> /run/systemd/generator/boot.mount (What=/sysroot/boot, Options=bind) — the real mount
    systemd-gpt-auto-generator -> /run/systemd/generator.late/boot.automount (Where=/boot, TimeoutIdleSec=120, described as "EFI System Partition Automount") — targeting the otherwise-unused ESP

    Hmm yes, this sounds like a pretty bad bug.

    Actually, one thing intermixed with this is that really our base images should have /efi as a directory, which I think the systemd generator would prefer, and that would help fix this I believe.

  3. cgwalters commented on Aug 25, 2026

    @cgwalters
    cgwalters
    on Aug 25, 2026
    Collaborator

    @stavros-k @ldobbelsteen can you guys post the output of lsblk? I especially want to know in this setup if you actually have a separate /boot partition or not, and whether it matches the XBOOTLDR partuuid.

  4. ldobbelsteen commented on Aug 25, 2026

    @ldobbelsteen
    ldobbelsteen
    on Aug 25, 2026

    Sure, see my output below. Note that I'm running Fedora Bootc (version 44.20260824.0 to be exact) with minimal changes layered on top that I don't think should affect this (except that I explicitly disabled SELinux). My /boot seems to be on the same partition.

    [lukas@<REDACTED>]~% ostree --version
libostree:
 Version: '2026.3'
 Git: 90aa9450120da1782591ff773d97afae47a40702
 Features:
  - inode64
  - initial-var
  - libcurl
  - libsoup3
  - gpgme
  - composefs
  - ex-fsverity
  - libarchive
  - selinux
  - openssl
  - sign-ed25519
  - sign-spki
  - libmount
  - systemd
  - release
  - p2p
[lukas@<REDACTED>]~% bootc --version
bootc 1.16.9
[lukas@<REDACTED>]~% lsblk -o NAME,SIZE,FSTYPE,PARTTYPE,PARTLABEL,PARTUUID,MOUNTPOINTS
NAME          SIZE FSTYPE  PARTTYPE                             PARTLABEL  PARTUUID                             MOUNTPOINTS
sda           7.5G iso9660
├─sda1       1021M iso9660 0x0                                             7a57ac1c-01
└─sda2        7.8M vfat    0xef                                            7a57ac1c-02
sdb          21.5G exfat
nvme0n1     238.5G
├─nvme0n1p1     1M         21686148-6449-6e6f-744e-656564454649 BIOS-BOOT  6fab00f0-8af7-43e4-8581-5196fdd579ce
├─nvme0n1p2   512M vfat    c12a7328-f81f-11d2-ba4b-00a0c93ec93b EFI-SYSTEM 28b0578f-8440-4800-b54f-9c686194638d
└─nvme0n1p3   238G xfs     4f68bce3-e8cd-4db1-96e7-fbcaf984b709 root       fafe54a7-cb64-4e85-99ef-6c6d2ff10e44 /var/lib/containers/storage/overlay
                                                                                                                /var
                                                                                                                /sysroot/ostree/deploy/default/var
                                                                                                                /sysroot
                                                                                                                /etc
nvme1n1     931.5G
└─nvme1n1p1 931.5G ext4    0fc63daf-8483-4772-8e79-3d69d8477de4 primary    ceea663f-ed7b-407b-becd-02734cdc9602 /run/media/storage
[lukas@<REDACTED>]~% sudo bootc status
error: Status: opendir(boot): Operation not permitted
[lukas@<REDACTED>]~% ls /boot
boot  bootupd-state.json  efi  grub2  loader  loader.0  ostree
[lukas@<REDACTED>]~% lsblk -o NAME,SIZE,FSTYPE,PARTTYPE,PARTLABEL,PARTUUID,MOUNTPOINTS
NAME          SIZE FSTYPE  PARTTYPE                             PARTLABEL  PARTUUID                             MOUNTPOINTS
sda           7.5G iso9660
├─sda1       1021M iso9660 0x0                                             7a57ac1c-01
└─sda2        7.8M vfat    0xef                                            7a57ac1c-02
sdb          21.5G exfat
nvme0n1     238.5G
├─nvme0n1p1     1M         21686148-6449-6e6f-744e-656564454649 BIOS-BOOT  6fab00f0-8af7-43e4-8581-5196fdd579ce
├─nvme0n1p2   512M vfat    c12a7328-f81f-11d2-ba4b-00a0c93ec93b EFI-SYSTEM 28b0578f-8440-4800-b54f-9c686194638d
└─nvme0n1p3   238G xfs     4f68bce3-e8cd-4db1-96e7-fbcaf984b709 root       fafe54a7-cb64-4e85-99ef-6c6d2ff10e44 /boot
                                                                                                                /var/lib/containers/storage/overlay
                                                                                                                /var
                                                                                                                /sysroot/ostree/deploy/default/var
                                                                                                                /sysroot
                                                                                                                /etc
nvme1n1     931.5G
└─nvme1n1p1 931.5G ext4    0fc63daf-8483-4772-8e79-3d69d8477de4 primary    ceea663f-ed7b-407b-becd-02734cdc9602 /run/media/storage
[lukas@<REDACTED>]~% sudo bootc status
● Booted image: <REDACTED>:latest
        Digest: sha256:602897f94511fd5f594a911ed22cd6ada40f3d8fc477b9cb75f88f6ac897582c (amd64)
       Version: 44.20260824.0 (2026-08-24T12:34:12Z)

  Rollback image: <REDACTED>:latest
          Digest: sha256:7dd430ef96aceb47eaa591230a8cc89ac733fe16f6c7be5b3e5f47e7ca1f0e12 (amd64)
         Version: 44.20260823.0 (2026-08-24T08:16:17Z)
   UpdateVersion: 44.20260823.0 (2026-08-24T08:26:31Z)
    UpdateDigest: sha256:f517bcaa5b788c4d04d3aa39612d18f43b928945503697a35ba6e1b21339475c
  5. cgwalters
    added
    triagedThis issue appears to be valid
    on Aug 25, 2026

  6. Johan-Liebert1 commented on Aug 26, 2026

    @Johan-Liebert1
    Johan-Liebert1
    on Aug 26, 2026
    Member

    This seems to be the same issue I faced while working on #2323

  7. cgwalters
    added a commit that references this issue on Aug 26, 2026

    unshare: Propagate /boot automount to mount ns

    4c8aa51
  8. cgwalters
    closed this as completedin #2411on Aug 26, 2026

  9. Johan-Liebert1 commented on Aug 27, 2026

    @Johan-Liebert1
    Johan-Liebert1
    on Aug 27, 2026
    Member

    Reopening because #2411 does not solve the competing boot mount issue

  10. Johan-Liebert1
    reopened this on Aug 27, 2026

  11. ignitedPotato commented on Sep 13, 2026

    @ignitedPotato
    ignitedPotato
    on Sep 13, 2026

    I ran into this issue as well.
    Unfortunately after applying the fix suggested by @stavros-k now ostree-finalize-staged.service fails with:

    error: Remounting /boot read-write: Invalid argument

  12. clement-dufour commented on Sep 13, 2026

    @clement-dufour
    clement-dufour
    on Sep 13, 2026

    I ran into this issue as well.
    Unfortunately after applying the fix suggested by @stavros-k now ostree-finalize-staged.service fails with:

    error: Remounting /boot read-write: Invalid argument

    I have been observing the same behavior for a few weeks now. In the meantime, I update manually.

    I would be happy to help with logs or testing if needed. 😊

  13. ericcurtin
    added a commit that references this issue on Sep 13, 2026

    Install the smoke VM from an OCI layout and keep /boot mounted

    0356c6a

  14. cgwalters commented on Sep 14, 2026

    @cgwalters
    cgwalters
    on Sep 14, 2026
    Collaborator

    cc #2440 which is in this space

  15. cgwalters-bot
    added a commit that references this issue on Sep 30, 2026

    tmt: Add test switching to a zstd:chunked image

    316b48c
  16. cgwalters
    added a commit that references this issue on Sep 30, 2026

    tmt: Add test switching to a zstd:chunked image

    4c8765c
  17. cgwalters-bot
    added a commit that references this issue on Oct 7, 2026

    generator: Mask gpt-auto's boot.automount over our /boot bind mount

    8fe1681
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triagedThis issue appears to be valid

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions