sysbox-fs zombie reaper can steal fusermount3 children during concurrent container creation

[stuartwdouglas]

We have been seeing intermittent OCI container creation failures and my agent thinks the following is the cause:

Summary

Under concurrent container creation/deletion, Sysbox 0.7.1 intermittently fails container pre-registration because the sysbox-fs zombie reaper collects a fusermount3 child before the FUSE library's exec.Cmd.Wait() does.

Environment

• Sysbox: 0.7.1
• sysbox-fs commit: c3d2ebc65102e32e74e383675f03b45556326888
• Architecture: arm64
• Kernel: 6.12
• Runtime: containerd 2.2.7
• FUSE helper: fusermount3

Observed failure

The following messages occurred consecutively in the same sysbox-fs process:

reaper: reaped pid 328923
fusermount: waitid: no child processes
Container pre-registration error: unable to initialize fuseServer for container 8f86022534b6: FuseServer InitWait error

Container creation subsequently succeeded on retry.

Root cause

The nsenter zombie reaper invokes process-wide wait4(-1):

https://github.com/nestybox/sysbox-fs/blob/c3d2ebc65102e32e74e383675f03b45556326888/nsenter/reaper.go#L88-L97

The pinned FUSE library independently launches fusermount3 and calls cmd.Wait():

https://github.com/nestybox/fuse/blob/5ce319439091b08f33b149cb2482039f1b0a24fe/mount_linux.go#L74-L107

FUSE startup does not participate in the reaper's synchronization. The global reaper can therefore collect the fusermount3 child first, causing cmd.Wait() to return ECHILD.

The FUSE library has an attempted no child processes workaround, but it is applied to cmd.Start() rather than cmd.Wait(). It also checks "wait: no child processes", while the observed error is "waitid: no child processes".

Additional state leak

ContainerPreRegister inserts the container into idTable and netns tracking before starting FUSE:

https://github.com/nestybox/sysbox-fs/blob/c3d2ebc65102e32e74e383675f03b45556326888/state/containerDB.go#L132-L176

When CreateFuseServer fails, that state and the created mountpoint directory are not rolled back:

https://github.com/nestybox/sysbox-fs/blob/c3d2ebc65102e32e74e383675f03b45556326888/state/containerDB.go#L176-L186

The failed container's empty /var/lib/sysboxfs/<container-id> directory remained after the error.

Related concurrency issue

ContainerPreRegister holds the global container-state write lock throughout synchronous FUSE startup. During a burst of registrations, later requests can remain queued long enough for runtime/CNI cleanup to remove their network namespace:

Container pre-registration error: <id> has invalid net-ns:
Error getting netns inode: stat /var/run/netns/cni-...: no such file or directory

The gRPC cancellation context is also not propagated into the callback, so pre-registration may continue after its caller has abandoned container creation.

Suggested fixes

1. Prevent the global zombie reaper from collecting children owned by exec.Cmd, including fusermount3.
2. Roll back idTable, netns tracking, mountpoint, and partial FUSE state when pre-registration fails.
3. Avoid holding the global container DB lock during FUSE process startup.
4. Pin or inspect the netns before waiting for the global lock.
5. Propagate gRPC cancellation through pre-registration.

This is related to #981. PR nestybox/sysbox-fs#113 prevents the entire daemon from deadlocking when FUSE startup fails, but it does not prevent the reaper from stealing the FUSE helper or clean up failed pre-registration state.