nvx-rs: attribute a clone's failures, record them, and reattach it

nvx-rs/README.md

@@ -536,6 +536,10 @@ explicit handle list.
 event until a deadline, and takes a bounded forced step through the VMM's
 verified identity only on request.
 
+`Vmm::stop_unless_exited` stops the VM the same way only if its VMM still runs
+as the stop begins, and otherwise stops nothing and keeps the runtime record of
+its launch, so that `reattach` finds that the VM exited without its holder.
+
 `Vmm::terminate` takes that forced step alone, within the caller's bound, for a
 VM whose guest the caller must not ask anything, such as one of another runtime
 identity.
@@ -678,11 +682,12 @@ store's names is refused.
 
 `ClonePlan::launch` then launches the clone as the launch that its lease
 reserves, once it has checked that the sandbox's clone binding records the
-lease, and returns a `Restoring`: it writes the clone's activation record in its
-canonical encoding, `clone-activation.bin`, from which a caller resends it
-unchanged; it listens for the VMM's restore-ready event, at the socket
-`restore.sock` on Linux, and at a named pipe of a random name on Windows; and it
-runs `supervisor::launch` with OpenVMM's restore of the clone's devices,
+lease, and that the sandbox records no failure of the clone, which fails once
+under its lease, and returns a `Restoring`: it writes the clone's activation
+record in its canonical encoding, `clone-activation.bin`, from which a caller
+resends it unchanged; it listens for the VMM's restore-ready event, at the
+socket `restore.sock` on Linux, and at a named pipe of a random name on Windows;
+and it runs `supervisor::launch` with OpenVMM's restore of the clone's devices,
 `--restore-snapshot` with the generation's snapshot in place, which no clone
 copies or changes, so that clones of one generation restore it at once,
 `--restore-memory` with the memory target, and `--restore-ready-path`, under the
@@ -728,6 +733,77 @@ before the commit leaves the launch's journal, whose recovery ends the VMM,
 while the lease lasts until the release, and a crash after it leaves a VM that
 reattaches, and keeps its lease while it runs.
 
+A failure of a clone is a `CloneFailure`, whose `Evidence` attributes it to an
+`Outcome`, at a `Stage`, once the clone got as far as its `Progress`: the
+request, for a denied authorization, a stale binding, invalid targets, format,
+settings, Activate, or mappings, or an image, a scratch, or a root that the
+template cannot serve, and for a supplied file of the wrong identity, length, or
+geometry, which the host's staging takes instead once the caller proves it,
+`CloneFailure::with_staging_proven`; the host, for an endpoint that cannot be
+bound, resource exhaustion, a host that cannot restore the template, another
+runtime, or a failure of its own operations; unknown, for a lost status, a
+timeout, a reset, a forced kill, or a broken protocol; the template, for a
+malformed generation, or a defect of its capture or repair that a diagnosis
+attributes to it, the only outcome that quarantines the generation, through its
+tombstone, `CloneFailure::quarantine`; and the repair stage, for a status of the
+clone's repair alone.
+
+OpenVMM's outcome report gives the status with which the guest ended the clone's
+VM, which `status_evidence` attributes, and which prevails over what the host or
+the caller saw, but not over evidence of the template, which shows the
+provenance that a status alone leaves to establish: a status of the repair at
+any point, and one of the activation before the start was sent.
+
+A VMM that exits before its restore-ready event, as OpenVMM does once it cannot
+restore the snapshot, after it closed the event's endpoint, which the sequence
+then gives ten seconds to exit, leaves the line with which OpenVMM reported its
+fatal error in its log, which the failure's message keeps, and whose time ABI
+code, if it says that the host cannot present the template's CPU profile, TSC,
+or LAPIC timer, attributes the failure to the host.
+
+From the launch on, a failure ends the clone's VMM through its verified
+identity, and the sandbox keeps the clone's progress log, `clone-progress.log`,
+which records each step from the spawn to the commit, after a first line that
+names the launch and the boot of the host, a few lines that it appends without a
+sync, and the record of its failure, `clone-failure.json`, JSON in format
+`FAILURE_FORMAT`, which it writes before the teardown, with the teardown
+pending, so that a crash of the caller during the teardown leaves it for a
+reattach to finish, and again once the teardown ended, with the stage, the
+evidence and the status, the original outcome, the transaction, the generation,
+the progress, and the teardown: no VM, an exit or a termination that the host
+verified, or an unresolved one, which keeps the launch's journal or runtime
+record, and the lease; a release removes both, and a sandbox that boots cold
+writes neither. A failure that cannot be recorded before its teardown keeps the
+launch's journal or runtime record, and the lease, so that a reattach still
+finds that the clone failed.
+
+A step or a check of the caller's own that fails once the clone is ready, or
+once its sandbox started, fails the clone with the caller's evidence through
+`Activated::fail` or `Started::fail`, which end the launch and record the
+failure as the sequence's own failures do.
+
+`reattach_clone` takes a clone over after the caller of its launch restarted: it
+reattaches by the identity of its VMM's process, checks a committed clone in a
+new session, in which the clone answers the byte-equal Activate of
+`clone-activation.bin` with the Ready that it answered once, and ends a clone
+that another runtime identity launched, or that fails the check, through its
+verified identity; it records the failure of a launch that a crash interrupted,
+but not the journal that a commit may leave beside its runtime record,
+classifies a VM that exited by its outcome report, which is the clone's VM's
+once its launch began, whether or not a crash left a record of its spawn,
+records a failure before it ends a VMM or clears the runtime record of one that
+exited, and fails, leaving them as they were, if it cannot, finishes a failure
+that a crash left pending as its teardown would have recorded it, keeping its
+transaction, resolves the teardown of an earlier failure with what the VMM left
+as it ended, its outcome report and OpenVMM's fatal error, attributes a status
+of the guest's as one that a workload may have given once the clone's progress
+log is lost, damaged, or was written during another boot of the host, whose
+crash may have lost its last lines, and reads nothing of the template store, so
+that a tombstone or a retirement of the generation leaves the clone its lease.
+
+`CloneRecords::read` reads what the sandbox keeps of a clone, for the operations
+that list, inspect, or remove clones.
+
 `Store::tombstone` quarantines a generation, which takes no new lease while its
 leases keep working, and `Store::retire` deletes it once its leases have
 drained, crash-safely, removing the records of leases that no clone binding

nvx-rs/src/host/platform/linux.rs

@@ -784,6 +784,17 @@ impl Directory {
     pub(crate) fn sync(&self) -> io::Result<()> {
         self.0.sync_all()
     }
+
+    /// Returns why the directory cannot hold a sandbox store, as [`check_store_dir`] returns it for
+    /// a path, if it cannot: unless it belongs to the current user with mode 0700.
+    pub(crate) fn store_dir_problem(&self) -> io::Result<Option<String>> {
+        let metadata = self.0.metadata()?;
+        Ok(store_dir_problem(
+            metadata.uid(),
+            metadata.mode(),
+            sys::effective_ids().0,
+        ))
+    }
 }
 
 /// Claims a launch's fresh log, which `file` holds open.
@@ -868,7 +879,8 @@ pub(crate) struct ReadyListener {
 
 /// Listens at the Unix socket `socket_path`, in place of a socket that an earlier launch left
 /// there, as the restore-ready endpoint of a clone's launch, and returns the endpoint, as
-/// OpenVMM's `--restore-ready-path` takes it, and its listener.
+/// OpenVMM's `--restore-ready-path` takes it, and its listener. The socket has mode 0600,
+/// whatever the umask, as [`set_ready_socket_mode`] gives it.
 pub(crate) fn listen_ready(socket_path: &Path) -> io::Result<(String, ReadyListener)> {
     let endpoint = socket_path
         .to_str()
@@ -879,6 +891,10 @@ pub(crate) fn listen_ready(socket_path: &Path) -> io::Result<(String, ReadyListe
         _ => {}
     }
     let socket = UnixListener::bind(socket_path)?;
+    // The umask may clear bits of the mode that the bind gives the socket, even its owner's
+    // permission to write it, without which a VMM that is not root cannot connect: the mode is
+    // set again before any VMM may connect.
+    set_ready_socket_mode(socket_path)?;
     // The host polls the socket before it accepts, so an accept never blocks.
     socket.set_nonblocking(true)?;
     let listener = ReadyListener {
@@ -965,6 +981,35 @@ impl ReadyListener {
     }
 }
 
+/// Gives the restore-ready socket at `socket_path`, which the caller bound, the mode 0600, whatever
+/// the umask: its owner, the user of the clone's VMM, may connect to it, which takes the
+/// permission to write it, and no one else may.
+///
+/// The socket is opened only to locate it, following no link at its name, and refused unless it
+/// is a socket.
+fn set_ready_socket_mode(socket_path: &Path) -> io::Result<()> {
+    use std::os::unix::fs::FileTypeExt;
+
+    use nix::fcntl::OFlag;
+
+    let socket = OpenOptions::new()
+        .read(true)
+        .custom_flags((OFlag::O_PATH | OFlag::O_NOFOLLOW).bits())
+        .open(socket_path)?;
+    if !socket.metadata()?.file_type().is_socket() {
+        return Err(io::Error::other(format!(
+            "{} is not the restore-ready socket that the launch bound",
+            socket_path.display()
+        )));
+    }
+    // A descriptor that only locates the socket cannot change its mode itself, but its link in
+    // /proc names exactly the socket that it opened.
+    fs::set_permissions(
+        Path::new("/proc/self/fd").join(socket.as_raw_fd().to_string()),
+        Permissions::from_mode(0o600),
+    )
+}
+
 /// Connects to a control endpoint served by the process `expected_pid`.
 ///
 /// The connection attempt never blocks: a listener whose backlog is full yields
@@ -1110,4 +1155,65 @@ mod tests {
         assert!(elapsed < Duration::from_secs(1), "{elapsed:?}");
         assert!(process.wait(Instant::now()).unwrap().unwrap());
     }
+
+    #[test]
+    fn a_restore_ready_socket_lets_its_owner_connect_whatever_the_umask() {
+        let directory = tempfile::tempdir().unwrap();
+        let path = directory.path().join("ready.sock");
+        // A umask that clears even the owner's permission to write what it creates, in a thread
+        // of the test's own, whose umask no other thread shares.
+        let (endpoint, _listener) = thread::scope(|scope| {
+            scope
+                .spawn(|| {
+                    nix::sched::unshare(nix::sched::CloneFlags::CLONE_FS).unwrap();
+                    nix::sys::stat::umask(nix::sys::stat::Mode::from_bits_truncate(0o277));
+                    listen_ready(&path)
+                })
+                .join()
+                .unwrap()
+        })
+        .unwrap();
+        let mode = fs::symlink_metadata(&path).unwrap().permissions().mode();
+        assert_eq!(mode & PERMISSION_BITS, 0o600, "{mode:o}");
+        UnixStream::connect(&endpoint).unwrap();
+    }
+
+    #[test]
+    fn a_restore_ready_client_that_closed_before_its_connection_was_accepted_is_read_to_its_end() {
+        let pid = std::process::id();
+        let receive_all = |listener: &mut ReadyListener| {
+            let mut received = Vec::new();
+            let deadline = Instant::now() + Duration::from_secs(10);
+            while listener
+                .receive(pid, &mut received, 64, Duration::from_millis(50))
+                .unwrap()
+            {
+                assert!(Instant::now() < deadline, "the endpoint did not close");
+            }
+            received
+        };
+        let directory = tempfile::tempdir().unwrap();
+        let path = directory.path().join("ready.sock");
+        // A client that sent nothing, as a VMM that fails as it starts, and one that sent its
+        // event, each closed before the listener accepted their connection: what they sent
+        // remains to read, and then the end of the connection.
+        for sent in [&b""[..], b"the event\n"] {
+            let (endpoint, mut listener) = listen_ready(&path).unwrap();
+            let mut client = UnixStream::connect(&endpoint).unwrap();
+            std::io::Write::write_all(&mut client, sent).unwrap();
+            drop(client);
+            assert_eq!(receive_all(&mut listener), sent);
+        }
+        // A client that closes once the listener accepted its connection.
+        let (endpoint, mut listener) = listen_ready(&path).unwrap();
+        let client = UnixStream::connect(&endpoint).unwrap();
+        let mut received = Vec::new();
+        assert!(
+            listener
+                .receive(pid, &mut received, 64, Duration::from_millis(50))
+                .unwrap()
+        );
+        drop(client);
+        assert_eq!(receive_all(&mut listener), b"");
+    }
 }

nvx-rs/src/host/platform/windows.rs

@@ -2137,7 +2137,7 @@ pub(crate) fn interrupted() -> bool {
 
 #[cfg(test)]
 mod tests {
-    use std::io::Read;
+    use std::io::{Read, Write};
 
     use super::*;
 
@@ -2165,6 +2165,43 @@ mod tests {
         end_process(false, 7, || panic!("terminated"), || Ok(true)).unwrap();
     }
 
+    #[test]
+    fn a_restore_ready_client_that_closed_before_its_connection_was_accepted_is_read_to_its_end() {
+        let pid = std::process::id();
+        let receive_all = |listener: &mut ReadyListener| {
+            let mut received = Vec::new();
+            let deadline = Instant::now() + Duration::from_secs(10);
+            while listener
+                .receive(pid, &mut received, 64, Duration::from_millis(50))
+                .unwrap()
+            {
+                assert!(Instant::now() < deadline, "the endpoint did not close");
+            }
+            received
+        };
+        // A client that sent nothing, as a VMM that fails as it starts, and one that sent its
+        // event, each closed before the listener accepted their connection: what they sent
+        // remains to read, and then the end of the pipe.
+        for sent in [&b""[..], b"the event\n"] {
+            let (endpoint, mut listener) = listen_ready(Path::new("unused")).unwrap();
+            let mut client = fs::OpenOptions::new().write(true).open(&endpoint).unwrap();
+            client.write_all(sent).unwrap();
+            drop(client);
+            assert_eq!(receive_all(&mut listener), sent);
+        }
+        // A client that closes once the listener accepted its connection.
+        let (endpoint, mut listener) = listen_ready(Path::new("unused")).unwrap();
+        let client = fs::OpenOptions::new().write(true).open(&endpoint).unwrap();
+        let mut received = Vec::new();
+        assert!(
+            listener
+                .receive(pid, &mut received, 64, Duration::from_millis(50))
+                .unwrap()
+        );
+        drop(client);
+        assert_eq!(receive_all(&mut listener), b"");
+    }
+
     #[test]
     fn windows_refuses_to_terminate_a_process_that_exited_with_access_denied() {
         let mut child = std::process::Command::new("ping")

nvx-rs/src/host/sandbox/distro.rs

@@ -337,6 +337,12 @@ impl DistroBinding {
         self.policy
     }
 
+    /// Returns the binding's version, which a [check](Self::check) requires to be
+    /// [`BINDING_VERSION`].
+    pub(crate) fn version(&self) -> u32 {
+        self.version
+    }
+
     /// Returns the binding of `layer`, with the seals of its image and manifest and `policy`.
     fn new(
         layer: &SandboxLayer,

nvx-rs/src/host/sandbox/scratch.rs

@@ -257,6 +257,12 @@ impl ScratchBinding {
         self.length
     }
 
+    /// Returns the binding's version, which a [check](Self::check) requires to be
+    /// [`SCRATCH_BINDING_VERSION`].
+    pub(crate) fn version(&self) -> u32 {
+        self.version
+    }
+
     /// Returns the binding of the image that `seal` seals.
     fn of(seal: &FileSeal) -> Self {
         Self {

nvx-rs/src/host/store/lifecycle/tests.rs

@@ -11,8 +11,8 @@ use std::time::Duration;
 
 use super::*;
 use crate::host::store::{
-    CAPABILITY_NAME, CLONE_ACTIVATION_NAME, CLONE_SCRATCH_NAME, CONSOLE_NAME, LOG_NAME,
-    OUTCOME_NAME, RESTORE_READY_NAME, SOCKET_NAME,
+    CAPABILITY_NAME, CLONE_ACTIVATION_NAME, CLONE_FAILURE_NAME, CLONE_PROGRESS_NAME,
+    CLONE_SCRATCH_NAME, CONSOLE_NAME, LOG_NAME, OUTCOME_NAME, RESTORE_READY_NAME, SOCKET_NAME,
 };
 use crate::host::test_support::released;
 
@@ -326,6 +326,8 @@ fn a_removal_removes_the_store_the_callers_files_and_the_directory() {
         CLONE_BINDING_NAME,
         CLONE_SCRATCH_NAME,
         CLONE_ACTIVATION_NAME,
+        CLONE_PROGRESS_NAME,
+        CLONE_FAILURE_NAME,
         RESTORE_READY_NAME,
         BOOT,
         ".runtime.json.12345.tmp",
@@ -417,6 +419,8 @@ fn write_clone_files(dir: &Path) {
         CLONE_BINDING_NAME,
         CLONE_SCRATCH_NAME,
         CLONE_ACTIVATION_NAME,
+        CLONE_PROGRESS_NAME,
+        CLONE_FAILURE_NAME,
         RESTORE_READY_NAME,
     ] {
         fs::write(dir.join(name), b"{}").unwrap();
@@ -536,6 +540,8 @@ fn linked_entries(more: &[&str]) -> Vec<String> {
     let mut names = vec![
         CLONE_ACTIVATION_NAME,
         CLONE_BINDING_NAME,
+        CLONE_FAILURE_NAME,
+        CLONE_PROGRESS_NAME,
         CLONE_SCRATCH_NAME,
         CONFIG,
         LOCK_NAME,