From bbeb7f596d08dc69ac0781a67fffdcddad73da97 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 13 Aug 2026 13:09:36 +0200 Subject: [PATCH 01/28] fuse: retry the read in fuse_write_begin() on DLM contention fuse_do_readpage() turns the DLM's "granting this read would deadlock" answer into AOP_TRUNCATED_PAGE: the caller is supposed to drop the page and come back, which lets the conflicting holder -- typically a page invalidation on this very node -- take the folio lock and make progress. ->read_folio() callers in filemap.c implement that contract. ->write_begin() callers do not. generic_perform_write() only breaks out of the copy loop on a negative return: status = a_ops->write_begin(file, mapping, pos, bytes, &folio, &fsdata); if (unlikely(status < 0)) break; offset = offset_in_folio(folio, pos); AOP_TRUNCATED_PAGE is 0x80001, so a partial-folio buffered write that loses the race walks straight into offset_in_folio() on the uninitialised folio that fuse_write_begin() never assigned. Handle the retry where the contract is understood. The cleanup path has already dropped the folio lock at that point, so looping back is exactly the recovery AOP_TRUNCATED_PAGE asks for, and the fresh READ each pass sends keeps the loop paced by the server rather than spinning on it. Reset err inside the loop as well: it doubled as the -ENOMEM the __filemap_get_folio() failure path returns, and would otherwise carry AOP_TRUNCATED_PAGE back out through the same hole on a retry. Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index cf1f528914c208..2560afa1cb06c6 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -2327,10 +2327,12 @@ static int fuse_write_begin(struct file *file, struct address_space *mapping, struct fuse_conn *fc = get_fuse_conn(file_inode(file)); struct folio *folio; loff_t fsize; - int err = -ENOMEM; + int err; WARN_ON(!fc->writeback_cache); +retry: + err = -ENOMEM; folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) @@ -2361,6 +2363,18 @@ static int fuse_write_begin(struct file *file, struct address_space *mapping, cleanup: folio_unlock(folio); folio_put(folio); + /* + * The DLM refused the read because a conflicting lock is held, and + * fuse_do_readpage() asked for the page to be dropped and the read + * retried. ->write_begin() has no way to pass that request on: + * generic_perform_write() only inspects negative returns, so an + * AOP_TRUNCATED_PAGE escaping here would be taken for success and + * the uninitialised *foliop dereferenced. Retry here instead, now + * that the folio lock the revoke was waiting for has been dropped. + * Each pass costs a server round trip, which throttles the loop. + */ + if (err == AOP_TRUNCATED_PAGE) + goto retry; error: return err; } From 5890520665ab0caaa54424a30155881628f9fbfd Mon Sep 17 00:00:00 2001 From: Bernd Schubert Date: Thu, 13 Aug 2026 13:10:06 +0200 Subject: [PATCH 02/28] fuse: use -EDEADLK for DLM lock error instead of -EAGAIN -EAGAIN is semantically overloaded for a DLM error and not self describing, switch to -EDEADLK. In order to allow a graceful daemon change, -EAGAIN is kept for now. [hbi: adapted -- this tree has no iomap buffered-write path, so only the fuse_do_readpage() translation applies and the Documentation/filesystems/fuse note the original also updates does not exist here. Both codes are additionally gated on fc->dlm: without a DLM connection there is no lock-ordering conflict to report, so an -EAGAIN arriving from such a server is an ordinary error the caller has to see, not a request to drop the page and retry.] Signed-off-by: Bernd Schubert Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 2560afa1cb06c6..6315d1e7c0ee13 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -947,7 +947,19 @@ static int fuse_do_readpage(struct file *file, struct page *page) fuse_read_args_fill(&ia, file, pos, desc.length, FUSE_READ); res = fuse_simple_request(fm, &ia.ap.args); if (res < 0) { - if (res == -EAGAIN) + /* + * The DLM subsystem refuses a READ whose grant would deadlock + * against a conflicting lock this node already holds. Ask the + * caller to drop the page and retry so the conflicting holder + * can drain first. + * + * -EDEADLK is the self-describing code the server reports; + * -EAGAIN is the legacy spelling, still accepted so an older + * daemon keeps working. Only a DLM connection may make that + * claim -- elsewhere -EAGAIN is a plain error from the server + * and must be passed through. + */ + if ((res == -EDEADLK || res == -EAGAIN) && fm->fc->dlm) res = AOP_TRUNCATED_PAGE; return res; } From affca4437f7e718295c6266bfd13aeff6e3dd7df Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Mon, 22 Jun 2026 08:54:19 +0200 Subject: [PATCH 03/28] fuse: drop BDI_CAP_STRICTLIMIT from fuse bdi setup Signed-off-by: Horst Birthelmer --- fs/fuse/inode.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 88c377e95bf569..85099a9a3f04f8 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -1783,7 +1783,7 @@ static int fuse_bdi_init(struct fuse_conn *fc, struct super_block *sb) /* fuse does it's own writeback accounting */ sb->s_bdi->capabilities &= ~BDI_CAP_WRITEBACK_ACCT; - sb->s_bdi->capabilities |= BDI_CAP_STRICTLIMIT; + sb->s_bdi->capabilities &= ~BDI_CAP_STRICTLIMIT; /* * For a single fuse filesystem use max 1% of dirty + From 00d60f0fc3fd314ab37f5fef05cd91bc50a9721a Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 13 Aug 2026 13:19:08 +0200 Subject: [PATCH 04/28] fuse: switch to direct IO on an inode-invalidation notify storm A FUSE_NOTIFY_INVAL_INODE data invalidation means another (remote) entity is modifying the file. Rather than react to a single notify, keep a per-inode moving average of how fast data invalidations arrive for the whole file: an EWMA of the inter-arrival interval, updated under fi->lock on every notify (fuse_notify_inval_hot()). The inode is latched only once the average spacing drops below an internal threshold (FUSE_NOTIFY_DIO_INTERVAL) while a local writer is open; a lone or occasional notify keeps the average high and does not trip the switch. The heuristic has no external knob -- its parameters (EWMA weight, threshold, seed) are source-level constants. Introduce the forced-direct-IO latch (FUSE_I_FORCE_DIO): - fuse_reverse_inval_inode() folds each data invalidation into the moving average and sets the latch when it trips with a local writer present; - fuse_file_{read,write}_iter() and fuse_cache_write_iter() route to the direct path while latched; fuse_dio_{wr_exclusive_lock,lock,unlock}() use the shared parallel-dio path and bypass the cached/uncached accounting; - fuse_file_io_open() opens new files uncached so they do not re-enter caching mode; - fuse_prepare_release() clears the latch once the last writer is gone and fuse_file_release() drops any clean folios a racing read repopulated; fuse_file_mmap() reverts to caching mode (a mapping needs the page cache). Latching to direct IO is only coherent if no buffered write can deposit dirty folios into the page cache after it has been dropped. Add a per-inode rw_semaphore, wb_inval_rwsem, to serialise the buffered-write page-cache dirtying against the latch transition. The writeback path holds it for read around the dirtying and re-checks the latch under it; fuse_reverse_inval_inode() holds it for write around its invalidate + latch set. The notification may be delivered by the same server thread that still owes a reply to an in-flight write holding the inode lock, so it takes the rwsem with a trylock and never blocks: if the writer has gone it skips the latch and only invalidates the notified range. The writer's read-side section stays free of server round-trips because under fc->dlm the partial-write RMW read is skipped. Holding the rwsem across the dirtying means the writeback branch of fuse_cache_write_iter() can no longer delegate to generic_file_write_iter(), which takes and drops the inode lock itself. Open-code it -- inode_lock(), generic_write_checks(), __generic_file_write_iter(), inode_unlock(), generic_write_sync() -- so the read side spans exactly the page-cache dirtying. Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 155 ++++++++++++++++++++++++++++++++++++++++++++--- fs/fuse/fuse_i.h | 56 +++++++++++++++++ fs/fuse/inode.c | 98 +++++++++++++++++++++++++++++- fs/fuse/iomode.c | 10 +++ 4 files changed, 308 insertions(+), 11 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 6315d1e7c0ee13..784df7d84fff01 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -410,6 +410,18 @@ static void fuse_prepare_release(struct fuse_inode *fi, struct fuse_file *ff, if (likely(fi)) { spin_lock(&fi->lock); list_del(&ff->write_entry); + /* + * Leave forced direct IO mode once the last writer is gone: with + * no local writer left there is no cached-write contention with + * the remote modifier that triggered the switch. Restore + * FUSE_I_CACHE_IO_MODE for any frozen cached opens. + */ + if (test_bit(FUSE_I_FORCE_DIO, &fi->state) && + list_empty(&fi->write_files)) { + clear_bit(FUSE_I_FORCE_DIO, &fi->state); + if (fi->iocachectr > 0) + set_bit(FUSE_I_CACHE_IO_MODE, &fi->state); + } spin_unlock(&fi->lock); } spin_lock(&fc->lock); @@ -448,9 +460,24 @@ void fuse_file_release(struct inode *inode, struct fuse_file *ff, struct fuse_inode *fi = get_fuse_inode(inode); struct fuse_release_args *ra = &ff->args->release_args; int opcode = isdir ? FUSE_RELEASEDIR : FUSE_RELEASE; + bool was_force_dio = test_bit(FUSE_I_FORCE_DIO, &fi->state); fuse_prepare_release(fi, ff, open_flags, opcode, false); + /* + * If this release dropped the last writer, fuse_prepare_release() + * cleared the forced-direct-IO latch (under fi->lock). Drop any clean + * folios a read racing the latch may have repopulated so they cannot be + * served stale once caching mode resumes. No inode lock or + * wb_inval_rwsem: release may run on the fuse server thread (async fput + * from aio completion), where blocking on a contended inode lock could + * stall the connection. Writes were routed direct while latched, so + * only clean folios exist and this invalidate is server-free; the last + * writer is gone, so no forced-dio writer can race the drop. + */ + if (was_force_dio && !test_bit(FUSE_I_FORCE_DIO, &fi->state)) + invalidate_inode_pages2(inode->i_mapping); + if (ra && ff->flock) { ra->inarg.release_flags |= FUSE_RELEASE_FLOCK_UNLOCK; ra->inarg.lock_owner = fuse_lock_owner_id(ff->fm->fc, id); @@ -1413,9 +1440,15 @@ static bool fuse_dio_wr_exclusive_lock(struct kiocb *iocb, struct iov_iter *from struct fuse_file *ff = file->private_data; struct inode *inode = file_inode(iocb->ki_filp); struct fuse_inode *fi = get_fuse_inode(inode); + bool force_dio = test_bit(FUSE_I_FORCE_DIO, &fi->state); - /* Server side has to advise that it supports parallel dio writes. */ - if (!(ff->open_flags & FOPEN_PARALLEL_DIRECT_WRITES)) + /* + * Server side has to advise that it supports parallel dio writes. + * When the inode is latched into forced direct IO, parallel writes are + * used unconditionally: the page cache has been flushed and is bypassed + * for this inode. + */ + if (!force_dio && !(ff->open_flags & FOPEN_PARALLEL_DIRECT_WRITES)) return true; /* @@ -1426,7 +1459,7 @@ static bool fuse_dio_wr_exclusive_lock(struct kiocb *iocb, struct iov_iter *from return true; /* shared locks are not allowed with parallel page cache IO */ - if (test_bit(FUSE_I_CACHE_IO_MODE, &fi->state)) + if (!force_dio && test_bit(FUSE_I_CACHE_IO_MODE, &fi->state)) return true; /* Parallel dio beyond EOF is not supported, at least for now. */ @@ -1453,9 +1486,14 @@ static void fuse_dio_lock(struct kiocb *iocb, struct iov_iter *from, * should be performed only after taking shared inode lock. * Previous past eof check was without inode lock and might * have raced, so check it again. + * + * Under the forced-dio latch the cached/uncached accounting is + * bypassed (the latch guarantees the cache is flushed and not + * repopulated), so only re-check the past-eof condition. */ if (fuse_io_past_eof(iocb, from) || - fuse_inode_uncached_io_start(fi, NULL) != 0) { + (!test_bit(FUSE_I_FORCE_DIO, &fi->state) && + fuse_inode_uncached_io_start(fi, NULL) != 0)) { inode_unlock_shared(inode); inode_lock(inode); *exclusive = true; @@ -1471,12 +1509,19 @@ static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive) if (exclusive) { inode_unlock(inode); } else { - /* Allow opens in caching mode after last parallel dio end */ - fuse_inode_uncached_io_end(fi); + /* + * Allow opens in caching mode after last parallel dio end. + * Skipped under the forced-dio latch, which never took an + * uncached_io reference in fuse_dio_lock(). + */ + if (!test_bit(FUSE_I_FORCE_DIO, &fi->state)) + fuse_inode_uncached_io_end(fi); inode_unlock_shared(inode); } } +static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from); + static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) { struct file *file = iocb->ki_filp; @@ -1486,6 +1531,18 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) struct inode *inode = mapping->host; ssize_t err, count; struct fuse_conn *fc = get_fuse_conn(inode); + struct fuse_inode *fi = get_fuse_inode(inode); + bool wb_guard = false; + + /* + * The inode may have been latched into forced direct IO -- by a + * NOTIFY_INVAL_INODE arriving while this inode is open for writing here + * -- after this write was routed to the cached path but before it took + * any lock. Re-route to the direct path (before taking a DLM lock) so + * we do not repopulate the page cache the latch just dropped. + */ + if (fuse_inode_force_dio(inode)) + return fuse_direct_write_iter(iocb, from); if (fc->writeback_cache) { /* Update size (EOF optimization) and mode (SUID clearing) */ @@ -1511,12 +1568,64 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) size_t length = iov_iter_count(from); fuse_get_dlm_write_lock(file, pos, length); } - return generic_file_write_iter(iocb, from); + + /* + * Open-code generic_file_write_iter() so that wb_inval_rwsem + * can be held for read across the page-cache dirtying: a + * concurrent NOTIFY_INVAL_INODE -- which latches the inode + * under the write side of that lock via a non-blocking trylock + * -- must not be able to strand the folios we are about to + * dirty. Re-check the latch under it (it may have been set + * while we blocked on the inode lock) and re-route to the + * direct path if it is now set; the DLM write lock taken above + * is harmless there, as the direct path does its own server + * coordination. The forced-direct-IO latch feature is only + * active under writeback+dlm, so the guard follows fc->dlm. + */ + inode_lock(inode); + if (fc->dlm) { + down_read(&fi->wb_inval_rwsem); + if (fuse_inode_force_dio(inode)) { + up_read(&fi->wb_inval_rwsem); + inode_unlock(inode); + return fuse_direct_write_iter(iocb, from); + } + } + written = generic_write_checks(iocb, from); + if (written > 0) + written = __generic_file_write_iter(iocb, from); + if (fc->dlm) + up_read(&fi->wb_inval_rwsem); + inode_unlock(inode); + if (written > 0) + written = generic_write_sync(iocb, written); + return written; } writethrough: inode_lock(inode); + /* + * The killpriv fallback lands here with the writeback cache still on, + * so it populates the page cache too and needs the same guard as the + * writeback branch above: hold wb_inval_rwsem for read across the + * page-cache population and re-check the latch under it, so a + * concurrent NOTIFY_INVAL_INODE cannot have the cache repopulated + * behind the invalidate it just did. Taken before + * task_io_account_write() so a re-route is not double-counted. A + * connection without the writeback cache never latches, and has no + * gate to take. + */ + wb_guard = fc->writeback_cache && fc->dlm; + if (wb_guard) { + down_read(&fi->wb_inval_rwsem); + if (fuse_inode_force_dio(inode)) { + up_read(&fi->wb_inval_rwsem); + inode_unlock(inode); + return fuse_direct_write_iter(iocb, from); + } + } + err = count = generic_write_checks(iocb, from); if (err <= 0) goto out; @@ -1541,6 +1650,8 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) written = fuse_perform_write(iocb, from); } out: + if (wb_guard) + up_read(&fi->wb_inval_rwsem); inode_unlock(inode); if (written > 0) written = generic_write_sync(iocb, written); @@ -1820,7 +1931,7 @@ static ssize_t fuse_file_read_iter(struct kiocb *iocb, struct iov_iter *to) return fuse_dax_read_iter(iocb, to); /* FOPEN_DIRECT_IO overrides FOPEN_PASSTHROUGH */ - if (ff->open_flags & FOPEN_DIRECT_IO) + if ((ff->open_flags & FOPEN_DIRECT_IO) || fuse_inode_force_dio(inode)) return fuse_direct_read_iter(iocb, to); else if (fuse_file_passthrough(ff)) return fuse_passthrough_read_iter(iocb, to); @@ -1841,7 +1952,7 @@ static ssize_t fuse_file_write_iter(struct kiocb *iocb, struct iov_iter *from) return fuse_dax_write_iter(iocb, from); /* FOPEN_DIRECT_IO overrides FOPEN_PASSTHROUGH */ - if (ff->open_flags & FOPEN_DIRECT_IO) + if ((ff->open_flags & FOPEN_DIRECT_IO) || fuse_inode_force_dio(inode)) return fuse_direct_write_iter(iocb, from); else if (fuse_file_passthrough(ff)) return fuse_passthrough_write_iter(iocb, from); @@ -2573,6 +2684,29 @@ static int fuse_file_mmap(struct file *file, struct vm_area_struct *vma) else if (fuse_inode_backing(get_fuse_inode(inode))) return -ENODEV; + /* + * If the inode was latched into forced direct IO after a remote-modify + * notification, a mapping needs the page cache, so revert to caching + * mode. Revert without the inode lock or wb_inval_rwsem: ->mmap runs + * under mmap_lock and the buffered write path holds both across a fault + * on the user buffer (which takes mmap_lock), so taking either here + * would invert lock order (ABBA). Clearing the latch and dropping the + * cache is sufficient -- writers re-check the latch and route to cached + * IO once it is clear, and in-flight parallel dio drains itself. Cached + * opens frozen while latched are still counted in iocachectr, so restore + * FUSE_I_CACHE_IO_MODE for them. + */ + if (fuse_inode_force_dio(inode)) { + struct fuse_inode *fi = get_fuse_inode(inode); + + spin_lock(&fi->lock); + clear_bit(FUSE_I_FORCE_DIO, &fi->state); + if (fi->iocachectr > 0) + set_bit(FUSE_I_CACHE_IO_MODE, &fi->state); + spin_unlock(&fi->lock); + invalidate_inode_pages2(file->f_mapping); + } + /* * FOPEN_DIRECT_IO handling is special compared to O_DIRECT, * as does not allow MAP_SHARED mmap without FUSE_DIRECT_IO_ALLOW_MMAP. @@ -3388,6 +3522,9 @@ void fuse_init_file_inode(struct inode *inode, unsigned int flags) fi->iocachectr = 0; init_waitqueue_head(&fi->page_waitq); init_waitqueue_head(&fi->direct_io_waitq); + init_rwsem(&fi->wb_inval_rwsem); + fi->notify_stamp = jiffies; + fi->notify_interval_ewma = FUSE_NOTIFY_EWMA_SEED << FUSE_NOTIFY_EWMA_SHIFT; if (IS_ENABLED(CONFIG_FUSE_DAX)) fuse_dax_inode_init(inode, flags); diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index 3b3897f7ddfc80..d2bac70fda7918 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -106,6 +106,20 @@ struct dlm_locked_area size_t size; }; +/* + * Force-DIO switch trigger: an exponentially weighted moving average of the + * interval (in jiffies) between FUSE_NOTIFY_INVAL_INODE data invalidations for + * a file. When the average spacing falls below FUSE_NOTIFY_DIO_INTERVAL -- a + * remote writer streaming invalidations -- and the file is open for writing + * here, it is latched into direct IO. These are the source-level (not + * externally tunable) parameters of the heuristic: EWMA weight 1/2^SHIFT, + * seeded and capped at SEED so it takes a short burst rather than a single + * notify to trip. + */ +#define FUSE_NOTIFY_DIO_INTERVAL max_t(unsigned long, HZ / 10, 1) +#define FUSE_NOTIFY_EWMA_SHIFT 2 +#define FUSE_NOTIFY_EWMA_SEED (2 * FUSE_NOTIFY_DIO_INTERVAL) + /** FUSE inode */ struct fuse_inode { /** Inode data */ @@ -173,6 +187,34 @@ struct fuse_inode { /* dlm locked areas we have sent lock requests for */ struct fuse_dlm_cache dlm_locked_areas; + + /* + * Serializes buffered-write page-cache dirtying against + * the forced-direct-IO latch transition driven by + * NOTIFY_INVAL_INODE (fuse_reverse_inval_inode()), which + * may be delivered by the same server thread that still + * owes a reply to an in-flight write holding the inode + * lock. The buffered writer holds this for read around + * the dirtying and re-checks the latch under it; the + * NOTIFY latch site takes it for write (trylock, never + * blocking) around its page-cache invalidate + latch set. + * Only regular files initialise it -- it shares storage + * with the readdir-cache union arm. + */ + struct rw_semaphore wb_inval_rwsem; + + /* + * Rate of FUSE_NOTIFY_INVAL_INODE data invalidations + * for this whole file: notify_stamp is the jiffies of + * the last one, notify_interval_ewma the EWMA of the + * inter-arrival interval (jiffies, scaled by + * 2^FUSE_NOTIFY_EWMA_SHIFT). A rapid stream (short + * average interval) with a local writer latches the + * inode into direct IO. Protected by fi->lock; regular + * files only (shares the readdir-cache union arm). + */ + unsigned long notify_stamp; + unsigned int notify_interval_ewma; }; /* readdir cache (directory only) */ @@ -238,6 +280,14 @@ enum { FUSE_I_BTIME, /* Wants or already has page cache IO */ FUSE_I_CACHE_IO_MODE, + /* + * Latched into direct IO: a NOTIFY_INVAL_INODE arrived while the file + * was open for writing here, so another (remote) entity is modifying it + * concurrently. Reads and writes are routed direct (shared-lock + * parallel dio) until the last writer closes or the inode is mmapped. + * See fuse_reverse_inval_inode()/fuse_file_io_open(). + */ + FUSE_I_FORCE_DIO, }; struct fuse_conn; @@ -1556,6 +1606,12 @@ void fuse_inode_uncached_io_end(struct fuse_inode *fi); int fuse_file_io_open(struct file *file, struct inode *inode); void fuse_file_io_release(struct fuse_file *ff, struct inode *inode); +/* Inode latched into forced direct IO after a remote-modify notification */ +static inline bool fuse_inode_force_dio(struct inode *inode) +{ + return test_bit(FUSE_I_FORCE_DIO, &get_fuse_inode(inode)->state); +} + /* file.c */ struct fuse_file *fuse_file_open(struct fuse_mount *fm, u64 nodeid, struct inode *inode, diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 85099a9a3f04f8..a6ca0cdc337f1a 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -745,6 +745,35 @@ static void fuse_invalidate_inode_entry(struct inode *inode) } } +/* + * Fold one FUSE_NOTIFY_INVAL_INODE data invalidation into the per-inode + * moving average of the notification inter-arrival interval and report whether + * the file is now "hot" -- notifications are arriving fast enough (short + * average interval) that a remote writer is repeatedly invalidating it. The + * average is an EWMA (weight 1/2^FUSE_NOTIFY_EWMA_SHIFT); the sample is clamped + * to FUSE_NOTIFY_EWMA_SEED so a notify after a long idle only cools the average + * and cannot overflow the accumulator. Must be called under fi->lock; called + * for every data invalidation so the average stays current even while no local + * writer is open. + */ +static bool fuse_notify_inval_hot(struct fuse_inode *fi) +{ + unsigned long now = jiffies; + unsigned long sample; + unsigned int avg; + + sample = min_t(unsigned long, now - fi->notify_stamp, + FUSE_NOTIFY_EWMA_SEED); + fi->notify_stamp = now; + + /* E += sample - (E >> SHIFT); avg = E >> SHIFT */ + fi->notify_interval_ewma += sample - + (fi->notify_interval_ewma >> FUSE_NOTIFY_EWMA_SHIFT); + avg = fi->notify_interval_ewma >> FUSE_NOTIFY_EWMA_SHIFT; + + return avg < FUSE_NOTIFY_DIO_INTERVAL; +} + int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, loff_t offset, loff_t len) { @@ -794,8 +823,73 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, pg_end == -1 ? 0 : (offset + len - 1)); - invalidate_inode_pages2_range(inode->i_mapping, - pg_start, pg_end); + /* + * A data invalidation means another (remote) entity is modifying + * the file. Keep a moving average of how fast these notifications + * arrive for the whole inode; when they come in a rapid stream -- + * a remote writer repeatedly invalidating the file -- and it is + * also open for writing here, latch the inode into direct IO: + * reads and writes are served direct (from the server) until the + * last writer closes or the inode is mmapped. A lone or occasional + * notify keeps the average high and does not trip the switch. Only + * under writeback+dlm, where the buffered write's RMW read is + * skipped so its wb_inval_rwsem read-side section is free of server + * round-trips. + * + * fuse_notify_inval_hot() updates the average under fi->lock and is + * called for every data invalidation so it stays current even while + * no local writer is open. When it trips, take wb_inval_rwsem for + * write so the buffered write path -- which holds it for read across + * its dirtying and re-checks the latch under it -- cannot strand + * dirty folios after the cache is dropped. Use a trylock and never + * block: this may run on the server thread that still owes an + * in-flight write (holding the inode lock) its reply, so blocking on + * the rwsem or the inode lock would deadlock. If the writer has + * gone, skip the latch this round (best effort); the invalidate + * still runs. Only regular files initialise the average and the + * rwsem (they share storage with the readdir-cache union arm), so + * gate on S_ISREG. When latched, drop the whole mapping rather than + * just the notified range, or dirty folios outside it would be + * invisible to the forced direct reads (stale read / lost write). + */ + if (S_ISREG(inode->i_mode) && fc->writeback_cache && fc->dlm && + !FUSE_IS_DAX(inode) && !fuse_inode_backing(fi) && + !mapping_mapped(inode->i_mapping)) { + bool hot, has_writer; + + spin_lock(&fi->lock); + hot = fuse_notify_inval_hot(fi); + has_writer = !list_empty(&fi->write_files); + spin_unlock(&fi->lock); + + if (hot && has_writer && !fuse_inode_force_dio(inode) && + down_write_trylock(&fi->wb_inval_rwsem)) { + bool latched = false; + + spin_lock(&fi->lock); + if (!list_empty(&fi->write_files)) { + set_bit(FUSE_I_FORCE_DIO, &fi->state); + latched = true; + } + spin_unlock(&fi->lock); + + if (latched) { + pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", + nodeid); + invalidate_inode_pages2(inode->i_mapping); + } else { + invalidate_inode_pages2_range(inode->i_mapping, + pg_start, pg_end); + } + up_write(&fi->wb_inval_rwsem); + } else { + invalidate_inode_pages2_range(inode->i_mapping, + pg_start, pg_end); + } + } else { + invalidate_inode_pages2_range(inode->i_mapping, + pg_start, pg_end); + } } iput(inode); return 0; diff --git a/fs/fuse/iomode.c b/fs/fuse/iomode.c index c99e285f3183ef..301294c8fb25f0 100644 --- a/fs/fuse/iomode.c +++ b/fs/fuse/iomode.c @@ -233,6 +233,16 @@ int fuse_file_io_open(struct file *file, struct inode *inode) !(ff->open_flags & FOPEN_PASSTHROUGH)) return 0; + /* + * The inode was latched into direct IO after a remote-modify + * notification arrived while it was open for writing here. Open this + * file uncached as well so its IO is routed direct and it does not + * re-enter caching mode. + */ + if (test_bit(FUSE_I_FORCE_DIO, &fi->state) && + !(ff->open_flags & FOPEN_PASSTHROUGH)) + return 0; + if (ff->open_flags & FOPEN_PASSTHROUGH) err = fuse_file_passthrough_open(inode, file); else From d33312f3a4a69007c73e00f7688fe862b03119a6 Mon Sep 17 00:00:00 2001 From: Joanne Koong Date: Thu, 13 Aug 2026 13:20:12 +0200 Subject: [PATCH 05/28] fuse: use default writeback accounting commit 0c58a97f919c ("fuse: remove tmp folio for writebacks and internal rb tree") removed temp folios for dirty page writeback. Consequently, fuse can now use the default writeback accounting. With switching fuse to use default writeback accounting, there are some added benefits. This updates wb->writeback_inodes tracking as well now and updates writeback throughput estimates after writeback completion. This commit also removes inc_wb_stat() and dec_wb_stat(). These have no callers anymore now that fuse does not call them. Signed-off-by: Joanne Koong Reviewed-by: David Hildenbrand Reviewed-by: Bernd Schubert Signed-off-by: Miklos Szeredi (cherry picked from commit 494d2f508883a6e5c4530e5c6b3c8b2bbfb7318d) Ported to the page-based writeback path this tree still carries: the manual accounting sits in fuse_writepage_finish_stat() and in both fuse_writepage_args_page_fill() and fuse_writepages_fill(), and each of those pairs 1:1 with the folio_start_writeback()/end_page_writeback() the generic accounting hooks, so dropping them is equivalent. The include/linux/backing-dev.h hunk removing the now-callerless inc_wb_stat()/dec_wb_stat() helpers is left out: only fuse ever used them, and fuse ships as a module here, so the header change would buy nothing but a full-tree rebuild. Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 16 +--------------- fs/fuse/inode.c | 2 -- 2 files changed, 1 insertion(+), 17 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 784df7d84fff01..8c032adc4dd178 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1999,14 +1999,6 @@ static void fuse_writepage_free(struct fuse_writepage_args *wpa) kfree(wpa); } -static void fuse_writepage_finish_stat(struct inode *inode, struct page *page) -{ - struct backing_dev_info *bdi = inode_to_bdi(inode); - - dec_wb_stat(&bdi->wb, WB_WRITEBACK); - wb_writeout_inc(&bdi->wb); -} - static void fuse_writepage_finish(struct fuse_writepage_args *wpa) { struct fuse_args_pages *ap = &wpa->ia.ap; @@ -2014,10 +2006,8 @@ static void fuse_writepage_finish(struct fuse_writepage_args *wpa) struct fuse_inode *fi = get_fuse_inode(inode); int i; - for (i = 0; i < ap->num_pages; i++) { - fuse_writepage_finish_stat(inode, ap->pages[i]); + for (i = 0; i < ap->num_pages; i++) end_page_writeback(ap->pages[i]); - } wake_up(&fi->page_waitq); } @@ -2199,14 +2189,11 @@ static void fuse_writepage_add_to_bucket(struct fuse_conn *fc, static void fuse_writepage_args_page_fill(struct fuse_writepage_args *wpa, struct folio *folio, uint32_t page_index) { - struct inode *inode = folio->mapping->host; struct fuse_args_pages *ap = &wpa->ia.ap; ap->pages[page_index] = &folio->page; ap->descs[page_index].offset = 0; ap->descs[page_index].length = PAGE_SIZE; - - inc_wb_stat(&inode_to_bdi(inode)->wb, WB_WRITEBACK); } static struct fuse_writepage_args *fuse_writepage_args_setup(struct folio *folio, @@ -2395,7 +2382,6 @@ static int fuse_writepages_fill(struct folio *folio, ap->descs[ap->num_pages].length = PAGE_SIZE; ap->pages[ap->num_pages] = &folio->page; ap->num_pages++; - inc_wb_stat(&inode_to_bdi(inode)->wb, WB_WRITEBACK); err = 0; if (!data->wpa) { diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index a6ca0cdc337f1a..7fa31fd2e75bfa 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -1875,8 +1875,6 @@ static int fuse_bdi_init(struct fuse_conn *fc, struct super_block *sb) if (err) return err; - /* fuse does it's own writeback accounting */ - sb->s_bdi->capabilities &= ~BDI_CAP_WRITEBACK_ACCT; sb->s_bdi->capabilities &= ~BDI_CAP_STRICTLIMIT; /* From fcacbc80d6b432f416692517db72965059fc0b1b Mon Sep 17 00:00:00 2001 From: Bernd Schubert Date: Thu, 13 Aug 2026 13:21:30 +0200 Subject: [PATCH 06/28] fuse: allow parallel direct writes past EOF Extending FOPEN_PARALLEL_DIRECT_WRITES writes were forced onto the exclusive inode lock, re-serializing the parallel phase. The exclusive lock only bundled "write + advance i_size + undo-on-failure" into one unit. But i_size is committed by fuse_write_update_attr() under fi->lock, only on a successful growing write and independent of the inode rwsem -- so shared-lock writers commit size correctly and have nothing to undo. Drop the past-EOF exclusive triggers and gate the whole-file fuse_do_truncate() rollback on holding the exclusive lock. Lock mode is passed to __fuse_direct_IO(); i_size is committed at the same point in every path, only the failure rollback differs: non-exclusive (relaxed, parallel): fuse_direct_write_iter fuse_dio_lock -> inode_lock_shared (exclusive=false) __fuse_direct_IO(.., false) fuse_direct_io() write to server fuse_write_update_attr() commit i_size (on success) no rollback exclusive (append / caching / !parallel): fuse_direct_write_iter fuse_dio_lock -> inode_lock (exclusive=true) __fuse_direct_IO(.., true) fuse_direct_io() write to server fuse_write_update_attr() commit i_size (on success) ret<0 & extend -> fuse_do_truncate() rollback exclusive (caching-mode O_DIRECT): fuse_cache_write_iter -> inode_lock (exclusive) generic_file_direct_write -> fuse_direct_IO __fuse_direct_IO(.., true) fuse_direct_io() write to server fuse_write_update_attr() commit i_size (on success) ret<0 & extend -> fuse_do_truncate() rollback [hbi: adapted -- kept this tree's local `&& iocb->ki_flags & IOCB_DIRECT` guards on the async routing in fuse_direct_{read,write}_iter, which the upstream branch does not have.] Signed-off-by: Bernd Schubert Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 53 ++++++++++++++++++++++++++------------------------ 1 file changed, 28 insertions(+), 25 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 8c032adc4dd178..a71e4564bbf730 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1424,13 +1424,6 @@ static ssize_t fuse_perform_write(struct kiocb *iocb, struct iov_iter *ii) return res; } -static bool fuse_io_past_eof(struct kiocb *iocb, struct iov_iter *iter) -{ - struct inode *inode = file_inode(iocb->ki_filp); - - return iocb->ki_pos + iov_iter_count(iter) > i_size_read(inode); -} - /* * @return true if an exclusive lock for direct IO writes is needed */ @@ -1462,10 +1455,6 @@ static bool fuse_dio_wr_exclusive_lock(struct kiocb *iocb, struct iov_iter *from if (!force_dio && test_bit(FUSE_I_CACHE_IO_MODE, &fi->state)) return true; - /* Parallel dio beyond EOF is not supported, at least for now. */ - if (fuse_io_past_eof(iocb, from)) - return true; - return false; } @@ -1484,16 +1473,15 @@ static void fuse_dio_lock(struct kiocb *iocb, struct iov_iter *from, * New parallal dio allowed only if inode is not in caching * mode and denies new opens in caching mode. This check * should be performed only after taking shared inode lock. - * Previous past eof check was without inode lock and might - * have raced, so check it again. * - * Under the forced-dio latch the cached/uncached accounting is - * bypassed (the latch guarantees the cache is flushed and not - * repopulated), so only re-check the past-eof condition. + * Under the forced-dio latch the uncached-io accounting is + * bypassed entirely -- fuse_dio_unlock() likewise skips + * fuse_inode_uncached_io_end() -- so do not take a reference + * here. An unbalanced start would drive fi->iocachectr + * permanently negative and hang the next caching-mode open. */ - if (fuse_io_past_eof(iocb, from) || - (!test_bit(FUSE_I_FORCE_DIO, &fi->state) && - fuse_inode_uncached_io_start(fi, NULL) != 0)) { + if (!test_bit(FUSE_I_FORCE_DIO, &fi->state) && + fuse_inode_uncached_io_start(fi, NULL) != 0) { inode_unlock_shared(inode); inode_lock(inode); *exclusive = true; @@ -1865,14 +1853,16 @@ static ssize_t __fuse_direct_read(struct fuse_io_priv *io, return res; } -static ssize_t fuse_direct_IO(struct kiocb *iocb, struct iov_iter *iter); +static ssize_t __fuse_direct_IO(struct kiocb *iocb, struct iov_iter *iter, + bool exclusive); static ssize_t fuse_direct_read_iter(struct kiocb *iocb, struct iov_iter *to) { ssize_t res; if (!is_sync_kiocb(iocb) && iocb->ki_flags & IOCB_DIRECT) { - res = fuse_direct_IO(iocb, to); + /* exclusive is unused on reads; rollback is write-only */ + res = __fuse_direct_IO(iocb, to, true); } else { struct fuse_io_priv io = FUSE_IO_PRIV_SYNC(iocb); @@ -1896,7 +1886,7 @@ static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from) if (res > 0) { task_io_account_write(res); if (!is_sync_kiocb(iocb) && iocb->ki_flags & IOCB_DIRECT) { - res = fuse_direct_IO(iocb, from); + res = __fuse_direct_IO(iocb, from, exclusive); } else { res = fuse_direct_io(&io, from, &iocb->ki_pos, FUSE_DIO_WRITE); @@ -3121,7 +3111,7 @@ static inline loff_t fuse_round_up(struct fuse_conn *fc, loff_t off) } static ssize_t -fuse_direct_IO(struct kiocb *iocb, struct iov_iter *iter) +__fuse_direct_IO(struct kiocb *iocb, struct iov_iter *iter, bool exclusive) { DECLARE_COMPLETION_ONSTACK(wait); ssize_t ret = 0; @@ -3215,14 +3205,27 @@ fuse_direct_IO(struct kiocb *iocb, struct iov_iter *iter) if (iov_iter_rw(iter) == WRITE) { fuse_write_update_attr(inode, pos, ret); - /* For extending writes we already hold exclusive lock */ - if (ret < 0 && offset + count > i_size) + /* + * Whole-file rollback is only safe under an exclusive lock. + * Parallel writers commit i_size only on success (nothing to + * undo); the server owns failed-extend cleanup. + */ + if (exclusive && ret < 0 && offset + count > i_size) fuse_do_truncate(file); } return ret; } +static ssize_t fuse_direct_IO(struct kiocb *iocb, struct iov_iter *iter) +{ + /* + * Only reached via generic_file_direct_write/read() + * (caching-mode O_DIRECT), which holds the inode lock exclusively. + */ + return __fuse_direct_IO(iocb, iter, true); +} + static int fuse_writeback_range(struct inode *inode, loff_t start, loff_t end) { int err = filemap_write_and_wait_range(inode->i_mapping, start, LLONG_MAX); From 0f696e4d7d78e3f7f228ed20820e18eca7e0c3bc Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 16 Jul 2026 14:02:56 +0200 Subject: [PATCH 07/28] fuse: balance uncached_io accounting under forced-DIO latch fuse_dio_lock() takes an uncached_io reference (via fuse_inode_uncached_io_start()) only when FUSE_I_FORCE_DIO is clear, while fuse_dio_unlock() decided whether to drop it by re-reading FUSE_I_FORCE_DIO. On this tree the latch is toggled asynchronously by the inode-invalidation notify-storm path, so the bit can differ between the lock and the unlock of a single direct write: - clear at lock (reference taken), set before unlock: the reference is never dropped, leaving fi->iocachectr permanently negative and hanging the next caching-mode open; - set at lock (no reference), cleared before unlock: fuse_inode_uncached_io_end() is called without a matching start, tripping WARN_ON(fi->iocachectr >= 0) and corrupting the counter. Record in fuse_dio_lock() whether a reference was actually taken and have fuse_dio_unlock() drop it based on that captured decision instead of re-testing the racy bit, so the accounting stays balanced regardless of any FORCE_DIO transition mid-write. Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index a71e4564bbf730..c7ebbeb711e492 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1459,7 +1459,7 @@ static bool fuse_dio_wr_exclusive_lock(struct kiocb *iocb, struct iov_iter *from } static void fuse_dio_lock(struct kiocb *iocb, struct iov_iter *from, - bool *exclusive) + bool *exclusive, bool *uncached) { struct inode *inode = file_inode(iocb->ki_filp); struct fuse_inode *fi = get_fuse_inode(inode); @@ -1473,23 +1473,20 @@ static void fuse_dio_lock(struct kiocb *iocb, struct iov_iter *from, * New parallal dio allowed only if inode is not in caching * mode and denies new opens in caching mode. This check * should be performed only after taking shared inode lock. - * - * Under the forced-dio latch the uncached-io accounting is - * bypassed entirely -- fuse_dio_unlock() likewise skips - * fuse_inode_uncached_io_end() -- so do not take a reference - * here. An unbalanced start would drive fi->iocachectr - * permanently negative and hang the next caching-mode open. */ - if (!test_bit(FUSE_I_FORCE_DIO, &fi->state) && - fuse_inode_uncached_io_start(fi, NULL) != 0) { - inode_unlock_shared(inode); - inode_lock(inode); - *exclusive = true; + if (!test_bit(FUSE_I_FORCE_DIO, &fi->state)) { + if (fuse_inode_uncached_io_start(fi, NULL) != 0) { + inode_unlock_shared(inode); + inode_lock(inode); + *exclusive = true; + } else { + *uncached = true; + } } } } -static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive) +static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive, bool uncached) { struct inode *inode = file_inode(iocb->ki_filp); struct fuse_inode *fi = get_fuse_inode(inode); @@ -1497,12 +1494,7 @@ static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive) if (exclusive) { inode_unlock(inode); } else { - /* - * Allow opens in caching mode after last parallel dio end. - * Skipped under the forced-dio latch, which never took an - * uncached_io reference in fuse_dio_lock(). - */ - if (!test_bit(FUSE_I_FORCE_DIO, &fi->state)) + if (uncached) fuse_inode_uncached_io_end(fi); inode_unlock_shared(inode); } @@ -1878,10 +1870,11 @@ static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from) struct fuse_io_priv io = FUSE_IO_PRIV_SYNC(iocb); struct address_space *mapping = inode->i_mapping; loff_t pos = iocb->ki_pos; + bool exclusive = false; + bool uncached = false; ssize_t res; - bool exclusive; - fuse_dio_lock(iocb, from, &exclusive); + fuse_dio_lock(iocb, from, &exclusive, &uncached); res = generic_write_checks(iocb, from); if (res > 0) { task_io_account_write(res); @@ -1903,7 +1896,7 @@ static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from) (pos + res - 1) >> PAGE_SHIFT); } } - fuse_dio_unlock(iocb, exclusive); + fuse_dio_unlock(iocb, exclusive, uncached); return res; } From 3488a13ce657e7a6e4ca4eb87e0f48853d885f73 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 13 Aug 2026 13:23:48 +0200 Subject: [PATCH 08/28] fuse: don't hold i_rwsem exclusively when doing buffered write On the DLM-serialised writeback path the DLM write lock already excludes writers cluster-wide, so disjoint local writers (MPI-IO / IOR) may take the inode rwsem shared instead of exclusive, mirroring fuse_dio_wr_exclusive_lock() on the direct path. O_APPEND, O_DIRECT and the killpriv writethrough fallback keep the exclusive lock. i_size is then no longer protected by an exclusive inode lock in that mode. Commit the EOF extension in fuse_write_end() under fi->lock -- the same way fuse_write_update_attr() does on the direct path -- instead of by its previous unlocked read-modify-write, which two concurrent extenders could lose an update to. The upstream branch this comes from writes through iomap, where the generic code owns the i_size update, and therefore has to claim the whole extension up front under fi->lock so iomap never touches i_size itself. Here ->write_end() is fuse's own, so the update can simply be made atomic where it already happens. Growing i_size just behind the write cursor rather than claiming the extension up front also keeps fuse_write_begin()'s beyond-EOF optimisation effective: folios wholly past EOF are zeroed locally instead of sending the server a read-modify-write READ for data that does not exist. Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 69 insertions(+), 5 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index c7ebbeb711e492..ff15323695699f 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1502,6 +1502,45 @@ static void fuse_dio_unlock(struct kiocb *iocb, bool exclusive, bool uncached) static ssize_t fuse_direct_write_iter(struct kiocb *iocb, struct iov_iter *from); +/* + * @return true if an exclusive inode lock is needed for a cached (buffered) + * write. + * + * Buffered writes normally hold the inode rwsem exclusively, serialising all + * writers even on disjoint ranges. The DLM-serialised writeback path is + * the exception: the DLM already excludes cluster-wide, and i_size is committed + * under fi->lock rather than the inode rwsem (see fuse_write_end()), so + * disjoint writers (MPI-IO / IOR) may share the lock. Mirrors + * fuse_dio_wr_exclusive_lock() for the direct path. + */ +static bool fuse_cache_wr_exclusive_lock(struct kiocb *iocb, bool writeback) +{ + struct inode *inode = file_inode(iocb->ki_filp); + struct fuse_conn *fc = get_fuse_conn(inode); + + /* Only the DLM-serialised writeback path relaxes the lock. */ + if (!fc->dlm || !writeback) + return true; + + /* O_DIRECT writes fall back to generic_file_direct_write(). */ + if (iocb->ki_flags & IOCB_DIRECT) + return true; + + /* Append needs the eventual EOF - always needs an exclusive lock. */ + if (iocb->ki_flags & IOCB_APPEND) + return true; + + return false; +} + +static void fuse_cache_wr_unlock(struct inode *inode, bool exclusive) +{ + if (exclusive) + inode_unlock(inode); + else + inode_unlock_shared(inode); +} + static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) { struct file *file = iocb->ki_filp; @@ -1513,6 +1552,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) struct fuse_conn *fc = get_fuse_conn(inode); struct fuse_inode *fi = get_fuse_inode(inode); bool wb_guard = false; + bool exclusive = true; /* * The inode may have been latched into forced direct IO -- by a @@ -1562,12 +1602,16 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) * coordination. The forced-direct-IO latch feature is only * active under writeback+dlm, so the guard follows fc->dlm. */ - inode_lock(inode); + exclusive = fuse_cache_wr_exclusive_lock(iocb, true); + if (exclusive) + inode_lock(inode); + else + inode_lock_shared(inode); if (fc->dlm) { down_read(&fi->wb_inval_rwsem); if (fuse_inode_force_dio(inode)) { up_read(&fi->wb_inval_rwsem); - inode_unlock(inode); + fuse_cache_wr_unlock(inode, exclusive); return fuse_direct_write_iter(iocb, from); } } @@ -1576,7 +1620,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) written = __generic_file_write_iter(iocb, from); if (fc->dlm) up_read(&fi->wb_inval_rwsem); - inode_unlock(inode); + fuse_cache_wr_unlock(inode, exclusive); if (written > 0) written = generic_write_sync(iocb, written); return written; @@ -2490,8 +2534,28 @@ static int fuse_write_end(struct file *file, struct address_space *mapping, folio_mark_uptodate(folio); } - if (pos > inode->i_size) - i_size_write(inode, pos); + /* + * On the DLM writeback path the inode rwsem may be held shared (see + * fuse_cache_wr_exclusive_lock()), so i_size is no longer serialised + * by it. Commit the extension monotonically under fi->lock -- the + * same way fuse_write_update_attr() does on the direct path -- rather + * than by an unlocked read-modify-write two concurrent extenders could + * lose an update to. + * + * Growing i_size just behind the write cursor, rather than claiming + * the whole extension up front, also keeps fuse_write_begin()'s + * beyond-EOF optimisation effective: folios wholly past EOF are zeroed + * locally instead of sending the server a read-modify-write READ for + * data that does not exist yet. + */ + if (pos > inode->i_size) { + struct fuse_inode *fi = get_fuse_inode(inode); + + spin_lock(&fi->lock); + if (pos > inode->i_size) + i_size_write(inode, pos); + spin_unlock(&fi->lock); + } folio_mark_dirty(folio); From 385d3ee00150c414dd72243eae2079eda9ced35b Mon Sep 17 00:00:00 2001 From: Hai Zhong Zhou Date: Thu, 13 Aug 2026 13:24:27 +0200 Subject: [PATCH 09/28] fuse: acquire dlm lock for the normal buffer read Acquire the dlm lock from fuse server for the normal buffer read path to ensure the distributed page cache across different nodes can be co-existing and consistency. More importantly, this change will correct the DLM lock and folio locks ordering for the buffer read path, thus can avoid the potential deadlock between the buffer read and page cache invalidation processes. [hbi: adapted -- the fuse_dlm_cache.{c,h} rename to fuse_get_dlm_lock() applies unmodified, but fuse_cache_write_iter() has a different shape in this tree (a `goto writethrough` fallback rather than a single writeback flag), so its call site was adjusted by hand.] Signed-off-by Hai Zhong Zhou Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 14 +++++++++++--- fs/fuse/fuse_dlm_cache.c | 22 ++++++++++++---------- fs/fuse/fuse_dlm_cache.h | 6 +++--- 3 files changed, 26 insertions(+), 16 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index ff15323695699f..b26e60d894e61b 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1137,7 +1137,8 @@ static void fuse_readahead(struct readahead_control *rac) static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) { - struct inode *inode = iocb->ki_filp->f_mapping->host; + struct file *file = iocb->ki_filp; + struct inode *inode = file->f_mapping->host; struct fuse_conn *fc = get_fuse_conn(inode); /* @@ -1153,6 +1154,12 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) return err; } + /* if we have dlm support acquire a read lock for the area + * we are reading from. */ + if (fc->writeback_cache && fc->dlm) + fuse_get_dlm_lock(file, iocb->ki_pos, + iov_iter_count(to), FUSE_PAGE_LOCK_READ); + return generic_file_read_iter(iocb, to); } @@ -1586,7 +1593,8 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) * get the performance benefits of 'parallel direct writes'. */ loff_t pos = file->f_flags & O_APPEND ? i_size_read(inode) + iocb->ki_pos : iocb->ki_pos; size_t length = iov_iter_count(from); - fuse_get_dlm_write_lock(file, pos, length); + fuse_get_dlm_lock(file, pos, length, + FUSE_PAGE_LOCK_WRITE); } /* @@ -2596,7 +2604,7 @@ static void fuse_vma_close(struct vm_area_struct *vma) /** * Request a DLM lock from the FUSE server. * - * This routine is similar to fuse_get_dlm_write_lock(), but it + * This routine is similar to fuse_get_dlm_lock(), but it * does not cache the DLM lock in the kernel. */ static int fuse_get_page_mkwrite_lock(struct file *file, loff_t offset, size_t length) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index d765dd8018cc6a..ea296a2e9ec89c 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -487,10 +487,14 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, } /** - * request a dlm lock from the fuse server + * fuse_get_dlm_lock - request a dlm lock from the fuse server + * @file: the file being accessed + * @offset: byte offset into the file (need not be page-aligned) + * @length: length of the region in bytes (need not be page-aligned) + * @mode: FUSE_PAGE_LOCK_READ or FUSE_PAGE_LOCK_WRITE */ -void fuse_get_dlm_write_lock(struct file *file, loff_t offset, - size_t length) +void fuse_get_dlm_lock(struct file *file, loff_t offset, + size_t length, enum fuse_page_lock_mode mode) { struct fuse_file *ff = file->private_data; struct inode *inode = file_inode(file); @@ -500,7 +504,7 @@ void fuse_get_dlm_write_lock(struct file *file, loff_t offset, uint64_t end = (offset + length - 1) | (PAGE_SIZE - 1); /* note that the offset and length don't have to be page aligned here - * but since we only get here on writeback caching we will send out + * but since we only get here on writeback caching we will send out * page aligned requests */ offset &= PAGE_MASK; @@ -513,8 +517,7 @@ void fuse_get_dlm_write_lock(struct file *file, loff_t offset, * at the same time. It is intentionally not protected * since a DLM implementation in the FUSE server should take care * of any races in lock requests */ - if (fuse_dlm_range_is_locked(fi, offset, - end, FUSE_PAGE_LOCK_WRITE)) + if (fuse_dlm_range_is_locked(fi, offset, end, mode)) return; /* we already have this area locked */ memset(&inarg, 0, sizeof(inarg)); @@ -522,7 +525,8 @@ void fuse_get_dlm_write_lock(struct file *file, loff_t offset, inarg.start = offset; inarg.end = end; - inarg.type = FUSE_DLM_LOCK_WRITE; + inarg.type = (mode == FUSE_PAGE_LOCK_WRITE) ? + FUSE_DLM_LOCK_WRITE : FUSE_DLM_LOCK_READ; args.opcode = FUSE_DLM_WB_LOCK; args.nodeid = get_node_id(inode); @@ -551,8 +555,6 @@ void fuse_get_dlm_write_lock(struct file *file, loff_t offset, return; } else { /* ignore any errors here, there is no way we can react appropriately */ - fuse_dlm_lock_range(fi, outarg.start, - outarg.end, - FUSE_PAGE_LOCK_WRITE); + fuse_dlm_lock_range(fi, outarg.start, outarg.end, mode); } } diff --git a/fs/fuse/fuse_dlm_cache.h b/fs/fuse/fuse_dlm_cache.h index 438d31d28b666e..5c3deaa3536866 100644 --- a/fs/fuse/fuse_dlm_cache.h +++ b/fs/fuse/fuse_dlm_cache.h @@ -43,8 +43,8 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, uint64_t start, bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, uint64_t end, enum fuse_page_lock_mode mode); -/* this is the interface to the filesystem */ -void fuse_get_dlm_write_lock(struct file *file, loff_t offset, - size_t length); +/* This is the interface to the filesystem */ +void fuse_get_dlm_lock(struct file *file, loff_t offset, + size_t length, enum fuse_page_lock_mode mode); #endif /* _FS_FUSE_DLM_CACHE_H */ From 4225b31f1c38ff337610572c346402ac6407ba74 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 13 Aug 2026 13:27:29 +0200 Subject: [PATCH 10/28] fuse: fence cached reads against NOTIFY invalidate with a percpu gate A FUSE_NOTIFY_INVAL_INODE is a coherency event: once the server signals a remote modify, no local read may return a page it has superseded. The invalidate ran unserialized against cache-serving reads, so a buffered read could hand back a stale folio it still held a reference to. Convert the per-inode wb_inval_rwsem to a percpu_rw_semaphore and take its read side around the cache-serving read as well as the existing buffered write. The read side is per-CPU, so it scales on a shared file; the NOTIFY takes the write side blocking, giving the invalidate priority -- it parks new readers, drains in-flight ones, then drops the cache. Every gated invalidate now runs under the write side, not just the storm-latching one. The gate is allocated only for writeback+dlm regular files and is NULL elsewhere (best-effort invalidate, as before). The blocking write side may run on the notify-delivering server thread, so it is safe only under a server that services request replies on other threads; redfs' dlm server provides that contract. Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 89 +++++++++++++++++++++++++--------- fs/fuse/fuse_i.h | 35 +++++++++----- fs/fuse/inode.c | 123 ++++++++++++++++++++++++++++++----------------- 3 files changed, 170 insertions(+), 77 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index b26e60d894e61b..b369d4d344215a 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1135,11 +1135,16 @@ static void fuse_readahead(struct readahead_control *rac) } } +static ssize_t fuse_direct_read_iter(struct kiocb *iocb, struct iov_iter *to); + static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) { struct file *file = iocb->ki_filp; struct inode *inode = file->f_mapping->host; struct fuse_conn *fc = get_fuse_conn(inode); + struct fuse_inode *fi = get_fuse_inode(inode); + struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; + ssize_t res; /* * In auto invalidate mode, always update attributes on read. @@ -1160,7 +1165,29 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) fuse_get_dlm_lock(file, iocb->ki_pos, iov_iter_count(to), FUSE_PAGE_LOCK_READ); - return generic_file_read_iter(iocb, to); + /* + * Fence the cache-serving read against a NOTIFY invalidate so we never + * hand back a folio the server has just superseded. The gate read side + * is per-CPU cheap; the NOTIFY holds the write side with priority. + * Re-check the forced-DIO latch under it: if a storm latched us while we + * waited on a pending writer, reroute to direct like the buffered write + * path, so we do not repopulate the cache the latch just dropped. + * wb_sem is NULL on non-writeback+dlm mounts (gate inactive). + */ + if (wb_sem) { + percpu_down_read(wb_sem); + if (fuse_inode_force_dio(inode)) { + percpu_up_read(wb_sem); + return fuse_direct_read_iter(iocb, to); + } + } + + res = generic_file_read_iter(iocb, to); + + if (wb_sem) + percpu_up_read(wb_sem); + + return res; } static void fuse_write_args_fill(struct fuse_io_args *ia, struct fuse_file *ff, @@ -1558,6 +1585,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) ssize_t err, count; struct fuse_conn *fc = get_fuse_conn(inode); struct fuse_inode *fi = get_fuse_inode(inode); + struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; bool wb_guard = false; bool exclusive = true; @@ -1600,25 +1628,25 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) /* * Open-code generic_file_write_iter() so that wb_inval_rwsem * can be held for read across the page-cache dirtying: a - * concurrent NOTIFY_INVAL_INODE -- which latches the inode - * under the write side of that lock via a non-blocking trylock - * -- must not be able to strand the folios we are about to - * dirty. Re-check the latch under it (it may have been set - * while we blocked on the inode lock) and re-route to the + * concurrent NOTIFY_INVAL_INODE -- which takes the write side + * of that gate (blocking, with priority) around its invalidate + * + latch set -- must not be able to strand the folios we are + * about to dirty. Re-check the latch under it (it may have been + * set while we blocked on the inode lock) and re-route to the * direct path if it is now set; the DLM write lock taken above * is harmless there, as the direct path does its own server - * coordination. The forced-direct-IO latch feature is only - * active under writeback+dlm, so the guard follows fc->dlm. + * coordination. wb_sem is NULL on mounts where the gate is + * inactive. */ exclusive = fuse_cache_wr_exclusive_lock(iocb, true); if (exclusive) inode_lock(inode); else inode_lock_shared(inode); - if (fc->dlm) { - down_read(&fi->wb_inval_rwsem); + if (wb_sem) { + percpu_down_read(wb_sem); if (fuse_inode_force_dio(inode)) { - up_read(&fi->wb_inval_rwsem); + percpu_up_read(wb_sem); fuse_cache_wr_unlock(inode, exclusive); return fuse_direct_write_iter(iocb, from); } @@ -1626,8 +1654,8 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) written = generic_write_checks(iocb, from); if (written > 0) written = __generic_file_write_iter(iocb, from); - if (fc->dlm) - up_read(&fi->wb_inval_rwsem); + if (wb_sem) + percpu_up_read(wb_sem); fuse_cache_wr_unlock(inode, exclusive); if (written > 0) written = generic_write_sync(iocb, written); @@ -1640,19 +1668,19 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) /* * The killpriv fallback lands here with the writeback cache still on, * so it populates the page cache too and needs the same guard as the - * writeback branch above: hold wb_inval_rwsem for read across the + * writeback branch above: hold the coherency gate for read across the * page-cache population and re-check the latch under it, so a * concurrent NOTIFY_INVAL_INODE cannot have the cache repopulated * behind the invalidate it just did. Taken before - * task_io_account_write() so a re-route is not double-counted. A - * connection without the writeback cache never latches, and has no - * gate to take. + * task_io_account_write() so a re-route is not double-counted. + * wb_sem is NULL on mounts where the gate is inactive, and such a + * connection never latches either. */ - wb_guard = fc->writeback_cache && fc->dlm; + wb_guard = !!wb_sem; if (wb_guard) { - down_read(&fi->wb_inval_rwsem); + percpu_down_read(wb_sem); if (fuse_inode_force_dio(inode)) { - up_read(&fi->wb_inval_rwsem); + percpu_up_read(wb_sem); inode_unlock(inode); return fuse_direct_write_iter(iocb, from); } @@ -1683,7 +1711,7 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) } out: if (wb_guard) - up_read(&fi->wb_inval_rwsem); + percpu_up_read(wb_sem); inode_unlock(inode); if (written > 0) written = generic_write_sync(iocb, written); @@ -3565,6 +3593,7 @@ static const struct address_space_operations fuse_file_aops = { void fuse_init_file_inode(struct inode *inode, unsigned int flags) { struct fuse_inode *fi = get_fuse_inode(inode); + struct fuse_conn *fc = get_fuse_conn(inode); inode->i_fop = &fuse_file_operations; inode->i_data.a_ops = &fuse_file_aops; @@ -3576,7 +3605,23 @@ void fuse_init_file_inode(struct inode *inode, unsigned int flags) fi->iocachectr = 0; init_waitqueue_head(&fi->page_waitq); init_waitqueue_head(&fi->direct_io_waitq); - init_rwsem(&fi->wb_inval_rwsem); + /* + * Coherency gate for the forced-direct-IO feature; only writeback+dlm + * regular files need it. A percpu_rw_semaphore embeds per-CPU state, + * so allocate it out of line and only when the mount can use it rather + * than paying it on every inode. On failure leave it NULL: the gate + * stays inactive (best-effort invalidate) and the inode is still usable. + */ + fi->wb_inval_rwsem = NULL; + if (fc->writeback_cache && fc->dlm) { + struct percpu_rw_semaphore *sem = kmalloc(sizeof(*sem), GFP_KERNEL); + + if (sem && percpu_init_rwsem(sem)) { + kfree(sem); + sem = NULL; + } + fi->wb_inval_rwsem = sem; + } fi->notify_stamp = jiffies; fi->notify_interval_ewma = FUSE_NOTIFY_EWMA_SEED << FUSE_NOTIFY_EWMA_SHIFT; diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index d2bac70fda7918..ef0a2be96ea426 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -189,19 +190,29 @@ struct fuse_inode { struct fuse_dlm_cache dlm_locked_areas; /* - * Serializes buffered-write page-cache dirtying against - * the forced-direct-IO latch transition driven by - * NOTIFY_INVAL_INODE (fuse_reverse_inval_inode()), which - * may be delivered by the same server thread that still - * owes a reply to an in-flight write holding the inode - * lock. The buffered writer holds this for read around - * the dirtying and re-checks the latch under it; the - * NOTIFY latch site takes it for write (trylock, never - * blocking) around its page-cache invalidate + latch set. - * Only regular files initialise it -- it shares storage - * with the readdir-cache union arm. + * Per-inode read/write coherency gate for the + * forced-direct-IO feature. Cache-serving buffered reads + * and buffered writes hold it for read; being a + * percpu_rw_semaphore the read side is per-CPU cheap and + * scales on a shared file. The NOTIFY invalidate + * (fuse_reverse_inval_inode()) holds it for write, which + * BLOCKS so the coherency notify has priority: it fences + * cache-serving reads (and buffered writes) out for the + * whole invalidate, so no folio a remote modify has + * superseded is ever handed back. + * + * The write side may run on the server thread delivering + * the notify, so a blocking writer is safe only under a + * server that services request replies on threads other + * than the one delivering the notify (see the NOTIFY site). + * + * Allocated out of line only for writeback+dlm regular + * files (it shares storage with the readdir-cache union + * arm); NULL on other mounts and on allocation failure, + * where the gate is inactive and the invalidate falls back + * to best-effort. */ - struct rw_semaphore wb_inval_rwsem; + struct percpu_rw_semaphore *wb_inval_rwsem; /* * Rate of FUSE_NOTIFY_INVAL_INODE data invalidations diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 7fa31fd2e75bfa..a4dee87d2bc56a 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -195,6 +195,23 @@ static void fuse_evict_inode(struct inode *inode) WARN_ON(!list_empty(&fi->queued_writes)); fuse_dlm_cache_release_locks(fi); } + + /* + * Free the coherency gate here rather than in ->free_inode: that runs + * from an RCU callback, where percpu_free_rwsem() may sleep in + * rcu_sync_dtor() if the write side has not fully quiesced. No user + * can remain by eviction time: gate readers hold a file reference and + * a concurrent notify holds an inode reference. wb_inval_rwsem lives + * in the regular-file union arm and is only ever allocated for regular + * files, so gate on S_ISREG (but not fuse_is_bad() -- bad-marked + * regular files still own a gate); a directory's overlapping + * readdir-cache fields must not be misread. + */ + if (S_ISREG(inode->i_mode) && fi->wb_inval_rwsem) { + percpu_free_rwsem(fi->wb_inval_rwsem); + kfree(fi->wb_inval_rwsem); + fi->wb_inval_rwsem = NULL; + } } static int fuse_reconfigure(struct fs_context *fsc) @@ -777,6 +794,7 @@ static bool fuse_notify_inval_hot(struct fuse_inode *fi) int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, loff_t offset, loff_t len) { + struct percpu_rw_semaphore *wb_sem = NULL; struct fuse_inode *fi; struct inode *inode; pgoff_t pg_start; @@ -825,67 +843,86 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, /* * A data invalidation means another (remote) entity is modifying - * the file. Keep a moving average of how fast these notifications - * arrive for the whole inode; when they come in a rapid stream -- - * a remote writer repeatedly invalidating the file -- and it is - * also open for writing here, latch the inode into direct IO: - * reads and writes are served direct (from the server) until the - * last writer closes or the inode is mmapped. A lone or occasional - * notify keeps the average high and does not trip the switch. Only - * under writeback+dlm, where the buffered write's RMW read is - * skipped so its wb_inval_rwsem read-side section is free of server - * round-trips. + * the file. Two things happen here: + * + * 1. Coherency. Drop the affected page-cache range so no local + * read returns a folio the remote modify has superseded. This + * runs under the write side of the per-inode coherency gate + * (wb_inval_rwsem), which fences cache-serving buffered reads + * and buffered writes out for the whole invalidate. Unlike the + * old best-effort trylock this BLOCKS -- the notify has + * priority: percpu_down_write() parks new gate readers, drains + * in-flight ones, then invalidates. A blocking writer here is + * safe only under a server that services request replies on + * threads other than the one delivering this notify: the write + * side waits for gate readers to drain, and a cache-miss read + * holds the read side across its FUSE_READ round-trip. redfs' + * dlm server provides that contract; a server that cannot must + * not enable writeback+dlm. * - * fuse_notify_inval_hot() updates the average under fi->lock and is - * called for every data invalidation so it stays current even while - * no local writer is open. When it trips, take wb_inval_rwsem for - * write so the buffered write path -- which holds it for read across - * its dirtying and re-checks the latch under it -- cannot strand - * dirty folios after the cache is dropped. Use a trylock and never - * block: this may run on the server thread that still owes an - * in-flight write (holding the inode lock) its reply, so blocking on - * the rwsem or the inode lock would deadlock. If the writer has - * gone, skip the latch this round (best effort); the invalidate - * still runs. Only regular files initialise the average and the - * rwsem (they share storage with the readdir-cache union arm), so - * gate on S_ISREG. When latched, drop the whole mapping rather than - * just the notified range, or dirty folios outside it would be - * invisible to the forced direct reads (stale read / lost write). + * 2. Latch. Keep a moving average (fuse_notify_inval_hot(), under + * fi->lock, updated for every data invalidation) of how fast + * these arrive; when they come in a rapid stream -- a remote + * writer repeatedly invalidating -- and the inode is also open + * for writing here, latch it into direct IO until the last + * writer closes or it is mmapped. When latched, drop the whole + * mapping rather than just the notified range, or dirty folios + * outside it would be invisible to the forced direct reads + * (stale read / lost write). + * + * The gate (and the average) exist only for writeback+dlm regular + * files, and not while mmapped; elsewhere wb_sem is NULL and the + * invalidate runs unserialized (best-effort), as before. */ - if (S_ISREG(inode->i_mode) && fc->writeback_cache && fc->dlm && - !FUSE_IS_DAX(inode) && !fuse_inode_backing(fi) && - !mapping_mapped(inode->i_mapping)) { - bool hot, has_writer; + if (S_ISREG(inode->i_mode) && fc->writeback_cache && + fc->dlm && !FUSE_IS_DAX(inode) && + !fuse_inode_backing(fi) && + !mapping_mapped(inode->i_mapping)) + wb_sem = fi->wb_inval_rwsem; + + if (wb_sem) { + bool hot, has_writer, latched = false; spin_lock(&fi->lock); hot = fuse_notify_inval_hot(fi); has_writer = !list_empty(&fi->write_files); spin_unlock(&fi->lock); - if (hot && has_writer && !fuse_inode_force_dio(inode) && - down_write_trylock(&fi->wb_inval_rwsem)) { - bool latched = false; + /* + * Priority write side: park new gate readers, + * drain in-flight ones, then invalidate. Blocks + * (unlike the old trylock) -- see the contract in + * the comment above. + */ + percpu_down_write(wb_sem); + if (hot && has_writer && + !fuse_inode_force_dio(inode)) { spin_lock(&fi->lock); if (!list_empty(&fi->write_files)) { set_bit(FUSE_I_FORCE_DIO, &fi->state); latched = true; } spin_unlock(&fi->lock); + } - if (latched) { - pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", - nodeid); - invalidate_inode_pages2(inode->i_mapping); - } else { - invalidate_inode_pages2_range(inode->i_mapping, - pg_start, pg_end); - } - up_write(&fi->wb_inval_rwsem); - } else { + /* + * Latched: drop the whole mapping (dirty folios + * outside the notified range would be invisible to + * the forced direct reads). Otherwise just the + * notified range. + */ + if (fuse_inode_force_dio(inode)) + invalidate_inode_pages2(inode->i_mapping); + else invalidate_inode_pages2_range(inode->i_mapping, pg_start, pg_end); - } + + percpu_up_write(wb_sem); + + if (latched) + pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", + nodeid); } else { invalidate_inode_pages2_range(inode->i_mapping, pg_start, pg_end); From 7896594c93e563af2e09ffabe97c5266b5fc1002 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Fri, 17 Jul 2026 15:51:24 +0200 Subject: [PATCH 11/28] fuse: satisfy DLM read lock requests from a held write lock The buffered read path now acquires a DLM read lock via fuse_get_dlm_lock(..., FUSE_PAGE_LOCK_READ). Before sending the request to the server, fuse_get_dlm_lock() calls fuse_dlm_range_is_locked() to skip regions we already hold. That coverage check compared the held lock mode for exact equality (range->mode != lock_mode), so a range we already hold with an exclusive WRITE lock was reported as not-locked for a READ request. Because fuse_dlm_lock_range() intentionally does not downgrade a WRITE lock on a read, the region stays WRITE-locked and every subsequent read re-requests a DLM read lock from the server. This made read-after-write and re-read workloads flood the server with redundant FUSE_DLM_WB_LOCK requests, never converging. A held WRITE lock (exclusive) subsumes a READ lock. Treat a range as uncovered only when the held mode is strictly weaker than the requested mode (range->mode < lock_mode). READ requests are now satisfied by either a READ or a WRITE lock, while WRITE requests still require an existing WRITE lock (upgrade otherwise), matching the compatibility rules already documented in fuse_dlm_lock_range(). Signed-off-by: Horst Birthelmer --- fs/fuse/fuse_dlm_cache.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index ea296a2e9ec89c..40eda6daf75cae 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -454,9 +454,16 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, /* Check if the entire range is covered */ while (range && current_start <= end) { - /* If we're checking for a specific mode, verify it matches */ - if (lock_mode && range->mode != lock_mode) { - /* Wrong lock mode */ + /* + * The held lock must be at least as strong as the one + * requested. A WRITE lock (exclusive) satisfies a READ + * request, so only treat the range as uncovered when the + * held mode is weaker than what we ask for. This avoids + * re-requesting a READ lock for a range we already hold + * a WRITE lock on (e.g. read-after-write). + */ + if (lock_mode && range->mode < lock_mode) { + /* Held lock is weaker than requested */ up_read(&cache->lock); return false; } From f2ba2c6865535ea7fb7ae7d9dc56fabcd4793bba Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 13 Aug 2026 13:30:30 +0200 Subject: [PATCH 12/28] fuse: zero-fill writes past the server EOF instead of sending READs A partial-folio buffered write reads the folio back from the server first. On the DLM writeback path that read is often pointless: a shared-lock writer commits i_size just behind its own cursor (see fuse_write_end()), so folios can sit inside i_size in ranges no writer has reached yet, and every one of them costs a FUSE_READ for a range that cannot hold data. Against a file opened write-only the server may even fail that read outright. Track fi->server_size, an upper bound on how far the server holds data: seeded from server-reported attributes, advanced when the server acknowledges data (writeback completion and fuse_write_update_attr()), and lowered on truncate. In fuse_write_begin(), when the folio starts at or past that bound and the page-granular DLM write lock covers it, zero-fill the folio locally and skip the READ. The three conditions together are what makes this safe: - fi->server_size bounds data materialized on the server; it is advanced in fuse_writepage_end() before the pages leave writeback, i.e. before they can go clean and be reclaimed, so a reclaimed range the server holds data in can never be zero-filled; - local data not yet acknowledged sits in uptodate folios, which fuse_write_begin() has already returned before reaching here; - the DLM write lock excludes data written by other nodes, and it is re-checked against the live lock tree so a revoked lock falls back to reading. Signed-off-by: Horst Birthelmer --- fs/fuse/dir.c | 10 ++++++++ fs/fuse/file.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/fuse/fuse_i.h | 13 ++++++++++ fs/fuse/inode.c | 14 +++++++++++ 4 files changed, 99 insertions(+) diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c index 3ec99cee3ef62d..71dc10c0d1167e 100644 --- a/fs/fuse/dir.c +++ b/fs/fuse/dir.c @@ -2083,7 +2083,10 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, */ if (fc->dlm && fc->writeback_cache) fuse_dlm_cache_release_locks(fi); + spin_lock(&fi->lock); + fi->server_size = 0; i_size_write(inode, 0); + spin_unlock(&fi->lock); truncate_pagecache(inode, 0); goto out; } @@ -2175,6 +2178,13 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, /* see the comment in fuse_change_attributes() */ if (!is_wb || is_truncate) i_size_write(inode, outarg.attr.size); + /* + * A truncate settles the size on the server; only shrink the + * server-materialized bound: growing just exposes zeros, which the + * bound need not cover (see fuse_write_begin()). + */ + if (is_truncate && (loff_t) outarg.attr.size < fi->server_size) + fi->server_size = outarg.attr.size; if (is_truncate) { /* NOTE: this may release/reacquire fi->lock */ diff --git a/fs/fuse/file.c b/fs/fuse/file.c index b369d4d344215a..addc029436dd16 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -333,6 +333,7 @@ static void fuse_truncate_update_attr(struct inode *inode, struct file *file) spin_lock(&fi->lock); fi->attr_version = atomic64_inc_return(&fc->attr_version); + fi->server_size = 0; i_size_write(inode, 0); spin_unlock(&fi->lock); file_update_time(file); @@ -1259,6 +1260,15 @@ bool fuse_write_update_attr(struct inode *inode, loff_t pos, ssize_t written) spin_lock(&fi->lock); fi->attr_version = atomic64_inc_return(&fc->attr_version); + if (written > 0 && S_ISREG(inode->i_mode)) { + /* + * The server acknowledged data up to @pos, keep the + * server-materialized bound in sync for the expansion + * zero-fill in fuse_write_begin(). + */ + if (pos > fi->server_size) + fi->server_size = pos; + } if (written > 0 && pos > inode->i_size) { i_size_write(inode, pos); ret = true; @@ -2164,6 +2174,20 @@ static void fuse_writepage_end(struct fuse_mount *fm, struct fuse_args *args, if (!fc->writeback_cache) fuse_invalidate_attr_mask(inode, FUSE_STATX_MODIFY); spin_lock(&fi->lock); + if (!error) { + struct fuse_write_in *inarg = &wpa->ia.write.in; + + /* + * The server acknowledged this writeback, so data up to the + * end of the request is materialized on the server. Advance + * the bound before the pages end writeback below, i.e. before + * they can go clean and be reclaimed, so that + * fuse_write_begin() can never zero-fill a reclaimed range + * the server holds data in. + */ + if ((loff_t) (inarg->offset + inarg->size) > fi->server_size) + fi->server_size = inarg->offset + inarg->size; + } fi->writectr--; fuse_writepage_finish(wpa); spin_unlock(&fi->lock); @@ -2525,6 +2549,43 @@ static int fuse_write_begin(struct file *file, struct address_space *mapping, folio_zero_segment(folio, 0, off); goto success; } + + /* + * The folio is inside i_size but may still sit in a range the server + * holds no data for: a shared-lock writer extends i_size past regions + * it has not written yet (see fuse_write_end()), and every such folio + * would otherwise be read back from the server although it cannot + * contain data. Zero-fill locally instead when the server is known to + * hold nothing in the range and we hold the DLM write lock covering + * it: + * + * - fi->server_size bounds the data materialized on the server + * (writeback and direct write acknowledgements, server + * attributes), + * - local data not yet acknowledged sits in uptodate folios, which + * are already handled above, + * - the page-granular DLM write lock excludes data written by other + * nodes, re-checked against the live lock tree so a revoked lock + * falls back to reading. + */ + if (fc->dlm) { + struct fuse_inode *fi = get_fuse_inode(mapping->host); + loff_t fpos = folio_pos(folio); + size_t fsz = folio_size(folio); + bool hole; + + spin_lock(&fi->lock); + hole = fpos >= fi->server_size; + spin_unlock(&fi->lock); + + if (hole && fuse_dlm_range_is_locked(fi, fpos, fpos + fsz - 1, + FUSE_PAGE_LOCK_WRITE)) { + folio_zero_range(folio, 0, fsz); + folio_mark_uptodate(folio); + goto success; + } + } + err = fuse_do_readpage(file, &folio->page); if (err) goto cleanup; @@ -3603,6 +3664,7 @@ void fuse_init_file_inode(struct inode *inode, unsigned int flags) fuse_dlm_cache_init(fi); fi->writectr = 0; fi->iocachectr = 0; + fi->server_size = 0; init_waitqueue_head(&fi->page_waitq); init_waitqueue_head(&fi->direct_io_waitq); /* diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index ef0a2be96ea426..26678ceecff108 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -189,6 +189,19 @@ struct fuse_inode { /* dlm locked areas we have sent lock requests for */ struct fuse_dlm_cache dlm_locked_areas; + /* + * Server-materialized size: an upper bound for how far + * the server holds file data. Seeded from + * server-reported attributes, advanced when the server + * acknowledges data (writeback completion, + * fuse_write_update_attr()), lowered again on + * truncate. A read-modify-write of a folio starting + * at or past this bound needs no READ request under a + * held DLM write lock: the server has no data there + * (see fuse_write_begin()). Protected by fi->lock. + */ + loff_t server_size; + /* * Per-inode read/write coherency gate for the * forced-direct-IO feature. Cache-serving buffered reads diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index a4dee87d2bc56a..5b6d79b2bdbea4 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -496,8 +496,10 @@ static void fuse_change_attributes_i(struct inode *inode, struct fuse_attr *attr loff_t oldsize; struct timespec64 old_mtime; bool have_size = !sx || (sx->mask & STATX_SIZE); + u64 srv_size; spin_lock(&fi->lock); + srv_size = attr->size; /* * In case of writeback_cache enabled, writes update mtime, ctime and @@ -523,6 +525,18 @@ static void fuse_change_attributes_i(struct inode *inode, struct fuse_attr *attr return; } + /* + * srv_size is the size the server reported before the writeback + * cache_mask above replaced attr->size with the local value. It + * bounds how far the server can hold data, letting the buffered write + * path zero-fill expansion read-modify-writes instead of sending READ + * requests, see fuse_write_begin(). Only ever grow it here: stale + * attributes were rejected above and truncation lowers it directly. + */ + if (have_size && S_ISREG(inode->i_mode) && + (loff_t) srv_size > fi->server_size) + fi->server_size = srv_size; + old_mtime = inode_get_mtime(inode); fuse_change_attributes_common(inode, attr, sx, attr_valid, cache_mask, evict_ctr); From 5ce4f970d61e66ae4743bbaa0a7c34e3272a6da7 Mon Sep 17 00:00:00 2001 From: Hai Zhong Zhou Date: Wed, 29 Jul 2026 06:38:18 +0000 Subject: [PATCH 13/28] fuse: only refresh size on cached write when opened with O_APPEND In fuse_cache_write_iter, writeback_cache mode was always refreshing STATX_SIZE along with STATX_MODE before a buffered write. The size refresh is only needed for the O_APPEND path, where the kernel must know the current EOF before extending the file. For ordinary writes, fetching size is unnecessary work and can race with concurrent writes and then impact writeback performance. Keep refreshing STATX_MODE in all cases so SUID clearing still sees an up-to-date mode. Request STATX_SIZE only when the file is opened with O_APPEND. Signed-off-by Hai Zhong Zhou --- fs/fuse/file.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index addc029436dd16..4150e6768938a0 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1610,9 +1610,12 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) return fuse_direct_write_iter(iocb, from); if (fc->writeback_cache) { - /* Update size (EOF optimization) and mode (SUID clearing) */ - err = fuse_update_attributes(mapping->host, file, - STATX_SIZE | STATX_MODE); + /* Update mode for SUID clearing, and also update size if the file + * is opened with O_APPEND mode. + */ + u32 request_mask = (file->f_flags & O_APPEND) ? + (STATX_SIZE | STATX_MODE) : STATX_MODE; + err = fuse_update_attributes(mapping->host, file, request_mask); if (err) return err; From 0f7f4199f4bc2d64eaefb20d738ddef873e89395 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Sat, 25 Jul 2026 12:45:45 +0200 Subject: [PATCH 14/28] fuse: seed DLM grant merging with an interval-tree lookup fuse_dlm_try_merge() locates the first merge candidate by walking from rb_first_cached() until it reaches the region just granted. The walk runs under the write-held cache rwsem on every fuse_dlm_lock_range() call, and the tree it walks holds every cached grant of the inode. Strided writers (IOR hard-write) accumulate grants that cannot merge with each other, so the tree keeps growing and every new grant pays a scan of all grants below it -- quadratic over the run, with fuse_dlm_range_is_locked() readers blocked behind each scan. Seed the merge with fuse_page_it_iter_first() on the region widened by one unit to each side instead; finding the lowest overlapping range is what the interval tree is there for. This also repairs two edge cases of the linear scan: a region starting at offset 0 made 'start - 1' wrap so the scan degenerated and merging was silently skipped, and a region ending at U64_MAX overflowed 'end + 1' in the loop bound, ending the merge after the first range. Both bounds now saturate. Signed-off-by: Horst Birthelmer --- fs/fuse/fuse_dlm_cache.c | 57 ++++++++++++++++++++++++++-------------- 1 file changed, 37 insertions(+), 20 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index 40eda6daf75cae..2ec072b86312f4 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -120,26 +120,25 @@ static void fuse_dlm_try_merge(struct fuse_dlm_cache *cache, uint64_t start, uint64_t end) { struct fuse_dlm_range *range, *next; - struct rb_node *node; + uint64_t first = start ? start - 1 : start; + uint64_t last = end < U64_MAX ? end + 1 : end; if (!cache) return; - /* Find the first range that might need merging */ - range = NULL; - node = rb_first_cached(&cache->ranges); - while (node) { - range = rb_entry(node, struct fuse_dlm_range, rb); - if (range->end >= start - 1) - break; - node = rb_next(node); - } - - if (!range || range->start > end + 1) - return; + /* + * Find the first range that might need merging. Directly adjacent + * ranges can merge, hence the region is widened by one unit to each + * side (saturating at the type bounds). This must stay an + * interval-tree lookup: the tree holds every cached grant of the + * inode and strided writers grow it for the lifetime of the file, + * so seeding the merge by walking from the tree minimum would make + * every new grant cost a full scan. + */ + range = fuse_page_it_iter_first(&cache->ranges, first, last); /* Try to merge ranges in and around the specified region */ - while (range && range->start <= end + 1) { + while (range && range->start <= last) { /* Get next range before we potentially modify the tree */ next = NULL; if (rb_next(&range->rb)) { @@ -150,11 +149,11 @@ static void fuse_dlm_try_merge(struct fuse_dlm_cache *cache, uint64_t start, /* Try to merge with next range if adjacent and same mode */ if (next && range->mode == next->mode && range->end + 1 == next->start) { - /* Merge ranges */ - range->end = next->end; - - /* Remove next from tree */ + /* Merge ranges: re-insert so __subtree_end is updated */ fuse_page_it_remove(next, &cache->ranges); + fuse_page_it_remove(range, &cache->ranges); + range->end = next->end; + fuse_page_it_insert(range, &cache->ranges); kfree(next); /* Continue with the same range */ @@ -188,6 +187,7 @@ int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, struct fuse_dlm_cache *cache = &inode->dlm_locked_areas; struct fuse_dlm_range *range, *new_range, *next; int lock_mode; + bool covered_to_end = false; int ret = 0; LIST_HEAD(to_lock); LIST_HEAD(to_upgrade); @@ -233,14 +233,17 @@ int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, } /* Move current_start past this range */ - current_start = max(current_start, range->end + 1); + if (range->end >= end) + covered_to_end = true; + else + current_start = max(current_start, range->end + 1); /* Move to next range */ range = next; } /* If there's a gap after the last range to the end, extend the range */ - if (current_start <= end) { + if (!covered_to_end && current_start <= end) { new_range = kmalloc(sizeof(*new_range), GFP_KERNEL); if (!new_range) { ret = -ENOMEM; @@ -322,13 +325,17 @@ static int fuse_dlm_punch_hole(struct fuse_dlm_cache *cache, uint64_t start, /* If the hole is at the beginning of the range */ if (start == range->start) { + fuse_page_it_remove(range, &cache->ranges); range->start = end + 1; + fuse_page_it_insert(range, &cache->ranges); goto out; } /* If the hole is at the end of the range */ if (end == range->end) { + fuse_page_it_remove(range, &cache->ranges); range->end = start - 1; + fuse_page_it_insert(range, &cache->ranges); goto out; } @@ -400,10 +407,14 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, break; } else if (start > range->start) { /* Adjust the end of the range */ + fuse_page_it_remove(range, &cache->ranges); range->end = start - 1; + fuse_page_it_insert(range, &cache->ranges); } else if (end < range->end) { /* Adjust the start of the range */ + fuse_page_it_remove(range, &cache->ranges); range->start = end + 1; + fuse_page_it_insert(range, &cache->ranges); } else { /* Complete overlap, remove the range */ fuse_page_it_remove(range, &cache->ranges); @@ -475,6 +486,12 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, return false; } + /* Covered through the end of the requested range? */ + if (range->end >= end) { + up_read(&cache->lock); + return true; + } + /* Move current_start past this range */ current_start = range->end + 1; From c48c1d153cdafbba21343a3343d8fab896b8908c Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 13 Aug 2026 13:35:48 +0200 Subject: [PATCH 15/28] fuse: re-validate the DLM grant after waiting on the coherency gate Both cached IO paths request their DLM lock first and then go to sleep on things a NOTIFY invalidate can be holding: the read path blocks on the coherency gate (writer priority), the write path additionally sleeps on a contended i_rwsem. A NOTIFY invalidate running in that window revokes exactly the lock just granted (fuse_dlm_unlock_range()), so the task wakes up and populates or dirties the page cache with no DLM coverage. Close the window without ever sending a FUSE_DLM_WB_LOCK request while holding the gate (a grant that had to wait on an invalidate delivered to this same client would deadlock against our own gate hold): - Drop the lock record under the gate write side in fuse_reverse_inval_inode(), so revocation and page drop are one atomic step with respect to the gate. - After entering the gate read side, re-check the grant against the live lock tree; if it was revoked while we waited, drop the gate, re-request, re-enter and check again. With the revoke now gated, passing the check means the lock cannot go away for the whole gate hold: a revoke arriving mid-operation parks until the IO is done. - Keep the write path's lock request ahead of the inode lock: the round trip must not capture the writer-priority i_rwsem for unbounded cluster-grant latency, and the in-gate re-validation already closes the grant-to-use window. Only O_APPEND moves below the lock, because its range is the current EOF -- stable only under the exclusive inode lock. This also fixes the append range itself: generic_write_checks() rewrites ki_pos to i_size for IOCB_APPEND, so the old 'i_size + ki_pos' double-counted (ki_pos is absolute, not relative) and locked a range disjoint from where the data lands. fuse_get_dlm_lock() now reports whether the grant is recorded, and the re-validation never re-requests a grant that failed, so it cannot spin (the read path seeds this from its pre-gate request instead of discarding that result). A grant the server issued but that could not be recorded (small-allocation -ENOMEM) reports FUSE_DLM_GRANT_UNRECORDED: coverage exists cluster-wide, so failing the IO would be wrong -- it proceeds, it just cannot re-validate. Empty ranges are trivially held, so a zero-length IO neither sends a doomed request nor spins in the retry loops. The write path returns a real failure to the caller instead of dirtying the cache without DLM coverage; only -ENOSYS still degrades to a plain cached write, since it means the server has no DLM at all and clears fc->dlm. The read path keeps falling through unlocked and additionally bounds its retry: a reader-only inode has no force-DIO latch to end a revoke storm, so after a few re-requests the read is served unlocked rather than looping in the kernel for the duration of the storm. Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 147 ++++++++++++++++++++++++++++++++++----- fs/fuse/fuse_dlm_cache.c | 102 ++++++++++++++++++++------- fs/fuse/fuse_dlm_cache.h | 17 ++++- fs/fuse/inode.c | 35 +++++++--- 4 files changed, 244 insertions(+), 57 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 4150e6768938a0..9ef19445fe4ae1 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1138,6 +1138,12 @@ static void fuse_readahead(struct readahead_control *rac) static ssize_t fuse_direct_read_iter(struct kiocb *iocb, struct iov_iter *to); +/* + * Bound on re-requesting a revoked DLM grant before a cached read is + * served unlocked; see fuse_cache_read_iter(). + */ +#define FUSE_DLM_READ_RETRIES 3 + static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) { struct file *file = iocb->ki_filp; @@ -1146,6 +1152,7 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) struct fuse_inode *fi = get_fuse_inode(inode); struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; ssize_t res; + int lock_err = 0; /* * In auto invalidate mode, always update attributes on read. @@ -1163,8 +1170,9 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) /* if we have dlm support acquire a read lock for the area * we are reading from. */ if (fc->writeback_cache && fc->dlm) - fuse_get_dlm_lock(file, iocb->ki_pos, - iov_iter_count(to), FUSE_PAGE_LOCK_READ); + lock_err = fuse_get_dlm_lock(file, iocb->ki_pos, + iov_iter_count(to), + FUSE_PAGE_LOCK_READ); /* * Fence the cache-serving read against a NOTIFY invalidate so we never @@ -1176,11 +1184,44 @@ static ssize_t fuse_cache_read_iter(struct kiocb *iocb, struct iov_iter *to) * wb_sem is NULL on non-writeback+dlm mounts (gate inactive). */ if (wb_sem) { + int tries = FUSE_DLM_READ_RETRIES; + +retry: percpu_down_read(wb_sem); if (fuse_inode_force_dio(inode)) { percpu_up_read(wb_sem); return fuse_direct_read_iter(iocb, to); } + /* + * The DLM lock was requested before entering the gate, and + * the NOTIFY invalidate we may just have waited on revokes + * locks under the gate write side. Re-check the grant here + * and re-request with the gate dropped, so a + * FUSE_DLM_WB_LOCK round trip never parks a pending + * invalidate behind our own gate hold. Once the check + * passes the lock cannot go away for the rest of the gate + * hold. A failed or unrecorded request falls through + * unlocked, as before: the retry is taken even then (the + * latch must be re-checked under the re-entered gate), so + * lock_err has to stay sticky across it -- seeded by the + * pre-gate request above -- or a grant that failed would + * be re-requested forever. The retry is also bounded: a + * remote writer can revoke each successful grant before + * the gate is re-entered, and a reader-only inode has no + * force-DIO latch to end such a storm, so after + * FUSE_DLM_READ_RETRIES re-requests the read is served + * unlocked rather than looping without bound. + */ + if (!lock_err && fc->dlm && tries-- > 0 && + !fuse_dlm_lock_is_held(fi, iocb->ki_pos, + iov_iter_count(to), + FUSE_PAGE_LOCK_READ)) { + percpu_up_read(wb_sem); + lock_err = fuse_get_dlm_lock(file, iocb->ki_pos, + iov_iter_count(to), + FUSE_PAGE_LOCK_READ); + goto retry; + } } res = generic_file_read_iter(iocb, to); @@ -1585,6 +1626,26 @@ static void fuse_cache_wr_unlock(struct inode *inode, bool exclusive) inode_unlock_shared(inode); } +/* + * Request the DLM write lock covering a cached write. -ENOSYS cleared + * fc->dlm: the server has no DLM, proceed as a plain cached write. Any + * other failure means the cache would be dirtied without DLM coverage - + * the caller must fail the write instead. A granted-but-unrecorded + * lock (positive return) is covered cluster-wide; proceed, but flag it + * so the in-gate re-validation skips a check an invisible grant could + * never pass. + */ +static int fuse_cache_wr_dlm_lock(struct file *file, loff_t pos, size_t len, + bool *unrecorded) +{ + int err = fuse_get_dlm_lock(file, pos, len, FUSE_PAGE_LOCK_WRITE); + + if (err < 0 && err != -ENOSYS) + return err; + *unrecorded = err > 0; + return 0; +} + static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) { struct file *file = iocb->ki_filp; @@ -1598,6 +1659,9 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; bool wb_guard = false; bool exclusive = true; + bool dlm_unrecorded = false; + loff_t dlm_pos = 0; + size_t dlm_len = 0; /* * The inode may have been latched into forced direct IO -- by a @@ -1625,22 +1689,31 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) goto writethrough; } - /* if we have dlm support acquire the lock for the area - * we are writing into */ - if (fc->dlm) { - /* note that a file opened with O_APPEND will have relative values - * in ki_pos. This code is here for convenience and for libfuse overlay test. - * Filesystems should handle O_APPEND with 'direct io' to additionally - * get the performance benefits of 'parallel direct writes'. */ - loff_t pos = file->f_flags & O_APPEND ? i_size_read(inode) + iocb->ki_pos : iocb->ki_pos; - size_t length = iov_iter_count(from); - fuse_get_dlm_lock(file, pos, length, - FUSE_PAGE_LOCK_WRITE); + exclusive = fuse_cache_wr_exclusive_lock(iocb, true); + + /* + * Request the DLM write lock before taking i_rwsem: the request + * is an unbounded cluster round trip, and holding the + * writer-priority rwsem across it would park a truncate -- and + * behind it every later writer -- for the duration. The + * grant-to-use window this leaves open is closed by the in-gate + * re-validation below. Only the append case must wait for the + * lock: its range depends on i_size, which is stable only under + * the exclusive inode lock. + */ + if (fc->dlm && !(iocb->ki_flags & IOCB_APPEND)) { + dlm_pos = iocb->ki_pos; + dlm_len = iov_iter_count(from); + + err = fuse_cache_wr_dlm_lock(file, dlm_pos, dlm_len, + &dlm_unrecorded); + if (err) + return err; } /* - * Open-code generic_file_write_iter() so that wb_inval_rwsem - * can be held for read across the page-cache dirtying: a + * Open-code generic_file_write_iter() so that the coherency + * gate can be held for read across the page-cache dirtying: a * concurrent NOTIFY_INVAL_INODE -- which takes the write side * of that gate (blocking, with priority) around its invalidate * + latch set -- must not be able to strand the folios we are @@ -1651,28 +1724,66 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) * coordination. wb_sem is NULL on mounts where the gate is * inactive. */ - exclusive = fuse_cache_wr_exclusive_lock(iocb, true); if (exclusive) inode_lock(inode); else inode_lock_shared(inode); + + /* note that this small code dup will save us a lot of headache later + * when appends are done concurrently without using parallel direct writes */ + if (fc->dlm && (iocb->ki_flags & IOCB_APPEND)) { + /* + * An append write lands at the current EOF no matter + * what ki_pos holds: generic_write_checks() rewrites + * ki_pos to i_size for IOCB_APPEND, and i_size is + * stable here because append writes hold the inode lock + * exclusive. Lock where the data will land. + */ + dlm_pos = i_size_read(inode); + dlm_len = iov_iter_count(from); + + err = fuse_cache_wr_dlm_lock(file, dlm_pos, dlm_len, + &dlm_unrecorded); + if (err) + goto wb_out; + } + if (wb_sem) { + wb_guard = true; +retry: percpu_down_read(wb_sem); if (fuse_inode_force_dio(inode)) { percpu_up_read(wb_sem); fuse_cache_wr_unlock(inode, exclusive); return fuse_direct_write_iter(iocb, from); } + if (fc->dlm && !dlm_unrecorded && + !fuse_dlm_lock_is_held(fi, dlm_pos, dlm_len, + FUSE_PAGE_LOCK_WRITE)) { + percpu_up_read(wb_sem); + err = fuse_cache_wr_dlm_lock(file, dlm_pos, + dlm_len, + &dlm_unrecorded); + if (err) { + /* The gate is already dropped; funnel + * the failure through the one audited + * exit. */ + wb_guard = false; + goto wb_out; + } + goto retry; + } } written = generic_write_checks(iocb, from); if (written > 0) written = __generic_file_write_iter(iocb, from); - if (wb_sem) +wb_out: + if (wb_guard) percpu_up_read(wb_sem); fuse_cache_wr_unlock(inode, exclusive); if (written > 0) written = generic_write_sync(iocb, written); - return written; + return written ? written : err; } writethrough: diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index 2ec072b86312f4..30d371bcf6e293 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -510,45 +510,83 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, return true; } +/** + * fuse_dlm_lock_is_held - check that a byte range is covered by a granted lock + * @fi: the fuse inode + * @offset: byte offset into the file (need not be page-aligned) + * @length: length of the region in bytes (need not be page-aligned) + * @mode: FUSE_PAGE_LOCK_READ or FUSE_PAGE_LOCK_WRITE + * + * Re-validation helper for fuse_get_dlm_lock() callers: checks the same + * page-aligned range a fuse_get_dlm_lock() call with these arguments + * requests, against the live lock tree. + */ +bool fuse_dlm_lock_is_held(struct fuse_inode *fi, loff_t offset, + size_t length, enum fuse_page_lock_mode mode) +{ + uint64_t end = (offset + length - 1) | (PAGE_SIZE - 1); + + /* + * An empty range needs no coverage. Reporting it held keeps the + * re-validating IO paths from re-requesting a lock the tree can + * never show (the page-aligned end would invert below). + */ + if (!length) + return true; + + return fuse_dlm_range_is_locked(fi, offset & PAGE_MASK, end, mode); +} + /** * fuse_get_dlm_lock - request a dlm lock from the fuse server * @file: the file being accessed * @offset: byte offset into the file (need not be page-aligned) * @length: length of the region in bytes (need not be page-aligned) * @mode: FUSE_PAGE_LOCK_READ or FUSE_PAGE_LOCK_WRITE + * + * Return: 0 when the range is covered by a recorded grant on return, + * FUSE_DLM_GRANT_UNRECORDED when the server granted the lock but + * recording it failed (covered cluster-wide, invisible to + * fuse_dlm_lock_is_held()), a negative error code otherwise. Callers + * re-validating the grant must not re-request on a nonzero return or + * they would spin. */ -void fuse_get_dlm_lock(struct file *file, loff_t offset, - size_t length, enum fuse_page_lock_mode mode) +int fuse_get_dlm_lock(struct file *file, loff_t offset, + size_t length, enum fuse_page_lock_mode mode) { struct fuse_file *ff = file->private_data; struct inode *inode = file_inode(file); struct fuse_conn *fc = get_fuse_conn(inode); struct fuse_inode *fi = get_fuse_inode(inode); struct fuse_mount *fm = ff->fm; - uint64_t end = (offset + length - 1) | (PAGE_SIZE - 1); - - /* note that the offset and length don't have to be page aligned here - * but since we only get here on writeback caching we will send out - * page aligned requests */ - offset &= PAGE_MASK; FUSE_ARGS(args); struct fuse_dlm_lock_in inarg; struct fuse_dlm_lock_out outarg; int err; + /* An empty range needs no lock. */ + if (!length) + return 0; + /* note that this can be run from different processes * at the same time. It is intentionally not protected * since a DLM implementation in the FUSE server should take care - * of any races in lock requests */ - if (fuse_dlm_range_is_locked(fi, offset, end, mode)) - return; /* we already have this area locked */ + * of any races in lock requests. + * The early exit uses the same helper the callers re-validate + * with, so this check and a later fuse_dlm_lock_is_held() can + * never disagree about what counts as covered. */ + if (fuse_dlm_lock_is_held(fi, offset, length, mode)) + return 0; /* we already have this area locked */ memset(&inarg, 0, sizeof(inarg)); inarg.fh = ff->fh; - inarg.start = offset; - inarg.end = end; + /* note that the offset and length don't have to be page aligned + * here but since we only get here on writeback caching we will + * send out page aligned requests */ + inarg.start = offset & PAGE_MASK; + inarg.end = (offset + length - 1) | (PAGE_SIZE - 1); inarg.type = (mode == FUSE_PAGE_LOCK_WRITE) ? FUSE_DLM_LOCK_WRITE : FUSE_DLM_LOCK_READ; @@ -564,21 +602,31 @@ void fuse_get_dlm_lock(struct file *file, loff_t offset, if (err == -ENOSYS) { /* fuse server does not support dlm, save the info */ fc->dlm = 0; - return; + return err; } if (err) - return; - else - if (inarg.start < outarg.start || - inarg.end > outarg.end) { - /* fuse server is seriously broken */ - pr_warn("fuse: dlm lock request for %llu:%llu returned %llu:%llu bytes\n", - inarg.start, inarg.end, outarg.start, outarg.end); - fuse_abort_conn(fc); - return; - } else { - /* ignore any errors here, there is no way we can react appropriately */ - fuse_dlm_lock_range(fi, outarg.start, outarg.end, mode); - } + return err; + + if (inarg.start < outarg.start || inarg.end > outarg.end) { + /* fuse server is seriously broken */ + pr_warn("fuse: dlm lock request for %llu:%llu returned %llu:%llu bytes\n", + inarg.start, inarg.end, outarg.start, outarg.end); + fuse_abort_conn(fc); + return -EIO; + } + + /* + * The server granted the lock; record it so + * fuse_dlm_lock_is_held() sees it. A failure to record + * (small-allocation -ENOMEM) does not undo the grant: coverage + * exists cluster-wide, only the local bookkeeping is missing. + * Report that as FUSE_DLM_GRANT_UNRECORDED so callers neither + * fail an IO that is actually covered nor keep re-requesting a + * grant that will not become visible. + */ + if (fuse_dlm_lock_range(fi, outarg.start, outarg.end, mode)) + return FUSE_DLM_GRANT_UNRECORDED; + + return 0; } diff --git a/fs/fuse/fuse_dlm_cache.h b/fs/fuse/fuse_dlm_cache.h index 5c3deaa3536866..b0b16c56e3b0b0 100644 --- a/fs/fuse/fuse_dlm_cache.h +++ b/fs/fuse/fuse_dlm_cache.h @@ -17,6 +17,15 @@ struct fuse_inode; /* Lock modes for page ranges */ enum fuse_page_lock_mode { FUSE_PAGE_LOCK_READ, FUSE_PAGE_LOCK_WRITE }; +/* + * fuse_get_dlm_lock() result: the server granted the lock but recording + * it locally failed, leaving the grant invisible to + * fuse_dlm_lock_is_held(). The IO is covered cluster-wide; the caller + * must proceed without re-validating (a re-request would spin) instead + * of failing the IO. + */ +#define FUSE_DLM_GRANT_UNRECORDED 1 + /* Page cache lock manager */ struct fuse_dlm_cache { /* Lock protecting the tree */ @@ -43,8 +52,12 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, uint64_t start, bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, uint64_t end, enum fuse_page_lock_mode mode); +/* Re-validate a fuse_get_dlm_lock() grant against the live lock tree */ +bool fuse_dlm_lock_is_held(struct fuse_inode *inode, loff_t offset, + size_t length, enum fuse_page_lock_mode mode); + /* This is the interface to the filesystem */ -void fuse_get_dlm_lock(struct file *file, loff_t offset, - size_t length, enum fuse_page_lock_mode mode); +int fuse_get_dlm_lock(struct file *file, loff_t offset, + size_t length, enum fuse_page_lock_mode mode); #endif /* _FS_FUSE_DLM_CACHE_H */ diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 5b6d79b2bdbea4..95d46323917529 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -845,16 +845,6 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, else pg_end = (offset + len - 1) >> PAGE_SHIFT; - if (fc->dlm && fc->writeback_cache) - /* Invalidate the range exactly as the fuse server requested - * except for the case where it sends -1. - * Note that this can lead to some inconsistencies if - * the fuse server sends unaligned data */ - fuse_dlm_unlock_range(fi, - offset, - pg_end == -1 ? 0 : - (offset + len - 1)); - /* * A data invalidation means another (remote) entity is modifying * the file. Two things happen here: @@ -910,6 +900,23 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, */ percpu_down_write(wb_sem); + /* + * Revoke the DLM lock range under the gate write + * side, atomically with the page drop: gate readers + * re-validate their grant right after entering, and + * a grant that passed that check must stay visible + * for their whole gate hold. + * The range is exactly what the fuse server + * requested except for the case where it sends -1. + * Note that this can lead to some inconsistencies + * if the fuse server sends unaligned data. + */ + if (fc->dlm && fc->writeback_cache) + fuse_dlm_unlock_range(fi, + offset, + pg_end == -1 ? 0 : + (offset + len - 1)); + if (hot && has_writer && !fuse_inode_force_dio(inode)) { spin_lock(&fi->lock); @@ -938,6 +945,14 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", nodeid); } else { + /* No gate on this inode (mmapped, DAX, backing or + * non-regular): drop the lock range unserialized, + * as before. */ + if (fc->dlm && fc->writeback_cache) + fuse_dlm_unlock_range(fi, + offset, + pg_end == -1 ? 0 : + (offset + len - 1)); invalidate_inode_pages2_range(inode->i_mapping, pg_start, pg_end); } From 42a13e9d7b8730e3207ada59190f324506544423 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 13 Aug 2026 13:36:45 +0200 Subject: [PATCH 16/28] fuse: fix the DLM revoke range of an inode invalidate The NOTIFY_INVAL_INODE revoke computed fuse_dlm_unlock_range(fi, offset, pg_end == -1 ? 0 : offset + len - 1) which is wrong at both degenerate ends: a to-EOF invalidate (len <= 0, e.g. a remote truncate) with offset > 0 becomes the inverted range [offset, 0] and removes nothing, so the revoked grant stays visible to the re-validating IO paths forever -- cached writes with no server-side lock, zero-filled RMW reads; and an invalidate of byte 0 (offset 0, len 1) becomes [0, 0], the "destroy everything" sentinel, wiping every grant of the inode. Map the range in one helper shared by the gated and the ungated branch: to-EOF revokes through U64_MAX, and the bounds widen to page boundaries to match how grants are recorded -- revoking too much only costs a re-request, too little leaves a stale grant. Drop the in-band (0, 0) sentinel: whole-file invalidates walk the normal removal path, release-all is fuse_dlm_cache_release_locks(), and an inverted range is rejected with -EINVAL instead of silently ignored. Signed-off-by: Horst Birthelmer --- fs/fuse/fuse_dlm_cache.c | 15 +++++++-------- fs/fuse/inode.c | 33 +++++++++++++++++++++------------ 2 files changed, 28 insertions(+), 20 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index 30d371bcf6e293..4714d48e6bc9b1 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -369,8 +369,12 @@ static int fuse_dlm_punch_hole(struct fuse_dlm_cache *cache, uint64_t start, * @start: Start page offset * @end: End page offset * - * Release locks on the specified range of pages. - * Note that if start and end are set to zero the cache is destroyed. + * Release locks on the specified range of pages. An inverted range is + * rejected rather than silently removing nothing: the callers revoke + * coverage, and a revoke that quietly keeps the grant alive would let + * the re-validating IO paths trust a lock the server has taken away. + * To drop every grant use fuse_dlm_cache_release_locks() (there is no + * in-band sentinel range for it). * * Return: 0 on success, negative error code on failure */ @@ -381,14 +385,9 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, struct fuse_dlm_range *range, *next; int ret = 0; - if (!cache) + if (!cache || start > end) return -EINVAL; - if (start == 0 && end == 0) { - fuse_dlm_cache_release_locks(inode); - return 0; - } - down_write(&cache->lock); /* Find all ranges that overlap with [start, end] */ diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 95d46323917529..27f39c06c7b84a 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -805,6 +805,25 @@ static bool fuse_notify_inval_hot(struct fuse_inode *fi) return avg < FUSE_NOTIFY_DIO_INTERVAL; } +/* + * Revoke the DLM grants backing an invalidated byte range. Grants are + * recorded page-aligned, so widen the revoke to page boundaries: dropping + * more than the server invalidated only costs a re-request, dropping less + * would leave a stale grant that fuse_dlm_lock_is_held() keeps trusting. + * len <= 0 means "invalidate to EOF" (see fuse_notify_inval_inode()) and + * revokes through U64_MAX -- it must not become an inverted range, which + * fuse_dlm_unlock_range() rejects without removing anything. + */ +static void fuse_dlm_revoke_inval_range(struct fuse_inode *fi, loff_t offset, + loff_t len) +{ + uint64_t start = (uint64_t)offset & PAGE_MASK; + uint64_t end = len <= 0 ? U64_MAX : + (((uint64_t)offset + len - 1) | (PAGE_SIZE - 1)); + + fuse_dlm_unlock_range(fi, start, end); +} + int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, loff_t offset, loff_t len) { @@ -906,16 +925,9 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, * re-validate their grant right after entering, and * a grant that passed that check must stay visible * for their whole gate hold. - * The range is exactly what the fuse server - * requested except for the case where it sends -1. - * Note that this can lead to some inconsistencies - * if the fuse server sends unaligned data. */ if (fc->dlm && fc->writeback_cache) - fuse_dlm_unlock_range(fi, - offset, - pg_end == -1 ? 0 : - (offset + len - 1)); + fuse_dlm_revoke_inval_range(fi, offset, len); if (hot && has_writer && !fuse_inode_force_dio(inode)) { @@ -949,10 +961,7 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, * non-regular): drop the lock range unserialized, * as before. */ if (fc->dlm && fc->writeback_cache) - fuse_dlm_unlock_range(fi, - offset, - pg_end == -1 ? 0 : - (offset + len - 1)); + fuse_dlm_revoke_inval_range(fi, offset, len); invalidate_inode_pages2_range(inode->i_mapping, pg_start, pg_end); } From 953ed785b2426d75dbb643a0b8049c64f0938058 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Tue, 28 Jul 2026 09:45:16 +0200 Subject: [PATCH 17/28] fuse: fence mmapped invalidates and local truncates with the coherency gate Two revocation paths bypassed the revoke-under-gate invariant the IO paths re-validate against: - fuse_reverse_inval_inode() skipped the gate once mapping_mapped() turned true, revoking concurrently with gate holders -- but fuse_cache_read_iter()/fuse_cache_write_iter() enter the gate unconditionally, so a single mmap() reopened the race. Keep the gate for mmapped inodes; only the force-DIO latch stays disabled for them (a mapping needs the page cache, and fuse_file_mmap() reverts any latch it races with). - The local truncates in fuse_do_setattr() -- the atomic-O_TRUNC open shortcut and the after-setattr trim -- revoked and dropped the cache with no gate at all, so an already re-validated reader could repopulate the truncated range. Take the gate write side around revoke + drop. This cannot deadlock: both run under exclusive i_rwsem, which no gate holder waits on (the write path takes i_rwsem before the gate, the read path never takes it). Signed-off-by: Horst Birthelmer --- fs/fuse/dir.c | 29 +++++++++++++++++++++++++++++ fs/fuse/inode.c | 19 ++++++++++++------- 2 files changed, 41 insertions(+), 7 deletions(-) diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c index 71dc10c0d1167e..69510a0947eecf 100644 --- a/fs/fuse/dir.c +++ b/fs/fuse/dir.c @@ -2076,11 +2076,26 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, WARN_ON(!(attr->ia_valid & ATTR_SIZE)); WARN_ON(attr->ia_size != 0); if (fc->atomic_o_trunc) { + struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; + /* * No need to send request to userspace, since actual * truncation has already been done by OPEN. But still * need to truncate page cache. + * + * Revoke and drop under the coherency gate write side, + * like the NOTIFY invalidate path: a gate reader that + * already re-validated its grant must not have the + * lock tree and the cache yanked mid-hold, or it + * would repopulate the truncated range trusting a + * grant that no longer exists. Waiting for gate + * readers here is safe: we hold i_rwsem exclusive, so + * no gate holder can be waiting on it (the write path + * takes i_rwsem before the gate, the read path never + * takes it). */ + if (wb_sem) + percpu_down_write(wb_sem); if (fc->dlm && fc->writeback_cache) fuse_dlm_cache_release_locks(fi); spin_lock(&fi->lock); @@ -2088,6 +2103,8 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, i_size_write(inode, 0); spin_unlock(&fi->lock); truncate_pagecache(inode, 0); + if (wb_sem) + percpu_up_write(wb_sem); goto out; } file = NULL; @@ -2198,11 +2215,23 @@ int fuse_do_setattr(struct mnt_idmap *idmap, struct dentry *dentry, */ if ((is_truncate || !is_wb) && S_ISREG(inode->i_mode) && oldsize != outarg.attr.size) { + struct percpu_rw_semaphore *wb_sem = fi->wb_inval_rwsem; + + /* + * Revoke and drop under the coherency gate write side; see + * the atomic-O_TRUNC branch above. i_rwsem is held + * exclusive here as well (setattr), so waiting out gate + * readers cannot deadlock. + */ + if (wb_sem) + percpu_down_write(wb_sem); if (fc->dlm && fc->writeback_cache) fuse_dlm_unlock_range(fi, outarg.attr.size & PAGE_MASK, -1); truncate_pagecache(inode, outarg.attr.size); invalidate_inode_pages2(mapping); + if (wb_sem) + percpu_up_write(wb_sem); } clear_bit(FUSE_I_SIZE_UNSTABLE, &fi->state); diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 27f39c06c7b84a..28a2bfadbfbb4a 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -894,13 +894,17 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, * (stale read / lost write). * * The gate (and the average) exist only for writeback+dlm regular - * files, and not while mmapped; elsewhere wb_sem is NULL and the - * invalidate runs unserialized (best-effort), as before. + * files; elsewhere wb_sem is NULL and the invalidate runs + * unserialized (best-effort), as before. An mmapped inode + * keeps the gate -- fuse_cache_read_iter() and + * fuse_cache_write_iter() enter it unconditionally and rely + * on the revoke staying fenced -- but is never latched: + * a mapping needs the page cache, and fuse_file_mmap() + * reverts any latch it races with. */ if (S_ISREG(inode->i_mode) && fc->writeback_cache && fc->dlm && !FUSE_IS_DAX(inode) && - !fuse_inode_backing(fi) && - !mapping_mapped(inode->i_mapping)) + !fuse_inode_backing(fi)) wb_sem = fi->wb_inval_rwsem; if (wb_sem) { @@ -930,6 +934,7 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, fuse_dlm_revoke_inval_range(fi, offset, len); if (hot && has_writer && + !mapping_mapped(inode->i_mapping) && !fuse_inode_force_dio(inode)) { spin_lock(&fi->lock); if (!list_empty(&fi->write_files)) { @@ -957,9 +962,9 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, pr_info_ratelimited("FUSE: inode %llu latched to direct IO on invalidation notify storm\n", nodeid); } else { - /* No gate on this inode (mmapped, DAX, backing or - * non-regular): drop the lock range unserialized, - * as before. */ + /* No gate on this inode (DAX, backing, non-regular, + * or the gate allocation failed): drop the lock + * range unserialized (best-effort), as before. */ if (fc->dlm && fc->writeback_cache) fuse_dlm_revoke_inval_range(fi, offset, len); invalidate_inode_pages2_range(inode->i_mapping, From 669269ebc72f8f6441bff7258e9926c8e2a8952f Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Tue, 28 Jul 2026 09:47:47 +0200 Subject: [PATCH 18/28] fuse: order grant recording against concurrent revokes A FUSE_DLM_WB_LOCK reply and a NOTIFY invalidate are serviced on different threads, so a revoke aimed at the grant a reply carries can be processed before fuse_get_dlm_lock() records it: the revoke finds nothing to remove, and the requester then records an already-dead grant that no later NOTIFY will target -- a permanent false positive for the re-validating IO paths. Add a revocation generation to the lock cache, bumped under the cache lock by every revoke path -- unconditionally, because the racing revoke sees an empty overlap precisely when the grant is in flight. fuse_get_dlm_lock() samples it before sending and records through fuse_dlm_lock_range_gen(), which refuses with -EAGAIN once the generation has moved; the grant is then re-requested instead of recorded, bounded so a revoke storm cannot pin the IO here (past the bound the failure reports like any request failure). Signed-off-by: Horst Birthelmer --- fs/fuse/fuse_dlm_cache.c | 110 +++++++++++++++++++++++++++++++++++---- fs/fuse/fuse_dlm_cache.h | 15 ++++++ 2 files changed, 115 insertions(+), 10 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index 4714d48e6bc9b1..960d51e7836a3c 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -31,6 +31,12 @@ struct fuse_dlm_range { #define FUSE_PCACHE_LK_READ 1 /* Shared read lock */ #define FUSE_PCACHE_LK_WRITE 2 /* Exclusive write lock */ +/* + * Bound on re-requesting a grant whose recording lost against a + * concurrent revoke; see fuse_get_dlm_lock(). + */ +#define FUSE_DLM_RECORD_TRIES 3 + /* Interval tree definitions for page ranges */ static inline uint64_t fuse_dlm_range_start(struct fuse_dlm_range *range) { @@ -63,6 +69,7 @@ int fuse_dlm_cache_init(struct fuse_inode *inode) init_rwsem(&cache->lock); cache->ranges = RB_ROOT_CACHED; + cache->revoke_gen = 0; return 0; } @@ -84,6 +91,7 @@ void fuse_dlm_cache_release_locks(struct fuse_inode *inode) /* Release all locks */ down_write(&cache->lock); + WRITE_ONCE(cache->revoke_gen, cache->revoke_gen + 1); while ((node = rb_first_cached(&cache->ranges)) != NULL) { range = rb_entry(node, struct fuse_dlm_range, rb); fuse_page_it_remove(range, &cache->ranges); @@ -166,11 +174,13 @@ static void fuse_dlm_try_merge(struct fuse_dlm_cache *cache, uint64_t start, } /** - * fuse_dlm_lock_range - Lock a range of pages + * __fuse_dlm_lock_range - Lock a range of pages * @cache: The page cache * @start: Start page offset * @end: End page offset * @mode: Lock mode (read or write) + * @genp: If non-NULL, the revocation generation sampled before the grant + * was requested; recording fails with -EAGAIN if it has moved * * Add a locked range on the specified range of pages. * If parts of the range are already locked, only add the remaining parts. @@ -181,8 +191,9 @@ static void fuse_dlm_try_merge(struct fuse_dlm_cache *cache, uint64_t start, * * Return: 0 on success, negative error code on failure */ -int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, - uint64_t end, enum fuse_page_lock_mode mode) +static int __fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, + uint64_t end, enum fuse_page_lock_mode mode, + const uint64_t *genp) { struct fuse_dlm_cache *cache = &inode->dlm_locked_areas; struct fuse_dlm_range *range, *new_range, *next; @@ -202,6 +213,17 @@ int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, down_write(&cache->lock); + /* + * A revoke was processed after @genp was sampled; the grant this + * record carries may be the very one it targeted (a revoke of a + * not-yet-recorded grant removes nothing and would never be + * retried). Refuse, the caller re-requests. + */ + if (genp && cache->revoke_gen != *genp) { + up_write(&cache->lock); + return -EAGAIN; + } + /* Find all ranges that overlap with [start, end] */ range = fuse_page_it_iter_first(&cache->ranges, start, end); while (range) { @@ -297,6 +319,35 @@ int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, return ret; } +int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, + uint64_t end, enum fuse_page_lock_mode mode) +{ + return __fuse_dlm_lock_range(inode, start, end, mode, NULL); +} + +int fuse_dlm_lock_range_gen(struct fuse_inode *inode, uint64_t start, + uint64_t end, enum fuse_page_lock_mode mode, + uint64_t gen) +{ + return __fuse_dlm_lock_range(inode, start, end, mode, &gen); +} + +/** + * fuse_dlm_revoke_gen - sample the revocation generation + * @inode: the fuse inode + * + * Sampled before a FUSE_DLM_WB_LOCK request leaves the client. The + * reply and a NOTIFY revoke can be serviced on different threads, so a + * revoke may be processed between the reply arriving and its grant + * being recorded. fuse_dlm_lock_range_gen() re-checks the generation + * under the cache lock and refuses to record a grant such a revoke may + * have already killed. + */ +uint64_t fuse_dlm_revoke_gen(struct fuse_inode *inode) +{ + return READ_ONCE(inode->dlm_locked_areas.revoke_gen); +} + /** * fuse_dlm_punch_hole - Punch a hole in a locked range * @cache: The page cache @@ -390,6 +441,14 @@ int fuse_dlm_unlock_range(struct fuse_inode *inode, down_write(&cache->lock); + /* + * Unconditional, even when nothing overlaps: the revoke racing + * with an in-flight grant finds an empty tree precisely because + * the grant is not recorded yet, and the bump is what makes the + * recording side notice (see fuse_dlm_lock_range_gen()). + */ + WRITE_ONCE(cache->revoke_gen, cache->revoke_gen + 1); + /* Find all ranges that overlap with [start, end] */ range = fuse_page_it_iter_first(&cache->ranges, start, end); while (range) { @@ -562,12 +621,15 @@ int fuse_get_dlm_lock(struct file *file, loff_t offset, FUSE_ARGS(args); struct fuse_dlm_lock_in inarg; struct fuse_dlm_lock_out outarg; + uint64_t gen; + int tries = FUSE_DLM_RECORD_TRIES; int err; /* An empty range needs no lock. */ if (!length) return 0; +restart: /* note that this can be run from different processes * at the same time. It is intentionally not protected * since a DLM implementation in the FUSE server should take care @@ -578,6 +640,16 @@ int fuse_get_dlm_lock(struct file *file, loff_t offset, if (fuse_dlm_lock_is_held(fi, offset, length, mode)) return 0; /* we already have this area locked */ + /* + * Sample the revocation generation before the request leaves. + * The reply and a NOTIFY revoke are serviced on different + * threads, so a revoke aimed at the grant this request returns + * can be processed before the grant is recorded below -- + * recording it anyway would resurrect a dead grant that no later + * NOTIFY will ever remove. + */ + gen = fuse_dlm_revoke_gen(fi); + memset(&inarg, 0, sizeof(inarg)); inarg.fh = ff->fh; @@ -617,14 +689,32 @@ int fuse_get_dlm_lock(struct file *file, loff_t offset, /* * The server granted the lock; record it so - * fuse_dlm_lock_is_held() sees it. A failure to record - * (small-allocation -ENOMEM) does not undo the grant: coverage - * exists cluster-wide, only the local bookkeeping is missing. - * Report that as FUSE_DLM_GRANT_UNRECORDED so callers neither - * fail an IO that is actually covered nor keep re-requesting a - * grant that will not become visible. + * fuse_dlm_lock_is_held() sees it. */ - if (fuse_dlm_lock_range(fi, outarg.start, outarg.end, mode)) + err = fuse_dlm_lock_range_gen(fi, outarg.start, outarg.end, mode, gen); + if (err == -EAGAIN) { + /* + * A revoke was processed while the request was in flight; + * the grant may already be dead, so re-request instead of + * recording it. Bounded: a revoke storm must not pin the + * IO here -- past the bound the failure is reported like + * any other request failure (the write path fails the + * write, the read path serves unlocked). + */ + if (--tries) + goto restart; + return -EAGAIN; + } + + /* + * A failure to record (small-allocation -ENOMEM) does not undo + * the grant: coverage exists cluster-wide, only the local + * bookkeeping is missing. Report that as + * FUSE_DLM_GRANT_UNRECORDED so callers neither fail an IO that + * is actually covered nor keep re-requesting a grant that will + * not become visible. + */ + if (err) return FUSE_DLM_GRANT_UNRECORDED; return 0; diff --git a/fs/fuse/fuse_dlm_cache.h b/fs/fuse/fuse_dlm_cache.h index b0b16c56e3b0b0..647a8c37c36095 100644 --- a/fs/fuse/fuse_dlm_cache.h +++ b/fs/fuse/fuse_dlm_cache.h @@ -32,6 +32,13 @@ struct fuse_dlm_cache { struct rw_semaphore lock; /* Interval tree of locked ranges */ struct rb_root_cached ranges; + /* + * Bumped under @lock by every revocation + * (fuse_dlm_unlock_range(), fuse_dlm_cache_release_locks()); + * lets fuse_get_dlm_lock() order recording a reply's grant + * against revokes processed while the reply was in flight. + */ + uint64_t revoke_gen; }; /* Initialize a page cache lock manager */ @@ -44,6 +51,14 @@ void fuse_dlm_cache_release_locks(struct fuse_inode *inode); int fuse_dlm_lock_range(struct fuse_inode *inode, uint64_t start, uint64_t end, enum fuse_page_lock_mode mode); +/* As above, but refuse (-EAGAIN) if a revoke ran since @gen was sampled */ +int fuse_dlm_lock_range_gen(struct fuse_inode *inode, uint64_t start, + uint64_t end, enum fuse_page_lock_mode mode, + uint64_t gen); + +/* Sample the revocation generation (see fuse_dlm_lock_range_gen()) */ +uint64_t fuse_dlm_revoke_gen(struct fuse_inode *inode); + /* Unlock a range of pages */ int fuse_dlm_unlock_range(struct fuse_inode *inode, uint64_t start, uint64_t end); From 205e1d7401c515dbb8199ca370be2c8253c18d7c Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Thu, 30 Jul 2026 14:22:16 +0200 Subject: [PATCH 19/28] fuse: complete pinned-header sends in ring task context On the pinned-header send path fuse_uring_dispatch_ent() calls io_uring_cmd_done() directly from the request submitter's context with IO_URING_F_UNLOCKED. Every fuse-uring command is marked cancelable, so io_uring_cmd_del_cancelable() has to take ctx->uring_lock from that foreign task on every request. The ring task holds this mutex for the whole ->uring_cmd() issue path (io_uring_enter() submission), where it also wakes the submitter of the request it just committed - before releasing the lock. The freshly woken submitter usually preempts the ring task on the same CPU, and its next dispatch then blocks on the very mutex its victim still holds. The preempted owner is merely runnable and gets no wakeup boost, so under CPU pressure this convoy costs milliseconds per request while the daemon's actual work is a few microseconds. Keep the copies into the pinned pages in the submitter's context - that is the point of the pinning - but defer the command completion to ring task task-work, like the non-pinned path already does. There io_uring_cmd_del_cancelable() runs under the task-work batch's already held uring_lock, and the submitter no longer touches ctx->uring_lock at all on the fast path (only the rare copy-failure fallback still can). Signed-off-by: Horst Birthelmer --- fs/fuse/dev_uring.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c index 6b2de8b3e4ec8e..1d372cede3ee69 100644 --- a/fs/fuse/dev_uring.c +++ b/fs/fuse/dev_uring.c @@ -1461,6 +1461,26 @@ static void fuse_uring_send_in_task(struct io_uring_cmd *cmd, fuse_uring_send(ent, cmd, err, issue_flags); } +/* + * The request was already copied to the ring buffer in the submitter's + * context, only the io_uring cmd completion is left to do. + * io_uring_cmd_done() must not run in the submitter's context as it would + * have to take ctx->uring_lock (io_uring_cmd_del_cancelable()) - a mutex + * the ring task holds across its whole submission path and frequently gets + * preempted under while the just-woken submitter runs. + */ +static void fuse_uring_send_prepared_in_task(struct io_uring_cmd *cmd, + unsigned int issue_flags) +{ + struct fuse_ring_ent *ent = uring_cmd_to_ring_ent(cmd); + int err = 0; + + if (unlikely(issue_flags & IO_URING_F_TASK_DEAD)) + err = -ECANCELED; + + fuse_uring_send(ent, cmd, err, issue_flags); +} + static struct fuse_ring_queue *fuse_uring_select_queue(struct fuse_ring *ring, bool background) { @@ -1560,7 +1580,9 @@ static void fuse_uring_dispatch_ent(struct fuse_ring_ent *ent, bool bg) IO_URING_F_UNLOCKED); return; } - fuse_uring_send(ent, cmd, 0, IO_URING_F_UNLOCKED); + uring_cmd_set_ring_ent(cmd, ent); + io_uring_cmd_complete_in_task(cmd, + fuse_uring_send_prepared_in_task); } } From 19c898c9dce7ffa26353a6922d71970ef4b990f6 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Wed, 12 Aug 2026 12:08:30 +0200 Subject: [PATCH 20/28] fuse: retry grant recording until it wins against revokes fuse_get_dlm_lock() re-requests the DLM lock when fuse_dlm_lock_range_gen() returns -EAGAIN, i.e. a revoke was processed while the grant request was in flight and the grant it returned may already be dead. That restart was bounded by FUSE_DLM_RECORD_TRIES, and past the bound the function returned -EAGAIN. Reporting that as a request failure is wrong: no one else holds the range at that point, the caller simply lost a race with a revoke, and the write path turns the error into a failed write. Retry unconditionally instead. Every pass issues a fresh FUSE_DLM_WB_LOCK round trip to the server, so a revoke storm throttles the loop rather than spinning it, and the loop ends as soon as one grant survives long enough to be recorded. Drop the now-unused bound and its counter. Signed-off-by: Horst Birthelmer --- fs/fuse/fuse_dlm_cache.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index 960d51e7836a3c..b0b17cbd3c3f7a 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -31,12 +31,6 @@ struct fuse_dlm_range { #define FUSE_PCACHE_LK_READ 1 /* Shared read lock */ #define FUSE_PCACHE_LK_WRITE 2 /* Exclusive write lock */ -/* - * Bound on re-requesting a grant whose recording lost against a - * concurrent revoke; see fuse_get_dlm_lock(). - */ -#define FUSE_DLM_RECORD_TRIES 3 - /* Interval tree definitions for page ranges */ static inline uint64_t fuse_dlm_range_start(struct fuse_dlm_range *range) { @@ -622,7 +616,6 @@ int fuse_get_dlm_lock(struct file *file, loff_t offset, struct fuse_dlm_lock_in inarg; struct fuse_dlm_lock_out outarg; uint64_t gen; - int tries = FUSE_DLM_RECORD_TRIES; int err; /* An empty range needs no lock. */ @@ -696,14 +689,14 @@ int fuse_get_dlm_lock(struct file *file, loff_t offset, /* * A revoke was processed while the request was in flight; * the grant may already be dead, so re-request instead of - * recording it. Bounded: a revoke storm must not pin the - * IO here -- past the bound the failure is reported like - * any other request failure (the write path fails the - * write, the read path serves unlocked). + * recording it. Retry until a grant survives long enough to + * be recorded: giving up here would hand the caller an error + * for a range no one else holds, and the write path turns + * that into a failed write. Each pass makes a fresh server + * round trip, so a revoke storm throttles this loop rather + * than spinning it. */ - if (--tries) - goto restart; - return -EAGAIN; + goto restart; } /* From 3aa1aad768c11c833d98917c0b093317f28637f5 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Wed, 12 Aug 2026 16:59:59 +0200 Subject: [PATCH 21/28] fuse: gate the notify-driven direct-IO latch behind a module parameter fuse_reverse_inval_inode() latches an inode into direct IO when a remote writer keeps invalidating a file that is also open for writing here. That trades the writeback cache away for as long as the latch holds, which only pays off on workloads that actually see such invalidation storms. Make it opt-in through a new 'enable_notify_dio' module parameter, default off. FUSE_I_FORCE_DIO is set in exactly one place, so gating that single site is enough: every other reference only tests or clears the bit, and with the bit never set those paths behave as they did before the latch existed. The moving average is still folded on every invalidation while the parameter is off, so enabling it at runtime takes effect on the next storm instead of after a warm-up. Clearing it stops new latches but leaves already-latched inodes to run out on the usual exits (last writer closes, or mmap). Signed-off-by: Horst Birthelmer --- fs/fuse/inode.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 28a2bfadbfbb4a..fc5e0b852120bb 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -36,6 +36,19 @@ static bool __read_mostly enable_compound; module_param(enable_compound, bool, 0644); MODULE_PARM_DESC(enable_uring, "Enable fuse compounds"); +/* + * Gate for the notify-driven direct-IO latch (see + * fuse_reverse_inval_inode()): when a remote writer keeps invalidating a + * file that is also open for writing here, the inode is switched to + * direct IO until its last writer closes. Off by default -- it trades + * the writeback cache away for the duration, which only pays off on + * workloads that actually see such storms. + */ +static bool __read_mostly enable_notify_dio; +module_param(enable_notify_dio, bool, 0644); +MODULE_PARM_DESC(enable_notify_dio, + "Latch a contended inode to direct IO on an invalidation notify storm"); + static struct kmem_cache *fuse_inode_cachep; struct list_head fuse_conn_list; DEFINE_MUTEX(fuse_mutex); @@ -891,7 +904,13 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, * writer closes or it is mmapped. When latched, drop the whole * mapping rather than just the notified range, or dirty folios * outside it would be invisible to the forced direct reads - * (stale read / lost write). + * (stale read / lost write). Latching is opt-in via the + * enable_notify_dio module parameter and off by default; the + * average is kept up to date either way, so enabling it at + * runtime takes effect on the next storm rather than after a + * warm-up. Clearing it at runtime stops new latches but lets + * already-latched inodes run out on the usual exits (last + * writer closes, or mmap). * * The gate (and the average) exist only for writeback+dlm regular * files; elsewhere wb_sem is NULL and the invalidate runs @@ -933,7 +952,7 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, if (fc->dlm && fc->writeback_cache) fuse_dlm_revoke_inval_range(fi, offset, len); - if (hot && has_writer && + if (enable_notify_dio && hot && has_writer && !mapping_mapped(inode->i_mapping) && !fuse_inode_force_dio(inode)) { spin_lock(&fi->lock); From 0a699238ee69bb60c1a70c8bbe93b5331305e58c Mon Sep 17 00:00:00 2001 From: Allison Henderson Date: Thu, 13 Aug 2026 13:40:52 +0200 Subject: [PATCH 22/28] fuse: re-decide the write lock mode after the DLM probe fuse_cache_write_iter() picks between the exclusive and the relaxed shared inode lock with fuse_cache_wr_exclusive_lock(), which returns "shared" only when fc->dlm is set. Since "fuse: re-validate the DLM grant after waiting on the coherency gate" that decision is made before the DLM write lock is requested, and the request itself can clear fc->dlm: a server that does not implement FUSE_DLM_WB_LOCK answers -ENOSYS, which fuse_get_dlm_lock() handles by clearing fc->dlm and which fuse_cache_wr_dlm_lock() reports as success. The write then proceeds in a state that was unreachable before: the shared lock was chosen believing DLM was active, but DLM is now known to be absent. The relaxed lock has no other justification -- the DLM is what excludes writers on disjoint ranges cluster-wide -- so without it concurrent buffered writers run with no serialisation at all. Re-evaluate the lock mode after the request, while no lock is held yet, so a server without DLM support gets the exclusive path back. Upstream this was found as generic/105, 123, 215, 246, 378, 423, 519 and 597 all failing with EBADF on the first write to a newly created file, and bisected to the commit named above. That specific failure was an iomap effect (a shared-lock write claims the i_size extension up front, which suppresses iomap's beyond-EOF zeroing while its fc->dlm-gated replacement also stops running, so an expanding write sends a READ past EOF). It cannot occur here -- fuse_write_end() grows i_size behind the cursor, so fuse_write_begin() keeps zeroing wholly-past-EOF folios itself -- but the unsound lock mode it exposed is the same, and is what this fixes. Signed-off-by: Allison Henderson Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 9ef19445fe4ae1..c1803ea3cc0bee 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1709,6 +1709,18 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) &dlm_unrecorded); if (err) return err; + + /* + * The request above may have found that the server has + * no DLM at all, in which case it cleared fc->dlm. The + * relaxed shared lock was chosen just before, while + * fc->dlm still read 1, and it is only sound under DLM: + * nothing else excludes a concurrent writer on a + * disjoint range, and the buffered write path no longer + * serialises them itself. Re-decide now, while no lock + * is held yet. + */ + exclusive = fuse_cache_wr_exclusive_lock(iocb, true); } /* From b510e66df065c703145fbf401747dea26e38af95 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Wed, 5 Aug 2026 15:20:18 +0200 Subject: [PATCH 23/28] fuse: mark writeback-initiated SETATTR with FATTR_WRITEBACK fuse_write_inode() -> fuse_flush_times() pushes out the mtime/ctime that the kernel owns locally while the writeback cache is on. On the wire that request is indistinguishable from a userspace "touch -m": both arrive as SETATTR with FATTR_MTIME | FATTR_CTIME | FATTR_FH, because trust_local_cmtime makes iattr_to_fattr() send CTIME whenever the writeback cache is enabled. A server that wants to handle a cache flush differently from an explicit attribute change - skipping a cluster-wide lock, merging rather than overwriting - has no way to tell them apart. Add FATTR_WRITEBACK, a control bit in fuse_setattr_in.valid alongside the existing non-attribute bits FATTR_FH, FATTR_LOCKOWNER and FATTR_KILL_SUIDGID. It selects no attribute, it only states that the request originates from writeback. Bit 30 is used rather than the next free one. libfuse mirrors the wire bits into its own FUSE_SET_ATTR_* namespace, where bits 12 to 17 are already taken by library-internal flags, and it masks incoming requests against that namespace; a bit picked from the low end would collide there and need translating on the way in. Bit 30 is free on both sides and clear of the sign bit of the int that the libfuse setattr operation takes, so one value works end to end. The bit is negotiated at INIT time with FUSE_SETATTR_WRITEBACK and is only set on a connection whose server asked for it, so servers that do not know the bit never receive it. Only ->write_inode() is marked. The other kernel-initiated SETATTR, fuse_do_truncate() rolling back a failed extending direct-IO write, is deliberately left unmarked: it is a size correction rather than an attribute writeback, and conflating the two would make the flag ambiguous. Signed-off-by: Horst Birthelmer --- fs/fuse/dir.c | 6 ++++++ fs/fuse/fuse_i.h | 3 +++ fs/fuse/inode.c | 5 ++++- include/uapi/linux/fuse.h | 14 ++++++++++++++ 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c index 69510a0947eecf..99af2777432201 100644 --- a/fs/fuse/dir.c +++ b/fs/fuse/dir.c @@ -2016,6 +2016,12 @@ int fuse_flush_times(struct inode *inode, struct fuse_file *ff) inarg.valid |= FATTR_FH; inarg.fh = ff->fh; } + /* + * This is ->write_inode() flushing times the kernel owns locally, not + * a userspace utimes(); let the server tell the two apart. + */ + if (fm->fc->setattr_writeback) + inarg.valid |= FATTR_WRITEBACK; fuse_setattr_fill(fm->fc, &args, inode, &inarg, &outarg); return fuse_simple_request(fm, &args); diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index 26678ceecff108..c6e9791105671d 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -840,6 +840,9 @@ struct fuse_conn { /* expire inode entries when doing inode invalidation */ unsigned expire_inode_entries:1; + /* mark writeback-initiated SETATTR requests with FATTR_WRITEBACK */ + unsigned setattr_writeback:1; + /* * The following bitfields are only for optimization purposes * and hence races in setting them will not cause malfunction diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index fc5e0b852120bb..19ceb73ed5b367 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -1836,6 +1836,8 @@ static void process_init_reply(struct fuse_mount *fm, struct fuse_args *args, fc->inval_inode_entries = 1; if (flags & FUSE_EXPIRE_INODE_ENTRY) fc->expire_inode_entries = 1; + if (flags & FUSE_SETATTR_WRITEBACK) + fc->setattr_writeback = 1; } else { ra_pages = fc->max_read / PAGE_SIZE; fc->no_lock = 1; @@ -1886,7 +1888,8 @@ static struct fuse_init_args *fuse_new_init(struct fuse_mount *fm) FUSE_SECURITY_CTX | FUSE_CREATE_SUPP_GROUP | FUSE_HAS_EXPIRE_ONLY | FUSE_DIRECT_IO_ALLOW_MMAP | FUSE_NO_EXPORT_SUPPORT | FUSE_INVAL_INODE_ENTRY | - FUSE_EXPIRE_INODE_ENTRY | FUSE_URING_REDUCED_Q; + FUSE_EXPIRE_INODE_ENTRY | FUSE_URING_REDUCED_Q | + FUSE_SETATTR_WRITEBACK; #ifdef CONFIG_FUSE_DAX if (fm->fc->dax) flags |= FUSE_MAP_ALIGNMENT; diff --git a/include/uapi/linux/fuse.h b/include/uapi/linux/fuse.h index 7902f034908cb8..d814a339442ff2 100644 --- a/include/uapi/linux/fuse.h +++ b/include/uapi/linux/fuse.h @@ -360,6 +360,17 @@ struct fuse_file_lock { #define FATTR_LOCKOWNER (1 << 9) #define FATTR_CTIME (1 << 10) #define FATTR_KILL_SUIDGID (1 << 11) +/* + * Not an attribute selector: marks the request as a kernel-initiated + * writeback of locally owned attributes rather than a userspace-initiated + * change. Only sent if the server negotiated FUSE_SETATTR_WRITEBACK. + * + * The bit is deliberately far above the sequentially allocated FATTR_* + * range: libfuse mirrors these bits into its own FUSE_SET_ATTR_* space, + * which has its own allocations from bit 12 upwards, and only a value that + * is free on both sides can be passed through without translation. + */ +#define FATTR_WRITEBACK (1 << 30) /** * Flags returned by the OPEN request @@ -441,6 +452,8 @@ struct fuse_file_lock { * optimal io-size alignment * FUSE_URING_REDUCED_Q: Client (kernel) supports less queues - Server is free * to register between 1 and nr-core io-uring queues + * FUSE_SETATTR_WRITEBACK: kernel marks writeback-initiated SETATTR requests + * with FATTR_WRITEBACK */ #define FUSE_ASYNC_READ (1 << 0) #define FUSE_POSIX_LOCKS (1 << 1) @@ -490,6 +503,7 @@ struct fuse_file_lock { #define FUSE_OVER_IO_URING (1ULL << 41) #define FUSE_ALIGN_PG_ORDER (1ULL << 50) +#define FUSE_SETATTR_WRITEBACK (1ULL << 58) #define FUSE_URING_REDUCED_Q (1ULL << 59) #define FUSE_INVAL_INODE_ENTRY (1ULL << 60) #define FUSE_EXPIRE_INODE_ENTRY (1ULL << 61) From eb0d730347a513af9c442fe70ed5ce3acd1095f7 Mon Sep 17 00:00:00 2001 From: Hai Zhong Zhou Date: Wed, 12 Aug 2026 09:32:34 +0000 Subject: [PATCH 24/28] fuse: disable local attribute cache override under DLM fuse_get_cache_mask() returned STATX_MTIME|CTIME|SIZE whenever writeback_cache was enabled, causing the kernel to trust its locally cached mtime/ctime/size over whatever the server returned in a GETATTR reply. This is unsafe under DLM: another node can hold a PW lock on the inode and modify its size/mtime independently, and the local writeback_cache values have no way of reflecting that. Under the DLM protocol, however, this override is unnecessary in the first place: a GETATTR always acquires a PR sattr lock, which forces every node holding a conflicting PW lock -- including the local node, for its own buffered writes -- to flush dirty pages before the server renders the reply. So under DLM the server's answer is always at least as fresh as anything cached locally, for all three attributes, not just size. Make fuse_get_cache_mask() return 0 whenever fc->dlm is set, when writeback_cache is enabled, so the kernel always trusts the server's attr/size reply in that case. The STATX_MTIME|CTIME|SIZE local-cache override remains only as a fallback for servers without DLM support, where no such flush-before-grant guarantee exists. Signed-off-by Hai Zhong Zhou --- fs/fuse/inode.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 19ceb73ed5b367..1d59cab6f792b9 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -493,7 +493,7 @@ u32 fuse_get_cache_mask(struct inode *inode) { struct fuse_conn *fc = get_fuse_conn(inode); - if (!fc->writeback_cache || !S_ISREG(inode->i_mode)) + if (!fc->writeback_cache || !S_ISREG(inode->i_mode) || fc->dlm) return 0; return STATX_MTIME | STATX_CTIME | STATX_SIZE; From 9e9a8e779ae66ec1fb4a6b119f8d0d9ed8a6dafb Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Fri, 14 Aug 2026 08:30:34 +0200 Subject: [PATCH 25/28] fuse: don't launder from a NOTIFY invalidate while writepages are frozen fuse_reverse_inval_inode() invalidates with invalidate_inode_pages2_range(), which waits out folios under writeback and launders dirty ones. Both need a FUSE_WRITE reply, and while fi->writectr < 0 none can arrive: fuse_flush_writepages() parks the request on fi->queued_writes until fuse_release_nowrite(). A truncate holds that freeze across its whole SETATTR, and the server revokes the truncated range from inside the SETATTR handler, so the notify blocks the very thread that owes the reply lifting the freeze. generic/014 deadlocks within seconds, in folio_wait_writeback() under fuse_launder_folio() under fuse_reverse_inval_inode(). fuse_do_setattr() already states the rule ("Only call invalidate_inode_pages2() after removing FUSE_NOWRITE, otherwise fuse_launder_folio() would deadlock"). Give the notify path the same: while frozen, use invalidate_mapping_pages(), which skips dirty and under-writeback folios instead of waiting on them. The stale clean folios still go, the DLM grant is revoked either way, and the freezes that span a request drop the cache themselves when they finish. Signed-off-by: Horst Birthelmer --- fs/fuse/inode.c | 43 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 1d59cab6f792b9..015ed1b3bec695 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -837,6 +837,40 @@ static void fuse_dlm_revoke_inval_range(struct fuse_inode *fi, loff_t offset, fuse_dlm_unlock_range(fi, start, end); } +/* + * Drop a page-cache range on behalf of a NOTIFY invalidate. + * + * invalidate_inode_pages2_range() waits out folios under writeback and + * launders dirty ones, both of which need a FUSE_WRITE reply. While + * writepages are frozen (fuse_set_nowrite(): truncate, O_TRUNC open, fsync, + * pre-SETATTR flush) no reply can arrive, because fuse_flush_writepages() + * parks the request on fi->queued_writes until fuse_release_nowrite(). A + * server that revokes from inside the handler it is revoking for then + * deadlocks against its own reply. fuse_do_setattr() states the same rule + * for its own invalidate. + * + * So while frozen use invalidate_mapping_pages(), which skips dirty and + * under-writeback folios and never blocks. The stale clean folios still + * go, and the freezes that span a request drop the cache themselves once + * they complete: fuse_do_setattr() invalidates the mapping after releasing + * the freeze, the O_TRUNC open path calls truncate_pagecache(). + */ +static void fuse_notify_invalidate_range(struct inode *inode, pgoff_t start, + pgoff_t end) +{ + struct fuse_inode *fi = get_fuse_inode(inode); + bool frozen; + + spin_lock(&fi->lock); + frozen = fi->writectr < 0; + spin_unlock(&fi->lock); + + if (frozen) + invalidate_mapping_pages(inode->i_mapping, start, end); + else + invalidate_inode_pages2_range(inode->i_mapping, start, end); +} + int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, loff_t offset, loff_t len) { @@ -970,10 +1004,10 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, * notified range. */ if (fuse_inode_force_dio(inode)) - invalidate_inode_pages2(inode->i_mapping); + fuse_notify_invalidate_range(inode, 0, -1); else - invalidate_inode_pages2_range(inode->i_mapping, - pg_start, pg_end); + fuse_notify_invalidate_range(inode, pg_start, + pg_end); percpu_up_write(wb_sem); @@ -986,8 +1020,7 @@ int fuse_reverse_inval_inode(struct fuse_conn *fc, u64 nodeid, * range unserialized (best-effort), as before. */ if (fc->dlm && fc->writeback_cache) fuse_dlm_revoke_inval_range(fi, offset, len); - invalidate_inode_pages2_range(inode->i_mapping, - pg_start, pg_end); + fuse_notify_invalidate_range(inode, pg_start, pg_end); } } iput(inode); From 5157748972127c92635be249e17de14ffbaa7e48 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Fri, 14 Aug 2026 10:58:21 +0200 Subject: [PATCH 26/28] fuse: do not kill suid from inside the coherency gate fuse_cache_write_iter() holds wb_inval_rwsem for read across __generic_file_write_iter(), which starts with file_remove_privs(). Without handle_killpriv[_v2] that asks the server (GETATTR, then SETATTR), and a server that invalidates the inode from inside such a handler blocks in percpu_down_write() draining the gate reader that is waiting for its reply. generic/193 hangs there. The gate only has to fence the page-cache dirtying, so run the privilege kill and the timestamp update before entering it, in both branches. Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 74 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 58 insertions(+), 16 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index c1803ea3cc0bee..4cdc6fffc39481 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1760,6 +1760,34 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) goto wb_out; } + written = generic_write_checks(iocb, from); + if (written <= 0) + goto wb_out; + + /* + * Kill suid/sgid and stamp the timestamps here, before the + * gate, instead of leaving them to + * __generic_file_write_iter(). file_remove_privs() is the one + * that reaches the server: without handle_killpriv[_v2] + * fuse_setattr() kills the bits by asking it (a FUSE_GETATTR to + * refresh the mode, then a FUSE_SETATTR, which for a writeback + * inode first flushes and freezes writepages), and + * security_inode_killpriv() can drop the capability xattr with + * another round trip. A server may have to invalidate this + * inode from inside such a handler; its NOTIFY_INVAL_INODE then + * blocks in percpu_down_write() draining a gate reader that is + * itself waiting for the reply. Nothing held under the gate may + * wait for the server. file_update_time() only marks the inode + * dirty, but stays next to it to keep the VFS order. + */ + err = file_remove_privs(file); + if (!err) + err = file_update_time(file); + if (err) { + written = err; + goto wb_out; + } + if (wb_sem) { wb_guard = true; retry: @@ -1786,9 +1814,15 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) goto retry; } } - written = generic_write_checks(iocb, from); - if (written > 0) - written = __generic_file_write_iter(iocb, from); + if (iocb->ki_flags & IOCB_DIRECT) { + written = generic_file_direct_write(iocb, from); + if (written < 0 || !iov_iter_count(from)) + goto wb_out; + written = direct_write_fallback(iocb, from, written, + generic_perform_write(iocb, from)); + } else { + written = generic_perform_write(iocb, from); + } wb_out: if (wb_guard) percpu_up_read(wb_sem); @@ -1801,13 +1835,33 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) writethrough: inode_lock(inode); + err = count = generic_write_checks(iocb, from); + if (err <= 0) + goto out; + + /* + * Kill suid/sgid and stamp the timestamps before entering the gate, + * for the reason given in the writeback branch: file_remove_privs() + * can issue a request, and a request must never be waited for under + * the gate. They run before the forced-DIO re-route below, so a + * re-routed write repeats them; neither has anything left to do the + * second time. + */ + err = file_remove_privs(file); + if (err) + goto out; + + err = file_update_time(file); + if (err) + goto out; + /* * The killpriv fallback lands here with the writeback cache still on, * so it populates the page cache too and needs the same guard as the * writeback branch above: hold the coherency gate for read across the * page-cache population and re-check the latch under it, so a * concurrent NOTIFY_INVAL_INODE cannot have the cache repopulated - * behind the invalidate it just did. Taken before + * behind the invalidate it just did. Still taken before * task_io_account_write() so a re-route is not double-counted. * wb_sem is NULL on mounts where the gate is inactive, and such a * connection never latches either. @@ -1822,20 +1876,8 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) } } - err = count = generic_write_checks(iocb, from); - if (err <= 0) - goto out; - task_io_account_write(count); - err = file_remove_privs(file); - if (err) - goto out; - - err = file_update_time(file); - if (err) - goto out; - if (iocb->ki_flags & IOCB_DIRECT) { written = generic_file_direct_write(iocb, from); if (written < 0 || !iov_iter_count(from)) From 3fc92d60d7ac2a4eb42587c0c24f1478103188e8 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Fri, 14 Aug 2026 12:00:26 +0200 Subject: [PATCH 27/28] fuse: keep cached size and times under a DLM write grant fuse_get_cache_mask() returns 0 once the connection has DLM, so every GETATTR reply overwrites i_size, mtime and ctime, and truncate_pagecache() then drops the tail the client still holds dirty. The only way a server can make that answer true is to revoke the client from inside the handler, which deadlocks against the coherency gate. A write grant already guarantees that no other node can touch the range, so keep the cached values while one is held: the size when the server reports less than i_size and [srv_size, i_size) is fully granted, mtime and ctime while the cache under the grant is still dirty. A remote truncate has to revoke first, so the smaller size that follows is applied as usual. The attribute-driven invalidation now keys off STATX_SIZE instead of the whole mask, so a reply that does shrink i_size still truncates the page cache when only the timestamps were served from the cache. Signed-off-by: Horst Birthelmer --- fs/fuse/fuse_dlm_cache.c | 31 +++++++++++++++ fs/fuse/fuse_dlm_cache.h | 3 ++ fs/fuse/inode.c | 83 +++++++++++++++++++++++++++++++++++----- 3 files changed, 107 insertions(+), 10 deletions(-) diff --git a/fs/fuse/fuse_dlm_cache.c b/fs/fuse/fuse_dlm_cache.c index b0b17cbd3c3f7a..bc6dbae2d5aeb0 100644 --- a/fs/fuse/fuse_dlm_cache.c +++ b/fs/fuse/fuse_dlm_cache.c @@ -562,6 +562,37 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, return true; } +/** + * fuse_dlm_write_grant_exists - does the inode hold an exclusive grant anywhere + * @fi: the fuse inode + * + * Unlike fuse_dlm_range_is_locked(), which asks whether one range is fully + * covered, this asks whether any part of the file is held exclusively. A + * client that holds a write grant may be sitting on dirty page cache the + * server has not seen, so its mtime and ctime run ahead of anything the + * server can report. + * + * Return: true if at least one recorded range is held for write + */ +bool fuse_dlm_write_grant_exists(struct fuse_inode *fi) +{ + struct fuse_dlm_cache *cache = &fi->dlm_locked_areas; + struct fuse_dlm_range *range; + bool held = false; + + down_read(&cache->lock); + for (range = fuse_dlm_find_overlapping(cache, 0, U64_MAX); range; + range = fuse_page_it_iter_next(range, 0, U64_MAX)) { + if (range->mode == FUSE_PCACHE_LK_WRITE) { + held = true; + break; + } + } + up_read(&cache->lock); + + return held; +} + /** * fuse_dlm_lock_is_held - check that a byte range is covered by a granted lock * @fi: the fuse inode diff --git a/fs/fuse/fuse_dlm_cache.h b/fs/fuse/fuse_dlm_cache.h index 647a8c37c36095..30fdbb26bd3daf 100644 --- a/fs/fuse/fuse_dlm_cache.h +++ b/fs/fuse/fuse_dlm_cache.h @@ -71,6 +71,9 @@ bool fuse_dlm_range_is_locked(struct fuse_inode *inode, uint64_t start, bool fuse_dlm_lock_is_held(struct fuse_inode *inode, loff_t offset, size_t length, enum fuse_page_lock_mode mode); +/* Is any part of the file held for write? */ +bool fuse_dlm_write_grant_exists(struct fuse_inode *inode); + /* This is the interface to the filesystem */ int fuse_get_dlm_lock(struct file *file, loff_t offset, size_t length, enum fuse_page_lock_mode mode); diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c index 015ed1b3bec695..b4b4cec2d65671 100644 --- a/fs/fuse/inode.c +++ b/fs/fuse/inode.c @@ -499,6 +499,68 @@ u32 fuse_get_cache_mask(struct inode *inode) return STATX_MTIME | STATX_CTIME | STATX_SIZE; } +/* + * Which cached attributes survive a server reply. + * + * Without DLM this is fuse_get_cache_mask(): with the writeback cache on, + * writes update mtime and ctime and may extend i_size locally, the server + * knows about none of it, so the cached values win. + * + * With DLM the server is the authority (fuse_get_cache_mask() returns 0), + * because another node may have changed the file behind us and only the + * server can say so. That holds for the parts of the file we do not own. A + * write grant means no other node can touch the range until we are revoked, + * so anything the server reports about it is at best as new as what we have, + * and older if we still have unwritten data there. Keep the cached values + * for exactly what the grant covers: + * + * - size, when the server reports less than i_size and the tail it does not + * know about, [srv_size, i_size), is entirely under a write grant. Taking + * the server's answer would shrink i_size and have truncate_pagecache() + * throw the unwritten tail away. + * - mtime and ctime, while a write grant covers unwritten data: our writes + * have stamped them locally and the server's stamps predate them. Only + * while the cache is actually dirty, not for as long as the grant lives: + * a grant is held until it is revoked or the inode is evicted, and past + * the writeback the server's stamps are the newer ones. Keeping ours + * beyond that would hide a remote chown or chmod indefinitely. + * + * A remote truncate cannot slip through. It has to revoke the grant first, + * and the revoke launders the tail and drops the grant, so by the time the + * smaller size is reported neither check holds and the server's answer is + * applied as usual. A grant the server made but that could not be recorded + * (FUSE_DLM_GRANT_UNRECORDED) is invisible to the lock tree and falls back to + * trusting the server, as before. + * + * Must be called without fi->lock: the lock tree query sleeps. + */ +static u32 fuse_attr_cache_mask(struct inode *inode, struct fuse_attr *attr, + bool have_size) +{ + struct fuse_conn *fc = get_fuse_conn(inode); + struct fuse_inode *fi = get_fuse_inode(inode); + u32 cache_mask = fuse_get_cache_mask(inode); + loff_t size = i_size_read(inode); + + if (cache_mask || !fc->dlm || !fc->writeback_cache || + !S_ISREG(inode->i_mode)) + return cache_mask; + + if (!fuse_dlm_write_grant_exists(fi)) + return cache_mask; + + if (mapping_tagged(inode->i_mapping, PAGECACHE_TAG_DIRTY) || + mapping_tagged(inode->i_mapping, PAGECACHE_TAG_WRITEBACK)) + cache_mask |= STATX_MTIME | STATX_CTIME; + + if (have_size && size > (loff_t) attr->size && + fuse_dlm_lock_is_held(fi, attr->size, size - attr->size, + FUSE_PAGE_LOCK_WRITE)) + cache_mask |= STATX_SIZE; + + return cache_mask; +} + static void fuse_change_attributes_i(struct inode *inode, struct fuse_attr *attr, struct fuse_statx *sx, u64 attr_valid, u64 attr_version, u64 evict_ctr) @@ -511,15 +573,11 @@ static void fuse_change_attributes_i(struct inode *inode, struct fuse_attr *attr bool have_size = !sx || (sx->mask & STATX_SIZE); u64 srv_size; + cache_mask = fuse_attr_cache_mask(inode, attr, have_size); + spin_lock(&fi->lock); srv_size = attr->size; - /* - * In case of writeback_cache enabled, writes update mtime, ctime and - * may update i_size. In these cases trust the cached value in the - * inode. - */ - cache_mask = fuse_get_cache_mask(inode); if (cache_mask & STATX_SIZE) attr->size = i_size_read(inode); @@ -566,11 +624,16 @@ static void fuse_change_attributes_i(struct inode *inode, struct fuse_attr *attr spin_unlock(&fi->lock); /* - * Only do page cache invalidation when cache_mask is not set - * (writeback_cache disabled) AND the relevant attributes (SIZE/MTIME) - * were actually returned by the server. + * Only do page cache invalidation when the size was not served from + * the cache (writeback_cache disabled, or no grant covering the tail) + * AND the relevant attributes (SIZE/MTIME) were actually returned by + * the server. This has to key off STATX_SIZE alone: i_size_write() + * above took the server's size for any mask without that bit, and the + * cache has to be truncated to match it. The mtime branch neutralises + * itself when STATX_MTIME is set, since attr->mtime then holds the + * value old_mtime was read from. */ - if (!cache_mask && S_ISREG(inode->i_mode)) { + if (!(cache_mask & STATX_SIZE) && S_ISREG(inode->i_mode)) { bool inval = false; bool have_mtime = !sx || (sx->mask & STATX_MTIME); From db0bd6bc7d9f7ee6b8fe57a096a0536d52e101b4 Mon Sep 17 00:00:00 2001 From: Horst Birthelmer Date: Fri, 14 Aug 2026 13:50:15 +0200 Subject: [PATCH 28/28] fuse: take i_rwsem exclusive when a write drops suid/sgid Without handle_killpriv[_v2], fuse_setattr() kills the bits by asking the server, and fuse_do_setattr() freezes writepages around that SETATTR. fuse_set_nowrite() asserts BUG_ON(fi->writectr < 0) under fi->lock, which assumes the caller holds i_rwsem exclusive: with the writeback cache and DLM, buffered writes hold it only shared. Two writers to a suid file can then both pass dentry_needs_remove_privs() before either has cleared the bits, and the second one hits the assert. It oopses inside spin_lock(&fi->lock), so fi->lock stays held and the i_rwsem read count leaks: the inode wedges and the box follows. The race window is a full GETATTR plus SETATTR, so it is not narrow, and an unprivileged user can set the bit on a file it owns. Send those writes down the writethrough branch, which takes i_rwsem exclusive, the way handle_killpriv_v2 writes already go. Scoped to DLM connections, since every other configuration already holds i_rwsem exclusive for a buffered write, and only writes that still find the bits set pay for it. Note that the writethrough branch takes no DLM lock, so those writes leave clean folios in the page cache without a grant covering them. That gap already exists for handle_killpriv_v2 and is not addressed here. Signed-off-by: Horst Birthelmer --- fs/fuse/file.c | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 4cdc6fffc39481..a9198a7eb283e2 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1683,11 +1683,33 @@ static ssize_t fuse_cache_write_iter(struct kiocb *iocb, struct iov_iter *from) if (err) return err; - if (fc->handle_killpriv_v2 && - setattr_should_drop_suidgid(idmap, - file_inode(file))) { + /* + * A write that drops suid/sgid goes down the writethrough + * branch, which holds i_rwsem exclusive. + * + * With handle_killpriv_v2 that is because the server does the + * killing from the WRITE itself. Without it, fuse_setattr() + * has to ask the server, and fuse_do_setattr() freezes + * writepages around that SETATTR: fuse_set_nowrite() asserts + * BUG_ON(fi->writectr < 0), which assumes an exclusive + * i_rwsem, and the DLM-relaxed buffered write path below holds + * it only shared. Two writers can both see the bits set + * before either has cleared them, and the second one would + * then oops inside spin_lock(&fi->lock). + * + * Only the DLM path needs the detour: everywhere else the + * buffered write already holds i_rwsem exclusive, so the two + * writers cannot overlap in the first place. + * + * The bits are read without the inode lock here, so a server + * attribute update can still set them between this test and + * file_remove_privs(). That leaves the same race, but only + * for writers whose mode changed underneath them, rather than + * for every write to a suid file. + */ + if ((fc->handle_killpriv_v2 || fc->dlm) && + setattr_should_drop_suidgid(idmap, file_inode(file))) goto writethrough; - } exclusive = fuse_cache_wr_exclusive_lock(iocb, true);