Conversation
This is especially needed for better ftrace analysis, for example to build histograms. So far the request unique was missing, because it was added after the first trace message. IDs/req-unique now might not come up perfectly sequentially anymore, but especially with cloned device or io-uring this did not work perfectly anyway. Signed-off-by: Bernd Schubert <bschubert@ddn.com> (imported from commit 4415892) (cherry picked from commit 02f0a37) [hbi: 7.0 assigns the unique in exported send helpers that also trace; keep them, assigning only when fuse_args_to_req() has not]
Sometimes the file offset alignment needs to be opt-in to achieve the optimum performance at the backend store. For example when ErasureCode [1] is used at the backend store, the optimum write performance is achieved when the WRITE request is aligned with the stripe size of ErasureCode. Otherwise a non-aligned WRITE request needs to be split at the stripe size boundary. It is quite costly to handle these split partial requests, as firstly the whole stripe to which the split partial request belongs needs to be read out, then overwrite the read stripe buffer with the request, and finally write the whole stripe back to the persistent storage. Thus the backend store can suffer severe performance degradation when WRITE requests can not fit into one stripe exactly. The write performance can be 10x slower when the request is 256KB in size given 4MB stripe size. Also there can be 50% performance degradation in theory if the request is not stripe boundary aligned. Besides, the conveyed test indicates that, the non-alignment issue becomes more severe when decreasing fuse's max_ratio, maybe partly because the background writeback now is more likely to run parallelly with the dirtier. fuse's max_ratio ratio of aligned WRITE requests ---------------- ------------------------------- 70 99.9% 40 74% 20 45% 10 20% With the patched version, which makes the alignment constraint opt-in when constructing WRITE requests, the ratio of aligned WRITE requests increases to 98% (previously 20%) when fuse's max_ratio is 10. fuse: fix alignment to work with redfs ubuntu - small fix to make the fuse alignment patch work with redfs ubuntu 6.8.x - add writeback_control to fuse_writepage_need_send() to make more accurate decisions about when to skip sending data - fix shift number for FUSE_ALIGN_PG_ORDER - remove test code [1] https://lore.kernel.org/linux-fsdevel/20240124070512.52207-1-jefflexu@linux.alibaba.com/T/#m9bce469998ea6e4f911555c6f7be1e077ce3d8b4 Signed-off-by: Jingbo Xu <jefflexu@linux.alibaba.com> Signed-off-by: Bernd Schubert <bschubert@ddn.com> Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (imported from commit 5e590a6) (cherry picked from commit 1ee8c80) [hbi: 7.0 shares fuse_folios_need_send() with the read path, so the alignment check goes in fuse_iomap_writeback_range()]
Simplify fuse_compound_req to hold only the pointers to the added fuse args and the request housekeeping. Simplify open+getattr call by using helper functions to fill out the fuse request parameters Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (imported from commit 1607a03) (imported from commit 9df5e4c) (imported from commit 9921bcd) (imported from commit 09d6f59) (cherry picked from commit 41b40bd)
In preparation for allowing partial attribute updates via statx (where only a subset of STATX_BASIC_STATS may be requested and returned), modify fuse_change_attributes_common() to be more selective about what it updates. Currently, fuse_change_attributes_common() unconditionally: 1. Clears ALL of STATX_BASIC_STATS from inval_mask 2. Updates fi->i_time (extending the attribute timeout) For some fuse-servers it might be benefitial to reduce the number of queries attributes and and attribute mask is one of the features of statx. With the all or nothing handling of fuse_change_attributes_common() that statx feature is impossible to be used. This commit adds the logic to: 1. Track which attributes were actually returned (via sx->mask for statx, or assume all STATX_BASIC_STATS for getattr) 2. Only clear those specific attributes from inval_mask 3. Only update fi->i_time when it's safe: when cache_mask is empty OR when all cache_mask attributes were included in the response The condition in fuse_do_statx() still requires ALL STATX_BASIC_STATS, so this commit has no functional change. A follow up commit will relax that condition to enable partial updates. Signed-off-by: Bernd Schubert <bernd@bsbernd.com> (cherry picked from commit 8034f26) (cherry picked from commit bf71659)
…attributes For permission checks via inode_permission(), we only need mode, uid, and gid attributes. Previously, we requested all STATX_BASIC_STATS, which was inefficient. This commit enables the optimization by: 1. Requesting only STATX_MODE | STATX_UID | STATX_GID for permission checks 2. Relaxing the condition in fuse_do_statx() from requiring all basic stats to accepting any subset of basic stats 3. Adding validation that the server returns at least what was requested The preparation commit ensures partial updates work correctly by only updating returned attributes and managing timeouts appropriately. Signed-off-by: Bernd Schubert <bernd@bsbernd.com> (cherry picked from commit 09ed47b) (cherry picked from commit 7a08fe8)
fuse_perm_getattr() requests only STATX_MODE | STATX_UID | STATX_GID. fi->i_time is only advanced when the response covers STATX_BASIC_STATS in full, so a perm-only refresh clears the perm bits from inval_mask but leaves i_time at its old (expired) value. Add fi->i_perm_time, updated whenever a refresh covers MODE|UID|GID (always for the non-statx getattr path; conditionally on returned_attrs for the statx path), and gate fuse_permission()'s sync decision on it instead of i_time. fuse_update_get_attr() continues to use i_time so a full stat() still re-queries the server when size/mtime/etc. expire. Fixes: 09ed47b ("fuse: Optimize statx for permission checks by requesting only needed attributes") Signed-off-by: Horst Birthelmer <horst@birthelmer.de> (cherry picked from commit a2d3971) (cherry picked from commit fa98cf9)
Don't assume that the name is null terminated in fuse_do_lookupx() That holds when name is taken from a dentry's d_name (dcache invariant), but the 6.14 d_revalidate signature change (commit 5be1fa8) makes the callback receive the walker's qstr, whose backing buffer is the pathname being resolved. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (cherry picked from commit e3501c1)
On the pinned-header send path fuse_uring_dispatch_ent() calls io_uring_cmd_done() directly from the request submitter's context with IO_URING_F_UNLOCKED. Every fuse-uring command is marked cancelable, so io_uring_cmd_del_cancelable() has to take ctx->uring_lock from that foreign task on every request. The ring task holds this mutex for the whole ->uring_cmd() issue path (io_uring_enter() submission), where it also wakes the submitter of the request it just committed - before releasing the lock. The freshly woken submitter usually preempts the ring task on the same CPU, and its next dispatch then blocks on the very mutex its victim still holds. The preempted owner is merely runnable and gets no wakeup boost, so under CPU pressure this convoy costs milliseconds per request while the daemon's actual work is a few microseconds. Keep the copies into the pinned pages in the submitter's context - that is the point of the pinning - but defer the command completion to ring task task-work, like the non-pinned path already does. There io_uring_cmd_del_cancelable() runs under the task-work batch's already held uring_lock, and the submitter no longer touches ctx->uring_lock at all on the fast path (only the rare copy-failure fallback still can). Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (cherry picked from commit 6470c72) [hbi: 7.0 task work takes io_req_tw_func_t; use tw.cancel in place of IO_URING_F_TASK_DEAD]
fuse_uring_commit_fetch() runs from the ->uring_cmd() issue path with ctx->uring_lock held by io_uring_enter(). Ending the committed request there means wake_up(&req->waitq) runs while the ring task still holds the mutex, and the freshly woken submitter typically preempts the ring task on the same CPU right away - leaving the lock held by a merely runnable task for however long the runqueue is backed up. wake_up_sync() would not help here: WF_SYNC only biases task placement in select_task_rq_fair(), check_preempt_wakeup_fair() never looks at it, so it cannot stop the wakee from preempting the waker. Instead defer fuse_request_end() to task work on the ring task. It runs after the submission path has dropped uring_lock - at latest on return to userspace, typically at io_cqring_wait() entry via io_run_task_work() - so the ring task finishes its critical section and the entry recycling in fuse_uring_next_fuse_req() before the submitter gets the CPU. This also moves the request ->end() callbacks off the locked section, shortening the uring_lock hold time. Fall back to completing in place when the command was issued from an unlocked context (io-wq punt) or task_work_add() fails because the task is exiting. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (cherry picked from commit b6ec467)
The append grant is taken at a sampled i_size, but the exclusive inode lock does not pin it: attribute replies move i_size under fi->lock alone. generic_write_checks() then rewrites ki_pos to the fresher i_size and the write lands past the granted range, dirtying the cache without DLM coverage. The in-gate re-validation checks the stale range and cannot catch it. Request the lock again for the final ki_pos when it moved, and let dlm_pos track it so the in-gate re-validation guards the range the write really lands in. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (cherry picked from commit c93c346)
Read-modify-writes of blocks past the server EOF go back to reading: the zero-fill needed fi->server_size verified against every recorded write grant, and the verification GETATTRs are server traffic and complexity the saved READs are not worth. Partial pages of expanding writes will be sent as separate FUSE_WRITE requests instead, which needs no read-modify-write at all. Remove the fi->server_size bound with its grow and shrink sites, the grant verification debt, and the zero-fill in fuse_iomap_read_folio_range(). Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (cherry picked from commit ea43157)
With the expansion zero-fill gone, every unaligned buffered write under DLM pays a read-modify-write READ per partial page, and past the server EOF that READ returns zero bytes: a wasted round trip for data that cannot exist. The write path cannot tell the wasted READ from a needed one without trusting a bound the grant may predate. Do not complete partial pages at all. Split the buffered write (fuse_dlm_buffered_write()): cache only the page-aligned interior, whole pages need no read-modify-write, and send the unaligned head and tail through the existing writethrough path (fuse_perform_write()). A partial page is written byte-exact and left non-uptodate, so nothing is read and neighbouring writers sharing a boundary page accumulate their bytes on the server. A sub-page write with no aligned interior goes fully through. The DLM write lock is taken over the whole range before i_rwsem, so all chunks are covered, and the pre-claimed i_size extension with its reconciliation keeps working on the chunk total. The read-modify-write path stays functional for the rare unaligned restart of the interior after a faulted user copy. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (cherry picked from commit ae035fb)
Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (cherry picked from commit 41b92e4)
fuse_dlm_buffered_write() sends the unaligned edges of a write to the server itself so no partly written block is dirtied, and cuts at PAGE_SIZE. Nothing checked that a block is a page: a fuseblk mount keeps whatever blksize= asked for. iomap then goes back for the remainder of an edge block and dirties what the write already sent. Refuse the connection at FUSE_INIT when a writeback cache is negotiated on a block that is not a page. st_blksize still reports what the server named. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (cherry picked from commit 7499e44) [hbi: 7.0 dropped fc->blkbits and never sets inode->i_blkbits from the server, so check sb->s_blocksize_bits and drop the per-inode pin]
Readahead fills the page cache past the range fuse_cache_read_iter() locked, so those folios get no revoke when a remote node writes them. Request a read grant over the whole window in fuse_readahead() before any folio is consumed, and skip the window when the request fails. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> (cherry picked from commit 8c2ff0e) [hbi: 7.0 reads ahead through iomap_readahead(); take the grant before it]
The record was an interval tree of ranges carrying a lock mode and a revoke generation, walked under one lock per inode. Splitting and merging ranges on every grant made it the contended structure on a file several threads write. Record coverage as two bitmaps per aligned file region instead, a bit per page for granted and for granted-for-write. A region is a fixed span, so recording a grant neither allocates nor rearranges anything, and the regions live in an xarray, so threads writing far apart never touch the same one. A grant still on the wire stays out of the bitmaps and waits on a pending list where a revoke marks it, which keeps a reply that crosses a revoke from recording coverage the server took back. Adapt the paths to it: reads and readahead take a read grant, a cached write takes the write grant for the range it dirties and sends its unaligned edges straight through, writeback confirms the grant for every run it sends, and a NOTIFY invalidate drops the record for the range it revokes before it drops the page cache. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_writeback_write_iter() published its AOP_TRUNCATED_PAGE retry state in a connection-wide xarray keyed by task pointer, and erased it again, once per buffered write. A task pointer is a high index, so every write built and tore down the whole node chain down to it under one lock shared by every writer of the mount, and its GFP_KERNEL allocation ran with i_rwsem held. iomap already carries a private pointer through to fuse_iomap_read_folio_range(), where fuse put the struct file. Put the file and the flag in one struct on the stack and pass that instead. The store could fail and take the write down with it, and the load could come back empty and drop a needed retry. Neither is reachable now. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A server answers a contended range with -EDEADLK, which fuse_do_readfolio() turns into AOP_TRUNCATED_PAGE so the read is retried. __fuse_get_dlm_lock() returned it instead, and its writeback caller has no way to hold on to a folio it reports an error for: iomap takes the dirty flag off before offering the folio and clears its dirty ranges whatever the callback returns, so the bytes are dropped without ever reaching the server. A NOTIFY invalidate then drops the now clean folio and the next read gets the pre-write content. Ask again on -EDEADLK and -EAGAIN, on the retry budget the killed-in- flight grant already uses. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
iomap clears the dirty flag before it offers a folio to ->writeback_range and clears its dirty ranges whatever the callback returns, and nothing puts either back. A run that reports an error has therefore dropped its bytes: the folio stays in the page cache, clean and uptodate, holding data the server never received, until an invalidate drops it and the next read returns what the server has. The comment claimed the opposite. Redirty the folio instead, for the grant and the allocation, while there is still a connection to send it on. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The DLM buffered write holds i_rwsem shared and claims its i_size extension before it dirties anything. fuse_attr_cache_mask() decides whether the server's size wins from an i_size it reads before that claim, and sleeps in the lock tree query before fuse_change_attributes_i() applies the answer under fi->lock. A GETATTR that left while i_size still matched the server's is therefore applied over every claim made since, and i_size drops by exactly the writes in flight. truncate_pagecache() then zeroes the tail of the page holding the new size and drops what is above, which writeback sends as zeros. Move attr_version on the claim so those replies are dropped, the same reason fuse_write_update_attr() moves it, and count the writers whose claim is not yet dirty for a reply that leaves after one. FUSE_I_SIZE_UNSTABLE cannot serve as the count: it is a single bit and every writer clears it. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_reverse_inval_inode() drives this inode's page cache, before the revoke to flush it and after the revoke to drop it, on the thread that wrote the NOTIFY into /dev/fuse. Both reach fuse_iomap_writeback_range(), which holds the folio locked and under writeback while fuse_dlm_regrant_range() sends FUSE_DLM_WB_LOCK and waits. The range is the one being revoked, so the server cannot answer until the revoke completes, and the revoke cannot complete until the handler returns. A folio dirtied under a grant a NOTIFY took away is reached this way, which fuse_cache_write_iter() says is allowed to happen. Mark the task across that work and report the run as a hole when the grant is not already held: the folio goes back on the dirty list and an ordinary writeback sends it with a grant of its own. Reporting a hole means the run needs the type reset that had no user before. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_iomap_writeback_range() runs with the folio locked and, since iomap_writeback_folio() starts the writeback before the run loop, under writeback as well. Asking the server for a grant there is the ordering Documentation/filesystems/fuse/fuse-AOP_TRUNCATED_PAGE-reason.txt exists to forbid: no cluster lock may be taken while a page lock is held. The read path has AOP_TRUNCATED_PAGE to unlock and retry with, and ->writeback_range has nothing of the sort, so the violation sits where the remedy does not reach. Skip a run whose grant has gone: report it as a hole, put the folio back on the dirty list and remember the range. fuse_iomap_writeback_submit() takes it back once the pass has let go of every folio, and the pass that follows sends it. Not from fuse_launder_folio(), which arrives with the folio locked by folio_unmap_invalidate(), nor from a revoke handler, which would ask for the range it is revoking. There the skip stands. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_dlm_request_begin() publishes the range asked for, and fuse_dlm_kill_pending() can only test that one, but the commit records what the server granted, which may reach FUSE_DLM_MAX_EXTRA_GRANT either side of it. A revoke processed while the request was on the wire and landing in that excess marks nothing: the request does not overlap it, and the shards hold no bit for it yet because the grant is not recorded. The excess is then recorded over the revoked range and no later NOTIFY takes it back, which is the case the pending list exists to catch. Publish the widest bounds the commit could record as well, and separate the two outcomes. A revoke over the range asked for still kills the grant. One over the excess alone leaves the range asked for recorded and drops the excess, which only costs a re-request. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The comment claimed the round trip happens before any folio of the window is locked. ->readahead is entered with all of them locked and readahead_folio() is what unlocks them, so it happens under them. Say so, and say why it does not close a cycle: the lock in the way belongs to the node the revoke is sent to, not to this one. Name the case that does not follow, a window this already holds part of. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_iomap_writeback_range() unions every run it had to skip, and fuse_iomap_writeback_submit() asks for the lot in one grant. A pass sweeping a large file skips runs gigabytes apart, so the union grows to the whole sweep and the request covers a range nothing wanted. Stop extending at one shard. The runs left out stay dirty and a later pass asks for them. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_writeback_redirty() dirtied the folio from inside ->writeback_range, and iomap_writeback_folio() runs iomap_clear_range_dirty() over the whole folio once that returns. For a folio one block wide there is no iomap_folio_state and the call does nothing, so it worked. A large folio has one, and the folio is left with the dirty flag and no dirty block under it: the next pass finds nothing to write and the folio goes clean with its bytes never sent. Hold the folio instead and dirty it once iomap has released it, on the next call or in the submit. fuse_writepage_need_send() also reached wbc->range_end with no wbc, which only a folio of a single run kept out of reach; guard it. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A cached write copies the bytes twice on their way to the server, into the folios and out of them into the ring the request is read from. A streamed file has no use for what the first copy leaves behind: the folios go to writeback unread and are dropped after it. Send such a write from write(2) itself, out of the caller's pages, once it carries FUSE_WRITE_STREAM_MIN bytes or more, which is what a record has to be worth saving the copy on. The cached range under the write is written back before it and dropped after, since the invalidate launders and a folio left behind would reach the server on top of these bytes. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A folio made uptodate from the server is served to every later reader, so the grant it was fetched under has to be held from the confirmation until the bytes are in the page cache. Without that a revoke sweeping the range leaves the fill behind it: the folio stays cached, uncovered, and the server sends no further notify for a lock this client no longer holds. Confirm the grant under a pin, as the write path does. Refused, or not covered, unlock the folio and back off with AOP_TRUNCATED_PAGE: neither the wait nor the grant request may be taken with a page lock held. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> [hbi: 7.0 fills through iomap, so the pin is dropped after iomap_read_folio() and after the merge path's own unlock; an async fill is covered by the span pin the readahead patch adds next]
A readahead reply lands in the page cache from the task that processes it, so the pin over the folios has to span the request: taken before it is sent and dropped once the folios are filled and unlocked. A revoke of the range waits for that and drops the folios after; one already draining refuses the pin and the window goes back unfilled. The node therefore outlives the pinning task, which fuse_dlm_unpin() cannot express. Add a span-owned pin, dropped by node and carrying no owner for the by-owner search to match. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> [hbi: 7.0 reads and readahead both go through iomap, so the pin sits in the shared fuse_send_readpages() and its -EAGAIN is left to the per run decline; the manual window loop the original breaks out of is gone]
A read grant may not be requested under a page lock, so every buffered read path asks for one before it enters the page cache: buffered read, splice read and read fault. The window covers what readahead may add beyond the read, bounded by the file. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
->readahead runs with every folio of the window locked, so the grant request it sent went out under those locks and a revoke of the window had to be given up on. The read now takes the grant before the page cache is entered; fill only what it covers. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
POSIX_FADV_WILLNEED and readahead(2) fill through ->readahead, which now fills only what a grant already covers. Take the grant for the advised range first, with no folio held. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A streamed write goes to the server out of the caller's pages, so its bytes are in no page cache and a revoke of the range finds nothing to flush. Hold the grant across the FUSE_WRITE, as the writethrough edges do. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
An O_DIRECT read or write neither fills nor dirties the page cache, so a grant over its range covers nothing and only conflicts with the rest of the cluster. Skip it on both sides. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_attr_cache_mask() decides whether the server's smaller size may be taken, then sleeps in the grant query before its answer is used. A write below EOF extends nothing, so it bumps neither fi->attr_version, which would drop the reply, nor fi->size_extenders, which would hold the size. Folios dirtied in the doomed range during that window are invisible to all three legs. Re-testing closer to the truncate is not enough on its own. truncate_pagecache() runs after fi->lock is dropped and holds nothing a writer holds, so a folio dirtied between the decision and the walk, or during the walk, is discarded anyway, with no error to report it: the bytes are gone and a later fsync() succeeds over the hole. A real truncate may discard the cache because fuse_set_nowrite() and i_rwsem hold the writers off; an attribute reply holds off nothing. So keep the folios, and re-test the size under fi->lock where nothing sleeps between the answer and its use. If the size was wrong the folios are written back and the size recovers, and fuse_flush_writepages() crops against a high water mark so nothing is clipped meanwhile. If it was right, the revoke that had to precede it already laundered and dropped the range, so there is nothing left to discard. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_writepage_need_send() takes its alignment bound from wbc->range_end, which a cyclic writeback does not set: write_cache_pages() runs to the end of the mapping there and leaves the field at zero. The bound is then zero as well, the test is true for every aligned index, and each folio is sent as a request of its own. Run to the end of the mapping when the pass is cyclic. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> [hbi: 7.0 keeps this cut in fuse_writeback_reached_alignment(), so only the cyclic bound moves over]
fuse_writeback_kick_stream() reads fi->write_stream_start twice in the same decision, once to round down into the start of the range and again to compare against its end. A writer moving the mark in between inverts the range the kick is given. Racing writers are meant to cost a kick rather than correctness, which needs each mark read once into a local. Take the end from this write's own position rather than reading back the mark just stored, and annotate the accesses. Still no lock: the run is a hint, and nothing waits for it. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The notify-driven latch required a local writer. Under the same storm a reader-only inode refills the page cache between two invalidations and has it dropped again before it can be read twice, so latch on any local opener and let the whole-mapping drop free the folios. The average behind the latch only folds on arrival, so an inode that stops being notified would stay uncached. Clear the latch at the top of the IO paths once the last invalidation is FUSE_NOTIFY_DIO_COLD old, and on release only for a writer. Keep the mapping empty while latched: splice reads copy through ->read_iter, readahead declines. The whole-mapping drop launders only what the mapping says may be dirty. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A read of the same buffer size arriving over and over is a reader working through the file a record at a time, and the folios it fills are dropped unread. Serve a streamed read of FUSE_READ_STREAM_MIN or more into the caller's own pages: one copy instead of two, no folios, and no read grant. What that gives up is the readahead of the next record, not the wait for this one. The detector is the one the write side uses, taking the field pair now so reads keep their own average, and the size is folded for every read that could be cached. Dirty folios over the range are sent first: a direct read does not look in the page cache, and fuse_direct_io() flushes only for a file opened FOPEN_DIRECT_IO. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_set_nowrite() biases an inode-wide counter and asserts BUG_ON(fi->writectr < 0) under fi->lock, which holds only with i_rwsem taken exclusive. Parallel direct writes hold it shared, so two of them reaching fuse_sync_writes() in fuse_direct_io() kill the second inside the spinlock, and every later user of the inode spins on it. Wait on the folios of the caller's own range instead when the lock is shared, which is the range the writeback test above already asked about. Reachable from the streamed write, and from any parallel write over a page cache that has not drained once the direct-IO latch is set. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The notify that sets the latch drops the mapping without an inode lock, so a cached write already past both latch checks keeps dirtying behind it. Writeback then puts those folios on the server on top of the direct writes that replace them, and direct reads miss them entirely. The per-range flush in the reroute paths covers only the range it is about to write, and only until the next dirty. Take i_rwsem exclusive once per latch instead. fuse_cache_write_iter() dirties under it, so holding it means every such writer has finished, and one that takes the lock afterwards rechecks the latch and reroutes before touching the cache. Flush and drop the mapping there and record it in FUSE_I_FORCE_DIO_DRAINED, which is cleared with the latch. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The cold check cleared the latch on a quiet inode even with a writer still on it, and nothing tracks the forced direct writes in flight, so a cached write could start beside one. Require write_files to be empty as well, which is what the last writer's release already waits for. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
The three paths that leave the forced direct IO latch dropped the mapping without writing it back first, so a folio a write racing the latch left dirty reached the server through invalidate_inode_pages2(), which launders one FUSE_WRITE per page. Send those with writeback, which batches them into max_write requests, and keep the error on the mapping for fsync. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
Currently fuse_iomap_writeback_submit discards the return code from fuse_dlm_regrant_range. Store the error in data->defer_err and return it to the caller. This can be used for sync write backs to know if it should continue waiting for the grant, or if the grant failed. Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
fuse_writepages() loops FUSE_WB_DEFER_PASSES times waiting for the deferred runs of a sync pass. If none of the passes manages to send them, it returns the rc of the last pass, which is 0. Under a storm, four passes may not be enough. close() calls fuse_flush(), which calls write_inode_now(); when that returns 0 with runs still deferred, close() returns 0, fuse_release() then unlinks the handle, and the dirty folios are left with an empty fi->write_files. Every later writeback of them (the flusher, a read-only handle's flush, the notify handler's launder) returns -EIO. Go round until a pass has nothing left to take back. Each pass is paced by the regrant, which the server answers once the revoke that refused the runs is done, and a regrant that fails is now reported from the submit instead of ignored, so the loop ends on it with the error visible to fsync and close. Only the silent case goes away. filp_close() releases the handle whatever ->flush returns, so a pass that ends on an error still leaves its folios dirty with no handle to send them from; what it no longer does is report those bytes written. The loop is bounded by the regrant rather than by a count, so end it on a fatal signal too, or a grant taken away as often as it is given leaves close() unkillable. Signed-off-by: Allison Henderson <allison.henderson@ddn.com> Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
A run whose grant was draining was reported to iomap as a hole and the folio redirtied only later, from fuse_writeback_redirty_done() at the next skipped folio or at submit. By then iomap_writeback_folio() had cleared the folio, ended its writeback and iomap_writepages() had unlocked it. Reclaim could not take it (the reference blocks the freeze), but invalidate_inode_pages2_range() does not look at references, and the revoke's own invalidation removed the folio in that window; folio_mark_dirty() then found no mapping and the bytes were gone with no error anywhere. Tracing showed the window open on every skipped run (32229 of 32232 deferred redirties found the folio clean) and folios dropped inside it at the offsets IOR reported. Count the deferred run on the folio as pending writeback, the way a queued run is, and report it back only after the folio is dirty again. A folio under writeback is neither reclaimed nor invalidated, and the revoke waits for it as it would for a send. Errors a run hits (no writer handle, a failed regrant, no memory) are deferred the same way and reported from the submit, rather than returned to iomap, which would end the folio's writeback with the bytes still to send. No writer handle is one of those deferrals, so take the handle with __fuse_write_file_get(): fuse_write_file_get()'s WARN_ON(!ff) fires on a state this path handles, and it has no other caller. Signed-off-by: Allison Henderson <allison.henderson@ddn.com> Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com> [hbi: 7.0 pre-charges write_bytes_pending with the whole folio in iomap_writeback_folio(), so the deferred run needs no charge of its own; iomap counts it submitted and leaves it to the finish below]
The unaligned head and tail of a buffered write went straight to the server so no partly written block was ever dirtied, at a synchronous request per edge. Cache the write whole instead and let iomap complete a partly covered block by reading the remainder back. The read is served under the write grant, which is page-rounded and exclusive, and a revoke drops the folio with it, so the block cannot reach the server stale. It goes out on the handle being written, which the server has to serve on a write-only open. fuse_perform_write() loses its cache flag with its last caller. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
Buffered writes under DLM share i_rwsem, so two of them over the same bytes are held apart by the folio lock alone and interleave a folio at a time. The grant does not order them: it is the node's, and every writer here holds the same one. Keep the extent of each write on a per inode list and wait for the overlapping entries published before it. Disjoint writers never wait. The list is node local and no revoke path consults it, so it may be held across a folio lock, a read back and a grant request alike. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_dlm_pin_write() and __fuse_get_dlm_lock() gave up after 16 passes and returned -EIO. Both counts are reached only by notifications arriving faster than the range is confirmed, which is contention and nothing worse, and the caller cannot tell the result from a real error. A write reports it to a task that has no reason to expect it. Ask again for as long as it takes. Every pass is a round trip, so the loops are paced by the server rather than spinning, and no caller holds a folio while a request is out. A fatal signal ends both, so a killed task and close() still get out. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
Every IO site spelled the same five lines: pin, confirm, unpin again if the range turned out uncovered. The order is the whole point and is easy to get wrong, and a pin left on a range no grant covers holds a revoke up for nothing. Fold it into fuse_dlm_trypin_held() and its span form and make the raw trypins static, so the header no longer offers a pin without a confirmation. fuse_dlm_pin() becomes the waiting trypin it always was. No change in behaviour. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
__fuse_get_dlm_lock() returns from the record without sending anything whenever the range is already covered, which is every writeback run and every folio of a write. It still built the request frame first: struct fuse_args, the in and out arguments and the pending node come to 392 bytes of stack, 136 of which gcc zeroes before the covered test is even reached. Move the request into fuse_dlm_send_lock() and keep it out of line. The lookup path is 24 bytes of stack and no zeroing. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_iomap_writeback_range() confirmed the grant under its pin and then called fuse_dlm_regrant_range() over the same range in the same mode, which can only find it held: a revoke drains the pins before fuse_dlm_unlock_range() removes anything, and one already draining would have refused the pin. So it was a second cache lock round per folio for an answer that cannot differ. The rule it stood for, never send bytes under a grant the server took away, is what the pin enforces. The submit side keeps its regrant: that one runs with no folio held and does ask. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_read_grant() asked for the read's range plus one readahead window,
and the fill paths decline whatever a grant does not cover. The window
readahead submits for a read is not the one the read sits in but the
next: page_cache_async_ra() fires on the marker of the current window
[s, s + W) and asks for [s + W, s + 2W), which ends up to W - count past
a grant of one window beyond the read. With a server that grants
exactly what is asked (redfs, memfs since 3fccf277) nearly the whole of
every readahead window was declined, given back unfilled, and refetched
one folio at a time by ->read_folio, each a synchronous request.
And what a read needs is not what it should ask for: the need moves
with every read, so a grant sized to it never covers the next read and
an exact server is asked once per read(2).
Cover two windows past the read, ask only when that is not already
held, and then ask for two windows more, so one grant carries the reads
across them. 32k-record stream, 128 MiB read, exact grants, np=1:
pages read one at a time requests MiB/s
one window past the read (before) 29024 of 32768 33123 85
two windows, asked on every read 0 4099 496
two windows, asked when not held 0 7 1800
wide grants (start..EOF) 0 4 1900
previous head 0 18 2000
A server that grants wider records more and is asked no more often than
before.
Signed-off-by: Allison Henderson <allison.henderson@ddn.com>
Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
__fuse_get_dlm_lock() waits for a contended range for as long as it takes, with a fatal signal the only way out. A writeback kworker never gets one, so a range revoked as fast as it is granted parks the flusher in fuse_dlm_regrant_range(). Nothing is coming for an inode the connection has given up on, so end the wait there as every other IO path on it does. fuse_dlm_pin_write() and fuse_writepages() ask through the same function and end with it. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
fuse_change_attributes_i() asks whether the range an attribute reply is about to truncate still holds unsent bytes, with fi->lock held. The lock excludes nothing the question is about, since a folio is dirtied without it, and a large clean range is walked a cached folio at a time under a spinlock. Ask before taking the lock, and record the i_size the walk was bounded by so the decision under the lock applies it to that range only. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
POSIX_FADV_WILLNEED, and readahead(2) with it, took the grant its readahead fills under and populated the window whatever came back. Without the grant every folio of that window is declined at the fill and dropped again, so report the failure and leave the page cache alone. -ENOSYS is a server with no DLM, and a positive return a grant the client could not record; neither ends the populate. The prefetching callers keep dropping it, which fuse_read_grant() now says: the fill confirms the grant itself and has a folio to report a failure on. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.