linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [GIT PULL] vfs netfs
@ 2024-09-13 16:56 Christian Brauner
  2024-09-16 10:28 ` Linus Torvalds
                   ` (6 more replies)
  0 siblings, 7 replies; 26+ messages in thread
From: Christian Brauner @ 2024-09-13 16:56 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: Christian Brauner, linux-fsdevel, linux-kernel

Hey Linus,

/* Summary */

This contains the work to improve read/write performance for the new
netfs library.

The main performance enhancing changes are:

    - Define a structure, struct folio_queue, and a new iterator type,
      ITER_FOLIOQ, to hold a buffer as a replacement for ITER_XARRAY. See
      that patch for questions about naming and form.

      ITER_FOLIOQ is provided as a replacement for ITER_XARRAY. The
      problem with an xarray is that accessing it requires the use of a
      lock (typically the RCU read lock) - and this means that we can't
      supply iterate_and_advance() with a step function that might sleep
      (crypto for example) without having to drop the lock between
      pages. ITER_FOLIOQ is the iterator for a chain of folio_queue
      structs, where each folio_queue holds a small list of folios. A
      folio_queue struct is a simpler structure than xarray and is not
      subject to concurrent manipulation by the VM. folio_queue is used
      rather than a bvec[] as it can form lists of indefinite size,
      adding to one end and removing from the other on the fly.

    - Provide a copy_folio_from_iter() wrapper.

    - Make cifs RDMA support ITER_FOLIOQ.

    - Use folio queues in the write-side helpers instead of xarrays.

    - Add a function to reset the iterator in a subrequest.

    - Simplify the write-side helpers to use sheaves to skip gaps rather
      than trying to work out where gaps are.

    - In afs, make the read subrequests asynchronous, putting them into work
      items to allow the next patch to do progressive unlocking/reading.

    - Overhaul the read-side helpers to improve performance.

    - Fix the caching of a partial block at the end of a file.

    - Allow a store to be cancelled.

Then some changes for cifs to make it use folio queues instead of
xarrays for crypto bufferage:

    - Use raw iteration functions rather than manually coding iteration when
      hashing data.

    - Switch to using folio_queue for crypto buffers.

    - Remove the xarray bits.

Make some adjustments to the /proc/fs/netfs/stats file such that:

    - All the netfs stats lines begin 'Netfs:' but change this to something
      a bit more useful.

    - Add a couple of stats counters to track the numbers of skips and waits
      on the per-inode writeback serialisation lock to make it easier to
      check for this as a source of performance loss.

Miscellaneous work:

    - Ensure that the sb_writers lock is taken around
      vfs_{set,remove}xattr() in the cachefiles code.

    - Reduce the number of conditional branches in netfs_perform_write().

    - Move the CIFS_INO_MODIFIED_ATTR flag to the netfs_inode struct and
      remove cifs_post_modify().

    - Move the max_len/max_nr_segs members from netfs_io_subrequest to
      netfs_io_request as they're only needed for one subreq at a time.

    - Add an 'unknown' source value for tracing purposes.

    - Remove NETFS_COPY_TO_CACHE as it's no longer used.

    - Set the request work function up front at allocation time.

    - Use bh-disabling spinlocks for rreq->lock as cachefiles completion may
      be run from block-filesystem DIO completion in softirq context.

    - Remove fs/netfs/io.c.

/* Testing */

gcc version 14.2.0 (Debian 14.2.0-3)
Debian clang version 16.0.6 (27+b1)

All patches are based on the vfs-6.11-rc7.fixes merge to bring in prerequisite
fixes in individual filesystems. All of this has been sitting in linux-next. No
build failures or warnings were observed.

/* Conflicts */

Merge conflicts with mainline
=============================

No known merge conflicts.

This has now a merge conflict with main due to some rather late cifs fixes.
This can be resolved by:

git rm fs/netfs/io.c

and then:

diff --cc fs/smb/client/cifssmb.c
index cfae2e918209,04f2a5441a89..d0df0c17b18f
--- a/fs/smb/client/cifssmb.c
+++ b/fs/smb/client/cifssmb.c
@@@ -1261,16 -1261,6 +1261,14 @@@ openRetry
        return rc;
  }

 +static void cifs_readv_worker(struct work_struct *work)
 +{
 +      struct cifs_io_subrequest *rdata =
 +              container_of(work, struct cifs_io_subrequest, subreq.work);
 +
-       netfs_subreq_terminated(&rdata->subreq,
-                               (rdata->result == 0 || rdata->result == -EAGAIN) ?
-                               rdata->got_bytes : rdata->result, true);
++      netfs_read_subreq_terminated(&rdata->subreq, rdata->result, false);
 +}
 +
  static void
  cifs_readv_callback(struct mid_q_entry *mid)
  {
@@@ -1323,21 -1306,11 +1321,23 @@@
                rdata->result = -EIO;
        }

 -      if (rdata->result == 0 || rdata->result == -EAGAIN)
 -              iov_iter_advance(&rdata->subreq.io_iter, rdata->got_bytes);
 +      if (rdata->result == -ENODATA) {
 +              __set_bit(NETFS_SREQ_HIT_EOF, &rdata->subreq.flags);
 +              rdata->result = 0;
 +      } else {
-               if (rdata->got_bytes < rdata->actual_len &&
-                   rdata->subreq.start + rdata->subreq.transferred + rdata->got_bytes ==
-                   ictx->remote_i_size) {
++              size_t trans = rdata->subreq.transferred + rdata->got_bytes;
++              if (trans < rdata->subreq.len &&
++                  rdata->subreq.start + trans == ictx->remote_i_size) {
 +                      __set_bit(NETFS_SREQ_HIT_EOF, &rdata->subreq.flags);
 +                      rdata->result = 0;
 +              }
 +      }
 +
        rdata->credits.value = 0;
+       rdata->subreq.transferred += rdata->got_bytes;
 -      netfs_read_subreq_terminated(&rdata->subreq, rdata->result, false);
++      trace_netfs_sreq(&rdata->subreq, netfs_sreq_trace_io_progress);
 +      INIT_WORK(&rdata->subreq.work, cifs_readv_worker);
 +      queue_work(cifsiod_wq, &rdata->subreq.work);
        release_mid(mid);
        add_credits(server, &credits, 0);
  }
diff --cc fs/smb/client/smb2pdu.c
index 88dc49d67037,95377bb91950..bb8ecbbe78af
--- a/fs/smb/client/smb2pdu.c
+++ b/fs/smb/client/smb2pdu.c
@@@ -4614,6 -4613,10 +4613,8 @@@ smb2_readv_callback(struct mid_q_entry
                              server->credits, server->in_flight,
                              0, cifs_trace_rw_credits_read_response_clear);
        rdata->credits.value = 0;
+       rdata->subreq.transferred += rdata->got_bytes;
 -      if (rdata->subreq.start + rdata->subreq.transferred >= rdata->subreq.rreq->i_size)
 -              __set_bit(NETFS_SREQ_HIT_EOF, &rdata->subreq.flags);
+       trace_netfs_sreq(&rdata->subreq, netfs_sreq_trace_io_progress);
        INIT_WORK(&rdata->subreq.work, smb2_readv_worker);
        queue_work(cifsiod_wq, &rdata->subreq.work);
        release_mid(mid);

Merge conflicts with other trees
================================

No known merge conflicts.

The following changes since commit 4356ab331c8f0dbed0f683abde345cd5503db1e4:

  Merge tag 'vfs-6.11-rc7.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs (2024-09-04 09:33:57 -0700)

are available in the Git repository at:

  git@gitolite.kernel.org:pub/scm/linux/kernel/git/vfs/vfs tags/vfs-6.12.netfs

for you to fetch changes up to 4b40d43d9f951d87ae8dc414c2ef5ae50303a266:

  docs: filesystems: corrected grammar of netfs page (2024-09-12 12:20:43 +0200)

Please consider pulling these changes from the signed vfs-6.12.netfs tag.

Thanks!
Christian

----------------------------------------------------------------
vfs-6.12.netfs

----------------------------------------------------------------
Christian Brauner (1):
      Merge branch 'netfs-writeback' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs into vfs.netfs

David Howells (24):
      cachefiles: Fix non-taking of sb_writers around set/removexattr
      netfs: Adjust labels in /proc/fs/netfs/stats
      netfs: Record contention stats for writeback lock
      netfs: Reduce number of conditional branches in netfs_perform_write()
      netfs, cifs: Move CIFS_INO_MODIFIED_ATTR to netfs_inode
      netfs: Move max_len/max_nr_segs from netfs_io_subrequest to netfs_io_stream
      netfs: Reserve netfs_sreq_source 0 as unset/unknown
      netfs: Remove NETFS_COPY_TO_CACHE
      netfs: Set the request work function upon allocation
      netfs: Use bh-disabling spinlocks for rreq->lock
      mm: Define struct folio_queue and ITER_FOLIOQ to handle a sequence of folios
      iov_iter: Provide copy_folio_from_iter()
      cifs: Provide the capability to extract from ITER_FOLIOQ to RDMA SGEs
      netfs: Use new folio_queue data type and iterator instead of xarray iter
      netfs: Provide an iterator-reset function
      netfs: Simplify the writeback code
      afs: Make read subreqs async
      netfs: Speed up buffered reading
      netfs: Remove fs/netfs/io.c
      cachefiles, netfs: Fix write to partial block at EOF
      netfs: Cancel dirty folios that have no storage destination
      cifs: Use iterate_and_advance*() routines directly for hashing
      cifs: Switch crypto buffer to use a folio_queue rather than an xarray
      cifs: Don't support ITER_XARRAY

Dennis Lam (1):
      docs: filesystems: corrected grammar of netfs page

 Documentation/filesystems/netfs_library.rst |   2 +-
 fs/9p/vfs_addr.c                            |  11 +-
 fs/afs/file.c                               |  30 +-
 fs/afs/fsclient.c                           |   9 +-
 fs/afs/write.c                              |   4 +-
 fs/afs/yfsclient.c                          |   9 +-
 fs/cachefiles/io.c                          |  19 +-
 fs/cachefiles/xattr.c                       |  34 +-
 fs/ceph/addr.c                              |  76 +--
 fs/netfs/Makefile                           |   4 +-
 fs/netfs/buffered_read.c                    | 766 ++++++++++++++++----------
 fs/netfs/buffered_write.c                   | 309 +++++------
 fs/netfs/direct_read.c                      | 147 ++++-
 fs/netfs/internal.h                         |  43 +-
 fs/netfs/io.c                               | 804 ----------------------------
 fs/netfs/iterator.c                         |  50 ++
 fs/netfs/main.c                             |   7 +-
 fs/netfs/misc.c                             |  94 ++++
 fs/netfs/objects.c                          |  16 +-
 fs/netfs/read_collect.c                     | 544 +++++++++++++++++++
 fs/netfs/read_pgpriv2.c                     | 264 +++++++++
 fs/netfs/read_retry.c                       | 256 +++++++++
 fs/netfs/stats.c                            |  27 +-
 fs/netfs/write_collect.c                    | 246 +++------
 fs/netfs/write_issue.c                      |  93 ++--
 fs/nfs/fscache.c                            |  19 +-
 fs/nfs/fscache.h                            |   7 +-
 fs/smb/client/cifsencrypt.c                 | 144 +----
 fs/smb/client/cifsglob.h                    |   4 +-
 fs/smb/client/cifssmb.c                     |   6 +-
 fs/smb/client/file.c                        |  96 ++--
 fs/smb/client/smb2ops.c                     | 219 ++++----
 fs/smb/client/smb2pdu.c                     |  27 +-
 fs/smb/client/smbdirect.c                   |  82 +--
 include/linux/folio_queue.h                 | 156 ++++++
 include/linux/iov_iter.h                    | 104 ++++
 include/linux/netfs.h                       |  46 +-
 include/linux/uio.h                         |  18 +
 include/trace/events/netfs.h                | 144 +++--
 lib/iov_iter.c                              | 240 ++++++++-
 lib/kunit_iov_iter.c                        | 259 +++++++++
 lib/scatterlist.c                           |  69 ++-
 42 files changed, 3520 insertions(+), 1984 deletions(-)
 delete mode 100644 fs/netfs/io.c
 create mode 100644 fs/netfs/read_collect.c
 create mode 100644 fs/netfs/read_pgpriv2.c
 create mode 100644 fs/netfs/read_retry.c
 create mode 100644 include/linux/folio_queue.h

^ permalink raw reply	[flat|nested] 26+ messages in thread
* [GIT PULL] vfs netfs
@ 2025-06-02 10:11 Christian Brauner
  2025-06-02 22:58 ` pr-tracker-bot
  0 siblings, 1 reply; 26+ messages in thread
From: Christian Brauner @ 2025-06-02 10:11 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: Christian Brauner, linux-fsdevel, linux-kernel

Hey Linus,

A quick word on the general timing for netfs pull requests going
forward. In contrast to most of the other work I tend to send the netfs
updates a little later and will probably continue to do so in the
future. In the past we had some issues with late-stage fixes or there
were complicated merge conflicts with other trees or mainline that were
a little unusal. So since there's more jitter in how consistently this
tree can be carried through a cycle I'm letting this reflect in the pull
request timing as well.

/* Summary */

This contains the netfs updates for this cycle:

- The main API document has been extensively updated/rewritten.

- Fix an oops in write-retry due to mis-resetting the I/O iterator.

- Fix the recording of transferred bytes for short DIO reads.

- Fix a request's work item to not require a reference, thereby avoiding
  the need to get rid of it in BH/IRQ context.

- Fix waiting and waking to be consistent about the waitqueue used.

- Remove NETFS_SREQ_SEEK_DATA_READ.

- Remove NETFS_INVALID_WRITE.

- Remove NETFS_ICTX_WRITETHROUGH.

- Remove NETFS_READ_HOLE_CLEAR.

- Reorder structs to eliminate holes.

- Remove netfs_io_request::ractl.

- Only provide proc_link field if CONFIG_PROC_FS=y.

- Remove folio_queue::marks3.

- Remove NETFS_RREQ_DONT_UNLOCK_FOLIOS.

- Remove NETFS_RREQ_BLOCKED.

- Fix undifferentiation of DIO reads from unbuffered reads.

generally sent
after vfs-6.16-rc1.misc

/* Testing */

gcc (Debian 14.2.0-19) 14.2.0
Debian clang version 19.1.7 (3)

No build failures or warnings were observed.

/* Conflicts */

Merge conflicts with mainline
=============================

No known conflicts.

Merge conflicts with other trees
================================

No known conflicts.

The following changes since commit f1745496d3fba34a2e16ef47d78903d7208c1214:

  netfs: Update main API document (2025-04-11 15:23:50 +0200)

are available in the Git repository at:

  git@gitolite.kernel.org:pub/scm/linux/kernel/git/vfs/vfs tags/vfs-6.16-rc1.netfs

for you to fetch changes up to db26d62d79e4068934ad0dccdb92715df36352b9:

  netfs: Fix undifferentiation of DIO reads from unbuffered reads (2025-05-23 10:35:03 +0200)

Please consider pulling these changes from the signed vfs-6.16-rc1.netfs tag.

Thanks!
Christian

----------------------------------------------------------------
vfs-6.16-rc1.netfs

----------------------------------------------------------------
Christian Brauner (3):
      Merge netfs API documentation updates
      Merge patch series "netfs: Miscellaneous cleanups"
      Merge patch series "netfs: Miscellaneous fixes"

David Howells (4):
      netfs: Fix oops in write-retry from mis-resetting the subreq iterator
      netfs: Fix the request's work item to not require a ref
      netfs: Fix wait/wake to be consistent about the waitqueue used
      netfs: Fix undifferentiation of DIO reads from unbuffered reads

Max Kellermann (10):
      fs/netfs: remove unused flag NETFS_SREQ_SEEK_DATA_READ
      fs/netfs: remove unused source NETFS_INVALID_WRITE
      fs/netfs: remove unused flag NETFS_ICTX_WRITETHROUGH
      fs/netfs: remove unused enum choice NETFS_READ_HOLE_CLEAR
      fs/netfs: reorder struct fields to eliminate holes
      fs/netfs: remove `netfs_io_request.ractl`
      fs/netfs: declare field `proc_link` only if CONFIG_PROC_FS=y
      folio_queue: remove unused field `marks3`
      fs/netfs: remove unused flag NETFS_RREQ_DONT_UNLOCK_FOLIOS
      fs/netfs: remove unused flag NETFS_RREQ_BLOCKED

Paulo Alcantara (1):
      netfs: Fix setting of transferred bytes with short DIO reads

 Documentation/core-api/folio_queue.rst      |   3 -
 Documentation/filesystems/netfs_library.rst |   5 -
 fs/9p/vfs_addr.c                            |   5 +-
 fs/afs/write.c                              |   9 +-
 fs/cachefiles/io.c                          |  16 +-
 fs/ceph/addr.c                              |   6 +-
 fs/erofs/fscache.c                          |   6 +-
 fs/netfs/buffered_read.c                    |  56 ++++---
 fs/netfs/buffered_write.c                   |   5 +-
 fs/netfs/direct_read.c                      |  16 +-
 fs/netfs/direct_write.c                     |  12 +-
 fs/netfs/fscache_io.c                       |  10 +-
 fs/netfs/internal.h                         |  42 ++++--
 fs/netfs/main.c                             |   1 +
 fs/netfs/misc.c                             | 219 ++++++++++++++++++++++++++++
 fs/netfs/objects.c                          |  48 +++---
 fs/netfs/read_collect.c                     | 199 +++++--------------------
 fs/netfs/read_pgpriv2.c                     |   4 +-
 fs/netfs/read_retry.c                       |  26 +---
 fs/netfs/read_single.c                      |   6 +-
 fs/netfs/write_collect.c                    |  83 ++++-------
 fs/netfs/write_issue.c                      |  38 ++---
 fs/netfs/write_retry.c                      |  19 +--
 fs/nfs/fscache.c                            |   1 +
 fs/smb/client/cifsproto.h                   |   3 +-
 fs/smb/client/cifssmb.c                     |   4 +-
 fs/smb/client/file.c                        |  10 +-
 fs/smb/client/smb2pdu.c                     |   4 +-
 include/linux/folio_queue.h                 |  42 ------
 include/linux/fscache.h                     |   5 +-
 include/linux/netfs.h                       |  45 +++---
 include/trace/events/netfs.h                |  11 +-
 net/9p/client.c                             |   6 +-
 33 files changed, 478 insertions(+), 487 deletions(-)

^ permalink raw reply	[flat|nested] 26+ messages in thread
* [GIT PULL] vfs netfs
@ 2025-01-18 12:55 Christian Brauner
  2025-01-20 18:59 ` pr-tracker-bot
  0 siblings, 1 reply; 26+ messages in thread
From: Christian Brauner @ 2025-01-18 12:55 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: Christian Brauner, linux-fsdevel, linux-kernel

Hey Linus,

/* Summary */

This contains read performance improvements and support for monolithic
single-blob objects that have to be read/written as such (e.g. AFS
directory contents). The implementation of the two parts is interwoven
as each makes the other possible.

- Read performance improvements

  The read performance improvements are intended to speed up some loss
  of performance detected in cifs and to a lesser extend in afs. The
  problem is that we queue too many work items during the collection of
  read results: each individual subrequest is collected by its own work
  item, and then they have to interact with each other when a series of
  subrequests don't exactly align with the pattern of folios that are
  being read by the overall request.

  Whilst the processing of the pages covered by individual subrequests
  as they complete potentially allows folios to be woken in parallel and
  with minimum delay, it can shuffle wakeups for sequential reads out of
  order - and that is the most common I/O pattern.

  The final assessment and cleanup of an operation is then held up until
  the last I/O completes - and for a synchronous sequential operation,
  this means the bouncing around of work items just adds latency.

  Two changes have been made to make this work:

  (1) All collection is now done in a single "work item" that works
      progressively through the subrequests as they complete (and also
      dispatches retries as necessary).

  (2) For readahead and AIO, this work item be done on a workqueue and
      can run in parallel with the ultimate consumer of the data; for
      synchronous direct or unbuffered reads, the collection is run in
      the application thread and not offloaded.

  Functions such as smb2_readv_callback() then just tell netfslib that
  the subrequest has terminated; netfslib does a minimal bit of
  processing on the spot - stat counting and tracing mostly - and then
  queues/wakes up the worker.  This simplifies the logic as the
  collector just walks sequentially through the subrequests as they
  complete and walks through the folios, if buffered, unlocking them as
  it goes.  It also keeps to a minimum the amount of latency injected
  into the filesystem's low-level I/O handling

  The way netfs supports filesystems using the deprecated PG_private_2
  flag is changed: folios are flagged and added to a write request as
  they complete and that takes care of scheduling the writes to the
  cache.  The originating read request can then just unlock the pages
  whatever happens.

- Single-blob object support

  Single-blob objects are files for which the content of the file must
  be read from or written to the server in a single operation because
  reading them in parts may yield inconsistent results.  AFS directories
  are an example of this as there exists the possibility that the
  contents are generated on the fly and would differ between reads or
  might change due to third party interference.

  Such objects will be written to and retrieved from the cache if one is
  present, though we allow/may need to propose multiple subrequests to
  do so. The important part is that read from/write to the *server* is
  monolithic.

  Single blob reading is, for the moment, fully synchronous and does
  result collection in the application thread and, also for the moment,
  the API is supplied the buffer in the form of a folio_queue chain
  rather than using the pagecache.

- Related afs changes

  This series makes a number of changes to the kafs filesystem,
  primarily in the area of directory handling:

  - AFS's FetchData RPC reply processing is made partially asynchronous
    which allows the netfs_io_request's outstanding operation counter to
    be removed as part of reducing the collection to a single work item.

  - Directory and symlink reading are plumbed through netfslib using the
    single-blob object API and are now cacheable with fscache. This also
    allows the afs_read struct to be eliminated and netfs_io_subrequest
    to be used directly instead.

  - Directory and symlink content are now stored in a folio_queue buffer
    rather than in the pagecache.  This means we don't require the RCU
    read lock and xarray iteration to access it, and folios won't
    randomly disappear under us because the VM wants them back.

  - The vnode operation lock is changed from a mutex struct to a private
    lock implementation. The problem is that the lock now needs to be
    dropped in a separate thread and mutexes don't permit that.

  - When a new directory or symlink is created, we now initialise it
    locally and mark it valid rather than downloading it (we know what
    it's likely to look like).

  - We now use the in-directory hashtable to reduce the number of
    entries we need to scan when doing a lookup.  The edit routines have
    to maintain the hash chains.

  - Cancellation (e.g. by signal) of an async call after the rxrpc_call
    has been set up is now offloaded to the worker thread as there will
    be a notification from rxrpc upon completion.  This avoids a double
    cleanup.

- A "rolling buffer" implementation is created to abstract out the two
  separate folio_queue chaining implementations I had (one for read and
  one for write).

- Functions are provided to create/extend a buffer in a folio_queue
  chain and tear it down again.  This is used to handle AFS directories,
  but could also be used to create bounce buffers for content crypto and
  transport crypto.

- The was_async argument is dropped from netfs_read_subreq_terminated().
  Instead we wake the read collection work item by either queuing it or
  waking up the app thread.

- We don't need to use BH-excluding locks when communicating between the
  issuing thread and the collection thread as neither of them now run in
  BH context.

- Also included are a number of new tracepoints; a split of the netfslib
  write collection code to put retrying into its own file (it gets more
  complicated with content encryption).

- There are also some minor fixes AFS included, including fixing the AFS
  directory format struct layout, reducing some directory
  over-invalidation and making afs_mkdir() translate EEXIST to ENOTEMPY
  (which is not available on all systems the servers support).

- Finally, there's a patch to try and detect entry into the folio unlock
  function with no folio_queue structs in the buffer (which isn't
  allowed in the cases that can get there).  This is a debugging patch,
  but should be minimal overhead.

/* Testing */

gcc version 14.2.0 (Debian 14.2.0-6)
Debian clang version 16.0.6 (27+b1)

No build failures or warnings were observed.

/* Conflicts */

Merge conflicts with mainline
=============================

There's a merge conflict that can be resolved as follows:

diff --cc fs/netfs/direct_write.c
index f9421f3e6d37,eded8afaa60b..000000000000
--- a/fs/netfs/direct_write.c
+++ b/fs/netfs/direct_write.c
@@@ -67,8 -67,8 +67,8 @@@ ssize_t netfs_unbuffered_write_iter_loc
                 * allocate a sufficiently large bvec array and may shorten the
                 * request.
                 */
 -              if (async || user_backed_iter(iter)) {
 +              if (user_backed_iter(iter)) {
-                       n = netfs_extract_user_iter(iter, len, &wreq->iter, 0);
+                       n = netfs_extract_user_iter(iter, len, &wreq->buffer.iter, 0);
                        if (n < 0) {
                                ret = n;
                                goto out;
@@@ -77,15 -77,8 +77,13 @@@
                        wreq->direct_bv_count = n;
                        wreq->direct_bv_unpin = iov_iter_extract_will_pin(iter);
                } else {
 +                      /* If this is a kernel-generated async DIO request,
 +                       * assume that any resources the iterator points to
 +                       * (eg. a bio_vec array) will persist till the end of
 +                       * the op.
 +                       */
-                       wreq->iter = *iter;
+                       wreq->buffer.iter = *iter;
                }
-
-               wreq->io_iter = wreq->iter;
        }

        __set_bit(NETFS_RREQ_USE_IO_ITER, &wreq->flags);
diff --cc fs/netfs/read_retry.c
index 16b676c68dcd,bf6f26525b0d..000000000000
--- a/fs/netfs/read_retry.c
+++ b/fs/netfs/read_retry.c
@@@ -142,25 -125,19 +125,20 @@@ static void netfs_retry_read_subrequest
                        __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags);
                        subreq->retry_count++;

-                       spin_lock_bh(&rreq->lock);
-                       list_add_tail(&subreq->rreq_link, &rreq->subrequests);
-                       subreq->prev_donated += rreq->prev_donated;
-                       rreq->prev_donated = 0;
                        trace_netfs_sreq(subreq, netfs_sreq_trace_retry);
-                       spin_unlock_bh(&rreq->lock);
-
-                       BUG_ON(!len);

                        /* Renegotiate max_len (rsize) */
+                       stream->sreq_max_len = subreq->len;
 -                      if (rreq->netfs_ops->prepare_read(subreq) < 0) {
 +                      if (rreq->netfs_ops->prepare_read &&
 +                          rreq->netfs_ops->prepare_read(subreq) < 0) {
                                trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed);
                                __set_bit(NETFS_SREQ_FAILED, &subreq->flags);
+                               goto abandon;
                        }

-                       part = umin(len, stream0->sreq_max_len);
-                       if (unlikely(rreq->io_streams[0].sreq_max_segs))
-                               part = netfs_limit_iter(&source, 0, part, stream0->sreq_max_segs);
+                       part = umin(len, stream->sreq_max_len);
+                       if (unlikely(stream->sreq_max_segs))
+                               part = netfs_limit_iter(&source, 0, part, stream->sreq_max_segs);
                        subreq->len = subreq->transferred + part;
                        subreq->io_iter = source;
                        iov_iter_truncate(&subreq->io_iter, part);

Merge conflicts with other trees
================================

No known conflicts.

The following changes since commit 5fe85a5c513344161cde33b79f8badc81b8aa8d3:

  Merge patch series "netfs, ceph, nfs, cachefiles: Miscellaneous fixes/changes" (2024-12-20 22:08:16 +0100)

are available in the Git repository at:

  git@gitolite.kernel.org:pub/scm/linux/kernel/git/vfs/vfs tags/vfs-6.14-rc1.netfs

for you to fetch changes up to 7a47db23a9f003614e15c687d2a5425c175a9ca8:

  Merge patch series "netfs: Read performance improvements and "single-blob" support" (2024-12-20 22:34:18 +0100)

Please consider pulling these changes from the signed vfs-6.14-rc1.netfs tag.

Thanks!
Christian

----------------------------------------------------------------
vfs-6.14-rc1.netfs

----------------------------------------------------------------
Christian Brauner (1):
      Merge patch series "netfs: Read performance improvements and "single-blob" support"

David Howells (31):
      netfs: Clean up some whitespace in trace header
      cachefiles: Clean up some whitespace in trace header
      netfs: Use a folio_queue allocation and free functions
      netfs: Add a tracepoint to log the lifespan of folio_queue structs
      netfs: Abstract out a rolling folio buffer implementation
      netfs: Make netfs_advance_write() return size_t
      netfs: Split retry code out of fs/netfs/write_collect.c
      netfs: Drop the error arg from netfs_read_subreq_terminated()
      netfs: Drop the was_async arg from netfs_read_subreq_terminated()
      netfs: Don't use bh spinlock
      afs: Don't use mutex for I/O operation lock
      afs: Fix EEXIST error returned from afs_rmdir() to be ENOTEMPTY
      afs: Fix directory format encoding struct
      netfs: Remove some extraneous directory invalidations
      cachefiles: Add some subrequest tracepoints
      cachefiles: Add auxiliary data trace
      afs: Add more tracepoints to do with tracking validity
      netfs: Add functions to build/clean a buffer in a folio_queue
      netfs: Add support for caching single monolithic objects such as AFS dirs
      afs: Make afs_init_request() get a key if not given a file
      afs: Use netfslib for directories
      afs: Use netfslib for symlinks, allowing them to be cached
      afs: Eliminate afs_read
      afs: Fix cleanup of immediately failed async calls
      afs: Make {Y,}FS.FetchData an asynchronous operation
      netfs: Change the read result collector to only use one work item
      afs: Make afs_mkdir() locally initialise a new directory's content
      afs: Use the contained hashtable to search a directory
      afs: Locally initialise the contents of a new symlink on creation
      afs: Add a tracepoint for afs_read_receive()
      netfs: Report on NULL folioq in netfs_writeback_unlock_folios()

 fs/9p/vfs_addr.c                  |   6 +-
 fs/afs/Makefile                   |   1 +
 fs/afs/callback.c                 |   4 +-
 fs/afs/dir.c                      | 809 ++++++++++++++++++++------------------
 fs/afs/dir_edit.c                 | 383 ++++++++++--------
 fs/afs/dir_search.c               | 227 +++++++++++
 fs/afs/file.c                     | 260 ++++++------
 fs/afs/fs_operation.c             | 113 +++++-
 fs/afs/fsclient.c                 |  62 +--
 fs/afs/inode.c                    | 140 ++++++-
 fs/afs/internal.h                 | 143 +++++--
 fs/afs/main.c                     |   2 +-
 fs/afs/mntpt.c                    |  22 +-
 fs/afs/rotate.c                   |   4 +-
 fs/afs/rxrpc.c                    |  37 +-
 fs/afs/super.c                    |   4 +-
 fs/afs/validation.c               |  31 +-
 fs/afs/vlclient.c                 |   1 +
 fs/afs/write.c                    |  16 +-
 fs/afs/xdr_fs.h                   |   2 +-
 fs/afs/yfsclient.c                |  49 +--
 fs/cachefiles/io.c                |   4 +
 fs/cachefiles/xattr.c             |   9 +-
 fs/ceph/addr.c                    |  22 +-
 fs/netfs/Makefile                 |   5 +-
 fs/netfs/buffered_read.c          | 290 +++++---------
 fs/netfs/direct_read.c            |  78 ++--
 fs/netfs/direct_write.c           |  10 +-
 fs/netfs/internal.h               |  41 +-
 fs/netfs/main.c                   |   6 +-
 fs/netfs/misc.c                   | 164 ++++----
 fs/netfs/objects.c                |  21 +-
 fs/netfs/read_collect.c           | 761 +++++++++++++++++++++--------------
 fs/netfs/read_pgpriv2.c           | 207 ++++------
 fs/netfs/read_retry.c             | 209 +++++-----
 fs/netfs/read_single.c            | 195 +++++++++
 fs/netfs/rolling_buffer.c         | 226 +++++++++++
 fs/netfs/stats.c                  |   4 +-
 fs/netfs/write_collect.c          | 281 +++----------
 fs/netfs/write_issue.c            | 241 +++++++++++-
 fs/netfs/write_retry.c            | 232 +++++++++++
 fs/nfs/fscache.c                  |   6 +-
 fs/nfs/fscache.h                  |   3 +-
 fs/smb/client/cifssmb.c           |  12 +-
 fs/smb/client/file.c              |   3 +-
 fs/smb/client/smb2ops.c           |   2 +-
 fs/smb/client/smb2pdu.c           |  15 +-
 include/linux/folio_queue.h       |  12 +-
 include/linux/netfs.h             |  54 ++-
 include/linux/rolling_buffer.h    |  61 +++
 include/trace/events/afs.h        | 210 +++++++++-
 include/trace/events/cachefiles.h | 185 ++++-----
 include/trace/events/netfs.h      | 229 +++++------
 lib/kunit_iov_iter.c              |   4 +-
 54 files changed, 3911 insertions(+), 2207 deletions(-)
 create mode 100644 fs/afs/dir_search.c
 create mode 100644 fs/netfs/read_single.c
 create mode 100644 fs/netfs/rolling_buffer.c
 create mode 100644 fs/netfs/write_retry.c
 create mode 100644 include/linux/rolling_buffer.h

^ permalink raw reply	[flat|nested] 26+ messages in thread
* [GIT PULL] vfs netfs
@ 2024-11-15 14:00 Christian Brauner
  2024-11-18 18:29 ` Linus Torvalds
  2024-11-18 19:49 ` pr-tracker-bot
  0 siblings, 2 replies; 26+ messages in thread
From: Christian Brauner @ 2024-11-15 14:00 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: Christian Brauner, linux-fsdevel, linux-kernel

Hey Linus,

A pidfs patch ended up in the branch and I didn't notice it. I decided
to leave it in here instead of rebasing the whole branch.

/* Summary */

This contains various fixes for the netfs library and related
filesystems and infrastructure:

afs:

    - Fix missing wire-up of afs_retry_request().

    - Fix the setting of the server responding flag in afs.

    - Fix possible infinite loop with unresponsive servers.

    - Remove unused struct and function prototype.

cachefiles:

    - Fix a dentry leak in cachefiles_open_file().

    - Fix incorrect length return value in cachefiles_ondemand_fd_write_iter().

    - Fix missing pos updates in cachefiles_ondemand_fd_write_iter().

    - Clean up in cachefiles_commit_tmpfile().

    - Fix NULL pointer dereference in object->file.

    - Add a memory barrier for FSCACHE_VOLUME_CREATING.

netfs:

    - Advance iterator correctly rather than jumping it.

    - Add folio_queue API documentation.

    - Fix the netfs_folio tracepoint to handle NULL mapping.

    - Remove call to folio_index().

    - Fix a few minor bugs in netfs_page_mkwrite().

    - Remove unnecessary references to pages.

pidfs:

    - Check for valid pid namespace.

/* Testing */

gcc version 14.2.0 (Debian 14.2.0-6)
Debian clang version 16.0.6 (27+b1)

All patches are based on v6.12-rc1 and have been sitting in linux-next.
No build failures or warnings were observed.

/* Conflicts */

Merge conflicts with mainline
=============================

No known conflicts.

Merge conflicts with other trees
================================

No known conflicts.

The following changes since commit 9852d85ec9d492ebef56dc5f229416c925758edc:

  Linux 6.12-rc1 (2024-09-29 15:06:19 -0700)

are available in the Git repository at:

  git@gitolite.kernel.org:pub/scm/linux/kernel/git/vfs/vfs tags/vfs-6.13.netfs

for you to fetch changes up to a4b2923376be062a243ac38762212a38485cfab1:

  Merge patch series "fscache/cachefiles: Some bugfixes" (2024-11-11 14:39:39 +0100)

Please consider pulling these changes from the signed vfs-6.13.netfs tag.

Thanks!
Christian

----------------------------------------------------------------
vfs-6.13.netfs

----------------------------------------------------------------
Baokun Li (1):
      cachefiles: fix dentry leak in cachefiles_open_file()

Christian Brauner (3):
      pidfs: check for valid pid namespace
      Merge patch series "Random netfs folio fixes"
      Merge patch series "fscache/cachefiles: Some bugfixes"

David Howells (5):
      afs: Fix missing wire-up of afs_retry_request()
      afs: Fix the setting of the server responding flag
      netfs: Advance iterator correctly rather than jumping it
      netfs: Add folio_queue API documentation
      netfs: Fix the netfs_folio tracepoint to handle NULL mapping

Marc Dionne (1):
      afs: Fix possible infinite loop with unresponsive servers

Matthew Wilcox (Oracle) (3):
      netfs: Remove call to folio_index()
      netfs: Fix a few minor bugs in netfs_page_mkwrite()
      netfs: Remove unnecessary references to pages

Thorsten Blum (1):
      afs: Remove unused struct and function prototype

Zizhi Wo (5):
      cachefiles: Fix incorrect length return value in cachefiles_ondemand_fd_write_iter()
      cachefiles: Fix missing pos updates in cachefiles_ondemand_fd_write_iter()
      cachefiles: Clean up in cachefiles_commit_tmpfile()
      cachefiles: Fix NULL pointer dereference in object->file
      netfs/fscache: Add a memory barrier for FSCACHE_VOLUME_CREATING

 Documentation/core-api/folio_queue.rst | 212 +++++++++++++++++++++++++++++++++
 fs/afs/afs_vl.h                        |   9 --
 fs/afs/file.c                          |   1 +
 fs/afs/fs_operation.c                  |   2 +-
 fs/afs/fs_probe.c                      |   4 +-
 fs/afs/rotate.c                        |  11 +-
 fs/cachefiles/interface.c              |  14 ++-
 fs/cachefiles/namei.c                  |  12 +-
 fs/cachefiles/ondemand.c               |  38 ++++--
 fs/netfs/buffered_read.c               |   8 +-
 fs/netfs/buffered_write.c              |  41 ++++---
 fs/netfs/fscache_volume.c              |   3 +-
 fs/netfs/write_issue.c                 |  12 +-
 fs/pidfs.c                             |   5 +-
 include/linux/folio_queue.h            | 168 ++++++++++++++++++++++++++
 include/trace/events/netfs.h           |   5 +-
 16 files changed, 475 insertions(+), 70 deletions(-)
 create mode 100644 Documentation/core-api/folio_queue.rst

^ permalink raw reply	[flat|nested] 26+ messages in thread
* [GIT PULL] vfs netfs
@ 2024-05-10 11:47 Christian Brauner
  2024-05-13 19:38 ` pr-tracker-bot
  0 siblings, 1 reply; 26+ messages in thread
From: Christian Brauner @ 2024-05-10 11:47 UTC (permalink / raw)
  To: Linus Torvalds; +Cc: Christian Brauner, linux-fsdevel, linux-kernel

Hey Linus,

/* Summary */
This reworks the netfslib writeback implementation so that pages read
from the cache are written to the cache through ->writepages(), thereby
allowing the fscache page flag to be retired.

The reworking also:

 (1) builds on top of the new writeback_iter() infrastructure;

 (2) makes it possible to use vectored write RPCs as discontiguous streams
     of pages can be accommodated;

 (3) makes it easier to do simultaneous content crypto and stream division.

 (4) provides support for retrying writes and re-dividing a stream;

 (5) replaces the ->launder_folio() op, so that ->writepages() is used
     instead;

 (6) uses mempools to allocate the netfs_io_request and netfs_io_subrequest
     structs to avoid allocation failure in the writeback path.

Some code that uses the fscache page flag is retained for compatibility
purposes with nfs and ceph. The code is switched to using the synonymous
private_2 label instead and marked with deprecation comments.

The merge commit contains additional details on the new algorithm that
I've left out of here as it would probably be excessively detailed.

On top of the netfslib infrastructure this contains the work to convert
cifs over to netfslib.

/* Testing */
clang: Debian clang version 16.0.6 (26)
gcc: (Debian 13.2.0-24)

All patches are based on v6.9-rc6 and have been sitting in linux-next.
No build failures or warnings were observed.

/* Conflicts */

No known conflicts.

The following changes since commit e67572cd2204894179d89bd7b984072f19313b03:

  Linux 6.9-rc6 (2024-04-28 13:47:24 -0700)

are available in the Git repository at:

  git@gitolite.kernel.org:pub/scm/linux/kernel/git/vfs/vfs tags/vfs-6.10.netfs

for you to fetch changes up to e2bc9f6cfbd62c72a93a70068daab8886bec32ce:

  Merge branch 'cifs-netfs' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs (2024-05-02 11:23:31 +0200)

Please consider pulling these changes from the signed vfs-6.10.netfs tag.

Thanks!
Christian

----------------------------------------------------------------
vfs-6.10.netfs

----------------------------------------------------------------
Christian Brauner (2):
      Merge branch 'netfs-writeback' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs
      Merge branch 'cifs-netfs' of ssh://gitolite.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs

David Howells (38):
      netfs: Update i_blocks when write committed to pagecache
      netfs: Replace PG_fscache by setting folio->private and marking dirty
      mm: Remove the PG_fscache alias for PG_private_2
      netfs: Remove deprecated use of PG_private_2 as a second writeback flag
      netfs: Make netfs_io_request::subreq_counter an atomic_t
      netfs: Use subreq_counter to allocate subreq debug_index values
      mm: Provide a means of invalidation without using launder_folio
      9p: Use alternative invalidation to using launder_folio
      afs: Use alternative invalidation to using launder_folio
      netfs: Remove ->launder_folio() support
      netfs: Use mempools for allocating requests and subrequests
      mm: Export writeback_iter()
      netfs: Switch to using unsigned long long rather than loff_t
      netfs: New writeback implementation
      netfs: Add some write-side stats and clean up some stat names
      netfs, afs: Implement helpers for new write code
      netfs, 9p: Implement helpers for new write code
      netfs, cachefiles: Implement helpers for new write code
      netfs: Cut over to using new writeback code
      netfs: Remove the old writeback code
      netfs: Miscellaneous tidy ups
      netfs, afs: Use writeback retry to deal with alternate keys
      cifs: Use alternative invalidation to using launder_folio
      cifs: Replace cifs_readdata with a wrapper around netfs_io_subrequest
      cifs: Replace cifs_writedata with a wrapper around netfs_io_subrequest
      cifs: Use more fields from netfs_io_subrequest
      cifs: Make wait_mtu_credits take size_t args
      cifs: Replace the writedata replay bool with a netfs sreq flag
      cifs: Move cifs_loose_read_iter() and cifs_file_write_iter() to file.c
      cifs: Set zero_point in the copy_file_range() and remap_file_range()
      cifs: Add mempools for cifs_io_request and cifs_io_subrequest structs
      cifs: Make add_credits_and_wake_if() clear deducted credits
      cifs: Implement netfslib hooks
      cifs: Cut over to using netfslib
      cifs: Remove some code that's no longer used, part 1
      cifs: Remove some code that's no longer used, part 2
      cifs: Remove some code that's no longer used, part 3
      cifs: Enable large folio support

 fs/9p/vfs_addr.c             |   60 +-
 fs/afs/file.c                |    8 +-
 fs/afs/internal.h            |    6 +-
 fs/afs/validation.c          |    4 +-
 fs/afs/write.c               |  189 +--
 fs/cachefiles/io.c           |   76 +-
 fs/ceph/addr.c               |   24 +-
 fs/ceph/inode.c              |    2 +
 fs/netfs/Makefile            |    3 +-
 fs/netfs/buffered_read.c     |   40 +-
 fs/netfs/buffered_write.c    |  829 ++-----------
 fs/netfs/direct_write.c      |   56 +-
 fs/netfs/fscache_io.c        |   14 +-
 fs/netfs/internal.h          |   55 +-
 fs/netfs/io.c                |  162 +--
 fs/netfs/main.c              |   55 +-
 fs/netfs/misc.c              |   10 +-
 fs/netfs/objects.c           |   81 +-
 fs/netfs/output.c            |  478 --------
 fs/netfs/stats.c             |   17 +-
 fs/netfs/write_collect.c     |  808 +++++++++++++
 fs/netfs/write_issue.c       |  684 +++++++++++
 fs/nfs/file.c                |    8 +-
 fs/nfs/fscache.h             |    6 +-
 fs/nfs/write.c               |    4 +-
 fs/smb/client/Kconfig        |    1 +
 fs/smb/client/cifsfs.c       |  124 +-
 fs/smb/client/cifsfs.h       |   11 +-
 fs/smb/client/cifsglob.h     |   65 +-
 fs/smb/client/cifsproto.h    |   12 +-
 fs/smb/client/cifssmb.c      |  120 +-
 fs/smb/client/file.c         | 2720 ++++++------------------------------------
 fs/smb/client/fscache.c      |  109 --
 fs/smb/client/fscache.h      |   54 -
 fs/smb/client/inode.c        |   45 +-
 fs/smb/client/smb2ops.c      |   10 +-
 fs/smb/client/smb2pdu.c      |  186 +--
 fs/smb/client/smb2proto.h    |    5 +-
 fs/smb/client/trace.h        |  144 ++-
 fs/smb/client/transport.c    |   17 +-
 include/linux/fscache.h      |   22 +-
 include/linux/netfs.h        |  197 +--
 include/linux/pagemap.h      |    2 +
 include/net/9p/client.h      |    2 +
 include/trace/events/netfs.h |  250 +++-
 mm/filemap.c                 |   60 +-
 mm/page-writeback.c          |    1 +
 net/9p/Kconfig               |    1 +
 net/9p/client.c              |   49 +
 49 files changed, 3298 insertions(+), 4588 deletions(-)
 delete mode 100644 fs/netfs/output.c
 create mode 100644 fs/netfs/write_collect.c
 create mode 100644 fs/netfs/write_issue.c

^ permalink raw reply	[flat|nested] 26+ messages in thread
end of thread, other threads:[~2025-06-02 22:57 UTC | newest]

Thread overview: 26+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-09-13 16:56 [GIT PULL] vfs netfs Christian Brauner
2024-09-16 10:28 ` Linus Torvalds
2024-09-16 11:09 ` pr-tracker-bot
2024-09-16 12:58 ` David Howells
2024-09-16 13:50 ` [PATCH] cifs: Fix cifs readv callback merge resolution issue David Howells
2024-09-16 14:08   ` Linus Torvalds
2024-09-16 15:33   ` David Howells
2024-09-16 16:49     ` Linus Torvalds
2024-09-16 14:02 ` [PATCH] cifs: Remove redundant setting of NETFS_SREQ_HIT_EOF David Howells
2024-09-26 17:40 ` [GIT PULL] vfs netfs Leon Romanovsky
2024-09-27  8:01 ` David Howells
2024-09-27 16:34   ` Leon Romanovsky
2024-09-27 20:31   ` David Howells
2024-09-28 10:11     ` Leon Romanovsky
  -- strict thread matches above, loose matches on Subject: below --
2025-06-02 10:11 Christian Brauner
2025-06-02 22:58 ` pr-tracker-bot
2025-01-18 12:55 Christian Brauner
2025-01-20 18:59 ` pr-tracker-bot
2024-11-15 14:00 Christian Brauner
2024-11-18 18:29 ` Linus Torvalds
2024-11-20  8:49   ` Christian Brauner
2024-11-20 17:09     ` Linus Torvalds
2024-11-20 19:53       ` Christian Brauner
2024-11-18 19:49 ` pr-tracker-bot
2024-05-10 11:47 Christian Brauner
2024-05-13 19:38 ` pr-tracker-bot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).