From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C79274B127D for ; Wed, 2 Sep 2026 17:38:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788370736; cv=none; b=Qhln+gK9stui30par+d1iLzs2ZQvD9aWI4JhxoS6tsK5gYL4cA9Psuuuam7KzTMPzvCGZd85Srz24zutNlNGmyoLXRYJOOHdK7eQjGpoxPEtGlx9YFJWKftyBwpYChqcwkbpfBaXGe1Y81CM58wd+EcvvLOlMDJaw+svHEXyQkA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788370736; c=relaxed/simple; bh=oGaR6QTxEZH57XPUo1J1nGfdm11hFSvqx52aa3JPeuc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:content-type; b=OXPq3jNoYwiYpA5ciHUAqYfDb/AapJzIX+o+8IJrPQAhLrv25YOHdUo/cHPOJlsnqsADvwZvAJrMljWpUmRQA679Ua3HeFTn/JPo81wlCcspkpKDs7c6xMwefRlWGKvjck9QOk57ZeWQ7RTwiu/DnqYccBmeKjGkLTV8VNwdE0s= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=XTSxupMD; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XTSxupMD" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788370722; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ie5bveMuVKHMUnVy+72KB4vW+d/beni6Ki+9oyTOXIw=; b=XTSxupMD7IXa5scRW1n7MfaJaVoRhRLRd+zeh+jt47ljzpq3x/gixX1YTy05Gcf0MPkTz3 ogtMLOvG2alVHzPh4n4kuyQSONlCHVmp+WDSW1ERfHEnjw0VlK3MoxbQkDzPz3JF7R3Gg8 moNKXcssLaiuqIoy4ifi3Nh3MGvdEAY= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-480-XwZONQApPBinqPRWylA9WA-1; Wed, 02 Sep 2026 13:38:38 -0400 X-MC-Unique: XwZONQApPBinqPRWylA9WA-1 X-Mimecast-MFC-AGG-ID: XwZONQApPBinqPRWylA9WA_1788370715 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 94C9118345AD; Wed, 2 Sep 2026 17:38:35 +0000 (UTC) Received: from warthog.com (unknown [10.44.32.158]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BB1D01955D8D; Wed, 2 Sep 2026 17:38:28 +0000 (UTC) From: David Howells To: Paulo Alcantara Cc: David Howells , Christian Brauner , Matthew Wilcox , Christoph Hellwig , Jens Axboe , Leon Romanovsky , Namjae Jeon , ChenXiaoSong , Marc Dionne , Stefan Metzmacher , Eric Van Hensbergen , Dominique Martinet , Ilya Dryomov , netfs@lists.linux.dev, linux-afs@lists.infradead.org, linux-cifs@vger.kernel.org, linux-nfs@vger.kernel.org, ceph-devel@vger.kernel.org, v9fs@lists.linux.dev, linux-erofs@lists.ozlabs.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v11 34/36] netfs: Combine prepare and issue ops Date: Wed, 2 Sep 2026 18:33:46 +0100 Message-ID: <20260902173350.3468672-35-dhowells@redhat.com> In-Reply-To: <20260902173350.3468672-1-dhowells@redhat.com> References: <20260902173350.3468672-1-dhowells@redhat.com> Precedence: bulk X-Mailing-List: v9fs@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-MFC-PROC-ID: hrpGy6_8GgGekoQJWnyax33ZAGInQ3Ds2t8_Ofj4mEI_1788370715 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit content-type: text/plain; charset="US-ASCII"; x-default=true Modify the way subrequests are generated in netfslib to simplify the way things work. Currently, netfslib creates a subrequest and then calls ->prepare_xxx() to prepare it and to enquire about how big it can be and to which source it sends a request (server or cache), then it progressively loads up the buffer and some time later finally issues it. This delay between prepare and issue, however, presents a couple of issues: firstly, it may hold server resources for a significant length of time (e.g. cifs SMB2+ credits), blocking other users; and secondly it holds memory or other local resources that could otherwise be used to perform some other task under low-memory conditions. Fix this by accumulating buffers to be involved in the I/O and then calling ->issue_xxx() to do all the preparation work and issuing, including calling back into netfslib to grab a slice of the so-far accumulated buffers. For buffered reads, instead of ->prepare_read() setting limits on the amount of bufferage to be accumulated, all read requests (including readahead) build the buffer list up front and then just go through dispatching slices of it. Each subreq bites off the slice it can handle. In a previous patch, the cache acquired a ->query_occupancy() method that allows netfs to find out where extents of holes or data are rather than sending each subreq to the cache first. For unbuffered/DIO read and writes, as for buffered reads, the buffer list is built up front and then slices of it are dispatched. For buffered writeback, ->estimate_write() is used to set a need-flush-by point for each stream, and then buffers are added into the list until enough is accumulated to flush a stream or a discontiguity is reached - at which point ->issue_write() is called and the estimate refreshed repeatedly until the remaining amount of accumulated buffers is less than the estimate. To this end, netfslib is changed in the following ways: (1) ->prepare_xxx(), buffer selection and ->issue_xxx() are now collapsed together such that one ->issue_xxx() call is made with the subrequest defined to the maximum extent; the filesystem/cache then reduces the length of the subrequest and calls back to netfslib to grab a slice of the buffer, which may reduce the subrequest further if a maximum segment limit is set. The filesystem/cache then dispatches the operation. (2) Retry buffer tracking is added to the netfs_io_request struct. This is then selected by the subrequest retry counter being non-zero. (3) The use of iov_iter is pushed down to the filesystem. Netfslib now provides the filesystem with a bvecq holding the buffer rather than an iov_iter. The bvecq can be duplicated and headers/trailers attached to hold protocol and several duplicated bvecqs can be linked together to create a compound operation. (4) If the ->issue_xxx() functions terminate with -ENOMEM, a flag is set on the request to abort further subrequest generation/retrying. If it fails early with any other error, generally the subreq will be extended as much as possible and failed with the error returned. (5) During writeback, netfslib now builds up an accumulation of buffered data before issuing writes on each stream (one server, one cache). It asks each stream for an estimate of how much data to accumulate before it next generates subrequests on the stream. The filesystem or cache is not required to use up all the data accumulated on a stream at that time unless the end of the pagecache is hit. (6) During read-gaps, in which there are two gaps on either end of a dirty streaming write page that need to be filled, a buffer is constructed consisting of the two ends plus a sink page repeated to cover the middle portion. This is passed to the server as a single write. For something like Ceph, this should probably be done either as a vectored/sparse read or as two separate reads (if different Ceph objects are involved). (7) During unbuffered/DIO read/write, there is a single contiguous file region to be read or written as a single stream. The dispatching function just creates subrequests and calls ->issue_xxx() repeatedly to eat through the bufferage. Signed-off-by: David Howells cc: Paulo Alcantara cc: Matthew Wilcox cc: Christoph Hellwig cc: netfs@lists.linux.dev cc: linux-fsdevel@vger.kernel.org --- fs/9p/vfs_addr.c | 43 ++++-- fs/afs/file.c | 27 ++-- fs/afs/fsclient.c | 8 +- fs/afs/internal.h | 5 +- fs/afs/write.c | 33 ++-- fs/afs/yfsclient.c | 6 +- fs/cachefiles/io.c | 230 ++++++++++++++++++++-------- fs/ceph/addr.c | 123 ++++++++------- fs/netfs/buffered_read.c | 281 ++++++++++++++++++++--------------- fs/netfs/direct_read.c | 84 ++++++----- fs/netfs/direct_write.c | 144 +++++++++--------- fs/netfs/fscache_io.c | 2 +- fs/netfs/internal.h | 44 ++++-- fs/netfs/main.c | 1 + fs/netfs/misc.c | 33 +++- fs/netfs/objects.c | 4 +- fs/netfs/read_collect.c | 38 ++--- fs/netfs/read_pgpriv2.c | 125 ++++++++++------ fs/netfs/read_retry.c | 208 ++++++++++++-------------- fs/netfs/read_single.c | 151 ++++++++++++------- fs/netfs/write_collect.c | 2 + fs/netfs/write_issue.c | 157 ++++++++++++------- fs/netfs/write_retry.c | 168 +++++++++++---------- fs/nfs/fscache.c | 24 ++- fs/smb/client/cifssmb.c | 13 +- fs/smb/client/file.c | 134 ++++++++--------- fs/smb/client/smb2ops.c | 9 +- fs/smb/client/smb2pdu.c | 28 +++- fs/smb/client/transport.c | 15 +- include/linux/netfs.h | 55 +++---- include/trace/events/netfs.h | 30 +++- net/9p/client.c | 8 +- 32 files changed, 1320 insertions(+), 913 deletions(-) diff --git a/fs/9p/vfs_addr.c b/fs/9p/vfs_addr.c index 9b163a13ec3f..fc5b9c677986 100644 --- a/fs/9p/vfs_addr.c +++ b/fs/9p/vfs_addr.c @@ -67,28 +67,51 @@ static int v9fs_estimate_write(struct netfs_io_request *wreq, /* * Issue a subrequest to write to the server. */ -static void v9fs_issue_write(struct netfs_io_subrequest *subreq) +static int v9fs_issue_write(struct netfs_io_subrequest *subreq) { + struct iov_iter iter; struct p9_fid *fid = subreq->rreq->netfs_priv; - int err, len; + int err, len = 0; - len = p9_client_write(fid, subreq->start, &subreq->io_iter, &err); + subreq->len = umin(subreq->len, fid->clnt->msize - P9_IOHDRSZ); + + err = netfs_prepare_write_buffer(subreq, INT_MAX); + if (err < 0) + return err; + /* After this point, must fail by termination. */ + + iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, subreq->len); + + len = p9_client_write(fid, subreq->start, &iter, &err); if (len > 0) __set_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags); + netfs_write_subrequest_terminated(subreq, len ?: err); + return 0; } /** * v9fs_issue_read - Issue a read from 9P * @subreq: The read to make */ -static void v9fs_issue_read(struct netfs_io_subrequest *subreq) +static int v9fs_issue_read(struct netfs_io_subrequest *subreq) { struct netfs_io_request *rreq = subreq->rreq; + struct iov_iter iter; struct p9_fid *fid = rreq->netfs_priv; char *target; unsigned long long pos = subreq->start + subreq->transferred; - int total = 0, err, len, n; + size_t len; + int total = 0, err; + + err = netfs_prepare_read_buffer(subreq, INT_MAX); + if (err < 0) + return err; + /* After this point, must fail by termination. */ + + iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, subreq->len); if (S_ISLNK(rreq->inode->i_mode)) { /* p9_client_readlink() must not be called for legacy protocols @@ -105,12 +128,13 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq) err = p9_client_readlink(fid, &target); if (err != 0) goto fill_subreq; - len = strlen(target); - n = copy_to_iter(target, len, &subreq->io_iter); + len = min(strlen(target), subreq->len); + total = copy_to_iter(target, len, &iter); kfree(target); - total = n; } else { - total = p9_client_read(fid, pos, &subreq->io_iter, &err); + trace_netfs_sreq(subreq, netfs_sreq_trace_submit); + + total = p9_client_read(fid, pos, &iter, &err); } fill_subreq: @@ -128,6 +152,7 @@ static void v9fs_issue_read(struct netfs_io_subrequest *subreq) subreq->error = err; netfs_read_subreq_terminated(subreq); + return 0; } /** diff --git a/fs/afs/file.c b/fs/afs/file.c index 4a412e0472ec..a6b61d609f02 100644 --- a/fs/afs/file.c +++ b/fs/afs/file.c @@ -332,11 +332,12 @@ void afs_fetch_data_immediate_cancel(struct afs_call *call) /* * Fetch file data from the volume. */ -static void afs_issue_read(struct netfs_io_subrequest *subreq) +static int afs_issue_read(struct netfs_io_subrequest *subreq) { struct afs_operation *op; struct afs_vnode *vnode = AFS_FS_I(subreq->rreq->inode); struct key *key = subreq->rreq->netfs_priv; + int ret; _enter("%s{%llx:%llu.%u},%x,,,", vnode->volume->name, @@ -345,11 +346,15 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq) vnode->fid.unique, key_serial(key)); + ret = netfs_prepare_read_buffer(subreq, INT_MAX); + if (ret < 0) + return ret; + /* After this point, must fail by termination. */ + op = afs_alloc_operation(key, vnode->volume); if (IS_ERR(op)) { - subreq->error = PTR_ERR(op); - netfs_read_subreq_terminated(subreq); - return; + ret = PTR_ERR(op); + goto failed; } afs_op_set_vnode(op, 0, vnode); @@ -364,20 +369,24 @@ static void afs_issue_read(struct netfs_io_subrequest *subreq) op->flags |= AFS_OPERATION_ASYNC; if (!afs_begin_vnode_operation(op)) { - subreq->error = afs_put_operation(op); - netfs_read_subreq_terminated(subreq); - return; + ret = afs_put_operation(op); + goto failed; } if (!afs_select_fileserver(op)) { - afs_end_read(op); - return; + afs_end_read(op); /* Error recorded here. */ + return 0; } afs_issue_read_call(op); } else { afs_do_sync_operation(op); } + return 0; +failed: + subreq->error = ret; + netfs_read_subreq_terminated(subreq); + return 0; } static int afs_init_request(struct netfs_io_request *rreq, struct file *file) diff --git a/fs/afs/fsclient.c b/fs/afs/fsclient.c index 1a3f186a6a11..9713551dd842 100644 --- a/fs/afs/fsclient.c +++ b/fs/afs/fsclient.c @@ -339,7 +339,9 @@ static int afs_deliver_fs_fetch_data(struct afs_call *call) if (call->remaining == 0) goto no_more_data; - call->iter = &subreq->io_iter; + iov_iter_bvec_queue(&call->def_iter, ITER_DEST, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, subreq->len); + call->iov_len = umin(call->remaining, subreq->len - subreq->transferred); call->unmarshall++; fallthrough; @@ -1088,7 +1090,7 @@ static void afs_fs_store_data64(struct afs_operation *op) if (!call) return afs_op_nomem(op); - call->write_iter = op->store.write_iter; + call->write_iter = &op->store.write_iter; /* marshall the parameters */ bp = call->request; @@ -1142,7 +1144,7 @@ void afs_fs_store_data(struct afs_operation *op) if (!call) return afs_op_nomem(op); - call->write_iter = op->store.write_iter; + call->write_iter = &op->store.write_iter; /* marshall the parameters */ bp = call->request; diff --git a/fs/afs/internal.h b/fs/afs/internal.h index ae2bc699bfde..5c536cc7b53c 100644 --- a/fs/afs/internal.h +++ b/fs/afs/internal.h @@ -914,7 +914,7 @@ struct afs_operation { afs_lock_type_t type; } lock; struct { - struct iov_iter *write_iter; + struct iov_iter write_iter; loff_t pos; loff_t size; loff_t i_size; @@ -1701,8 +1701,7 @@ extern int afs_check_volume_status(struct afs_volume *, struct afs_operation *); int afs_estimate_write(struct netfs_io_request *wreq, struct netfs_io_stream *stream, struct netfs_write_estimate *estimate); -void afs_prepare_write(struct netfs_io_subrequest *subreq); -void afs_issue_write(struct netfs_io_subrequest *subreq); +int afs_issue_write(struct netfs_io_subrequest *subreq); void afs_begin_writeback(struct netfs_io_request *wreq); void afs_retry_request(struct netfs_io_request *wreq, struct netfs_io_stream *stream); extern int afs_writepages(struct address_space *, struct writeback_control *); diff --git a/fs/afs/write.c b/fs/afs/write.c index a0c9916f594b..46f81d9e7d7f 100644 --- a/fs/afs/write.c +++ b/fs/afs/write.c @@ -99,20 +99,6 @@ int afs_estimate_write(struct netfs_io_request *wreq, return 0; } -/* - * Prepare a subrequest to write to the server. This sets the max_len - * parameter. - */ -void afs_prepare_write(struct netfs_io_subrequest *subreq) -{ - struct netfs_io_stream *stream = &subreq->rreq->io_streams[subreq->stream_nr]; - - //if (test_bit(NETFS_SREQ_RETRYING, &subreq->flags)) - // subreq->max_len = 512 * 1024; - //else - stream->sreq_max_len = 256 * 1024 * 1024; -} - /* * Issue a subrequest to write to the server. */ @@ -156,12 +142,15 @@ static void afs_issue_write_worker(struct work_struct *work) op->flags |= AFS_OPERATION_UNINTR; op->ops = &afs_store_data_operation; + trace_netfs_sreq(subreq, netfs_sreq_trace_submit); afs_begin_vnode_operation(op); - op->store.write_iter = &subreq->io_iter; op->store.i_size = umax(pos + len, netfs_read_remote_i_size(&vnode->netfs.inode)); op->mtime = inode_get_mtime(&vnode->netfs.inode); + iov_iter_bvec_queue(&op->store.write_iter, ITER_SOURCE, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, subreq->len); + afs_wait_for_operation(op); ret = afs_put_operation(op); switch (ret) { @@ -185,11 +174,21 @@ static void afs_issue_write_worker(struct work_struct *work) netfs_write_subrequest_terminated(subreq, ret < 0 ? ret : subreq->len); } -void afs_issue_write(struct netfs_io_subrequest *subreq) +int afs_issue_write(struct netfs_io_subrequest *subreq) { + int ret; + + if (subreq->len > 256 * 1024 * 1024) + subreq->len = 256 * 1024 * 1024; + ret = netfs_prepare_write_buffer(subreq, INT_MAX); + if (ret < 0) + return ret; + /* After this point, must fail by termination. */ + subreq->work.func = afs_issue_write_worker; if (!queue_work(system_dfl_wq, &subreq->work)) WARN_ON_ONCE(1); + return 0; } /* @@ -200,6 +199,8 @@ void afs_begin_writeback(struct netfs_io_request *wreq) { if (S_ISREG(wreq->inode->i_mode)) afs_get_writeback_key(wreq); + + wreq->io_streams[0].avail = true; } /* diff --git a/fs/afs/yfsclient.c b/fs/afs/yfsclient.c index d941179730a9..52c588092050 100644 --- a/fs/afs/yfsclient.c +++ b/fs/afs/yfsclient.c @@ -385,7 +385,9 @@ static int yfs_deliver_fs_fetch_data64(struct afs_call *call) if (call->remaining == 0) goto no_more_data; - call->iter = &subreq->io_iter; + iov_iter_bvec_queue(&call->def_iter, ITER_DEST, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, subreq->len); + call->iov_len = min(call->remaining, subreq->len - subreq->transferred); call->unmarshall++; fallthrough; @@ -1357,7 +1359,7 @@ void yfs_fs_store_data(struct afs_operation *op) if (!call) return afs_op_nomem(op); - call->write_iter = op->store.write_iter; + call->write_iter = &op->store.write_iter; /* marshall the parameters */ bp = call->request; diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c index 91b69be78c6b..a1580385d5fb 100644 --- a/fs/cachefiles/io.c +++ b/fs/cachefiles/io.c @@ -197,12 +197,131 @@ static int cachefiles_read(struct netfs_cache_resources *cres, return ret; } +/* + * Handle completion of a read from the cache issued by netfslib. + */ +static void cachefiles_issue_read_complete(struct kiocb *iocb, long ret) +{ + struct cachefiles_kiocb *ki = container_of(iocb, struct cachefiles_kiocb, iocb); + struct netfs_io_subrequest *subreq = ki->subreq; + struct inode *inode = file_inode(ki->iocb.ki_filp); + + _enter("%ld", ret); + + if (ret < 0) { + subreq->error = -ESTALE; + trace_cachefiles_io_error(ki->object, inode, ret, + cachefiles_trace_read_error); + } + + if (ret >= 0) { + if (ki->object->cookie->inval_counter == ki->inval_counter) { + subreq->error = 0; + if (ret > 0) { + subreq->transferred += ret; + __set_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags); + } + } else { + subreq->error = -ESTALE; + } + } + + netfs_read_subreq_terminated(subreq); + cachefiles_put_kiocb(ki); +} + +/* + * Issue a read operation to the cache. We assume that the cache is readable + * as the caller should have called ->query_occupancy() first. + */ +static int cachefiles_issue_read(struct netfs_io_subrequest *subreq) +{ + struct netfs_cache_resources *cres = &subreq->rreq->cache_resources; + struct cachefiles_object *object = cachefiles_cres_object(cres); + struct cachefiles_kiocb *ki; + struct iov_iter iter; + struct file *file = cachefiles_cres_file(cres); + unsigned int old_nofs; + ssize_t ret; + + fscache_count_read(); + + _enter("%pD,%lli,%llx,%zx/%llx", + file, file_inode(file)->i_ino, subreq->start, subreq->len, + i_size_read(file_inode(file))); + + if (subreq->len > MAX_RW_COUNT) + subreq->len = MAX_RW_COUNT; + + ret = netfs_prepare_read_buffer(subreq, BIO_MAX_VECS); + if (ret < 0) + return ret; + /* After this point, must fail by termination. */ + + iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, subreq->len); + + ret = -ENOMEM; + ki = kzalloc_obj(struct cachefiles_kiocb); + if (!ki) + goto failed; + + refcount_set(&ki->ki_refcnt, 2); + ki->iocb.ki_filp = file; + ki->iocb.ki_pos = subreq->start; + ki->iocb.ki_flags = IOCB_DIRECT; + ki->iocb.ki_ioprio = get_current_ioprio(); + ki->iocb.ki_complete = cachefiles_issue_read_complete; + ki->object = object; + ki->inval_counter = cres->inval_counter; + ki->subreq = subreq; + ki->was_async = true; + + get_file(ki->iocb.ki_filp); + cachefiles_grab_object(object, cachefiles_obj_get_ioreq); + + trace_cachefiles_read(object, file_inode(file), ki->iocb.ki_pos, subreq->len); + old_nofs = memalloc_nofs_save(); + ret = cachefiles_inject_read_error(); + if (ret == 0) + ret = vfs_iocb_iter_read(file, &ki->iocb, &iter); + memalloc_nofs_restore(old_nofs); + + switch (ret) { + case -EIOCBQUEUED: + break; + + case -ERESTARTSYS: + case -ERESTARTNOINTR: + case -ERESTARTNOHAND: + case -ERESTART_RESTARTBLOCK: + /* There's no easy way to restart the syscall since other AIO's + * may be already running. Just fail this IO with EINTR. + */ + ret = -EINTR; + fallthrough; + default: + ki->was_async = false; + cachefiles_issue_read_complete(&ki->iocb, ret); + break; + } + + cachefiles_put_kiocb(ki); + _leave(" = %zd", ret); + return 0; +failed: + subreq->error = ret; + netfs_read_subreq_terminated(subreq); + return 0; +} + /* * Query the occupancy of the cache in a region, returning the extent of the - * next two chunks of cached data and the next hole. + * next two chunks of cached data and the next hole. The occupancy map is + * preloaded to show just one giant hole. */ -static int cachefiles_query_occupancy(struct netfs_cache_resources *cres, - struct fscache_occupancy *occ) +static void cachefiles_query_occupancy(struct netfs_cache_resources *cres, + struct fscache_occupancy *occ) { struct cachefiles_object *object; struct inode *inode; @@ -212,7 +331,7 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres, int i; if (!fscache_wait_for_operation(cres, FSCACHE_WANT_READ)) - return -ENOBUFS; + return; object = cachefiles_cres_object(cres); file = cachefiles_cres_file(cres); @@ -247,7 +366,7 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres, ret = vfs_llseek(file, occ->query_from, SEEK_DATA); if (IS_ERR_VALUE_LL(ret)) { if (ret != -ENXIO) - return ret; + goto done; occ->query_from = ULLONG_MAX; goto done; } @@ -260,7 +379,7 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres, ret = vfs_llseek(file, occ->query_from, SEEK_HOLE); if (IS_ERR_VALUE_LL(ret)) { if (ret != -ENXIO) - return ret; + goto done; occ->query_from = ULLONG_MAX; goto done; } @@ -273,7 +392,6 @@ static int cachefiles_query_occupancy(struct netfs_cache_resources *cres, done: _debug("query[0] %llx-%llx", occ->cached_from[0], occ->cached_to[0]); _debug("query[1] %llx-%llx", occ->cached_from[1], occ->cached_to[1]); - return 0; } /* @@ -590,9 +708,9 @@ int __cachefiles_prepare_write(struct cachefiles_object *object, cachefiles_has_space_for_write); } -static int cachefiles_prepare_write(struct netfs_cache_resources *cres, - uoff_t *_start, size_t *_len, size_t upper_len, - uoff_t i_size, bool no_space_allocated_yet) +static int cachefiles_prepare_write_old(struct netfs_cache_resources *cres, + uoff_t *_start, size_t *_len, size_t upper_len, + uoff_t i_size, bool no_space_allocated_yet) { struct cachefiles_object *object = cachefiles_cres_object(cres); struct cachefiles_cache *cache = object->volume->cache; @@ -623,80 +741,63 @@ static int cachefiles_estimate_write(struct netfs_io_request *wreq, return 0; } -static void cachefiles_prepare_write_subreq(struct netfs_io_subrequest *subreq) +static int cachefiles_issue_write(struct netfs_io_subrequest *subreq) { struct netfs_io_request *wreq = subreq->rreq; struct netfs_cache_resources *cres = &wreq->cache_resources; - struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr]; - - _enter("W=%x[%x] %llx", wreq->debug_id, subreq->debug_index, subreq->start); + struct cachefiles_object *object = cachefiles_cres_object(cres); + struct cachefiles_cache *cache = object->volume->cache; + struct iov_iter iter; + const struct cred *saved_cred; + uoff_t start = subreq->start; + size_t len = subreq->len; + int ret = -EINVAL; - stream->sreq_max_len = MAX_RW_COUNT; - stream->sreq_max_segs = BIO_MAX_VECS; + _enter("W=%x[%x] %llx-%llx", + wreq->debug_id, subreq->debug_index, start, start + len - 1); if (!cachefiles_cres_file(cres)) { if (!fscache_wait_for_operation(cres, FSCACHE_WANT_WRITE)) - return netfs_prepare_write_failed(subreq); + return -ENOBUFS; if (!cachefiles_cres_file(cres)) - return netfs_prepare_write_failed(subreq); + return -ENOBUFS; } -} -static void cachefiles_issue_write(struct netfs_io_subrequest *subreq) -{ - struct netfs_io_request *wreq = subreq->rreq; - struct netfs_cache_resources *cres = &wreq->cache_resources; - struct cachefiles_object *object = cachefiles_cres_object(cres); - struct cachefiles_cache *cache = object->volume->cache; - const struct cred *saved_cred; - size_t off, pre, post, len = subreq->len; - uoff_t start = subreq->start; - int ret; + ret = netfs_prepare_write_buffer(subreq, BIO_MAX_VECS); + if (ret < 0) + return ret; + /* After this point, must fail by termination. */ - _enter("W=%x[%x] %llx-%llx", - wreq->debug_id, subreq->debug_index, start, start + len - 1); + /* The buffer extraction func may round out start and end. */ + start = subreq->start; + len = subreq->len; - /* We need to start on the cache granularity boundary */ - off = start & (cache->bsize - 1); - if (off) { - pre = cache->bsize - off; - if (pre >= len) { - fscache_count_dio_misfit(); - netfs_write_subrequest_terminated(subreq, len); - return; - } - subreq->transferred += pre; - start += pre; - len -= pre; - iov_iter_advance(&subreq->io_iter, pre); - } - - /* We also need to end on the cache granularity boundary */ - post = len & (cache->bsize - 1); - if (post) { - len -= post; - if (len == 0) { - fscache_count_dio_misfit(); - netfs_write_subrequest_terminated(subreq, post); - return; - } - iov_iter_truncate(&subreq->io_iter, len); + /* We need to start and end on cache granularity boundaries. */ + if (WARN_ON_ONCE(start & (cache->bsize - 1)) || + WARN_ON_ONCE(len & (cache->bsize - 1))) { + fscache_count_dio_misfit(); + ret = -EIO; + goto failed; } + iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, len); + trace_netfs_sreq(subreq, netfs_sreq_trace_cache_prepare); cachefiles_begin_secure(cache, &saved_cred); ret = __cachefiles_prepare_write(object, cachefiles_cres_file(cres), &start, &len, len, true); cachefiles_end_secure(cache, saved_cred); - if (ret < 0) { - netfs_write_subrequest_terminated(subreq, ret); - return; - } + if (ret < 0) + goto failed; trace_netfs_sreq(subreq, netfs_sreq_trace_cache_write); - cachefiles_write(&subreq->rreq->cache_resources, - subreq->start, &subreq->io_iter, + cachefiles_write(&subreq->rreq->cache_resources, subreq->start, &iter, netfs_write_subrequest_terminated, subreq); + return 0; +failed: + netfs_write_subrequest_terminated(subreq, ret); + return 0; } /* @@ -753,6 +854,7 @@ static void cachefiles_collect_write(struct netfs_io_request *wreq, WARN_ON(new_size & (cache->bsize - 1)) || WARN_ON(start & (cache->bsize - 1)) || WARN_ON(len & (cache->bsize - 1))) { + trace_netfs_rreq(wreq, netfs_rreq_trace_cache_align_error); trace_cachefiles_io_error(object, inode, -EIO, cachefiles_trace_alignment_error); cachefiles_remove_object_xattr(cache, object, file->f_path.dentry); @@ -907,10 +1009,10 @@ static const struct netfs_cache_ops cachefiles_netfs_cache_ops = { .end_operation = cachefiles_end_operation, .read = cachefiles_read, .write = cachefiles_write, + .issue_read = cachefiles_issue_read, .issue_write = cachefiles_issue_write, - .prepare_write = cachefiles_prepare_write, - .prepare_write_subreq = cachefiles_prepare_write_subreq, .estimate_write = cachefiles_estimate_write, + .prepare_write_old = cachefiles_prepare_write_old, .query_occupancy = cachefiles_query_occupancy, .collect_write = cachefiles_collect_write, }; diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c index 6586f6c1dc73..b13c071bd574 100644 --- a/fs/ceph/addr.c +++ b/fs/ceph/addr.c @@ -274,7 +274,7 @@ static void finish_netfs_read(struct ceph_osd_request *req) ceph_dec_osd_stopping_blocker(fsc->mdsc); } -static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq) +static int ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq) { struct netfs_io_request *rreq = subreq->rreq; struct inode *inode = rreq->inode; @@ -283,7 +283,8 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq) struct ceph_mds_request *req; struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(inode->i_sb); struct ceph_inode_info *ci = ceph_inode(inode); - ssize_t err = 0; + struct iov_iter iter; + ssize_t err; size_t len; int mode; @@ -292,8 +293,20 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq) __set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags); __clear_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags); - if (subreq->start >= inode->i_size) + err = netfs_prepare_read_buffer(subreq, INT_MAX); + if (err < 0) + return err; + /* After this point, must fail by termination. */ + + if (subreq->start >= inode->i_size) { + __set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags); + err = 0; goto out; + } + + iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, + subreq->len); /* We need to fetch the inline data. */ mode = ceph_try_to_choose_auth_mds(inode, CEPH_STAT_CAP_INLINE_DATA); @@ -302,11 +315,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq) err = PTR_ERR(req); goto out; } + req->r_ino1 = ci->i_vino; req->r_args.getattr.mask = cpu_to_le32(CEPH_STAT_CAP_INLINE_DATA); req->r_num_caps = 2; trace_netfs_sreq(subreq, netfs_sreq_trace_submit); + err = ceph_mdsc_do_request(mdsc, NULL, req); if (err < 0) goto out; @@ -316,11 +331,13 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq) if (iinfo->inline_version == CEPH_INLINE_NONE) { /* The data got uninlined */ ceph_mdsc_put_request(req); - return false; + __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags); + err = -EAGAIN; + goto out; } len = min_t(size_t, iinfo->inline_len - subreq->start, subreq->len); - err = copy_to_iter(iinfo->inline_data + subreq->start, len, &subreq->io_iter); + err = copy_to_iter(iinfo->inline_data + subreq->start, len, &iter); if (err == 0) { err = -EFAULT; } else { @@ -333,26 +350,10 @@ static bool ceph_netfs_issue_op_inline(struct netfs_io_subrequest *subreq) subreq->error = err; trace_netfs_sreq(subreq, netfs_sreq_trace_io_progress); netfs_read_subreq_terminated(subreq); - return true; -} - -static int ceph_netfs_prepare_read(struct netfs_io_subrequest *subreq) -{ - struct netfs_io_request *rreq = subreq->rreq; - struct inode *inode = rreq->inode; - struct ceph_inode_info *ci = ceph_inode(inode); - struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode); - u64 objno, objoff; - u32 xlen; - - /* Truncate the extent at the end of the current block */ - ceph_calc_file_object_mapping(&ci->i_layout, subreq->start, subreq->len, - &objno, &objoff, &xlen); - rreq->io_streams[0].sreq_max_len = umin(xlen, fsc->mount_options->rsize); return 0; } -static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) +static int ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) { struct netfs_io_request *rreq = subreq->rreq; struct inode *inode = rreq->inode; @@ -361,19 +362,18 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) struct ceph_client *cl = fsc->client; struct ceph_osd_request *req = NULL; struct ceph_vino vino = ceph_vino(inode); - int err; - u64 len; + struct iov_iter iter; + u64 objno, objoff, len, off = subreq->start; + u32 maxlen; + int err = -EIO; bool sparse = IS_ENCRYPTED(inode) || ceph_test_mount_opt(fsc, SPARSEREAD); - u64 off = subreq->start; int extent_cnt; - if (ceph_inode_is_shutdown(inode)) { - err = -EIO; - goto out; - } + if (ceph_inode_is_shutdown(inode)) + return -EIO; - if (ceph_has_inline_data(ci) && ceph_netfs_issue_op_inline(subreq)) - return; + if (ceph_has_inline_data(ci)) + return ceph_netfs_issue_op_inline(subreq); // TODO: This rounding here is slightly dodgy. It *should* work, for // now, as the cache only deals in blocks that are a multiple of @@ -383,26 +383,48 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) len = subreq->len; ceph_fscrypt_adjust_off_and_len(inode, &off, &len); + /* Truncate the extent at the end of the current block */ + ceph_calc_file_object_mapping(&ci->i_layout, subreq->start, len, + &objno, &objoff, &maxlen); + maxlen = min(maxlen, fsc->mount_options->rsize); + len = min(len, maxlen); + subreq->len = len; + + /* Grab a slice of read buffer. This may shrink the subreq. */ + err = netfs_prepare_read_buffer(subreq, INT_MAX); + if (err < 0) + return err; + /* After this point, must fail by termination. */ + + /* Create a request. In theory, this may shrink the request again, but + * it shouldn't since we calculated the object size above and already + * shrank to that, but if it does, we'll just end up doing a short read + * and retrying to get the rest. + */ + len = subreq->len; req = ceph_osdc_new_request(&fsc->client->osdc, &ci->i_layout, vino, off, &len, 0, 1, sparse ? CEPH_OSD_OP_SPARSE_READ : CEPH_OSD_OP_READ, CEPH_OSD_FLAG_READ, NULL, ci->i_truncate_seq, ci->i_truncate_size, false); if (IS_ERR(req)) { err = PTR_ERR(req); - req = NULL; - goto out; + goto failed_noput; } if (sparse) { extent_cnt = __ceph_sparse_read_ext_count(inode, len); err = ceph_alloc_sparse_ext_map(&req->r_ops[0], extent_cnt); if (err) - goto out; + goto failed; } doutc(cl, "%llx.%llx pos=%llu orig_len=%zu len=%llu\n", ceph_vinop(inode), subreq->start, subreq->len, len); + iov_iter_bvec_queue(&iter, ITER_DEST, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, + subreq->len); + /* * FIXME: For now, use CEPH_OSD_DATA_TYPE_PAGES instead of _ITER for * encrypted inodes. We'd need infrastructure that handles an iov_iter @@ -421,13 +443,11 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) * ceph_msg_data_cursor_init() triggers BUG_ON() in the case * if msg->sparse_read_total > msg->data_length. */ - subreq->io_iter.count = len; - - err = iov_iter_get_pages_alloc2(&subreq->io_iter, &pages, len, &page_off); + err = iov_iter_get_pages_alloc2(&iter, &pages, len, &page_off); if (err < 0) { doutc(cl, "%llx.%llx failed to allocate pages, %d\n", ceph_vinop(inode), err); - goto out; + goto eio; } /* should always give us a page-aligned read */ @@ -438,12 +458,10 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) osd_req_op_extent_osd_data_pages(req, 0, pages, len, 0, false, false); } else { - osd_req_op_extent_osd_iter(req, 0, &subreq->io_iter); - } - if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) { - err = -EIO; - goto out; + osd_req_op_extent_osd_iter(req, 0, &iter); } + if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) + goto eio; req->r_callback = finish_netfs_read; req->r_priv = subreq; req->r_inode = inode; @@ -451,19 +469,22 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) trace_netfs_sreq(subreq, netfs_sreq_trace_submit); ceph_osdc_start_request(req->r_osdc, req); -out: ceph_osdc_put_request(req); - if (err) { - subreq->error = err; - netfs_read_subreq_terminated(subreq); - } - doutc(cl, "%llx.%llx result %d\n", ceph_vinop(inode), err); + doutc(cl, "%llx.%llx result -EIOCBQUEUED\n", ceph_vinop(inode)); + return 0; +eio: + err = -EIO; +failed: + ceph_osdc_put_request(req); +failed_noput: + subreq->error = err; + netfs_read_subreq_terminated(subreq); + return 0; } static int ceph_init_request(struct netfs_io_request *rreq, struct file *file) { struct inode *inode = rreq->inode; - struct ceph_fs_client *fsc = ceph_inode_to_fs_client(inode); struct ceph_client *cl = ceph_inode_to_client(inode); int got = 0, want = CEPH_CAP_FILE_CACHE; struct ceph_netfs_request_data *priv; @@ -515,7 +536,6 @@ static int ceph_init_request(struct netfs_io_request *rreq, struct file *file) priv->caps = got; rreq->netfs_priv = priv; - rreq->io_streams[0].sreq_max_len = fsc->mount_options->rsize; out: if (ret < 0) { @@ -543,7 +563,6 @@ static void ceph_netfs_free_request(struct netfs_io_request *rreq) const struct netfs_request_ops ceph_netfs_ops = { .init_request = ceph_init_request, .free_request = ceph_netfs_free_request, - .prepare_read = ceph_netfs_prepare_read, .issue_read = ceph_netfs_issue_read, .expand_readahead = ceph_netfs_expand_readahead, .check_write_begin = ceph_netfs_check_write_begin, diff --git a/fs/netfs/buffered_read.c b/fs/netfs/buffered_read.c index 5178d9bba453..224d57b4ad39 100644 --- a/fs/netfs/buffered_read.c +++ b/fs/netfs/buffered_read.c @@ -100,70 +100,99 @@ static int netfs_begin_cache_read(struct netfs_io_request *rreq, struct netfs_in } /* - * netfs_prepare_read_iterator - Prepare the subreq iterator for I/O - * @subreq: The subrequest to be set up - * - * Prepare the I/O iterator representing the read buffer on a subrequest for - * the filesystem to use for I/O (it can be passed directly to a socket). This - * is intended to be called from the ->issue_read() method once the filesystem - * has trimmed the request to the size it wants. - * - * Returns the limited size if successful and -ENOMEM if insufficient memory - * available. + * Prepare the I/O buffer on a buffered read subrequest for the filesystem to + * use as a bvec queue. */ -static ssize_t netfs_prepare_read_iterator(struct netfs_io_subrequest *subreq) +static int netfs_prepare_buffered_read_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) { struct netfs_io_request *rreq = subreq->rreq; struct netfs_io_stream *stream = &rreq->io_streams[0]; ssize_t extracted; - size_t rsize = subreq->len; - if (subreq->source == NETFS_DOWNLOAD_FROM_SERVER) - rsize = umin(rsize, stream->sreq_max_len); + _enter("R=%08x[%x] l=%zx s=%u", + rreq->debug_id, subreq->debug_index, subreq->len, max_segs); + + bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor); + bvecq_pos_set(&subreq->content, &subreq->dispatch_pos); + extracted = bvecq_slice(&stream->dispatch_cursor, subreq->len, + max_segs, &subreq->nr_segs); - bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor); - extracted = bvecq_slice(&rreq->dispatch_cursor, rsize, - stream->sreq_max_segs, &subreq->nr_segs); - if (extracted < rsize) { + if (extracted < subreq->len) { subreq->len = extracted; trace_netfs_sreq(subreq, netfs_sreq_trace_limited); } + stream->buffered -= extracted; + stream->issue_from = subreq->start + subreq->len; + rreq->submitted = stream->issue_from; - return subreq->len; + if (!stream->buffered) + netfs_all_subreqs_queued(rreq); + return 0; } -/* - * Issue a read against the cache. - * - Eats the caller's ref on subreq. +/** + * netfs_prepare_read_buffer - Get the buffer for a subrequest + * @subreq: The subrequest to get the buffer for + * @max_segs: Maximum number of segments in buffer (or INT_MAX) + * + * Extract a slice of buffer from the stream and attach it to the subrequest as + * a bio_vec queue. The maximum amount of data attached is set by + * @subreq->len, but this may be shortened if @max_segs would be exceeded. + * + * [!] NOTE: This must be run in the same thread as ->issue_read() was called + * in as we access the readahead_control struct if there is one. */ -static void netfs_read_cache_to_pagecache(struct netfs_io_request *rreq, - struct netfs_io_subrequest *subreq) +int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) { - struct netfs_cache_resources *cres = &rreq->cache_resources; - - netfs_stat(&netfs_n_rh_read); - cres->ops->read(cres, subreq->start, &subreq->io_iter, NETFS_READ_HOLE_IGNORE, - netfs_cache_read_terminated, subreq); + switch (subreq->rreq->origin) { + case NETFS_READAHEAD: + case NETFS_READPAGE: + case NETFS_READ_FOR_WRITE: + if (subreq->retry_count) + return netfs_prepare_buffered_read_retry_buffer(subreq, max_segs); + return netfs_prepare_buffered_read_buffer(subreq, max_segs); + + case NETFS_UNBUFFERED_READ: + case NETFS_DIO_READ: + case NETFS_READ_GAPS: + return netfs_prepare_unbuffered_read_buffer(subreq, max_segs); + case NETFS_READ_SINGLE: + return netfs_prepare_read_single_buffer(subreq, max_segs); + default: + WARN_ON_ONCE(1); + return -EIO; + } } +EXPORT_SYMBOL(netfs_prepare_read_buffer); -int netfs_read_query_cache(struct netfs_io_request *rreq, struct fscache_occupancy *occ) +void netfs_read_query_cache(struct netfs_io_request *rreq, struct fscache_occupancy *occ) { struct netfs_cache_resources *cres = &rreq->cache_resources; occ->granularity = PAGE_SIZE; if (occ->query_from >= occ->query_to) - return 0; + return; if (!cres->ops) - return 0; + return; occ->query_from = round_up(occ->query_from, occ->granularity); - return cres->ops->query_occupancy(cres, occ); + cres->ops->query_occupancy(cres, occ); } -void netfs_queue_read(struct netfs_io_request *rreq, - struct netfs_io_subrequest *subreq) +/* + * Allocate and prepare a read subrequest. + */ +struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq, + enum netfs_io_source source) { + struct netfs_io_subrequest *subreq; struct netfs_io_stream *stream = &rreq->io_streams[0]; + subreq = netfs_alloc_subrequest(rreq, source); + if (!subreq) + return subreq; + __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags); /* We add to the end of the list whilst the collector may be walking @@ -173,40 +202,56 @@ void netfs_queue_read(struct netfs_io_request *rreq, spin_lock(&rreq->lock); /* Write IN_PROGRESS before pointer to new subreq */ list_add_tail_release(&subreq->rreq_link, &stream->subrequests); - if (list_is_first(&subreq->rreq_link, &stream->subrequests)) { - if (!stream->active) { - stream->collected_to = subreq->start; - /* Store list pointers before active flag */ - smp_store_release(&stream->active, true); - } - } + if (!stream->active) + /* Store list pointers before active flag */ + smp_store_release(&stream->active, true); spin_unlock(&rreq->lock); + return subreq; } static void netfs_issue_read(struct netfs_io_request *rreq, struct netfs_io_subrequest *subreq) { - bvecq_pos_set(&subreq->content, &subreq->dispatch_pos); - iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq, - subreq->content.slot, subreq->content.offset, subreq->len); + struct netfs_cache_resources *cres = &rreq->cache_resources; + struct netfs_io_stream *stream = &rreq->io_streams[0]; + int ret; + + _enter("R=%08x[%x]", rreq->debug_id, subreq->debug_index); switch (subreq->source) { case NETFS_DOWNLOAD_FROM_SERVER: - rreq->netfs_ops->issue_read(subreq); - break; + ret = rreq->netfs_ops->issue_read(subreq); + if (ret < 0) + goto fail; + return; case NETFS_READ_FROM_CACHE: - netfs_read_cache_to_pagecache(rreq, subreq); - break; + netfs_stat(&netfs_n_rh_read); + ret = cres->ops->issue_read(subreq); + if (ret < 0) + goto fail; + return; default: - bvecq_zero(&subreq->content, subreq->len); + WARN_ON_ONCE(1); + fallthrough; + case NETFS_FILL_WITH_ZEROES: + stream->issue_from = subreq->start + subreq->len; + stream->buffered -= subreq->len; + trace_netfs_sreq(subreq, netfs_sreq_trace_submit); + if (!stream->buffered) + netfs_all_subreqs_queued(rreq); + bvecq_zero(&stream->dispatch_cursor, subreq->len); subreq->transferred = subreq->len; subreq->error = 0; - iov_iter_zero(subreq->len, &subreq->io_iter); - subreq->transferred = subreq->len; - netfs_read_subreq_terminated(subreq); - break; + return netfs_read_subreq_terminated(subreq); } + +fail: + /* Ownership of subreq was returned to us. */ + trace_netfs_sreq(subreq, netfs_sreq_trace_fail); + stream->buffered -= subreq->len; + subreq->error = ret; + netfs_read_subreq_terminated(subreq); } /* @@ -282,23 +327,21 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) .cached_to[1] = ULLONG_MAX, }; struct fscache_occupancy *occ = &_occ; + struct netfs_io_stream *stream = &rreq->io_streams[0]; struct bvecq_pos mark_cursor; - ssize_t size = rreq->len; - uoff_t start = rreq->start; + bool need_mark = true; int ret = 0; _enter("R=%08x", rreq->debug_id); bvecq_pos_set(&mark_cursor, &rreq->load_cursor); - bvecq_pos_set(&rreq->dispatch_cursor, &rreq->load_cursor); + bvecq_pos_set(&stream->dispatch_cursor, &rreq->load_cursor); do { - int (*prepare_read)(struct netfs_io_subrequest *subreq) = NULL; struct netfs_io_subrequest *subreq; enum netfs_io_source source; - ssize_t slice; - uoff_t hole_to, cache_to; - size_t len = size; + uoff_t hole_to, cache_to, start, stop; + size_t len; bool copy = false; /* If we don't have any, find out the next couple of data @@ -308,7 +351,7 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) */ hole_to = occ->cached_from[0]; cache_to = occ->cached_to[0]; - if (start >= cache_to) { + if (stream->issue_from >= cache_to) { /* Extent exhausted; shuffle down. */ int i; @@ -320,41 +363,61 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) occ->cached_from[i] = ULLONG_MAX; occ->cached_to[i] = ULLONG_MAX; - if (occ->cached_from[0] != ULLONG_MAX) + if (occ->cached_from[0] != ULLONG_MAX) { + need_mark = true; continue; + } - /* Get new extents */ - ret = netfs_read_query_cache(rreq, occ); - if (ret < 0) - break; + /* Wait for the cache to enter the readable state and + * get more extents. If the cache doesn't become + * readable, this leaves the default negative extent in + * place. + */ + netfs_read_query_cache(rreq, occ); + need_mark = true; continue; } + start = stream->issue_from; + stop = stream->issue_from + stream->buffered; + + /* See if the cache indicated this should be cached. */ + if (need_mark && rreq->cache_resources.ops) { + if (stream->issue_from < hole_to) + netfs_mark_copy_to_cache(rreq, &mark_cursor, + hole_to - stream->issue_from, true); + if (hole_to < cache_to) + netfs_mark_copy_to_cache(rreq, &mark_cursor, + cache_to - hole_to, false); + } + need_mark = false; + uoff_t zero_point = netfs_read_zero_point(rreq->inode); uoff_t zlimit = umin(zero_point, rreq->i_size); _debug("rsub %llx %llx-%llx", start, hole_to, cache_to); - if (start >= hole_to && start < cache_to) { + if (stream->issue_from >= hole_to && stream->issue_from < cache_to) { /* Overlap with a cached region, where the cache may * record a block of zeroes. */ - _debug("cached s=%llx c=%llx l=%zx", start, cache_to, size); - len = umin(cache_to - start, size); - len = round_up(len, occ->granularity); + _debug("cached s=%llx c=%llx l=%zx", + stream->issue_from, cache_to, stream->buffered); + len = umin(cache_to - stream->issue_from, stream->buffered); if (occ->cached_type[0] == FSCACHE_EXTENT_ZERO) { source = NETFS_FILL_WITH_ZEROES; netfs_stat(&netfs_n_rh_zero); } else { + len = round_up(len, occ->granularity); source = NETFS_READ_FROM_CACHE; - prepare_read = rreq->cache_resources.ops->prepare_read; } - } else if (start >= zlimit && size > 0) { + } else if (start >= zlimit && + start < stop) { /* If this range lies beyond the zero-point, that part * can just be cleared locally. */ - _debug("zero %llx-%llx", start, start + size); - len = size; + _debug("zero %llx-%llx", start, stop); + len = stream->buffered; source = NETFS_FILL_WITH_ZEROES; if (rreq->cache_resources.ops) copy = true; @@ -364,10 +427,10 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) * this range lies beyond the zero-point or the EOF, * that part can just be cleared locally. */ - uoff_t limit = min3(zlimit, start + size, hole_to); + uoff_t limit = min3(zlimit, stop, hole_to); _debug("limit %llx %llx", rreq->i_size, zero_point); - _debug("download %llx-%llx", start, start + size); + _debug("download %llx-%llx", start, stop); len = umin(limit - start, ULONG_MAX); source = NETFS_DOWNLOAD_FROM_SERVER; if (rreq->cache_resources.ops) @@ -377,54 +440,22 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) if (len == 0) { pr_err("ZERO-LEN READ: R=%08x l=%zx/%zx s=%llx z=%llx i=%llx", - rreq->debug_id, len, size, + rreq->debug_id, len, stream->buffered, start, zero_point, rreq->i_size); break; } - subreq = netfs_alloc_subrequest(rreq, source); + subreq = netfs_alloc_read_subrequest(rreq, source); if (!subreq) { ret = -ENOMEM; break; } - subreq->start = start; - subreq->len = size; + subreq->start = start; + subreq->len = len; if (copy) __set_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags); - netfs_queue_read(rreq, subreq); - - rreq->io_streams[0].sreq_max_len = MAX_RW_COUNT; - rreq->io_streams[0].sreq_max_segs = INT_MAX; - - if (prepare_read) { - ret = prepare_read(subreq); - if (ret < 0) { - netfs_cancel_read(subreq, ret); - break; - } - trace_netfs_sreq(subreq, netfs_sreq_trace_prepare); - } - - slice = netfs_prepare_read_iterator(subreq); - if (slice < 0) { - ret = slice; - netfs_cancel_read(subreq, ret); - break; - } - start += slice; - size -= slice; - if (size <= 0) - netfs_all_subreqs_queued(rreq); - - /* See if the cache indicated this should be cached. */ - if (mark_cursor.bvecq) { - copy = test_bit(NETFS_SREQ_COPY_TO_CACHE, &subreq->flags); - netfs_mark_copy_to_cache(rreq, &mark_cursor, slice, copy); - } - - trace_netfs_sreq(subreq, netfs_sreq_trace_submit); netfs_issue_read(rreq, subreq); netfs_maybe_bulk_drop_ra_refs(rreq); @@ -433,19 +464,20 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) if (test_bit(NETFS_RREQ_FAILED, &rreq->flags)) break; cond_resched(); - } while (size > 0); + } while (stream->buffered > 0); - if (unlikely(size > 0)) { + if (unlikely(!netfs_are_all_subreqs_queued(rreq))) { netfs_all_subreqs_queued(rreq); netfs_wake_collector(rreq); } /* Defer error return as we may need to wait for outstanding I/O. */ - cmpxchg(&rreq->error, 0, ret); + if (ret < 0) + cmpxchg(&rreq->error, 0, ret); bvecq_pos_unset(&mark_cursor); bvecq_pos_unset(&rreq->load_cursor); - bvecq_pos_unset(&rreq->dispatch_cursor); + bvecq_pos_unset(&stream->dispatch_cursor); } /** @@ -466,17 +498,22 @@ static void netfs_read_to_pagecache(struct netfs_io_request *rreq) void netfs_readahead(struct readahead_control *ractl) { struct netfs_io_request *rreq; + struct netfs_io_stream *stream; struct netfs_inode *ictx = netfs_inode(ractl->mapping->host); ssize_t added; uoff_t start = readahead_pos(ractl); size_t size = readahead_length(ractl); int ret; + _enter(""); + rreq = netfs_alloc_request(ractl->mapping, ractl->file, start, size, NETFS_READAHEAD); if (IS_ERR(rreq)) return; + stream = &rreq->io_streams[0]; + __set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &rreq->flags); ret = netfs_begin_cache_read(rreq, ictx); @@ -501,9 +538,9 @@ void netfs_readahead(struct readahead_control *ractl) __set_bit(NETFS_RREQ_NEED_PUT_RA_REFS, &rreq->flags); rreq->submitted = rreq->start + added; - rreq->cleaned_to = rreq->start; bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor); netfs_read_set_unlock_at(rreq); + stream->buffered = added; netfs_read_to_pagecache(rreq); netfs_maybe_bulk_drop_ra_refs(rreq); @@ -519,18 +556,20 @@ EXPORT_SYMBOL(netfs_readahead); */ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct folio *folio) { + struct netfs_io_stream *stream = &rreq->io_streams[0]; struct bvecq *bq; size_t fsize = folio_size(folio); if (bvecq_buffer_init(&rreq->load_cursor, rreq->gfp, false) < 0) return -ENOMEM; + bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor); bq = rreq->load_cursor.bvecq; bvec_set_folio(&bq->bv[0], folio, fsize, 0); bvecq_filled_to(bq, 1); rreq->submitted = rreq->start + fsize; rreq->progress_at = fsize; - bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor); + stream->buffered = fsize; return 0; } @@ -540,6 +579,7 @@ static int netfs_create_singular_buffer(struct netfs_io_request *rreq, struct fo static int netfs_read_gaps(struct file *file, struct folio *folio) { struct netfs_io_request *rreq; + struct netfs_io_stream *stream; struct address_space *mapping = folio->mapping; struct netfs_group *group = netfs_folio_group(folio); struct netfs_folio *finfo = netfs_folio_info(folio); @@ -561,6 +601,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio) ret = PTR_ERR(rreq); goto alloc_error; } + stream = &rreq->io_streams[0]; ret = netfs_begin_cache_read(rreq, ctx); if (ret == -ENOMEM || ret == -EINTR || ret == -ERESTARTSYS) @@ -615,6 +656,7 @@ static int netfs_read_gaps(struct file *file, struct folio *folio) bvecq_filled_to(bq, slot); rreq->submitted = rreq->start + flen; + stream->buffered = flen; netfs_read_to_pagecache(rreq); @@ -693,6 +735,7 @@ int netfs_read_folio(struct file *file, struct folio *folio) goto discard; netfs_read_to_pagecache(rreq); + ret = netfs_wait_for_read(rreq); netfs_put_request(rreq, netfs_rreq_trace_put_return); return ret < 0 ? ret : 0; diff --git a/fs/netfs/direct_read.c b/fs/netfs/direct_read.c index 40407bf9aa3f..db72814e954a 100644 --- a/fs/netfs/direct_read.c +++ b/fs/netfs/direct_read.c @@ -16,6 +16,32 @@ #include #include "internal.h" +int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) +{ + struct netfs_io_request *rreq = subreq->rreq; + struct netfs_io_stream *stream = &rreq->io_streams[0]; + size_t len; + + bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor); + bvecq_pos_set(&subreq->content, &stream->dispatch_cursor); + len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs, + &subreq->nr_segs); + + if (len < subreq->len) { + subreq->len = len; + trace_netfs_sreq(subreq, netfs_sreq_trace_limited); + } + + stream->buffered -= subreq->len; + stream->issue_from += subreq->len; + rreq->submitted = stream->issue_from; + + if (stream->buffered == 0) + netfs_all_subreqs_queued(rreq); + return 0; +} + /* * Perform a read to a buffer from the server, slicing up the region to be read * according to the network rsize. @@ -23,17 +49,15 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq) { struct netfs_io_stream *stream = &rreq->io_streams[0]; - ssize_t size = rreq->len; - uoff_t start = rreq->start; - int ret; - bvecq_pos_set(&rreq->dispatch_cursor, &rreq->load_cursor); - bvecq_pos_transfer(&rreq->collect_cursor, &rreq->load_cursor); + bvecq_pos_set(&rreq->collect_cursor, &rreq->load_cursor); + bvecq_pos_transfer(&stream->dispatch_cursor, &rreq->load_cursor); do { struct netfs_io_subrequest *subreq; + int ret; - subreq = netfs_alloc_subrequest(rreq, NETFS_DOWNLOAD_FROM_SERVER); + subreq = netfs_alloc_read_subrequest(rreq, NETFS_DOWNLOAD_FROM_SERVER); if (!subreq) { /* Stash the error in the request if there's not * already an error set. @@ -42,51 +66,32 @@ static void netfs_dispatch_unbuffered_reads(struct netfs_io_request *rreq) break; } - subreq->start = start; - subreq->len = size; - - netfs_queue_read(rreq, subreq); + subreq->start = stream->issue_from; + subreq->len = stream->buffered; netfs_stat(&netfs_n_rh_download); - if (rreq->netfs_ops->prepare_read) { - ret = rreq->netfs_ops->prepare_read(subreq); - if (ret < 0) { - netfs_cancel_read(subreq, ret); - break; - } - } - - bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor); - bvecq_pos_set(&subreq->content, &rreq->dispatch_cursor); - subreq->len = bvecq_slice(&rreq->dispatch_cursor, - umin(size, stream->sreq_max_len), - stream->sreq_max_segs, - &subreq->nr_segs); - - size -= subreq->len; - start += subreq->len; - rreq->submitted += subreq->len; - if (size <= 0) - netfs_all_subreqs_queued(rreq); - iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq, - subreq->content.slot, subreq->content.offset, subreq->len); - - rreq->netfs_ops->issue_read(subreq); + ret = rreq->netfs_ops->issue_read(subreq); + if (ret < 0) { + stream->buffered -= subreq->len; + subreq->error = ret; + netfs_read_subreq_terminated(subreq); + break; + } if (test_bit(NETFS_RREQ_PAUSE, &rreq->flags)) netfs_wait_for_paused_read(rreq); if (test_bit(NETFS_RREQ_FAILED, &rreq->flags)) break; cond_resched(); - } while (size > 0); + } while (stream->buffered > 0); - if (unlikely(size > 0)) { + if (unlikely(stream->buffered > 0)) { netfs_all_subreqs_queued(rreq); netfs_wake_collector(rreq); } - bvecq_pos_unset(&rreq->dispatch_cursor); + bvecq_pos_unset(&stream->dispatch_cursor); } /* @@ -137,6 +142,7 @@ static ssize_t netfs_unbuffered_read(struct netfs_io_request *rreq, bool sync) ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *iter) { struct netfs_io_request *rreq; + struct netfs_io_stream *stream; ssize_t ret; size_t orig_count = iov_iter_count(iter); bool sync = is_sync_kiocb(iocb); @@ -161,6 +167,8 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i netfs_stat(&netfs_n_rh_dio_read); trace_netfs_read(rreq, rreq->start, rreq->len, netfs_read_trace_dio_read); + stream = &rreq->io_streams[0]; + /* If this is an async op, we have to keep track of the destination * buffer for ourselves as the caller's iterator will be trashed when * we return. @@ -176,6 +184,8 @@ ssize_t netfs_unbuffered_read_iter_locked(struct kiocb *iocb, struct iov_iter *i goto error_put; rreq->len = ret; + stream->buffered = ret; + stream->issue_from = rreq->start; // TODO: Set up bounce buffer if needed diff --git a/fs/netfs/direct_write.c b/fs/netfs/direct_write.c index 64b444e24d79..b7e931f9123a 100644 --- a/fs/netfs/direct_write.c +++ b/fs/netfs/direct_write.c @@ -9,6 +9,34 @@ #include #include "internal.h" +/* + * Prepare the buffer for an unbuffered/DIO write. + */ +int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) +{ + struct netfs_io_stream *stream = &subreq->rreq->io_streams[subreq->stream_nr]; + size_t len; + + bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor); + bvecq_pos_set(&subreq->content, &stream->dispatch_cursor); + len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs, + &subreq->nr_segs); + + if (len < subreq->len) { + subreq->len = len; + trace_netfs_sreq(subreq, netfs_sreq_trace_limited); + } + + // TODO: Wait here for completion of prev subreq + + stream->issue_from += subreq->len; + stream->buffered -= subreq->len; + if (stream->buffered == 0) + netfs_all_subreqs_queued(subreq->rreq); + return 0; +} + /* * Perform the cleanup rituals after an unbuffered write is complete. */ @@ -74,9 +102,9 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq, wreq->transferred += subreq->transferred; if (subreq->transferred < subreq->len) { - bvecq_pos_unset(&wreq->dispatch_cursor); - bvecq_pos_transfer(&wreq->dispatch_cursor, &subreq->dispatch_pos); - bvecq_pos_advance(&wreq->dispatch_cursor, subreq->transferred); + bvecq_pos_unset(&stream->dispatch_cursor); + bvecq_pos_transfer(&stream->dispatch_cursor, &subreq->dispatch_pos); + bvecq_pos_advance(&stream->dispatch_cursor, subreq->transferred); } stream->collected_to = subreq->start + subreq->transferred; @@ -85,6 +113,7 @@ static void netfs_unbuffered_write_collect(struct netfs_io_request *wreq, trace_netfs_collect_stream(wreq, stream); trace_netfs_collect_state(wreq, wreq->collected_to, 0); + /* TODO: Progressively clean up wreq->direct_bq */ } /* @@ -103,68 +132,49 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq) _enter("%llx", wreq->len); - bvecq_pos_set(&wreq->dispatch_cursor, &wreq->load_cursor); - bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor); + stream->issue_from = wreq->start; + stream->buffered = wreq->len; + bvecq_pos_set(&stream->dispatch_cursor, &wreq->load_cursor); if (wreq->origin == NETFS_DIO_WRITE) inode_dio_begin(wreq->inode); - stream->collected_to = wreq->start; - for (;;) { bool retry = false; if (!subreq) { - netfs_prepare_write(wreq, stream, wreq->start + wreq->transferred); - subreq = stream->construct; + subreq = netfs_alloc_write_subreq(wreq, stream); if (!subreq) { - wreq->error = -ENOMEM; ret = -ENOMEM; - break; + goto failed; } - stream->construct = NULL; - } else { - bvecq_pos_set(&subreq->dispatch_pos, &wreq->dispatch_cursor); - } - - /* Check if (re-)preparation failed. */ - if (unlikely(test_bit(NETFS_SREQ_FAILED, &subreq->flags))) { - netfs_write_subrequest_terminated(subreq, subreq->error); - wreq->error = subreq->error; - break; } - subreq->len = bvecq_slice(&wreq->dispatch_cursor, stream->sreq_max_len, - stream->sreq_max_segs, &subreq->nr_segs); - bvecq_pos_set(&subreq->content, &subreq->dispatch_pos); - - iov_iter_bvec_queue(&subreq->io_iter, ITER_SOURCE, - subreq->content.bvecq, subreq->content.slot, - subreq->content.offset, - subreq->len); - - if (!iov_iter_count(&subreq->io_iter)) { - pr_warn("netfs: Unexpected zero-length slice R=%08x\n", - wreq->debug_id); - __set_bit(NETFS_SREQ_FAILED, &subreq->flags); - netfs_write_subrequest_terminated(subreq, -EIO); - wreq->error = -EIO; - break; + ret = stream->issue_write(subreq); + if (ret < 0) { + /* Ownership of subreq was returned to us. Note that + * ->dispatch_pos may or may not be initialised. + */ + trace_netfs_sreq(subreq, netfs_sreq_trace_fail); + netfs_write_subrequest_terminated(subreq, ret); + list_del_init(&subreq->rreq_link); + netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed); + subreq = NULL; + goto failed; } - trace_netfs_sreq(subreq, netfs_sreq_trace_submit); - stream->issue_write(subreq); - - /* Async, need to wait. */ - netfs_wait_for_in_progress_stream(wreq, stream); - - if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) { + ret = netfs_wait_for_in_progress_subreq(wreq, subreq); + if (ret < 0) { + if (ret != -EAGAIN) { + /* Don't need to lock here as the collection is + * done in this thread. + */ + list_del_init(&subreq->rreq_link); + netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed); + subreq = NULL; + goto failed; + } retry = true; - } else if (test_bit(NETFS_SREQ_FAILED, &subreq->flags)) { - wreq->error = subreq->error; - netfs_see_subrequest(subreq, netfs_sreq_trace_see_failed); - subreq = NULL; - break; } if (!retry) { @@ -180,20 +190,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq) continue; } - /* We need to retry the last subrequest, so first reset the - * iterator, taking into account what, if anything, we managed - * to transfer. + /* We need to retry the last subrequest, so first wind back the + * buffer position. */ subreq->error = -EAGAIN; trace_netfs_sreq(subreq, netfs_sreq_trace_retry); bvecq_pos_unset(&subreq->content); - bvecq_pos_unset(&wreq->dispatch_cursor); - bvecq_pos_transfer(&wreq->dispatch_cursor, &subreq->dispatch_pos); + bvecq_pos_unset(&stream->dispatch_cursor); + bvecq_pos_transfer(&stream->dispatch_cursor, &subreq->dispatch_pos); + stream->issue_from -= subreq->len - subreq->transferred; + stream->buffered += subreq->len - subreq->transferred; if (subreq->transferred > 0) { - wreq->transferred += subreq->transferred; - bvecq_pos_advance(&wreq->dispatch_cursor, subreq->transferred); + wreq->transferred += subreq->transferred; + bvecq_pos_advance(&stream->dispatch_cursor, subreq->transferred); } if (stream->source == NETFS_UPLOAD_TO_SERVER && @@ -202,25 +213,21 @@ static int netfs_unbuffered_write(struct netfs_io_request *wreq) __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags); __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags); - __clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags); __clear_bit(NETFS_SREQ_FAILED, &subreq->flags); - subreq->start = wreq->start + wreq->transferred; - subreq->len = wreq->len - wreq->transferred; + __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags); + subreq->start = stream->issue_from; + subreq->len = stream->buffered; subreq->transferred = 0; subreq->retry_count += 1; - stream->sreq_max_len = UINT_MAX; - stream->sreq_max_segs = INT_MAX; netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit); - if (stream->prepare_write) - stream->prepare_write(subreq); __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags); netfs_stat(&netfs_n_wh_retry_write_subreq); } - bvecq_pos_unset(&wreq->dispatch_cursor); - bvecq_pos_unset(&wreq->load_cursor); +failed: + bvecq_pos_unset(&stream->dispatch_cursor); netfs_unbuffered_write_done(wreq); _leave(" = %d", ret); return ret; @@ -264,6 +271,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter * if (IS_ERR(wreq)) return PTR_ERR(wreq); + wreq->len = iov_iter_count(iter); wreq->io_streams[0].avail = true; trace_netfs_write(wreq, (iocb->ki_flags & IOCB_DIRECT ? netfs_write_trace_dio_write : @@ -274,9 +282,7 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter * * we have to save the source buffer as the iterator is only * good until we return. In such a case, extract an iterator * to represent as much of the the output buffer as we can - * manage. Note that the extraction might not be able to - * allocate a sufficiently large bvec array and may shorten the - * request. + * manage. Note that the extraction may shorten the request. */ ssize_t n = netfs_extract_iter(iter, len, INT_MAX, &wreq->load_cursor.bvecq, 0, wreq->gfp); @@ -291,8 +297,6 @@ ssize_t netfs_unbuffered_write_iter_locked(struct kiocb *iocb, struct iov_iter * wreq->load_cursor.bvecq->max_slots); } - __set_bit(NETFS_RREQ_USE_IO_ITER, &wreq->flags); - /* Copy the data into the bounce buffer and encrypt it. */ // TODO diff --git a/fs/netfs/fscache_io.c b/fs/netfs/fscache_io.c index 056a2bae5d99..0b85bd33dfe5 100644 --- a/fs/netfs/fscache_io.c +++ b/fs/netfs/fscache_io.c @@ -239,7 +239,7 @@ void __fscache_write_to_cache(struct fscache_cookie *cookie, fscache_access_io_write) < 0) goto abandon_free; - ret = cres->ops->prepare_write(cres, &start, &len, len, i_size, false); + ret = cres->ops->prepare_write_old(cres, &start, &len, len, i_size, false); if (ret < 0) goto abandon_end; diff --git a/fs/netfs/internal.h b/fs/netfs/internal.h index e1e051ef60b4..d2b507e70a69 100644 --- a/fs/netfs/internal.h +++ b/fs/netfs/internal.h @@ -22,10 +22,10 @@ /* * buffered_read.c */ -int netfs_read_query_cache(struct netfs_io_request *rreq, - struct fscache_occupancy *occ); -void netfs_queue_read(struct netfs_io_request *rreq, - struct netfs_io_subrequest *subreq); +void netfs_read_query_cache(struct netfs_io_request *rreq, + struct fscache_occupancy *occ); +struct netfs_io_subrequest *netfs_alloc_read_subrequest(struct netfs_io_request *rreq, + enum netfs_io_source source); void netfs_cache_read_terminated(void *priv, ssize_t transferred_or_error); int netfs_prefetch_for_write(struct file *file, struct folio *folio, size_t offset, size_t len); @@ -36,6 +36,18 @@ int netfs_prefetch_for_write(struct file *file, struct folio *folio, void netfs_update_i_size(struct netfs_inode *ctx, struct inode *inode, uoff_t pos, size_t copied); +/* + * direct_read.c + */ +int netfs_prepare_unbuffered_read_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs); + +/* + * direct_write.c + */ +int netfs_prepare_unbuffered_write_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs); + /* * main.c */ @@ -74,6 +86,8 @@ struct bvecq *netfs_buffer_make_space(struct netfs_io_request *rreq, enum netfs_bvecq_trace trace); void netfs_wake_collector(struct netfs_io_request *rreq); void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq); +int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq, + struct netfs_io_subrequest *subreq); void netfs_wait_for_in_progress_stream(struct netfs_io_request *rreq, struct netfs_io_stream *stream); ssize_t netfs_wait_for_read(struct netfs_io_request *rreq); @@ -157,9 +171,18 @@ static inline bool netfs_using_pgpriv2(const struct netfs_io_request *rreq) /* * read_retry.c */ +int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs); +int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq); void netfs_retry_reads(struct netfs_io_request *rreq); void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq); +/* + * read_single.c + */ +int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs); + /* * stats.c */ @@ -230,26 +253,19 @@ void netfs_write_collection_worker(struct work_struct *work); /* * write_issue.c */ +struct netfs_writethrough; struct netfs_io_request *netfs_create_write_req(struct address_space *mapping, struct file *file, uoff_t start, enum netfs_io_origin origin); struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wreq, struct netfs_io_stream *stream); -void netfs_prepare_write(struct netfs_io_request *wreq, - struct netfs_io_stream *stream, - uoff_t start); -void netfs_reissue_write(struct netfs_io_stream *stream, - struct netfs_io_subrequest *subreq); -void netfs_issue_write(struct netfs_io_request *wreq, - struct netfs_io_stream *stream); -size_t netfs_advance_write(struct netfs_io_request *wreq, - struct netfs_io_stream *stream, - uoff_t start, size_t len, bool to_eof); /* * write_retry.c */ +int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs); void netfs_retry_writes(struct netfs_io_request *wreq); /* diff --git a/fs/netfs/main.c b/fs/netfs/main.c index b8da5e85cc67..3736a2a7662d 100644 --- a/fs/netfs/main.c +++ b/fs/netfs/main.c @@ -18,6 +18,7 @@ MODULE_DESCRIPTION("Network fs support"); MODULE_AUTHOR("Red Hat, Inc."); MODULE_LICENSE("GPL"); +EXPORT_TRACEPOINT_SYMBOL(netfs_rreq); EXPORT_TRACEPOINT_SYMBOL(netfs_sreq); unsigned netfs_debug; diff --git a/fs/netfs/misc.c b/fs/netfs/misc.c index 4fa09560fb4b..a729e51a8ab2 100644 --- a/fs/netfs/misc.c +++ b/fs/netfs/misc.c @@ -250,6 +250,37 @@ void netfs_subreq_clear_in_progress(struct netfs_io_subrequest *subreq) netfs_wake_collector(rreq); } +/* + * Wait for a subrequest to come to completion. + */ +int netfs_wait_for_in_progress_subreq(struct netfs_io_request *rreq, + struct netfs_io_subrequest *subreq) +{ + if (netfs_check_subreq_in_progress(subreq)) { + DEFINE_WAIT(myself); + + trace_netfs_rreq(rreq, netfs_rreq_trace_wait_quiesce); + for (;;) { + prepare_to_wait(&rreq->waitq, &myself, TASK_UNINTERRUPTIBLE); + + if (!netfs_check_subreq_in_progress(subreq)) + break; + + trace_netfs_sreq(subreq, netfs_sreq_trace_wait_for); + schedule(); + } + + trace_netfs_rreq(rreq, netfs_rreq_trace_waited_quiesce); + finish_wait(&rreq->waitq, &myself); + } + + if (test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) + return -EAGAIN; + if (test_bit(NETFS_SREQ_FAILED, &subreq->flags)) + return subreq->error; + return 0; +} + /* * Wait for all outstanding I/O in a stream to quiesce. */ @@ -380,7 +411,7 @@ static ssize_t netfs_wait_for_in_progress(struct netfs_io_request *rreq, case NETFS_UNBUFFERED_WRITE: break; default: - if (rreq->submitted < rreq->len) { + if (rreq->transferred < rreq->len) { trace_netfs_failure(rreq, NULL, ret, netfs_fail_short_read); ret = -EIO; } diff --git a/fs/netfs/objects.c b/fs/netfs/objects.c index 64fbb6e6c445..fdd7fe6122da 100644 --- a/fs/netfs/objects.c +++ b/fs/netfs/objects.c @@ -55,8 +55,6 @@ struct netfs_io_request *netfs_alloc_request(struct address_space *mapping, rreq->i_size = i_size_read(inode); rreq->debug_id = atomic_inc_return(&debug_ids); rreq->wsize = INT_MAX; - rreq->io_streams[0].sreq_max_len = ULONG_MAX; - rreq->io_streams[0].sreq_max_segs = 0; spin_lock_init(&rreq->lock); init_waitqueue_head(&rreq->waitq); refcount_set(&rreq->ref, 2); @@ -150,8 +148,8 @@ static void netfs_deinit_request(struct netfs_io_request *rreq) if (rreq->cache_resources.ops) rreq->cache_resources.ops->end_operation(&rreq->cache_resources); bvecq_pos_unset(&rreq->load_cursor); - bvecq_pos_unset(&rreq->dispatch_cursor); bvecq_pos_unset(&rreq->collect_cursor); + bvecq_pos_unset(&rreq->retry_cursor); bvecq_put(rreq->spare); while (rreq->writebacks) { struct netfs_writeback *wback = rreq->writebacks; diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c index 11cc1ac37fc5..17af3470ab73 100644 --- a/fs/netfs/read_collect.c +++ b/fs/netfs/read_collect.c @@ -35,6 +35,7 @@ static void netfs_clear_unread(struct netfs_io_subrequest *subreq) if (subreq->start + subreq->transferred >= subreq->rreq->i_size) __set_bit(NETFS_SREQ_HIT_EOF, &subreq->flags); + trace_netfs_rreq(subreq->rreq, netfs_rreq_trace_zero_unread); } /* @@ -415,8 +416,7 @@ static void netfs_collect_read_results(struct netfs_io_request *rreq) stream->collected_to = front->start + transferred; rreq->collected_to = stream->collected_to; - if (front->start + transferred >= unlock_at || - test_bit(NETFS_SREQ_HIT_EOF, &front_flags)) + if (front->start + transferred >= unlock_at) netfs_read_unlock_folios(rreq, ¬es); } else { stream->collected_to = front->start + transferred; @@ -508,31 +508,6 @@ static void netfs_rreq_assess_dio(struct netfs_io_request *rreq) inode_dio_end(rreq->inode); } -/* - * Do processing after reading a monolithic single object. - */ -static void netfs_rreq_assess_single(struct netfs_io_request *rreq) -{ - struct netfs_io_stream *stream = &rreq->io_streams[0]; - - if (!rreq->error && stream->source == NETFS_DOWNLOAD_FROM_SERVER && - fscache_resources_valid(&rreq->cache_resources)) { - trace_netfs_rreq(rreq, netfs_rreq_trace_dirty); - netfs_single_mark_inode_dirty(rreq->inode); - } - - if (rreq->iocb) { - rreq->iocb->ki_pos += rreq->transferred; - if (rreq->iocb->ki_complete) { - trace_netfs_rreq(rreq, netfs_rreq_trace_ki_complete); - rreq->iocb->ki_complete( - rreq->iocb, rreq->error ? rreq->error : rreq->transferred); - } - } - if (rreq->netfs_ops->done) - rreq->netfs_ops->done(rreq); -} - /* * Perform the collection of subrequests and folios. * @@ -566,7 +541,7 @@ bool netfs_read_collection(struct netfs_io_request *rreq) netfs_rreq_assess_dio(rreq); break; case NETFS_READ_SINGLE: - netfs_rreq_assess_single(rreq); + WARN_ON_ONCE(1); break; default: break; @@ -700,6 +675,11 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq) } else if (test_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags)) { __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags); trace_netfs_sreq(subreq, netfs_sreq_trace_partial_read); + } else if (subreq->source == NETFS_READ_FROM_CACHE) { + netfs_stat(&netfs_n_rh_read_failed); + __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags); + subreq->error = -ENODATA; + trace_netfs_sreq(subreq, netfs_sreq_trace_short); } else { __set_bit(NETFS_SREQ_FAILED, &subreq->flags); subreq->error = -ENODATA; @@ -718,6 +698,8 @@ void netfs_read_subreq_terminated(struct netfs_io_subrequest *subreq) if (unlikely(subreq->error < 0)) { trace_netfs_failure(rreq, subreq, subreq->error, netfs_fail_read); + if (subreq->error == -ENOMEM) + set_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags); if (subreq->source == NETFS_READ_FROM_CACHE) { netfs_stat(&netfs_n_rh_read_failed); __set_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags); diff --git a/fs/netfs/read_pgpriv2.c b/fs/netfs/read_pgpriv2.c index 6a81a394eeac..8c5093820b07 100644 --- a/fs/netfs/read_pgpriv2.c +++ b/fs/netfs/read_pgpriv2.c @@ -13,8 +13,37 @@ #include #include "internal.h" +int netfs_prepare_pgpriv2_write_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) +{ + struct netfs_io_request *creq = subreq->rreq; + struct netfs_io_stream *stream = &creq->io_streams[1]; + size_t len; + + bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor); + bvecq_pos_set(&subreq->content, &stream->dispatch_cursor); + len = bvecq_slice(&stream->dispatch_cursor, subreq->len, max_segs, + &subreq->nr_segs); + + if (len < subreq->len) { + subreq->len = len; + trace_netfs_sreq(subreq, netfs_sreq_trace_limited); + } + + // TODO: Wait here for completion of prev subreq + + stream->issue_from += subreq->len; + stream->buffered -= subreq->len; + if (stream->buffered == 0) + netfs_all_subreqs_queued(creq); + return 0; +} + /* - * [DEPRECATED] Copy a folio to the cache with PG_private_2 set. + * [DEPRECATED] Copy a folio to the cache with PG_private_2 set. Note that the + * folio won't necessarily be contiguous with the previous one as there might + * be a mixture of folios read from the cache and downloaded from the server + * (or just zeroed). */ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio *folio) { @@ -24,7 +53,6 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio size_t dio_size = PAGE_SIZE; size_t fsize = folio_size(folio), flen = fsize; uoff_t fpos = folio_pos(folio), i_size; - bool to_eof = false; _enter(""); @@ -44,12 +72,8 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio if (fpos + fsize > creq->i_size) creq->i_size = i_size; - if (flen > i_size - fpos) { + if (flen > i_size - fpos) flen = i_size - fpos; - to_eof = true; - } else if (flen == i_size - fpos) { - to_eof = true; - } flen = round_up(flen, dio_size); @@ -63,7 +87,7 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio */ queue = creq->load_cursor.bvecq; if (bvecq_is_full(queue) || - (fpos != creq->last_end && creq->last_end > 0 && queue->nr_slots > 0)) { + (fpos != cache->last_end && cache->last_end > 0 && queue->nr_slots > 0)) { bvecq_buffer_append(&creq->load_cursor, creq->spare); creq->spare = NULL; @@ -78,38 +102,10 @@ static void netfs_pgpriv2_copy_folio(struct netfs_io_request *creq, struct folio bvecq_filled_to(queue, slot); creq->load_cursor.slot = slot; creq->load_cursor.offset = 0; - creq->last_end = fpos + flen; - - bvecq_pos_nudge(&creq->dispatch_cursor); - - cache->submit_off = 0; - cache->submit_len = flen; + cache->last_end = fpos + flen; + trace_netfs_wback(creq, folio, 0); - /* Attach the folio to one or more subrequests. For a big folio, we - * could end up with thousands of subrequests if the wsize is small - - * but we might need to wait during the creation of subrequests for - * network resources (eg. SMB credits). - */ - do { - ssize_t part; - - creq->dispatch_cursor.offset = cache->submit_off; - - atomic64_set(&cache->issued_to, fpos + cache->submit_off); - part = netfs_advance_write(creq, cache, fpos + cache->submit_off, - cache->submit_len, to_eof); - cache->submit_off += part; - if (part > cache->submit_len) - cache->submit_len = 0; - else - cache->submit_len -= part; - } while (cache->submit_len > 0); - - bvecq_pos_step(&creq->dispatch_cursor); - atomic64_set(&cache->issued_to, fpos + fsize); - - if (flen < fsize) - netfs_issue_write(creq, cache); + cache->buffered += flen; } /* @@ -119,6 +115,7 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache( struct netfs_io_request *rreq, struct folio *folio) { struct netfs_io_request *creq; + struct netfs_io_stream *cache; if (!fscache_resources_valid(&rreq->cache_resources)) goto cancel; @@ -128,13 +125,14 @@ static struct netfs_io_request *netfs_pgpriv2_begin_copy_to_cache( if (IS_ERR(creq)) goto cancel; - if (!creq->io_streams[1].avail) + cache = &creq->io_streams[1]; + if (!cache->avail) goto cancel_put; if (bvecq_buffer_init(&creq->load_cursor, creq->gfp, false) < 0) goto cancel_put; - bvecq_pos_set(&creq->dispatch_cursor, &creq->load_cursor); - bvecq_pos_set(&creq->collect_cursor, &creq->dispatch_cursor); + bvecq_pos_set(&cache->dispatch_cursor, &creq->load_cursor); + bvecq_pos_set(&creq->collect_cursor, &creq->load_cursor); __set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &creq->flags); trace_netfs_copy2cache(rreq, creq); @@ -180,24 +178,61 @@ void netfs_pgpriv2_copy_to_cache(struct netfs_io_request *rreq, struct folio *fo netfs_pgpriv2_copy_folio(creq, folio); } +/* + * Issue all pending writes on the cache stream. + */ +static void netfs_pgpriv2_issue_stream(struct netfs_io_request *wreq, + struct netfs_io_stream *stream) +{ + atomic64_set_release(&stream->issued_to, wreq->start); + + do { + struct netfs_io_subrequest *subreq; + int ret; + + subreq = netfs_alloc_write_subreq(wreq, stream); + if (!subreq) + break; + + ret = stream->issue_write(subreq); + if (ret < 0) { + /* Ownership of subreq was returned to us. Punt the + * error to the collector and stop the issuance of new + * subreqs. + */ + trace_netfs_sreq(subreq, netfs_sreq_trace_fail); + stream->buffered -= subreq->len; + netfs_write_subrequest_terminated(subreq, ret); + break; + } + /* We no longer own subreq. */ + + if (test_bit(NETFS_RREQ_SAW_ENOMEM, &wreq->flags)) + break; + + } while (stream->buffered > 0); + + netfs_all_subreqs_queued(wreq); +} + /* * [DEPRECATED] End writing to the cache, flushing out any outstanding writes. */ void netfs_pgpriv2_end_copy_to_cache(struct netfs_io_request *rreq) { struct netfs_io_request *creq = rreq->copy_to_cache; + struct netfs_io_stream *stream = &creq->io_streams[1]; if (IS_ERR_OR_NULL(creq)) return; - netfs_issue_write(creq, &creq->io_streams[1]); - netfs_all_subreqs_queued(creq); + netfs_pgpriv2_issue_stream(creq, stream); trace_netfs_rreq(rreq, netfs_rreq_trace_end_copy_to_cache); if (list_empty_careful(&creq->io_streams[1].subrequests)) netfs_wake_collector(creq); netfs_put_request(creq, netfs_rreq_trace_put_return); - creq->copy_to_cache = NULL; + rreq->copy_to_cache = NULL; } /* diff --git a/fs/netfs/read_retry.c b/fs/netfs/read_retry.c index 08b0bf526759..e2afff62d380 100644 --- a/fs/netfs/read_retry.c +++ b/fs/netfs/read_retry.c @@ -9,20 +9,55 @@ #include #include "internal.h" -static void netfs_reissue_read(struct netfs_io_request *rreq, - struct netfs_io_subrequest *subreq) +/* + * Prepare the I/O buffer on a buffered read subrequest for the filesystem to + * use as a bvec queue. + */ +int netfs_prepare_buffered_read_retry_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) { - bvecq_pos_unset(&subreq->content); + struct netfs_io_request *rreq = subreq->rreq; + size_t len; + + bvecq_pos_set(&subreq->dispatch_pos, &rreq->retry_cursor); bvecq_pos_set(&subreq->content, &subreq->dispatch_pos); - iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq, - subreq->content.slot, subreq->content.offset, subreq->len); - iov_iter_advance(&subreq->io_iter, subreq->transferred); + len = bvecq_slice(&rreq->retry_cursor, subreq->len, max_segs, + &subreq->nr_segs); + if (len < subreq->len) { + subreq->len = len; + trace_netfs_sreq(subreq, netfs_sreq_trace_limited); + } + rreq->retry_buffered -= subreq->len; + rreq->retry_start += subreq->len; + return 0; +} - subreq->error = 0; +/* + * Reset the state of the subrequest and discard any buffering so that we can + * retry (where this may include sending it to the server instead of the + * cache). + */ +int netfs_reset_for_read_retry(struct netfs_io_subrequest *subreq) +{ + trace_netfs_sreq(subreq, netfs_sreq_trace_retry); + + if (subreq->retry_count > 3) { + trace_netfs_sreq(subreq, netfs_sreq_trace_too_many_retries); + return subreq->error; + } + + subreq->retry_count++; __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags); + __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags); + __clear_bit(NETFS_SREQ_FAILED, &subreq->flags); __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags); - netfs_stat(&netfs_n_rh_retry_read_subreq); - subreq->rreq->netfs_ops->issue_read(subreq); + bvecq_pos_unset(&subreq->content); + bvecq_pos_unset(&subreq->dispatch_pos); + subreq->error = 0; + subreq->transferred = 0; + netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit); + netfs_stat(&netfs_n_wh_retry_write_subreq); + return 0; } /* @@ -33,8 +68,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) { struct netfs_io_subrequest *subreq; struct netfs_io_stream *stream = &rreq->io_streams[0]; - struct bvecq_pos dispatch_cursor = {}; struct list_head *next; + int ret; _enter("R=%x", rreq->debug_id); @@ -44,46 +79,19 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) if (rreq->netfs_ops->retry_request) rreq->netfs_ops->retry_request(rreq, NULL); - /* If there's no renegotiation to do, just resend each retryable subreq - * up to the first permanently failed one. - */ - if (!rreq->netfs_ops->prepare_read && - !rreq->cache_resources.ops) { - list_for_each_entry(subreq, &stream->subrequests, rreq_link) { - if (test_bit(NETFS_SREQ_FAILED, &subreq->flags)) - break; - if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) { - subreq->retry_count++; - netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit); - netfs_reissue_read(rreq, subreq); - } - } - return; - } + /* Read pointer to subreq before reading subreq state. */ + next = smp_load_acquire(&stream->subrequests.next); - /* Okay, we need to renegotiate all the download requests and flip any - * failed cache reads over to being download requests and negotiate - * those also. All fully successful subreqs have been removed from the - * list and any spare data from those has been donated. - * - * What we do is decant the list and rebuild it one subreq at a time so - * that we don't end up with donations jumping over a gap we're busy - * populating with smaller subrequests. In the event that the subreq - * we just launched finishes before we insert the next subreq, it'll - * fill in rreq->prev_donated instead. - * - * Note: Alternatively, we could split the tail subrequest right before - * we reissue it and fix up the donations under lock. + /* Renegotiate all the download requests and flip any failed cache + * reads over to being download requests and negotiate those also. */ - next = stream->subrequests.next; - do { struct netfs_io_subrequest *from, *to, *tmp; - uoff_t start, len; - size_t part; - bool boundary = false, subreq_superfluous = false; + uoff_t start; + size_t len; + bool subreq_superfluous = false; - bvecq_pos_unset(&dispatch_cursor); + bvecq_pos_unset(&rreq->retry_cursor); /* Go through the subreqs and find the next span of contiguous * buffer that we then rejig (cifs, for example, needs the @@ -98,8 +106,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) rreq->debug_id, from->debug_index, from->start, from->transferred, from->len); - if (test_bit(NETFS_SREQ_FAILED, &from->flags) || - !test_bit(NETFS_SREQ_NEED_RETRY, &from->flags)) { + if (!test_bit(NETFS_SREQ_NEED_RETRY, &from->flags)) { subreq = from; goto abandon; } @@ -113,20 +120,21 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) subreq = list_entry(next, struct netfs_io_subrequest, rreq_link); if (subreq->start != start + len || subreq->transferred > 0 || - test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) || !test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) break; to = subreq; len += to->len; } - _debug(" - range: %llx-%llx %llx", start, start + len - 1, len); + _debug(" - range: %llx-%llx %zx", start, start + len - 1, len); /* Determine the set of buffers we're going to use. Each - * subreq gets a subset of a single overall contiguous buffer. + * subreq takes a subset of a single overall contiguous buffer. */ - bvecq_pos_transfer(&dispatch_cursor, &from->dispatch_pos); - bvecq_pos_advance(&dispatch_cursor, from->transferred); + bvecq_pos_transfer(&rreq->retry_cursor, &from->dispatch_pos); + bvecq_pos_advance(&rreq->retry_cursor, from->transferred); + rreq->retry_start = start; + rreq->retry_buffered = len; from->transferred = 0; /* Work through the sublist. The chain of buffers we're going @@ -135,51 +143,30 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) */ subreq = from; list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) { - if (!len) { + if (rreq->retry_buffered == 0) { subreq_superfluous = true; break; } subreq->source = NETFS_DOWNLOAD_FROM_SERVER; - subreq->start = start; - subreq->len = len; - __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags); - __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags); - subreq->retry_count++; - subreq->transferred = 0; + subreq->start = rreq->retry_start; + subreq->len = rreq->retry_buffered; - bvecq_pos_unset(&subreq->content); - bvecq_pos_unset(&subreq->dispatch_pos); - bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor); - - trace_netfs_sreq(subreq, netfs_sreq_trace_retry); - - /* Renegotiate max_len (rsize) */ - stream->sreq_max_len = len; - stream->sreq_max_segs = INT_MAX; - if (rreq->netfs_ops->prepare_read && - rreq->netfs_ops->prepare_read(subreq) < 0) { - trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed); + ret = netfs_reset_for_read_retry(subreq); + if (ret < 0) { __set_bit(NETFS_SREQ_FAILED, &subreq->flags); + rreq->error = ret; goto abandon; } - part = bvecq_slice(&dispatch_cursor, - umin(len, stream->sreq_max_len), - stream->sreq_max_segs, - &subreq->nr_segs); - subreq->len = part; - - len -= part; - start += part; - if (!len) { - if (boundary) - __set_bit(NETFS_SREQ_BOUNDARY, &subreq->flags); - } else { - __clear_bit(NETFS_SREQ_BOUNDARY, &subreq->flags); + netfs_stat(&netfs_n_rh_download); + ret = rreq->netfs_ops->issue_read(subreq); + if (ret < 0) { + subreq->error = ret; + netfs_read_subreq_terminated(subreq); + goto abandon_after; } - - netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit); - netfs_reissue_read(rreq, subreq); + if (test_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags)) + goto abandon_after; if (subreq == to) { subreq_superfluous = false; break; @@ -189,7 +176,7 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) /* If we managed to use fewer subreqs, we can discard the * excess; if we used the same number, then we're done. */ - if (!len) { + if (rreq->retry_buffered == 0) { if (!subreq_superfluous) continue; list_for_each_entry_safe_from(subreq, tmp, @@ -207,7 +194,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) } /* We ran out of subrequests, so we need to allocate some more - * and insert them after. + * and insert them after. They must start with being marked + * for retry to switch to the retry cursor. */ do { subreq = netfs_alloc_subrequest(rreq, NETFS_DOWNLOAD_FROM_SERVER); @@ -215,8 +203,8 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) subreq = to; goto abandon_after; } - subreq->start = start; - subreq->len = len; + subreq->start = rreq->retry_start; + subreq->len = rreq->retry_buffered; subreq->stream_nr = stream->stream_nr; subreq->retry_count = 1; @@ -224,43 +212,32 @@ static void netfs_retry_read_subrequests(struct netfs_io_request *rreq) refcount_read(&subreq->ref), netfs_sreq_trace_new); + __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags); + spin_lock(&rreq->lock); + /* Write IN_PROGRESS before pointer to new subreq */ + smp_wmb(); list_add(&subreq->rreq_link, &to->rreq_link); spin_unlock(&rreq->lock); to = subreq; trace_netfs_sreq(subreq, netfs_sreq_trace_retry); - stream->sreq_max_len = umin(len, rreq->rsize); - stream->sreq_max_segs = INT_MAX; - netfs_stat(&netfs_n_rh_download); - if (rreq->netfs_ops->prepare_read(subreq) < 0) { - trace_netfs_sreq(subreq, netfs_sreq_trace_reprep_failed); - __set_bit(NETFS_SREQ_FAILED, &subreq->flags); - goto abandon; - } - - bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor); - part = bvecq_slice(&dispatch_cursor, - umin(len, stream->sreq_max_len), - stream->sreq_max_segs, - &subreq->nr_segs); - subreq->len = part; - - len -= part; - start += part; - if (!len && boundary) { - __set_bit(NETFS_SREQ_BOUNDARY, &to->flags); - boundary = false; + ret = rreq->netfs_ops->issue_read(subreq); + if (ret < 0) { + subreq->error = ret; + netfs_read_subreq_terminated(subreq); + goto abandon_after; } + if (test_bit(NETFS_RREQ_SAW_ENOMEM, &rreq->flags)) + goto abandon_after; - netfs_reissue_read(rreq, subreq); - } while (len); + } while (rreq->retry_buffered > 0); } while (!list_is_head(next, &stream->subrequests)); out: - bvecq_pos_unset(&dispatch_cursor); + bvecq_pos_unset(&rreq->retry_cursor); return; /* If we hit an error, fail all remaining incomplete subrequests */ @@ -335,6 +312,7 @@ void netfs_unlock_abandoned_read_pages(struct netfs_io_request *rreq) } trace_netfs_folio(folio, netfs_folio_trace_abandon); folio_unlock(folio); + p->bv[slot].bv_page = NULL; } } } diff --git a/fs/netfs/read_single.c b/fs/netfs/read_single.c index 46dc22c3859b..8636ae423709 100644 --- a/fs/netfs/read_single.c +++ b/fs/netfs/read_single.c @@ -16,6 +16,22 @@ #include #include "internal.h" +int netfs_prepare_read_single_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) +{ + struct netfs_io_request *rreq = subreq->rreq; + struct netfs_io_stream *stream = &rreq->io_streams[0]; + + bvecq_pos_set(&subreq->dispatch_pos, &rreq->load_cursor); + bvecq_pos_set(&subreq->content, &subreq->dispatch_pos); + + stream->buffered = 0; + stream->issue_from += subreq->len; + rreq->submitted = stream->issue_from; + netfs_all_subreqs_queued(rreq); + return 0; +} + /** * netfs_single_mark_inode_dirty - Mark a single, monolithic object inode dirty * @inode: The inode to mark @@ -58,17 +74,6 @@ static int netfs_single_begin_cache_read(struct netfs_io_request *rreq, struct n return fscache_begin_read_operation(&rreq->cache_resources, netfs_i_cookie(ctx)); } -static void netfs_single_read_cache(struct netfs_io_request *rreq, - struct netfs_io_subrequest *subreq) -{ - struct netfs_cache_resources *cres = &rreq->cache_resources; - - _enter("R=%08x[%x]", rreq->debug_id, subreq->debug_index); - netfs_stat(&netfs_n_rh_read); - cres->ops->read(cres, subreq->start, &subreq->io_iter, NETFS_READ_HOLE_FAIL, - netfs_cache_read_terminated, subreq); -} - /* * Perform a read to a buffer from the cache or the server. Only a single * subreq is permitted as the object must be fetched in a single transaction. @@ -84,70 +89,84 @@ static int netfs_single_dispatch_read(struct netfs_io_request *rreq) .cached_to[1] = ULLONG_MAX, }; struct netfs_io_subrequest *subreq; - int ret = 0; + int ret; + + netfs_read_query_cache(rreq, &occ); - subreq = netfs_alloc_subrequest(rreq, NETFS_DOWNLOAD_FROM_SERVER); + subreq = netfs_alloc_read_subrequest(rreq, NETFS_DOWNLOAD_FROM_SERVER); if (!subreq) return -ENOMEM; - subreq->source = NETFS_DOWNLOAD_FROM_SERVER; subreq->start = 0; subreq->len = rreq->len; - bvecq_pos_set(&subreq->dispatch_pos, &rreq->dispatch_cursor); - bvecq_pos_set(&subreq->content, &rreq->dispatch_cursor); - - iov_iter_bvec_queue(&subreq->io_iter, ITER_DEST, subreq->content.bvecq, - subreq->content.slot, subreq->content.offset, subreq->len); - - netfs_queue_read(rreq, subreq); + trace_netfs_sreq(subreq, netfs_sreq_trace_prepare); /* Try to use the cache if the cache content matches the size of the * remote file. */ - netfs_read_query_cache(rreq, &occ); if (occ.cached_from[0] == 0 && - occ.cached_to[0] >= rreq->len) + occ.cached_to[0] >= rreq->len) { + struct netfs_cache_resources *cres = &rreq->cache_resources; + subreq->source = NETFS_READ_FROM_CACHE; + netfs_stat(&netfs_n_rh_read); + ret = cres->ops->issue_read(subreq); + if (ret < 0) { + subreq->error = ret; + netfs_read_subreq_terminated(subreq); + } + + ret = netfs_wait_for_in_progress_subreq(rreq, subreq); + if (ret == 0) + goto success; /* 0 -> No copy to cache */ + if (ret == -ENOMEM) + goto cancel; + + /* Didn't manage to retrieve from the cache, so toss it to the + * server instead. + */ + if (netfs_reset_for_read_retry(subreq) < 0) + goto cancel; + } - switch (subreq->source) { - case NETFS_DOWNLOAD_FROM_SERVER: + /* Try to read from the server. */ + for (;;) { + subreq->source = NETFS_DOWNLOAD_FROM_SERVER; netfs_stat(&netfs_n_rh_download); - if (rreq->netfs_ops->prepare_read) { - ret = rreq->netfs_ops->prepare_read(subreq); - if (ret < 0) - goto cancel; + ret = rreq->netfs_ops->issue_read(subreq); + if (ret < 0) { + subreq->error = ret; + netfs_read_subreq_terminated(subreq); } - netfs_all_subreqs_queued(rreq); - rreq->netfs_ops->issue_read(subreq); - rreq->submitted += subreq->len; - break; - case NETFS_READ_FROM_CACHE: - if (rreq->cache_resources.ops->prepare_read) { - ret = rreq->cache_resources.ops->prepare_read(subreq); - if (ret < 0) - goto cancel; + ret = netfs_wait_for_in_progress_subreq(rreq, subreq); + if (ret == 0) { + ret = 1; /* 1 -> Schedule copy to cache. */ + goto success; } - - netfs_all_subreqs_queued(rreq); - trace_netfs_sreq(subreq, netfs_sreq_trace_submit); - netfs_single_read_cache(rreq, subreq); - rreq->submitted += subreq->len; - ret = 0; - break; - default: - pr_warn("Unexpected single-read source %u\n", subreq->source); - WARN_ON_ONCE(true); - ret = -EIO; - goto cancel; + if (ret == -ENOMEM) + goto cancel; + if (ret != -EAGAIN) + goto failed; + if (netfs_reset_for_read_retry(subreq) < 0) + goto cancel; } +success: + rreq->transferred = subreq->transferred; + list_del_init(&subreq->rreq_link); + netfs_put_subrequest(subreq, netfs_sreq_trace_put_consumed); return ret; cancel: - netfs_cancel_read(subreq, ret); - netfs_all_subreqs_queued(rreq); - netfs_wake_collector(rreq); + rreq->error = ret; + list_del_init(&subreq->rreq_link); + netfs_put_subrequest(subreq, netfs_sreq_trace_put_cancel); + return ret; +failed: + rreq->error = ret; + list_del_init(&subreq->rreq_link); + netfs_put_subrequest(subreq, netfs_sreq_trace_put_failed); return ret; } @@ -179,7 +198,7 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite if (IS_ERR(rreq)) return PTR_ERR(rreq); - ret = netfs_extract_iter(iter, rreq->len, INT_MAX, &rreq->dispatch_cursor.bvecq, + ret = netfs_extract_iter(iter, rreq->len, INT_MAX, &rreq->load_cursor.bvecq, 0, rreq->gfp); if (ret < 0) goto cleanup_free; @@ -197,9 +216,29 @@ ssize_t netfs_read_single(struct inode *inode, struct file *file, struct iov_ite netfs_stat(&netfs_n_rh_read_single); trace_netfs_read(rreq, 0, rreq->len, netfs_read_trace_read_single); - netfs_single_dispatch_read(rreq); + ret = netfs_single_dispatch_read(rreq); + + trace_netfs_rreq(rreq, netfs_rreq_trace_complete); + if (ret >= 0) { + task_io_account_read(rreq->transferred); + + if (ret == 1 && + fscache_resources_valid(&rreq->cache_resources)) { + trace_netfs_rreq(rreq, netfs_rreq_trace_dirty); + netfs_single_mark_inode_dirty(rreq->inode); + } + ret = rreq->transferred; + } + + if (rreq->netfs_ops->done) + rreq->netfs_ops->done(rreq); + + netfs_wake_rreq_flag(rreq, NETFS_RREQ_IN_PROGRESS, netfs_rreq_trace_wake_ip); + /* As we cleared NETFS_RREQ_IN_PROGRESS, we acquired its ref. */ + netfs_put_request(rreq, netfs_rreq_trace_put_work_ip); + + trace_netfs_rreq(rreq, netfs_rreq_trace_done); - ret = netfs_wait_for_read(rreq); netfs_put_request(rreq, netfs_rreq_trace_put_return); return ret; diff --git a/fs/netfs/write_collect.c b/fs/netfs/write_collect.c index b8caf15b2a8c..678058d77308 100644 --- a/fs/netfs/write_collect.c +++ b/fs/netfs/write_collect.c @@ -566,6 +566,8 @@ void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error) if (IS_ERR_VALUE(transferred_or_error)) { subreq->error = transferred_or_error; + if (transferred_or_error == -ENOMEM) + set_bit(NETFS_RREQ_SAW_ENOMEM, &wreq->flags); switch (subreq->source) { case NETFS_WRITE_TO_CACHE: diff --git a/fs/netfs/write_issue.c b/fs/netfs/write_issue.c index 08bc84182847..0a79da073f03 100644 --- a/fs/netfs/write_issue.c +++ b/fs/netfs/write_issue.c @@ -68,6 +68,9 @@ struct netfs_wb_params { struct netfs_write_estimate estimates[NR_IO_STREAMS]; }; +static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs); + /* * Kill all dirty folios in the event of an unrecoverable error, starting with * a locked folio we've already obtained from writeback_iter(). @@ -148,7 +151,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping, wreq->io_streams[0].source = NETFS_UPLOAD_TO_SERVER; wreq->io_streams[0].applicable = NOTE_UPLOAD; wreq->io_streams[0].estimate_write = ictx->ops->estimate_write; - wreq->io_streams[0].prepare_write = ictx->ops->prepare_write; wreq->io_streams[0].issue_write = ictx->ops->issue_write; wreq->io_streams[0].collected_to = start; wreq->io_streams[0].transferred = 0; @@ -162,7 +164,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping, wreq->io_streams[1].avail = true; wreq->io_streams[1].active = true; wreq->io_streams[1].estimate_write = wreq->cache_resources.ops->estimate_write; - wreq->io_streams[1].prepare_write = wreq->cache_resources.ops->prepare_write_subreq; wreq->io_streams[1].issue_write = wreq->cache_resources.ops->issue_write; wreq->io_streams[1].alignment = wreq->cache_resources.dio_size; } @@ -170,19 +171,6 @@ struct netfs_io_request *netfs_create_write_req(struct address_space *mapping, return wreq; } -/** - * netfs_prepare_write_failed - Note write preparation failed - * @subreq: The subrequest to mark - * - * Mark a subrequest to note that preparation for write failed. - */ -void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq) -{ - __set_bit(NETFS_SREQ_FAILED, &subreq->flags); - trace_netfs_sreq(subreq, netfs_sreq_trace_prep_failed); -} -EXPORT_SYMBOL(netfs_prepare_write_failed); - /* * Allocate and prepare a write subrequest. Will only return NULL if not * performing writeback; if performing writeback, mempools may be accessed and @@ -234,6 +222,7 @@ struct netfs_io_subrequest *netfs_alloc_write_subreq(struct netfs_io_request *wr return subreq; } +#if 0 // TODO: Remove old stuff /* * Prepare a write subrequest. We need to allocate a new subrequest * if we don't have one. @@ -295,6 +284,7 @@ void netfs_prepare_write(struct netfs_io_request *wreq, stream->construct = subreq; } +#endif /* * Advance the state of the amount of data buffered on a stream. @@ -341,6 +331,44 @@ static int netfs_prepare_buffered_write_buffer(struct netfs_io_subrequest *subre return 0; } +/** + * netfs_prepare_write_buffer - Get the buffer for a subrequest + * @subreq: The subrequest to get the buffer for + * @max_segs: Maximum number of segments in buffer (or INT_MAX) + * + * Extract a slice of buffer from the stream and attach it to the subrequest as + * a bio_vec queue. The maximum amount of data attached is set by + * @subreq->len, but this may be shortened if @max_segs would be exceeded. + */ +int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) +{ + struct netfs_io_request *rreq = subreq->rreq; + + switch (rreq->origin) { + case NETFS_WRITEBACK: + if (test_bit(NETFS_RREQ_RETRYING, &rreq->flags)) + return netfs_prepare_write_retry_buffer(subreq, max_segs); + return netfs_prepare_buffered_write_buffer(subreq, max_segs); + + case NETFS_UNBUFFERED_WRITE: + case NETFS_DIO_WRITE: + return netfs_prepare_unbuffered_write_buffer(subreq, max_segs); + + case NETFS_WRITEBACK_SINGLE: + return netfs_prepare_write_single_buffer(subreq, max_segs); + + case NETFS_PGPRIV2_COPY_TO_CACHE: + return netfs_prepare_pgpriv2_write_buffer(subreq, max_segs); + + default: + WARN_ON_ONCE(1); + return -EIO; + } +} +EXPORT_SYMBOL(netfs_prepare_write_buffer); + +#if 0 // TODO: Remove old stuff /* * Set the I/O iterator for the filesystem/cache to use and dispatch the I/O * operation. The operation may be asynchronous and should call @@ -395,8 +423,8 @@ void netfs_reissue_write(struct netfs_io_stream *stream, netfs_do_issue_write(stream, subreq); } -void netfs_issue_write(struct netfs_io_request *wreq, - struct netfs_io_stream *stream) +static void netfs_issue_write(struct netfs_io_request *wreq, + struct netfs_io_stream *stream) { struct netfs_io_subrequest *subreq = stream->construct; @@ -491,6 +519,7 @@ static int netfs_prep_and_issue_subreq(struct netfs_io_request *wreq, stream->issue_write(subreq); return 0; } +#endif /* * Issue writes for a stream. @@ -523,7 +552,7 @@ static void netfs_writeback_flush(struct netfs_io_request *wreq, return; } - ret = netfs_prep_and_issue_subreq(wreq, stream, subreq); + ret = stream->issue_write(subreq); if (ret < 0) { /* Ownership of subreq was returned to us. */ trace_netfs_sreq(subreq, netfs_sreq_trace_fail); @@ -651,6 +680,7 @@ static void netfs_writeback_add_folio_to_stream(struct netfs_io_request *wreq, wreq->load_cursor.slot--; trace_netfs_bv_slot(wreq->load_cursor.bvecq, wreq->load_cursor.slot - 1); + trace_netfs_wback(wreq, folio, params->notes); for (int s = 0; s < NR_IO_STREAMS; s++) { struct netfs_io_stream *stream = &wreq->io_streams[s]; @@ -846,31 +876,6 @@ static void netfs_writeback_folio(struct netfs_io_request *wreq, goto out; } -#if 0 // TODO: Remove -/* - * End the issuing of writes, letting the collector know we're done. - */ -static void netfs_end_issue_write(struct netfs_io_request *wreq) -{ - bool needs_poke = true; - - netfs_all_subreqs_queued(wreq); - - for (int s = 0; s < NR_IO_STREAMS; s++) { - struct netfs_io_stream *stream = &wreq->io_streams[s]; - - if (!stream->active) - continue; - if (!list_empty(&stream->subrequests)) - needs_poke = false; - netfs_issue_write(wreq, stream); - } - - if (needs_poke) - netfs_wake_collector(wreq); -} -#endif - /* * Write some of the pending data back to the server */ @@ -957,6 +962,25 @@ int netfs_writepages(struct address_space *mapping, } EXPORT_SYMBOL(netfs_writepages); +/* + * Prepare a buffer for a single monolithic write. + */ +static int netfs_prepare_write_single_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) +{ + struct netfs_io_request *wreq = subreq->rreq; + struct netfs_io_stream *stream = &wreq->io_streams[subreq->stream_nr]; + + bvecq_pos_set(&subreq->dispatch_pos, &stream->dispatch_cursor); + bvecq_pos_set(&subreq->content, &subreq->dispatch_pos); + + stream->buffered = 0; + stream->issue_from = subreq->len; + wreq->submitted = subreq->len; + netfs_all_subreqs_queued(wreq); + return 0; +} + /** * netfs_writeback_single - Write back a monolithic payload * @mapping: The mapping to write from @@ -968,6 +992,11 @@ EXPORT_SYMBOL(netfs_writepages); * cache. There's a maximum of one subrequest per stream. The buffer should * be rounded out sufficiently that it can accommodate cache DIO rounding. * + * This is normally only used to write to the cache (for AFS directories and + * symlinks); it doesn't normally write to the server as well. The filesystem + * can override that by setting NETFS_RREQ_UPLOAD_TO_SERVER when the request is + * initialised. + * * Return: 0 if successful; 1 if skipped due to lock conflict and WB_SYNC_NONE; * or a negative error code. * the cache. There's a maximum of one subrequest per stream. @@ -981,6 +1010,11 @@ int netfs_writeback_single(struct address_space *mapping, size_t clen; int ret; + _enter("%zx,%zx", iov_iter_count(iter), len); + + if (!len) + return 0; + if (!netfs_wb_begin(ictx, wbc->sync_mode == WB_SYNC_NONE)) { /* The VFS will have undirtied the inode. */ netfs_single_mark_inode_dirty(&ictx->inode); @@ -992,9 +1026,9 @@ int netfs_writeback_single(struct address_space *mapping, ret = PTR_ERR(wreq); goto couldnt_start; } + wreq->len = len; clen = len; - if (wreq->cache_resources.dio_size > 1) { clen = round_up(len, wreq->cache_resources.dio_size); if (clen > iov_iter_count(iter)) { @@ -1003,7 +1037,7 @@ int netfs_writeback_single(struct address_space *mapping, } } - ret = netfs_extract_iter(iter, clen, INT_MAX, &wreq->dispatch_cursor.bvecq, + ret = netfs_extract_iter(iter, clen, INT_MAX, &wreq->load_cursor.bvecq, 0, wreq->gfp); if (ret < 0) goto cleanup_free; @@ -1012,12 +1046,14 @@ int netfs_writeback_single(struct address_space *mapping, goto cleanup_free; } - bvecq_pos_set(&wreq->collect_cursor, &wreq->dispatch_cursor); - __set_bit(NETFS_RREQ_OFFLOAD_COLLECTION, &wreq->flags); trace_netfs_write(wreq, netfs_write_trace_writeback_single); netfs_stat(&netfs_n_wh_writepages); + /* This normally just writes to the cache; if the filesystem wants to + * write to the server too, it must set UPLOAD_TO_SERVER in + * ->init_request(). + */ if (test_bit(NETFS_RREQ_UPLOAD_TO_SERVER, &wreq->flags)) wreq->netfs_ops->begin_writeback(wreq); @@ -1028,14 +1064,29 @@ int netfs_writeback_single(struct address_space *mapping, if (!stream->avail) continue; - netfs_prepare_write(wreq, stream, 0); - - subreq = stream->construct; - subreq->len = wreq->len; + stream->issue_from = 0; + stream->buffered = len; if (stream->source == NETFS_WRITE_TO_CACHE) - subreq->len = clen; + stream->buffered = clen; - netfs_issue_write(wreq, stream); + subreq = netfs_alloc_write_subreq(wreq, stream); + if (!subreq) { + ret = -ENOMEM; + break; + } + + bvecq_pos_set(&stream->dispatch_cursor, &wreq->load_cursor); + + ret = stream->issue_write(subreq); + if (ret < 0) { + /* Ownership of subreq was returned to us. */ + trace_netfs_sreq(subreq, netfs_sreq_trace_fail); + stream->buffered -= subreq->len; + netfs_write_subrequest_terminated(subreq, ret); + /* Punt the error to the collector. */ + } + + bvecq_pos_unset(&stream->dispatch_cursor); } wreq->submitted = wreq->len; diff --git a/fs/netfs/write_retry.c b/fs/netfs/write_retry.c index 7fa09e190203..67bf7c764cae 100644 --- a/fs/netfs/write_retry.c +++ b/fs/netfs/write_retry.c @@ -12,13 +12,44 @@ #include "internal.h" /* - * Perform retries on the streams that need it. + * Prepare the write buffer for a retry. We can't necessarily reuse the write + * buffer from the previous run of a subrequest because the filesystem is + * permitted to modify it (add headers/trailers, encrypt it). Further, the + * subrequest may now be a different size (e.g. cifs has to negotiate for + * maximum transfer size). Also, we can't look at *stream as that may still + * refer to the source material being broken up into original subrequests. + */ +int netfs_prepare_write_retry_buffer(struct netfs_io_subrequest *subreq, + unsigned int max_segs) +{ + struct netfs_io_request *wreq = subreq->rreq; + size_t len; + + bvecq_pos_set(&subreq->dispatch_pos, &wreq->retry_cursor); + bvecq_pos_set(&subreq->content, &wreq->retry_cursor); + len = bvecq_slice(&wreq->retry_cursor, subreq->len, max_segs, &subreq->nr_segs); + + if (len < subreq->len) { + subreq->len = len; + trace_netfs_sreq(subreq, netfs_sreq_trace_limited); + } + + wreq->retry_start += len; + wreq->retry_buffered -= len; + if (wreq->retry_buffered == 0) + bvecq_pos_unset(&wreq->retry_cursor); + return 0; +} + +/* + * Perform retries on the streams that need it. This only has to deal with + * buffered writes; unbuffered write retry is handled in direct_write.c. */ static void netfs_retry_write_stream(struct netfs_io_request *wreq, struct netfs_io_stream *stream) { - struct bvecq_pos dispatch_cursor = {}; struct list_head *next; + int ret; _enter("R=%x[%x:]", wreq->debug_id, stream->stream_nr); @@ -33,30 +64,15 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, if (unlikely(stream->failed)) return; - /* If there's no renegotiation to do, just resend each failed subreq. */ - if (!stream->prepare_write) { - struct netfs_io_subrequest *subreq; - - list_for_each_entry(subreq, &stream->subrequests, rreq_link) { - if (test_bit(NETFS_SREQ_FAILED, &subreq->flags)) - break; - if (__test_and_clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) { - netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit); - netfs_reissue_write(stream, subreq); - } - } - return; - } - - next = stream->subrequests.next; + /* Read pointer to subreq before reading subreq state. */ + next = smp_load_acquire(&stream->subrequests.next); do { struct netfs_io_subrequest *subreq = NULL, *from, *to, *tmp; uoff_t start, len; - size_t part; - bool boundary = false; + bool subreq_superfluous = false; - bvecq_pos_unset(&dispatch_cursor); + bvecq_pos_unset(&wreq->retry_cursor); /* Go through the stream and find the next span of contiguous * data that we then rejig (cifs, for example, needs the wsize @@ -80,7 +96,6 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, subreq = list_entry(next, struct netfs_io_subrequest, rreq_link); if (subreq->start != start + len || subreq->transferred > 0 || - test_bit(NETFS_SREQ_BOUNDARY, &subreq->flags) || !test_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags)) break; to = subreq; @@ -90,8 +105,10 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, /* Determine the set of buffers we're going to use. Each * subreq gets a subset of a single overall contiguous buffer. */ - bvecq_pos_transfer(&dispatch_cursor, &from->dispatch_pos); - bvecq_pos_advance(&dispatch_cursor, from->transferred); + bvecq_pos_transfer(&wreq->retry_cursor, &from->dispatch_pos); + bvecq_pos_advance(&wreq->retry_cursor, from->transferred); + wreq->retry_start = start; + wreq->retry_buffered = len; /* Work through the sublist. The chain of buffers we're going * to fill is attached to dispatch_cursor and we need to read @@ -99,47 +116,50 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, */ subreq = from; list_for_each_entry_from(subreq, &stream->subrequests, rreq_link) { - if (!len) + if (!wreq->retry_buffered) { + subreq_superfluous = true; break; - - subreq->start = start; - subreq->len = len; - __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags); - trace_netfs_sreq(subreq, netfs_sreq_trace_retry); - subreq->transferred = 0; + } bvecq_pos_unset(&subreq->content); bvecq_pos_unset(&subreq->dispatch_pos); - /* Renegotiate max_len (wsize) */ - stream->sreq_max_len = len; - stream->sreq_max_segs = INT_MAX; - stream->prepare_write(subreq); - - bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor); - part = bvecq_slice(&dispatch_cursor, - umin(len, stream->sreq_max_len), - stream->sreq_max_segs, - &subreq->nr_segs); - subreq->len = part; - - len -= part; - start += part; - if (len && subreq == to && - __test_and_clear_bit(NETFS_SREQ_BOUNDARY, &to->flags)) - boundary = true; - + __clear_bit(NETFS_SREQ_MADE_PROGRESS, &subreq->flags); + __clear_bit(NETFS_SREQ_NEED_RETRY, &subreq->flags); + __clear_bit(NETFS_SREQ_FAILED, &subreq->flags); + __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags); + subreq->start = wreq->retry_start; + subreq->len = wreq->retry_buffered; + subreq->transferred = 0; + subreq->retry_count += 1; + subreq->error = 0; + + netfs_stat(&netfs_n_wh_retry_write_subreq); + trace_netfs_sreq(subreq, netfs_sreq_trace_retry); netfs_get_subrequest(subreq, netfs_sreq_trace_get_resubmit); - netfs_reissue_write(stream, subreq); - if (subreq == to) + ret = stream->issue_write(subreq); + if (ret < 0) { + /* Ownership of subreq was returned to us. + * Expand the subreq to consume the entire + * remaining amount to be retried and fail it. + */ + trace_netfs_sreq(subreq, netfs_sreq_trace_fail); + wreq->retry_buffered -= subreq->len; + netfs_write_subrequest_terminated(subreq, ret); + break; + } + + if (subreq == to) { + subreq_superfluous = false; break; + } } /* If we managed to use fewer subreqs, we can discard the * excess; if we used the same number, then we're done. */ - if (!len) { - if (subreq == to) + if (!wreq->retry_buffered) { + if (!subreq_superfluous) continue; list_for_each_entry_safe_from(subreq, tmp, &stream->subrequests, rreq_link) { @@ -160,6 +180,7 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, do { subreq = netfs_alloc_subrequest(wreq, stream->source); subreq->start = start; + subreq->len = wreq->retry_buffered; subreq->stream_nr = to->stream_nr; subreq->retry_count = 1; @@ -168,49 +189,40 @@ static void netfs_retry_write_stream(struct netfs_io_request *wreq, netfs_sreq_trace_new); trace_netfs_sreq(subreq, netfs_sreq_trace_split); + __set_bit(NETFS_SREQ_IN_PROGRESS, &subreq->flags); spin_lock(&wreq->lock); + /* Write IN_PROGRESS before pointer to new subreq */ + smp_wmb(); list_add(&subreq->rreq_link, &to->rreq_link); spin_unlock(&wreq->lock); to = subreq; - trace_netfs_sreq(subreq, netfs_sreq_trace_retry); - stream->sreq_max_len = len; - stream->sreq_max_segs = INT_MAX; switch (stream->source) { case NETFS_UPLOAD_TO_SERVER: netfs_stat(&netfs_n_wh_upload); - stream->sreq_max_len = umin(len, wreq->wsize); break; default: WARN_ON_ONCE(1); } - stream->prepare_write(subreq); - - bvecq_pos_set(&subreq->dispatch_pos, &dispatch_cursor); - part = bvecq_slice(&dispatch_cursor, - umin(len, stream->sreq_max_len), - stream->sreq_max_segs, - &subreq->nr_segs); - subreq->len = subreq->transferred + part; - - len -= part; - start += part; - if (!len && boundary) { - __set_bit(NETFS_SREQ_BOUNDARY, &to->flags); - boundary = false; - } - - netfs_reissue_write(stream, subreq); - if (!len) + trace_netfs_sreq(subreq, netfs_sreq_trace_retry); + ret = stream->issue_write(subreq); + if (ret < 0) { + /* Ownership of subreq was returned to us. + * Expand the subreq to consume the entire + * remaining amount to be retried and fail it. + */ + trace_netfs_sreq(subreq, netfs_sreq_trace_fail); + wreq->retry_buffered -= subreq->len; + netfs_write_subrequest_terminated(subreq, ret); break; - - } while (len); + } + } while (wreq->retry_buffered > 0); } while (!list_is_head(next, &stream->subrequests)); out: - bvecq_pos_unset(&dispatch_cursor); + bvecq_pos_unset(&wreq->retry_cursor); } /* diff --git a/fs/nfs/fscache.c b/fs/nfs/fscache.c index 9b7fdad4a920..fb1441f88661 100644 --- a/fs/nfs/fscache.c +++ b/fs/nfs/fscache.c @@ -23,6 +23,7 @@ #include "iostat.h" #include "fscache.h" #include "nfstrace.h" +#include #define NFS_MAX_KEY_LEN 1000 @@ -273,8 +274,6 @@ static int nfs_netfs_init_request(struct netfs_io_request *rreq, struct file *fi rreq->debug_id = atomic_inc_return(&nfs_netfs_debug_id); /* [DEPRECATED] Use PG_private_2 to mark folio being written to the cache. */ __set_bit(NETFS_RREQ_USE_PGPRIV2, &rreq->flags); - rreq->io_streams[0].sreq_max_len = NFS_SB(rreq->inode->i_sb)->rsize; - return 0; } @@ -296,8 +295,9 @@ static struct nfs_netfs_io_data *nfs_netfs_alloc(struct netfs_io_subrequest *sre return netfs; } -static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq) +static int nfs_netfs_issue_read(struct netfs_io_subrequest *sreq) { + struct netfs_io_request *rreq = sreq->rreq; struct nfs_netfs_io_data *netfs; struct nfs_pageio_descriptor pgio; struct inode *inode = sreq->rreq->inode; @@ -307,6 +307,16 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq) pgoff_t start, last; int err; + if (sreq->len > NFS_SB(rreq->inode->i_sb)->rsize) + sreq->len = NFS_SB(rreq->inode->i_sb)->rsize; + + err = netfs_prepare_read_buffer(sreq, INT_MAX); + if (err < 0) { + sreq->error = err; + return err; + } + /* After this point, must fail by termination. */ + start = (sreq->start + sreq->transferred) >> PAGE_SHIFT; last = ((sreq->start + sreq->len - sreq->transferred - 1) >> PAGE_SHIFT); @@ -316,12 +326,15 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq) netfs = nfs_netfs_alloc(sreq); if (!netfs) { sreq->error = -ENOMEM; - return netfs_read_subreq_terminated(sreq); + netfs_read_subreq_terminated(sreq); + return 0; } + trace_netfs_sreq(sreq, netfs_sreq_trace_submit); + pgio.pg_netfs = netfs; /* used in completion */ - xa_for_each_range(&sreq->rreq->mapping->i_pages, idx, page, start, last) { + xa_for_each_range(&rreq->mapping->i_pages, idx, page, start, last) { /* nfs_read_add_folio() may schedule() due to pNFS layout and other RPCs */ err = nfs_read_add_folio(&pgio, ctx, page_folio(page)); if (err < 0) { @@ -332,6 +345,7 @@ static void nfs_netfs_issue_read(struct netfs_io_subrequest *sreq) out: nfs_pageio_complete_read(&pgio); nfs_netfs_put(netfs); + return 0; } void nfs_netfs_initiate_read(struct nfs_pgio_header *hdr) diff --git a/fs/smb/client/cifssmb.c b/fs/smb/client/cifssmb.c index 1f77512252e7..37d576d38837 100644 --- a/fs/smb/client/cifssmb.c +++ b/fs/smb/client/cifssmb.c @@ -1467,8 +1467,7 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid) struct cifs_tcon *tcon = tlink_tcon(rdata->req->cfile->tlink); struct inode *inode = &ictx->inode; struct smb_rqst rqst = { .rq_iov = rdata->iov, - .rq_nvec = 1, - .rq_iter = rdata->subreq.io_iter }; + .rq_nvec = 1}; struct cifs_credits credits = { .value = 1, .instance = 0, @@ -1482,6 +1481,11 @@ cifs_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid) __func__, mid->mid, mid->mid_state, rdata->result, rdata->subreq.len); + if (rdata->got_bytes) + iov_iter_bvec_queue(&rqst.rq_iter, ITER_DEST, + rdata->subreq.content.bvecq, rdata->subreq.content.slot, + rdata->subreq.content.offset, rdata->subreq.len); + switch (mid->mid_state) { case MID_RESPONSE_RECEIVED: /* result already set, check signature */ @@ -2007,7 +2011,10 @@ cifs_async_writev(struct cifs_io_subrequest *wdata) rqst.rq_iov = iov; rqst.rq_nvec = 1; - rqst.rq_iter = wdata->subreq.io_iter; + + iov_iter_bvec_queue(&rqst.rq_iter, ITER_SOURCE, + wdata->subreq.content.bvecq, wdata->subreq.content.slot, + wdata->subreq.content.offset, wdata->subreq.len); cifs_dbg(FYI, "async write at %llu %zu bytes\n", wdata->subreq.start, wdata->subreq.len); diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c index 7ad464782439..54db4e3a83fd 100644 --- a/fs/smb/client/file.c +++ b/fs/smb/client/file.c @@ -53,21 +53,23 @@ static int cifs_estimate_write(struct netfs_io_request *wreq, } /* - * Prepare a subrequest to upload to the server. We need to allocate credits - * so that we know the maximum amount of data that we can include in it. + * Issue a subrequest to upload to the server. */ -static void cifs_prepare_write(struct netfs_io_subrequest *subreq) +static int cifs_issue_write(struct netfs_io_subrequest *subreq) { struct cifs_io_subrequest *wdata = container_of(subreq, struct cifs_io_subrequest, subreq); struct cifs_io_request *req = wdata->req; - struct netfs_io_stream *stream = &req->rreq.io_streams[subreq->stream_nr]; struct TCP_Server_Info *server; struct cifsFileInfo *open_file = req->cfile; - struct cifs_sb_info *cifs_sb = CIFS_SB(wdata->rreq->inode->i_sb); - size_t wsize = req->rreq.wsize; + struct cifs_sb_info *cifs_sb = CIFS_SB(subreq->rreq->inode->i_sb); + unsigned int max_segs = INT_MAX; + size_t len; int rc; + if (cifs_forced_shutdown(cifs_sb)) + return smb_EIO(smb_eio_trace_forced_shutdown); + if (!wdata->have_xid) { wdata->xid = get_xid(); wdata->have_xid = true; @@ -86,18 +88,16 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq) if (rc < 0) { if (rc == -EAGAIN) goto retry; - subreq->error = rc; - return netfs_prepare_write_failed(subreq); + return rc; } } - rc = server->ops->wait_mtu_credits(server, wsize, &stream->sreq_max_len, - &wdata->credits); - if (rc < 0) { - subreq->error = rc; - return netfs_prepare_write_failed(subreq); - } + len = umin(subreq->len, cifs_sb->ctx->wsize); + rc = server->ops->wait_mtu_credits(server, len, &len, &wdata->credits); + if (rc < 0) + return rc; + subreq->len = len; wdata->credits.rreq_debug_id = subreq->rreq->debug_id; wdata->credits.rreq_debug_index = subreq->debug_index; wdata->credits.in_flight_check = 1; @@ -113,46 +113,34 @@ static void cifs_prepare_write(struct netfs_io_subrequest *subreq) const struct smbdirect_socket_parameters *sp = smbd_get_parameters(server->smbd_conn); - stream->sreq_max_segs = sp->max_frmr_depth; + max_segs = sp->max_frmr_depth; } #endif -} -/* - * Issue a subrequest to upload to the server. - */ -static void cifs_issue_write(struct netfs_io_subrequest *subreq) -{ - struct cifs_io_subrequest *wdata = - container_of(subreq, struct cifs_io_subrequest, subreq); - struct cifs_sb_info *sbi = CIFS_SB(subreq->rreq->inode->i_sb); - int rc; - - if (cifs_forced_shutdown(sbi)) { - rc = smb_EIO(smb_eio_trace_forced_shutdown); - goto fail; + rc = netfs_prepare_write_buffer(subreq, max_segs); + if (rc < 0) { + add_credits_and_wake_if(wdata->server, &wdata->credits, 0); + return rc; } + /* After this point, must fail by termination. */ - rc = adjust_credits(wdata->server, wdata, cifs_trace_rw_credits_issue_write_adjust); + rc = adjust_credits(server, wdata, cifs_trace_rw_credits_issue_write_adjust); if (rc) - goto fail; + goto fail_with_credits; rc = -EAGAIN; if (wdata->req->cfile->invalidHandle) - goto fail; + goto fail_with_credits; wdata->server->ops->async_writev(wdata); -out: - return; + return 0; -fail: +fail_with_credits: if (rc == -EAGAIN) trace_netfs_sreq(subreq, netfs_sreq_trace_retry); - else - trace_netfs_sreq(subreq, netfs_sreq_trace_fail); add_credits_and_wake_if(wdata->server, &wdata->credits, 0); cifs_write_subrequest_terminated(wdata, rc); - goto out; + return 0; } static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq) @@ -161,17 +149,25 @@ static void cifs_netfs_invalidate_cache(struct netfs_io_request *wreq) } /* - * Negotiate the size of a read operation on behalf of the netfs library. + * Issue a read operation on behalf of the netfs helper functions. We're asked + * to make a read of a certain size at a point in the file. We are permitted + * to only read a portion of that, but as long as we read something, the netfs + * helper will call us again so that we can issue another read. */ -static int cifs_prepare_read(struct netfs_io_subrequest *subreq) +static int cifs_issue_read(struct netfs_io_subrequest *subreq) { struct netfs_io_request *rreq = subreq->rreq; struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq); struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq); - struct TCP_Server_Info *server; + struct TCP_Server_Info *server = rdata->server; struct cifs_sb_info *cifs_sb = CIFS_SB(rreq->inode->i_sb); - size_t size; - int rc = 0; + unsigned int max_segs = INT_MAX; + size_t len; + int rc; + + cifs_dbg(FYI, "%s: op=%08x[%x] mapping=%p len=%zu/%zu\n", + __func__, rreq->debug_id, subreq->debug_index, rreq->mapping, + subreq->transferred, subreq->len); if (!rdata->have_xid) { rdata->xid = get_xid(); @@ -185,17 +181,15 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq) cifs_negotiate_rsize(server, cifs_sb->ctx, tlink_tcon(req->cfile->tlink)); - rc = server->ops->wait_mtu_credits(server, cifs_sb->ctx->rsize, - &size, &rdata->credits); + len = umin(subreq->len, cifs_sb->ctx->rsize); + rc = server->ops->wait_mtu_credits(server, len, &len, &rdata->credits); if (rc) return rc; - rreq->io_streams[0].sreq_max_len = size; - - rdata->credits.in_flight_check = 1; + subreq->len = len; rdata->credits.rreq_debug_id = rreq->debug_id; rdata->credits.rreq_debug_index = subreq->debug_index; - + rdata->credits.in_flight_check = 1; trace_smb3_rw_credits(rdata->rreq->debug_id, rdata->subreq.debug_index, rdata->credits.value, @@ -207,40 +201,27 @@ static int cifs_prepare_read(struct netfs_io_subrequest *subreq) const struct smbdirect_socket_parameters *sp = smbd_get_parameters(server->smbd_conn); - rreq->io_streams[0].sreq_max_segs = sp->max_frmr_depth; + max_segs = sp->max_frmr_depth; } #endif - return 0; -} -/* - * Issue a read operation on behalf of the netfs helper functions. We're asked - * to make a read of a certain size at a point in the file. We are permitted - * to only read a portion of that, but as long as we read something, the netfs - * helper will call us again so that we can issue another read. - */ -static void cifs_issue_read(struct netfs_io_subrequest *subreq) -{ - struct netfs_io_request *rreq = subreq->rreq; - struct cifs_io_subrequest *rdata = container_of(subreq, struct cifs_io_subrequest, subreq); - struct cifs_io_request *req = container_of(subreq->rreq, struct cifs_io_request, rreq); - struct TCP_Server_Info *server = rdata->server; - int rc = 0; - - cifs_dbg(FYI, "%s: op=%08x[%x] mapping=%p len=%zu/%zu\n", - __func__, rreq->debug_id, subreq->debug_index, rreq->mapping, - subreq->transferred, subreq->len); + rc = netfs_prepare_read_buffer(subreq, max_segs); + if (rc < 0) { + add_credits_and_wake_if(rdata->server, &rdata->credits, 0); + return rc; + } + /* After this point, must fail by termination. */ rc = adjust_credits(server, rdata, cifs_trace_rw_credits_issue_read_adjust); if (rc) - goto failed; + goto fail_with_credits; if (req->cfile->invalidHandle) { do { rc = cifs_reopen_file(req->cfile, true); } while (rc == -EAGAIN); if (rc) - goto failed; + goto fail_with_credits; } if (subreq->rreq->origin != NETFS_UNBUFFERED_READ && @@ -248,15 +229,22 @@ static void cifs_issue_read(struct netfs_io_subrequest *subreq) __set_bit(NETFS_SREQ_CLEAR_TAIL, &subreq->flags); trace_netfs_sreq(subreq, netfs_sreq_trace_submit); + rc = rdata->server->ops->async_readv(rdata); if (rc) goto failed; - return; + return 0; +fail_with_credits: + if (rc == -EAGAIN) + trace_netfs_sreq(subreq, netfs_sreq_trace_retry); + else + trace_netfs_sreq(subreq, netfs_sreq_trace_fail); failed: add_credits_and_wake_if(rdata->server, &rdata->credits, 0); subreq->error = rc; netfs_read_subreq_terminated(subreq); + return 0; } /* @@ -371,12 +359,10 @@ const struct netfs_request_ops cifs_req_ops = { .init_request = cifs_init_request, .free_request = cifs_free_request, .free_subrequest = cifs_free_subrequest, - .prepare_read = cifs_prepare_read, .issue_read = cifs_issue_read, .done = cifs_rreq_done, .begin_writeback = cifs_begin_writeback, .estimate_write = cifs_estimate_write, - .prepare_write = cifs_prepare_write, .issue_write = cifs_issue_write, .invalidate_cache = cifs_netfs_invalidate_cache, }; diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c index 97823bcf198c..1f3266f89cce 100644 --- a/fs/smb/client/smb2ops.c +++ b/fs/smb/client/smb2ops.c @@ -4884,6 +4884,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid, unsigned int cur_page_idx; unsigned int pad_len; struct cifs_io_subrequest *rdata = mid->callback_data; + struct iov_iter iter; struct smb2_hdr *shdr = (struct smb2_hdr *)buf; size_t copied; bool use_rdma_mr = false; @@ -4956,6 +4957,10 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid, pad_len = data_offset - server->vals->read_rsp_size; + iov_iter_bvec_queue(&iter, ITER_DEST, + rdata->subreq.content.bvecq, rdata->subreq.content.slot, + rdata->subreq.content.offset, rdata->subreq.len); + if (buf_len <= data_offset) { /* read response payload is in pages */ cur_page_idx = pad_len / PAGE_SIZE; @@ -4985,7 +4990,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid, /* Copy the data to the output I/O iterator. */ rdata->result = cifs_copy_bvecq_to_iter(buffer, data_len, - cur_off, &rdata->subreq.io_iter); + cur_off, &iter); if (rdata->result != 0) { if (is_offloaded) mid->mid_state = MID_RESPONSE_MALFORMED; @@ -4999,7 +5004,7 @@ handle_read_data(struct TCP_Server_Info *server, struct mid_q_entry *mid, buf_len >= end_off) { /* read response payload is in buf */ WARN_ONCE(buffer, "read data can be either in buf or in buffer"); - copied = copy_to_iter(buf + data_offset, data_len, &rdata->subreq.io_iter); + copied = copy_to_iter(buf + data_offset, data_len, &iter); if (copied == 0) return smb_EIO2(smb_eio_trace_rx_copy_to_iter, copied, data_len); rdata->got_bytes = copied; diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c index 4ce165e40657..085fdc7d8bec 100644 --- a/fs/smb/client/smb2pdu.c +++ b/fs/smb/client/smb2pdu.c @@ -4592,9 +4592,13 @@ smb2_new_read_req(void **buf, unsigned int *total_len, */ if (rdata && smb3_use_rdma_offload(io_parms)) { struct smbdirect_buffer_descriptor_v1 *v1; + struct iov_iter iter; bool need_invalidate = server->dialect == SMB30_PROT_ID; - rdata->mr = smbd_register_mr(server->smbd_conn, &rdata->subreq.io_iter, + iov_iter_bvec_queue(&iter, ITER_DEST, + rdata->subreq.content.bvecq, rdata->subreq.content.slot, + rdata->subreq.content.offset, rdata->subreq.len); + rdata->mr = smbd_register_mr(server->smbd_conn, &iter, true, need_invalidate); if (!rdata->mr) return -EAGAIN; @@ -4658,9 +4662,10 @@ smb2_readv_callback(struct TCP_Server_Info *server, struct mid_q_entry *mid) unsigned int rreq_debug_id = rdata->rreq->debug_id; unsigned int subreq_debug_index = rdata->subreq.debug_index; - if (rdata->got_bytes) { - rqst.rq_iter = rdata->subreq.io_iter; - } + if (rdata->got_bytes) + iov_iter_bvec_queue(&rqst.rq_iter, ITER_DEST, + rdata->subreq.content.bvecq, rdata->subreq.content.slot, + rdata->subreq.content.offset, rdata->subreq.len); WARN_ONCE(rdata->server != server, "rdata server %p != mid server %p", @@ -5148,7 +5153,9 @@ smb2_async_writev(struct cifs_io_subrequest *wdata) goto out; rqst.rq_iov = iov; - rqst.rq_iter = wdata->subreq.io_iter; + iov_iter_bvec_queue(&rqst.rq_iter, ITER_SOURCE, + wdata->subreq.content.bvecq, wdata->subreq.content.slot, + wdata->subreq.content.offset, wdata->subreq.len); rqst.rq_iov[0].iov_len = total_len - 1; rqst.rq_iov[0].iov_base = (char *)req; @@ -5187,9 +5194,14 @@ smb2_async_writev(struct cifs_io_subrequest *wdata) */ if (smb3_use_rdma_offload(io_parms)) { struct smbdirect_buffer_descriptor_v1 *v1; + struct iov_iter iter; bool need_invalidate = server->dialect == SMB30_PROT_ID; - wdata->mr = smbd_register_mr(server->smbd_conn, &wdata->subreq.io_iter, + iov_iter_bvec_queue(&iter, ITER_SOURCE, + wdata->subreq.content.bvecq, wdata->subreq.content.slot, + wdata->subreq.content.offset, wdata->subreq.len); + + wdata->mr = smbd_register_mr(server->smbd_conn, &iter, false, need_invalidate); if (!wdata->mr) { rc = -EAGAIN; @@ -5226,8 +5238,8 @@ smb2_async_writev(struct cifs_io_subrequest *wdata) smb2_set_replay(server, &rqst); } - cifs_dbg(FYI, "async write at %llu %u bytes iter=%zx\n", - io_parms->offset, io_parms->length, iov_iter_count(&wdata->subreq.io_iter)); + cifs_dbg(FYI, "async write at %llu %u bytes len=%zx\n", + io_parms->offset, io_parms->length, wdata->subreq.len); if (wdata->credits.value > 0) { shdr->CreditCharge = cpu_to_le16(DIV_ROUND_UP(wdata->subreq.len, diff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c index fdf4e50c27ce..be2f6b909c34 100644 --- a/fs/smb/client/transport.c +++ b/fs/smb/client/transport.c @@ -1267,12 +1267,19 @@ cifs_readv_receive(struct TCP_Server_Info *server, struct mid_q_entry *mid) } #ifdef CONFIG_CIFS_SMB_DIRECT - if (rdata->mr) + if (rdata->mr) { length = data_len; /* An RDMA read is already done. */ - else + } else { +#endif + struct iov_iter iter; + + iov_iter_bvec_queue(&iter, ITER_DEST, rdata->subreq.content.bvecq, + rdata->subreq.content.slot, rdata->subreq.content.offset, + data_len); + length = cifs_read_iter_from_socket(server, &iter, data_len); +#ifdef CONFIG_CIFS_SMB_DIRECT + } #endif - length = cifs_read_iter_from_socket(server, &rdata->subreq.io_iter, - data_len); if (length > 0) rdata->got_bytes += length; server->total_read += length; diff --git a/include/linux/netfs.h b/include/linux/netfs.h index 23c13eb178fb..88d53cd8c4ce 100644 --- a/include/linux/netfs.h +++ b/include/linux/netfs.h @@ -162,23 +162,17 @@ struct netfs_write_estimate { struct netfs_io_stream { /* Submission tracking (main dispatch only; not retry) */ struct bvecq_pos dispatch_cursor; /* Point from which buffers are dispatched */ - struct netfs_io_subrequest *construct; /* Op being constructed */ uoff_t issue_from; /* Current issue point */ uoff_t last_end; /* End file pos of last folio added */ size_t buffered; /* Amount in buffer */ size_t post_gap; /* Length of partial folio tail */ - size_t sreq_max_len; /* Maximum size of a subrequest */ - unsigned int sreq_max_segs; /* 0 or max number of segments in an iterator */ - unsigned int submit_off; /* Folio offset we're submitting from */ - unsigned int submit_len; /* Amount of data left to submit */ unsigned int alignment; /* Required alignment */ u8 applicable; /* What sources are applicable (NOTE_* mask) */ bool buffering; /* T if buffering on this stream */ int (*estimate_write)(struct netfs_io_request *wreq, struct netfs_io_stream *stream, struct netfs_write_estimate *estimate); - void (*prepare_write)(struct netfs_io_subrequest *subreq); - void (*issue_write)(struct netfs_io_subrequest *subreq); + int (*issue_write)(struct netfs_io_subrequest *subreq); atomic64_t issued_to; /* Point to which can be considered issued */ /* Collection tracking */ @@ -223,15 +217,14 @@ struct netfs_io_subrequest { struct list_head rreq_link; /* Link in rreq->subrequests */ struct bvecq_pos dispatch_pos; /* Bookmark in the combined queue of the start */ struct bvecq_pos content; /* The (copied) content of the subrequest */ - struct iov_iter io_iter; /* Iterator for this subrequest */ uoff_t start; /* Where to start the I/O */ size_t len; /* Size of the I/O */ size_t post_gap; /* Length of partial folio tail */ size_t transferred; /* Amount of data transferred */ + unsigned int nr_segs; /* Number of segments in content */ refcount_t ref; short error; /* 0 or error that occurred */ unsigned short debug_index; /* Index in list (for debugging output) */ - unsigned int nr_segs; /* Number of segments in content */ u8 retry_count; /* The number of retries (0 on initial pass) */ enum netfs_io_source source; /* Where to read from/write to */ unsigned char stream_nr; /* I/O stream this belongs to */ @@ -239,7 +232,6 @@ struct netfs_io_subrequest { #define NETFS_SREQ_COPY_TO_CACHE 0 /* Set if should copy the data to the cache */ #define NETFS_SREQ_CLEAR_TAIL 1 /* Set if the rest of the read should be cleared */ #define NETFS_SREQ_MADE_PROGRESS 4 /* Set if we transferred at least some data */ -#define NETFS_SREQ_BOUNDARY 6 /* Set if ends on hard boundary (eg. ceph object) */ #define NETFS_SREQ_HIT_EOF 7 /* Set if short due to EOF */ #define NETFS_SREQ_IN_PROGRESS 8 /* Unlocked when the subrequest completes */ #define NETFS_SREQ_NEED_RETRY 9 /* Set if the filesystem requests a retry */ @@ -288,12 +280,11 @@ struct netfs_io_request { struct netfs_group *group; /* Writeback group being written back */ struct bvecq *spare; /* Advance allocation of bvecq */ struct bvecq_pos load_cursor; /* Point at which new folios are loaded in */ - struct bvecq_pos dispatch_cursor; /* Point from which buffers are dispatched */ struct bvecq_pos collect_cursor; /* Clear-up point of I/O buffer */ + struct bvecq_pos retry_cursor; /* Point from which retries are dispatched */ wait_queue_head_t waitq; /* Processor waiter */ void *netfs_priv; /* Private data for the netfs */ void *netfs_priv2; /* Private data for the netfs */ - uoff_t last_end; /* End pos of last folio submitted */ uoff_t submitted; /* Amount submitted for I/O so far */ uoff_t len; /* Length of the request */ size_t transferred; /* Amount to be indicated as transferred */ @@ -305,6 +296,8 @@ struct netfs_io_request { uoff_t cache_coll_to; /* Point the cache has collected to */ uoff_t cleaned_to; /* Position we've cleaned folios to */ uoff_t abandon_to; /* Position to abandon folios to */ + uoff_t retry_start; /* Position to retry from */ + size_t retry_buffered; /* Amount of data to retry */ const struct folio *no_unlock_folio; /* Don't unlock this folio after read */ gfp_t gfp; /* GFP flags to use */ unsigned int debug_id; @@ -323,6 +316,7 @@ struct netfs_io_request { #define NETFS_RREQ_RETRYING 4 /* Set if we're in the retry path */ #define NETFS_RREQ_SHORT_TRANSFER 5 /* Set if we have a short transfer */ #define NETFS_RREQ_ABANDON_REQ 6 /* Set if the request is to be abandoned */ +#define NETFS_RREQ_SAW_ENOMEM 7 /* Set if we encounted ENOMEM */ #define NETFS_RREQ_CACHE_STOP 8 /* Set to stop caching (ENOBUFS or error) */ #define NETFS_RREQ_CACHE_ERROR 9 /* Set if we got an error from the cache */ #define NETFS_RREQ_CANCEL_CACHING 10 /* Set to cancel caching */ @@ -350,8 +344,7 @@ struct netfs_request_ops { /* Read request handling */ void (*expand_readahead)(struct netfs_io_request *rreq); - int (*prepare_read)(struct netfs_io_subrequest *subreq); - void (*issue_read)(struct netfs_io_subrequest *subreq); + int (*issue_read)(struct netfs_io_subrequest *subreq); bool (*is_still_valid)(struct netfs_io_request *rreq); int (*check_write_begin)(struct file *file, uoff_t pos, unsigned len, struct folio **foliop, void **_fsdata); @@ -366,8 +359,7 @@ struct netfs_request_ops { int (*estimate_write)(struct netfs_io_request *wreq, struct netfs_io_stream *stream, struct netfs_write_estimate *estimate); - void (*prepare_write)(struct netfs_io_subrequest *subreq); - void (*issue_write)(struct netfs_io_subrequest *subreq); + int (*issue_write)(struct netfs_io_subrequest *subreq); void (*retry_request)(struct netfs_io_request *wreq, struct netfs_io_stream *stream); void (*invalidate_cache)(struct netfs_io_request *wreq); }; @@ -407,8 +399,11 @@ struct netfs_cache_ops { struct netfs_io_stream *stream, struct netfs_write_estimate *estimate); + /* Read data from the cache for a netfs subrequest. */ + int (*issue_read)(struct netfs_io_subrequest *subreq); + /* Write data to the cache from a netfs subrequest. */ - void (*issue_write)(struct netfs_io_subrequest *subreq); + int (*issue_write)(struct netfs_io_subrequest *subreq); /* Expand readahead request */ void (*expand_readahead)(struct netfs_cache_resources *cres, @@ -416,30 +411,18 @@ struct netfs_cache_ops { uoff_t *_len, uoff_t i_size); - /* Prepare a read operation, shortening it to a cached/uncached - * boundary as appropriate. - */ - int (*prepare_read)(struct netfs_io_subrequest *subreq); - - /* Prepare a write subrequest, working out if we're allowed to do it - * and finding out the maximum amount of data to gather before - * attempting to submit. If we're not permitted to do it, the - * subrequest should be marked failed. - */ - void (*prepare_write_subreq)(struct netfs_io_subrequest *subreq); - /* Prepare a write operation, working out what part of the write we can * actually do. */ - int (*prepare_write)(struct netfs_cache_resources *cres, - uoff_t *_start, size_t *_len, size_t upper_len, - uoff_t i_size, bool no_space_allocated_yet); + int (*prepare_write_old)(struct netfs_cache_resources *cres, + uoff_t *_start, size_t *_len, size_t upper_len, + uoff_t i_size, bool no_space_allocated_yet); /* Query the occupancy of the cache in a region, returning where the * next chunk of data starts and how long it is. */ - int (*query_occupancy)(struct netfs_cache_resources *cres, - struct fscache_occupancy *occ); + void (*query_occupancy)(struct netfs_cache_resources *cres, + struct fscache_occupancy *occ); /* Collect the result of buffered writeback to the cache. This * includes copying a read to the cache. block_type is one of: @@ -476,7 +459,6 @@ int netfs_writeback_single(struct address_space *mapping, struct iov_iter *iter, size_t len); /* Address operations API */ -struct readahead_control; void netfs_readahead(struct readahead_control *); int netfs_read_folio(struct file *, struct folio *); int netfs_write_begin(struct netfs_inode *, struct file *, @@ -503,7 +485,8 @@ void netfs_put_subrequest(struct netfs_io_subrequest *subreq, ssize_t netfs_extract_iter(struct iov_iter *orig, size_t max_len, size_t max_pages, struct bvecq **_bvecq_head, iov_iter_extraction_t extraction_flags, gfp_t gfp); -void netfs_prepare_write_failed(struct netfs_io_subrequest *subreq); +int netfs_prepare_read_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs); +int netfs_prepare_write_buffer(struct netfs_io_subrequest *subreq, unsigned int max_segs); void netfs_write_subrequest_terminated(void *_op, ssize_t transferred_or_error); int netfs_start_io_read(struct inode *inode); diff --git a/include/trace/events/netfs.h b/include/trace/events/netfs.h index ea4721cc41ac..f43a212e4a79 100644 --- a/include/trace/events/netfs.h +++ b/include/trace/events/netfs.h @@ -49,6 +49,7 @@ #define netfs_rreq_traces \ EM(netfs_rreq_trace_all_queued, "ALL-Q ") \ EM(netfs_rreq_trace_assess, "ASSESS ") \ + EM(netfs_rreq_trace_cache_align_error, "CA-ALN!") \ EM(netfs_rreq_trace_cache_cancelled, "CA-CNCL") \ EM(netfs_rreq_trace_cache_failed, "CA-FAIL") \ EM(netfs_rreq_trace_cache_fail_collect, "CA-F-CO") \ @@ -85,7 +86,8 @@ EM(netfs_rreq_trace_waited_quiesce, "DONE-QUIESCE") \ EM(netfs_rreq_trace_wake_ip, "WAKE-IP") \ EM(netfs_rreq_trace_wake_queue, "WAKE-Q ") \ - E_(netfs_rreq_trace_write_done, "WR-DONE") + EM(netfs_rreq_trace_write_done, "WR-DONE") \ + E_(netfs_rreq_trace_zero_unread, "ZERO-UR") #define netfs_sreq_sources \ EM(NETFS_SOURCE_UNKNOWN, "----") \ @@ -134,6 +136,7 @@ EM(netfs_sreq_trace_superfluous, "SPRFL") \ EM(netfs_sreq_trace_terminated, "TERM ") \ EM(netfs_sreq_trace_too_much, "!TOOM") \ + EM(netfs_sreq_trace_too_many_retries, "!RETR") \ EM(netfs_sreq_trace_wait_for, "_WAIT") \ EM(netfs_sreq_trace_write, "WRITE") \ EM(netfs_sreq_trace_write_skip, "SKIP ") \ @@ -530,6 +533,31 @@ TRACE_EVENT(netfs_folio, __print_symbolic(__entry->why, netfs_folio_traces)) ); +TRACE_EVENT(netfs_wback, + TP_PROTO(struct netfs_io_request *wreq, struct folio *folio, unsigned int notes), + + TP_ARGS(wreq, folio, notes), + + TP_STRUCT__entry( + __field(pgoff_t, index) + __field(unsigned int, wreq) + __field(unsigned int, nr) + __field(unsigned int, notes) + ), + + TP_fast_assign( + __entry->wreq = wreq->debug_id; + __entry->notes = notes; + __entry->index = folio->index; + __entry->nr = folio_nr_pages(folio); + ), + + TP_printk("R=%08x ix=%05lx-%05lx n=%02x", + __entry->wreq, + __entry->index, __entry->index + __entry->nr - 1, + __entry->notes) + ); + TRACE_EVENT(netfs_write_iter, TP_PROTO(const struct kiocb *iocb, const struct iov_iter *from), diff --git a/net/9p/client.c b/net/9p/client.c index ef64546c6d52..68d82f677a7a 100644 --- a/net/9p/client.c +++ b/net/9p/client.c @@ -1564,6 +1564,7 @@ void p9_client_write_subreq(struct netfs_io_subrequest *subreq) { struct netfs_io_request *wreq = subreq->rreq; + struct iov_iter iter; struct p9_fid *fid = wreq->netfs_priv; struct p9_client *clnt = fid->clnt; struct p9_req_t *req; @@ -1574,14 +1575,17 @@ p9_client_write_subreq(struct netfs_io_subrequest *subreq) p9_debug(P9_DEBUG_9P, ">>> TWRITE fid %d offset %llu len %d\n", fid->fid, start, len); + iov_iter_bvec_queue(&iter, ITER_SOURCE, subreq->content.bvecq, + subreq->content.slot, subreq->content.offset, subreq->len); + /* Don't bother zerocopy for small IO (< 1024) */ if (clnt->trans_mod->zc_request && len > 1024) { - req = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, &subreq->io_iter, + req = p9_client_zc_rpc(clnt, P9_TWRITE, NULL, &iter, 0, wreq->len, P9_ZC_HDR_SZ, "dqd", fid->fid, start, len); } else { req = p9_client_rpc(clnt, P9_TWRITE, "dqV", fid->fid, - start, len, &subreq->io_iter); + start, len, &iter); } if (IS_ERR(req)) { netfs_write_subrequest_terminated(subreq, PTR_ERR(req));