From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54233C83F1B for ; Mon, 14 Jul 2025 13:17:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C1C86B009E; Mon, 14 Jul 2025 09:17:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FD3A6B009F; Mon, 14 Jul 2025 09:17:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09C986B00A0; Mon, 14 Jul 2025 09:17:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D97926B009E for ; Mon, 14 Jul 2025 09:17:29 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6E61BC01A5 for ; Mon, 14 Jul 2025 13:17:29 +0000 (UTC) X-FDA: 83662921818.09.478A8F8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 9C4851C0015 for ; Mon, 14 Jul 2025 13:17:27 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TYG2GQQl; spf=pass (imf18.hostedemail.com: domain of bfoster@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=bfoster@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752499047; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3d7+aTpFG2q5k8m7QCJzVExxCJHPn2TXr1hZPgcAlxA=; b=luaWuNbDcx12Qy69ev2xRsoWJ6dA5hw5Nwn5LvHmFx0BUWalxAWvXS2m0MXFtNY8CZjuL7 4ljKW7C0cdDJ4VOgJPvAJZHKstE5sgk3CAAaVz2rUS8TMx+Iyp+fHhgrBz7Wq7uwKtrM6I YxDtMjuDrdMbCB3fIOEqeglX9BHer7E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752499047; a=rsa-sha256; cv=none; b=XpziY1EP1hrjE3wR3sWtEiMZ5vZ0REesNnvtoJIyTyX9BLSmJqOJR6wjDfeYbUcLMbY7M8 kzAwJkbopcWbjEPN6zNupyGcUKny01/VuqmYUxTUGkslqlfY7oGv5A3ZaWM9VdRMVlTAiB clKxMa1KhKAcxUaCypW7+DYmTIPchpQ= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TYG2GQQl; spf=pass (imf18.hostedemail.com: domain of bfoster@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=bfoster@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1752499047; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3d7+aTpFG2q5k8m7QCJzVExxCJHPn2TXr1hZPgcAlxA=; b=TYG2GQQlAzN+5c03PFG1Z9zG5hL9W8b5cy0ZRwqBFayJ66sl9jX2zlEa1jODjC6mrAAvVU r4n7aanrSs8Z4FaqUTiHlejiLLE7EOYSQKqpa9X3IVefWz6iSz4VlKgWG4qaXZKWhxoIO9 Fi9uYT+GKKV2JTC4+1vw1TwSTwRzwJI= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-644-jF7E60NgPpqWDXe2R5oyFg-1; Mon, 14 Jul 2025 09:17:23 -0400 X-MC-Unique: jF7E60NgPpqWDXe2R5oyFg-1 X-Mimecast-MFC-AGG-ID: jF7E60NgPpqWDXe2R5oyFg_1752499042 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 49D5319560BA; Mon, 14 Jul 2025 13:17:22 +0000 (UTC) Received: from bfoster.redhat.com (unknown [10.22.64.43]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 160D61800285; Mon, 14 Jul 2025 13:17:20 +0000 (UTC) From: Brian Foster To: linux-fsdevel@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-mm@kvack.org, hch@infradead.org, djwong@kernel.org, willy@infradead.org Subject: [PATCH v2 3/7] iomap: optional zero range dirty folio processing Date: Mon, 14 Jul 2025 09:20:55 -0400 Message-ID: <20250714132059.288129-4-bfoster@redhat.com> In-Reply-To: <20250714132059.288129-1-bfoster@redhat.com> References: <20250714132059.288129-1-bfoster@redhat.com> MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 X-Rspamd-Queue-Id: 9C4851C0015 X-Rspam-User: X-Rspamd-Server: rspam09 X-Stat-Signature: gs88wma6ca551dizrjpbux51m4zdgshg X-HE-Tag: 1752499047-36466 X-HE-Meta: U2FsdGVkX195PDSNS2iDZRJvOWcV+qYeM9n1Ym8UD5r1VHz0cW4yBZ1t1y9OyCjadgC2RZhlRO0t6cEjoiyEmIMKOMhoUe9NHoD94XXpSj4UwtRazNPmNEm9D4MIWtVhlqeWFALClL01S3Gl1ZTO8rzy4G7+z3T16uyNO9sqEzFCbDlNu/byucxNAGYM7mRD8URiTGonfK/50SB5KOOWiZ3SmrYLbjZWt/jtWxASK5bLx+nVDCnrxtb2vj3bifvngRpwPyH4YfJSLkO2ZL5mgxBJEzy6DF8YrsTtodetMO74r+2MqRZQrFFrhFH7DkhnwUxWMJMJGSiJ5Eyah0PeDPSWaHhydOW3LMkBTM1o2RyP6aFJ35CnBEgPylo3HxSATpvLzcsd9re5OfLe5EGwgkiWC+nOWHdpEVcZSTaWfceO5C4Y1B4Bz9z69twR1HjCzgycXCOfgfIpi1vx2YDHoXUSh7gfy2bBrHbAc5gRWzhH2qzRY0OwxPOPQVzVG0Xd1r2+T3nhFf8QNTPC9K0K7eTHOH1vlXiL725blzeZPstp3BRtsVJrZ5/J+NOyy1LfSDpzzIrs1V+/xgRiHc9Q1Oyn9fUm87F6okn+9ikQHq0tlRaaeHqEH1FuxzcAOQQ8Xi5FC98A7AaErkW50bGaUIUxndLhb+UXGbMzlgoCX5VtvKb45KoHsOqbH8nm/aYcSx6+F8vQY7BEXr7nYzZUI6/59aroxVPxE6QAwLcrabAh9iXdw4nu0L8Bt2kg+HwNRMZFIzvHorm6XWZ5z6gvjoa0ANPzgMckHn1lPZEPtxNjWsQY5JZVB1pORG/Wgi0U+91gKJs1flrQ1DiPD/5RkAq2TM0M+3BSJv9oNfrlxAl/+c96+lE9Kj0dOo/a5qdOSsvu2krarcoUVrKx3HJ7s1pc3vrT40mXj4n1f8pPEG/T3ZNDN6JamHFaX02pZo1hwjkelnO0HnmZVJxyGCz uySnIWFQ Oc04dkObfEMffo27aKXYtqAnEVMsd6zupRfDddPgLiuENci/VTNpy3oivqnp1vKc8SUsDpMY85WALYzA6m6CNy4FsHYkp6WC3+p/UqGlpNYOxIZ4gXj720tILkQ4Ugj7bjE4qTW5ajWuexCft2Bzpq+ZsmWyqaPhp2abUr27FKu5unh93LbYO2RJStU4Z5ZlpFnTt41wTqoQJ/XSAMGo0IxsD0/nwRCOOoq2i3Vqw5oCA/8LJJ+vBo7A0qmLEGeELrKpKIY6f4oP/rNDVIc/EiZBgLa/zZqEHg/oEUHcjBQEfNY9HedsQACDguJPxCbITMN0HP/tTcLdewdQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The only way zero range can currently process unwritten mappings with dirty pagecache is to check whether the range is dirty before mapping lookup and then flush when at least one underlying mapping is unwritten. This ordering is required to prevent iomap lookup from racing with folio writeback and reclaim. Since zero range can skip ranges of unwritten mappings that are clean in cache, this operation can be improved by allowing the filesystem to provide a set of dirty folios that require zeroing. In turn, rather than flush or iterate file offsets, zero range can iterate on folios in the batch and advance over clean or uncached ranges in between. Add a folio_batch in struct iomap and provide a helper for fs' to populate the batch at lookup time. Update the folio lookup path to return the next folio in the batch, if provided, and advance the iter if the folio starts beyond the current offset. Signed-off-by: Brian Foster --- fs/iomap/buffered-io.c | 89 +++++++++++++++++++++++++++++++++++++++--- fs/iomap/iter.c | 6 +++ include/linux/iomap.h | 4 ++ 3 files changed, 94 insertions(+), 5 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 38da2fa6e6b0..194e3cc0857f 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -750,6 +750,28 @@ static struct folio *__iomap_get_folio(struct iomap_iter *iter, size_t len) if (!mapping_large_folio_support(iter->inode->i_mapping)) len = min_t(size_t, len, PAGE_SIZE - offset_in_page(pos)); + if (iter->fbatch) { + struct folio *folio = folio_batch_next(iter->fbatch); + + if (!folio) + return NULL; + + /* + * The folio mapping generally shouldn't have changed based on + * fs locks, but be consistent with filemap lookup and retry + * the iter if it does. + */ + folio_lock(folio); + if (unlikely(folio->mapping != iter->inode->i_mapping)) { + iter->iomap.flags |= IOMAP_F_STALE; + folio_unlock(folio); + return NULL; + } + + folio_get(folio); + return folio; + } + if (folio_ops && folio_ops->get_folio) return folio_ops->get_folio(iter, pos, len); else @@ -811,6 +833,8 @@ static int iomap_write_begin(struct iomap_iter *iter, struct folio **foliop, int status = 0; len = min_not_zero(len, *plen); + *foliop = NULL; + *plen = 0; if (fatal_signal_pending(current)) return -EINTR; @@ -819,6 +843,15 @@ static int iomap_write_begin(struct iomap_iter *iter, struct folio **foliop, if (IS_ERR(folio)) return PTR_ERR(folio); + /* + * No folio means we're done with a batch. We still have range to + * process so return and let the caller iterate and refill the batch. + */ + if (!folio) { + WARN_ON_ONCE(!iter->fbatch); + return 0; + } + /* * Now we have a locked folio, before we do anything with it we need to * check that the iomap we have cached is not stale. The inode extent @@ -839,6 +872,21 @@ static int iomap_write_begin(struct iomap_iter *iter, struct folio **foliop, } } + /* + * The folios in a batch may not be contiguous. If we've skipped + * forward, advance the iter to the pos of the current folio. If the + * folio starts beyond the end of the mapping, it may have been trimmed + * since the lookup for whatever reason. Return a NULL folio to + * terminate the op. + */ + if (folio_pos(folio) > iter->pos) { + len = min_t(u64, folio_pos(folio) - iter->pos, + iomap_length(iter)); + status = iomap_iter_advance(iter, &len); + if (status || !len) + goto out_unlock; + } + pos = iomap_trim_folio_range(iter, folio, poffset, &len); if (srcmap->type == IOMAP_INLINE) @@ -1377,6 +1425,12 @@ static int iomap_zero_iter(struct iomap_iter *iter, bool *did_zero) if (iter->iomap.flags & IOMAP_F_STALE) break; + /* a NULL folio means we're done with a folio batch */ + if (!folio) { + status = iomap_iter_advance_full(iter); + break; + } + /* warn about zeroing folios beyond eof that won't write back */ WARN_ON_ONCE(folio_pos(folio) > iter->inode->i_size); @@ -1398,6 +1452,26 @@ static int iomap_zero_iter(struct iomap_iter *iter, bool *did_zero) return status; } +loff_t +iomap_fill_dirty_folios( + struct iomap_iter *iter, + loff_t offset, + loff_t length) +{ + struct address_space *mapping = iter->inode->i_mapping; + pgoff_t start = offset >> PAGE_SHIFT; + pgoff_t end = (offset + length - 1) >> PAGE_SHIFT; + + iter->fbatch = kmalloc(sizeof(struct folio_batch), GFP_KERNEL); + if (!iter->fbatch) + return offset + length; + folio_batch_init(iter->fbatch); + + filemap_get_folios_dirty(mapping, &start, end, iter->fbatch); + return (start << PAGE_SHIFT); +} +EXPORT_SYMBOL_GPL(iomap_fill_dirty_folios); + int iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero, const struct iomap_ops *ops, void *private) @@ -1426,7 +1500,7 @@ iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero, * flushing on partial eof zeroing, special case it to zero the * unaligned start portion if already dirty in pagecache. */ - if (off && + if (!iter.fbatch && off && filemap_range_needs_writeback(mapping, pos, pos + plen - 1)) { iter.len = plen; while ((ret = iomap_iter(&iter, ops)) > 0) @@ -1442,13 +1516,18 @@ iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero, * if dirty and the fs returns a mapping that might convert on * writeback. */ - range_dirty = filemap_range_needs_writeback(inode->i_mapping, - iter.pos, iter.pos + iter.len - 1); + range_dirty = filemap_range_needs_writeback(mapping, iter.pos, + iter.pos + iter.len - 1); while ((ret = iomap_iter(&iter, ops)) > 0) { const struct iomap *srcmap = iomap_iter_srcmap(&iter); - if (srcmap->type == IOMAP_HOLE || - srcmap->type == IOMAP_UNWRITTEN) { + if (WARN_ON_ONCE(iter.fbatch && + srcmap->type != IOMAP_UNWRITTEN)) + return -EIO; + + if (!iter.fbatch && + (srcmap->type == IOMAP_HOLE || + srcmap->type == IOMAP_UNWRITTEN)) { s64 status; if (range_dirty) { diff --git a/fs/iomap/iter.c b/fs/iomap/iter.c index 6ffc6a7b9ba5..89bd5951a6fd 100644 --- a/fs/iomap/iter.c +++ b/fs/iomap/iter.c @@ -9,6 +9,12 @@ static inline void iomap_iter_reset_iomap(struct iomap_iter *iter) { + if (iter->fbatch) { + folio_batch_release(iter->fbatch); + kfree(iter->fbatch); + iter->fbatch = NULL; + } + iter->status = 0; memset(&iter->iomap, 0, sizeof(iter->iomap)); memset(&iter->srcmap, 0, sizeof(iter->srcmap)); diff --git a/include/linux/iomap.h b/include/linux/iomap.h index 522644d62f30..0b9b460b2873 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -9,6 +9,7 @@ #include #include #include +#include struct address_space; struct fiemap_extent_info; @@ -239,6 +240,7 @@ struct iomap_iter { unsigned flags; struct iomap iomap; struct iomap srcmap; + struct folio_batch *fbatch; void *private; }; @@ -345,6 +347,8 @@ void iomap_invalidate_folio(struct folio *folio, size_t offset, size_t len); bool iomap_dirty_folio(struct address_space *mapping, struct folio *folio); int iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len, const struct iomap_ops *ops); +loff_t iomap_fill_dirty_folios(struct iomap_iter *iter, loff_t offset, + loff_t length); int iomap_zero_range(struct inode *inode, loff_t pos, loff_t len, bool *did_zero, const struct iomap_ops *ops, void *private); int iomap_truncate_page(struct inode *inode, loff_t pos, bool *did_zero, -- 2.50.0