[PATCH 0/3] Add and use bdev_getblk()

[PATCH 0/3] Add and use bdev_getblk()

[Matthew Wilcox (Oracle)]

This patch series fixes a bug reported by Hui Zhu; see proposed
patches v1 and v2:
https://lore.kernel.org/linux-fsdevel/20230811035705.3296-1-teawaterz@linux.alibaba.com/
https://lore.kernel.org/linux-fsdevel/20230811071519.1094-1-teawaterz@linux.alibaba.com/

I decided to go in a rather different direction for this fix, and
fix a related problem at the same time.  I don't think there's any
urgency to rush this into Linus' tree, nor have I marked it for stable.
Reasonable people may disagree.

Matthew Wilcox (Oracle) (3):
  buffer: Pass GFP flags to folio_alloc_buffers()
  buffer: Hoist GFP flags from grow_dev_page() to __getblk_gfp()
  ext4: Use bdev_getblk() to avoid memory reclaim in readahead path

 fs/buffer.c                 | 75 +++++++++++++++++++++++--------------
 fs/ext4/super.c             |  3 +-
 include/linux/buffer_head.h |  4 +-
 3 files changed, 52 insertions(+), 30 deletions(-)

-- 
2.40.1

[PATCH 1/3] buffer: Pass GFP flags to folio_alloc_buffers()

[Matthew Wilcox (Oracle)]

Instead of creating entirely new flags, inherit them from grow_dev_page().
The other callers create the same flags that this function used to create.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 fs/buffer.c                 | 17 +++++++++--------
 include/linux/buffer_head.h |  2 +-
 2 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/fs/buffer.c b/fs/buffer.c
index bd091329026c..7326acc29541 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -920,16 +920,12 @@ int remove_inode_buffers(struct inode *inode)
  * which may not fail from ordinary buffer allocations.
  */
 struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,
-					bool retry)
+					gfp_t gfp)
 {
 	struct buffer_head *bh, *head;
-	gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT;
 	long offset;
 	struct mem_cgroup *memcg, *old_memcg;
 
-	if (retry)
-		gfp |= __GFP_NOFAIL;
-
 	/* The folio lock pins the memcg */
 	memcg = folio_memcg(folio);
 	old_memcg = set_active_memcg(memcg);
@@ -972,7 +968,11 @@ EXPORT_SYMBOL_GPL(folio_alloc_buffers);
 struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size,
 				       bool retry)
 {
-	return folio_alloc_buffers(page_folio(page), size, retry);
+	gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT;
+	if (retry)
+		gfp |= __GFP_NOFAIL;
+
+	return folio_alloc_buffers(page_folio(page), size, gfp);
 }
 EXPORT_SYMBOL_GPL(alloc_page_buffers);
 
@@ -1074,7 +1074,7 @@ grow_dev_page(struct block_device *bdev, sector_t block,
 			goto failed;
 	}
 
-	bh = folio_alloc_buffers(folio, size, true);
+	bh = folio_alloc_buffers(folio, size, gfp_mask);
 
 	/*
 	 * Link the folio to the buffers and initialise them.  Take the
@@ -1665,8 +1665,9 @@ void folio_create_empty_buffers(struct folio *folio, unsigned long blocksize,
 				unsigned long b_state)
 {
 	struct buffer_head *bh, *head, *tail;
+	gfp_t gfp = GFP_NOFS | __GFP_ACCOUNT | __GFP_NOFAIL;
 
-	head = folio_alloc_buffers(folio, blocksize, true);
+	head = folio_alloc_buffers(folio, blocksize, gfp);
 	bh = head;
 	do {
 		bh->b_state |= b_state;
diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
index 6cb3e9af78c9..d17efb8b7976 100644
--- a/include/linux/buffer_head.h
+++ b/include/linux/buffer_head.h
@@ -200,7 +200,7 @@ void folio_set_bh(struct buffer_head *bh, struct folio *folio,
 		  unsigned long offset);
 bool try_to_free_buffers(struct folio *);
 struct buffer_head *folio_alloc_buffers(struct folio *folio, unsigned long size,
-					bool retry);
+					gfp_t gfp);
 struct buffer_head *alloc_page_buffers(struct page *page, unsigned long size,
 		bool retry);
 void create_empty_buffers(struct page *, unsigned long,
-- 
2.40.1

[PATCH 2/3] buffer: Hoist GFP flags from grow_dev_page() to __getblk_gfp()

[Matthew Wilcox (Oracle)]

grow_dev_page() is only called by grow_buffers().  grow_buffers()
is only called by __getblk_slow() and __getblk_slow() is only called
from __getblk_gfp(), so it is safe to move the GFP flags setting
all the way up.  With that done, add a new bdev_getblk() entry point
that leaves the GFP flags the way the caller specified them.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 fs/buffer.c                 | 60 ++++++++++++++++++++++++-------------
 include/linux/buffer_head.h |  2 ++
 2 files changed, 41 insertions(+), 21 deletions(-)

diff --git a/fs/buffer.c b/fs/buffer.c
index 7326acc29541..122b7d16befb 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -1048,20 +1048,11 @@ grow_dev_page(struct block_device *bdev, sector_t block,
 	struct buffer_head *bh;
 	sector_t end_block;
 	int ret = 0;
-	gfp_t gfp_mask;
-
-	gfp_mask = mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS) | gfp;
-
-	/*
-	 * XXX: __getblk_slow() can not really deal with failure and
-	 * will endlessly loop on improvised global reclaim.  Prefer
-	 * looping in the allocator rather than here, at least that
-	 * code knows what it's doing.
-	 */
-	gfp_mask |= __GFP_NOFAIL;
 
 	folio = __filemap_get_folio(inode->i_mapping, index,
-			FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp_mask);
+			FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp);
+	if (IS_ERR(folio))
+		return PTR_ERR(folio);
 
 	bh = folio_buffers(folio);
 	if (bh) {
@@ -1074,7 +1065,9 @@ grow_dev_page(struct block_device *bdev, sector_t block,
 			goto failed;
 	}
 
-	bh = folio_alloc_buffers(folio, size, gfp_mask);
+	bh = folio_alloc_buffers(folio, size, gfp);
+	if (!bh)
+		goto failed;
 
 	/*
 	 * Link the folio to the buffers and initialise them.  Take the
@@ -1426,24 +1419,49 @@ __find_get_block(struct block_device *bdev, sector_t block, unsigned size)
 }
 EXPORT_SYMBOL(__find_get_block);
 
+/**
+ * bdev_getblk - Get a buffer_head in a block device's buffer cache.
+ * @bdev: The block device.
+ * @block: The block number.
+ * @size: The size of buffer_heads for this @bdev.
+ * @gfp: The memory allocation flags to use.
+ *
+ * In contrast to __getblk_gfp(), the @gfp flags must be all of the flags;
+ * they are not augmented with the mapping's GFP flags.
+ *
+ * Return: The buffer head, or NULL if memory could not be allocated.
+ */
+struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block,
+		unsigned size, gfp_t gfp)
+{
+	struct buffer_head *bh = __find_get_block(bdev, block, size);
+
+	might_alloc(gfp);
+	if (bh)
+		return bh;
+
+	return __getblk_slow(bdev, block, size, gfp);
+}
+EXPORT_SYMBOL(bdev_getblk);
+
 /*
  * __getblk_gfp() will locate (and, if necessary, create) the buffer_head
  * which corresponds to the passed block_device, block and size. The
  * returned buffer has its reference count incremented.
- *
- * __getblk_gfp() will lock up the machine if grow_dev_page's
- * try_to_free_buffers() attempt is failing.  FIXME, perhaps?
  */
 struct buffer_head *
 __getblk_gfp(struct block_device *bdev, sector_t block,
 	     unsigned size, gfp_t gfp)
 {
-	struct buffer_head *bh = __find_get_block(bdev, block, size);
+	gfp |= mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS);
 
-	might_sleep();
-	if (bh == NULL)
-		bh = __getblk_slow(bdev, block, size, gfp);
-	return bh;
+	/*
+	 * Prefer looping in the allocator rather than here, at least that
+	 * code knows what it's doing.
+	 */
+	gfp |= __GFP_NOFAIL;
+
+	return bdev_getblk(bdev, block, size, gfp);
 }
 EXPORT_SYMBOL(__getblk_gfp);
 
diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
index d17efb8b7976..01110db9213c 100644
--- a/include/linux/buffer_head.h
+++ b/include/linux/buffer_head.h
@@ -233,6 +233,8 @@ void __wait_on_buffer(struct buffer_head *);
 wait_queue_head_t *bh_waitq_head(struct buffer_head *bh);
 struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block,
 			unsigned size);
+struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block,
+		unsigned size, gfp_t gfp);
 struct buffer_head *__getblk_gfp(struct block_device *bdev, sector_t block,
 				  unsigned size, gfp_t gfp);
 void __brelse(struct buffer_head *);
-- 
2.40.1

[PATCH 3/3] ext4: Use bdev_getblk() to avoid memory reclaim in readahead path

[Matthew Wilcox (Oracle)]

sb_getblk_gfp adds __GFP_NOFAIL, which is unnecessary for readahead;
we're quite comfortable with the possibility that we may not get a bh
back.  Switch to bdev_getblk() which does not include __GFP_NOFAIL.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reported-by: Hui Zhu <teawater@antgroup.com>
Link: https://lore.kernel.org/linux-fsdevel/20230811035705.3296-1-teawaterz@linux.alibaba.com/
---
 fs/ext4/super.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index c94ebf704616..48524314be97 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -254,7 +254,8 @@ struct buffer_head *ext4_sb_bread_unmovable(struct super_block *sb,
 
 void ext4_sb_breadahead_unmovable(struct super_block *sb, sector_t block)
 {
-	struct buffer_head *bh = sb_getblk_gfp(sb, block, 0);
+	struct buffer_head *bh = bdev_getblk(sb->s_bdev, block,
+			sb->s_blocksize, GFP_NOWAIT);
 
 	if (likely(bh)) {
 		if (trylock_buffer(bh))
-- 
2.40.1

Re: [PATCH 3/3] ext4: Use bdev_getblk() to avoid memory reclaim in readahead path

[Matthew Wilcox]

On Fri, Aug 11, 2023 at 05:15:28PM +0100, Matthew Wilcox (Oracle) wrote:
> +	struct buffer_head *bh = bdev_getblk(sb->s_bdev, block,
> +			sb->s_blocksize, GFP_NOWAIT);

Doh, I missed the __GFP_ACCOUNT from that.  Will wait a day or two
for other feedback.

[PATCH 4/3] buffer: Use bdev_getblk() in __breadahead()

[Matthew Wilcox]

It occurs to me that this would also be useful and I'll include it in
the next version:

diff --git a/fs/buffer.c b/fs/buffer.c
index 122b7d16befb..b551a5b1196b 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -1470,7 +1470,9 @@ EXPORT_SYMBOL(__getblk_gfp);
  */
 void __breadahead(struct block_device *bdev, sector_t block, unsigned size)
 {
-       struct buffer_head *bh = __getblk(bdev, block, size);
+       struct buffer_head *bh = bdev_getblk(bdev, block, size,
+                       GFP_NOWAIT | __GFP_MOVABLE | __GFP_ACCOUNT);
+
        if (likely(bh)) {
                bh_readahead(bh, REQ_RAHEAD);
                brelse(bh);

Re: [PATCH 2/3] buffer: Hoist GFP flags from grow_dev_page() to __getblk_gfp()

[Jan Kara]

On Fri 11-08-23 17:15:27, Matthew Wilcox (Oracle) wrote:
> grow_dev_page() is only called by grow_buffers().  grow_buffers()
> is only called by __getblk_slow() and __getblk_slow() is only called
> from __getblk_gfp(), so it is safe to move the GFP flags setting
> all the way up.  With that done, add a new bdev_getblk() entry point
> that leaves the GFP flags the way the caller specified them.
> 
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>

Can't we just finish this gfp parameter conversion for all the users?
There are five __getblk_gfp() users, three in buffer_head.h directly
generate gfp mask, two (__bread_gfp() and sb_getblk_gfp()) pass it from the
caller. All three __bread_gfp() callers are in buffer_head.h and directly
generate gfp mask. sb_getblk_gfp() has five callers, all in ext4 and easily
convertable as well.

This results not only in cleaner code but also just checking
sb_getblk_gfp() callers shows how confused they currently are about the gfp
argument (passing NOFS, NOFAIL and other pointless flags). Secondly, we can
keep using sb_getblk_gfp() from the filesystems instead of having to decide
between sb_getblk_gfp() and bdev_getblk().

If you don't have time for this, I guess I can find some...

								Honza

> ---
>  fs/buffer.c                 | 60 ++++++++++++++++++++++++-------------
>  include/linux/buffer_head.h |  2 ++
>  2 files changed, 41 insertions(+), 21 deletions(-)
> 
> diff --git a/fs/buffer.c b/fs/buffer.c
> index 7326acc29541..122b7d16befb 100644
> --- a/fs/buffer.c
> +++ b/fs/buffer.c
> @@ -1048,20 +1048,11 @@ grow_dev_page(struct block_device *bdev, sector_t block,
>  	struct buffer_head *bh;
>  	sector_t end_block;
>  	int ret = 0;
> -	gfp_t gfp_mask;
> -
> -	gfp_mask = mapping_gfp_constraint(inode->i_mapping, ~__GFP_FS) | gfp;
> -
> -	/*
> -	 * XXX: __getblk_slow() can not really deal with failure and
> -	 * will endlessly loop on improvised global reclaim.  Prefer
> -	 * looping in the allocator rather than here, at least that
> -	 * code knows what it's doing.
> -	 */
> -	gfp_mask |= __GFP_NOFAIL;
>  
>  	folio = __filemap_get_folio(inode->i_mapping, index,
> -			FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp_mask);
> +			FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp);
> +	if (IS_ERR(folio))
> +		return PTR_ERR(folio);
>  
>  	bh = folio_buffers(folio);
>  	if (bh) {
> @@ -1074,7 +1065,9 @@ grow_dev_page(struct block_device *bdev, sector_t block,
>  			goto failed;
>  	}
>  
> -	bh = folio_alloc_buffers(folio, size, gfp_mask);
> +	bh = folio_alloc_buffers(folio, size, gfp);
> +	if (!bh)
> +		goto failed;
>  
>  	/*
>  	 * Link the folio to the buffers and initialise them.  Take the
> @@ -1426,24 +1419,49 @@ __find_get_block(struct block_device *bdev, sector_t block, unsigned size)
>  }
>  EXPORT_SYMBOL(__find_get_block);
>  
> +/**
> + * bdev_getblk - Get a buffer_head in a block device's buffer cache.
> + * @bdev: The block device.
> + * @block: The block number.
> + * @size: The size of buffer_heads for this @bdev.
> + * @gfp: The memory allocation flags to use.
> + *
> + * In contrast to __getblk_gfp(), the @gfp flags must be all of the flags;
> + * they are not augmented with the mapping's GFP flags.
> + *
> + * Return: The buffer head, or NULL if memory could not be allocated.
> + */
> +struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block,
> +		unsigned size, gfp_t gfp)
> +{
> +	struct buffer_head *bh = __find_get_block(bdev, block, size);
> +
> +	might_alloc(gfp);
> +	if (bh)
> +		return bh;
> +
> +	return __getblk_slow(bdev, block, size, gfp);
> +}
> +EXPORT_SYMBOL(bdev_getblk);
> +
>  /*
>   * __getblk_gfp() will locate (and, if necessary, create) the buffer_head
>   * which corresponds to the passed block_device, block and size. The
>   * returned buffer has its reference count incremented.
> - *
> - * __getblk_gfp() will lock up the machine if grow_dev_page's
> - * try_to_free_buffers() attempt is failing.  FIXME, perhaps?
>   */
>  struct buffer_head *
>  __getblk_gfp(struct block_device *bdev, sector_t block,
>  	     unsigned size, gfp_t gfp)
>  {
> -	struct buffer_head *bh = __find_get_block(bdev, block, size);
> +	gfp |= mapping_gfp_constraint(bdev->bd_inode->i_mapping, ~__GFP_FS);
>  
> -	might_sleep();
> -	if (bh == NULL)
> -		bh = __getblk_slow(bdev, block, size, gfp);
> -	return bh;
> +	/*
> +	 * Prefer looping in the allocator rather than here, at least that
> +	 * code knows what it's doing.
> +	 */
> +	gfp |= __GFP_NOFAIL;
> +
> +	return bdev_getblk(bdev, block, size, gfp);
>  }
>  EXPORT_SYMBOL(__getblk_gfp);
>  
> diff --git a/include/linux/buffer_head.h b/include/linux/buffer_head.h
> index d17efb8b7976..01110db9213c 100644
> --- a/include/linux/buffer_head.h
> +++ b/include/linux/buffer_head.h
> @@ -233,6 +233,8 @@ void __wait_on_buffer(struct buffer_head *);
>  wait_queue_head_t *bh_waitq_head(struct buffer_head *bh);
>  struct buffer_head *__find_get_block(struct block_device *bdev, sector_t block,
>  			unsigned size);
> +struct buffer_head *bdev_getblk(struct block_device *bdev, sector_t block,
> +		unsigned size, gfp_t gfp);
>  struct buffer_head *__getblk_gfp(struct block_device *bdev, sector_t block,
>  				  unsigned size, gfp_t gfp);
>  void __brelse(struct buffer_head *);
> -- 
> 2.40.1
> 
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [PATCH 2/3] buffer: Hoist GFP flags from grow_dev_page() to __getblk_gfp()

[Matthew Wilcox]

On Thu, Sep 14, 2023 at 11:16:25AM +0200, Jan Kara wrote:
> On Fri 11-08-23 17:15:27, Matthew Wilcox (Oracle) wrote:
> > grow_dev_page() is only called by grow_buffers().  grow_buffers()
> > is only called by __getblk_slow() and __getblk_slow() is only called
> > from __getblk_gfp(), so it is safe to move the GFP flags setting
> > all the way up.  With that done, add a new bdev_getblk() entry point
> > that leaves the GFP flags the way the caller specified them.
> > 
> > Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> 
> Can't we just finish this gfp parameter conversion for all the users?
> There are five __getblk_gfp() users, three in buffer_head.h directly
> generate gfp mask, two (__bread_gfp() and sb_getblk_gfp()) pass it from the
> caller. All three __bread_gfp() callers are in buffer_head.h and directly
> generate gfp mask. sb_getblk_gfp() has five callers, all in ext4 and easily
> convertable as well.
> 
> This results not only in cleaner code but also just checking
> sb_getblk_gfp() callers shows how confused they currently are about the gfp
> argument (passing NOFS, NOFAIL and other pointless flags). Secondly, we can
> keep using sb_getblk_gfp() from the filesystems instead of having to decide
> between sb_getblk_gfp() and bdev_getblk().

I didn't do __bread_gfp() because it's basically an internal interface.
All users call sb_bread(), sb_bread_unmovable() or __bread().
It doesn't seem worth doing.  Now, if we start to see people actually
using __bread_gfp() outside of those three interfaces, I'd agree we need
to make it use GFP flags properly.

BTW, Andrew has taken the bdev_getblk() series into the mm tree, so
testing that tree might be a good idea for the ext4 developers (and other
filesystems; an earlier revision of this patchset  had a bug which would
have only affected nilfs2).