[PATCH] block: fix bio_alloc_bioset() percpu cache fallback for non-reclaim contexts

[Joseph Qi <joseph.qi@linux.alibaba.com>]

bio_alloc_bioset() was restructured in commit b520c4eef83d ("block:
split bio_alloc_bioset more clearly into a fast and slowpath") to
separate fast and slow paths. However, the restructuring introduced a
regression for GFP_ATOMIC callers when the per-CPU bio cache is enabled.

Before the restructuring, a percpu cache miss would fall through to the
mempool allocation path, which internally performs a slab allocation.
After the restructuring, a percpu cache miss leaves bio as NULL and
falls through to:

    if (unlikely(!bio)) {
        if (!(saved_gfp & __GFP_DIRECT_RECLAIM))
            return NULL;
        ...
    }

This immediately returns NULL for GFP_ATOMIC callers without ever
attempting a slab allocation, even when there is plenty of free memory.
The comment says "non-blocking mempool allocations just go back to the
slab allocation", but for the percpu-cache path the slab was never tried
in the first place.

This causes virtio-pmem flush (async_pmem_flush) to fail with -ENOMEM
whenever the percpu bio cache happens to be empty (common right after
boot), making the device effectively unmountable:

    Buffer I/O error on dev pmem0, logical block 0, lost sync page write

Fix this by restructuring the allocation so that the percpu cache is
tried first when applicable, and the slab allocation is always attempted
as a common fallback path when the bio is still NULL.

Fixes: b520c4eef83d ("block: split bio_alloc_bioset more clearly into a fast and slowpath")
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
---
 block/bio.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/block/bio.c b/block/bio.c
index f2a5f4d0a9672..9e7939861b94a 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -553,6 +553,11 @@ struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs,
 		 */
 		opf |= REQ_ALLOC_CACHE;
 		bio = bio_alloc_percpu_cache(bs);
+		if (!bio) {
+			p = kmem_cache_alloc(bs->bio_slab, gfp);
+			if (p)
+				bio = p + bs->front_pad;
+		}
 	} else {
 		opf &= ~REQ_ALLOC_CACHE;
 		p = kmem_cache_alloc(bs->bio_slab, gfp);
-- 
2.39.3