* [PATCH 6.12.y 2/6] nvdimm: pmem: keep PREFLUSH before data writes
2026-09-11 1:13 ` [PATCH 6.12.y 1/6] nvdimm: preserve flush callback -ENOMEM Sasha Levin
@ 2026-09-11 1:13 ` Sasha Levin
2026-09-11 1:13 ` [PATCH 6.12.y 3/6] nvdimm: virtio_pmem: stop allocating child flush bio Sasha Levin
` (3 subsequent siblings)
4 siblings, 0 replies; 7+ messages in thread
From: Sasha Levin @ 2026-09-11 1:13 UTC (permalink / raw)
To: stable; +Cc: Li Chen, Michael S. Tsirkin, Sasha Levin
From: Li Chen <me@linux.beauty>
[ Upstream commit c644a2f8fef5618fcf453c591177700fd07dd024 ]
pmem_submit_bio() records a REQ_PREFLUSH error, but continues to copy the
bio data and can later overwrite the error with a successful REQ_FUA flush.
That lets data writes run after a failed preflush and can complete the bio
successfully despite the failed ordering barrier.
Run the REQ_PREFLUSH flush synchronously before touching the bio data and
complete the bio with the flush error if it fails. Keep asynchronous flush
chaining for REQ_FUA. At that point, data copy has completed and the parent
bio can wait for the chained flush bio.
Signed-off-by: Li Chen <me@linux.beauty>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Message-ID: <20260630092338.2094628-3-me@linux.beauty>
Stable-dep-of: e57140944b5a ("nvdimm: virtio_pmem: refcount requests for token lifetime")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/nvdimm/pmem.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 210fb77f51ba0..58f335dd57464 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -209,8 +209,14 @@ static void pmem_submit_bio(struct bio *bio)
struct pmem_device *pmem = bio->bi_bdev->bd_disk->private_data;
struct nd_region *nd_region = to_region(pmem);
- if (bio->bi_opf & REQ_PREFLUSH)
- ret = nvdimm_flush(nd_region, bio);
+ if (bio->bi_opf & REQ_PREFLUSH) {
+ ret = nvdimm_flush(nd_region, NULL);
+ if (ret) {
+ bio->bi_status = errno_to_blk_status(ret);
+ bio_endio(bio);
+ return;
+ }
+ }
do_acct = blk_queue_io_stat(bio->bi_bdev->bd_disk->queue);
if (do_acct)
@@ -230,7 +236,7 @@ static void pmem_submit_bio(struct bio *bio)
if (do_acct)
bio_end_io_acct(bio, start);
- if (bio->bi_opf & REQ_FUA)
+ if ((bio->bi_opf & REQ_FUA) && !bio->bi_status)
ret = nvdimm_flush(nd_region, bio);
if (ret)
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 6.12.y 3/6] nvdimm: virtio_pmem: stop allocating child flush bio
2026-09-11 1:13 ` [PATCH 6.12.y 1/6] nvdimm: preserve flush callback -ENOMEM Sasha Levin
2026-09-11 1:13 ` [PATCH 6.12.y 2/6] nvdimm: pmem: keep PREFLUSH before data writes Sasha Levin
@ 2026-09-11 1:13 ` Sasha Levin
2026-09-11 1:13 ` [PATCH 6.12.y 4/6] nvdimm: virtio_pmem: always wake -ENOSPC waiters Sasha Levin
` (2 subsequent siblings)
4 siblings, 0 replies; 7+ messages in thread
From: Sasha Levin @ 2026-09-11 1:13 UTC (permalink / raw)
To: stable; +Cc: Li Chen, Michael S. Tsirkin, Sasha Levin
From: Li Chen <me@linux.beauty>
[ Upstream commit 40f356e610df95728074b1fc2e2ccb54ca1b5659 ]
pmem_submit_bio() passes the parent bio to nvdimm_flush() for
REQ_FUA. For virtio-pmem this makes async_pmem_flush() allocate
and submit a child PREFLUSH bio chained to the parent.
That child allocation is in the block submit path. Making it
blocking with GFP_NOIO can consume the same global bio mempool that
submit_bio() uses, while making it GFP_ATOMIC can fail under
pressure. A forced failure of the child allocation produced:
virtio_pmem: forcing child bio allocation failure for test
Buffer I/O error on dev pmem0, logical block 0, lost sync page write
EXT4-fs (pmem0): I/O error while writing superblock
EXT4-fs (pmem0): mount failed
Avoid the child bio without turning REQ_FUA into a synchronous
submit-path wait. Let provider flush callbacks return
NVDIMM_FLUSH_ASYNC after taking ownership of parent bio completion.
pmem_submit_bio() returns in that case, and virtio-pmem queues an
ordered WQ_MEM_RECLAIM work item that runs the existing host flush
path and completes the parent bio.
This keeps the asynchronous completion model of the child-bio path
while removing the child bio allocation from the submit path.
Signed-off-by: Li Chen <me@linux.beauty>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Message-ID: <20260630092338.2094628-5-me@linux.beauty>
Backport notes for 6.12:
Drop the freeze callback hunk: this tree does not have the upstream
virtio-pmem freeze/restore callbacks. Keep the workqueue allocation,
probe error cleanup, and removal drain without adding those callbacks.
Use the available kmalloc_obj() helper for the existing request allocation,
matching upstream without changing its GFP_KERNEL behavior. This keeps the
following GFP_NOIO dependency applicable. Correct the existing "repsonse"
comment typo to match the context of the wait-flag dependency. The remaining
request-wakeup and wait-flag dependencies are still needed before the
refcount target.
Stable-dep-of: e57140944b5a ("nvdimm: virtio_pmem: refcount requests for token lifetime")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/nvdimm/nd_virtio.c | 58 +++++++++++++++++++++++++-----------
drivers/nvdimm/pmem.c | 5 +++-
drivers/nvdimm/region_devs.c | 2 ++
drivers/nvdimm/virtio_pmem.c | 14 ++++++++-
drivers/nvdimm/virtio_pmem.h | 4 +++
include/linux/libnvdimm.h | 9 ++++++
6 files changed, 72 insertions(+), 20 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index e8ac7425c97cd..8e16b7780be1a 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -9,6 +9,12 @@
#include "virtio_pmem.h"
#include "nd.h"
+struct virtio_pmem_flush_work {
+ struct work_struct work;
+ struct nd_region *nd_region;
+ struct bio *bio;
+};
+
/* The interrupt handler */
void virtio_pmem_host_ack(struct virtqueue *vq)
{
@@ -55,7 +61,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
return -EIO;
}
- req_data = kmalloc(sizeof(*req_data), GFP_KERNEL);
+ req_data = kmalloc_obj(*req_data);
if (!req_data)
return -ENOMEM;
@@ -98,7 +104,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
dev_info(&vdev->dev, "failed to send command to virtio pmem device\n");
err = -EIO;
} else {
- /* A host repsonse results in "host_ack" getting called */
+ /* A host response results in "host_ack" getting called */
wait_event(req_data->host_acked, req_data->done);
err = le32_to_cpu(req_data->resp.ret);
}
@@ -107,30 +113,46 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
return err;
};
+static void virtio_pmem_flush_work(struct work_struct *work)
+{
+ struct virtio_pmem_flush_work *flush;
+ int err;
+
+ flush = container_of(work, struct virtio_pmem_flush_work, work);
+ err = virtio_pmem_flush(flush->nd_region);
+ if (err > 0)
+ err = -EIO;
+ if (err)
+ flush->bio->bi_status = errno_to_blk_status(err);
+ bio_endio(flush->bio);
+ kfree(flush);
+}
+
/* The asynchronous flush callback function */
int async_pmem_flush(struct nd_region *nd_region, struct bio *bio)
{
- /*
- * Create child bio for asynchronous flush and chain with
- * parent bio. Otherwise directly call nd_region flush.
- */
- if (bio && bio->bi_iter.bi_sector != -1) {
- struct bio *child = bio_alloc(bio->bi_bdev, 0,
- REQ_OP_WRITE | REQ_PREFLUSH,
- GFP_ATOMIC);
+ struct virtio_device *vdev = nd_region->provider_data;
+ struct virtio_pmem *vpmem = vdev->priv;
+ struct virtio_pmem_flush_work *flush;
+ int err;
- if (!child)
+ if (bio && bio->bi_iter.bi_sector != -1) {
+ flush = kmalloc_obj(*flush, GFP_NOIO);
+ if (!flush)
return -ENOMEM;
- bio_clone_blkg_association(child, bio);
- child->bi_iter.bi_sector = -1;
- bio_chain(child, bio);
- submit_bio(child);
- return 0;
+
+ INIT_WORK(&flush->work, virtio_pmem_flush_work);
+ flush->nd_region = nd_region;
+ flush->bio = bio;
+ queue_work(vpmem->flush_wq, &flush->work);
+ return NVDIMM_FLUSH_ASYNC;
}
- if (virtio_pmem_flush(nd_region))
+
+ err = virtio_pmem_flush(nd_region);
+ if (err > 0)
return -EIO;
- return 0;
+ return err;
};
EXPORT_SYMBOL_GPL(async_pmem_flush);
MODULE_DESCRIPTION("Virtio Persistent Memory Driver");
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 58f335dd57464..53f6587674d34 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -236,8 +236,11 @@ static void pmem_submit_bio(struct bio *bio)
if (do_acct)
bio_end_io_acct(bio, start);
- if ((bio->bi_opf & REQ_FUA) && !bio->bi_status)
+ if ((bio->bi_opf & REQ_FUA) && !bio->bi_status) {
ret = nvdimm_flush(nd_region, bio);
+ if (ret == NVDIMM_FLUSH_ASYNC)
+ return;
+ }
if (ret)
bio->bi_status = errno_to_blk_status(ret);
diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c
index 4e24dccb2ff86..97ad92fbc8e49 100644
--- a/drivers/nvdimm/region_devs.c
+++ b/drivers/nvdimm/region_devs.c
@@ -1113,6 +1113,8 @@ int nvdimm_flush(struct nd_region *nd_region, struct bio *bio)
rc = generic_nvdimm_flush(nd_region);
else {
rc = nd_region->flush(nd_region, bio);
+ if (rc > 0)
+ return rc;
if (rc && rc != -ENOMEM)
rc = -EIO;
}
diff --git a/drivers/nvdimm/virtio_pmem.c b/drivers/nvdimm/virtio_pmem.c
index 89d3d4fc3ce89..94f324ffdc80e 100644
--- a/drivers/nvdimm/virtio_pmem.c
+++ b/drivers/nvdimm/virtio_pmem.c
@@ -67,10 +67,17 @@ static int virtio_pmem_probe(struct virtio_device *vdev)
mutex_init(&vpmem->flush_lock);
vpmem->vdev = vdev;
vdev->priv = vpmem;
+ vpmem->flush_wq = alloc_ordered_workqueue("virtio-pmem-flush",
+ WQ_MEM_RECLAIM);
+ if (!vpmem->flush_wq) {
+ err = -ENOMEM;
+ goto out_err;
+ }
+
err = init_vq(vpmem);
if (err) {
dev_err(&vdev->dev, "failed to initialize virtio pmem vq's\n");
- goto out_err;
+ goto out_wq;
}
if (virtio_has_feature(vdev, VIRTIO_PMEM_F_SHMEM_REGION)) {
@@ -131,6 +138,8 @@ static int virtio_pmem_probe(struct virtio_device *vdev)
nvdimm_bus_unregister(vpmem->nvdimm_bus);
out_vq:
vdev->config->del_vqs(vdev);
+out_wq:
+ destroy_workqueue(vpmem->flush_wq);
out_err:
return err;
}
@@ -138,10 +147,13 @@ static int virtio_pmem_probe(struct virtio_device *vdev)
static void virtio_pmem_remove(struct virtio_device *vdev)
{
struct nvdimm_bus *nvdimm_bus = dev_get_drvdata(&vdev->dev);
+ struct virtio_pmem *vpmem = vdev->priv;
nvdimm_bus_unregister(nvdimm_bus);
+ drain_workqueue(vpmem->flush_wq);
vdev->config->del_vqs(vdev);
virtio_reset_device(vdev);
+ destroy_workqueue(vpmem->flush_wq);
}
static unsigned int features[] = {
diff --git a/drivers/nvdimm/virtio_pmem.h b/drivers/nvdimm/virtio_pmem.h
index f72cf17f9518f..e6dfc10ce0762 100644
--- a/drivers/nvdimm/virtio_pmem.h
+++ b/drivers/nvdimm/virtio_pmem.h
@@ -15,6 +15,7 @@
#include <linux/libnvdimm.h>
#include <linux/mutex.h>
#include <linux/spinlock.h>
+#include <linux/workqueue.h>
struct virtio_pmem_request {
struct virtio_pmem_req req;
@@ -39,6 +40,9 @@ struct virtio_pmem {
/* Serialize flush requests to the device. */
struct mutex flush_lock;
+ /* Complete asynchronous FUA flushes outside the submit path. */
+ struct workqueue_struct *flush_wq;
+
/* nvdimm bus registers virtio pmem device */
struct nvdimm_bus *nvdimm_bus;
struct nvdimm_bus_descriptor nd_desc;
diff --git a/include/linux/libnvdimm.h b/include/linux/libnvdimm.h
index e772aae718431..04043b7d5712a 100644
--- a/include/linux/libnvdimm.h
+++ b/include/linux/libnvdimm.h
@@ -122,6 +122,15 @@ struct nd_mapping_desc {
};
struct nd_region;
+
+/*
+ * Provider flush callback return values:
+ * 0: flush completed synchronously
+ * <0: flush failed
+ * >0: flush completion was queued and @bio will be completed later
+ */
+#define NVDIMM_FLUSH_ASYNC 1
+
struct nd_region_desc {
struct resource *res;
struct nd_mapping_desc *mapping;
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 6.12.y 4/6] nvdimm: virtio_pmem: always wake -ENOSPC waiters
2026-09-11 1:13 ` [PATCH 6.12.y 1/6] nvdimm: preserve flush callback -ENOMEM Sasha Levin
2026-09-11 1:13 ` [PATCH 6.12.y 2/6] nvdimm: pmem: keep PREFLUSH before data writes Sasha Levin
2026-09-11 1:13 ` [PATCH 6.12.y 3/6] nvdimm: virtio_pmem: stop allocating child flush bio Sasha Levin
@ 2026-09-11 1:13 ` Sasha Levin
2026-09-11 1:13 ` [PATCH 6.12.y 5/6] nvdimm: virtio_pmem: use READ_ONCE()/WRITE_ONCE() for wait flags Sasha Levin
2026-09-11 1:13 ` [PATCH 6.12.y 6/6] nvdimm: virtio_pmem: refcount requests for token lifetime Sasha Levin
4 siblings, 0 replies; 7+ messages in thread
From: Sasha Levin @ 2026-09-11 1:13 UTC (permalink / raw)
To: stable; +Cc: Li Chen, Michael S. Tsirkin, Sasha Levin
From: Li Chen <me@linux.beauty>
[ Upstream commit 811808761e19fdea1c25b7c76734b8945f758f27 ]
virtio_pmem_host_ack() reclaims virtqueue descriptors with
virtqueue_get_buf(). The -ENOSPC waiter wakeup is tied to completing the
returned token. If token completion is skipped for any reason, reclaimed
descriptors may not wake a waiter and the submitter may sleep forever
waiting for a free slot. Always wake one -ENOSPC waiter for each virtqueue
completion before touching the returned token.
Signed-off-by: Li Chen <me@linux.beauty>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Message-ID: <20260630092338.2094628-7-me@linux.beauty>
Stable-dep-of: e57140944b5a ("nvdimm: virtio_pmem: refcount requests for token lifetime")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/nvdimm/nd_virtio.c | 25 ++++++++++++++++---------
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 8e16b7780be1a..75183a177d75d 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -15,26 +15,33 @@ struct virtio_pmem_flush_work {
struct bio *bio;
};
+static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
+{
+ struct virtio_pmem_request *req_buf;
+
+ if (list_empty(&vpmem->req_list))
+ return;
+
+ req_buf = list_first_entry(&vpmem->req_list,
+ struct virtio_pmem_request, list);
+ req_buf->wq_buf_avail = true;
+ wake_up(&req_buf->wq_buf);
+ list_del(&req_buf->list);
+}
+
/* The interrupt handler */
void virtio_pmem_host_ack(struct virtqueue *vq)
{
struct virtio_pmem *vpmem = vq->vdev->priv;
- struct virtio_pmem_request *req_data, *req_buf;
+ struct virtio_pmem_request *req_data;
unsigned long flags;
unsigned int len;
spin_lock_irqsave(&vpmem->pmem_lock, flags);
while ((req_data = virtqueue_get_buf(vq, &len)) != NULL) {
+ virtio_pmem_wake_one_waiter(vpmem);
req_data->done = true;
wake_up(&req_data->host_acked);
-
- if (!list_empty(&vpmem->req_list)) {
- req_buf = list_first_entry(&vpmem->req_list,
- struct virtio_pmem_request, list);
- req_buf->wq_buf_avail = true;
- wake_up(&req_buf->wq_buf);
- list_del(&req_buf->list);
- }
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
}
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 6.12.y 5/6] nvdimm: virtio_pmem: use READ_ONCE()/WRITE_ONCE() for wait flags
2026-09-11 1:13 ` [PATCH 6.12.y 1/6] nvdimm: preserve flush callback -ENOMEM Sasha Levin
` (2 preceding siblings ...)
2026-09-11 1:13 ` [PATCH 6.12.y 4/6] nvdimm: virtio_pmem: always wake -ENOSPC waiters Sasha Levin
@ 2026-09-11 1:13 ` Sasha Levin
2026-09-11 1:13 ` [PATCH 6.12.y 6/6] nvdimm: virtio_pmem: refcount requests for token lifetime Sasha Levin
4 siblings, 0 replies; 7+ messages in thread
From: Sasha Levin @ 2026-09-11 1:13 UTC (permalink / raw)
To: stable; +Cc: Li Chen, Pankaj Gupta, Michael S. Tsirkin, Sasha Levin
From: Li Chen <me@linux.beauty>
[ Upstream commit 08e72a5ba1ab9dc0adf993ff0f4d606a1e3445a8 ]
Use READ_ONCE()/WRITE_ONCE() for the wait_event() flags (done and
wq_buf_avail). They are observed by waiters without pmem_lock, so make
the accesses explicit single loads/stores and avoid compiler
reordering/caching across the wait/wake paths.
Acked-by: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Signed-off-by: Li Chen <me@linux.beauty>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Message-ID: <20260630092338.2094628-8-me@linux.beauty>
Stable-dep-of: e57140944b5a ("nvdimm: virtio_pmem: refcount requests for token lifetime")
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/nvdimm/nd_virtio.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 75183a177d75d..4f40cf85feb40 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -24,9 +24,9 @@ static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
req_buf = list_first_entry(&vpmem->req_list,
struct virtio_pmem_request, list);
- req_buf->wq_buf_avail = true;
+ list_del_init(&req_buf->list);
+ WRITE_ONCE(req_buf->wq_buf_avail, true);
wake_up(&req_buf->wq_buf);
- list_del(&req_buf->list);
}
/* The interrupt handler */
@@ -40,7 +40,7 @@ void virtio_pmem_host_ack(struct virtqueue *vq)
spin_lock_irqsave(&vpmem->pmem_lock, flags);
while ((req_data = virtqueue_get_buf(vq, &len)) != NULL) {
virtio_pmem_wake_one_waiter(vpmem);
- req_data->done = true;
+ WRITE_ONCE(req_data->done, true);
wake_up(&req_data->host_acked);
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
@@ -72,7 +72,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
if (!req_data)
return -ENOMEM;
- req_data->done = false;
+ WRITE_ONCE(req_data->done, false);
init_waitqueue_head(&req_data->host_acked);
init_waitqueue_head(&req_data->wq_buf);
INIT_LIST_HEAD(&req_data->list);
@@ -93,12 +93,12 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
GFP_ATOMIC)) == -ENOSPC) {
dev_info(&vdev->dev, "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
- req_data->wq_buf_avail = false;
+ WRITE_ONCE(req_data->wq_buf_avail, false);
list_add_tail(&req_data->list, &vpmem->req_list);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
/* A host response results in "host_ack" getting called */
- wait_event(req_data->wq_buf, req_data->wq_buf_avail);
+ wait_event(req_data->wq_buf, READ_ONCE(req_data->wq_buf_avail));
spin_lock_irqsave(&vpmem->pmem_lock, flags);
}
err1 = virtqueue_kick(vpmem->req_vq);
@@ -112,7 +112,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
err = -EIO;
} else {
/* A host response results in "host_ack" getting called */
- wait_event(req_data->host_acked, req_data->done);
+ wait_event(req_data->host_acked, READ_ONCE(req_data->done));
err = le32_to_cpu(req_data->resp.ret);
}
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 6.12.y 6/6] nvdimm: virtio_pmem: refcount requests for token lifetime
2026-09-11 1:13 ` [PATCH 6.12.y 1/6] nvdimm: preserve flush callback -ENOMEM Sasha Levin
` (3 preceding siblings ...)
2026-09-11 1:13 ` [PATCH 6.12.y 5/6] nvdimm: virtio_pmem: use READ_ONCE()/WRITE_ONCE() for wait flags Sasha Levin
@ 2026-09-11 1:13 ` Sasha Levin
4 siblings, 0 replies; 7+ messages in thread
From: Sasha Levin @ 2026-09-11 1:13 UTC (permalink / raw)
To: stable; +Cc: Li Chen, Michael S. Tsirkin, Sasha Levin
From: Li Chen <me@linux.beauty>
[ Upstream commit e57140944b5a47a7fd5a142faab29a02af040bc8 ]
KASAN reports slab-use-after-free in __wake_up_common():
BUG: KASAN: slab-use-after-free in __wake_up_common+0x114/0x160
Read of size 8 at addr ffff88810fdcb710 by task swapper/0/0
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted
6.19.0-next-20260220-00006-g1eae5f204ec3 #4 PREEMPT(full)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux
1.17.0-2-2 04/01/2014
Call Trace:
<IRQ>
dump_stack_lvl+0x6d/0xb0
print_report+0x170/0x4e2
? __pfx__raw_spin_lock_irqsave+0x10/0x10
? __virt_addr_valid+0x1dc/0x380
kasan_report+0xbc/0xf0
? __wake_up_common+0x114/0x160
? __wake_up_common+0x114/0x160
__wake_up_common+0x114/0x160
? __pfx__raw_spin_lock_irqsave+0x10/0x10
__wake_up+0x36/0x60
virtio_pmem_host_ack+0x11d/0x3b0
? sched_balance_domains+0x29f/0xb00
? __pfx_virtio_pmem_host_ack+0x10/0x10
? _raw_spin_lock_irqsave+0x98/0x100
? __pfx__raw_spin_lock_irqsave+0x10/0x10
vring_interrupt+0x1c9/0x5e0
? __pfx_vp_interrupt+0x10/0x10
vp_vring_interrupt+0x87/0x100
? __pfx_vp_interrupt+0x10/0x10
__handle_irq_event_percpu+0x17f/0x550
? __pfx__raw_spin_lock+0x10/0x10
handle_irq_event+0xab/0x1c0
handle_fasteoi_irq+0x276/0xae0
__common_interrupt+0x65/0x130
common_interrupt+0x78/0xa0
</IRQ>
virtio_pmem_host_ack() wakes a request that has already been freed by the
submitter.
This happens when the request token is still reachable via the virtqueue,
but virtio_pmem_flush() returns and frees it.
Fix the token lifetime by refcounting struct virtio_pmem_request.
virtio_pmem_flush() holds a submitter reference, and the virtqueue holds an
extra reference once the request is queued. The completion path drops the
virtqueue reference, and the submitter drops its reference before
returning.
Fixes: 6e84200c0a29 ("virtio-pmem: Add virtio pmem driver")
Cc: stable@vger.kernel.org
Signed-off-by: Li Chen <me@linux.beauty>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Message-ID: <20260630092338.2094628-9-me@linux.beauty>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/nvdimm/nd_virtio.c | 34 +++++++++++++++++++++++++++++-----
drivers/nvdimm/virtio_pmem.h | 2 ++
2 files changed, 31 insertions(+), 5 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 4f40cf85feb40..cbc0f53398f09 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -15,6 +15,14 @@ struct virtio_pmem_flush_work {
struct bio *bio;
};
+static void virtio_pmem_req_release(struct kref *kref)
+{
+ struct virtio_pmem_request *req;
+
+ req = container_of(kref, struct virtio_pmem_request, kref);
+ kfree(req);
+}
+
static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
{
struct virtio_pmem_request *req_buf;
@@ -42,6 +50,7 @@ void virtio_pmem_host_ack(struct virtqueue *vq)
virtio_pmem_wake_one_waiter(vpmem);
WRITE_ONCE(req_data->done, true);
wake_up(&req_data->host_acked);
+ kref_put(&req_data->kref, virtio_pmem_req_release);
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
}
@@ -72,6 +81,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
if (!req_data)
return -ENOMEM;
+ kref_init(&req_data->kref);
WRITE_ONCE(req_data->done, false);
init_waitqueue_head(&req_data->host_acked);
init_waitqueue_head(&req_data->wq_buf);
@@ -89,10 +99,23 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
* to req_list and wait for host_ack to wake us up when free
* slots are available.
*/
- while ((err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data,
- GFP_ATOMIC)) == -ENOSPC) {
-
- dev_info(&vdev->dev, "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
+ for (;;) {
+ err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data,
+ GFP_ATOMIC);
+ if (!err) {
+ /*
+ * Take the virtqueue reference while @pmem_lock is
+ * held so completion cannot run concurrently.
+ */
+ kref_get(&req_data->kref);
+ break;
+ }
+
+ if (err != -ENOSPC)
+ break;
+
+ dev_info_ratelimited(&vdev->dev,
+ "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
WRITE_ONCE(req_data->wq_buf_avail, false);
list_add_tail(&req_data->list, &vpmem->req_list);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
@@ -101,6 +124,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
wait_event(req_data->wq_buf, READ_ONCE(req_data->wq_buf_avail));
spin_lock_irqsave(&vpmem->pmem_lock, flags);
}
+
err1 = virtqueue_kick(vpmem->req_vq);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
/*
@@ -116,7 +140,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
err = le32_to_cpu(req_data->resp.ret);
}
- kfree(req_data);
+ kref_put(&req_data->kref, virtio_pmem_req_release);
return err;
};
diff --git a/drivers/nvdimm/virtio_pmem.h b/drivers/nvdimm/virtio_pmem.h
index e6dfc10ce0762..3af92588bd9d1 100644
--- a/drivers/nvdimm/virtio_pmem.h
+++ b/drivers/nvdimm/virtio_pmem.h
@@ -12,12 +12,14 @@
#include <linux/module.h>
#include <uapi/linux/virtio_pmem.h>
+#include <linux/kref.h>
#include <linux/libnvdimm.h>
#include <linux/mutex.h>
#include <linux/spinlock.h>
#include <linux/workqueue.h>
struct virtio_pmem_request {
+ struct kref kref;
struct virtio_pmem_req req;
struct virtio_pmem_resp resp;
--
2.53.0
^ permalink raw reply related [flat|nested] 7+ messages in thread