* Re: [PATCH] virtio-blk: use little-endian types for the zoned fields
From: Stefan Hajnoczi @ 2026-06-18 15:18 UTC (permalink / raw)
To: Michael Bommarito
Cc: Michael S . Tsirkin, Jason Wang, Stefano Garzarella,
Dmitry Fomichev, Damien Le Moal, Jens Axboe, Paolo Bonzini,
virtualization, linux-block, linux-kernel
In-Reply-To: <20260617151727.4071754-1-michael.bommarito@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2446 bytes --]
On Wed, Jun 17, 2026 at 11:17:27AM -0400, Michael Bommarito wrote:
> The zoned block-device fields in the virtio-blk header are typed
> __virtio{32,64}, so their endianness follows VIRTIO_F_VERSION_1. The
> zoned feature is only defined for VIRTIO 1.x devices, and the virtio
> specification defines all of its fields as little-endian. Commit
> b16a1756c716 ("virtio_blk: mark all zone fields LE") tagged them
> __le* for exactly this reason, but commit f1ba4e674feb ("virtio-blk:
> fix to match virtio spec") re-applied the reviewed version of the
> original zoned series -- which predated b16a1756 -- and silently
> restored the __virtio* typing together with the matching
> virtio*_to_cpu() / virtio_cread() accessors in the driver.
>
> Restore the little-endian typing for the zoned configuration-space
> characteristics, the zone descriptor, the zone report header and the
> ZONE_APPEND in-header sector, and read them with le*_to_cpu() and
> virtio_cread_le() to match.
>
> There is no functional change on any spec-compliant device: zoned
> requires VIRTIO_F_VERSION_1, and for a VERSION_1 device
> virtio*_to_cpu() is identical to le*_to_cpu(). The change makes the
> uapi types describe the actual wire format and removes a latent
> endianness mismatch for a (non-conformant) legacy device on a
> big-endian guest.
>
> Fixes: f1ba4e674feb ("virtio-blk: fix to match virtio spec")
> Suggested-by: Michael S. Tsirkin <mst@redhat.com>
> Assisted-by: Claude:claude-opus-4-8
> Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
> ---
> Testing:
> - Builds with no new warnings; sparse endian-clean (C=2,
> __CHECK_ENDIAN__, CONFIG_BLK_DEV_ZONED=y) both before and after.
> - Booted under QEMU with a host-managed zoned device exposed through
> virtio-blk. Zone revalidation, blkzone report and a sequential
> write / write-pointer check return correct values; blktests zbd
> device tests 001-006 (sysfs+ioctl, report zone, reset, write split,
> write ordering, revalidate) pass, with results identical before and
> after this change -- expected, since on a VIRTIO_F_VERSION_1 device
> virtio*_to_cpu() == le*_to_cpu().
>
> drivers/block/virtio_blk.c | 38 +++++++++++++++------------------
> include/uapi/linux/virtio_blk.h | 18 ++++++++--------
> 2 files changed, 26 insertions(+), 30 deletions(-)
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [PATCH net v4] virtio-net: fix len check in receive_big()
From: patchwork-bot+netdevbpf @ 2026-06-19 0:50 UTC (permalink / raw)
To: Xiang Mei
Cc: mst, jasowang, xuanzhuo, eperezma, andrew+netdev, davem, edumazet,
kuba, pabeni, netdev, virtualization, linux-kernel,
minhquangbui99, bestswngs
In-Reply-To: <20260616042837.2249468-1-xmei5@asu.edu>
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:
On Mon, 15 Jun 2026 21:28:37 -0700 you wrote:
> receive_big() bounds the device-announced length by
> (big_packets_num_skbfrags + 1) * PAGE_SIZE. That is still too loose:
> add_recvbuf_big() sets sg[1] to start at offset
> sizeof(struct padded_vnet_hdr) into the first page, so the chain
> actually carries hdr_len + (PAGE_SIZE - sizeof(padded_vnet_hdr)) +
> big_packets_num_skbfrags * PAGE_SIZE bytes -- 20 bytes less than the
> check allows for the common hdr_len == 12 case.
>
> [...]
Here is the summary with links:
- [net,v4] virtio-net: fix len check in receive_big()
https://git.kernel.org/netdev/net/c/9e5ad06ea826
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply
* [PATCH v2] vdpa_sim: fix cleanup after worker creation failure
From: Linfeng Sun @ 2026-06-20 10:09 UTC (permalink / raw)
To: Michael S . Tsirkin, Jason Wang
Cc: Xuan Zhuo, Eugenio Pérez, virtualization, linux-kernel,
Linfeng Sun
vdpasim_create() leaves vdpasim->worker as an ERR_PTR when
kthread_run_worker() fails. The error path then drops the device
reference, which releases the partially initialized simulator.
vdpasim_free() unconditionally passes the worker pointer to
kthread_destroy_worker(), so the ERR_PTR is dereferenced and can trigger
a general protection fault.
Store the worker error, clear the pointer, and only clean up the worker
when it was successfully initialized. Also make the release path tolerate
partially initialized objects by guarding virtqueue and IOTLB cleanup,
since the same release path can be reached from other initialization
failures.
I found this bug myself, though the patch was written with AI assistance.
Fixes: 76acfa7bc54f ("vdpa_sim: use kthread worker")
Assisted-by: OpenAI-Codex:GPT-5
Reviewed-by: Eugenio Pérez <eperezma@redhat.com>
Signed-off-by: Linfeng Sun <slf@hdu.edu.cn>
---
v1 -> v2:
- Remove the unnecessary vdpasim->iommu_pt check from IOTLB cleanup.
- Update the commit message to clarify the cleanup path for partially
initialized devices.
- Add Fixes and Assisted-by tags.
- Link to v1: https://lore.kernel.org/r/20260612105054.1850453-1-slf@hdu.edu.cn/
drivers/vdpa/vdpa_sim/vdpa_sim.c | 25 +++++++++++++++++--------
1 file changed, 17 insertions(+), 8 deletions(-)
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index 8cb1cc2ea139..2aa34df4d0ad 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -231,8 +231,11 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr,
kthread_init_work(&vdpasim->work, vdpasim_work_fn);
vdpasim->worker = kthread_run_worker(0, "vDPA sim worker: %s",
dev_attr->name);
- if (IS_ERR(vdpasim->worker))
+ if (IS_ERR(vdpasim->worker)) {
+ ret = PTR_ERR(vdpasim->worker);
+ vdpasim->worker = NULL;
goto err_iommu;
+ }
mutex_init(&vdpasim->mutex);
spin_lock_init(&vdpasim->iommu_lock);
@@ -742,18 +745,24 @@ static void vdpasim_free(struct vdpa_device *vdpa)
struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
int i;
- kthread_cancel_work_sync(&vdpasim->work);
- kthread_destroy_worker(vdpasim->worker);
+ if (vdpasim->worker) {
+ kthread_cancel_work_sync(&vdpasim->work);
+ kthread_destroy_worker(vdpasim->worker);
+ }
- for (i = 0; i < vdpasim->dev_attr.nvqs; i++) {
- vringh_kiov_cleanup(&vdpasim->vqs[i].out_iov);
- vringh_kiov_cleanup(&vdpasim->vqs[i].in_iov);
+ if (vdpasim->vqs) {
+ for (i = 0; i < vdpasim->dev_attr.nvqs; i++) {
+ vringh_kiov_cleanup(&vdpasim->vqs[i].out_iov);
+ vringh_kiov_cleanup(&vdpasim->vqs[i].in_iov);
+ }
}
vdpasim->dev_attr.free(vdpasim);
- for (i = 0; i < vdpasim->dev_attr.nas; i++)
- vhost_iotlb_reset(&vdpasim->iommu[i]);
+ if (vdpasim->iommu) {
+ for (i = 0; i < vdpasim->dev_attr.nas; i++)
+ vhost_iotlb_reset(&vdpasim->iommu[i]);
+ }
kfree(vdpasim->iommu);
kfree(vdpasim->iommu_pt);
kfree(vdpasim->vqs);
--
2.43.0
^ permalink raw reply related
* [PATCH] drm/virtio: bound EDID block reads to the response buffer
From: Bryam Vargas via B4 Relay @ 2026-06-21 2:43 UTC (permalink / raw)
To: Dmitry Osipenko, David Airlie, Gerd Hoffmann
Cc: linux-kernel, Gurchetan Singh, Chia-I Wu, dri-devel,
virtualization
From: Bryam Vargas <hexlabsecurity@proton.me>
virtio_get_edid_block() validates the read offset only against the
device-supplied resp->size field, never against the fixed-size resp->edid
array. The EDID block index is driven by the device-supplied extension
count, so a malicious virtio-gpu backend can advertise a large size
together with a high block count and read far past the array into adjacent
kernel memory, which is then surfaced in the parsed EDID (an out-of-bounds
read / info leak).
Also reject any read whose end exceeds the size of the edid array.
Conforming EDID responses stay within the array and are unaffected.
Fixes: b4b01b4995fb ("drm/virtio: add edid support")
Cc: stable@vger.kernel.org
Signed-off-by: Bryam Vargas <hexlabsecurity@proton.me>
---
drivers/gpu/drm/virtio/virtgpu_vq.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/virtio/virtgpu_vq.c b/drivers/gpu/drm/virtio/virtgpu_vq.c
index 67865810a2e7..c8b9475a7472 100644
--- a/drivers/gpu/drm/virtio/virtgpu_vq.c
+++ b/drivers/gpu/drm/virtio/virtgpu_vq.c
@@ -897,7 +897,8 @@ static int virtio_get_edid_block(void *data, u8 *buf,
struct virtio_gpu_resp_edid *resp = data;
size_t start = block * EDID_LENGTH;
- if (start + len > le32_to_cpu(resp->size))
+ if (start + len > le32_to_cpu(resp->size) ||
+ start + len > sizeof(resp->edid))
return -EINVAL;
memcpy(buf, resp->edid + start, len);
return 0;
---
base-commit: 1a3746ccbb0a97bed3c06ccde6b880013b1dddc1
change-id: 20260620-b4-disp-22bba7bf-47d80f0c083b
Best regards,
--
Bryam Vargas <hexlabsecurity@proton.me>
^ permalink raw reply related
* [PATCH] crypto: virtio - bound the device-reported akcipher result
From: Bryam Vargas via B4 Relay @ 2026-06-21 2:44 UTC (permalink / raw)
To: Herbert Xu, Jason Wang, Michael S. Tsirkin, Gonglei
Cc: virtualization, Xuan Zhuo, linux-crypto, linux-kernel,
Eugenio Pérez, David S. Miller
From: Bryam Vargas <hexlabsecurity@proton.me>
length
virtio_crypto_dataq_akcipher_callback() sets the result length from the
device-reported response length without bounding it to the destination
buffer, which was allocated for the original request length.
sg_copy_from_buffer() then reads that many bytes from the destination
buffer; a backend reporting a larger length over-reads adjacent kernel
heap into the caller's scatterlist (an out-of-bounds read).
Clamp the reported length to the originally requested destination length.
A conforming device reports no more than that, so valid results are
unaffected.
Fixes: a36bd0ad9fbf ("virtio-crypto: adjust dst_len at ops callback")
Cc: stable@vger.kernel.org
Signed-off-by: Bryam Vargas <hexlabsecurity@proton.me>
---
drivers/crypto/virtio/virtio_crypto_akcipher_algs.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
index d8d452cac391..64ea141f018c 100644
--- a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
+++ b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
@@ -88,7 +88,8 @@ static void virtio_crypto_dataq_akcipher_callback(struct virtio_crypto_request *
}
/* actual length may be less than dst buffer */
- akcipher_req->dst_len = len - sizeof(vc_req->status);
+ akcipher_req->dst_len = min_t(unsigned int, len - sizeof(vc_req->status),
+ akcipher_req->dst_len);
sg_copy_from_buffer(akcipher_req->dst, sg_nents(akcipher_req->dst),
vc_akcipher_req->dst_buf, akcipher_req->dst_len);
virtio_crypto_akcipher_finalize_req(vc_akcipher_req, akcipher_req, error);
---
base-commit: 1a3746ccbb0a97bed3c06ccde6b880013b1dddc1
change-id: 20260620-b4-disp-27caeeac-5b8b67962fdd
Best regards,
--
Bryam Vargas <hexlabsecurity@proton.me>
^ permalink raw reply related
* Re: [PATCH] crypto: virtio - bound the device-reported akcipher result
From: Michael S. Tsirkin @ 2026-06-21 5:33 UTC (permalink / raw)
To: hexlabsecurity
Cc: Herbert Xu, Jason Wang, Gonglei, virtualization, Xuan Zhuo,
linux-crypto, linux-kernel, Eugenio Pérez, David S. Miller
In-Reply-To: <20260620-b4-disp-27caeeac-v1-1-956e8f9c4f01@proton.me>
On Sat, Jun 20, 2026 at 09:44:21PM -0500, Bryam Vargas via B4 Relay wrote:
> From: Bryam Vargas <hexlabsecurity@proton.me>
>
> length
some kind of corruption here.
> virtio_crypto_dataq_akcipher_callback() sets the result length from the
> device-reported response length without bounding it to the destination
> buffer, which was allocated for the original request length.
> sg_copy_from_buffer() then reads that many bytes from the destination
> buffer; a backend reporting a larger length over-reads adjacent kernel
> heap into the caller's scatterlist (an out-of-bounds read).
>
> Clamp the reported length to the originally requested destination length.
> A conforming device reports no more than that, so valid results are
> unaffected.
>
> Fixes: a36bd0ad9fbf ("virtio-crypto: adjust dst_len at ops callback")
> Cc: stable@vger.kernel.org
> Signed-off-by: Bryam Vargas <hexlabsecurity@proton.me>
> ---
> drivers/crypto/virtio/virtio_crypto_akcipher_algs.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
> index d8d452cac391..64ea141f018c 100644
> --- a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
> +++ b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
> @@ -88,7 +88,8 @@ static void virtio_crypto_dataq_akcipher_callback(struct virtio_crypto_request *
> }
>
> /* actual length may be less than dst buffer */
> - akcipher_req->dst_len = len - sizeof(vc_req->status);
> + akcipher_req->dst_len = min_t(unsigned int, len - sizeof(vc_req->status),
> + akcipher_req->dst_len);
> sg_copy_from_buffer(akcipher_req->dst, sg_nents(akcipher_req->dst),
> vc_akcipher_req->dst_buf, akcipher_req->dst_len);
> virtio_crypto_akcipher_finalize_req(vc_akcipher_req, akcipher_req, error);
>
> ---
> base-commit: 1a3746ccbb0a97bed3c06ccde6b880013b1dddc1
> change-id: 20260620-b4-disp-27caeeac-5b8b67962fdd
>
> Best regards,
> --
> Bryam Vargas <hexlabsecurity@proton.me>
>
^ permalink raw reply
* [PATCH v6 00/12] nvdimm: virtio_pmem: fix request lifetime and converge broken queue failures
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel
Hi,
The nvdimm flush helper currently converts any non-zero provider flush
callback error to -EIO. That hides useful errno values from providers. For
example, virtio-pmem may fail flush allocation with -ENOMEM, but that is
currently reported as -EIO by nvdimm_flush().
The raw failure seen in the local mkfs sanity test was:
wipefs: /dev/pmem0: cannot flush modified buffers: Input/output error
mkfs.ext4: Input/output error while writing out and closing file system
nd_region region0: dbg: nvdimm_flush rc=-5
The first five patches keep provider flush errors intact, make
pmem_submit_bio() honor a failed REQ_PREFLUSH before copying data, keep
dataless bios out of the data loop, and avoid allocating a child flush bio
for virtio-pmem REQ_FUA handling. REQ_PREFLUSH and REQ_FUA are now issued
synchronously from pmem_submit_bio(). After that, virtio-pmem only allocates
its request object for the actual provider flush, and that allocation uses
GFP_NOIO so reclaim does not recurse into filesystem or block IO.
The rest of the series addresses virtio-pmem request lifetime and broken
virtqueue handling. The virtio-pmem flush path uses a virtqueue cookie/token
to carry a per-request context through completion. Under broken virtqueue /
notify failure conditions, the submitter can return and free the request
object while the host/backend may still complete the published request. The
IRQ completion handler then dereferences freed memory when waking waiters,
which is reported by KASAN as a slab-use-after-free and may manifest as lock
corruption (e.g. "BUG: spinlock already unlocked") without KASAN.
In addition, the flush path has two wait sites: one for virtqueue descriptor
availability (-ENOSPC from virtqueue_add_sgs()) and one for request
completion. If the virtqueue becomes broken, forward progress is no longer
guaranteed and these waiters may sleep indefinitely unless the driver
converges the failure and wakes all wait sites. This version also orders
response publication with release/acquire, keeps DMA_FROM_DEVICE response
storage away from CPU-owned request fields, and wakes the in-flight
completion waiter when a queue is marked broken.
This series addresses these issues:
1/12 nvdimm: preserve flush callback errors
Return provider flush callback errors directly from nvdimm_flush().
2/12 nvdimm: pmem: keep PREFLUSH before data writes
Run REQ_PREFLUSH synchronously before copying data and fail the bio if the
flush fails.
3/12 nvdimm: pmem: guard data loop for dataless bios
Keep flush-only bios out of the data copy loop.
4/12 nvdimm: virtio_pmem: stop allocating child flush bio
Flush REQ_FUA synchronously instead of allocating a chained child bio.
5/12 nvdimm: virtio_pmem: use GFP_NOIO for flush requests
Use GFP_NOIO for the virtio-pmem request allocation.
6/12 nvdimm: virtio_pmem: always wake -ENOSPC waiters
Wake one -ENOSPC waiter for each reclaimed used buffer, decoupled from
token completion.
7/12 nvdimm: virtio_pmem: use READ_ONCE()/WRITE_ONCE() for wait flags
Use READ_ONCE()/WRITE_ONCE() for the wait_event() flags (done and
wq_buf_avail).
8/12 nvdimm: virtio_pmem: refcount requests for token lifetime
Refcount request objects so the token lifetime spans the window where it is
reachable through the virtqueue until completion/drain drops the virtqueue
reference.
9/12 nvdimm: virtio_pmem: publish done with release/acquire
Order response publication before the submitter observes request completion.
10/12 nvdimm: virtio_pmem: isolate DMA request buffers
Keep the DMA_FROM_DEVICE response buffer away from CPU-owned request fields.
11/12 nvdimm: virtio_pmem: converge broken virtqueue to -EIO
Track a device-level broken state to converge broken/notify failures to -EIO:
wake -ENOSPC waiters, wake the in-flight completion waiter, fail-fast new
requests, and report errors after the queue is marked broken.
12/12 nvdimm: virtio_pmem: drain requests in freeze
Drain outstanding requests in freeze() after resetting the device so waiters
do not sleep indefinitely and virtqueue_detach_unused_buf() only runs on a
quiesced queue.
The original repros were on QEMU x86_64 with a virtio-pmem device exported
as /dev/pmem0. For this v6 reroll, the series applies to v7.1-rc7 and to
local next/master at 4fa3f5fabb30 ("Add linux-next specific files for
20260616").
Thanks,
Li Chen
Changelog:
v5->v6:
- Address Sashiko review feedback:
- Add a data-loop guard for dataless bios in pmem_submit_bio().
- Replace the child flush bio allocation with synchronous FUA flushing.
- Keep GFP_NOIO only for the virtio-pmem request allocation.
- Publish request completion with release/acquire ordering.
- Isolate the DMA_FROM_DEVICE response buffer from CPU-owned fields.
- Wake the in-flight host-completion waiter when marking the queue broken.
- Clear req_vq after del_vqs() and make drain tolerate a NULL queue.
v4->v5:
- Address review feedback about REQ_PREFLUSH ordering and active virtqueue
detach.
- Add 2/8 so a failed REQ_PREFLUSH fails the bio before any data copy, and
make REQ_PREFLUSH use a synchronous provider flush instead of a deferred
child bio.
- Rework broken-queue handling so runtime failure marking only stops new
submissions and wakes local -ENOSPC waiters; used/unused token draining is
done after device reset in remove() and freeze().
- Remove the broken-state shortcut from the host-completion wait so the
submitter never reads an uninitialized response field.
- Keep the raw broken-virtqueue dmesg in 7/8 while updating the teardown
rationale.
- Renumber the old virtio-pmem fixes after the new pmem PREFLUSH patch.
v3->v4:
- Rebased the series onto v7.1-rc7 so it applies cleanly to Linux 7.1-rc7.
- Update the allocation site in 6/7 from kmalloc(sizeof(*req_data),
GFP_KERNEL) to kmalloc_obj(*req_data) to match current nvdimm code.
- Add 1/7 to preserve provider flush callback errors in nvdimm_flush().
- Include the GFP_NOIO child flush bio allocation fix as 2/7.
- Renumber the old request lifetime and broken virtqueue fixes after the two
new flush error patches.
v2->v3:
- Split patch 1 as suggested by Pankaj Gupta: keep the waiter wakeup
ordering change in 1/5 and move READ_ONCE()/WRITE_ONCE() updates to
2/5 (no functional change intended).
- Add log report to commit msg.
- Fold the export fix into 4/5 to keep the series bisectable when
CONFIG_VIRTIO_PMEM=m.
v1->v2:
- Add the export patch to fix compile issue.
Links:
v5: https://lore.kernel.org/all/20260617122442.2118957-1-me@linux.beauty/
v4: https://lore.kernel.org/all/20260609120726.1714780-1-me@linux.beauty/
v3: https://lore.kernel.org/all/20260226025712.2236279-1-me@linux.beauty/#t
v2: https://lore.kernel.org/all/20251225042915.334117-1-me@linux.beauty/
v1: https://www.spinics.net/lists/kernel/msg5974818.html
Li Chen (12):
nvdimm: preserve flush callback errors
nvdimm: pmem: keep PREFLUSH before data writes
nvdimm: pmem: guard data loop for dataless bios
nvdimm: virtio_pmem: stop allocating child flush bio
nvdimm: virtio_pmem: use GFP_NOIO for flush requests
nvdimm: virtio_pmem: always wake -ENOSPC waiters
nvdimm: virtio_pmem: use READ_ONCE()/WRITE_ONCE() for wait flags
nvdimm: virtio_pmem: refcount requests for token lifetime
nvdimm: virtio_pmem: publish done with release/acquire
nvdimm: virtio_pmem: isolate DMA request buffers
nvdimm: virtio_pmem: converge broken virtqueue to -EIO
nvdimm: virtio_pmem: drain requests in freeze
drivers/nvdimm/nd_virtio.c | 224 +++++++++++++++++++++++++++--------
drivers/nvdimm/pmem.c | 52 ++++----
drivers/nvdimm/region_devs.c | 6 +-
drivers/nvdimm/virtio_pmem.c | 51 +++++++-
drivers/nvdimm/virtio_pmem.h | 18 ++-
5 files changed, 270 insertions(+), 81 deletions(-)
--
2.52.0
^ permalink raw reply
* [PATCH v6 01/12] nvdimm: preserve flush callback errors
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
nvdimm_flush() currently converts any non-zero provider flush error to
-EIO. That loses useful errno values from provider callbacks.
A local virtio-pmem mkfs sanity test showed the masking clearly:
wipefs: /dev/pmem0: cannot flush modified buffers: Input/output error
mkfs.ext4: Input/output error while writing out and closing file system
nd_region region0: dbg: nvdimm_flush rc=-5
The virtio-pmem callback can return -ENOMEM when async_pmem_flush() fails
to allocate a child flush bio, but nvdimm_flush() hides that as -EIO before
pmem_submit_bio() converts it to a block status.
Return the provider callback error directly. The generic flush path still
returns 0, and pmem_submit_bio() already handles errno-to-blk_status
conversion for bio completion.
Signed-off-by: Li Chen <me@linux.beauty>
---
v3->v4:
- New patch.
drivers/nvdimm/region_devs.c | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/drivers/nvdimm/region_devs.c b/drivers/nvdimm/region_devs.c
index e35c2e18518f0..0cd96503c0596 100644
--- a/drivers/nvdimm/region_devs.c
+++ b/drivers/nvdimm/region_devs.c
@@ -1114,10 +1114,8 @@ int nvdimm_flush(struct nd_region *nd_region, struct bio *bio)
if (!nd_region->flush)
rc = generic_nvdimm_flush(nd_region);
- else {
- if (nd_region->flush(nd_region, bio))
- rc = -EIO;
- }
+ else
+ rc = nd_region->flush(nd_region, bio);
return rc;
}
--
2.52.0
^ permalink raw reply related
* [PATCH v6 02/12] nvdimm: pmem: keep PREFLUSH before data writes
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
pmem_submit_bio() records a REQ_PREFLUSH error, but continues to copy the
bio data and can later overwrite the error with a successful REQ_FUA flush.
That lets data writes run after a failed preflush and can complete the bio
successfully despite the failed ordering barrier.
Run the REQ_PREFLUSH flush synchronously before touching the bio data and
complete the bio with the flush error if it fails. Keep asynchronous flush
chaining for REQ_FUA. At that point, data copy has completed and the parent
bio can wait for the chained flush bio.
Signed-off-by: Li Chen <me@linux.beauty>
---
Changes in v5:
- New patch.
drivers/nvdimm/pmem.c | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 92c67fbbc1c85..05d3de33e2706 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -208,8 +208,14 @@ static void pmem_submit_bio(struct bio *bio)
struct pmem_device *pmem = bio->bi_bdev->bd_disk->private_data;
struct nd_region *nd_region = to_region(pmem);
- if (bio->bi_opf & REQ_PREFLUSH)
- ret = nvdimm_flush(nd_region, bio);
+ if (bio->bi_opf & REQ_PREFLUSH) {
+ ret = nvdimm_flush(nd_region, NULL);
+ if (ret) {
+ bio->bi_status = errno_to_blk_status(ret);
+ bio_endio(bio);
+ return;
+ }
+ }
do_acct = blk_queue_io_stat(bio->bi_bdev->bd_disk->queue);
if (do_acct)
@@ -229,7 +235,7 @@ static void pmem_submit_bio(struct bio *bio)
if (do_acct)
bio_end_io_acct(bio, start);
- if (bio->bi_opf & REQ_FUA)
+ if ((bio->bi_opf & REQ_FUA) && !bio->bi_status)
ret = nvdimm_flush(nd_region, bio);
if (ret)
--
2.52.0
^ permalink raw reply related
* [PATCH v6 03/12] nvdimm: pmem: guard data loop for dataless bios
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
pmem_submit_bio() handles flush-only bios before and after the data
loop. Keep dataless bios out of bio_for_each_segment() so the data path
only walks bios that actually carry bvec data.
Signed-off-by: Li Chen <me@linux.beauty>
---
Changes in v6:
- New patch.
drivers/nvdimm/pmem.c | 36 +++++++++++++++++++++---------------
1 file changed, 21 insertions(+), 15 deletions(-)
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 05d3de33e2706..82ee1ddb3a445 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -217,23 +217,29 @@ static void pmem_submit_bio(struct bio *bio)
}
}
- do_acct = blk_queue_io_stat(bio->bi_bdev->bd_disk->queue);
- if (do_acct)
- start = bio_start_io_acct(bio);
- bio_for_each_segment(bvec, bio, iter) {
- if (op_is_write(bio_op(bio)))
- rc = pmem_do_write(pmem, bvec.bv_page, bvec.bv_offset,
- iter.bi_sector, bvec.bv_len);
- else
- rc = pmem_do_read(pmem, bvec.bv_page, bvec.bv_offset,
- iter.bi_sector, bvec.bv_len);
- if (rc) {
- bio->bi_status = rc;
- break;
+ if (bio_has_data(bio)) {
+ do_acct = blk_queue_io_stat(bio->bi_bdev->bd_disk->queue);
+ if (do_acct)
+ start = bio_start_io_acct(bio);
+ bio_for_each_segment(bvec, bio, iter) {
+ if (op_is_write(bio_op(bio)))
+ rc = pmem_do_write(pmem, bvec.bv_page,
+ bvec.bv_offset,
+ iter.bi_sector,
+ bvec.bv_len);
+ else
+ rc = pmem_do_read(pmem, bvec.bv_page,
+ bvec.bv_offset,
+ iter.bi_sector,
+ bvec.bv_len);
+ if (rc) {
+ bio->bi_status = rc;
+ break;
+ }
}
+ if (do_acct)
+ bio_end_io_acct(bio, start);
}
- if (do_acct)
- bio_end_io_acct(bio, start);
if ((bio->bi_opf & REQ_FUA) && !bio->bi_status)
ret = nvdimm_flush(nd_region, bio);
--
2.52.0
^ permalink raw reply related
* [PATCH v6 04/12] nvdimm: virtio_pmem: stop allocating child flush bio
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
pmem_submit_bio() passes the parent bio to nvdimm_flush() for
REQ_FUA. For virtio-pmem this makes async_pmem_flush() allocate
and submit a child PREFLUSH bio chained to the parent.
That child allocation is in the block submit path. Making it
blocking with GFP_NOIO can consume the same global bio mempool that
submit_bio() uses, while making it GFP_ATOMIC can fail under
pressure. A forced failure of the child allocation produced:
virtio_pmem: forcing child bio allocation failure for test
Buffer I/O error on dev pmem0, logical block 0, lost sync page write
EXT4-fs (pmem0): I/O error while writing superblock
EXT4-fs (pmem0): mount failed
Avoid the child bio completely. Flush FUA synchronously, like
REQ_PREFLUSH, then complete the parent after the flush. Since no
child bio can be created, async_pmem_flush() now only issues the
virtio flush and preserves negative errno values.
Signed-off-by: Li Chen <me@linux.beauty>
---
Changes in v6:
- Replace the child bio allocation fix with synchronous FUA flushing.
drivers/nvdimm/nd_virtio.c | 22 ++++------------------
drivers/nvdimm/pmem.c | 2 +-
2 files changed, 5 insertions(+), 19 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 4176046627beb..4b2e9c47af0f5 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -110,27 +110,13 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
/* The asynchronous flush callback function */
int async_pmem_flush(struct nd_region *nd_region, struct bio *bio)
{
- /*
- * Create child bio for asynchronous flush and chain with
- * parent bio. Otherwise directly call nd_region flush.
- */
- if (bio && bio->bi_iter.bi_sector != -1) {
- struct bio *child = bio_alloc(bio->bi_bdev, 0,
- REQ_OP_WRITE | REQ_PREFLUSH,
- GFP_ATOMIC);
+ int err;
- if (!child)
- return -ENOMEM;
- bio_clone_blkg_association(child, bio);
- child->bi_iter.bi_sector = -1;
- bio_chain(child, bio);
- submit_bio(child);
- return 0;
- }
- if (virtio_pmem_flush(nd_region))
+ err = virtio_pmem_flush(nd_region);
+ if (err > 0)
return -EIO;
- return 0;
+ return err;
};
EXPORT_SYMBOL_GPL(async_pmem_flush);
MODULE_DESCRIPTION("Virtio Persistent Memory Driver");
diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c
index 82ee1ddb3a445..058d2739c95a1 100644
--- a/drivers/nvdimm/pmem.c
+++ b/drivers/nvdimm/pmem.c
@@ -242,7 +242,7 @@ static void pmem_submit_bio(struct bio *bio)
}
if ((bio->bi_opf & REQ_FUA) && !bio->bi_status)
- ret = nvdimm_flush(nd_region, bio);
+ ret = nvdimm_flush(nd_region, NULL);
if (ret)
bio->bi_status = errno_to_blk_status(ret);
--
2.52.0
^ permalink raw reply related
* [PATCH v6 05/12] nvdimm: virtio_pmem: use GFP_NOIO for flush requests
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
virtio_pmem_flush() can run from pmem_submit_bio() while filesystem IO
is waiting on the flush completion. The request object allocation can
sleep, but it should not enter filesystem or block IO reclaim from this
flush path.
Use GFP_NOIO for the request allocation. The virtqueue descriptor
allocation still uses GFP_ATOMIC because it runs under pmem_lock.
Signed-off-by: Li Chen <me@linux.beauty>
---
Changes in v6:
- New patch; keep GFP_NOIO only for the virtio-pmem request allocation.
drivers/nvdimm/nd_virtio.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 4b2e9c47af0f5..91ca144607531 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -55,7 +55,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
return -EIO;
}
- req_data = kmalloc_obj(*req_data);
+ req_data = kmalloc_obj(*req_data, GFP_NOIO);
if (!req_data)
return -ENOMEM;
--
2.52.0
^ permalink raw reply related
* [PATCH v6 06/12] nvdimm: virtio_pmem: always wake -ENOSPC waiters
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
virtio_pmem_host_ack() reclaims virtqueue descriptors with
virtqueue_get_buf(). The -ENOSPC waiter wakeup is tied to completing the
returned token. If token completion is skipped for any reason, reclaimed
descriptors may not wake a waiter and the submitter may sleep forever
waiting for a free slot. Always wake one -ENOSPC waiter for each virtqueue
completion before touching the returned token.
Signed-off-by: Li Chen <me@linux.beauty>
---
v2->v3:
- Split out the waiter wakeup ordering change from READ_ONCE()/WRITE_ONCE()
updates (now patch 4/7), per Pankaj's suggestion.
v3->v4:
- Rebased onto v7.1-rc7 and renumbered after the flush error patches.
drivers/nvdimm/nd_virtio.c | 25 ++++++++++++++++---------
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 91ca144607531..8ed4d6b3a9284 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -9,26 +9,33 @@
#include "virtio_pmem.h"
#include "nd.h"
+static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
+{
+ struct virtio_pmem_request *req_buf;
+
+ if (list_empty(&vpmem->req_list))
+ return;
+
+ req_buf = list_first_entry(&vpmem->req_list,
+ struct virtio_pmem_request, list);
+ req_buf->wq_buf_avail = true;
+ wake_up(&req_buf->wq_buf);
+ list_del(&req_buf->list);
+}
+
/* The interrupt handler */
void virtio_pmem_host_ack(struct virtqueue *vq)
{
struct virtio_pmem *vpmem = vq->vdev->priv;
- struct virtio_pmem_request *req_data, *req_buf;
+ struct virtio_pmem_request *req_data;
unsigned long flags;
unsigned int len;
spin_lock_irqsave(&vpmem->pmem_lock, flags);
while ((req_data = virtqueue_get_buf(vq, &len)) != NULL) {
+ virtio_pmem_wake_one_waiter(vpmem);
req_data->done = true;
wake_up(&req_data->host_acked);
-
- if (!list_empty(&vpmem->req_list)) {
- req_buf = list_first_entry(&vpmem->req_list,
- struct virtio_pmem_request, list);
- req_buf->wq_buf_avail = true;
- wake_up(&req_buf->wq_buf);
- list_del(&req_buf->list);
- }
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
}
--
2.52.0
^ permalink raw reply related
* [PATCH v6 07/12] nvdimm: virtio_pmem: use READ_ONCE()/WRITE_ONCE() for wait flags
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
Use READ_ONCE()/WRITE_ONCE() for the wait_event() flags (done and
wq_buf_avail). They are observed by waiters without pmem_lock, so make
the accesses explicit single loads/stores and avoid compiler
reordering/caching across the wait/wake paths.
Signed-off-by: Li Chen <me@linux.beauty>
---
v2->v3:
- Split out READ_ONCE()/WRITE_ONCE() updates from patch 3/7 (no functional
change intended).
v3->v4:
- Rebased onto v7.1-rc7 and renumbered after the flush error patches.
drivers/nvdimm/nd_virtio.c | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 8ed4d6b3a9284..da829e9f4bdff 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -18,9 +18,9 @@ static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
req_buf = list_first_entry(&vpmem->req_list,
struct virtio_pmem_request, list);
- req_buf->wq_buf_avail = true;
+ list_del_init(&req_buf->list);
+ WRITE_ONCE(req_buf->wq_buf_avail, true);
wake_up(&req_buf->wq_buf);
- list_del(&req_buf->list);
}
/* The interrupt handler */
@@ -34,7 +34,7 @@ void virtio_pmem_host_ack(struct virtqueue *vq)
spin_lock_irqsave(&vpmem->pmem_lock, flags);
while ((req_data = virtqueue_get_buf(vq, &len)) != NULL) {
virtio_pmem_wake_one_waiter(vpmem);
- req_data->done = true;
+ WRITE_ONCE(req_data->done, true);
wake_up(&req_data->host_acked);
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
@@ -66,7 +66,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
if (!req_data)
return -ENOMEM;
- req_data->done = false;
+ WRITE_ONCE(req_data->done, false);
init_waitqueue_head(&req_data->host_acked);
init_waitqueue_head(&req_data->wq_buf);
INIT_LIST_HEAD(&req_data->list);
@@ -87,12 +87,12 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
GFP_ATOMIC)) == -ENOSPC) {
dev_info(&vdev->dev, "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
- req_data->wq_buf_avail = false;
+ WRITE_ONCE(req_data->wq_buf_avail, false);
list_add_tail(&req_data->list, &vpmem->req_list);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
/* A host response results in "host_ack" getting called */
- wait_event(req_data->wq_buf, req_data->wq_buf_avail);
+ wait_event(req_data->wq_buf, READ_ONCE(req_data->wq_buf_avail));
spin_lock_irqsave(&vpmem->pmem_lock, flags);
}
err1 = virtqueue_kick(vpmem->req_vq);
@@ -106,7 +106,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
err = -EIO;
} else {
/* A host response results in "host_ack" getting called */
- wait_event(req_data->host_acked, req_data->done);
+ wait_event(req_data->host_acked, READ_ONCE(req_data->done));
err = le32_to_cpu(req_data->resp.ret);
}
--
2.52.0
^ permalink raw reply related
* [PATCH v6 08/12] nvdimm: virtio_pmem: refcount requests for token lifetime
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, stable, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
KASAN reports slab-use-after-free in __wake_up_common():
BUG: KASAN: slab-use-after-free in __wake_up_common+0x114/0x160
Read of size 8 at addr ffff88810fdcb710 by task swapper/0/0
CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted
6.19.0-next-20260220-00006-g1eae5f204ec3 #4 PREEMPT(full)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Arch Linux
1.17.0-2-2 04/01/2014
Call Trace:
<IRQ>
dump_stack_lvl+0x6d/0xb0
print_report+0x170/0x4e2
? __pfx__raw_spin_lock_irqsave+0x10/0x10
? __virt_addr_valid+0x1dc/0x380
kasan_report+0xbc/0xf0
? __wake_up_common+0x114/0x160
? __wake_up_common+0x114/0x160
__wake_up_common+0x114/0x160
? __pfx__raw_spin_lock_irqsave+0x10/0x10
__wake_up+0x36/0x60
virtio_pmem_host_ack+0x11d/0x3b0
? sched_balance_domains+0x29f/0xb00
? __pfx_virtio_pmem_host_ack+0x10/0x10
? _raw_spin_lock_irqsave+0x98/0x100
? __pfx__raw_spin_lock_irqsave+0x10/0x10
vring_interrupt+0x1c9/0x5e0
? __pfx_vp_interrupt+0x10/0x10
vp_vring_interrupt+0x87/0x100
? __pfx_vp_interrupt+0x10/0x10
__handle_irq_event_percpu+0x17f/0x550
? __pfx__raw_spin_lock+0x10/0x10
handle_irq_event+0xab/0x1c0
handle_fasteoi_irq+0x276/0xae0
__common_interrupt+0x65/0x130
common_interrupt+0x78/0xa0
</IRQ>
virtio_pmem_host_ack() wakes a request that has already been freed by the
submitter.
This happens when the request token is still reachable via the virtqueue,
but virtio_pmem_flush() returns and frees it.
Fix the token lifetime by refcounting struct virtio_pmem_request.
virtio_pmem_flush() holds a submitter reference, and the virtqueue holds an
extra reference once the request is queued. The completion path drops the
virtqueue reference, and the submitter drops its reference before
returning.
Fixes: 6e84200c0a29 ("virtio-pmem: Add virtio pmem driver")
Cc: stable@vger.kernel.org
Signed-off-by: Li Chen <me@linux.beauty>
---
v2->v3:
- Add raw KASAN report to the patch description.
- Drop timestamps from the embedded report.
v3->v4:
- Rebased onto v7.1-rc7 and renumbered after the flush error patches.
drivers/nvdimm/nd_virtio.c | 34 +++++++++++++++++++++++++++++-----
drivers/nvdimm/virtio_pmem.h | 2 ++
2 files changed, 31 insertions(+), 5 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index da829e9f4bdff..7b6761adf28bc 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -9,6 +9,14 @@
#include "virtio_pmem.h"
#include "nd.h"
+static void virtio_pmem_req_release(struct kref *kref)
+{
+ struct virtio_pmem_request *req;
+
+ req = container_of(kref, struct virtio_pmem_request, kref);
+ kfree(req);
+}
+
static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
{
struct virtio_pmem_request *req_buf;
@@ -36,6 +44,7 @@ void virtio_pmem_host_ack(struct virtqueue *vq)
virtio_pmem_wake_one_waiter(vpmem);
WRITE_ONCE(req_data->done, true);
wake_up(&req_data->host_acked);
+ kref_put(&req_data->kref, virtio_pmem_req_release);
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
}
@@ -66,6 +75,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
if (!req_data)
return -ENOMEM;
+ kref_init(&req_data->kref);
WRITE_ONCE(req_data->done, false);
init_waitqueue_head(&req_data->host_acked);
init_waitqueue_head(&req_data->wq_buf);
@@ -83,10 +93,23 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
* to req_list and wait for host_ack to wake us up when free
* slots are available.
*/
- while ((err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data,
- GFP_ATOMIC)) == -ENOSPC) {
-
- dev_info(&vdev->dev, "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
+ for (;;) {
+ err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data,
+ GFP_ATOMIC);
+ if (!err) {
+ /*
+ * Take the virtqueue reference while @pmem_lock is
+ * held so completion cannot run concurrently.
+ */
+ kref_get(&req_data->kref);
+ break;
+ }
+
+ if (err != -ENOSPC)
+ break;
+
+ dev_info_ratelimited(&vdev->dev,
+ "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
WRITE_ONCE(req_data->wq_buf_avail, false);
list_add_tail(&req_data->list, &vpmem->req_list);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
@@ -95,6 +118,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
wait_event(req_data->wq_buf, READ_ONCE(req_data->wq_buf_avail));
spin_lock_irqsave(&vpmem->pmem_lock, flags);
}
+
err1 = virtqueue_kick(vpmem->req_vq);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
/*
@@ -110,7 +134,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
err = le32_to_cpu(req_data->resp.ret);
}
- kfree(req_data);
+ kref_put(&req_data->kref, virtio_pmem_req_release);
return err;
};
diff --git a/drivers/nvdimm/virtio_pmem.h b/drivers/nvdimm/virtio_pmem.h
index f72cf17f9518f..1017e498c9b4c 100644
--- a/drivers/nvdimm/virtio_pmem.h
+++ b/drivers/nvdimm/virtio_pmem.h
@@ -12,11 +12,13 @@
#include <linux/module.h>
#include <uapi/linux/virtio_pmem.h>
+#include <linux/kref.h>
#include <linux/libnvdimm.h>
#include <linux/mutex.h>
#include <linux/spinlock.h>
struct virtio_pmem_request {
+ struct kref kref;
struct virtio_pmem_req req;
struct virtio_pmem_resp resp;
--
2.52.0
^ permalink raw reply related
* [PATCH v6 09/12] nvdimm: virtio_pmem: publish done with release/acquire
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
virtio_pmem_host_ack() publishes the device response by setting done and
waking the submitter. The submitter reads resp.ret after wait_event()
observes done.
Use smp_store_release() on done and smp_load_acquire() in the wait
condition so the response read is ordered after completion.
Signed-off-by: Li Chen <me@linux.beauty>
---
Changes in v6:
- New patch.
drivers/nvdimm/nd_virtio.c | 19 ++++++++++++++++---
1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 7b6761adf28bc..35d36bd36a526 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -17,6 +17,19 @@ static void virtio_pmem_req_release(struct kref *kref)
kfree(req);
}
+static void virtio_pmem_signal_done(struct virtio_pmem_request *req)
+{
+ /* Pairs with smp_load_acquire() in virtio_pmem_req_done(). */
+ smp_store_release(&req->done, true);
+ wake_up(&req->host_acked);
+}
+
+static bool virtio_pmem_req_done(struct virtio_pmem_request *req)
+{
+ /* Pairs with smp_store_release() in virtio_pmem_signal_done(). */
+ return smp_load_acquire(&req->done);
+}
+
static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
{
struct virtio_pmem_request *req_buf;
@@ -42,8 +55,7 @@ void virtio_pmem_host_ack(struct virtqueue *vq)
spin_lock_irqsave(&vpmem->pmem_lock, flags);
while ((req_data = virtqueue_get_buf(vq, &len)) != NULL) {
virtio_pmem_wake_one_waiter(vpmem);
- WRITE_ONCE(req_data->done, true);
- wake_up(&req_data->host_acked);
+ virtio_pmem_signal_done(req_data);
kref_put(&req_data->kref, virtio_pmem_req_release);
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
@@ -130,7 +142,8 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
err = -EIO;
} else {
/* A host response results in "host_ack" getting called */
- wait_event(req_data->host_acked, READ_ONCE(req_data->done));
+ wait_event(req_data->host_acked,
+ virtio_pmem_req_done(req_data));
err = le32_to_cpu(req_data->resp.ret);
}
--
2.52.0
^ permalink raw reply related
* [PATCH v6 10/12] nvdimm: virtio_pmem: isolate DMA request buffers
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
The virtio-pmem request object stores wait queues, flags, and list
pointers next to buffers mapped for virtqueue DMA. The response buffer is
mapped DMA_FROM_DEVICE, so non-coherent DMA invalidation must not share a
cache line with CPU-owned fields.
Keep the request buffer outside the DMA-from-device group and wrap only
the response buffer with __dma_from_device_group_begin/end.
Signed-off-by: Li Chen <me@linux.beauty>
---
Changes in v6:
- New patch.
drivers/nvdimm/virtio_pmem.h | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/drivers/nvdimm/virtio_pmem.h b/drivers/nvdimm/virtio_pmem.h
index 1017e498c9b4c..23bff40249c1b 100644
--- a/drivers/nvdimm/virtio_pmem.h
+++ b/drivers/nvdimm/virtio_pmem.h
@@ -10,6 +10,7 @@
#ifndef _LINUX_VIRTIO_PMEM_H
#define _LINUX_VIRTIO_PMEM_H
+#include <linux/dma-mapping.h>
#include <linux/module.h>
#include <uapi/linux/virtio_pmem.h>
#include <linux/kref.h>
@@ -19,8 +20,6 @@
struct virtio_pmem_request {
struct kref kref;
- struct virtio_pmem_req req;
- struct virtio_pmem_resp resp;
/* Wait queue to process deferred work after ack from host */
wait_queue_head_t host_acked;
@@ -30,6 +29,11 @@ struct virtio_pmem_request {
wait_queue_head_t wq_buf;
bool wq_buf_avail;
struct list_head list;
+
+ struct virtio_pmem_req req;
+ __dma_from_device_group_begin(resp);
+ struct virtio_pmem_resp resp;
+ __dma_from_device_group_end(resp);
};
struct virtio_pmem {
--
2.52.0
^ permalink raw reply related
* [PATCH v6 11/12] nvdimm: virtio_pmem: converge broken virtqueue to -EIO
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
dmesg reports virtqueue failure and device reset:
virtio_pmem virtio2: failed to send command to
virtio pmem device, no free slots in the virtqueue
virtio_pmem virtio2: virtio pmem device
needs a reset
virtio_pmem_flush() can wait for a free virtqueue descriptor (-ENOSPC).
It can also wait for host completion. If the request virtqueue breaks,
those waiters may never make progress. One example is notify failure from
virtqueue_kick().
Track a device-level broken state and converge the failure to -EIO. New
requests fail fast, -ENOSPC waiters are unlinked and woken, and the
currently submitted request is woken so its host_acked waiter can return
without waiting forever for host completion. Completed requests are forced
to report an error after the queue is marked broken.
Do not detach unused buffers from an active virtqueue. Runtime
broken-queue handling only stops new submissions and wakes local waiters.
Removal resets the device first. It then drains request tokens. After
that, the device no longer owns the buffers when the virtqueue reference
is dropped.
Closes: https://lore.kernel.org/r/202512250116.ewtzlD0g-lkp@intel.com/
Signed-off-by: Li Chen <me@linux.beauty>
---
Changes in v6:
- Wake the in-flight host-completion waiter when marking the queue broken.
- Track req_inflight and clear it on completion/drain paths.
- Return -EIO if the queue breaks before a host response is observed.
Changes in v5:
- Split broken marking from token draining.
- Do not call virtqueue_detach_unused_buf() on an active queue.
- Reset the device before draining tokens in remove().
- Do not let the host-completion wait return only because the device is
marked broken.
v2->v3:
- Add raw dmesg excerpt to the patch description.
- Drop timestamps from the embedded dmesg.
- Fold the CONFIG_VIRTIO_PMEM=m export fix into this patch.
v3->v4:
- Rebased onto v7.1-rc7 and renumbered after the flush error patches.
- Use kmalloc_obj(*req_data) at the allocation site to match current nvdimm
code.
drivers/nvdimm/nd_virtio.c | 117 +++++++++++++++++++++++++++++++----
drivers/nvdimm/virtio_pmem.c | 15 ++++-
drivers/nvdimm/virtio_pmem.h | 8 +++
3 files changed, 127 insertions(+), 13 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 35d36bd36a526..fb9391ebc46e7 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -30,6 +30,12 @@ static bool virtio_pmem_req_done(struct virtio_pmem_request *req)
return smp_load_acquire(&req->done);
}
+static void virtio_pmem_complete_err(struct virtio_pmem_request *req)
+{
+ req->resp.ret = cpu_to_le32(1);
+ virtio_pmem_signal_done(req);
+}
+
static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
{
struct virtio_pmem_request *req_buf;
@@ -44,6 +50,63 @@ static void virtio_pmem_wake_one_waiter(struct virtio_pmem *vpmem)
wake_up(&req_buf->wq_buf);
}
+static void virtio_pmem_wake_all_waiters(struct virtio_pmem *vpmem)
+{
+ struct virtio_pmem_request *req, *tmp;
+
+ list_for_each_entry_safe(req, tmp, &vpmem->req_list, list) {
+ list_del_init(&req->list);
+ WRITE_ONCE(req->wq_buf_avail, true);
+ wake_up(&req->wq_buf);
+ }
+}
+
+static void virtio_pmem_clear_inflight(struct virtio_pmem *vpmem,
+ struct virtio_pmem_request *req)
+{
+ if (vpmem->req_inflight == req)
+ vpmem->req_inflight = NULL;
+}
+
+static void virtio_pmem_wake_inflight(struct virtio_pmem *vpmem)
+{
+ struct virtio_pmem_request *req = vpmem->req_inflight;
+
+ if (req)
+ wake_up(&req->host_acked);
+}
+
+void virtio_pmem_mark_broken(struct virtio_pmem *vpmem)
+{
+ if (!READ_ONCE(vpmem->broken)) {
+ WRITE_ONCE(vpmem->broken, true);
+ dev_err_once(&vpmem->vdev->dev, "virtqueue is broken\n");
+ }
+
+ virtio_pmem_wake_inflight(vpmem);
+ virtio_pmem_wake_all_waiters(vpmem);
+}
+EXPORT_SYMBOL_GPL(virtio_pmem_mark_broken);
+
+void virtio_pmem_drain(struct virtio_pmem *vpmem)
+{
+ struct virtio_pmem_request *req;
+ unsigned int len;
+
+ while ((req = virtqueue_get_buf(vpmem->req_vq, &len)) != NULL) {
+ virtio_pmem_clear_inflight(vpmem, req);
+ virtio_pmem_complete_err(req);
+ kref_put(&req->kref, virtio_pmem_req_release);
+ }
+
+ while ((req = virtqueue_detach_unused_buf(vpmem->req_vq)) != NULL) {
+ virtio_pmem_clear_inflight(vpmem, req);
+ virtio_pmem_complete_err(req);
+ kref_put(&req->kref, virtio_pmem_req_release);
+ }
+}
+EXPORT_SYMBOL_GPL(virtio_pmem_drain);
+
/* The interrupt handler */
void virtio_pmem_host_ack(struct virtqueue *vq)
{
@@ -54,8 +117,12 @@ void virtio_pmem_host_ack(struct virtqueue *vq)
spin_lock_irqsave(&vpmem->pmem_lock, flags);
while ((req_data = virtqueue_get_buf(vq, &len)) != NULL) {
+ virtio_pmem_clear_inflight(vpmem, req_data);
virtio_pmem_wake_one_waiter(vpmem);
- virtio_pmem_signal_done(req_data);
+ if (READ_ONCE(vpmem->broken))
+ virtio_pmem_complete_err(req_data);
+ else
+ virtio_pmem_signal_done(req_data);
kref_put(&req_data->kref, virtio_pmem_req_release);
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
@@ -83,6 +150,9 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
return -EIO;
}
+ if (READ_ONCE(vpmem->broken))
+ return -EIO;
+
req_data = kmalloc_obj(*req_data, GFP_NOIO);
if (!req_data)
return -ENOMEM;
@@ -99,13 +169,18 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
sgs[1] = &ret;
spin_lock_irqsave(&vpmem->pmem_lock, flags);
- /*
- * If virtqueue_add_sgs returns -ENOSPC then req_vq virtual
- * queue does not have free descriptor. We add the request
- * to req_list and wait for host_ack to wake us up when free
- * slots are available.
- */
+ /*
+ * If virtqueue_add_sgs returns -ENOSPC then req_vq virtual
+ * queue does not have free descriptor. We add the request
+ * to req_list and wait for host_ack to wake us up when free
+ * slots are available.
+ */
for (;;) {
+ if (READ_ONCE(vpmem->broken)) {
+ err = -EIO;
+ break;
+ }
+
err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data,
GFP_ATOMIC);
if (!err) {
@@ -114,6 +189,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
* held so completion cannot run concurrently.
*/
kref_get(&req_data->kref);
+ vpmem->req_inflight = req_data;
break;
}
@@ -127,24 +203,41 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
/* A host response results in "host_ack" getting called */
- wait_event(req_data->wq_buf, READ_ONCE(req_data->wq_buf_avail));
+ wait_event(req_data->wq_buf,
+ READ_ONCE(req_data->wq_buf_avail) ||
+ READ_ONCE(vpmem->broken));
spin_lock_irqsave(&vpmem->pmem_lock, flags);
+
+ if (READ_ONCE(vpmem->broken))
+ break;
}
- err1 = virtqueue_kick(vpmem->req_vq);
+ if (err == -EIO || virtqueue_is_broken(vpmem->req_vq))
+ virtio_pmem_mark_broken(vpmem);
+
+ err1 = true;
+ if (!err && !READ_ONCE(vpmem->broken)) {
+ err1 = virtqueue_kick(vpmem->req_vq);
+ if (!err1)
+ virtio_pmem_mark_broken(vpmem);
+ }
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
/*
* virtqueue_add_sgs failed with error different than -ENOSPC, we can't
* do anything about that.
*/
- if (err || !err1) {
+ if (READ_ONCE(vpmem->broken) || err || !err1) {
dev_info(&vdev->dev, "failed to send command to virtio pmem device\n");
err = -EIO;
} else {
/* A host response results in "host_ack" getting called */
wait_event(req_data->host_acked,
- virtio_pmem_req_done(req_data));
- err = le32_to_cpu(req_data->resp.ret);
+ virtio_pmem_req_done(req_data) ||
+ READ_ONCE(vpmem->broken));
+ if (virtio_pmem_req_done(req_data))
+ err = le32_to_cpu(req_data->resp.ret);
+ else
+ err = -EIO;
}
kref_put(&req_data->kref, virtio_pmem_req_release);
diff --git a/drivers/nvdimm/virtio_pmem.c b/drivers/nvdimm/virtio_pmem.c
index 77b1966619059..b272e9279ef23 100644
--- a/drivers/nvdimm/virtio_pmem.c
+++ b/drivers/nvdimm/virtio_pmem.c
@@ -25,6 +25,8 @@ static int init_vq(struct virtio_pmem *vpmem)
spin_lock_init(&vpmem->pmem_lock);
INIT_LIST_HEAD(&vpmem->req_list);
+ vpmem->req_inflight = NULL;
+ WRITE_ONCE(vpmem->broken, false);
return 0;
};
@@ -138,10 +140,21 @@ static int virtio_pmem_probe(struct virtio_device *vdev)
static void virtio_pmem_remove(struct virtio_device *vdev)
{
struct nvdimm_bus *nvdimm_bus = dev_get_drvdata(&vdev->dev);
+ struct virtio_pmem *vpmem = vdev->priv;
+ unsigned long flags;
+
+ spin_lock_irqsave(&vpmem->pmem_lock, flags);
+ virtio_pmem_mark_broken(vpmem);
+ spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
+
+ virtio_reset_device(vdev);
+
+ spin_lock_irqsave(&vpmem->pmem_lock, flags);
+ virtio_pmem_drain(vpmem);
+ spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
nvdimm_bus_unregister(nvdimm_bus);
vdev->config->del_vqs(vdev);
- virtio_reset_device(vdev);
}
static int virtio_pmem_freeze(struct virtio_device *vdev)
diff --git a/drivers/nvdimm/virtio_pmem.h b/drivers/nvdimm/virtio_pmem.h
index 23bff40249c1b..bc7de2b328985 100644
--- a/drivers/nvdimm/virtio_pmem.h
+++ b/drivers/nvdimm/virtio_pmem.h
@@ -52,6 +52,12 @@ struct virtio_pmem {
/* List to store deferred work if virtqueue is full */
struct list_head req_list;
+ /* Request currently owned by the virtqueue. */
+ struct virtio_pmem_request *req_inflight;
+
+ /* Fail fast and wake waiters if the request virtqueue is broken. */
+ bool broken;
+
/* Synchronize virtqueue data */
spinlock_t pmem_lock;
@@ -61,5 +67,7 @@ struct virtio_pmem {
};
void virtio_pmem_host_ack(struct virtqueue *vq);
+void virtio_pmem_mark_broken(struct virtio_pmem *vpmem);
+void virtio_pmem_drain(struct virtio_pmem *vpmem);
int async_pmem_flush(struct nd_region *nd_region, struct bio *bio);
#endif
--
2.52.0
^ permalink raw reply related
* [PATCH v6 12/12] nvdimm: virtio_pmem: drain requests in freeze
From: Li Chen @ 2026-06-21 13:02 UTC (permalink / raw)
To: Pankaj Gupta, Dan Williams, Vishal Verma, Dave Jiang, Ira Weiny,
Alison Schofield, virtualization, nvdimm
Cc: linux-kernel, Li Chen
In-Reply-To: <20260621130246.2973254-1-me@linux.beauty>
virtio_pmem_freeze() currently deletes virtqueues and resets the device
without waking threads waiting for a virtqueue descriptor or a host
completion.
Mark the request virtqueue broken before reset. This makes new submissions
fail fast and lets -ENOSPC waiters leave the wait list. Reset the device
before draining used and unused request tokens, then delete the virtqueues.
This wakes waiters with -EIO. It also keeps the detach call on a quiesced
device.
Clear req_vq after del_vqs(), and make drain tolerate a NULL queue, so
remove after freeze does not dereference a stale virtqueue pointer.
Signed-off-by: Li Chen <me@linux.beauty>
---
Changes in v6:
- Clear req_vq after del_vqs() and make drain tolerate a NULL queue.
Changes in v5:
- Reset the device before draining used and unused request tokens.
- Use the split broken-marking and post-reset drain helpers.
v2->v3:
- No change.
v3->v4:
- Rebased onto v7.1-rc7 and renumbered after the flush error patches.
drivers/nvdimm/nd_virtio.c | 3 +++
drivers/nvdimm/virtio_pmem.c | 36 +++++++++++++++++++++++++++++++-----
2 files changed, 34 insertions(+), 5 deletions(-)
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index fb9391ebc46e7..ce4032dc07628 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -93,6 +93,9 @@ void virtio_pmem_drain(struct virtio_pmem *vpmem)
struct virtio_pmem_request *req;
unsigned int len;
+ if (!vpmem->req_vq)
+ return;
+
while ((req = virtqueue_get_buf(vpmem->req_vq, &len)) != NULL) {
virtio_pmem_clear_inflight(vpmem, req);
virtio_pmem_complete_err(req);
diff --git a/drivers/nvdimm/virtio_pmem.c b/drivers/nvdimm/virtio_pmem.c
index b272e9279ef23..fef792f725db2 100644
--- a/drivers/nvdimm/virtio_pmem.c
+++ b/drivers/nvdimm/virtio_pmem.c
@@ -17,11 +17,16 @@ static struct virtio_device_id id_table[] = {
/* Initialize virt queue */
static int init_vq(struct virtio_pmem *vpmem)
{
+ int err;
+
/* single vq */
vpmem->req_vq = virtio_find_single_vq(vpmem->vdev,
virtio_pmem_host_ack, "flush_queue");
- if (IS_ERR(vpmem->req_vq))
- return PTR_ERR(vpmem->req_vq);
+ if (IS_ERR(vpmem->req_vq)) {
+ err = PTR_ERR(vpmem->req_vq);
+ vpmem->req_vq = NULL;
+ return err;
+ }
spin_lock_init(&vpmem->pmem_lock);
INIT_LIST_HEAD(&vpmem->req_list);
@@ -31,6 +36,15 @@ static int init_vq(struct virtio_pmem *vpmem)
return 0;
};
+static void virtio_pmem_del_vqs(struct virtio_pmem *vpmem)
+{
+ if (!vpmem->req_vq)
+ return;
+
+ vpmem->vdev->config->del_vqs(vpmem->vdev);
+ vpmem->req_vq = NULL;
+}
+
static int virtio_pmem_validate(struct virtio_device *vdev)
{
struct virtio_shm_region shm_reg;
@@ -132,7 +146,7 @@ static int virtio_pmem_probe(struct virtio_device *vdev)
virtio_reset_device(vdev);
nvdimm_bus_unregister(vpmem->nvdimm_bus);
out_vq:
- vdev->config->del_vqs(vdev);
+ virtio_pmem_del_vqs(vpmem);
out_err:
return err;
}
@@ -154,14 +168,26 @@ static void virtio_pmem_remove(struct virtio_device *vdev)
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
nvdimm_bus_unregister(nvdimm_bus);
- vdev->config->del_vqs(vdev);
+ virtio_pmem_del_vqs(vpmem);
}
static int virtio_pmem_freeze(struct virtio_device *vdev)
{
- vdev->config->del_vqs(vdev);
+ struct virtio_pmem *vpmem = vdev->priv;
+ unsigned long flags;
+
+ spin_lock_irqsave(&vpmem->pmem_lock, flags);
+ virtio_pmem_mark_broken(vpmem);
+ spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
+
virtio_reset_device(vdev);
+ spin_lock_irqsave(&vpmem->pmem_lock, flags);
+ virtio_pmem_drain(vpmem);
+ spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
+
+ virtio_pmem_del_vqs(vpmem);
+
return 0;
}
--
2.52.0
^ permalink raw reply related
* Re: [PATCH net-next v3] virtio-net: xsk: support tx wake up
From: Jakub Kicinski @ 2026-06-21 22:06 UTC (permalink / raw)
To: xuanzhuo
Cc: Menglong Dong, eperezma, mst, jasowang, andrew+netdev, davem,
edumazet, pabeni, netdev, virtualization, linux-kernel
In-Reply-To: <20260616115912.513183-1-dongml2@chinatelecom.cn>
On Tue, 16 Jun 2026 19:59:12 +0800 Menglong Dong wrote:
> For now, XDP_RING_NEED_WAKEUP is not supported properly by the virtio-net
> in the tx path for example: we set xsk_set_tx_need_wakeup() in
> virtnet_xsk_xmit(), but we didn't call xsk_clear_tx_need_wakeup()
> anywhere, which means the user will call send() for every packet.
>
> We call xsk_set_tx_need_wakeup() after virtnet_xsk_xmit_batch() if sq->vq
> is empty, as we can't be wakeup by the skb_xmit_done() in this case.
> Otherwise, we will clear the wakeup flag.
>
> Race condition is considered for tx path.
Seems to follow what mlx5 does so presumably this is fine but IDK if
there's anything virtio-specific that we need to be worried about.
Xuan Zhuo, please TAL?
--
mping: VIRTIO NET DRIVER
^ permalink raw reply
* Re: [PATCH net-next v3] virtio-net: xsk: support tx wake up
From: Michael S. Tsirkin @ 2026-06-21 22:31 UTC (permalink / raw)
To: Menglong Dong
Cc: xuanzhuo, eperezma, jasowang, andrew+netdev, davem, edumazet,
kuba, pabeni, netdev, virtualization, linux-kernel
In-Reply-To: <20260616115912.513183-1-dongml2@chinatelecom.cn>
On Tue, Jun 16, 2026 at 07:59:12PM +0800, Menglong Dong wrote:
> For now, XDP_RING_NEED_WAKEUP is not supported properly by the virtio-net
> in the tx path for example: we set xsk_set_tx_need_wakeup() in
> virtnet_xsk_xmit(), but we didn't call xsk_clear_tx_need_wakeup()
> anywhere, which means the user will call send() for every packet.
>
> We call xsk_set_tx_need_wakeup() after virtnet_xsk_xmit_batch() if sq->vq
> is empty, as we can't be wakeup by the skb_xmit_done() in this case.
> Otherwise, we will clear the wakeup flag.
>
> Race condition is considered for tx path.
>
> Fixes: 89f86675cb03 ("virtio_net: xsk: tx: support xmit xsk buffer")
> Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
thanks for the patch! yes something to improve.
> ---
> v3:
> - remove the confusing comment
>
> v2:
> - add the Fixes tag
> ---
> drivers/net/virtio_net.c | 23 +++++++++++++++++++----
> 1 file changed, 19 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index f4adcfee7a80..6e099edef6e9 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1440,8 +1440,9 @@ static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool,
> struct virtnet_info *vi = sq->vq->vdev->priv;
> struct virtnet_sq_free_stats stats = {};
> struct net_device *dev = vi->dev;
> + int sent, vring_size;
> + bool need_wakeup;
> u64 kicks = 0;
> - int sent;
>
> /* Avoid to wakeup napi meanless, so call __free_old_xmit instead of
> * free_old_xmit().
> @@ -1451,8 +1452,25 @@ static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool,
> if (stats.xsk)
> xsk_tx_completed(sq->xsk_pool, stats.xsk);
>
> + vring_size = virtqueue_get_vring_size(sq->vq);
> + need_wakeup = xsk_uses_need_wakeup(pool);
> +
> + if (need_wakeup && vring_size == sq->vq->num_free)
> + xsk_set_tx_need_wakeup(pool);
> +
why are we doing this here?
the check after virtnet_xsk_xmit_batch not enough?
I vaguely think it's some kind of race we are closing?
Pls add a comment to explain.
> sent = virtnet_xsk_xmit_batch(sq, pool, budget, &kicks);
>
> + if (need_wakeup) {
> + if (vring_size == sq->vq->num_free)
> + /* we can't wake up by ourself, and it should be done
> + * by the user.
> + */
> + xsk_set_tx_need_wakeup(pool);
> + else
> + /* we can wake up from skb_xmit_done() */
> + xsk_clear_tx_need_wakeup(pool);
But what if we don't have get tx napi so no wakeup in skb_xmit_done?
> + }
> +
> if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
> check_sq_full_and_disable(vi, vi->dev, sq);
>
> @@ -1470,9 +1488,6 @@ static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool,
> u64_stats_add(&sq->stats.xdp_tx, sent);
> u64_stats_update_end(&sq->stats.syncp);
>
> - if (xsk_uses_need_wakeup(pool))
> - xsk_set_tx_need_wakeup(pool);
> -
> return sent;
> }
>
> --
> 2.54.0
^ permalink raw reply
* Re: [PATCH net-next v3] virtio-net: xsk: support tx wake up
From: Xuan Zhuo @ 2026-06-22 2:40 UTC (permalink / raw)
To: Menglong Dong
Cc: mst, jasowang, andrew+netdev, davem, edumazet, kuba, pabeni,
netdev, virtualization, linux-kernel, eperezma
In-Reply-To: <20260616115912.513183-1-dongml2@chinatelecom.cn>
On Tue, 16 Jun 2026 19:59:12 +0800, Menglong Dong <menglong8.dong@gmail.com> wrote:
> For now, XDP_RING_NEED_WAKEUP is not supported properly by the virtio-net
> in the tx path for example: we set xsk_set_tx_need_wakeup() in
> virtnet_xsk_xmit(), but we didn't call xsk_clear_tx_need_wakeup()
> anywhere, which means the user will call send() for every packet.
>
> We call xsk_set_tx_need_wakeup() after virtnet_xsk_xmit_batch() if sq->vq
> is empty, as we can't be wakeup by the skb_xmit_done() in this case.
> Otherwise, we will clear the wakeup flag.
>
> Race condition is considered for tx path.
>
> Fixes: 89f86675cb03 ("virtio_net: xsk: tx: support xmit xsk buffer")
This is not a bug, so we do not need this.
And you post this to net-next.
> Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn>
> ---
> v3:
> - remove the confusing comment
>
> v2:
> - add the Fixes tag
> ---
> drivers/net/virtio_net.c | 23 +++++++++++++++++++----
> 1 file changed, 19 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index f4adcfee7a80..6e099edef6e9 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1440,8 +1440,9 @@ static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool,
> struct virtnet_info *vi = sq->vq->vdev->priv;
> struct virtnet_sq_free_stats stats = {};
> struct net_device *dev = vi->dev;
> + int sent, vring_size;
> + bool need_wakeup;
> u64 kicks = 0;
> - int sent;
>
> /* Avoid to wakeup napi meanless, so call __free_old_xmit instead of
> * free_old_xmit().
> @@ -1451,8 +1452,25 @@ static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool,
> if (stats.xsk)
> xsk_tx_completed(sq->xsk_pool, stats.xsk);
>
> + vring_size = virtqueue_get_vring_size(sq->vq);
> + need_wakeup = xsk_uses_need_wakeup(pool);
> +
> + if (need_wakeup && vring_size == sq->vq->num_free)
> + xsk_set_tx_need_wakeup(pool);
You need to comment this.
> +
> sent = virtnet_xsk_xmit_batch(sq, pool, budget, &kicks);
>
> + if (need_wakeup) {
> + if (vring_size == sq->vq->num_free)
> + /* we can't wake up by ourself, and it should be done
> + * by the user.
> + */
> + xsk_set_tx_need_wakeup(pool);
> + else
> + /* we can wake up from skb_xmit_done() */
> + xsk_clear_tx_need_wakeup(pool);
> + }
> +
> if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
> check_sq_full_and_disable(vi, vi->dev, sq);
After fixed above comments, you can add:
Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Thanks.
>
> @@ -1470,9 +1488,6 @@ static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool,
> u64_stats_add(&sq->stats.xdp_tx, sent);
> u64_stats_update_end(&sq->stats.syncp);
>
> - if (xsk_uses_need_wakeup(pool))
> - xsk_set_tx_need_wakeup(pool);
> -
> return sent;
> }
>
> --
> 2.54.0
>
^ permalink raw reply
* [PATCH v3 0/7] Prepare mutable list iterators to cache cursor state
From: Kaitao Cheng @ 2026-06-22 4:05 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Jens Axboe, Tejun Heo,
Alexander Viro, Christian Brauner, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Johannes Weiner, Peter Zijlstra,
Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Juri Lelli, Vincent Guittot, Paul Moore,
Andy Shevchenko, Paul E. McKenney, Shakeel Butt,
Christian König
Cc: David Howells, Simona Vetter, Randy Dunlap, Luca Ceresoli,
Philipp Stanner, linux-block, linux-kernel, cgroups,
linux-ntfs-dev, linux-fsdevel, io-uring, audit, bpf, netdev,
dri-devel, linux-perf-users, linux-trace-kernel, kexec,
live-patching, linux-modules, linux-crypto, linux-pm, rcu,
sched-ext, linux-mm, virtualization, damon, llvm, chengkaitao
From: chengkaitao <chengkaitao@kylinos.cn>
The list_for_each*_safe() helpers are used when the loop body may remove
the current entry. Their current interface, however, forces every caller
to define a temporary cursor outside the macro and pass it in, even when
the caller never uses that cursor directly. For most call sites this
extra cursor is just boilerplate required by the macro implementation.
This is awkward because the saved next pointer is an internal detail of
the iteration. Callers that only remove or move the current entry do not
need to spell it out.
The _safe() suffix has also caused confusion. Christian Koenig pointed
out that the name is easy to read as a thread-safe variant, especially
for beginners, even though it only means that the iterator keeps enough
state to tolerate removal of the current entry. He suggested _mutable()
as a clearer description of what the loop permits.
Add *_mutable() iterator variants for list, hlist and llist. The new
helpers are variadic and support both forms. In the common case, the
caller omits the temporary cursor and the macro creates a unique internal
cursor with typeof(pos) and __UNIQUE_ID(). If a loop really needs an
explicit temporary cursor, the caller can still pass it and the helper
keeps the existing *_safe() behaviour.
For example, a call site may use the shorter form:
list_for_each_entry_mutable(pos, head, member)
or keep the explicit temporary cursor form:
list_for_each_entry_mutable(pos, tmp, head, member)
The existing *_safe() helpers remain available for compatibility. This
series only converts users in mm, block, kernel, init and io_uring. If
this approach looks acceptable, the remaining users can be converted in
follow-up series.
Changes in v3 (Christian König, Andy Shevchenko):
- Convert safe list walks to mutable iterators
Changes in v2 (Muchun Song, Andy Shevchenko):
- Drop the list_for_each_entry_mutable*() helpers from v1 and make the
cursor change directly in the existing list_for_each_entry*() helpers.
- Open-code special list walks that rely on updating the loop cursor in
the body, preserving their existing traversal semantics.
Link to v2:
https://lore.kernel.org/all/20260609061347.93688-1-kaitao.cheng@linux.dev/
Link to v1:
https://lore.kernel.org/all/20260529082149.76764-1-kaitao.cheng@linux.dev/
Kaitao Cheng (7):
list: Add mutable iterator variants
llist: Add mutable iterator variants
mm: Use mutable list iterators
block: Use mutable list iterators
kernel: Use mutable list iterators
initramfs: Use mutable list iterator
io_uring: Use mutable list iterators
block/bfq-iosched.c | 17 +-
block/blk-cgroup.c | 12 +-
block/blk-flush.c | 4 +-
block/blk-iocost.c | 18 +-
block/blk-mq.c | 8 +-
block/blk-throttle.c | 4 +-
block/kyber-iosched.c | 4 +-
block/partitions/ldm.c | 8 +-
block/sed-opal.c | 4 +-
include/linux/list.h | 269 ++++++++++++++++++++++++----
include/linux/llist.h | 81 +++++++--
init/initramfs.c | 5 +-
io_uring/cancel.c | 6 +-
io_uring/poll.c | 3 +-
io_uring/rw.c | 4 +-
io_uring/timeout.c | 8 +-
io_uring/uring_cmd.c | 3 +-
kernel/audit_tree.c | 4 +-
kernel/audit_watch.c | 16 +-
kernel/auditfilter.c | 4 +-
kernel/auditsc.c | 4 +-
kernel/bpf/arena.c | 10 +-
kernel/bpf/arraymap.c | 8 +-
kernel/bpf/bpf_local_storage.c | 3 +-
kernel/bpf/bpf_lru_list.c | 25 ++-
kernel/bpf/btf.c | 18 +-
kernel/bpf/cgroup.c | 7 +-
kernel/bpf/cpumap.c | 4 +-
kernel/bpf/devmap.c | 10 +-
kernel/bpf/helpers.c | 8 +-
kernel/bpf/local_storage.c | 4 +-
kernel/bpf/memalloc.c | 16 +-
kernel/bpf/offload.c | 8 +-
kernel/bpf/states.c | 4 +-
kernel/bpf/stream.c | 4 +-
kernel/bpf/verifier.c | 6 +-
kernel/cgroup/cgroup-v1.c | 4 +-
kernel/cgroup/cgroup.c | 54 +++---
kernel/cgroup/dmem.c | 12 +-
kernel/cgroup/rdma.c | 8 +-
kernel/events/core.c | 44 +++--
kernel/events/uprobes.c | 12 +-
kernel/exit.c | 8 +-
kernel/fail_function.c | 4 +-
kernel/gcov/clang.c | 4 +-
kernel/irq_work.c | 4 +-
kernel/kexec_core.c | 4 +-
kernel/kprobes.c | 16 +-
kernel/livepatch/core.c | 4 +-
kernel/livepatch/core.h | 4 +-
kernel/liveupdate/kho_block.c | 4 +-
kernel/liveupdate/luo_flb.c | 4 +-
kernel/locking/rwsem.c | 2 +-
kernel/locking/test-ww_mutex.c | 2 +-
kernel/module/main.c | 11 +-
kernel/padata.c | 4 +-
kernel/power/snapshot.c | 8 +-
kernel/power/wakelock.c | 4 +-
kernel/printk/printk.c | 11 +-
kernel/ptrace.c | 4 +-
kernel/rcu/rcutorture.c | 3 +-
kernel/rcu/tasks.h | 9 +-
kernel/rcu/tree.c | 6 +-
kernel/resource.c | 4 +-
kernel/sched/core.c | 4 +-
kernel/sched/ext.c | 22 +--
kernel/sched/fair.c | 28 +--
kernel/sched/topology.c | 4 +-
kernel/sched/wait.c | 4 +-
kernel/seccomp.c | 4 +-
kernel/signal.c | 11 +-
kernel/smp.c | 4 +-
kernel/taskstats.c | 8 +-
kernel/time/clockevents.c | 6 +-
kernel/time/clocksource.c | 4 +-
kernel/time/posix-cpu-timers.c | 4 +-
kernel/time/posix-timers.c | 3 +-
kernel/torture.c | 3 +-
kernel/trace/bpf_trace.c | 4 +-
kernel/trace/ftrace.c | 49 +++--
kernel/trace/ring_buffer.c | 25 ++-
kernel/trace/trace.c | 12 +-
kernel/trace/trace_dynevent.c | 6 +-
kernel/trace/trace_dynevent.h | 5 +-
kernel/trace/trace_events.c | 35 ++--
kernel/trace/trace_events_filter.c | 4 +-
kernel/trace/trace_events_hist.c | 8 +-
kernel/trace/trace_events_trigger.c | 17 +-
kernel/trace/trace_events_user.c | 16 +-
kernel/trace/trace_stat.c | 4 +-
kernel/user-return-notifier.c | 3 +-
kernel/workqueue.c | 16 +-
mm/backing-dev.c | 8 +-
mm/balloon.c | 8 +-
mm/cma.c | 4 +-
mm/compaction.c | 4 +-
mm/damon/core.c | 4 +-
mm/damon/sysfs-schemes.c | 4 +-
mm/dmapool.c | 4 +-
mm/huge_memory.c | 8 +-
mm/hugetlb.c | 56 +++---
mm/hugetlb_vmemmap.c | 16 +-
mm/khugepaged.c | 14 +-
mm/kmemleak.c | 7 +-
mm/ksm.c | 25 +--
mm/list_lru.c | 4 +-
mm/memcontrol-v1.c | 8 +-
mm/memory-failure.c | 12 +-
mm/memory-tiers.c | 4 +-
mm/migrate.c | 23 ++-
mm/mmu_notifier.c | 9 +-
mm/page_alloc.c | 8 +-
mm/page_reporting.c | 2 +-
mm/percpu.c | 11 +-
mm/pgtable-generic.c | 4 +-
mm/rmap.c | 10 +-
mm/shmem.c | 9 +-
mm/slab_common.c | 14 +-
mm/slub.c | 33 ++--
mm/swapfile.c | 4 +-
mm/userfaultfd.c | 12 +-
mm/vmalloc.c | 24 +--
mm/vmscan.c | 7 +-
mm/zsmalloc.c | 4 +-
124 files changed, 875 insertions(+), 681 deletions(-)
--
2.43.0
^ permalink raw reply
* [PATCH v3 1/7] list: Add mutable iterator variants
From: Kaitao Cheng @ 2026-06-22 4:05 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Jens Axboe, Tejun Heo,
Alexander Viro, Christian Brauner, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Johannes Weiner, Peter Zijlstra,
Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Juri Lelli, Vincent Guittot, Paul Moore,
Andy Shevchenko, Paul E. McKenney, Shakeel Butt,
Christian König
Cc: David Howells, Simona Vetter, Randy Dunlap, Luca Ceresoli,
Philipp Stanner, linux-block, linux-kernel, cgroups,
linux-ntfs-dev, linux-fsdevel, io-uring, audit, bpf, netdev,
dri-devel, linux-perf-users, linux-trace-kernel, kexec,
live-patching, linux-modules, linux-crypto, linux-pm, rcu,
sched-ext, linux-mm, virtualization, damon, llvm, Kaitao Cheng
In-Reply-To: <20260622040533.29824-1-kaitao.cheng@linux.dev>
From: Kaitao Cheng <chengkaitao@kylinos.cn>
The list_for_each*_safe() helpers are used when the loop body may
remove the current entry. Their API exposes the temporary cursor at
every call site, even though most users only need it for the iterator
implementation and never reference it in the loop body.
Add *_mutable() variants for list and hlist iteration. The new helpers
support both forms: callers may keep passing an explicit temporary cursor
when they need to inspect or reset it, or omit it and let the helper use
a unique internal cursor.
This makes call sites that only mutate the list through the current entry
less noisy, while keeping the existing *_safe() helpers available for
compatibility.
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
---
include/linux/list.h | 269 +++++++++++++++++++++++++++++++++++++------
1 file changed, 231 insertions(+), 38 deletions(-)
diff --git a/include/linux/list.h b/include/linux/list.h
index 09d979976b3b..1081def7cea9 100644
--- a/include/linux/list.h
+++ b/include/linux/list.h
@@ -7,6 +7,7 @@
#include <linux/stddef.h>
#include <linux/poison.h>
#include <linux/const.h>
+#include <linux/args.h>
#include <asm/barrier.h>
@@ -763,28 +764,72 @@ static inline void list_splice_tail_init(struct list_head *list,
#define list_for_each_prev(pos, head) \
for (pos = (head)->prev; !list_is_head(pos, (head)); pos = pos->prev)
-/**
- * list_for_each_safe - iterate over a list safe against removal of list entry
- * @pos: the &struct list_head to use as a loop cursor.
- * @n: another &struct list_head to use as temporary storage
- * @head: the head for your list.
+/*
+ * list_for_each_safe is an old interface, use list_for_each_mutable instead.
*/
#define list_for_each_safe(pos, n, head) \
for (pos = (head)->next, n = pos->next; \
!list_is_head(pos, (head)); \
pos = n, n = pos->next)
+#define __list_for_each_mutable_internal(pos, tmp, head) \
+ for (typeof(pos) tmp = (pos = (head)->next)->next; \
+ !list_is_head(pos, (head)); \
+ pos = tmp, tmp = pos->next)
+
+#define __list_for_each_mutable1(pos, head) \
+ __list_for_each_mutable_internal(pos, __UNIQUE_ID(next), head)
+
+#define __list_for_each_mutable2(pos, next, head) \
+ list_for_each_safe(pos, next, head)
+
/**
- * list_for_each_prev_safe - iterate over a list backwards safe against removal of list entry
+ * list_for_each_mutable - iterate over a list safe against entry removal
* @pos: the &struct list_head to use as a loop cursor.
- * @n: another &struct list_head to use as temporary storage
- * @head: the head for your list.
+ * @...: either (head) or (next, head)
+ *
+ * next: another &struct list_head to use as optional temporary storage.
+ * The temporary cursor is internal unless explicitly supplied by
+ * the caller.
+ * head: the head for your list.
+ */
+#define list_for_each_mutable(pos, ...) \
+ CONCATENATE(__list_for_each_mutable, COUNT_ARGS(__VA_ARGS__)) \
+ (pos, __VA_ARGS__)
+
+/*
+ * list_for_each_prev_safe is an old interface, use list_for_each_prev_mutable instead.
*/
#define list_for_each_prev_safe(pos, n, head) \
for (pos = (head)->prev, n = pos->prev; \
!list_is_head(pos, (head)); \
pos = n, n = pos->prev)
+#define __list_for_each_prev_mutable_internal(pos, tmp, head) \
+ for (typeof(pos) tmp = (pos = (head)->prev)->prev; \
+ !list_is_head(pos, (head)); \
+ pos = tmp, tmp = pos->prev)
+
+#define __list_for_each_prev_mutable1(pos, head) \
+ __list_for_each_prev_mutable_internal(pos, __UNIQUE_ID(prev), head)
+
+#define __list_for_each_prev_mutable2(pos, prev, head) \
+ list_for_each_prev_safe(pos, prev, head)
+
+/**
+ * list_for_each_prev_mutable - iterate over a list backwards safe against entry removal
+ * @pos: the &struct list_head to use as a loop cursor.
+ * @...: either (head) or (prev, head)
+ *
+ * prev: another &struct list_head to use as optional temporary storage.
+ * The temporary cursor is internal unless explicitly supplied by
+ * the caller.
+ * head: the head for your list.
+ */
+#define list_for_each_prev_mutable(pos, ...) \
+ CONCATENATE(__list_for_each_prev_mutable, COUNT_ARGS(__VA_ARGS__)) \
+ (pos, __VA_ARGS__)
+
/**
* list_count_nodes - count nodes in the list
* @head: the head for your list.
@@ -895,12 +940,8 @@ static inline size_t list_count_nodes(struct list_head *head)
for (; !list_entry_is_head(pos, head, member); \
pos = list_prev_entry(pos, member))
-/**
- * list_for_each_entry_safe - iterate over list of given type safe against removal of list entry
- * @pos: the type * to use as a loop cursor.
- * @n: another type * to use as temporary storage
- * @head: the head for your list.
- * @member: the name of the list_head within the struct.
+/*
+ * list_for_each_entry_safe is an old interface, use list_for_each_entry_mutable instead.
*/
#define list_for_each_entry_safe(pos, n, head, member) \
for (pos = list_first_entry(head, typeof(*pos), member), \
@@ -908,15 +949,36 @@ static inline size_t list_count_nodes(struct list_head *head)
!list_entry_is_head(pos, head, member); \
pos = n, n = list_next_entry(n, member))
+#define __list_for_each_entry_mutable_internal(pos, tmp, head, member) \
+ for (typeof(pos) tmp = list_next_entry(pos = \
+ list_first_entry(head, typeof(*pos), member), member); \
+ !list_entry_is_head(pos, head, member); \
+ pos = tmp, tmp = list_next_entry(tmp, member))
+
+#define __list_for_each_entry_mutable2(pos, head, member) \
+ __list_for_each_entry_mutable_internal(pos, __UNIQUE_ID(next), head, member)
+
+#define __list_for_each_entry_mutable3(pos, next, head, member) \
+ list_for_each_entry_safe(pos, next, head, member)
+
/**
- * list_for_each_entry_safe_continue - continue list iteration safe against removal
+ * list_for_each_entry_mutable - iterate over a list safe against entry removal
* @pos: the type * to use as a loop cursor.
- * @n: another type * to use as temporary storage
- * @head: the head for your list.
- * @member: the name of the list_head within the struct.
+ * @...: either (head, member) or (next, head, member)
*
- * Iterate over list of given type, continuing after current point,
- * safe against removal of list entry.
+ * next: another type * to use as optional temporary storage. The
+ * temporary cursor is internal unless explicitly supplied by the
+ * caller.
+ * head: the head for your list.
+ * member: the name of the list_head within the struct.
+ */
+#define list_for_each_entry_mutable(pos, ...) \
+ CONCATENATE(__list_for_each_entry_mutable, COUNT_ARGS(__VA_ARGS__)) \
+ (pos, __VA_ARGS__)
+
+/*
+ * list_for_each_entry_safe_continue is an old interface,
+ * use list_for_each_entry_mutable_continue instead.
*/
#define list_for_each_entry_safe_continue(pos, n, head, member) \
for (pos = list_next_entry(pos, member), \
@@ -924,30 +986,79 @@ static inline size_t list_count_nodes(struct list_head *head)
!list_entry_is_head(pos, head, member); \
pos = n, n = list_next_entry(n, member))
+#define __list_for_each_entry_mutable_continue_internal(pos, tmp, head, member) \
+ for (typeof(pos) tmp = list_next_entry(pos = \
+ list_next_entry(pos, member), member); \
+ !list_entry_is_head(pos, head, member); \
+ pos = tmp, tmp = list_next_entry(tmp, member))
+
+#define __list_for_each_entry_mutable_continue2(pos, head, member) \
+ __list_for_each_entry_mutable_continue_internal(pos, \
+ __UNIQUE_ID(next), head, member)
+
+#define __list_for_each_entry_mutable_continue3(pos, next, head, member) \
+ list_for_each_entry_safe_continue(pos, next, head, member)
+
/**
- * list_for_each_entry_safe_from - iterate over list from current point safe against removal
+ * list_for_each_entry_mutable_continue - continue list iteration safe against removal
* @pos: the type * to use as a loop cursor.
- * @n: another type * to use as temporary storage
- * @head: the head for your list.
- * @member: the name of the list_head within the struct.
+ * @...: either (head, member) or (next, head, member)
*
- * Iterate over list of given type from current point, safe against
- * removal of list entry.
+ * next: another type * to use as optional temporary storage. The
+ * temporary cursor is internal unless explicitly supplied by the
+ * caller.
+ * head: the head for your list.
+ * member: the name of the list_head within the struct.
+ *
+ * Iterate over list of given type, continuing after current point,
+ * safe against removal of list entry.
+ */
+#define list_for_each_entry_mutable_continue(pos, ...) \
+ CONCATENATE(__list_for_each_entry_mutable_continue, \
+ COUNT_ARGS(__VA_ARGS__))(pos, __VA_ARGS__)
+
+/*
+ * list_for_each_entry_safe_from is an old interface,
+ * use list_for_each_entry_mutable_from instead.
*/
#define list_for_each_entry_safe_from(pos, n, head, member) \
for (n = list_next_entry(pos, member); \
!list_entry_is_head(pos, head, member); \
pos = n, n = list_next_entry(n, member))
+#define __list_for_each_entry_mutable_from_internal(pos, tmp, head, member) \
+ for (typeof(pos) tmp = list_next_entry(pos, member); \
+ !list_entry_is_head(pos, head, member); \
+ pos = tmp, tmp = list_next_entry(tmp, member))
+
+#define __list_for_each_entry_mutable_from2(pos, head, member) \
+ __list_for_each_entry_mutable_from_internal(pos, \
+ __UNIQUE_ID(next), head, member)
+
+#define __list_for_each_entry_mutable_from3(pos, next, head, member) \
+ list_for_each_entry_safe_from(pos, next, head, member)
+
/**
- * list_for_each_entry_safe_reverse - iterate backwards over list safe against removal
+ * list_for_each_entry_mutable_from - iterate over list from current point safe against removal
* @pos: the type * to use as a loop cursor.
- * @n: another type * to use as temporary storage
- * @head: the head for your list.
- * @member: the name of the list_head within the struct.
+ * @...: either (head, member) or (next, head, member)
*
- * Iterate backwards over list of given type, safe against removal
- * of list entry.
+ * next: another type * to use as optional temporary storage. The
+ * temporary cursor is internal unless explicitly supplied by the
+ * caller.
+ * head: the head for your list.
+ * member: the name of the list_head within the struct.
+ *
+ * Iterate over list of given type from current point, safe against
+ * removal of list entry.
+ */
+#define list_for_each_entry_mutable_from(pos, ...) \
+ CONCATENATE(__list_for_each_entry_mutable_from, \
+ COUNT_ARGS(__VA_ARGS__))(pos, __VA_ARGS__)
+
+/*
+ * list_for_each_entry_safe_reverse is an old interface,
+ * use list_for_each_entry_mutable_reverse instead.
*/
#define list_for_each_entry_safe_reverse(pos, n, head, member) \
for (pos = list_last_entry(head, typeof(*pos), member), \
@@ -955,6 +1066,37 @@ static inline size_t list_count_nodes(struct list_head *head)
!list_entry_is_head(pos, head, member); \
pos = n, n = list_prev_entry(n, member))
+#define __list_for_each_entry_mutable_reverse_internal(pos, tmp, head, member) \
+ for (typeof(pos) tmp = list_prev_entry(pos = \
+ list_last_entry(head, typeof(*pos), member), member); \
+ !list_entry_is_head(pos, head, member); \
+ pos = tmp, tmp = list_prev_entry(tmp, member))
+
+#define __list_for_each_entry_mutable_reverse2(pos, head, member) \
+ __list_for_each_entry_mutable_reverse_internal(pos, \
+ __UNIQUE_ID(prev), head, member)
+
+#define __list_for_each_entry_mutable_reverse3(pos, prev, head, member) \
+ list_for_each_entry_safe_reverse(pos, prev, head, member)
+
+/**
+ * list_for_each_entry_mutable_reverse - iterate backwards over list safe against removal
+ * @pos: the type * to use as a loop cursor.
+ * @...: either (head, member) or (prev, head, member)
+ *
+ * prev: another type * to use as optional temporary storage. The
+ * temporary cursor is internal unless explicitly supplied by the
+ * caller.
+ * head: the head for your list.
+ * member: the name of the list_head within the struct.
+ *
+ * Iterate backwards over list of given type, safe against removal
+ * of list entry.
+ */
+#define list_for_each_entry_mutable_reverse(pos, ...) \
+ CONCATENATE(__list_for_each_entry_mutable_reverse, \
+ COUNT_ARGS(__VA_ARGS__))(pos, __VA_ARGS__)
+
/**
* list_safe_reset_next - reset a stale list_for_each_entry_safe loop
* @pos: the loop cursor used in the list_for_each_entry_safe loop
@@ -1189,6 +1331,31 @@ static inline void hlist_splice_init(struct hlist_head *from,
for (pos = (head)->first; pos && ({ n = pos->next; 1; }); \
pos = n)
+#define __hlist_for_each_mutable_internal(pos, tmp, head) \
+ for (typeof(pos) tmp = (pos = (head)->first) ? pos->next : NULL; \
+ pos; \
+ pos = tmp, tmp = pos ? pos->next : NULL)
+
+#define __hlist_for_each_mutable1(pos, head) \
+ __hlist_for_each_mutable_internal(pos, __UNIQUE_ID(next), head)
+
+#define __hlist_for_each_mutable2(pos, next, head) \
+ hlist_for_each_safe(pos, next, head)
+
+/**
+ * hlist_for_each_mutable - iterate over a hlist safe against entry removal
+ * @pos: the &struct hlist_node to use as a loop cursor.
+ * @...: either (head) or (next, head)
+ *
+ * next: another &struct hlist_node to use as optional temporary storage.
+ * The temporary cursor is internal unless explicitly supplied by
+ * the caller.
+ * head: the head for your hlist.
+ */
+#define hlist_for_each_mutable(pos, ...) \
+ CONCATENATE(__hlist_for_each_mutable, COUNT_ARGS(__VA_ARGS__)) \
+ (pos, __VA_ARGS__)
+
#define hlist_entry_safe(ptr, type, member) \
({ typeof(ptr) ____ptr = (ptr); \
____ptr ? hlist_entry(____ptr, type, member) : NULL; \
@@ -1224,18 +1391,44 @@ static inline void hlist_splice_init(struct hlist_head *from,
for (; pos; \
pos = hlist_entry_safe((pos)->member.next, typeof(*(pos)), member))
-/**
- * hlist_for_each_entry_safe - iterate over list of given type safe against removal of list entry
- * @pos: the type * to use as a loop cursor.
- * @n: a &struct hlist_node to use as temporary storage
- * @head: the head for your list.
- * @member: the name of the hlist_node within the struct.
+/*
+ * hlist_for_each_entry_safe is an old interface, use hlist_for_each_entry_mutable instead.
*/
#define hlist_for_each_entry_safe(pos, n, head, member) \
for (pos = hlist_entry_safe((head)->first, typeof(*pos), member);\
pos && ({ n = pos->member.next; 1; }); \
pos = hlist_entry_safe(n, typeof(*pos), member))
+#define __hlist_for_each_entry_mutable_internal(pos, tmp, head, member) \
+ for (struct hlist_node *tmp = (pos = \
+ hlist_entry_safe((head)->first, typeof(*pos), member)) ? \
+ pos->member.next : NULL; \
+ pos; \
+ pos = hlist_entry_safe((tmp), typeof(*pos), member), \
+ tmp = pos ? pos->member.next : NULL)
+
+#define __hlist_for_each_entry_mutable2(pos, head, member) \
+ __hlist_for_each_entry_mutable_internal(pos, \
+ __UNIQUE_ID(next), head, member)
+
+#define __hlist_for_each_entry_mutable3(pos, next, head, member) \
+ hlist_for_each_entry_safe(pos, next, head, member)
+
+/**
+ * hlist_for_each_entry_mutable - iterate over hlist safe against entry removal
+ * @pos: the type * to use as a loop cursor.
+ * @...: either (head, member) or (next, head, member)
+ *
+ * next: a &struct hlist_node to use as optional temporary storage. The
+ * temporary cursor is internal unless explicitly supplied by the
+ * caller.
+ * head: the head for your hlist.
+ * member: the name of the hlist_node within the struct.
+ */
+#define hlist_for_each_entry_mutable(pos, ...) \
+ CONCATENATE(__hlist_for_each_entry_mutable, \
+ COUNT_ARGS(__VA_ARGS__))(pos, __VA_ARGS__)
+
/**
* hlist_count_nodes - count nodes in the hlist
* @head: the head for your hlist.
--
2.43.0
^ permalink raw reply related
* [PATCH v3 2/7] llist: Add mutable iterator variants
From: Kaitao Cheng @ 2026-06-22 4:05 UTC (permalink / raw)
To: Andrew Morton, David Hildenbrand, Jens Axboe, Tejun Heo,
Alexander Viro, Christian Brauner, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Johannes Weiner, Peter Zijlstra,
Ingo Molnar, Arnaldo Carvalho de Melo, Namhyung Kim,
Thomas Gleixner, Juri Lelli, Vincent Guittot, Paul Moore,
Andy Shevchenko, Paul E. McKenney, Shakeel Butt,
Christian König
Cc: David Howells, Simona Vetter, Randy Dunlap, Luca Ceresoli,
Philipp Stanner, linux-block, linux-kernel, cgroups,
linux-ntfs-dev, linux-fsdevel, io-uring, audit, bpf, netdev,
dri-devel, linux-perf-users, linux-trace-kernel, kexec,
live-patching, linux-modules, linux-crypto, linux-pm, rcu,
sched-ext, linux-mm, virtualization, damon, llvm, Kaitao Cheng
In-Reply-To: <20260622040533.29824-1-kaitao.cheng@linux.dev>
From: Kaitao Cheng <chengkaitao@kylinos.cn>
llist_for_each_safe() and llist_for_each_entry_safe() require callers to
provide a temporary cursor even when the cursor is only needed by the
iterator itself. This makes call sites noisier than necessary for the
common case where the loop body may remove the current entry but does
not otherwise inspect the saved next pointer.
Add llist_for_each_mutable() and llist_for_each_entry_mutable() variants
that support both forms. Callers may omit the temporary cursor and let
the helper create an internal unique cursor, or keep passing an explicit
cursor when the loop needs to inspect or reset it.
Keep the existing safe helpers as compatibility wrappers so current users
continue to build unchanged while new code can use the shorter mutable
form.
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
---
include/linux/llist.h | 81 ++++++++++++++++++++++++++++++++++---------
1 file changed, 65 insertions(+), 16 deletions(-)
diff --git a/include/linux/llist.h b/include/linux/llist.h
index 8846b7709669..1c6f12411d5e 100644
--- a/include/linux/llist.h
+++ b/include/linux/llist.h
@@ -49,6 +49,7 @@
*/
#include <linux/atomic.h>
+#include <linux/args.h>
#include <linux/container_of.h>
#include <linux/stddef.h>
#include <linux/types.h>
@@ -143,12 +144,33 @@ static inline bool llist_on_list(const struct llist_node *node)
#define llist_for_each(pos, node) \
for ((pos) = (node); pos; (pos) = (pos)->next)
+/*
+ * llist_for_each_safe is an old interface, use llist_for_each_mutable instead.
+ */
+#define llist_for_each_safe(pos, n, node) \
+ for ((pos) = (node); (pos) && ((n) = (pos)->next, true); (pos) = (n))
+
+#define __llist_for_each_mutable_internal(pos, tmp, node) \
+ for (typeof(pos) tmp = ((pos) = (node)) ? (pos)->next : NULL; \
+ (pos); \
+ (pos) = tmp, tmp = (pos) ? (pos)->next : NULL)
+
+#define __llist_for_each_mutable1(pos, node) \
+ __llist_for_each_mutable_internal(pos, __UNIQUE_ID(next), node)
+
+#define __llist_for_each_mutable2(pos, next, node) \
+ llist_for_each_safe(pos, next, node)
+
/**
- * llist_for_each_safe - iterate over some deleted entries of a lock-less list
- * safe against removal of list entry
+ * llist_for_each_mutable - iterate over some deleted entries of a lock-less list
+ * safe against removal of list entry
* @pos: the &struct llist_node to use as a loop cursor
- * @n: another &struct llist_node to use as temporary storage
- * @node: the first entry of deleted list entries
+ * @...: either (node) or (next, node)
+ *
+ * next: another &struct llist_node to use as optional temporary storage.
+ * The temporary cursor is internal unless explicitly supplied by
+ * the caller.
+ * node: the first entry of deleted list entries
*
* In general, some entries of the lock-less list can be traversed
* safely only after being deleted from list, so start with an entry
@@ -159,8 +181,9 @@ static inline bool llist_on_list(const struct llist_node *node)
* you want to traverse from the oldest to the newest, you must
* reverse the order by yourself before traversing.
*/
-#define llist_for_each_safe(pos, n, node) \
- for ((pos) = (node); (pos) && ((n) = (pos)->next, true); (pos) = (n))
+#define llist_for_each_mutable(pos, ...) \
+ CONCATENATE(__llist_for_each_mutable, COUNT_ARGS(__VA_ARGS__)) \
+ (pos, __VA_ARGS__)
/**
* llist_for_each_entry - iterate over some deleted entries of lock-less list of given type
@@ -182,13 +205,41 @@ static inline bool llist_on_list(const struct llist_node *node)
member_address_is_nonnull(pos, member); \
(pos) = llist_entry((pos)->member.next, typeof(*(pos)), member))
+/*
+ * llist_for_each_entry_safe is an old interface, use llist_for_each_entry_mutable instead.
+ */
+#define llist_for_each_entry_safe(pos, n, node, member) \
+ for (pos = llist_entry((node), typeof(*pos), member); \
+ member_address_is_nonnull(pos, member) && \
+ (n = llist_entry(pos->member.next, typeof(*n), member), true); \
+ pos = n)
+
+#define __llist_for_each_entry_mutable_internal(pos, tmp, node, member) \
+ for (typeof(pos) tmp = ((pos) = llist_entry((node), typeof(*pos), member), \
+ member_address_is_nonnull(pos, member) ? \
+ llist_entry((pos)->member.next, typeof(*pos), member) : NULL); \
+ member_address_is_nonnull(pos, member); \
+ (pos) = tmp, tmp = member_address_is_nonnull(pos, member) ? \
+ llist_entry((pos)->member.next, typeof(*pos), member) : NULL)
+
+#define __llist_for_each_entry_mutable2(pos, node, member) \
+ __llist_for_each_entry_mutable_internal(pos, __UNIQUE_ID(next), node, member)
+
+#define __llist_for_each_entry_mutable3(pos, next, node, member) \
+ llist_for_each_entry_safe(pos, next, node, member)
+
/**
- * llist_for_each_entry_safe - iterate over some deleted entries of lock-less list of given type
- * safe against removal of list entry
+ * llist_for_each_entry_mutable - iterate over some deleted entries of
+ * lock-less list of given type safe against
+ * removal of list entry
* @pos: the type * to use as a loop cursor.
- * @n: another type * to use as temporary storage
- * @node: the first entry of deleted list entries.
- * @member: the name of the llist_node with the struct.
+ * @...: either (node, member) or (next, node, member)
+ *
+ * next: another type * to use as optional temporary storage. The
+ * temporary cursor is internal unless explicitly supplied by the
+ * caller.
+ * node: the first entry of deleted list entries.
+ * member: the name of the llist_node with the struct.
*
* In general, some entries of the lock-less list can be traversed
* safely only after being removed from list, so start with an entry
@@ -199,11 +250,9 @@ static inline bool llist_on_list(const struct llist_node *node)
* you want to traverse from the oldest to the newest, you must
* reverse the order by yourself before traversing.
*/
-#define llist_for_each_entry_safe(pos, n, node, member) \
- for (pos = llist_entry((node), typeof(*pos), member); \
- member_address_is_nonnull(pos, member) && \
- (n = llist_entry(pos->member.next, typeof(*n), member), true); \
- pos = n)
+#define llist_for_each_entry_mutable(pos, ...) \
+ CONCATENATE(__llist_for_each_entry_mutable, \
+ COUNT_ARGS(__VA_ARGS__))(pos, __VA_ARGS__)
/**
* llist_empty - tests whether a lock-less list is empty
--
2.43.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox