[PATCH] vhost: fix caching attributes of MMIO regions by setting them explicitly

[PATCH] vhost: fix caching attributes of MMIO regions by setting them explicitly

[Kommula Shiva Shankar]

Explicitly set non-cached caching attributes for MMIO regions.
Default write-back mode can cause CPU to cache device memory,
causing invalid reads and unpredictable behavior.

Invalid read and write issues were observed on ARM64 when mapping the
notification area to userspace via mmap.

Signed-off-by: Kommula Shiva Shankar <kshankar@marvell.com>
Acked-by: Jason Wang <jasowang@redhat.com>
---
Originally sent to net-next, now redirected to vhost tree
per Jason Wang's suggestion. 

 drivers/vhost/vdpa.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 05a481e4c385..b0179e8567ab 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -1527,6 +1527,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma)
 	if (vma->vm_end - vma->vm_start != notify.size)
 		return -ENOTSUPP;
 
+	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
 	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
 	vma->vm_ops = &vhost_vdpa_vm_ops;
 	return 0;
-- 
2.48.1

Re: [EXTERNAL] [PATCH] vhost: fix caching attributes of MMIO regions by setting them explicitly

[Shiva Shankar Kommula]

Hi Michael, 
Just a ping on this patch. Would appreciate your review when you get a chance.

Thanks


> On 2 Jan 2026, at 12:27, Kommula Shiva Shankar <kshankar@marvell.com> wrote:
> 
> Prioritize security for external emails:
> Confirm sender and content safety before clicking links or opening attachments
> Report Suspicious
> Explicitly set non-cached caching attributes for MMIO regions.
> Default write-back mode can cause CPU to cache device memory,
> causing invalid reads and unpredictable behavior.
> 
> Invalid read and write issues were observed on ARM64 when mapping the
> notification area to userspace via mmap.
> 
> Signed-off-by: Kommula Shiva Shankar <kshankar@marvell.com>
> Acked-by: Jason Wang <jasowang@redhat.com>
> ---
> Originally sent to net-next, now redirected to vhost tree
> per Jason Wang's suggestion. 
> 
> drivers/vhost/vdpa.c | 1 +
> 1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
> index 05a481e4c385..b0179e8567ab 100644
> --- a/drivers/vhost/vdpa.c
> +++ b/drivers/vhost/vdpa.c
> @@ -1527,6 +1527,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma)
> if (vma->vm_end - vma->vm_start != notify.size)
> return -ENOTSUPP;
> 
> + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
> vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
> vma->vm_ops = &vhost_vdpa_vm_ops;
> return 0;
> -- 
> 2.48.1
> 
> 
> 
> 

Re: [PATCH] vhost: fix caching attributes of MMIO regions by setting them explicitly

[Michael S. Tsirkin]

On Fri, Jan 02, 2026 at 12:27:03PM +0530, Kommula Shiva Shankar wrote:
> Explicitly set non-cached caching attributes for MMIO regions.
> Default write-back mode can cause CPU to cache device memory,
> causing invalid reads and unpredictable behavior.
> 
> Invalid read and write issues were observed on ARM64 when mapping the
> notification area to userspace via mmap.

device memory in question is the VQ kick, yes?
So if it is cached, the kick can get delayed, but how
is this causing "invalid read and write issues"?
What is read/written exactly?

> 
> Signed-off-by: Kommula Shiva Shankar <kshankar@marvell.com>
> Acked-by: Jason Wang <jasowang@redhat.com>

I also worry a bit about regressing on other hardware.
Cc nvidia guys.


> ---
> Originally sent to net-next, now redirected to vhost tree
> per Jason Wang's suggestion. 
> 
>  drivers/vhost/vdpa.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
> index 05a481e4c385..b0179e8567ab 100644
> --- a/drivers/vhost/vdpa.c
> +++ b/drivers/vhost/vdpa.c
> @@ -1527,6 +1527,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma)
>  	if (vma->vm_end - vma->vm_start != notify.size)
>  		return -ENOTSUPP;
>  
> +	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
>  	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
>  	vma->vm_ops = &vhost_vdpa_vm_ops;
>  	return 0;
> -- 
> 2.48.1

Re: [EXTERNAL] Re: [PATCH] vhost: fix caching attributes of MMIO regions by setting them explicitly

[Shiva Shankar Kommula]

________________________________________
From: Michael S. Tsirkin <mst@redhat.com>
Sent: Tuesday, January 13, 2026 13:00
To: Shiva Shankar Kommula
Cc: jasowang@redhat.com; virtualization@lists.linux.dev; eperezma@redhat.com; kvm@vger.kernel.org; netdev@vger.kernel.org; Jerin Jacob; Nithin Kumar Dabilpuram; Srujana Challa; dtatulea@nvidia.com; jgg@nvidia.com
Subject: [EXTERNAL] Re: [PATCH] vhost: fix caching attributes of MMIO regions by setting them explicitly

On Fri, Jan 02, 2026 at 12: 27: 03PM +0530, Kommula Shiva Shankar wrote: > Explicitly set non-cached caching attributes for MMIO regions. > Default write-back mode can cause CPU to cache device memory, > causing invalid reads and unpredictable
ZjQcmQRYFpfptBannerStart
Prioritize security for external emails:
Confirm sender and content safety before clicking links or opening attachments
<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/CRVmXkqW!tc3Z1f8UYnX61G-8-Z36D0TwOyXGKE_ybqkwtAsNGONThDzce1-BzQFJIDlDn_iWLJpe2FHKyeofP3u77neSt706ScpI5Ec$>
Report Suspicious

ZjQcmQRYFpfptBannerEnd

On Fri, Jan 02, 2026 at 12:27:03PM +0530, Kommula Shiva Shankar wrote:
> Explicitly set non-cached caching attributes for MMIO regions.
> Default write-back mode can cause CPU to cache device memory,
> causing invalid reads and unpredictable behavior.
>
> Invalid read and write issues were observed on ARM64 when mapping the
> notification area to userspace via mmap.

> device memory in question is the VQ kick, yes?
> So if it is cached, the kick can get delayed, but how
> is this causing "invalid read and write issues"?
> What is read/written exactly?
This is the VQ notification address. 
on ARM64, when memory mapped as write back, notification writes never reached the device.
Reads on the device side sees stale values. 

>
> Signed-off-by: Kommula Shiva Shankar <kshankar@marvell.com>
> Acked-by: Jason Wang <jasowang@redhat.com>

> I also worry a bit about regressing on other hardware.
> Cc nvidia guys.


> ---
> Originally sent to net-next, now redirected to vhost tree
> per Jason Wang's suggestion.
>
>  drivers/vhost/vdpa.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
> index 05a481e4c385..b0179e8567ab 100644
> --- a/drivers/vhost/vdpa.c
> +++ b/drivers/vhost/vdpa.c
> @@ -1527,6 +1527,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma)
>       if (vma->vm_end - vma->vm_start != notify.size)
>               return -ENOTSUPP;
>
> +     vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
>       vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);
>       vma->vm_ops = &vhost_vdpa_vm_ops;
>       return 0;
> --
> 2.48.1

Re: [PATCH] vhost: fix caching attributes of MMIO regions by setting them explicitly

[Jason Gunthorpe]

On Tue, Jan 13, 2026 at 02:30:13AM -0500, Michael S. Tsirkin wrote:
> > Signed-off-by: Kommula Shiva Shankar <kshankar@marvell.com>
> > Acked-by: Jason Wang <jasowang@redhat.com>
> 
> I also worry a bit about regressing on other hardware.
> Cc nvidia guys.

> > +	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
> >  	vm_flags_set(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP);

This is definitely required and correct if notify.addr comes from a
PCI BAR address.

You need to trace the origin of that memory in all the drivers to
determine if it is OK or not.

For instance mlx5 is:

	kick_addr = mdev->bar_addr + offset;
	res->phys_kick_addr = kick_addr;
[..]
	addr = (phys_addr_t)ndev->mvdev.res.phys_kick_addr;

"bar_addr" is PCI memory so this patch is correct and required for
mlx5.

ifcvf:
                        hw->notify_base_pa = pci_resource_start(pdev, cap.bar) +
                                        le32_to_cpu(cap.offset);
[..]
                hw->vring[i].notify_pa = hw->notify_base_pa +
                        notify_off * hw->notify_off_multiplier;
[..]
	area.addr = vf->vring[idx].notify_pa;

octep:

                        oct_hw->notify_base_pa = pci_resource_start(pdev, cap.bar) +
                                                 le32_to_cpu(cap.offset);
[..]
                oct_hw->vqs[i].notify_pa = oct_hw->notify_base_pa +
                        notify_off * oct_hw->notify_off_multiplier;
[..]
	area.addr = oct_hw->vqs[idx].notify_pa;

pds:
 No idea, it is messed up though:
	area.addr = pdsv->vqs[qid].notify_pa;
    struct pds_vdpa_vq_info {
	dma_addr_t notify_pa;
 Can't cast dma_addr_t to phys_addr_t!

virtio_pci:
 Also messed up:
  	notify.addr = vp_vdpa->vring[qid].notify_pa;
    struct vp_vring {
	resource_size_t notify_pa;
 phys_addr is not a resource_size_t

Guessing pds and virtio_pci are also both fine, even if I gave up trying to
figure out where notify_pa gets set from in the end.

So the patch is OK

Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>

Jason