* Re: [PATCH v2 3/4] iommu/hyperv: Add para-virtualized IOMMU support for Hyper-V guest
From: Yu Zhang @ 2026-07-10 7:34 UTC (permalink / raw)
To: Michael Kelley
Cc: linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org,
iommu@lists.linux.dev, linux-pci@vger.kernel.org,
linux-arch@vger.kernel.org, wei.liu@kernel.org, kys@microsoft.com,
haiyangz@microsoft.com, decui@microsoft.com, longli@microsoft.com,
joro@8bytes.org, will@kernel.org, robin.murphy@arm.com,
bhelgaas@google.com, kwilczynski@kernel.org,
lpieralisi@kernel.org, mani@kernel.org, robh@kernel.org,
arnd@arndb.de, jgg@ziepe.ca, jacob.pan@linux.microsoft.com,
tgopinath@linux.microsoft.com,
easwar.hariharan@linux.microsoft.com, mrathor@linux.microsoft.com
In-Reply-To: <SN6PR02MB4157253E030D477FD91B7E26D4FE2@SN6PR02MB4157.namprd02.prod.outlook.com>
On Thu, Jul 09, 2026 at 07:08:26PM +0000, Michael Kelley wrote:
> From: Yu Zhang <zhangyu1@linux.microsoft.com> Sent: Thursday, July 2, 2026 9:05 AM
> >
> > Add a para-virtualized IOMMU driver for Linux guests running on Hyper-V.
> > This driver implements stage-1 IO translation within the guest OS.
> > It integrates with the Linux IOMMU core, utilizing Hyper-V hypercalls
> > for:
> > - Capability discovery
> > - Domain allocation, configuration, and deallocation
> > - Device attachment and detachment
> > - IOTLB invalidation
> >
> > The driver constructs x86-compatible stage-1 IO page tables in the
> > guest memory using consolidated IO page table helpers. This allows
> > the guest to manage stage-1 translations independently of vendor-
> > specific drivers (like Intel VT-d or AMD IOMMU).
> >
> > Hyper-V consumes this stage-1 IO page table when a device domain is
> > created and configured, and nests it with the host's stage-2 IO page
> > tables, therefore eliminating the VM exits for guest IOMMU mapping
> > operations. For unmapping operations, VM exits to perform the IOTLB
> > flush are still unavoidable.
> >
> > To identify a device in its hypercall interface, the driver looks up the
> > logical device ID prefix registered for the device's PCI domain (see the
> > logical device ID registry in hv_common.c) and combines it with the PCI
> > function number of the endpoint device.
> >
> > Co-developed-by: Wei Liu <wei.liu@kernel.org>
> > Signed-off-by: Wei Liu <wei.liu@kernel.org>
> > Co-developed-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> > Signed-off-by: Easwar Hariharan <easwar.hariharan@linux.microsoft.com>
> > Signed-off-by: Yu Zhang <zhangyu1@linux.microsoft.com>
> > ---
> > arch/x86/hyperv/hv_init.c | 4 +
> > arch/x86/include/asm/mshyperv.h | 4 +
> > drivers/iommu/Kconfig | 1 +
> > drivers/iommu/hyperv/Kconfig | 16 +
> > drivers/iommu/hyperv/Makefile | 1 +
> > drivers/iommu/hyperv/iommu.c | 620 ++++++++++++++++++++++++++++++++
> > drivers/iommu/hyperv/iommu.h | 51 +++
> > 7 files changed, 697 insertions(+)
> > create mode 100644 drivers/iommu/hyperv/Kconfig
> > create mode 100644 drivers/iommu/hyperv/iommu.c
> > create mode 100644 drivers/iommu/hyperv/iommu.h
> >
> > diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
> > index 55a8b6de2865..094f9f7ddb72 100644
> > --- a/arch/x86/hyperv/hv_init.c
> > +++ b/arch/x86/hyperv/hv_init.c
> > @@ -578,6 +578,10 @@ void __init hyperv_init(void)
> > old_setup_percpu_clockev = x86_init.timers.setup_percpu_clockev;
> > x86_init.timers.setup_percpu_clockev = hv_stimer_setup_percpu_clockev;
> >
> > +#ifdef CONFIG_HYPERV_PVIOMMU
> > + x86_init.iommu.iommu_init = hv_iommu_init;
> > +#endif
> > +
>
> This approach to .iommu_init is a bit different from the Intel VT-d and
> AMD IOMMU initialization. Those cases detect the existence of the
> IOMMU first via a "detect" function that is called in pci_iommu_alloc().
> If the detect function finds an IOMMU, it sets .iommu_init. Any
> reason not to use the same approach for the Hyper-V pvIOMMU?
> One problem with exactly the same approach is that Hyper-V
> hypercalls aren't set up at the time pci_iommu_alloc() runs.
Yes. That's why I did not follow Intel VT-d and AMD IOMMU's approach -
the hv_hypercall_pg is not ready yet.
> So you'd have to call the "detect" function here in hyperv_init(),
> and have the detect function set .iommu_init if pvIOMMU
> support is present.
>
The detecion of the presense and capabilities of the pvIOMMU are done
in one hypercall. But I guess we can:
- do the HVCALL_GET_IOMMU_CAPABILITIES in hyperv_init();
- check the presense and only set .iommu_init to hyperv_iommu_init()
if pvIOMMU is present;
- and then do other capalibities check in hv_iommu_init();
- only give the error log if an pvIOMMU is present yet its capabilities
are not legal.
So below errors will not be printed for guest kernels built with
CONFIG_HYPERV_PVIOMMU and running on a host w/o one.
> While the code currently in this patch works, it generates boot
> time errors if the kernel is built with CONFIG_HYPERV_PVIOMMU
> but run in a guest on a host without pvIOMMU support:
>
> [ 0.101673] Hyper-V pvIOMMU: HVCALL_GET_IOMMU_CAPABILITIES failed, status 2
> [ 0.101675] Hyper-V pvIOMMU: HVCALL_GET_IOMMU_CAPABILITIES failed: -22
>
> We really don't want errors if it's just the case that there's no
> pvIOMMU support. A less alarming message (at INFO level instead
> of ERROR level) about running without an IOMMU might be OK, but
> perhaps is unnecessary since you have an INFO message if the
> pvIOMMU is found and successfully initialized.
>
> > hv_apic_init();
> >
> > x86_init.pci.arch_init = hv_pci_init;
> > diff --git a/arch/x86/include/asm/mshyperv.h b/arch/x86/include/asm/mshyperv.h
> > index f64393e853ee..20d947c2c758 100644
> > --- a/arch/x86/include/asm/mshyperv.h
> > +++ b/arch/x86/include/asm/mshyperv.h
> > @@ -313,6 +313,10 @@ static inline void mshv_vtl_return_hypercall(void) {}
> > static inline void __mshv_vtl_return_call(struct mshv_vtl_cpu_context *vtl0) {}
> > #endif
> >
> > +#ifdef CONFIG_HYPERV_PVIOMMU
> > +int __init hv_iommu_init(void);
> > +#endif
> > +
> > #include <asm-generic/mshyperv.h>
> >
> > #endif
> > diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
> > index 6e07bd69467a..0d128f377929 100644
> > --- a/drivers/iommu/Kconfig
> > +++ b/drivers/iommu/Kconfig
> > @@ -195,6 +195,7 @@ config MSM_IOMMU
> > source "drivers/iommu/amd/Kconfig"
> > source "drivers/iommu/arm/Kconfig"
> > source "drivers/iommu/intel/Kconfig"
> > +source "drivers/iommu/hyperv/Kconfig"
> > source "drivers/iommu/iommufd/Kconfig"
> > source "drivers/iommu/riscv/Kconfig"
> >
> > diff --git a/drivers/iommu/hyperv/Kconfig b/drivers/iommu/hyperv/Kconfig
> > new file mode 100644
> > index 000000000000..8b6abbaaf9b8
> > --- /dev/null
> > +++ b/drivers/iommu/hyperv/Kconfig
> > @@ -0,0 +1,16 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +# HyperV paravirtualized IOMMU support
> > +config HYPERV_PVIOMMU
> > + bool "Microsoft Hypervisor para-virtualized IOMMU support"
> > + depends on X86_64 && HYPERV
> > + select IOMMU_API
> > + select GENERIC_PT
> > + select IOMMU_PT
> > + select IOMMU_PT_X86_64
> > + select IOMMU_IOVA
> > + default HYPERV
> > + help
> > + Para-virtualized IOMMU driver for Linux guests running on
> > + Microsoft Hyper-V. Provides DMA remapping and IOTLB
> > + flush support to enable DMA isolation for devices
>
> I think this is specifically "PCI devices", right? VMBus devices
> that do DMA (storvsc and netvsc) don't use the pvIOMMU.
> The "assigned to the guest" phrase pretty much implies "PCI",
> but it would be clearer to be explicit.
>
Right. It is specifically "PCI devices".
B.R.
Yu
> > + assigned to the guest.
> > diff --git a/drivers/iommu/hyperv/Makefile b/drivers/iommu/hyperv/Makefile
> > index 6ef0ef97f3dd..fefb409d976b 100644
> > --- a/drivers/iommu/hyperv/Makefile
> > +++ b/drivers/iommu/hyperv/Makefile
> > @@ -1,2 +1,3 @@
> > # SPDX-License-Identifier: GPL-2.0
> > obj-$(CONFIG_IRQ_REMAP) += hv-irq-remap-x86.o
> > +obj-$(CONFIG_HYPERV_PVIOMMU) += iommu.o
> > diff --git a/drivers/iommu/hyperv/iommu.c b/drivers/iommu/hyperv/iommu.c
> > new file mode 100644
> > index 000000000000..254136946404
> > --- /dev/null
> > +++ b/drivers/iommu/hyperv/iommu.c
> > @@ -0,0 +1,620 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +
> > +/*
> > + * Hyper-V IOMMU driver.
> > + *
> > + * Copyright (C) 2019, 2024-2026 Microsoft, Inc.
> > + */
> > +
> > +#define pr_fmt(fmt) "Hyper-V pvIOMMU: " fmt
> > +#define dev_fmt(fmt) pr_fmt(fmt)
> > +
> > +#include <linux/iommu.h>
> > +#include <linux/pci.h>
> > +#include <linux/dma-map-ops.h>
> > +#include <linux/generic_pt/iommu.h>
> > +#include <linux/pci-ats.h>
> > +
> > +#include <asm/iommu.h>
> > +#include <asm/hypervisor.h>
> > +#include <asm/mshyperv.h>
> > +
> > +#include "iommu.h"
> > +#include "../iommu-pages.h"
> > +
> > +struct hv_iommu_dev *hv_iommu_device;
> > +
> > +/*
> > + * Identity and blocking domains are static singletons: identity is a 1:1
> > + * passthrough with no page table, blocking rejects all DMA. Neither holds
> > + * per-IOMMU state, so one instance suffices even with multiple vIOMMUs.
> > + */
> > +static const struct iommu_domain_ops hv_iommu_identity_domain_ops;
> > +static const struct iommu_domain_ops hv_iommu_blocking_domain_ops;
> > +static struct iommu_ops hv_iommu_ops;
> > +
> > +static struct hv_iommu_domain hv_identity_domain = {
> > + .domain = {
> > + .type = IOMMU_DOMAIN_IDENTITY,
> > + .ops = &hv_iommu_identity_domain_ops,
> > + .owner = &hv_iommu_ops,
> > + },
> > +};
> > +static struct hv_iommu_domain hv_blocking_domain = {
> > + .domain = {
> > + .type = IOMMU_DOMAIN_BLOCKED,
> > + .ops = &hv_iommu_blocking_domain_ops,
> > + .owner = &hv_iommu_ops,
> > + },
> > +};
> > +
> > +static inline bool hv_iommu_present(u64 cap)
> > +{
> > + return cap & HV_IOMMU_CAP_PRESENT;
> > +}
> > +
> > +static inline bool hv_iommu_s1_domain_supported(u64 cap)
> > +{
> > + return cap & HV_IOMMU_CAP_S1;
> > +}
> > +
> > +static inline bool hv_iommu_5lvl_supported(u64 cap)
> > +{
> > + return cap & HV_IOMMU_CAP_S1_5LVL;
> > +}
> > +
> > +static inline bool hv_iommu_ats_supported(u64 cap)
> > +{
> > + return cap & HV_IOMMU_CAP_ATS;
> > +}
> > +
> > +static int hv_create_device_domain(struct hv_iommu_domain *hv_domain, u32 domain_stage)
> > +{
> > + int ret;
> > + u64 status;
> > + unsigned long flags;
> > + struct hv_input_create_device_domain *input;
> > +
> > + ret = ida_alloc_range(&hv_iommu_device->domain_ids,
> > + hv_iommu_device->first_domain, hv_iommu_device->last_domain,
> > + GFP_KERNEL);
> > + if (ret < 0)
> > + return ret;
> > +
> > + hv_domain->device_domain.partition_id = HV_PARTITION_ID_SELF;
> > + hv_domain->device_domain.domain_id.type = domain_stage;
> > + hv_domain->device_domain.domain_id.id = ret;
> > + hv_domain->hv_iommu = hv_iommu_device;
> > +
> > + local_irq_save(flags);
> > +
> > + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> > + memset(input, 0, sizeof(*input));
> > + input->device_domain = hv_domain->device_domain;
> > + input->create_device_domain_flags.forward_progress_required = 1;
> > + input->create_device_domain_flags.inherit_owning_vtl = 0;
> > + status = hv_do_hypercall(HVCALL_CREATE_DEVICE_DOMAIN, input, NULL);
> > +
> > + local_irq_restore(flags);
> > +
> > + if (!hv_result_success(status)) {
> > + pr_err("HVCALL_CREATE_DEVICE_DOMAIN failed, status %lld\n", status);
> > + ida_free(&hv_iommu_device->domain_ids, hv_domain->device_domain.domain_id.id);
> > + }
> > +
> > + return hv_result_to_errno(status);
> > +}
> > +
> > +static void hv_delete_device_domain(struct hv_iommu_domain *hv_domain)
> > +{
> > + u64 status;
> > + unsigned long flags;
> > + struct hv_input_delete_device_domain *input;
> > +
> > + local_irq_save(flags);
> > +
> > + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> > + memset(input, 0, sizeof(*input));
> > + input->device_domain = hv_domain->device_domain;
> > + status = hv_do_hypercall(HVCALL_DELETE_DEVICE_DOMAIN, input, NULL);
> > +
> > + local_irq_restore(flags);
> > +
> > + if (!hv_result_success(status))
> > + pr_err("HVCALL_DELETE_DEVICE_DOMAIN failed, status %lld\n", status);
> > +
> > + ida_free(&hv_domain->hv_iommu->domain_ids, hv_domain->device_domain.domain_id.id);
> > +}
> > +
> > +static bool hv_iommu_capable(struct device *dev, enum iommu_cap cap)
> > +{
> > + switch (cap) {
> > + case IOMMU_CAP_CACHE_COHERENCY:
> > + return true;
> > + case IOMMU_CAP_DEFERRED_FLUSH:
> > + return true;
> > + default:
> > + return false;
> > + }
> > +}
> > +
> > +static void hv_flush_device_domain(struct hv_iommu_domain *hv_domain)
> > +{
> > + u64 status;
> > + unsigned long flags;
> > + struct hv_input_flush_device_domain *input;
> > +
> > + local_irq_save(flags);
> > +
> > + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> > + memset(input, 0, sizeof(*input));
> > + input->device_domain = hv_domain->device_domain;
> > + status = hv_do_hypercall(HVCALL_FLUSH_DEVICE_DOMAIN, input, NULL);
> > +
> > + local_irq_restore(flags);
> > +
> > + if (!hv_result_success(status))
> > + pr_err("HVCALL_FLUSH_DEVICE_DOMAIN failed, status %lld\n", status);
> > +}
> > +
> > +static int hv_iommu_attach_dev(struct iommu_domain *domain, struct device *dev,
> > + struct iommu_domain *old)
> > +{
> > + u64 status;
> > + u32 prefix;
> > + unsigned long flags;
> > + struct pci_dev *pdev;
> > + struct hv_input_attach_device_domain *input;
> > + struct hv_iommu_endpoint *vdev = dev_iommu_priv_get(dev);
> > + struct hv_iommu_domain *hv_domain = to_hv_iommu_domain(domain);
> > + int ret;
> > +
> > + if (vdev->hv_domain == hv_domain)
> > + return 0;
> > +
> > + pdev = to_pci_dev(dev);
> > + dev_dbg(dev, "attaching to domain %d\n",
> > + hv_domain->device_domain.domain_id.id);
> > +
> > + ret = hv_iommu_lookup_logical_dev_id(pci_domain_nr(pdev->bus), &prefix);
> > + if (ret) {
> > + dev_err(&pdev->dev, "no IOMMU registration for vPCI bus\n");
> > + return ret;
> > + }
> > +
> > + local_irq_save(flags);
> > +
> > + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> > + memset(input, 0, sizeof(*input));
> > + input->device_domain = hv_domain->device_domain;
> > + input->device_id.as_uint64 = (u64)prefix | PCI_FUNC(pdev->devfn);
> > + status = hv_do_hypercall(HVCALL_ATTACH_DEVICE_DOMAIN, input, NULL);
> > +
> > + local_irq_restore(flags);
> > +
> > + if (!hv_result_success(status))
> > + pr_err("HVCALL_ATTACH_DEVICE_DOMAIN failed, status %lld\n", status);
> > + else
> > + vdev->hv_domain = hv_domain;
> > +
> > + return hv_result_to_errno(status);
> > +}
> > +
> > +static int hv_iommu_blocking_attach_dev(struct iommu_domain *domain,
> > + struct device *dev,
> > + struct iommu_domain *old)
> > +{
> > + int ret = hv_iommu_attach_dev(domain, dev, old);
> > +
> > + /*
> > + * Attaching to the blocking domain only asks the hypervisor to
> > + * disable translation and IOPF for the device, so it cannot fail
> > + * unless there is a driver or hypervisor bug. Return the hypercall
> > + * status rather than 0 so that a failure on the DMA ownership claim
> > + * path (VFIO/iommufd) fails the claim instead of leaving the device
> > + * unblocked. WARN since such a failure indicates a bug.
> > + */
> > + WARN_ON(ret);
> > + return ret;
> > +}
> > +
> > +static int hv_iommu_get_logical_device_property(struct device *dev,
> > + u32 code,
> > + struct hv_output_get_logical_device_property *property)
> > +{
> > + u64 status;
> > + u32 prefix;
> > + unsigned long flags;
> > + int ret;
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > + struct hv_input_get_logical_device_property *input;
> > + struct hv_output_get_logical_device_property *output;
> > +
> > + ret = hv_iommu_lookup_logical_dev_id(pci_domain_nr(pdev->bus), &prefix);
> > + if (ret)
> > + return ret;
> > +
> > + local_irq_save(flags);
> > +
> > + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> > + output = (struct hv_output_get_logical_device_property *)(input + 1);
> > + memset(input, 0, sizeof(*input));
> > + input->partition_id = HV_PARTITION_ID_SELF;
> > + input->logical_device_id = (u64)prefix | PCI_FUNC(pdev->devfn);
> > + input->code = code;
> > + status = hv_do_hypercall(HVCALL_GET_LOGICAL_DEVICE_PROPERTY, input, output);
> > + *property = *output;
> > +
> > + local_irq_restore(flags);
> > +
> > + if (!hv_result_success(status))
> > + pr_err("HVCALL_GET_LOGICAL_DEVICE_PROPERTY failed, status %lld\n", status);
> > +
> > + return hv_result_to_errno(status);
> > +}
> > +
> > +static struct iommu_device *hv_iommu_probe_device(struct device *dev)
> > +{
> > + struct pci_dev *pdev;
> > + struct hv_iommu_endpoint *vdev;
> > + struct hv_output_get_logical_device_property device_iommu_property = {0};
> > +
> > + if (!dev_is_pci(dev))
> > + return ERR_PTR(-ENODEV);
> > +
> > + pdev = to_pci_dev(dev);
> > +
> > + if (hv_iommu_get_logical_device_property(dev,
> > + HV_LOGICAL_DEVICE_PROPERTY_PVIOMMU,
> > + &device_iommu_property) ||
> > + !(device_iommu_property.device_iommu & HV_DEVICE_IOMMU_ENABLED))
> > + return ERR_PTR(-ENODEV);
> > +
> > + vdev = kzalloc_obj(*vdev, GFP_KERNEL);
> > + if (!vdev)
> > + return ERR_PTR(-ENOMEM);
> > +
> > + vdev->dev = dev;
> > + vdev->hv_iommu = hv_iommu_device;
> > + dev_iommu_priv_set(dev, vdev);
> > +
> > + if (hv_iommu_ats_supported(hv_iommu_device->cap) &&
> > + pci_ats_supported(pdev))
> > + pci_enable_ats(pdev, __ffs(hv_iommu_device->pgsize_bitmap));
> > +
> > + return &vdev->hv_iommu->iommu;
> > +}
> > +
> > +static void hv_iommu_release_device(struct device *dev)
> > +{
> > + struct hv_iommu_endpoint *vdev = dev_iommu_priv_get(dev);
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > +
> > + if (pdev->ats_enabled)
> > + pci_disable_ats(pdev);
> > +
> > + dev_iommu_priv_set(dev, NULL);
> > +
> > + kfree(vdev);
> > +}
> > +
> > +static struct iommu_group *hv_iommu_device_group(struct device *dev)
> > +{
> > + if (dev_is_pci(dev))
> > + return pci_device_group(dev);
> > +
> > + WARN_ON_ONCE(1);
> > + return generic_device_group(dev);
> > +}
> > +
> > +static int hv_configure_device_domain(struct hv_iommu_domain *hv_domain, u32 domain_type)
> > +{
> > + u64 status;
> > + unsigned long flags;
> > + struct pt_iommu_x86_64_hw_info pt_info;
> > + struct hv_input_configure_device_domain *input;
> > +
> > + local_irq_save(flags);
> > +
> > + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> > + memset(input, 0, sizeof(*input));
> > + input->device_domain = hv_domain->device_domain;
> > + input->settings.flags.blocked = (domain_type == IOMMU_DOMAIN_BLOCKED);
> > + /*
> > + * Clearing translation_enabled bypasses translation (DMA uses the GPA
> > + * directly), which only suits identity. The hypervisor requires paging
> > + * and blocked domains to keep it set.
> > + */
> > + input->settings.flags.translation_enabled = (domain_type != IOMMU_DOMAIN_IDENTITY);
> > +
> > + if (domain_type & __IOMMU_DOMAIN_PAGING) {
> > + pt_iommu_x86_64_hw_info(&hv_domain->pt_iommu_x86_64, &pt_info);
> > + input->settings.page_table_root = pt_info.gcr3_pt;
> > + input->settings.flags.first_stage_paging_mode =
> > + pt_info.levels == 5;
> > + }
> > + status = hv_do_hypercall(HVCALL_CONFIGURE_DEVICE_DOMAIN, input, NULL);
> > +
> > + local_irq_restore(flags);
> > +
> > + if (!hv_result_success(status))
> > + pr_err("HVCALL_CONFIGURE_DEVICE_DOMAIN failed, status %lld\n", status);
> > +
> > + return hv_result_to_errno(status);
> > +}
> > +
> > +static int __init hv_initialize_static_domains(void)
> > +{
> > + int ret;
> > + struct hv_iommu_domain *hv_domain;
> > +
> > + /* Default stage-1 identity domain */
> > + hv_domain = &hv_identity_domain;
> > +
> > + ret = hv_create_device_domain(hv_domain, HV_DEVICE_DOMAIN_TYPE_S1);
> > + if (ret)
> > + return ret;
> > +
> > + ret = hv_configure_device_domain(hv_domain, IOMMU_DOMAIN_IDENTITY);
> > + if (ret)
> > + goto delete_identity_domain;
> > +
> > + /* Default stage-1 blocked domain */
> > + hv_domain = &hv_blocking_domain;
> > +
> > + ret = hv_create_device_domain(hv_domain, HV_DEVICE_DOMAIN_TYPE_S1);
> > + if (ret)
> > + goto delete_identity_domain;
> > +
> > + ret = hv_configure_device_domain(hv_domain, IOMMU_DOMAIN_BLOCKED);
> > + if (ret)
> > + goto delete_blocked_domain;
> > +
> > + return 0;
> > +
> > +delete_blocked_domain:
> > + hv_delete_device_domain(&hv_blocking_domain);
> > +delete_identity_domain:
> > + hv_delete_device_domain(&hv_identity_domain);
> > + return ret;
> > +}
> > +
> > +/* x86 architectural MSI address range */
> > +#define INTERRUPT_RANGE_START (0xfee00000)
> > +#define INTERRUPT_RANGE_END (0xfeefffff)
>
> These same constants are also defined in the Intel and AMD
> IOMMU drivers. Bonus points for creating a common definition
> in a .h file that can be shared by all the drivers. :-)
>
> > +static void hv_iommu_get_resv_regions(struct device *dev,
> > + struct list_head *head)
> > +{
> > + struct iommu_resv_region *region;
> > +
> > + region = iommu_alloc_resv_region(INTERRUPT_RANGE_START,
> > + INTERRUPT_RANGE_END - INTERRUPT_RANGE_START + 1,
> > + 0, IOMMU_RESV_MSI, GFP_KERNEL);
> > + if (!region)
> > + return;
> > +
> > + list_add_tail(®ion->list, head);
> > +}
> > +
> > +static void hv_iommu_flush_iotlb_all(struct iommu_domain *domain)
> > +{
> > + hv_flush_device_domain(to_hv_iommu_domain(domain));
> > +}
> > +
> > +static void hv_iommu_iotlb_sync(struct iommu_domain *domain,
> > + struct iommu_iotlb_gather *iotlb_gather)
> > +{
> > + hv_flush_device_domain(to_hv_iommu_domain(domain));
> > +
> > + iommu_put_pages_list(&iotlb_gather->freelist);
> > +}
> > +
> > +static void hv_iommu_paging_domain_free(struct iommu_domain *domain)
> > +{
> > + struct hv_iommu_domain *hv_domain = to_hv_iommu_domain(domain);
> > +
> > + /* Free all remaining mappings */
> > + pt_iommu_deinit(&hv_domain->pt_iommu);
> > +
> > + hv_delete_device_domain(hv_domain);
> > +
> > + kfree(hv_domain);
> > +}
> > +
> > +static const struct iommu_domain_ops hv_iommu_identity_domain_ops = {
> > + .attach_dev = hv_iommu_attach_dev,
> > +};
> > +
> > +static const struct iommu_domain_ops hv_iommu_blocking_domain_ops = {
> > + .attach_dev = hv_iommu_blocking_attach_dev,
> > +};
> > +
> > +static const struct iommu_domain_ops hv_iommu_paging_domain_ops = {
> > + .attach_dev = hv_iommu_attach_dev,
> > + IOMMU_PT_DOMAIN_OPS(x86_64),
> > + .flush_iotlb_all = hv_iommu_flush_iotlb_all,
> > + .iotlb_sync = hv_iommu_iotlb_sync,
> > + .free = hv_iommu_paging_domain_free,
> > +};
> > +
> > +static struct iommu_domain *hv_iommu_domain_alloc_paging(struct device *dev)
> > +{
> > + int ret;
> > + struct hv_iommu_domain *hv_domain;
> > + struct pt_iommu_x86_64_cfg cfg = {};
> > +
> > + hv_domain = kzalloc_obj(*hv_domain, GFP_KERNEL);
> > + if (!hv_domain)
> > + return ERR_PTR(-ENOMEM);
> > +
> > + ret = hv_create_device_domain(hv_domain, HV_DEVICE_DOMAIN_TYPE_S1);
> > + if (ret)
> > + goto err_free;
> > +
> > + hv_domain->pt_iommu.nid = dev_to_node(dev);
> > +
> > + cfg.common.hw_max_vasz_lg2 = hv_iommu_device->max_iova_width;
> > + cfg.common.hw_max_oasz_lg2 = 52;
> > + cfg.top_level = (hv_iommu_device->max_iova_width > 48) ? 4 : 3;
> > +
> > + ret = pt_iommu_x86_64_init(&hv_domain->pt_iommu_x86_64, &cfg, GFP_KERNEL);
> > + if (ret)
> > + goto err_delete_domain;
> > +
> > + /* Constrain to page sizes the hypervisor supports */
> > + hv_domain->domain.pgsize_bitmap &= hv_iommu_device->pgsize_bitmap;
> > +
> > + hv_domain->domain.ops = &hv_iommu_paging_domain_ops;
> > +
> > + ret = hv_configure_device_domain(hv_domain, __IOMMU_DOMAIN_PAGING);
> > + if (ret)
> > + goto err_pt_deinit;
> > +
> > + return &hv_domain->domain;
> > +
> > +err_pt_deinit:
> > + pt_iommu_deinit(&hv_domain->pt_iommu);
> > +err_delete_domain:
> > + hv_delete_device_domain(hv_domain);
> > +err_free:
> > + kfree(hv_domain);
> > + return ERR_PTR(ret);
> > +}
> > +
> > +static struct iommu_ops hv_iommu_ops = {
> > + .capable = hv_iommu_capable,
> > + .domain_alloc_paging = hv_iommu_domain_alloc_paging,
> > + .probe_device = hv_iommu_probe_device,
> > + .release_device = hv_iommu_release_device,
> > + .device_group = hv_iommu_device_group,
> > + .get_resv_regions = hv_iommu_get_resv_regions,
> > + .owner = THIS_MODULE,
> > + .identity_domain = &hv_identity_domain.domain,
> > + .blocked_domain = &hv_blocking_domain.domain,
> > + .release_domain = &hv_blocking_domain.domain,
> > +};
> > +
> > +static int hv_iommu_detect(struct hv_output_get_iommu_capabilities *hv_iommu_cap)
> > +{
> > + u64 status;
> > + unsigned long flags;
> > + struct hv_input_get_iommu_capabilities *input;
> > + struct hv_output_get_iommu_capabilities *output;
> > +
> > + local_irq_save(flags);
> > +
> > + input = *this_cpu_ptr(hyperv_pcpu_input_arg);
> > + output = (struct hv_output_get_iommu_capabilities *)(input + 1);
> > + memset(input, 0, sizeof(*input));
> > + input->partition_id = HV_PARTITION_ID_SELF;
> > + status = hv_do_hypercall(HVCALL_GET_IOMMU_CAPABILITIES, input, output);
> > + *hv_iommu_cap = *output;
> > +
> > + local_irq_restore(flags);
> > +
> > + if (!hv_result_success(status))
> > + pr_err("HVCALL_GET_IOMMU_CAPABILITIES failed, status %lld\n", status);
> > +
> > + return hv_result_to_errno(status);
> > +}
> > +
> > +static void __init hv_init_iommu_device(struct hv_iommu_dev *hv_iommu,
> > + struct hv_output_get_iommu_capabilities *hv_iommu_cap)
> > +{
> > + ida_init(&hv_iommu->domain_ids);
> > +
> > + hv_iommu->cap = hv_iommu_cap->iommu_cap;
> > + hv_iommu->max_iova_width = hv_iommu_cap->max_iova_width;
> > + if (!hv_iommu_5lvl_supported(hv_iommu->cap) &&
> > + hv_iommu->max_iova_width > 48) {
> > + pr_info("5-level paging not supported, limiting iova width to 48.\n");
> > + hv_iommu->max_iova_width = 48;
> > + }
> > +
> > + hv_iommu->geometry = (struct iommu_domain_geometry) {
> > + .aperture_start = 0,
> > + .aperture_end = (((u64)1) << hv_iommu->max_iova_width) - 1,
> > + .force_aperture = true,
> > + };
> > +
> > + hv_iommu->first_domain = HV_DEVICE_DOMAIN_ID_DEFAULT + 1;
> > + hv_iommu->last_domain = HV_DEVICE_DOMAIN_ID_NULL - 1;
> > + hv_iommu->pgsize_bitmap = hv_iommu_cap->pgsize_bitmap;
> > + hv_iommu_device = hv_iommu;
> > +}
> > +
> > +int __init hv_iommu_init(void)
> > +{
> > + int ret = 0;
> > + struct hv_iommu_dev *hv_iommu = NULL;
> > + struct hv_output_get_iommu_capabilities hv_iommu_cap = {0};
> > +
> > + if (no_iommu || iommu_detected)
> > + return -ENODEV;
> > +
> > + if (!hv_is_hyperv_initialized())
> > + return -ENODEV;
> > +
> > + ret = hv_iommu_detect(&hv_iommu_cap);
> > + if (ret) {
> > + pr_err("HVCALL_GET_IOMMU_CAPABILITIES failed: %d\n", ret);
>
> hv_iommu_detect() already outputs an error message in the failure case.
>
> > + return -ENODEV;
> > + }
> > +
> > + if (!hv_iommu_present(hv_iommu_cap.iommu_cap) ||
> > + !hv_iommu_s1_domain_supported(hv_iommu_cap.iommu_cap)) {
> > + pr_err("IOMMU capabilities not sufficient: cap=0x%llx\n",
> > + hv_iommu_cap.iommu_cap);
> > + return -ENODEV;
> > + }
> > +
> > + /*
> > + * The page table code only maps x86 page sizes (4K/2M/1G); require the
> > + * hypervisor to advertise a non-empty subset of exactly those.
> > + */
> > + if (!hv_iommu_cap.pgsize_bitmap ||
> > + (hv_iommu_cap.pgsize_bitmap & ~(u64)(SZ_4K | SZ_2M | SZ_1G))) {
> > + pr_err("unsupported page sizes: pgsize_bitmap=0x%llx\n",
> > + hv_iommu_cap.pgsize_bitmap);
> > + return -ENODEV;
> > + }
> > +
> > + iommu_detected = 1;
> > + pci_request_acs();
> > +
> > + hv_iommu = kzalloc_obj(*hv_iommu, GFP_KERNEL);
> > + if (!hv_iommu)
> > + return -ENOMEM;
> > +
> > + hv_init_iommu_device(hv_iommu, &hv_iommu_cap);
> > +
> > + ret = hv_initialize_static_domains();
> > + if (ret) {
> > + pr_err("static domains init failed: %d\n", ret);
> > + goto err_free;
> > + }
> > +
> > + ret = iommu_device_sysfs_add(&hv_iommu->iommu, NULL, NULL, "%s", "hv-iommu");
> > + if (ret) {
> > + pr_err("iommu_device_sysfs_add failed: %d\n", ret);
> > + goto err_delete_static_domains;
> > + }
> > +
> > + ret = iommu_device_register(&hv_iommu->iommu, &hv_iommu_ops, NULL);
> > + if (ret) {
> > + pr_err("iommu_device_register failed: %d\n", ret);
> > + goto err_sysfs_remove;
> > + }
> > +
> > + pr_info("successfully initialized\n");
> > + return 0;
> > +
> > +err_sysfs_remove:
> > + iommu_device_sysfs_remove(&hv_iommu->iommu);
> > +err_delete_static_domains:
> > + hv_delete_device_domain(&hv_blocking_domain);
> > + hv_delete_device_domain(&hv_identity_domain);
> > +err_free:
> > + kfree(hv_iommu);
> > + return ret;
> > +}
> > diff --git a/drivers/iommu/hyperv/iommu.h b/drivers/iommu/hyperv/iommu.h
> > new file mode 100644
> > index 000000000000..3a9f40fa2403
> > --- /dev/null
> > +++ b/drivers/iommu/hyperv/iommu.h
> > @@ -0,0 +1,51 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +
> > +/*
> > + * Hyper-V IOMMU driver.
> > + *
> > + * Copyright (C) 2024-2025, Microsoft, Inc.
> > + *
> > + */
> > +
> > +#ifndef _HYPERV_IOMMU_H
> > +#define _HYPERV_IOMMU_H
> > +
> > +struct hv_iommu_dev {
> > + struct iommu_device iommu;
> > + struct ida domain_ids;
> > +
> > + /* Device configuration */
> > + u8 max_iova_width;
> > + u8 max_pasid_width;
> > + u64 cap;
> > + u64 pgsize_bitmap;
> > +
> > + struct iommu_domain_geometry geometry;
> > + u64 first_domain;
> > + u64 last_domain;
> > +};
> > +
> > +struct hv_iommu_domain {
> > + union {
> > + struct iommu_domain domain;
> > + struct pt_iommu pt_iommu;
> > + struct pt_iommu_x86_64 pt_iommu_x86_64;
> > + };
> > + struct hv_iommu_dev *hv_iommu;
> > + struct hv_input_device_domain device_domain;
> > + u64 pgsize_bitmap;
> > +};
> > +
> > +PT_IOMMU_CHECK_DOMAIN(struct hv_iommu_domain, pt_iommu, domain);
> > +PT_IOMMU_CHECK_DOMAIN(struct hv_iommu_domain, pt_iommu_x86_64.iommu, domain);
> > +
> > +struct hv_iommu_endpoint {
> > + struct device *dev;
> > + struct hv_iommu_dev *hv_iommu;
> > + struct hv_iommu_domain *hv_domain;
> > +};
> > +
> > +#define to_hv_iommu_domain(d) \
> > + container_of(d, struct hv_iommu_domain, domain)
> > +
> > +#endif /* _HYPERV_IOMMU_H */
> > --
> > 2.52.0
> >
>
^ permalink raw reply
* [PATCH net-next] net: mana: Add debug knob to skip TX timeout recovery reset
From: Aditya Garg @ 2026-07-10 13:22 UTC (permalink / raw)
To: kys, haiyangz, wei.liu, decui, longli, andrew+netdev, davem,
edumazet, kuba, pabeni, kotaranov, horms, ernis, dipayanroy,
gargaditya, ssengar, gargaditya, linux-hyperv, netdev,
linux-kernel, linux-rdma
Add a per-port debugfs boolean "tx_timeout_skip_reset" that, when
enabled, makes mana_tx_timeout() log the TX timeout and return without
queueing the per-port detach/attach recovery work.
This is a debug-only aid for bringup and qualification: skipping the
recovery reset keeps the device and queue state intact so a TX timeout
can be correlated with hardware telemetry. The knob defaults to false,
so production recovery behaviour is unchanged.
Signed-off-by: Aditya Garg <gargaditya@linux.microsoft.com>
Reviewed-by: Haiyang Zhang <haiyangz@microsoft.com>
Reviewed-by: Dipayaan Roy <dipayanroy@linux.microsoft.com>
---
drivers/net/ethernet/microsoft/mana/mana_en.c | 14 ++++++++++++++
include/net/mana/mana.h | 3 +++
2 files changed, 17 insertions(+)
diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index 89e7f59f635d..a8c329bdbacf 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -892,6 +892,17 @@ static void mana_tx_timeout(struct net_device *netdev, unsigned int txqueue)
struct mana_context *ac = apc->ac;
struct gdma_context *gc = ac->gdma_dev->gdma_context;
+ /* Debug knob for bringup/qualification: when set, log the timeout and
+ * skip the reset so the failing state is preserved for telemetry.
+ * Disabled by default; production behaviour is unchanged.
+ */
+ if (READ_ONCE(apc->tx_timeout_skip_reset)) {
+ netdev_warn(netdev,
+ "TX timeout on queue %u: reset skipped (tx_timeout_skip_reset enabled)\n",
+ txqueue);
+ return;
+ }
+
/* Already in service, hence tx queue reset is not required.*/
if (test_bit(GC_IN_SERVICE, &gc->flags))
return;
@@ -3486,6 +3497,9 @@ static int mana_init_port(struct net_device *ndev)
&apc->steer_cqe_coalescing);
debugfs_create_u32("current_speed", 0400, apc->mana_port_debugfs,
&apc->speed);
+ debugfs_create_bool("tx_timeout_skip_reset", 0600,
+ apc->mana_port_debugfs,
+ &apc->tx_timeout_skip_reset);
return 0;
reset_apc:
diff --git a/include/net/mana/mana.h b/include/net/mana/mana.h
index 226b61504596..4d041fb8437f 100644
--- a/include/net/mana/mana.h
+++ b/include/net/mana/mana.h
@@ -530,6 +530,9 @@ struct mana_port_context {
struct net_device *ndev;
struct work_struct queue_reset_work;
+ /* Debug knob to log TX timeout but skip recovery reset */
+ bool tx_timeout_skip_reset;
+
u8 mac_addr[ETH_ALEN];
struct mana_eq *eqs;
--
2.43.0
^ permalink raw reply related
* Re: [PATCH v7 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Jason Gunthorpe @ 2026-07-10 16:27 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <178345362182.660027.12809852179204464964.stgit@skinsburskii>
On Tue, Jul 07, 2026 at 12:47:01PM -0700, Stanislav Kinsburskii wrote:
> +int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
> + unsigned long timeout)
> +{
> + struct mm_struct *mm = range->notifier->mm;
> + unsigned long deadline = 0;
> + int locked, ret;
> +
> + if (timeout)
> + deadline = jiffies + timeout;
> +
> + do {
> + if (fatal_signal_pending(current))
> + return -EINTR;
> +
> + if (timeout && time_after(jiffies, deadline))
> + return -EBUSY;
I really dislike there is a timeout here, HMM is supposed to be more
deterministic. GUP doesn't have a timeout, what is this about?
Jason
^ permalink raw reply
* Re: [PATCH v7 1/8] mm/hmm: move page fault handling out of walk callbacks
From: Jason Gunthorpe @ 2026-07-10 16:31 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <178345361483.660027.16455119612963295072.stgit@skinsburskii>
On Tue, Jul 07, 2026 at 12:46:54PM -0700, Stanislav Kinsburskii wrote:
> hmm_range_fault() currently triggers page faults from inside the page-table
> walk callbacks: hmm_vma_walk_pmd(), hmm_vma_walk_pud(),
> hmm_vma_walk_hugetlb_entry() and the pte-level helper all call
> hmm_vma_fault(), which in turn calls handle_mm_fault() while the walker
> still holds nested locks. The pte spinlock is dropped explicitly by each
> caller, and the hugetlb path manually drops and retakes
> hugetlb_vma_lock_read around the fault to dodge a deadlock against the walk
> framework's unconditional unlock.
>
> This layering does not extend cleanly to fault handlers that may release
> mmap_lock (VM_FAULT_RETRY, VM_FAULT_COMPLETED). If the lock is dropped
> while walk_page_range() is mid-traversal, the VMA can be freed before the
> walk framework's matching hugetlb_vma_unlock_read(), turning that unlock
> into a use-after-free.
>
> Split the responsibilities the way get_user_pages() does. Walk callbacks
> become inspect-only: when they detect a range that needs to be faulted in,
> they record it in struct hmm_vma_walk and return a private sentinel
> (HMM_FAULT_PENDING). The outer loop in hmm_range_fault() then drops out of
> walk_page_range(), invokes a new helper hmm_do_fault() that calls
> handle_mm_fault() with only mmap_lock held, and restarts the walk so the
> now-present entries are collected into hmm_pfns.
>
> No functional change for existing callers. As a side effect the hugetlb
> callback no longer needs the hugetlb_vma_{un}lock_read dance, and every
> fault-path exit from the callbacks now releases the pte spinlock on a
> single, common path. This refactor is also a precursor for adding an
> unlockable variant of hmm_range_fault() in a follow-up patch.
>
> Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
> ---
> mm/hmm.c | 118 +++++++++++++++++++++++++++++++++++++++-----------------------
> 1 file changed, 75 insertions(+), 43 deletions(-)
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Jason
^ permalink raw reply
* Re: [PATCH v7 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Jason Gunthorpe @ 2026-07-10 16:38 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <178345362182.660027.12809852179204464964.stgit@skinsburskii>
On Tue, Jul 07, 2026 at 12:47:01PM -0700, Stanislav Kinsburskii wrote:
> hmm_range_fault() requires the caller to hold the mmap read lock for the
> duration of the call. This is incompatible with mappings whose fault
> handler may release the mmap lock, notably userfaultfd-managed regions,
> where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED
> after dropping the lock. Drivers that need to populate device page tables
> for such mappings have no way to do so today.
sashiko could not apply v7 for some reason but the remarks on v6
seemed meaningful, did you see them were they delt with?
https://sashiko.dev/#/patchset/178336023903.504354.7500950448226027718.stgit%40skinsburskii
> diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst
> index 7d61b7a8b65b..70885f153d03 100644
> --- a/Documentation/mm/hmm.rst
> +++ b/Documentation/mm/hmm.rst
> @@ -208,6 +208,69 @@ invalidate() callback. That lock must be held before calling
> mmu_interval_read_retry() to avoid any race with a concurrent CPU page table
> update.
>
> +Dropping the mmap lock during page faults
> +=========================================
> +
> +Some VMAs have fault handlers that need to release the mmap lock while
> +servicing a fault (for example, regions managed by ``userfaultfd``).
> +``hmm_range_fault()`` cannot be used on such mappings because it must hold the
> +mmap lock for the duration of the call. Drivers that need to support them
> +should call::
Given the majority of callers use this API it should probably be the
focus of the documentation and example, regulate the existing API to a
'BTW if you really need the mmap lock, and you really shouldn't, this
exists too'
> @@ -32,6 +32,7 @@
>
> struct hmm_vma_walk {
> struct hmm_range *range;
> + int *locked;
Let's use bool if you have to respin this
> @@ -651,37 +663,33 @@ static int hmm_do_fault(struct mm_struct *mm,
> fault_flags |= FAULT_FLAG_WRITE;
> }
>
> - for (; addr < end; addr += PAGE_SIZE)
> - if (handle_mm_fault(vma, addr, fault_flags, NULL) &
> - VM_FAULT_ERROR)
> - return -EFAULT;
> + for (; addr < end; addr += PAGE_SIZE) {
> + vm_fault_t ret;
> +
> + ret = handle_mm_fault(vma, addr, fault_flags, NULL);
> +
> + if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
> + *hmm_vma_walk->locked = 0;
> + return HMM_FAULT_UNLOCKED;
> + }
> +
> + if (ret & VM_FAULT_ERROR) {
> + int err = vm_fault_to_errno(ret, 0);
> +
> + if (err)
> + return err;
> + BUG();
Linux will be upset if he sees this.
if (WARN_ON(!err))
err = -EINVAL
> +/**
> + * hmm_range_fault - try to fault some address in a virtual address range
> + * @range: argument structure
> + *
> + * Returns 0 on success or one of the following error codes:
> + *
> + * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
> + * (e.g., device file vma).
> + * -ENOMEM: Out of memory.
> + * -EPERM: Invalid permission (e.g., asking for write and range is read
> + * only).
> + * -EBUSY: The range has been invalidated and the caller needs to wait for
> + * the invalidation to finish.
> + * -EFAULT: A page was requested to be valid and could not be made valid
> + * ie it has no backing VMA or it is illegal to access
> + *
> + * This is similar to get_user_pages(), except that it can read the page tables
> + * without mutating them (ie causing faults).
> + *
> + * The mmap lock must be held by the caller and will remain held on return.
> + * For a variant that allows the mmap lock to be dropped during faults (e.g.,
> + * for userfaultfd support), see hmm_range_fault_unlocked_timeout().
> + */
Add a comment discourging anyone from using this function and prefer
hmm_range_fault_unlocked_timeout()
Other than the concern about the timeout and minor nits this looks
fine
Jason
^ permalink raw reply
* Re: [PATCH v7 4/8] mshv: Use hmm_range_fault_unlocked_timeout() for region faults
From: Jason Gunthorpe @ 2026-07-10 16:39 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <178345363584.660027.14063544694872741718.stgit@skinsburskii>
On Tue, Jul 07, 2026 at 12:47:15PM -0700, Stanislav Kinsburskii wrote:
> MSHV currently faults movable memory regions by taking mmap_read_lock()
> around hmm_range_fault(). That prevents the fault path from handling VMAs
> whose fault handlers need to drop mmap_lock, such as userfaultfd-backed
> mappings.
>
> Use hmm_range_fault_unlocked_timeout() instead. Passing a timeout of 0
> preserves MSHV's existing unbounded retry behavior while letting the HMM
> helper own mmap_lock acquisition and refresh range->notifier_seq internally
> before walking the range. After the fault succeeds, MSHV still takes
> mreg_mutex and checks mmu_interval_read_retry() before installing the pages
> into the region, so the existing invalidation synchronization is preserved.
>
> Fold the small fault-and-lock helper into mshv_region_range_fault(), since
> the remaining retry path is just the standard "fault, take the driver lock,
> check the interval notifier sequence" pattern.
>
> Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
> ---
> drivers/hv/mshv_regions.c | 54 ++++++++-------------------------------------
> 1 file changed, 10 insertions(+), 44 deletions(-)
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Jason
^ permalink raw reply
* Re: [PATCH v7 5/8] drm/nouveau: Use hmm_range_fault_unlocked_timeout() for SVM faults
From: Jason Gunthorpe @ 2026-07-10 16:40 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <178345364273.660027.15274510603185163961.stgit@skinsburskii>
On Tue, Jul 07, 2026 at 12:47:22PM -0700, Stanislav Kinsburskii wrote:
> nouveau_range_fault() takes mmap_read_lock() only to call
> hmm_range_fault(). It also keeps a single HMM_RANGE_DEFAULT_TIMEOUT
> deadline across both HMM -EBUSY retries and post-fault
> mmu_interval_read_retry() retries.
>
> Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns
> the mmap lock and refreshes range->notifier_seq for its internal retries.
> Nouveau keeps its existing absolute deadline in the outer loop and passes
> the remaining jiffies to the helper for each fault attempt, so retries
> caused by mmu_interval_read_retry() do not reset the overall retry budget.
>
> Nouveau still validates the interval notifier sequence while holding
> svmm->mutex before programming the GPU mapping.
>
> Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
> ---
> drivers/gpu/drm/nouveau/nouveau_svm.c | 11 +++--------
> 1 file changed, 3 insertions(+), 8 deletions(-)
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> - range.notifier_seq = mmu_interval_read_begin(range.notifier);
> - mmap_read_lock(mm);
> - ret = hmm_range_fault(&range);
> - mmap_read_unlock(mm);
> - if (ret) {
> - if (ret == -EBUSY)
> - continue;
> + ret = hmm_range_fault_unlocked_timeout(&range,
> + max_t(long, timeout - jiffies, 1UL));
Avoid max_t, 1L should be a signed long.
Though I'm even more concerned about the timeout if drivers are now
wrapping the whole thing in a timeout... That's not sensible at all
Jason
^ permalink raw reply
* Re: [PATCH v7 6/8] RDMA/umem: Use hmm_range_fault_unlocked_timeout() for ODP faults
From: Jason Gunthorpe @ 2026-07-10 16:43 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <178345364975.660027.8790629832830633290.stgit@skinsburskii>
On Tue, Jul 07, 2026 at 12:47:29PM -0700, Stanislav Kinsburskii wrote:
> ib_umem_odp_map_dma_and_lock() takes mmap_read_lock() only around
> hmm_range_fault(), then retries -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT
> expires.
>
> Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns
> the mmap lock and refreshes range->notifier_seq for its internal retries.
> ODP keeps using HMM_RANGE_DEFAULT_TIMEOUT for each HMM fault attempt,
> while interval invalidation retries continue to be handled by the existing
> outer loop.
>
> ODP still validates the interval notifier sequence while holding umem_mutex
> before DMA mapping pages.
>
> Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
> ---
> drivers/infiniband/core/umem_odp.c | 18 +++++-------------
> 1 file changed, 5 insertions(+), 13 deletions(-)
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Jason
^ permalink raw reply
* Re: [PATCH v7 7/8] accel/amdxdna: Use hmm_range_fault_unlocked_timeout() for range population
From: Jason Gunthorpe @ 2026-07-10 16:43 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <178345365679.660027.16671418103486907555.stgit@skinsburskii>
On Tue, Jul 07, 2026 at 12:47:36PM -0700, Stanislav Kinsburskii wrote:
> aie2_populate_range() takes mmap_read_lock() only around hmm_range_fault().
> It keeps a single HMM_RANGE_DEFAULT_TIMEOUT deadline for the populate pass
> and retries -EBUSY until that deadline expires.
>
> Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns
> the mmap lock and refreshes mapp->range.notifier_seq for its internal
> retries. Pass the remaining jiffies from the existing deadline to HMM,
> while preserving the driver's existing outer loop for interval invalidation
> retries and for selecting the next invalid mapping.
>
> Keep returning -ETIME when the retry budget expires, matching the driver's
> existing timeout error convention.
>
> Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
> ---
> drivers/accel/amdxdna/aie2_ctx.c | 17 +++--------------
> 1 file changed, 3 insertions(+), 14 deletions(-)
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Jason
^ permalink raw reply
* Re: [PATCH v7 8/8] drm/gpusvm: Use hmm_range_fault_unlocked_timeout() for range faults
From: Jason Gunthorpe @ 2026-07-10 16:44 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <178345366389.660027.12986386801605494596.stgit@skinsburskii>
On Tue, Jul 07, 2026 at 12:47:43PM -0700, Stanislav Kinsburskii wrote:
> Several GPU SVM paths take mmap_read_lock() only to call hmm_range_fault(),
> then retry -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT expires. Those paths use
> MMU interval notifiers whose mm matches the mm that was locked for the HMM
> fault.
>
> Use hmm_range_fault_unlocked_timeout() for those faults and pass the
> remaining retry budget to HMM. The helper owns mmap_lock acquisition and
> refreshes range->notifier_seq internally for each retry, while GPU SVM
> keeps its existing driver-lock validation with mmu_interval_read_retry()
> after a successful fault.
>
> Leave drm_gpusvm_check_pages() on hmm_range_fault() because that path is
> called with the mmap lock already held by its caller.
>
> Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
> ---
> drivers/gpu/drm/drm_gpusvm.c | 52 ++++++------------------------------------
> 1 file changed, 7 insertions(+), 45 deletions(-)
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
> - mmap_read_unlock(range->gpusvm->mm);
> + err = hmm_range_fault_unlocked_timeout(&hmm_range,
> + max_t(long, timeout - jiffies, 1));
Same remark about max_t
Jason
^ permalink raw reply
* Re: [PATCH v7 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Jason Gunthorpe @ 2026-07-10 16:47 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <20260710162749.GP118978@ziepe.ca>
On Fri, Jul 10, 2026 at 01:27:49PM -0300, Jason Gunthorpe wrote:
> On Tue, Jul 07, 2026 at 12:47:01PM -0700, Stanislav Kinsburskii wrote:
>
> > +int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
> > + unsigned long timeout)
> > +{
> > + struct mm_struct *mm = range->notifier->mm;
> > + unsigned long deadline = 0;
> > + int locked, ret;
> > +
> > + if (timeout)
> > + deadline = jiffies + timeout;
> > +
> > + do {
> > + if (fatal_signal_pending(current))
> > + return -EINTR;
> > +
> > + if (timeout && time_after(jiffies, deadline))
> > + return -EBUSY;
>
> I really dislike there is a timeout here, HMM is supposed to be more
> deterministic. GUP doesn't have a timeout, what is this about?
It looks like you've moved the timeout processing related to the mmnu
notifier sequence from the callers into this helper. I'm fine with
that, but maybe add some comments that this timeout is helping
implement the mmu notifiers, and we do expect that the HMM part will
not timeout.
Though it is a little hard to see how your stated purpose of enabling
userfaultfd is going to work, aren't you pretty much guarenteed to hit
a timeout if the userfaultfd process is adversly scheduled? That's
going to end up broken.
So, maybe the deadline should be resetting after every handled fault?
ie the timeout really is only about the mmu notifier and we don't
count the time spent handling faults or walking?
Jason
^ permalink raw reply
* Re: [PATCH v7 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Stanislav Kinsburskii @ 2026-07-10 16:48 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <20260710162749.GP118978@ziepe.ca>
On Fri, Jul 10, 2026 at 01:27:49PM -0300, Jason Gunthorpe wrote:
> On Tue, Jul 07, 2026 at 12:47:01PM -0700, Stanislav Kinsburskii wrote:
>
> > +int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
> > + unsigned long timeout)
> > +{
> > + struct mm_struct *mm = range->notifier->mm;
> > + unsigned long deadline = 0;
> > + int locked, ret;
> > +
> > + if (timeout)
> > + deadline = jiffies + timeout;
> > +
> > + do {
> > + if (fatal_signal_pending(current))
> > + return -EINTR;
> > +
> > + if (timeout && time_after(jiffies, deadline))
> > + return -EBUSY;
>
> I really dislike there is a timeout here, HMM is supposed to be more
> deterministic. GUP doesn't have a timeout, what is this about?
>
The timeout was added because this version makes the unlocked helper own
the internal retry loop, including -EBUSY retries.
Several existing HMM users already bound those retries with
HMM_RANGE_DEFAULT_TIMEOUT, so the timeout argument was meant to avoid
silently turning those call sites into unbounded waits.
That said, I agree this mixes driver retry policy into HMM. A cleaner
split may be to make the new helper timeout-free and have it retry
internally only when mmap_lock was dropped by the fault path, since that
is the part HMM must hide for userfaultfd-style faults.
If the walk fails with the normal HMM invalidation -EBUSY, the helper
would return -EBUSY to the caller, preserving the existing driver
timeout loops and keeping HMM deterministic, closer to GUP.
Is it better from your POV?
THanks,
Stanislav
> Jason
^ permalink raw reply
* Re: [PATCH v7 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Stanislav Kinsburskii @ 2026-07-10 17:02 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <20260710164743.GX118978@ziepe.ca>
On Fri, Jul 10, 2026 at 01:47:43PM -0300, Jason Gunthorpe wrote:
> On Fri, Jul 10, 2026 at 01:27:49PM -0300, Jason Gunthorpe wrote:
> > On Tue, Jul 07, 2026 at 12:47:01PM -0700, Stanislav Kinsburskii wrote:
> >
> > > +int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
> > > + unsigned long timeout)
> > > +{
> > > + struct mm_struct *mm = range->notifier->mm;
> > > + unsigned long deadline = 0;
> > > + int locked, ret;
> > > +
> > > + if (timeout)
> > > + deadline = jiffies + timeout;
> > > +
> > > + do {
> > > + if (fatal_signal_pending(current))
> > > + return -EINTR;
> > > +
> > > + if (timeout && time_after(jiffies, deadline))
> > > + return -EBUSY;
> >
> > I really dislike there is a timeout here, HMM is supposed to be more
> > deterministic. GUP doesn't have a timeout, what is this about?
>
> It looks like you've moved the timeout processing related to the mmnu
> notifier sequence from the callers into this helper. I'm fine with
> that, but maybe add some comments that this timeout is helping
> implement the mmu notifiers, and we do expect that the HMM part will
> not timeout.
>
> Though it is a little hard to see how your stated purpose of enabling
> userfaultfd is going to work, aren't you pretty much guarenteed to hit
> a timeout if the userfaultfd process is adversly scheduled? That's
> going to end up broken.
>
The main customer for this new feature I have in mind is the MSHV driver
which backs VMs memory with HMM, requires userfaultfd support for
post-copy live migration and fast restore and it doesn't timeout.
I agree, that this current timeout value used by the other callers might
not be enough to repopulate the mappings with userfaultfd, but there
drivers would get -EFAULT for uderfaultfd-backed mappings without this
change anyway, so getting -EBUSY with the change instead doesn't look
like a significant change to the behaviour from my POV.
> So, maybe the deadline should be resetting after every handled fault?
> ie the timeout really is only about the mmu notifier and we don't
> count the time spent handling faults or walking?
>
The timeout was inherited from existing HMM users rather than introduced
as a new HMM policy. Some GPU drivers use HMM_RANGE_DEFAULT_TIMEOUT as a
budget for the whole range population operation, including HMM retries
and subsequent driver mapping work.
Moving retry handling into the unlocked helper would otherwise hide
repeated -EBUSY returns from those callers and silently turn their
bounded operation into an unbounded one. So the timeout argument is
there to preserve existing caller semantics during conversion.
I'd say it's up to the caller to provide big enough timeout for
userfaultfd to succeed.
Thanks,
Stanislav
> Jason
^ permalink raw reply
* Re: [PATCH v7 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Stanislav Kinsburskii @ 2026-07-10 17:06 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <20260710163835.GR118978@ziepe.ca>
On Fri, Jul 10, 2026 at 01:38:35PM -0300, Jason Gunthorpe wrote:
> On Tue, Jul 07, 2026 at 12:47:01PM -0700, Stanislav Kinsburskii wrote:
> > hmm_range_fault() requires the caller to hold the mmap read lock for the
> > duration of the call. This is incompatible with mappings whose fault
> > handler may release the mmap lock, notably userfaultfd-managed regions,
> > where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED
> > after dropping the lock. Drivers that need to populate device page tables
> > for such mappings have no way to do so today.
>
> sashiko could not apply v7 for some reason but the remarks on v6
> seemed meaningful, did you see them were they delt with?
>
Yes, I dealt with them.
> https://sashiko.dev/#/patchset/178336023903.504354.7500950448226027718.stgit%40skinsburskii
>
> > diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst
> > index 7d61b7a8b65b..70885f153d03 100644
> > --- a/Documentation/mm/hmm.rst
> > +++ b/Documentation/mm/hmm.rst
> > @@ -208,6 +208,69 @@ invalidate() callback. That lock must be held before calling
> > mmu_interval_read_retry() to avoid any race with a concurrent CPU page table
> > update.
> >
> > +Dropping the mmap lock during page faults
> > +=========================================
> > +
> > +Some VMAs have fault handlers that need to release the mmap lock while
> > +servicing a fault (for example, regions managed by ``userfaultfd``).
> > +``hmm_range_fault()`` cannot be used on such mappings because it must hold the
> > +mmap lock for the duration of the call. Drivers that need to support them
> > +should call::
>
> Given the majority of callers use this API it should probably be the
> focus of the documentation and example, regulate the existing API to a
> 'BTW if you really need the mmap lock, and you really shouldn't, this
> exists too'
>
Sure, I'll update the doc to reflect it this way.
> > @@ -32,6 +32,7 @@
> >
> > struct hmm_vma_walk {
> > struct hmm_range *range;
> > + int *locked;
>
> Let's use bool if you have to respin this
>
Sure.
> > @@ -651,37 +663,33 @@ static int hmm_do_fault(struct mm_struct *mm,
> > fault_flags |= FAULT_FLAG_WRITE;
> > }
> >
> > - for (; addr < end; addr += PAGE_SIZE)
> > - if (handle_mm_fault(vma, addr, fault_flags, NULL) &
> > - VM_FAULT_ERROR)
> > - return -EFAULT;
> > + for (; addr < end; addr += PAGE_SIZE) {
> > + vm_fault_t ret;
> > +
> > + ret = handle_mm_fault(vma, addr, fault_flags, NULL);
> > +
> > + if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
> > + *hmm_vma_walk->locked = 0;
> > + return HMM_FAULT_UNLOCKED;
> > + }
> > +
> > + if (ret & VM_FAULT_ERROR) {
> > + int err = vm_fault_to_errno(ret, 0);
> > +
> > + if (err)
> > + return err;
> > + BUG();
>
> Linux will be upset if he sees this.
>
> if (WARN_ON(!err))
> err = -EINVAL
>
It will. I copied it from GUP.
I'll change it the way you propose it.
> > +/**
> > + * hmm_range_fault - try to fault some address in a virtual address range
> > + * @range: argument structure
> > + *
> > + * Returns 0 on success or one of the following error codes:
> > + *
> > + * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
> > + * (e.g., device file vma).
> > + * -ENOMEM: Out of memory.
> > + * -EPERM: Invalid permission (e.g., asking for write and range is read
> > + * only).
> > + * -EBUSY: The range has been invalidated and the caller needs to wait for
> > + * the invalidation to finish.
> > + * -EFAULT: A page was requested to be valid and could not be made valid
> > + * ie it has no backing VMA or it is illegal to access
> > + *
> > + * This is similar to get_user_pages(), except that it can read the page tables
> > + * without mutating them (ie causing faults).
> > + *
> > + * The mmap lock must be held by the caller and will remain held on return.
> > + * For a variant that allows the mmap lock to be dropped during faults (e.g.,
> > + * for userfaultfd support), see hmm_range_fault_unlocked_timeout().
> > + */
>
> Add a comment discourging anyone from using this function and prefer
> hmm_range_fault_unlocked_timeout()
>
Will do.
Thanks,
Stanislav
> Other than the concern about the timeout and minor nits this looks
> fine
>
> Jason
^ permalink raw reply
* Re: [PATCH v7 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Jason Gunthorpe @ 2026-07-10 18:03 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <alElv0EKjLQXMNK8@skinsburskii>
On Fri, Jul 10, 2026 at 10:02:55AM -0700, Stanislav Kinsburskii wrote:
> The main customer for this new feature I have in mind is the MSHV driver
> which backs VMs memory with HMM, requires userfaultfd support for
> post-copy live migration and fast restore and it doesn't timeout.
>
> I agree, that this current timeout value used by the other callers might
> not be enough to repopulate the mappings with userfaultfd, but there
> drivers would get -EFAULT for uderfaultfd-backed mappings without this
> change anyway, so getting -EBUSY with the change instead doesn't look
> like a significant change to the behaviour from my POV.
It sounds like it won't be reliable either then.
> > So, maybe the deadline should be resetting after every handled fault?
> > ie the timeout really is only about the mmu notifier and we don't
> > count the time spent handling faults or walking?
>
> The timeout was inherited from existing HMM users rather than introduced
> as a new HMM policy. Some GPU drivers use HMM_RANGE_DEFAULT_TIMEOUT as a
> budget for the whole range population operation, including HMM retries
> and subsequent driver mapping work.
Yes, because we always had a timeout around the notifier because that
scheme can sort of live lock. The timeout was to protect that only, ie
limit the number of notifier retries.
Expanding the timeout to be outside what is bounded by the notifier
retry is not right, and heavy stuff like mapping should be done after
the hmm side succeeds and the notifiers concluded so they can rely on
normal locking instead.
This is why I'm suggesting to reset the deadline as hmm makes forward
progress, we really only want to bound the notifier retry loop not
anything else.
Jason
^ permalink raw reply
* Re: [PATCH v7 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Stanislav Kinsburskii @ 2026-07-10 18:07 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, surenb, tzimmermann, vbabka,
wei.liu, dri-devel, linux-mm, linux-doc, linux-hyperv,
linux-kernel, linux-kselftest, linux-rdma
In-Reply-To: <20260710180312.GD1803712@ziepe.ca>
On Fri, Jul 10, 2026 at 03:03:12PM -0300, Jason Gunthorpe wrote:
> On Fri, Jul 10, 2026 at 10:02:55AM -0700, Stanislav Kinsburskii wrote:
> > The main customer for this new feature I have in mind is the MSHV driver
> > which backs VMs memory with HMM, requires userfaultfd support for
> > post-copy live migration and fast restore and it doesn't timeout.
> >
> > I agree, that this current timeout value used by the other callers might
> > not be enough to repopulate the mappings with userfaultfd, but there
> > drivers would get -EFAULT for uderfaultfd-backed mappings without this
> > change anyway, so getting -EBUSY with the change instead doesn't look
> > like a significant change to the behaviour from my POV.
>
> It sounds like it won't be reliable either then.
>
> > > So, maybe the deadline should be resetting after every handled fault?
> > > ie the timeout really is only about the mmu notifier and we don't
> > > count the time spent handling faults or walking?
> >
> > The timeout was inherited from existing HMM users rather than introduced
> > as a new HMM policy. Some GPU drivers use HMM_RANGE_DEFAULT_TIMEOUT as a
> > budget for the whole range population operation, including HMM retries
> > and subsequent driver mapping work.
>
> Yes, because we always had a timeout around the notifier because that
> scheme can sort of live lock. The timeout was to protect that only, ie
> limit the number of notifier retries.
>
> Expanding the timeout to be outside what is bounded by the notifier
> retry is not right, and heavy stuff like mapping should be done after
> the hmm side succeeds and the notifiers concluded so they can rely on
> normal locking instead.
>
> This is why I'm suggesting to reset the deadline as hmm makes forward
> progress, we really only want to bound the notifier retry loop not
> anything else.
>
Sure, I'll modify accordingly.
Thanks,
Stanislav
> Jason
^ permalink raw reply
* RE: [PATCH net-next v3] net: mana: Add handler for sriov configure
From: Haiyang Zhang @ 2026-07-10 19:19 UTC (permalink / raw)
To: Haiyang Zhang, linux-hyperv@vger.kernel.org,
netdev@vger.kernel.org, KY Srinivasan, Wei Liu, Dexuan Cui,
Long Li, Andrew Lunn, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Erni Sri Satya Vennela,
Dipayaan Roy, Aditya Garg, Shradha Gupta,
linux-kernel@vger.kernel.org
Cc: Paul Rosswurm
In-Reply-To: <20260708205924.2408673-1-haiyangz@linux.microsoft.com>
> -----Original Message-----
> From: Haiyang Zhang <haiyangz@linux.microsoft.com>
> Sent: Wednesday, July 8, 2026 4:59 PM
> To: linux-hyperv@vger.kernel.org; netdev@vger.kernel.org; KY Srinivasan
> <kys@microsoft.com>; Haiyang Zhang <haiyangz@microsoft.com>; Wei Liu
> <wei.liu@kernel.org>; Dexuan Cui <DECUI@microsoft.com>; Long Li
> <longli@microsoft.com>; Andrew Lunn <andrew+netdev@lunn.ch>; David S.
> Miller <davem@davemloft.net>; Eric Dumazet <edumazet@google.com>; Jakub
> Kicinski <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; Simon Horman
> <horms@kernel.org>; Erni Sri Satya Vennela <ernis@linux.microsoft.com>;
> Dipayaan Roy <dipayanroy@linux.microsoft.com>; Aditya Garg
> <gargaditya@linux.microsoft.com>; Shradha Gupta
> <shradhagupta@linux.microsoft.com>; linux-kernel@vger.kernel.org
> Cc: Paul Rosswurm <paulros@microsoft.com>
> Subject: [PATCH net-next v3] net: mana: Add handler for sriov configure
>
> From: Haiyang Zhang <haiyangz@microsoft.com>
>
> Add callback function for the pci_driver / sriov_configure.
>
> It asks the NIC to provide certain number of VFs, or disable
> VFs if the request is zero.
>
> Signed-off-by: Haiyang Zhang <haiyangz@microsoft.com>
> ---
> v3:
> Updated sriov disabling paths suggested by Paolo Abeni
>
> v2:
> No longer change VF autoprobe as discussed with Leon Romanovsky and
> Bjorn Helgaas.
>
> ---
> .../net/ethernet/microsoft/mana/gdma_main.c | 26 +++++++++++++++++++
> 1 file changed, 26 insertions(+)
>
> diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c
> b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> index aef3b77229c1..80a9118a90bc 100644
> --- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
> +++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
> @@ -2456,6 +2456,8 @@ static void mana_gd_remove(struct pci_dev *pdev)
> {
> struct gdma_context *gc = pci_get_drvdata(pdev);
>
> + pci_disable_sriov(pdev);
> +
> mana_rdma_remove(&gc->mana_ib);
> mana_remove(&gc->mana, false);
>
> @@ -2517,6 +2519,8 @@ static void mana_gd_shutdown(struct pci_dev *pdev)
>
> dev_info(&pdev->dev, "Shutdown was called\n");
>
> + pci_disable_sriov(pdev);
> +
I will remove this unnecessary pci_disable_sriov() as found by AI review,
and submit an updated patch.
- Haiyang
^ permalink raw reply
* [PATCH net-next v4] net: mana: Add handler for sriov configure
From: Haiyang Zhang @ 2026-07-10 19:27 UTC (permalink / raw)
To: linux-hyperv, netdev, K. Y. Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui, Long Li, Andrew Lunn, David S. Miller, Eric Dumazet,
Jakub Kicinski, Paolo Abeni, Simon Horman, Erni Sri Satya Vennela,
Dipayaan Roy, Aditya Garg, Shradha Gupta, linux-kernel
Cc: paulros
From: Haiyang Zhang <haiyangz@microsoft.com>
Add callback function for the pci_driver / sriov_configure.
It asks the NIC to provide certain number of VFs, or disable
VFs if the request is zero.
Signed-off-by: Haiyang Zhang <haiyangz@microsoft.com>
---
v4:
Removed unnecessary pci_disable_sriov() from mana_gd_shutdown() as
suggested by AI review.
v3:
Updated sriov disabling paths suggested by Paolo Abeni
v2:
No longer change VF autoprobe as discussed with Leon Romanovsky and Bjorn Helgaas.
---
.../net/ethernet/microsoft/mana/gdma_main.c | 24 +++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/drivers/net/ethernet/microsoft/mana/gdma_main.c b/drivers/net/ethernet/microsoft/mana/gdma_main.c
index aef3b77229c1..a38d4bb74621 100644
--- a/drivers/net/ethernet/microsoft/mana/gdma_main.c
+++ b/drivers/net/ethernet/microsoft/mana/gdma_main.c
@@ -2456,6 +2456,8 @@ static void mana_gd_remove(struct pci_dev *pdev)
{
struct gdma_context *gc = pci_get_drvdata(pdev);
+ pci_disable_sriov(pdev);
+
mana_rdma_remove(&gc->mana_ib);
mana_remove(&gc->mana, false);
@@ -2525,6 +2527,27 @@ static void mana_gd_shutdown(struct pci_dev *pdev)
pci_disable_device(pdev);
}
+static int mana_sriov_configure(struct pci_dev *pdev, int numvfs)
+{
+ int err = 0;
+
+ dev_info(&pdev->dev, "Requested num VFs: %d\n", numvfs);
+
+ if (numvfs > 0) {
+ err = pci_enable_sriov(pdev, numvfs);
+ } else {
+ if (pci_vfs_assigned(pdev)) {
+ dev_warn(&pdev->dev,
+ "Cannot disable SR-IOV while VFs are assigned\n");
+ return -EPERM;
+ }
+
+ pci_disable_sriov(pdev);
+ }
+
+ return err ? err : numvfs;
+}
+
static const struct pci_device_id mana_id_table[] = {
{ PCI_DEVICE(PCI_VENDOR_ID_MICROSOFT, MANA_PF_DEVICE_ID) },
{ PCI_DEVICE(PCI_VENDOR_ID_MICROSOFT, MANA_PF2_DEVICE_ID) },
@@ -2540,6 +2563,7 @@ static struct pci_driver mana_driver = {
.suspend = mana_gd_suspend,
.resume = mana_gd_resume,
.shutdown = mana_gd_shutdown,
+ .sriov_configure = mana_sriov_configure,
};
static int __init mana_driver_init(void)
--
2.34.1
^ permalink raw reply related
* [PATCH v8 0/8] mm/hmm: Add mmap lock-drop support for userfaultfd-backed mappings
From: Stanislav Kinsburskii @ 2026-07-10 21:26 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
This series extends the HMM framework to support userfaultfd-backed memory
by allowing the mmap read lock to be dropped during hmm_range_fault().
Some page fault handlers — most notably userfaultfd — require the mmap lock
to be released so that userspace can resolve the fault. The current HMM
interface never sets FAULT_FLAG_ALLOW_RETRY, making it impossible to fault
in pages from userfaultfd-registered regions.
This series follows the established int *locked pattern from
get_user_pages_remote() in mm/gup.c. A new helper function,
hmm_range_fault_locked(), accepts an int *locked parameter. When the
mmap lock is dropped during fault resolution (VM_FAULT_RETRY or
VM_FAULT_COMPLETED), the function returns 0 with *locked = 0, signalling
the caller to restart its walk. The existing hmm_range_fault() is
refactored into a thin wrapper that passes NULL, preserving current
behavior for all existing callers.
Possible approaches to lift this limitation are documented in
Documentation/mm/hmm.rst.
Changes in v8:
- Make hmm_range_fault_unlocked_timeout() the primary documented HMM
range-fault API, and move hmm_range_fault() into the “use only if the
caller really must hold mmap_lock” category.
- Clarify that the timeout is a retry budget for repeated mmu-notifier
invalidation retries. HMM does not interrupt an in-progress page fault
when the timeout expires.
- Restart the retry timeout only when handle_mm_fault() dropped
mmap_lock, because that indicates a lock-dropping fault handler such as
userfaultfd made progress. Ordinary -EBUSY retries keep the existing
deadline.
- Remove the attempted timeout selftest. The remaining selftest covers the
intended userfaultfd path by resolving missing-page faults through
HMM_DMIRROR_READ_UNLOCKED and hmm_range_fault_unlocked_timeout(..., 0).
Changes in v7:
- Replaced the unlocked HMM API with
hmm_range_fault_unlocked_timeout(). The helper now takes a timeout in
jiffies, with 0 meaning retry indefinitely.
- Moved -EBUSY retry handling into the HMM helper for the unlocked path.
The helper refreshes range->notifier_seq internally before each retry.
- Switched the unlocked path to mmap_read_lock_killable() and return
-EINTR if mmap lock acquisition is interrupted or a fatal signal is
pending during retry handling.
- Removed the redundant non-timeout hmm_range_fault_unlocked() interface.
- Updated Documentation/mm/hmm.rst and kernel-doc to describe the timeout API
and the intended caller pattern.
- Updated the HMM selftests to use hmm_range_fault_unlocked_timeout()
only, including coverage for the finite-timeout path.
- Added in-tree users of the new helper:
- mshv
- nouveau
- RDMA/umem
- amdxdna
- drm/gpusvm
- Preserved each converted driver’s existing timeout convention:
- unbounded retry where the old code retried indefinitely,
- HMM_RANGE_DEFAULT_TIMEOUT where the old code used that budget,
- existing driver-specific timeout return values such as -ETIME.
- Used max_t(long, timeout - jiffies, 1) when passing remaining time from
absolute jiffies deadlines to avoid unsigned underflow while keeping a
minimum one-jiffy retry window.
- Left callers on hmm_range_fault() when they already need to hold
mmap_lock across surrounding work, such as drm_gpusvm_check_pages().
Changes in v6:
- Reworked the new API from the external int *locked pattern to
hmm_range_fault_unlocked(), which owns mmap_read_lock() internally.
- Changed the dropped-lock contract: hmm_range_fault_unlocked() now returns
-EBUSY when the mmap lock is dropped, and callers restart with a fresh
mmu_interval_read_begin() sequence.
- Kept hmm_range_fault() as the locked variant for existing users, preserving
its caller-held mmap lock contract.
- Added an in-tree user by converting the MSHV region fault path to
hmm_range_fault_unlocked().
- Updated Documentation/mm/hmm.rst and kernel-doc to describe the unlocked
helper and retry pattern.
- Updated commit messages to match the new API and return semantics.
- Kept the userfaultfd HMM selftest using the test_hmm unlocked read ioctl
path.
Changes in v5:
- Rework hmm_range_fault_unlockable() retry handling to retry
VM_FAULT_RETRY internally with FAULT_FLAG_TRIED set, matching the
fixup_user_fault() pattern and avoiding repeated first-retry lock drops.
- Distinguish VM_FAULT_RETRY from VM_FAULT_COMPLETED: retry faults now
reacquire the mmap lock internally, while completed faults return to the
caller with *locked = 0 so the caller can restart with a fresh notifier
sequence.
- Document the two *locked return states, including the -EINTR case when a
fatal signal is pending after the mmap lock has already been dropped.
- Update comments around HMM_FAULT_UNLOCKED and the HMM fault loop to match
the current hmm_range_fault_unlockable() implementation.
Changes in v4:
- Rebased on 7.2-rc1
Changes in v3:
- Return -EFAULT from dmirror_fault_unlockable() when the mirrored mm can
no longer be pinned.
- Add an eventfd stop signal for the userfaultfd handler thread to avoid
waiting for the poll timeout on successful test completion.
Changes in v2:
- Split into a preparatory refactor (new patch 1) that moves
handle_mm_fault() out of the walk callbacks, plus a smaller feature
patch on top. Suggested by David Hildenbrand.
- Hugetlb regions are now supported on the unlockable path; the v1
-EFAULT short-circuit and the hugetlb_vma_lock_read drop/retake
dance are gone.
- Distinct internal sentinels for "needs fault" (HMM_FAULT_PENDING)
and "lock dropped" (HMM_FAULT_UNLOCKED).
- Outer loop now re-walks after a successful internal fault so the
faulted pfns end up in range->hmm_pfns.
- Kernel-doc on hmm_range_fault_unlockable() and the
Documentation/mm/hmm.rst example match the implementation.
- Dropped the mshv driver conversion (v1 patch 2); will post
separately.
- Selftest converted to drive the path through test_hmm with a
userfaultfd handler (new HMM_DMIRROR_READ_UNLOCKABLE ioctl).
---
Stanislav Kinsburskii (8):
mm/hmm: move page fault handling out of walk callbacks
mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
selftests/mm: add HMM test for mmap lock-dropping faults
mshv: Use hmm_range_fault_unlocked_timeout() for region faults
drm/nouveau: Use hmm_range_fault_unlocked_timeout() for SVM faults
RDMA/umem: Use hmm_range_fault_unlocked_timeout() for ODP faults
accel/amdxdna: Use hmm_range_fault_unlocked_timeout() for range population
drm/gpusvm: Use hmm_range_fault_unlocked_timeout() for range faults
Documentation/mm/hmm.rst | 76 +++++++--
drivers/accel/amdxdna/aie2_ctx.c | 17 --
drivers/gpu/drm/drm_gpusvm.c | 52 +-----
drivers/gpu/drm/nouveau/nouveau_svm.c | 12 -
drivers/hv/mshv_regions.c | 54 +------
drivers/infiniband/core/umem_odp.c | 18 +-
include/linux/hmm.h | 2
lib/test_hmm.c | 107 +++++++++++++
lib/test_hmm_uapi.h | 1
mm/hmm.c | 259 +++++++++++++++++++++++++-------
tools/testing/selftests/mm/hmm-tests.c | 150 +++++++++++++++++++
11 files changed, 541 insertions(+), 207 deletions(-)
^ permalink raw reply
* [PATCH v8 1/8] mm/hmm: move page fault handling out of walk callbacks
From: Stanislav Kinsburskii @ 2026-07-10 21:26 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178371866223.900500.12312667138651735591.stgit@skinsburskii>
hmm_range_fault() currently triggers page faults from inside the page-table
walk callbacks: hmm_vma_walk_pmd(), hmm_vma_walk_pud(),
hmm_vma_walk_hugetlb_entry() and the pte-level helper all call
hmm_vma_fault(), which in turn calls handle_mm_fault() while the walker
still holds nested locks. The pte spinlock is dropped explicitly by each
caller, and the hugetlb path manually drops and retakes
hugetlb_vma_lock_read around the fault to dodge a deadlock against the walk
framework's unconditional unlock.
This layering does not extend cleanly to fault handlers that may release
mmap_lock (VM_FAULT_RETRY, VM_FAULT_COMPLETED). If the lock is dropped
while walk_page_range() is mid-traversal, the VMA can be freed before the
walk framework's matching hugetlb_vma_unlock_read(), turning that unlock
into a use-after-free.
Split the responsibilities the way get_user_pages() does. Walk callbacks
become inspect-only: when they detect a range that needs to be faulted in,
they record it in struct hmm_vma_walk and return a private sentinel
(HMM_FAULT_PENDING). The outer loop in hmm_range_fault() then drops out of
walk_page_range(), invokes a new helper hmm_do_fault() that calls
handle_mm_fault() with only mmap_lock held, and restarts the walk so the
now-present entries are collected into hmm_pfns.
No functional change for existing callers. As a side effect the hugetlb
callback no longer needs the hugetlb_vma_{un}lock_read dance, and every
fault-path exit from the callbacks now releases the pte spinlock on a
single, common path. This refactor is also a precursor for adding an
unlockable variant of hmm_range_fault() in a follow-up patch.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
---
mm/hmm.c | 118 +++++++++++++++++++++++++++++++++++++++-----------------------
1 file changed, 75 insertions(+), 43 deletions(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index e5c1f4deed24..bc9361a715fa 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -33,8 +33,17 @@
struct hmm_vma_walk {
struct hmm_range *range;
unsigned long last;
+ unsigned long end;
+ unsigned int required_fault;
};
+/*
+ * Internal sentinel returned by walk callbacks when they need a page fault.
+ * The callback stores end/required_fault in hmm_vma_walk; the outer loop
+ * consumes the sentinel and never propagates it to the caller.
+ */
+#define HMM_FAULT_PENDING -EAGAIN
+
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -60,37 +69,25 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
}
/*
- * hmm_vma_fault() - fault in a range lacking valid pmd or pte(s)
- * @addr: range virtual start address (inclusive)
- * @end: range virtual end address (exclusive)
- * @required_fault: HMM_NEED_* flags
- * @walk: mm_walk structure
- * Return: -EBUSY after page fault, or page fault error
+ * hmm_record_fault() - record a range that needs to be faulted in
*
- * This function will be called whenever pmd_none() or pte_none() returns true,
- * or whenever there is no page directory covering the virtual address range.
+ * Called by the walk callbacks when they discover that part of the range
+ * needs a page fault. The callback records what to fault and returns
+ * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of
+ * walk_page_range() and invokes handle_mm_fault() from a context where no
+ * page-table or hugetlb_vma_lock is held.
*/
-static int hmm_vma_fault(unsigned long addr, unsigned long end,
- unsigned int required_fault, struct mm_walk *walk)
+static int hmm_record_fault(unsigned long addr, unsigned long end,
+ unsigned int required_fault,
+ struct mm_walk *walk)
{
struct hmm_vma_walk *hmm_vma_walk = walk->private;
- struct vm_area_struct *vma = walk->vma;
- unsigned int fault_flags = FAULT_FLAG_REMOTE;
WARN_ON_ONCE(!required_fault);
hmm_vma_walk->last = addr;
-
- if (required_fault & HMM_NEED_WRITE_FAULT) {
- if (!(vma->vm_flags & VM_WRITE))
- return -EPERM;
- fault_flags |= FAULT_FLAG_WRITE;
- }
-
- for (; addr < end; addr += PAGE_SIZE)
- if (handle_mm_fault(vma, addr, fault_flags, NULL) &
- VM_FAULT_ERROR)
- return -EFAULT;
- return -EBUSY;
+ hmm_vma_walk->end = end;
+ hmm_vma_walk->required_fault = required_fault;
+ return HMM_FAULT_PENDING;
}
static unsigned int hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk,
@@ -174,7 +171,7 @@ static int hmm_vma_walk_hole(unsigned long addr, unsigned long end,
return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR);
}
if (required_fault)
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
return hmm_pfns_fill(addr, end, range, 0);
}
@@ -209,7 +206,7 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr,
required_fault =
hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags);
if (required_fault)
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) {
@@ -328,7 +325,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
fault:
pte_unmap(ptep);
/* Fault any virtual address we were asked to fault */
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
}
#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
@@ -371,7 +368,7 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
npages, 0);
if (required_fault) {
if (softleaf_is_device_private(entry))
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
else
return -EFAULT;
}
@@ -517,7 +514,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end,
npages, cpu_flags);
if (required_fault) {
spin_unlock(ptl);
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
}
pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
@@ -564,21 +561,8 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask,
required_fault =
hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, cpu_flags);
if (required_fault) {
- int ret;
-
spin_unlock(ptl);
- hugetlb_vma_unlock_read(vma);
- /*
- * Avoid deadlock: drop the vma lock before calling
- * hmm_vma_fault(), which will itself potentially take and
- * drop the vma lock. This is also correct from a
- * protection point of view, because there is no further
- * use here of either pte or ptl after dropping the vma
- * lock.
- */
- ret = hmm_vma_fault(addr, end, required_fault, walk);
- hugetlb_vma_lock_read(vma);
- return ret;
+ return hmm_record_fault(addr, end, required_fault, walk);
}
pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT);
@@ -637,6 +621,44 @@ static const struct mm_walk_ops hmm_walk_ops = {
.walk_lock = PGWALK_RDLOCK,
};
+/*
+ * hmm_do_fault - fault in a range recorded by a walk callback
+ *
+ * Called from the outer loop in hmm_range_fault() after a callback
+ * returned HMM_FAULT_PENDING. At this point we hold only mmap_lock;
+ * the page-table spinlock and any hugetlb_vma_lock acquired by the walk
+ * framework have already been released by the unwind.
+ *
+ * Returns -EBUSY on success (all pages faulted, caller should re-walk).
+ * Returns a negative errno on failure.
+ */
+static int hmm_do_fault(struct mm_struct *mm,
+ struct hmm_vma_walk *hmm_vma_walk)
+{
+ unsigned long addr = hmm_vma_walk->last;
+ unsigned long end = hmm_vma_walk->end;
+ unsigned int required_fault = hmm_vma_walk->required_fault;
+ unsigned int fault_flags = FAULT_FLAG_REMOTE;
+ struct vm_area_struct *vma;
+
+ vma = vma_lookup(mm, addr);
+ if (!vma)
+ return -EFAULT;
+
+ if (required_fault & HMM_NEED_WRITE_FAULT) {
+ if (!(vma->vm_flags & VM_WRITE))
+ return -EPERM;
+ fault_flags |= FAULT_FLAG_WRITE;
+ }
+
+ for (; addr < end; addr += PAGE_SIZE)
+ if (handle_mm_fault(vma, addr, fault_flags, NULL) &
+ VM_FAULT_ERROR)
+ return -EFAULT;
+
+ return -EBUSY;
+}
+
/**
* hmm_range_fault - try to fault some address in a virtual address range
* @range: argument structure
@@ -674,6 +696,16 @@ int hmm_range_fault(struct hmm_range *range)
return -EBUSY;
ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
&hmm_walk_ops, &hmm_vma_walk);
+ /*
+ * When HMM_FAULT_PENDING is returned a walk callback
+ * recorded a range that needs handle_mm_fault();
+ * hmm_do_fault() runs the fault outside walk_page_range()
+ * (so no page-table or hugetlb_vma_lock is held) and
+ * returns -EBUSY so the loop re-walks and picks up the
+ * now-present entries.
+ */
+ if (ret == HMM_FAULT_PENDING)
+ ret = hmm_do_fault(mm, &hmm_vma_walk);
/*
* When -EBUSY is returned the loop restarts with
* hmm_vma_walk.last set to an address that has not been stored
^ permalink raw reply related
* [PATCH v8 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Stanislav Kinsburskii @ 2026-07-10 21:26 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178371866223.900500.12312667138651735591.stgit@skinsburskii>
hmm_range_fault() requires the caller to hold the mmap read lock for the
duration of the call. This is incompatible with mappings whose fault
handler may release the mmap lock, notably userfaultfd-managed regions,
where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED
after dropping the lock. Drivers that need to populate device page tables
for such mappings have no way to do so today.
Add hmm_range_fault_unlocked_timeout() for callers that do not need to hold
mmap_lock across any work outside the HMM fault itself. The helper takes
mmap_read_lock_killable() internally, calls the common HMM fault
implementation, and releases the lock before returning if it is still held.
The timeout is specified in jiffies; passing 0 retries indefinitely, while
a non-zero timeout makes the helper return -EBUSY when the retry budget
expires.
When handle_mm_fault() drops mmap_lock, or when the range is invalidated,
hmm_range_fault_unlocked_timeout() refreshes range->notifier_seq and
retries the walk internally. If the lock was dropped, the retry deadline is
also restarted because a lock-dropping fault handler made progress.
Ordinary -EBUSY retries keep the existing deadline, preserving the caller's
timeout policy for repeated mmu-notifier invalidations.
The caller only needs to perform the usual post-success
mmu_interval_read_retry() check while holding its update lock before
consuming the pfns. If mmap_lock acquisition is interrupted or a fatal
signal is pending during retry handling, -EINTR is returned instead.
The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and
FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked_timeout(). The
existing hmm_range_fault() path still passes no locked state, does not
allow handle_mm_fault() to drop mmap_lock, and remains a thin wrapper
preserving the existing API contract for current callers.
The previous refactor that moved page fault handling out of the page-table
walk callbacks is what makes this change small. Faults now run after
walk_page_range() has unwound, with only mmap_lock held, so dropping it
does not interact with the walker's pte spinlock or hugetlb_vma_lock.
Hugetlb regions therefore participate in the unlocked path uniformly with
PTE- and PMD-level mappings; no special case is required.
Documentation/mm/hmm.rst is updated with a description of the new API and
the recommended caller pattern.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
Documentation/mm/hmm.rst | 76 +++++++++++++++------
include/linux/hmm.h | 2 +
mm/hmm.c | 165 ++++++++++++++++++++++++++++++++++++++--------
3 files changed, 192 insertions(+), 51 deletions(-)
diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst
index 7d61b7a8b65b..4e5a750748ae 100644
--- a/Documentation/mm/hmm.rst
+++ b/Documentation/mm/hmm.rst
@@ -156,42 +156,57 @@ During the ops->invalidate() callback the device driver must perform the
update action to the range (mark range read only, or fully unmap, etc.). The
device must complete the update before the driver callback returns.
-When the device driver wants to populate a range of virtual addresses, it can
-use::
+When the device driver wants to populate a range of virtual addresses, the
+normal interface is::
- int hmm_range_fault(struct hmm_range *range);
+ int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
+ unsigned long timeout);
It will trigger a page fault on missing or read-only entries if write access is
requested (see below). Page faults use the generic mm page fault code path just
-like a CPU page fault. The usage pattern is::
+like a CPU page fault.
+
+The caller must not hold ``mmap_read_lock`` before the call.
+``hmm_range_fault_unlocked_timeout()`` takes the mmap read lock internally and
+allows ``handle_mm_fault()`` to drop it during fault handling. This is required
+for VMAs whose fault handlers may release the mmap lock, for example regions
+managed by ``userfaultfd``.
+
+If the mmap lock is dropped or the range is invalidated, the function refreshes
+``range->notifier_seq`` and restarts the walk internally. ``-EINTR`` is returned
+if mmap lock acquisition is interrupted or a fatal signal is pending during
+retry handling.
+
+The timeout is specified in jiffies; passing ``0`` means retry indefinitely. The
+timeout exists to preserve caller policy for repeated mmu-notifier invalidation
+and is checked between retry attempts. HMM does not interrupt page fault
+handling when the timeout expires, but returns ``-EBUSY`` if the retry budget is
+exhausted before a stable range is obtained.
+
+The usage pattern is::
int driver_populate_range(...)
{
struct hmm_range range;
+ unsigned long timeout;
...
+ timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
range.notifier = &interval_sub;
range.start = ...;
range.end = ...;
range.hmm_pfns = ...;
- if (!mmget_not_zero(interval_sub->notifier.mm))
+ if (!mmget_not_zero(interval_sub.mm))
return -EFAULT;
again:
- range.notifier_seq = mmu_interval_read_begin(&interval_sub);
- mmap_read_lock(mm);
- ret = hmm_range_fault(&range);
- if (ret) {
- mmap_read_unlock(mm);
- if (ret == -EBUSY)
- goto again;
- return ret;
- }
- mmap_read_unlock(mm);
+ ret = hmm_range_fault_unlocked_timeout(&range, timeout);
+ if (ret)
+ goto out_put;
take_lock(driver->update);
- if (mmu_interval_read_retry(&ni, range.notifier_seq) {
+ if (mmu_interval_read_retry(&interval_sub, range.notifier_seq)) {
release_lock(driver->update);
goto again;
}
@@ -200,7 +215,11 @@ like a CPU page fault. The usage pattern is::
* under the update lock */
release_lock(driver->update);
- return 0;
+ ret = 0;
+
+ out_put:
+ mmput(interval_sub.mm);
+ return ret;
}
The driver->update lock is the same lock that the driver takes inside its
@@ -208,6 +227,19 @@ invalidate() callback. That lock must be held before calling
mmu_interval_read_retry() to avoid any race with a concurrent CPU page table
update.
+Holding the mmap lock across HMM faults
+=======================================
+
+Most callers should use ``hmm_range_fault_unlocked_timeout()``. If a driver
+really needs to hold the mmap lock across work outside HMM, it can use::
+
+ int hmm_range_fault(struct hmm_range *range);
+
+The mmap lock must be held by the caller and will remain held on return. This
+interface cannot support VMAs whose fault handlers need to drop the mmap lock.
+New callers should prefer ``hmm_range_fault_unlocked_timeout()`` unless they
+have a specific requirement to keep the mmap lock held across the call.
+
Leverage default_flags and pfn_flags_mask
=========================================
@@ -221,8 +253,8 @@ permission, it sets::
range->default_flags = HMM_PFN_REQ_FAULT;
range->pfn_flags_mask = 0;
-and calls hmm_range_fault() as described above. This will fill fault all pages
-in the range with at least read permission.
+and calls the HMM range fault helper as described above. This will fault
+all pages in the range with at least read permission.
Now let's say the driver wants to do the same except for one page in the range for
which it wants to have write permission. Now driver set::
@@ -236,9 +268,9 @@ address == range->start + (index_of_write << PAGE_SHIFT) it will fault with
write permission i.e., if the CPU pte does not have write permission set then HMM
will call handle_mm_fault().
-After hmm_range_fault completes the flag bits are set to the current state of
-the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be set if the page is
-writable.
+After the HMM range fault helper completes the flag bits are set to the
+current state of the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be
+set if the page is writable.
Represent and manage device memory from core kernel point of view
diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index db75ffc949a7..6f04e3932f5b 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -123,6 +123,8 @@ struct hmm_range {
* Please see Documentation/mm/hmm.rst for how to use the range API.
*/
int hmm_range_fault(struct hmm_range *range);
+int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
+ unsigned long timeout);
/*
* HMM_RANGE_DEFAULT_TIMEOUT - default timeout (ms) when waiting for a range
diff --git a/mm/hmm.c b/mm/hmm.c
index bc9361a715fa..fc2e1cd0cb22 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -32,6 +32,7 @@
struct hmm_vma_walk {
struct hmm_range *range;
+ bool *locked;
unsigned long last;
unsigned long end;
unsigned int required_fault;
@@ -44,6 +45,14 @@ struct hmm_vma_walk {
*/
#define HMM_FAULT_PENDING -EAGAIN
+/*
+ * Internal sentinel returned by hmm_do_fault() when handle_mm_fault()
+ * completes a page fault with the mmap lock dropped. hmm_do_fault() sets
+ * *locked = false; the outer loop consumes the sentinel and never propagates
+ * it to the caller.
+ */
+#define HMM_FAULT_UNLOCKED -ENOLCK
+
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -73,9 +82,9 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
*
* Called by the walk callbacks when they discover that part of the range
* needs a page fault. The callback records what to fault and returns
- * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of
- * walk_page_range() and invokes handle_mm_fault() from a context where no
- * page-table or hugetlb_vma_lock is held.
+ * HMM_FAULT_PENDING; the outer loop in hmm_range_fault_locked() drops
+ * back out of walk_page_range() and invokes handle_mm_fault() from a context
+ * where no page-table or hugetlb_vma_lock is held.
*/
static int hmm_record_fault(unsigned long addr, unsigned long end,
unsigned int required_fault,
@@ -624,7 +633,7 @@ static const struct mm_walk_ops hmm_walk_ops = {
/*
* hmm_do_fault - fault in a range recorded by a walk callback
*
- * Called from the outer loop in hmm_range_fault() after a callback
+ * Called from the outer loop in hmm_range_fault_locked() after a callback
* returned HMM_FAULT_PENDING. At this point we hold only mmap_lock;
* the page-table spinlock and any hugetlb_vma_lock acquired by the walk
* framework have already been released by the unwind.
@@ -641,6 +650,9 @@ static int hmm_do_fault(struct mm_struct *mm,
unsigned int fault_flags = FAULT_FLAG_REMOTE;
struct vm_area_struct *vma;
+ if (hmm_vma_walk->locked)
+ fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+
vma = vma_lookup(mm, addr);
if (!vma)
return -EFAULT;
@@ -651,37 +663,34 @@ static int hmm_do_fault(struct mm_struct *mm,
fault_flags |= FAULT_FLAG_WRITE;
}
- for (; addr < end; addr += PAGE_SIZE)
- if (handle_mm_fault(vma, addr, fault_flags, NULL) &
- VM_FAULT_ERROR)
- return -EFAULT;
+ for (; addr < end; addr += PAGE_SIZE) {
+ vm_fault_t ret;
+
+ ret = handle_mm_fault(vma, addr, fault_flags, NULL);
+
+ if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
+ *hmm_vma_walk->locked = false;
+ return HMM_FAULT_UNLOCKED;
+ }
+
+ if (ret & VM_FAULT_ERROR) {
+ int err = vm_fault_to_errno(ret, 0);
+
+ if (WARN_ON(!err))
+ err = -EINVAL;
+
+ return err;
+ }
+ }
return -EBUSY;
}
-/**
- * hmm_range_fault - try to fault some address in a virtual address range
- * @range: argument structure
- *
- * Returns 0 on success or one of the following error codes:
- *
- * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
- * (e.g., device file vma).
- * -ENOMEM: Out of memory.
- * -EPERM: Invalid permission (e.g., asking for write and range is read
- * only).
- * -EBUSY: The range has been invalidated and the caller needs to wait for
- * the invalidation to finish.
- * -EFAULT: A page was requested to be valid and could not be made valid
- * ie it has no backing VMA or it is illegal to access
- *
- * This is similar to get_user_pages(), except that it can read the page tables
- * without mutating them (ie causing faults).
- */
-int hmm_range_fault(struct hmm_range *range)
+static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
{
struct hmm_vma_walk hmm_vma_walk = {
.range = range,
+ .locked = locked,
.last = range->start,
};
struct mm_struct *mm = range->notifier->mm;
@@ -704,8 +713,14 @@ int hmm_range_fault(struct hmm_range *range)
* returns -EBUSY so the loop re-walks and picks up the
* now-present entries.
*/
- if (ret == HMM_FAULT_PENDING)
+ if (ret == HMM_FAULT_PENDING) {
ret = hmm_do_fault(mm, &hmm_vma_walk);
+ if (ret == HMM_FAULT_UNLOCKED) {
+ if (fatal_signal_pending(current))
+ return -EINTR;
+ return -EBUSY;
+ }
+ }
/*
* When -EBUSY is returned the loop restarts with
* hmm_vma_walk.last set to an address that has not been stored
@@ -715,8 +730,100 @@ int hmm_range_fault(struct hmm_range *range)
} while (ret == -EBUSY);
return ret;
}
+
+/**
+ * hmm_range_fault - try to fault some address in a virtual address range
+ * @range: argument structure
+ *
+ * Returns 0 on success or one of the following error codes:
+ *
+ * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
+ * (e.g., device file vma).
+ * -ENOMEM: Out of memory.
+ * -EPERM: Invalid permission (e.g., asking for write and range is read
+ * only).
+ * -EBUSY: The range has been invalidated and the caller needs to wait for
+ * the invalidation to finish.
+ * -EFAULT: A page was requested to be valid and could not be made valid
+ * ie it has no backing VMA or it is illegal to access
+ *
+ * This is similar to get_user_pages(), except that it can read the page tables
+ * without mutating them (ie causing faults).
+ *
+ * The mmap lock must be held by the caller and will remain held on return.
+ * New users should prefer hmm_range_fault_unlocked_timeout() unless they
+ * specifically need to keep the mmap lock held across the call. This helper
+ * cannot support VMAs whose fault handlers need to drop the mmap lock.
+ */
+int hmm_range_fault(struct hmm_range *range)
+{
+ return hmm_range_fault_locked(range, NULL);
+}
EXPORT_SYMBOL(hmm_range_fault);
+/**
+ * hmm_range_fault_unlocked_timeout - fault in a range with a retry timeout
+ * @range: argument structure
+ * @timeout: timeout in jiffies for internal -EBUSY retries, or 0 to retry
+ * indefinitely
+ *
+ * The caller must not hold the mmap lock. The function takes the mmap read
+ * lock internally and allows handle_mm_fault() to drop it during faults. If
+ * the mmap lock is dropped or the range is invalidated, the function refreshes
+ * range->notifier_seq and restarts the walk internally.
+ *
+ * Passing 0 for @timeout retries indefinitely. A non-zero @timeout is a caller
+ * policy limit for repeated mmu-notifier invalidation retries. HMM does not
+ * interrupt page fault handling when the timeout expires, but returns -EBUSY
+ * if the retry budget is exhausted before a stable range is obtained.
+ *
+ * Returns 0 on success or one of the error codes documented for
+ * hmm_range_fault(). -EINTR is returned if mmap_lock acquisition is
+ * interrupted or a fatal signal is pending during retry handling.
+ */
+int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
+ unsigned long timeout)
+{
+ struct mm_struct *mm = range->notifier->mm;
+ unsigned long deadline = 0;
+ bool locked = false;
+ int ret;
+
+ do {
+ if (fatal_signal_pending(current))
+ return -EINTR;
+
+ if (timeout) {
+ /*
+ * If the previous fault dropped mmap_lock, then the fault
+ * handler made progress. Restart the retry timeout in that
+ * case, but keep the existing deadline for ordinary -EBUSY
+ * retries.
+ */
+ if (!locked)
+ deadline = jiffies + timeout;
+
+ if (time_after(jiffies, deadline))
+ return -EBUSY;
+ }
+
+ range->notifier_seq =
+ mmu_interval_read_begin(range->notifier);
+
+ ret = mmap_read_lock_killable(mm);
+ if (ret)
+ return ret;
+
+ locked = true;
+ ret = hmm_range_fault_locked(range, &locked);
+ if (locked)
+ mmap_read_unlock(mm);
+ } while (ret == -EBUSY);
+
+ return ret;
+}
+EXPORT_SYMBOL(hmm_range_fault_unlocked_timeout);
+
/**
* hmm_dma_map_alloc - Allocate HMM map structure
* @dev: device to allocate structure for
^ permalink raw reply related
* [PATCH v8 3/8] selftests/mm: add HMM test for mmap lock-dropping faults
From: Stanislav Kinsburskii @ 2026-07-10 21:26 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178371866223.900500.12312667138651735591.stgit@skinsburskii>
Add test_hmm coverage for the HMM lock-dropping fault path. The test module
gets a new HMM_DMIRROR_READ_UNLOCKED ioctl that calls
hmm_range_fault_unlocked_timeout() with a timeout of 0, exercising the
unbounded retry mode while allowing the mmap lock to be dropped during
fault handling.
Add a userfaultfd_read selftest that registers an anonymous mapping with
UFFDIO_REGISTER_MODE_MISSING, services the faults from a handler thread
with UFFDIO_COPY, and verifies that HMM can read back the data supplied by
the handler. This exercises the path where handle_mm_fault() drops
mmap_lock and hmm_range_fault_unlocked_timeout() restarts the walk
internally.
Assisted-by: GitHub-Copilot:claude-opus-4.6
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
lib/test_hmm.c | 107 +++++++++++++++++++++++
lib/test_hmm_uapi.h | 1
tools/testing/selftests/mm/hmm-tests.c | 150 ++++++++++++++++++++++++++++++++
3 files changed, 257 insertions(+), 1 deletion(-)
diff --git a/lib/test_hmm.c b/lib/test_hmm.c
index 45c0cb992218..6205fb313bd0 100644
--- a/lib/test_hmm.c
+++ b/lib/test_hmm.c
@@ -389,6 +389,67 @@ static int dmirror_range_fault(struct dmirror *dmirror,
return ret;
}
+static int dmirror_range_fault_unlocked(struct dmirror *dmirror,
+ struct hmm_range *range,
+ unsigned long timeout)
+{
+ int ret;
+
+ while (true) {
+ ret = hmm_range_fault_unlocked_timeout(range, timeout);
+ if (ret)
+ goto out;
+
+ mutex_lock(&dmirror->mutex);
+ if (mmu_interval_read_retry(range->notifier,
+ range->notifier_seq)) {
+ mutex_unlock(&dmirror->mutex);
+ continue;
+ }
+ break;
+ }
+
+ ret = dmirror_do_fault(dmirror, range);
+
+ mutex_unlock(&dmirror->mutex);
+out:
+ return ret;
+}
+
+static int dmirror_fault_unlocked(struct dmirror *dmirror,
+ unsigned long start,
+ unsigned long end, bool write,
+ unsigned long timeout)
+{
+ struct mm_struct *mm = dmirror->notifier.mm;
+ unsigned long addr;
+ unsigned long pfns[32];
+ struct hmm_range range = {
+ .notifier = &dmirror->notifier,
+ .hmm_pfns = pfns,
+ .pfn_flags_mask = 0,
+ .default_flags =
+ HMM_PFN_REQ_FAULT | (write ? HMM_PFN_REQ_WRITE : 0),
+ .dev_private_owner = dmirror->mdevice,
+ };
+ int ret = 0;
+
+ if (!mmget_not_zero(mm))
+ return -EFAULT;
+
+ for (addr = start; addr < end; addr = range.end) {
+ range.start = addr;
+ range.end = min(addr + (ARRAY_SIZE(pfns) << PAGE_SHIFT), end);
+
+ ret = dmirror_range_fault_unlocked(dmirror, &range, timeout);
+ if (ret)
+ break;
+ }
+
+ mmput(mm);
+ return ret;
+}
+
static int dmirror_fault(struct dmirror *dmirror, unsigned long start,
unsigned long end, bool write)
{
@@ -488,6 +549,48 @@ static int dmirror_read(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd)
return ret;
}
+static int dmirror_read_unlocked(struct dmirror *dmirror,
+ struct hmm_dmirror_cmd *cmd,
+ unsigned long timeout)
+{
+ struct dmirror_bounce bounce;
+ unsigned long start, end;
+ unsigned long size = cmd->npages << PAGE_SHIFT;
+ int ret;
+
+ start = cmd->addr;
+ end = start + size;
+ if (end < start)
+ return -EINVAL;
+
+ ret = dmirror_bounce_init(&bounce, start, size);
+ if (ret)
+ return ret;
+
+ while (1) {
+ mutex_lock(&dmirror->mutex);
+ ret = dmirror_do_read(dmirror, start, end, &bounce);
+ mutex_unlock(&dmirror->mutex);
+ if (ret != -ENOENT)
+ break;
+
+ start = cmd->addr + (bounce.cpages << PAGE_SHIFT);
+ ret = dmirror_fault_unlocked(dmirror, start, end, false, timeout);
+ if (ret)
+ break;
+ cmd->faults++;
+ }
+
+ if (ret == 0) {
+ if (copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr,
+ bounce.size))
+ ret = -EFAULT;
+ }
+ cmd->cpages = bounce.cpages;
+ dmirror_bounce_fini(&bounce);
+ return ret;
+}
+
static int dmirror_do_write(struct dmirror *dmirror, unsigned long start,
unsigned long end, struct dmirror_bounce *bounce)
{
@@ -1572,7 +1675,9 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp,
dmirror->flags = cmd.npages;
ret = 0;
break;
-
+ case HMM_DMIRROR_READ_UNLOCKED:
+ ret = dmirror_read_unlocked(dmirror, &cmd, 0);
+ break;
default:
return -EINVAL;
}
diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h
index f94c6d457338..ea9b0ec404fb 100644
--- a/lib/test_hmm_uapi.h
+++ b/lib/test_hmm_uapi.h
@@ -38,6 +38,7 @@ struct hmm_dmirror_cmd {
#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0)
diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftests/mm/hmm-tests.c
index 6fccbdab02ee..5acb728666f8 100644
--- a/tools/testing/selftests/mm/hmm-tests.c
+++ b/tools/testing/selftests/mm/hmm-tests.c
@@ -29,6 +29,10 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
#include <sys/time.h>
+#include <sys/syscall.h>
+#include <sys/eventfd.h>
+#include <linux/userfaultfd.h>
+#include <poll.h>
/*
* This is a private UAPI to the kernel test module so it isn't exported
@@ -2952,4 +2956,150 @@ TEST_F_TIMEOUT(hmm, benchmark_thp_migration, 120)
&thp_results, ®ular_results);
}
}
+/*
+ * Test that HMM can fault in pages backed by userfaultfd using the
+ * hmm_range_fault_unlocked_timeout() path with no timeout. This exercises
+ * the lock-drop retry logic in the HMM framework.
+ */
+struct uffd_thread_args {
+ int uffd;
+ int stop_fd;
+ void *page_buffer;
+ unsigned long page_size;
+};
+
+static void *uffd_handler_thread(void *arg)
+{
+ struct uffd_thread_args *args = arg;
+ struct uffd_msg msg;
+ struct uffdio_copy copy;
+ struct pollfd pollfd[2];
+ int ret;
+
+ pollfd[0].fd = args->uffd;
+ pollfd[0].events = POLLIN;
+ pollfd[1].fd = args->stop_fd;
+ pollfd[1].events = POLLIN;
+
+ while (1) {
+ ret = poll(pollfd, 2, -1);
+ if (ret <= 0)
+ break;
+ if (pollfd[1].revents)
+ break;
+ if (!(pollfd[0].revents & POLLIN))
+ break;
+
+ ret = read(args->uffd, &msg, sizeof(msg));
+ if (ret != sizeof(msg))
+ break;
+
+ if (msg.event != UFFD_EVENT_PAGEFAULT)
+ break;
+
+ /* Fill the page with a known pattern */
+ memset(args->page_buffer, 0xAB, args->page_size);
+
+ copy.dst = msg.arg.pagefault.address & ~(args->page_size - 1);
+ copy.src = (unsigned long)args->page_buffer;
+ copy.len = args->page_size;
+ copy.mode = 0;
+ copy.copy = 0;
+
+ ret = ioctl(args->uffd, UFFDIO_COPY, ©);
+ if (ret < 0)
+ break;
+ }
+
+ return NULL;
+}
+
+TEST_F(hmm, userfaultfd_read)
+{
+ struct hmm_buffer *buffer;
+ struct uffd_thread_args uffd_args;
+ unsigned long npages;
+ unsigned long size;
+ unsigned long i;
+ unsigned char *ptr;
+ pthread_t thread;
+ int uffd;
+ int stop_fd;
+ int ret;
+ struct uffdio_api api;
+ struct uffdio_register reg;
+ uint64_t stop = 1;
+ ssize_t nwrite;
+
+ npages = 4;
+ size = npages << self->page_shift;
+
+ /* Create userfaultfd */
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (uffd < 0)
+ SKIP(return, "userfaultfd not available");
+
+ api.api = UFFD_API;
+ api.features = 0;
+ ret = ioctl(uffd, UFFDIO_API, &api);
+ ASSERT_EQ(ret, 0);
+
+ buffer = malloc(sizeof(*buffer));
+ ASSERT_NE(buffer, NULL);
+
+ buffer->fd = -1;
+ buffer->size = size;
+ buffer->mirror = malloc(size);
+ ASSERT_NE(buffer->mirror, NULL);
+
+ /* Create anonymous mapping */
+ buffer->ptr = mmap(NULL, size,
+ PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS,
+ -1, 0);
+ ASSERT_NE(buffer->ptr, MAP_FAILED);
+
+ /* Register the region with userfaultfd */
+ reg.range.start = (unsigned long)buffer->ptr;
+ reg.range.len = size;
+ reg.mode = UFFDIO_REGISTER_MODE_MISSING;
+ ret = ioctl(uffd, UFFDIO_REGISTER, ®);
+ ASSERT_EQ(ret, 0);
+
+ /* Set up the handler thread */
+ uffd_args.uffd = uffd;
+ stop_fd = eventfd(0, EFD_CLOEXEC);
+ ASSERT_GE(stop_fd, 0);
+ uffd_args.stop_fd = stop_fd;
+ uffd_args.page_buffer = malloc(self->page_size);
+ ASSERT_NE(uffd_args.page_buffer, NULL);
+ uffd_args.page_size = self->page_size;
+
+ ret = pthread_create(&thread, NULL, uffd_handler_thread, &uffd_args);
+ ASSERT_EQ(ret, 0);
+
+ /*
+ * Use the unlocked read path which allows the mmap lock to be
+ * dropped during the fault, enabling userfaultfd resolution.
+ */
+ ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_READ_UNLOCKED,
+ buffer, npages);
+ ASSERT_EQ(ret, 0);
+ ASSERT_EQ(buffer->cpages, npages);
+
+ /* Verify the device read the data filled by the uffd handler */
+ ptr = buffer->mirror;
+ for (i = 0; i < size; ++i)
+ ASSERT_EQ(ptr[i], (unsigned char)0xAB);
+
+ nwrite = write(stop_fd, &stop, sizeof(stop));
+ ASSERT_EQ(nwrite, sizeof(stop));
+ pthread_join(thread, NULL);
+ close(stop_fd);
+ free(uffd_args.page_buffer);
+ close(uffd);
+ hmm_buffer_free(buffer);
+}
+
+
TEST_HARNESS_MAIN
^ permalink raw reply related
* [PATCH v8 4/8] mshv: Use hmm_range_fault_unlocked_timeout() for region faults
From: Stanislav Kinsburskii @ 2026-07-10 21:26 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178371866223.900500.12312667138651735591.stgit@skinsburskii>
MSHV currently faults movable memory regions by taking mmap_read_lock()
around hmm_range_fault(). That prevents the fault path from handling VMAs
whose fault handlers need to drop mmap_lock, such as userfaultfd-backed
mappings.
Use hmm_range_fault_unlocked_timeout() instead. Passing a timeout of 0
preserves MSHV's existing unbounded retry behavior while letting the HMM
helper own mmap_lock acquisition and refresh range->notifier_seq internally
before walking the range. After the fault succeeds, MSHV still takes
mreg_mutex and checks mmu_interval_read_retry() before installing the pages
into the region, so the existing invalidation synchronization is preserved.
Fold the small fault-and-lock helper into mshv_region_range_fault(), since
the remaining retry path is just the standard "fault, take the driver lock,
check the interval notifier sequence" pattern.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/hv/mshv_regions.c | 54 ++++++++-------------------------------------
1 file changed, 10 insertions(+), 44 deletions(-)
diff --git a/drivers/hv/mshv_regions.c b/drivers/hv/mshv_regions.c
index 6d65e5b42152..dddaade31b5d 100644
--- a/drivers/hv/mshv_regions.c
+++ b/drivers/hv/mshv_regions.c
@@ -381,46 +381,6 @@ int mshv_region_get(struct mshv_mem_region *region)
return kref_get_unless_zero(®ion->mreg_refcount);
}
-/**
- * mshv_region_hmm_fault_and_lock - Handle HMM faults and lock the memory region
- * @region: Pointer to the memory region structure
- * @range: Pointer to the HMM range structure
- *
- * This function performs the following steps:
- * 1. Reads the notifier sequence for the HMM range.
- * 2. Acquires a read lock on the memory map.
- * 3. Handles HMM faults for the specified range.
- * 4. Releases the read lock on the memory map.
- * 5. If successful, locks the memory region mutex.
- * 6. Verifies if the notifier sequence has changed during the operation.
- * If it has, releases the mutex and returns -EBUSY to match with
- * hmm_range_fault() return code for repeating.
- *
- * Return: 0 on success, a negative error code otherwise.
- */
-static int mshv_region_hmm_fault_and_lock(struct mshv_mem_region *region,
- struct hmm_range *range)
-{
- int ret;
-
- range->notifier_seq = mmu_interval_read_begin(range->notifier);
- mmap_read_lock(region->mreg_mni.mm);
- ret = hmm_range_fault(range);
- mmap_read_unlock(region->mreg_mni.mm);
- if (ret)
- return ret;
-
- mutex_lock(®ion->mreg_mutex);
-
- if (mmu_interval_read_retry(range->notifier, range->notifier_seq)) {
- mutex_unlock(®ion->mreg_mutex);
- cond_resched();
- return -EBUSY;
- }
-
- return 0;
-}
-
/**
* mshv_region_range_fault - Handle memory range faults for a given region.
* @region: Pointer to the memory region structure.
@@ -452,13 +412,19 @@ static int mshv_region_range_fault(struct mshv_mem_region *region,
range.start = region->start_uaddr + page_offset * HV_HYP_PAGE_SIZE;
range.end = range.start + page_count * HV_HYP_PAGE_SIZE;
- do {
- ret = mshv_region_hmm_fault_and_lock(region, &range);
- } while (ret == -EBUSY);
-
+again:
+ ret = hmm_range_fault_unlocked_timeout(&range, 0);
if (ret)
goto out;
+ mutex_lock(®ion->mreg_mutex);
+
+ if (mmu_interval_read_retry(range.notifier, range.notifier_seq)) {
+ mutex_unlock(®ion->mreg_mutex);
+ cond_resched();
+ goto again;
+ }
+
for (i = 0; i < page_count; i++)
region->mreg_pages[page_offset + i] = hmm_pfn_to_page(pfns[i]);
^ permalink raw reply related
* [PATCH v8 5/8] drm/nouveau: Use hmm_range_fault_unlocked_timeout() for SVM faults
From: Stanislav Kinsburskii @ 2026-07-10 21:26 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178371866223.900500.12312667138651735591.stgit@skinsburskii>
nouveau_range_fault() takes mmap_read_lock() only to call
hmm_range_fault(). It also keeps a single HMM_RANGE_DEFAULT_TIMEOUT
deadline across both HMM -EBUSY retries and post-fault
mmu_interval_read_retry() retries.
Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns
the mmap lock and refreshes range->notifier_seq for its internal retries.
Nouveau keeps its existing absolute deadline in the outer loop and passes
the remaining jiffies to the helper for each fault attempt, so retries
caused by mmu_interval_read_retry() do not reset the overall retry budget.
Nouveau still validates the interval notifier sequence while holding
svmm->mutex before programming the GPU mapping.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/gpu/drm/nouveau/nouveau_svm.c | 12 ++++--------
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c
index dcc92131488e..4cfb6eb7c771 100644
--- a/drivers/gpu/drm/nouveau/nouveau_svm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_svm.c
@@ -683,15 +683,11 @@ static int nouveau_range_fault(struct nouveau_svmm *svmm,
goto out;
}
- range.notifier_seq = mmu_interval_read_begin(range.notifier);
- mmap_read_lock(mm);
- ret = hmm_range_fault(&range);
- mmap_read_unlock(mm);
- if (ret) {
- if (ret == -EBUSY)
- continue;
+ ret = hmm_range_fault_unlocked_timeout(&range,
+ max(timeout - jiffies,
+ 1L));
+ if (ret)
goto out;
- }
mutex_lock(&svmm->mutex);
if (mmu_interval_read_retry(range.notifier,
^ permalink raw reply related
* [PATCH v8 6/8] RDMA/umem: Use hmm_range_fault_unlocked_timeout() for ODP faults
From: Stanislav Kinsburskii @ 2026-07-10 21:27 UTC (permalink / raw)
To: airlied, akhilesh, akpm, corbet, dakr, david, decui, haiyangz,
jgg, kees, kys, leon, liam, lizhi.hou, ljs, longli, lyude,
maarten.lankhorst, mamin506, mhocko, mripard, nouveau, ogabbay,
oleg, rppt, shuah, simona, skhan, skinsburskii, surenb,
tzimmermann, vbabka, wei.liu, skinsburskii
Cc: dri-devel, linux-mm, linux-doc, linux-hyperv, linux-kernel,
linux-kselftest, linux-rdma
In-Reply-To: <178371866223.900500.12312667138651735591.stgit@skinsburskii>
ib_umem_odp_map_dma_and_lock() takes mmap_read_lock() only around
hmm_range_fault(), then retries -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT
expires.
Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns
the mmap lock and refreshes range->notifier_seq for its internal retries.
ODP keeps using HMM_RANGE_DEFAULT_TIMEOUT for each HMM fault attempt,
while interval invalidation retries continue to be handled by the existing
outer loop.
ODP still validates the interval notifier sequence while holding umem_mutex
before DMA mapping pages.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
---
drivers/infiniband/core/umem_odp.c | 18 +++++-------------
1 file changed, 5 insertions(+), 13 deletions(-)
diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c
index 404fa1cc3254..9cc21cd762d9 100644
--- a/drivers/infiniband/core/umem_odp.c
+++ b/drivers/infiniband/core/umem_odp.c
@@ -329,7 +329,7 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt,
struct mm_struct *owning_mm = umem_odp->umem.owning_mm;
int pfn_index, dma_index, ret = 0, start_idx;
unsigned int page_shift, hmm_order, pfn_start_idx;
- unsigned long num_pfns, current_seq;
+ unsigned long num_pfns;
struct hmm_range range = {};
unsigned long timeout;
@@ -363,26 +363,18 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt,
}
range.hmm_pfns = &(umem_odp->map.pfn_list[pfn_start_idx]);
- timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
retry:
- current_seq = range.notifier_seq =
- mmu_interval_read_begin(&umem_odp->notifier);
-
- mmap_read_lock(owning_mm);
- ret = hmm_range_fault(&range);
- mmap_read_unlock(owning_mm);
- if (unlikely(ret)) {
- if (ret == -EBUSY && !time_after(jiffies, timeout))
- goto retry;
+ ret = hmm_range_fault_unlocked_timeout(&range, timeout);
+ if (unlikely(ret))
goto out_put_mm;
- }
start_idx = (range.start - ib_umem_start(umem_odp)) >> page_shift;
dma_index = start_idx;
mutex_lock(&umem_odp->umem_mutex);
- if (mmu_interval_read_retry(&umem_odp->notifier, current_seq)) {
+ if (mmu_interval_read_retry(&umem_odp->notifier, range.notifier_seq)) {
mutex_unlock(&umem_odp->umem_mutex);
goto retry;
}
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox