Linux Documentation
 help / color / mirror / Atom feed
From: "David Matlack" <dmatlack@google.com>
To: <mhonap@nvidia.com>, <alex@shazbot.org>, <jgg@ziepe.ca>,
	<ankita@nvidia.com>, <jic23@kernel.org>, <dave.jiang@intel.com>,
	<alejandro.lucero-palau@amd.com>, <smadhavan@nvidia.com>,
	<corbet@lwn.net>, <skhan@linuxfoundation.org>,
	<dave@stgolabs.net>, <alison.schofield@intel.com>,
	<vishal.l.verma@intel.com>, <iweiny@kernel.org>,
	<ming.li@zohomail.com>, <yishaih@nvidia.com>,
	<skolothumtho@nvidia.com>, <kevin.tian@intel.com>,
	<bhelgaas@google.com>, <dmatlack@google.com>, <kees@kernel.org>,
	<gustavoars@kernel.org>
Cc: <cjia@nvidia.com>, <kjaju@nvidia.com>, <vsethi@nvidia.com>,
	<zhiw@nvidia.com>, <linux-doc@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <kvm@vger.kernel.org>,
	<linux-cxl@vger.kernel.org>, <linux-pci@vger.kernel.org>,
	<linux-kselftest@vger.kernel.org>,
	<linux-hardening@vger.kernel.org>
Subject: Re: [PATCH v6 31/31] selftests/vfio: Add CXL Type-2 passthrough tests
Date: Mon, 16 Nov 2026 09:52:54 -0800	[thread overview]
Message-ID: <DMWTS6IPSI2R.3EJQC130JYVW3@google.com> (raw)
In-Reply-To: <20261016161839.2765592-32-mhonap@nvidia.com>

On Fri Oct 16, 2026 at 9:18 AM PDT, mhonap wrote:
> Exercise the vfio-cxl contract on a bound Type-2 device: the CXL
> device-info flag and the two vendor-type regions, the HDM memory mmap
> (including a 2 MB huge fault), the read-only trapped decoder region
> against the live host-committed decoder, and the guest's DVSEC reset
> doorbell and manual HDM discovery.

I tried running this test in a VM with the emulated cxl-type2 device
from your QEMU RFC [1], on an x86 q35 machine with intel-iommu. Out of
the box the device binds as plain vfio-pci and the fixture skips
everything. v6 now requires that the host can do CPU cache invalidation
for CXL Reset, and cpu_cache_has_invalidate_memregion() returns false in
a VM (x86 refuses it under a hypervisor, and arm64 has no cache
maintenance provider in a guest).

The QEMU RFC reports 19/19 passing in an Arm guest, but that was against
v5, which didn't have this requirement. How did you run the v6 version
of this test without hardware? Did you use something like
CONFIG_CXL_REGION_INVALIDATION_TEST? That option only covers region
management in drivers/cxl/core/region.c, and depends on CXL_REGION, so
it doesn't help here.

I added a local test-only bypass (a CONFIG_CXL_RESET_INVALIDATION_TEST
that skips the invalidation in cxl_reset_check() and
cxl_hdm_quiesce_prepare()/cxl_hdm_ranges_invalidate()), I get 26 passes
and 1 skip (pm_d3hot_restores_decoder, since cxl-type2 has no PM
capability in QEMU). I think something like that belongs in the
cxl_reset series. Otherwise this test can only be run by people with
real Type-2 hardware.

[1] https://lore.kernel.org/linux-cxl/20261106035911.1280066-1-mhonap@nvidia.com/

> Check that the decoder register region reads fail with EIO while Memory
> Space is disabled, and that the decoder reads back committed and HDM
> access works after low power entry and exit.

The series says a Type-2 device loses its committed HDM decoder in D3hot
and D3cold (patches 17 and 30), but the emulated cxl-type2 device in
your QEMU RFC has no PM capability, so on it these tests never take the
decoder restore path in vfio_cxl_power_up(). Please make the emulated
device behave the same way as the real hardware across D3hot (PM
capability, No_Soft_Reset, and whether the decoder is lost), so that the
selftest exercises the same vfio-cxl code paths in a VM as it does on
hardware.

To help get that right: what does lspci -vv show for the PM capability
of the device you tested with (NoSoftRst+ or NoSoftRst-), and did you
observe the decoder actually being lost?

> Add a partial-mmap path to the vfio selftest library so the trapped
> component BAR maps its sparse ranges, and an __iommu_map_file() helper for
> the dma-buf mapping.

Please split the library changes into separate commits ahead of the
test: __iommu_map_file(), the sparse mmap support in vfio_pci_bar_map(),
and each of the new libvfio helpers suggested below. That way each one
can be reviewed on its own, and the final commit only adds the test, its
Makefile entry, and the MAINTAINERS line.

This series is already quite long (31 patches), and with the library
changes split out the selftest changes will grow to several patches.
Consider sending all the VFIO selftests patches as a separate series
that depends on this one.

> +/* Return the sparse-mmap capability in @info, or NULL if the region has none. */
> +static struct vfio_region_info_cap_sparse_mmap *
> +vfio_pci_sparse_mmap_cap(struct vfio_region_info *info)
> +{
> +	struct vfio_info_cap_header *hdr;
> +	u32 offset;
> +
> +	if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS))
> +		return NULL;
> +
> +	for (offset = info->cap_offset; offset; offset = hdr->next) {
> +		hdr = (void *)info + offset;
> +		if (hdr->id == VFIO_REGION_INFO_CAP_SPARSE_MMAP)
> +			return (struct vfio_region_info_cap_sparse_mmap *)hdr;
> +	}
> +
> +	return NULL;
> +}

This walks the region info capability chain looking for one ID, which is
the same thing find_region_cap() does in the test. Please add one
generic libvfio helper that looks up a region info capability by ID, and
use it in both places.

> + * Usage: ./vfio_cxl_type2_test <BDF>  (or export VFIO_SELFTESTS_BDF=<BDF>).
> + * The device must be bound to vfio-pci with the vfio-cxl module available.

Please document how to run this test in more detail: the hardware
requirements, how to set up the emulated QEMU device, and the kernel
config needed (VFIO_CXL, IOMMUFD, any test-only bypass), so others can
reproduce your results.

> +/* vfio-pci region-offset packing is kernel-internal, not UAPI; define locally. */
> +#ifndef VFIO_PCI_OFFSET_SHIFT
> +#define VFIO_PCI_OFFSET_SHIFT		40
> +#endif
> +#ifndef VFIO_PCI_INDEX_TO_OFFSET
> +#define VFIO_PCI_INDEX_TO_OFFSET(i)	((uint64_t)(i) << VFIO_PCI_OFFSET_SHIFT)
> +#endif

As the comment says, this is kernel-internal and not uAPI, so the test
shouldn't depend on it. Please use vfio_region_info.offset from
VFIO_DEVICE_GET_REGION_INFO instead, as comp_bar_sparse_mmap already
does for the BAR. Better yet, use region access helpers in libvfio that
take a region index and apply the offset internally (see below), so the
test never computes region offsets itself.

> +/* Find a CXL region by subtype; returns the region index or -1, @buf left holding its info. */
> +static int find_cxl_region(int fd, uint32_t nregions, uint32_t subtype,
> +			   void *buf, size_t bufsz)
> +{
> +	uint32_t i;
> +
> +	for (i = 0; i < nregions; i++) {
> +		struct vfio_region_info *ri = buf;
> +		const struct vfio_region_info_cap_type *t;
> +		const struct vfio_info_cap_header *hdr;
> +
> +		memset(buf, 0, bufsz);
> +		ri->argsz = bufsz;
> +		ri->index = i;
> +		if (ioctl(fd, VFIO_DEVICE_GET_REGION_INFO, ri))
> +			continue;

This is a generic lookup of a region by type and subtype. Only the CXL
vendor ID is specific. Please make it a libvfio helper (e.g.
vfio_pci_find_region(device, type, subtype)) built on the existing
region info code in libvfio.

More generally, please replace the raw ioctl() calls throughout the test
with libvfio helpers: VFIO_DEVICE_GET_INFO, VFIO_DEVICE_GET_REGION_INFO,
and VFIO_DEVICE_FEATURE for DMA_BUF and LOW_POWER_ENTRY/EXIT. Please
follow the existing convention, e.g. __vfio_pci_device_reset() and
vfio_pci_device_reset(): a double-underscore variant that returns -errno
for the tests that expect failures or skip when something isn't
supported, and a plain variant that asserts success.

> +/* Walk the PCI extended capability list for the CXL Device DVSEC. */
> +static uint16_t find_cxl_dvsec(struct vfio_pci_device *dev)
> +{

Same here, the extended capability walk and the DVSEC match aren't
CXL-specific. Please consider vfio_pci_find_ext_capability() and
vfio_pci_find_dvsec(device, vendor, id) helpers in libvfio, with the
test passing in the CXL IDs.

> +/* Read config space from sysfs, which is the hardware, not the vfio view. */
> +static int sysfs_config_read(uint32_t off, void *buf, size_t len)
> +{
> +	char path[128];
> +	ssize_t n;
> +	int fd;
> +
> +	snprintf(path, sizeof(path), "/sys/bus/pci/devices/%s/config",
> +		 device_bdf);
> +	fd = open(path, O_RDONLY);
> +	if (fd < 0)
> +		return -errno;
> +	n = pread(fd, buf, len, off);
> +	close(fd);
> +	return n == (ssize_t)len ? 0 : -EIO;
> +}

This and sysfs_config() further down do the same thing. Please replace
both with one sysfs config access helper in lib/sysfs.c, modeled after
the existing vfio config helpers: a sysfs_config_access(bdf, write,
offset, size, data) core with sysfs_config_read{b,w,l}() and
sysfs_config_write{b,w,l}() wrappers that assert on failure, like
vfio_pci_config_access() and vfio_pci_config_read{b,w,l}().

Taking the BDF as a parameter also gets helpers off the device_bdf
global. Please only use device_bdf where the device is opened
(FIXTURE_SETUP() and close_restores_open_time_command). Everywhere else
can use self->dev->bdf.

> +/* The trapped block starts at the HDM decoder registers; CTRL 0 reads back. */
> +TEST_F(vfio_cxl, comp_regs_hdm_read)
> +{
> +	uint64_t ctrl = self->comp_off + CXL_HDM_DECODER0_CTRL_OFFSET(0);
> +	uint32_t val = 0;
> +
> +	ASSERT_GE(self->comp_size, CXL_HDM_DECODER0_CTRL_OFFSET(0) + 4);
> +	ASSERT_EQ((ssize_t)sizeof(val),
> +		  pread(self->dev->fd, &val, sizeof(val), ctrl));
> +}

Please add region access helpers to libvfio instead of open-coding
pread()/pwrite()/mmap() on the device fd, which this test does in about
50 places. Something modeled on the config space helpers:
__vfio_pci_region_access(device, index, write, offset, size, data)
returning the number of bytes or -errno, vfio_pci_region_access()
asserting success, and typed vfio_pci_region_read{b,w,l,q}() and
vfio_pci_region_write{b,w,l,q}() wrappers. Then this becomes:

	val = vfio_pci_region_readl(self->dev, self->comp_idx,
				    CXL_HDM_DECODER0_CTRL_OFFSET(0));

The double-underscore variant covers the expected failures, like
comp_regs_read_needs_memory and the unaligned and out of range tests.
vfio_pci_config_access() could become a wrapper around the same code
using VFIO_PCI_CONFIG_REGION_INDEX. libvfio would need to cache the
region info for all regions, not just config space and the BARs, so it
can apply the offset reported by VFIO_DEVICE_GET_REGION_INFO.

Similarly, __vfio_pci_region_mmap() and vfio_pci_region_mmap() could
cover the HDM region mappings and the expected mmap failures over the
decoder block, and vfio_pci_bar_map() could be built on them.

> +/* Find a standard capability through the vfio config view. */
> +static uint8_t find_std_cap(struct vfio_pci_device *dev, uint8_t id)

Please make this a libvfio helper too, e.g.
vfio_pci_find_capability(device, cap_id).

      reply	other threads:[~2026-11-16 17:52 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-16 16:18 [PATCH v6 00/31] vfio/pci: Add CXL Type-2 device passthrough support mhonap
2026-10-16 16:18 ` [PATCH v6 01/31] cxl/regs: Split the BAR block request and ioremap helpers mhonap
2026-10-16 16:18 ` [PATCH v6 02/31] cxl/regs: Let a BAR-owning driver own the component register block mhonap
2026-10-16 16:18 ` [PATCH v6 03/31] cxl: Move component register defines to uapi/cxl/cxl_regs.h mhonap
2026-11-04 23:52   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 04/31] PCI/CXL: Let the driver that owns the HDM range run CXL Reset mhonap
2026-11-04 23:54   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 05/31] PCI: Refuse bus and slot reset below a CXL port with SBR masked mhonap
2026-11-04 23:53   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 06/31] vfio/pci: Add the CXL provider ops registration interface mhonap
2026-11-04 23:53   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 07/31] vfio/pci: Detect CXL devices and load the CXL provider on demand mhonap
2026-11-04 23:53   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 08/31] vfio/pci: Add a generic excluded-range list mhonap
2026-11-06 15:59   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 09/31] vfio/pci: Advertise the mmap areas around excluded ranges mhonap
2026-11-06 20:39   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 10/31] vfio/pci: Refuse ioeventfds and dma-buf exports over " mhonap
2026-11-06 20:52   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 11/31] vfio/pci: Virtualize the CXL DVSEC in vfio_pci_config.c mhonap
2026-11-06 22:10   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 12/31] vfio/pci: Call the CXL open and close hooks around device use mhonap
2026-11-06 22:51   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 13/31] vfio/pci: Bracket PCI resets with the CXL reset hooks mhonap
2026-11-06 23:44   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 14/31] vfio/pci: Provide an opt-out for the CXL Type-2 extensions mhonap
2026-10-16 16:18 ` [PATCH v6 15/31] vfio/pci: Route every reset point through a try-reset helper mhonap
2026-11-04 23:52   ` Alex Williamson
2026-10-16 16:18 ` [PATCH v6 16/31] vfio/pci: Revoke and re-arm one device-specific region mhonap
2026-10-16 16:18 ` [PATCH v6 17/31] vfio/pci: Call the device provider across low-power transitions mhonap
2026-10-16 16:18 ` [PATCH v6 18/31] vfio/cxl: Add the vfio-cxl provider module skeleton mhonap
2026-10-16 16:18 ` [PATCH v6 19/31] vfio/cxl: Create the CXL memdev and set media ready at bind mhonap
2026-10-16 16:18 ` [PATCH v6 20/31] vfio/cxl: Own the whole component register BAR mhonap
2026-10-16 16:18 ` [PATCH v6 21/31] vfio/cxl: Expose the HDM memory region to the guest mhonap
2026-10-16 16:18 ` [PATCH v6 22/31] vfio/cxl: Allow mmap of the HDM memory region mhonap
2026-10-16 16:18 ` [PATCH v6 23/31] vfio/cxl: Contain HDM memory errors with memory_failure() mhonap
2026-10-16 16:18 ` [PATCH v6 24/31] vfio/cxl: Expose the HDM decoder registers read-only to the guest mhonap
2026-10-16 16:18 ` [PATCH v6 25/31] vfio/cxl: Exclude the HDM decoder registers from direct BAR access mhonap
2026-10-16 16:18 ` [PATCH v6 26/31] vfio/cxl: Restore the HDM decoder after a hot reset mhonap
2026-10-16 16:18 ` [PATCH v6 27/31] vfio/cxl: Describe the CXL device and decoder geometry to userspace mhonap
2026-10-16 16:18 ` [PATCH v6 28/31] vfio/cxl: Export the HDM memory region as a dma-buf mhonap
2026-10-16 16:18 ` [PATCH v6 29/31] vfio/cxl: Run the CXL reset at the vfio reset points mhonap
2026-10-16 16:18 ` [PATCH v6 30/31] Documentation: vfio-pci: Document CXL Type-2 device passthrough mhonap
2026-10-16 16:18 ` [PATCH v6 31/31] selftests/vfio: Add CXL Type-2 passthrough tests mhonap
2026-11-16 17:52   ` David Matlack [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMWTS6IPSI2R.3EJQC130JYVW3@google.com \
    --to=dmatlack@google.com \
    --cc=alejandro.lucero-palau@amd.com \
    --cc=alex@shazbot.org \
    --cc=alison.schofield@intel.com \
    --cc=ankita@nvidia.com \
    --cc=bhelgaas@google.com \
    --cc=cjia@nvidia.com \
    --cc=corbet@lwn.net \
    --cc=dave.jiang@intel.com \
    --cc=dave@stgolabs.net \
    --cc=gustavoars@kernel.org \
    --cc=iweiny@kernel.org \
    --cc=jgg@ziepe.ca \
    --cc=jic23@kernel.org \
    --cc=kees@kernel.org \
    --cc=kevin.tian@intel.com \
    --cc=kjaju@nvidia.com \
    --cc=kvm@vger.kernel.org \
    --cc=linux-cxl@vger.kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-hardening@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=mhonap@nvidia.com \
    --cc=ming.li@zohomail.com \
    --cc=skhan@linuxfoundation.org \
    --cc=skolothumtho@nvidia.com \
    --cc=smadhavan@nvidia.com \
    --cc=vishal.l.verma@intel.com \
    --cc=vsethi@nvidia.com \
    --cc=yishaih@nvidia.com \
    --cc=zhiw@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox