Re: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for grace hopper

[Alex Williamson <alex.williamson@redhat.com>]

On Sun, 15 Oct 2023 22:00:47 +0530
<ankita@nvidia.com> wrote:
> +static ssize_t nvgrace_gpu_vfio_pci_read(struct vfio_device *core_vdev,
> +					  char __user *buf, size_t count, loff_t *ppos)
> +{
> +	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);
> +	struct nvgrace_gpu_vfio_pci_core_device *nvdev = container_of(
> +		core_vdev, struct nvgrace_gpu_vfio_pci_core_device, core_device.vdev);
> +	int ret;
> +
> +	if (index == VFIO_PCI_BAR2_REGION_INDEX) {
> +		ret = nvgrace_gpu_memmap(nvdev);
> +		if (ret)
> +			return ret;
> +
> +		return nvgrace_gpu_read_mem(buf, count, ppos, nvdev);
> +	}

After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated
as an IO Port BAR, it occurs to me that there's no config space
emulation of BAR2 (or BAR3) here.  Doesn't this mean that QEMU registers
the BAR as 32-bit, non-prefetchable?  ie. VFIOBAR.type & .mem64 are
wrong?

I'd certainly expect this to be emulated as a 64-bit, prefetchable BAR
and the commit log indicates the intention is that this is exposed as a
64-bit BAR.

We also need to decide how strictly variant drivers need to emulate
vfio_pci_config_rw with respect to BAR sizing, where the core code
provides emulation of sizing and Yishai's virtio driver only emulates
the IO port indicator bit.  QEMU doesn't really need this, but the
vfio-pci implementation sets the precedent that this behavior is
provided and could be used by other userspace drivers.  It's essentially
just providing a masked buffer to service reads and writes to the BAR2
and BAR3 config address here.  Thanks,

Alex

> +
> +	return vfio_pci_core_read(core_vdev, buf, count, ppos);
> +}