Re: [PATCH v7 0/3] vfio/pci: s390: Fix issues preventing VFIO_PCI_MMAP=y for s390 and enable it

[Bjorn Helgaas <helgaas@kernel.org>]

On Wed, Feb 26, 2025 at 01:07:44PM +0100, Niklas Schnelle wrote:
> With the introduction of memory I/O (MIO) instructions enbaled in commit
> 71ba41c9b1d9 ("s390/pci: provide support for MIO instructions") s390
> gained support for direct user-space access to mapped PCI resources.
> Even without those however user-space can access mapped PCI resources
> via the s390 specific MMIO syscalls. There is thus nothing fundamentally
> preventing s390 from supporting VFIO_PCI_MMAP, allowing user-space
> drivers to access PCI resources without going through the pread()
> interface. To actually enable VFIO_PCI_MMAP a few issues need fixing
> however.
> 
> Firstly the s390 MMIO syscalls do not cause a page fault when
> follow_pte() fails due to the page not being present. This breaks
> vfio-pci's mmap() handling which lazily maps on first access.
> 
> Secondly on s390 there is a virtual PCI device called ISM which has
> a few oddities. For one it claims to have a 256 TiB PCI BAR (not a typo)
> which leads to any attempt to mmap() it fail with the following message:
> 
>     vmap allocation for size 281474976714752 failed: use vmalloc=<size> to increase size
> 
> Even if one tried to map this BAR only partially the mapping would not
> be usable on systems with MIO support enabled. So just block mapping
> BARs which don't fit between IOREMAP_START and IOREMAP_END. Solve this
> by keeping the vfio-pci mmap() blocking behavior around for this
> specific device via a PCI quirk and new pdev->non_mappable_bars
> flag.
> 
> As noted by Alex Williamson With mmap() enabled in vfio-pci it makes
> sense to also enable HAVE_PCI_MMAP with the same restriction for pdev->
> non_mappable_bars. So this is added in patch 3 and I tested this with
> another small test program.
> 
> Note:
> For your convenience the code is also available in the tagged
> b4/vfio_pci_mmap branch on my git.kernel.org site below:
> https://git.kernel.org/pub/scm/linux/kernel/git/niks/linux.git/
> 
> Thanks,
> Niklas
> 
> Link: https://lore.kernel.org/all/c5ba134a1d4f4465b5956027e6a4ea6f6beff969.camel@linux.ibm.com/
> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> ---
> Changes in v7:
> - Move all s390 changes, except for a one-lineer to set pdev->
>   non_mappable_bars for all devices, to the third patch (Bjorn)
> - Move checks in pci-sysfs.c and proc.c to the second patch (Bjorn)
> - Only set ARCH_GENERIC_PCI_MMAP_RESOURCES not HAVE_PCI_MMAP following
>   the recommendation for new architectures in
>   Documentation/PCI/sysfs-pci.rst. This only enables the sysfs but not
>   the proc interface.
> - Link to v6: https://lore.kernel.org/r/20250214-vfio_pci_mmap-v6-0-6f300cb63a7e@linux.ibm.com
> 
> Changes in v6:
> - Add a patch to also enable PCI resource mmap() via sysfs and proc
>   exlcluding pdev->non_mappable_bars devices (Alex Williamson)
> - Added Acks
> - Link to v5: https://lore.kernel.org/r/20250212-vfio_pci_mmap-v5-0-633ca5e056da@linux.ibm.com
> 
> Changes in v5:
> - Instead of relying on the existing pdev->non_compliant_bars introduce
>   a new pdev->non_mappable_bars flag. This replaces the VFIO_PCI_MMAP
>   Kconfig option and makes it per-device. This is necessary to not break
>   upcoming vfio-pci use of ISM devices (Julian Ruess)
> - Squash the removal of VFIO_PCI_MMAP into the second commit as this
>   is now where its only use goes away.
> - Switch to using follow_pfnmap_start() in MMIO syscall page fault
>   handling to match upstream changes
> - Dropped R-b's because the changes are significant
> - Link to v4: https://lore.kernel.org/r/20240626-vfio_pci_mmap-v4-0-7f038870f022@linux.ibm.com
> 
> Changes in v4:
> - Overhauled and split up patch 2 which caused errors on ppc due to
>   unexported __kernel_io_end. Replaced it with a minimal s390 PCI fixup
>   harness to set pdev->non_compliant_bars for ISM plus ignoring devices
>   with this flag in vfio-pci. Idea for using PCI quirks came from
>   Christoph Hellwig, thanks. Dropped R-bs for patch 2 accordingly.
> - Rebased on v6.10-rc5 which includes the vfio-pci mmap fault handler
>   fix to the issue I stumbled over independently in v3
> - Link to v3: https://lore.kernel.org/r/20240529-vfio_pci_mmap-v3-0-cd217d019218@linux.ibm.com
> 
> Changes in v3:
> - Rebased on v6.10-rc1 requiring change to follow_pte() call
> - Use current->mm for fixup_user_fault() as seems more common
> - Collected new trailers
> - Link to v2: https://lore.kernel.org/r/20240523-vfio_pci_mmap-v2-0-0dc6c139a4f1@linux.ibm.com
> 
> Changes in v2:
> - Changed last patch to remove VFIO_PCI_MMAP instead of just enabling it
>   for s390 as it is unconditionally true with s390 supporting PCI resource mmap() (Jason)
> - Collected R-bs from Jason
> - Link to v1: https://lore.kernel.org/r/20240521-vfio_pci_mmap-v1-0-2f6315e0054e@linux.ibm.com
> 
> ---
> Niklas Schnelle (3):
>       s390/pci: Fix s390_mmio_read/write syscall page fault handling
>       PCI: s390: Introduce pdev->non_mappable_bars and replace VFIO_PCI_MMAP
>       PCI: s390: Support mmap() of PCI resources except for ISM devices
> 
>  arch/s390/Kconfig                |  4 +---
>  arch/s390/include/asm/pci.h      |  3 +++
>  arch/s390/pci/Makefile           |  2 +-
>  arch/s390/pci/pci_fixup.c        | 23 +++++++++++++++++++++++
>  arch/s390/pci/pci_mmio.c         | 18 +++++++++++++-----
>  drivers/pci/pci-sysfs.c          |  4 ++++
>  drivers/pci/proc.c               |  4 ++++
>  drivers/s390/net/ism_drv.c       |  1 -
>  drivers/vfio/pci/Kconfig         |  4 ----
>  drivers/vfio/pci/vfio_pci_core.c |  2 +-
>  include/linux/pci.h              |  1 +
>  include/linux/pci_ids.h          |  1 +
>  12 files changed, 52 insertions(+), 15 deletions(-)

Applied to pci/resource for v6.15, thanks!

I updated the subject lines to all start with "s390/pci" since that's
where all the interesting bits are and there's only a single instance
of "PCI: s390" in the history.