linuxppc-dev.lists.ozlabs.org archive mirror
 help / color / mirror / Atom feed
From: Oliver O'Halloran <oohall@gmail.com>
To: linuxppc-dev@lists.ozlabs.org
Cc: aik@ozlabs.ru
Subject: Make PowerNV IOMMU group setup saner (and fix it for hotpug)
Date: Mon,  6 Apr 2020 13:07:38 +1000	[thread overview]
Message-ID: <20200406030745.24595-1-oohall@gmail.com> (raw)

Currently on PowerNV the IOMMU group of a device is initialised in
boot-time fixup which runs after devices are probed. Because this is
only run at boot time hotplugged devices do not recieve an iommu group
assignment which prevents them from being passed through to a guest.

This series fixes that by moving the point where IOMMU groups are
registered to when we configure DMA for a PE, and moves the point where
we add a device to the PE's IOMMU group into the per-device DMA setup
callback for IODA phbs (pnv_pci_ioda_dma_dev_setup()). This change means
that we'll do group setup for hotplugged devices and that we can remove
the hack we have for VFs which are currently added to their group
via a bus notifier.

With this change there's no longer any per-device setup that needs to
run in a fixup for ordinary PCI devices. The exception is, as per usual,
NVLink devices. For those the GPU and any of it's NVLink devices need
to be in a "compound" IOMMU group which keeps the DMA address spaces
of each device in sync with it's attached devices. As a result that
setup can only be done when both the NVLink devices and the GPU device
has been probed, so that setup is still done in the fixup. Sucks, but
it's still an improvement.

Boot tested on a witherspoon with 6xGPUs and it didn't crash so it must
be good.



             reply	other threads:[~2020-04-06  3:11 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-06  3:07 Oliver O'Halloran [this message]
2020-04-06  3:07 ` [PATCH 1/7] powerpc/powernv/npu: Clean up compound table group initialisation Oliver O'Halloran
2020-04-06  9:48   ` Alexey Kardashevskiy
2020-06-09  5:29   ` Michael Ellerman
2020-04-06  3:07 ` [PATCH 2/7] powerpc/powernv/iov: Don't add VFs to iommu group during PE config Oliver O'Halloran
2020-04-06  9:49   ` Alexey Kardashevskiy
2020-04-06  3:07 ` [PATCH 3/7] powerpc/powernv/pci: Register iommu group at PE DMA setup Oliver O'Halloran
2020-04-06  9:51   ` Alexey Kardashevskiy
2020-04-06  3:07 ` [PATCH 4/7] powerpc/powernv/pci: Add device to iommu group during dma_dev_setup() Oliver O'Halloran
2020-04-06  9:51   ` Alexey Kardashevskiy
2020-04-06  3:07 ` [PATCH 5/7] powerpc/powernv/pci: Delete old iommu recursive iommu setup Oliver O'Halloran
2020-04-06  9:53   ` Alexey Kardashevskiy
2020-04-06  3:07 ` [PATCH 6/7] powerpc/powernv/pci: Move tce size parsing to pci-ioda-tce.c Oliver O'Halloran
2020-04-06  9:53   ` Alexey Kardashevskiy
2020-04-06  3:07 ` [PATCH 7/7] powerpc/powernv/npu: Move IOMMU group setup into npu-dma.c Oliver O'Halloran
2020-04-06  9:54   ` Alexey Kardashevskiy
2020-04-06  9:56 ` Make PowerNV IOMMU group setup saner (and fix it for hotpug) Alexey Kardashevskiy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200406030745.24595-1-oohall@gmail.com \
    --to=oohall@gmail.com \
    --cc=aik@ozlabs.ru \
    --cc=linuxppc-dev@lists.ozlabs.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).