Re: Unable to pass SATA controller to VM with intel_iommu=igfx_off

[Alex Williamson <alex.williamson@redhat.com>]

On Sat, 6 Jan 2018 12:55:36 +0100
Binarus <lists@binarus.de> wrote:

> Dear all,
> 
> in my first two messages from today, I have described how intel_iommu=on makes the AHCI / SATA driver misbehave when trying to initialize an additional PCIe SATA controller card on one of our servers, and how intel_iommu=igfx_off changed that situation.

For the system you're using there is no integrated graphics on the
processor, so "igfx_off" does nothing.  However, using that as the only
parameter to intel_iommu= is basically the same as not providing an
intel_iommu= option at all, ie. the IOMMU is not enabled, thus...
 
> This message is to report that I can't pass through the SATA controller in question to a VM when the kernel has been booted with intel_iommu=igfx_off. Things are going wrong already during the boot process. dmesg | grep vfio says:
> 
> [    1.074239] vfio-pci: probe of 0000:02:00.0 failed with error -22
> [    1.074309] vfio_pci: add [1b4b:9128[ffff:ffff]] class 0x000000/00000000

vfio-pci won't probe the device because it doesn't have an IOMMU.

> 1b4b:9128 is indeed the correct Vendor / Device ID, and it finds the device correctly (bus 2, device 0, function 0 is exactly the place where the controller resides, according to lspci after having booted without any kernel parameters). But although it finds the device, it fails when probing it (I admit that I have no clue what this means).

Marvell SATA controllers are notoriously bad with IOMMUs, they perform
DMA using a requester ID other than the actual PCI address of the
device.  For instance, the SATA controller is often at PCI function
address 0, but function address 1 is used for the requester ID.  The
IOMMU uses the requester ID to index the DMA translations for the
device, so getting this fundamental aspect of the transaction wrong in
hardware is often fatal for doing any DMA at all when the IOMMU is
enabled.  This should cause all sorts of "DMAR" faults during boot when
VT-d is enabled.

We do have quirks to handle various Marvell chips, but not this one.
If you're interested in building a kernel, I can provide a patch to
test, though I suspect the reason we don't already have a patch for
this device might be conflicting results.  Your other option is to pick
a SATA controller that isn't so broken.  Thanks,

Alex