Re: Device is ineligible for IOMMU domain attach due to platform RMRR requirement

[Alex Williamson <alex.williamson@redhat.com>]

On Fri, 2015-03-06 at 00:20 -0500, Steven DuChene wrote:
> I am attempting on ubuntu 14.04 to configure PCI passthrough of a NVidia 
> K40 GPU card that is plugged into a HP DL580 rack mounted server.
> I have done all of the pre-work I normally have done in the past with 
> pci-stub, vfio and etc but when I try an execute a qemu-system-x86_64 
> command that works on a similar version of debian, I get the following 
> error in the dmesg:
> 
> Device is ineligible for IOMMU domain attach due to platform RMRR 
> requirement. Contact your platform vendor.
> 
> I have read through the patch description from Alex at:
> 
> http://lists.linuxfoundation.org/pipermail/iommu/2014-June/008816.html
> 
> and I have read the IOMMU documentation at:
> 
> https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt
> 
> but I am still not really understanding if or what the fix is for this.
> 
> The ubuntu 14.04 system where I am getting this error is running 
> 3.16.0-30-generic
> The debian system where I can do similar PCI passthrough of a NVidia K2 
> GPU device is running a 3.14.29-4 kernel.
> 
> Can anyone provide any insight into an fix or workaround for this?

Hi Steven,

The issue is that VT-d RMRRs are a platform imposed requirement that a
device continue to have identity mapped access to a platform defined
memory region at all times.  This requirement is fundamentally
incompatible with PCI device assignment where the address space of the
assigned device is defined by the VM.  The VT-d specification hints at
this restriction (8.4):

        The RMRR regions are expected to be used for legacy usages (such
        as USB, UMA Graphics, etc.) requiring reserved memory. Platform
        designers should avoid or limit use of reserved memory regions
        since these require system software to create holes in the DMA
        virtual address range available to system software and its
        drivers.

In order to support assignment of such devices and continue to honor the
RMRR, reserved memory regions would need to be imposed on the guest.
Doing this has a number of issues and it's not clear that it enables any
usable configurations due to the lack of isolation often implied by the
RMRRs.  RMRRs themselves imply some sort of communication conduit to the
platform, which it's also not clear should be allowed for a guest owned
device.

We also cannot continue the previous behavior of simply ignoring RMRRs
for assigned devices.  Not only does the platform require us to honor
them, failing to do so could have implication for both the platform and
the VM health and integrity.

As indicated by the dmesg warning, users encountering this problem
should contact their platform vendor, which is really the only course of
action that I can recommend.  Only the platform vendor can tell you why
they've imposed this requirement for the device and potentially offer a
remedy to remove that requirement.  KVM is not the first hypervisor to
impose this restriction for such devices.  The referenced patch was
tagged for stable, so you can expect that this change will eventually
trickle through all the distributions.  Sorry for the trouble, but it
really was a necessary change.  Thanks,

Alex