Re: [PATCH v2] Intel IOMMU patch to reprocess RMRR info

[Shuah Khan <shuah.khan-VXdhtT5mjnY@public.gmane.org>]

On Fri, 2012-09-28 at 20:15 +0100, David Woodhouse wrote:
> On Fri, 2012-09-28 at 12:36 -0400, Linda Knippers wrote:
> > I can only speak to the HP servers.  We have been shipping devices
> > 'for a while' that provide sensor-type data to the platform.  The
> > device does DMA writes to a range of memory (the RMRR) and
> > iLO does DMA reads of that data.
> > 
> > This works in general but not when the 'iommu=pt' boot option is
> > used.  This patch associates the RMRR with the devices when
> > they are moved out of the "si" domain.
> 
> That much makes sense, I think, because they're moved out of the
> hardware SI domain *early*, when we realise they're actually only
> capable of 32-bit DMA and we have >4GiB of RAM. Right?

Correct. By default all devices are added to SI domain assuming that
these devices are 64-bit devices. When we detect the device is a 32-bit
device based on the requested dma-mask, it gets removed from SI domain,
hence looses its RMRR association. In the meantime dma continues causing
DMA errors. This patch is re-processing RMRRs for the device in question
and doing re-assignment.

> 
> It sounds like this isn't a case of the device being used by a native
> driver or given to KVM, and subsequently released. This is just booting
> up and losing the RMRR regions on a device which the OS *hasn't* really
> touched. So that should be fixed.
> 
> > Based on Alex's comments about moving RMRR devices between domains,
> > it sounds like we also have a problem without the 'iommu=pt' boot
> > option if someone assigns one of those devices to a guest.
> 
> Yeah... but why *would* they? What possible reason would we have to
> assign either the sensor device, or the iLO, to a KVM guest. Or to have
> a native driver that attempts to do DMA from them?
> 
> Obviously, in an ideal world we'd have proper native drivers for the
> sensor device. But I'm guessing that's not the case here; it's used by
> the firmware and we're not supposed to be touching it?
> 
> And yes, obviously a better hardware design (from the OS/IOMMU point of
> view) would be to have a path for the sensor data that *doesn't* go via
> host RAM and thus via the IOMMU twice. But while that's a lesson that's
> hopefully been learned and will be implemented in future, we have to
> deal with the existing hardware and its (ab)use of RMRRs.
> 

Right. We do have hardware that is relying on being able to do dma from
devices to a system RAM via RMRR.

-- Shuah