IOMMU and graphics cards

IOMMU and graphics cards

[Joerg Roedel]

Hi David,

as I have seen the VT-d code implements a workaround for broken graphics
card drivers. What it does, when enabled, is giving each grahics device
direct access to all physical memory. I really don't like to implement
this but a similar workaround for the AMD IOMMU seems to be necessary.
The biggest problem here is that this kind of workaround disables device
isolation for graphics cards. Device isolation is the main reason for
using an IOMMU in an unvirtualized environment. So if this workaround is
enabled it is as good as disable the IOMMU at all.
Thats why I think a kernel compile option for this workaround is not
sufficient. Distributors will probably enable this in their kernels
which also disables device isolation even if the user don't want to use
these broken drivers.
I think we should change that and provide a better way which allows to
enable this workaround only if it is required (and it should be
transparent to the user which IOMMU is built into the system).
We have several options to do this:

	* Implement a kernel command line option to enable/disable the
	  workaround (what should be default?)
	* Use the IOMMU-API and write a kernel module which creates a
	  direct mapped protection domain and assigns the graphics
	  cards to it (need to be done carefully to not break graphics 
	  drivers which do everything right and use the DMA-API)
	* Any other great idea?

So what do you (and all the others reading this :-) think? I would
prefer the way of implementing a module but there may also be reasons
against this. I would like this to be disussed before I implement the
workaround for the AMD IOMMU.

Thanks,

	Joerg

-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Jochen Polster, Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

Re: IOMMU and graphics cards

[David Woodhouse]

On Tue, 2009-04-28 at 17:05 +0200, Joerg Roedel wrote:
> Hi David,
> 
> as I have seen the VT-d code implements a workaround for broken graphics
> card drivers. What it does, when enabled, is giving each grahics device
> direct access to all physical memory. I really don't like to implement
> this but a similar workaround for the AMD IOMMU seems to be necessary.
> The biggest problem here is that this kind of workaround disables device
> isolation for graphics cards. Device isolation is the main reason for
> using an IOMMU in an unvirtualized environment. So if this workaround is
> enabled it is as good as disable the IOMMU at all.

For that device, yes. The IOMMU still catches errors from _other_
devices, of course.

But I agree that it's a crap thing to have to do, and there are probably
a number of ways we could make it slightly less crap. For a start, we
could at least refrain from mapping the kernel text -- the card has no
business DMAing there, whatever happens.

> Thats why I think a kernel compile option for this workaround is not
> sufficient. Distributors will probably enable this in their kernels
> which also disables device isolation even if the user don't want to use
> these broken drivers.

Yes, that makes a lot of sense.

Is the reason you're doing this actually because of broken drivers? Or
just because of the performance implications? I've heard both cited as
reasons for the 1:1 mapping... and if it's mostly the former, then
perhaps the best way for me to help you is to stop enabling the option
in Fedora (rawhide, at least), so that the buggy drivers get _fixed_ and
you don't get stuck with "it works on Intel, why can't you make it
work?" reports?

> I think we should change that and provide a better way which allows to
> enable this workaround only if it is required (and it should be
> transparent to the user which IOMMU is built into the system).
> We have several options to do this:
> 
> 	* Implement a kernel command line option to enable/disable the
> 	  workaround (what should be default?)
> 	* Use the IOMMU-API and write a kernel module which creates a
> 	  direct mapped protection domain and assigns the graphics
> 	  cards to it (need to be done carefully to not break graphics 
> 	  drivers which do everything right and use the DMA-API)
> 	* Any other great idea?
> 
> So what do you (and all the others reading this :-) think? I would
> prefer the way of implementing a module but there may also be reasons
> against this. I would like this to be disussed before I implement the
> workaround for the AMD IOMMU.

I think I also prefer your second option. I don't like having this as a
hack in the IOMMU code.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: IOMMU and graphics cards

[Joerg Roedel]

On Tue, Apr 28, 2009 at 04:28:50PM +0100, David Woodhouse wrote:
> On Tue, 2009-04-28 at 17:05 +0200, Joerg Roedel wrote:
> 
> For that device, yes. The IOMMU still catches errors from _other_
> devices, of course.

Yes, it is in effect for other devices. But since its a security feature
it only makes sense if it covers all devices.

> Is the reason you're doing this actually because of broken drivers? Or
> just because of the performance implications? I've heard both cited as
> reasons for the 1:1 mapping... and if it's mostly the former, then
> perhaps the best way for me to help you is to stop enabling the option
> in Fedora (rawhide, at least), so that the buggy drivers get _fixed_ and
> you don't get stuck with "it works on Intel, why can't you make it
> work?" reports?

Currently some graphics drivers don't work with IOMMU enabled because
they don't use the DMA-API. The silently assume that device addess ==
physical address.
I don't know how it is solved for VT-d but for AMD IOMMU the available
DMA memory address space is limited per device. This is a problem for
graphics card drivers because, as developers told me, they may need
gigabytes of DMA memory. I am currently thinking about ways to fix that
without enlarging the DMA address space for each device (which would be
a huge waste of memory). But unless this problem isn't solved the
drivers won't be fixed, I guess.
I guess the DRM code in the kernel may have the same problem with IOMMU
enabled?

> > 	* Implement a kernel command line option to enable/disable the
> > 	  workaround (what should be default?)
> > 	* Use the IOMMU-API and write a kernel module which creates a
> > 	  direct mapped protection domain and assigns the graphics
> > 	  cards to it (need to be done carefully to not break graphics 
> > 	  drivers which do everything right and use the DMA-API)
> > 	* Any other great idea?
> 
> I think I also prefer your second option. I don't like having this as a
> hack in the IOMMU code.

Ok, so I will implement such a module and send the code around for
discussion.

	Joerg

-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Jochen Polster, Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

Re: IOMMU and graphics cards

[FUJITA Tomonori]

On Tue, 28 Apr 2009 18:04:48 +0200
Joerg Roedel <joerg.roedel@amd.com> wrote:

> On Tue, Apr 28, 2009 at 04:28:50PM +0100, David Woodhouse wrote:
> > On Tue, 2009-04-28 at 17:05 +0200, Joerg Roedel wrote:
> > 
> > For that device, yes. The IOMMU still catches errors from _other_
> > devices, of course.
> 
> Yes, it is in effect for other devices. But since its a security feature
> it only makes sense if it covers all devices.

For me, IOMMUs are not about security but accessing all memory space
(there are still lots of devices that are not capable of 64-bit DMA)
but I agree that this workaround is bad.


> > Is the reason you're doing this actually because of broken drivers? Or
> > just because of the performance implications? I've heard both cited as
> > reasons for the 1:1 mapping... and if it's mostly the former, then
> > perhaps the best way for me to help you is to stop enabling the option
> > in Fedora (rawhide, at least), so that the buggy drivers get _fixed_ and
> > you don't get stuck with "it works on Intel, why can't you make it
> > work?" reports?
> 
> Currently some graphics drivers don't work with IOMMU enabled because
> they don't use the DMA-API. The silently assume that device addess ==
> physical address.
> I don't know how it is solved for VT-d but for AMD IOMMU the available
> DMA memory address space is limited per device. This is a problem for
> graphics card drivers because, as developers told me, they may need
> gigabytes of DMA memory. I am currently thinking about ways to fix that
> without enlarging the DMA address space for each device (which would be
> a huge waste of memory).

Do you meant that AMD IOMMU code use the bitmap each device to manage
address space so enlarging the DMA address space wastes memory?


>  But unless this problem isn't solved the
> drivers won't be fixed, I guess.
> I guess the DRM code in the kernel may have the same problem with IOMMU
> enabled?

Looks like the DRM code uses the DMA API.


> > > 	* Implement a kernel command line option to enable/disable the
> > > 	  workaround (what should be default?)
> > > 	* Use the IOMMU-API and write a kernel module which creates a
> > > 	  direct mapped protection domain and assigns the graphics
> > > 	  cards to it (need to be done carefully to not break graphics 
> > > 	  drivers which do everything right and use the DMA-API)
> > > 	* Any other great idea?
> > 
> > I think I also prefer your second option. I don't like having this as a
> > hack in the IOMMU code.
> 
> Ok, so I will implement such a module and send the code around for
> discussion.

Yeah, the kernel boot option or the config option don't make sense
since distributions always need to enable the workaround.

btw, what graphic drivers are broken? No chance to fix them?

Re: IOMMU and graphics cards

[Joerg Roedel]

On Thu, May 07, 2009 at 04:47:14PM +0900, FUJITA Tomonori wrote:
> On Tue, 28 Apr 2009 18:04:48 +0200
> Joerg Roedel <joerg.roedel@amd.com> wrote:
> > 
> > Yes, it is in effect for other devices. But since its a security feature
> > it only makes sense if it covers all devices.
> 
> For me, IOMMUs are not about security but accessing all memory space
> (there are still lots of devices that are not capable of 64-bit DMA)
> but I agree that this workaround is bad.

Yeah true. Another major reason for having an IOMMU is to enlarge the
physical address range devices can access.

> Do you meant that AMD IOMMU code use the bitmap each device to manage
> address space so enlarging the DMA address space wastes memory?

Its not only the bitmap. There are also the page tables that would waste
memory if the aperture in the AMD IOMMU driver is enlarged to, say, 4GB.
Most devices don't need such a large amount of DMA address space. As I
found out VT-d does not have this problem because they implemented
another allocator which can cover all possible memory.
For the AMD IOMMU I currently implement a kind of a dynamically growing
bitmap allocator to fix this problem.

> >  But unless this problem isn't solved the
> > drivers won't be fixed, I guess.
> > I guess the DRM code in the kernel may have the same problem with IOMMU
> > enabled?
> 
> Looks like the DRM code uses the DMA API.

The proprietary drivers make problems so far. For the ATI one I am in
contact with the developers to try to fix it. But I can't do anything
about the other proprietary driver I am aware of :-(

Joerg

-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

Re: IOMMU and graphics cards

[David Woodhouse]

On Thu, 2009-05-07 at 12:01 +0200, Joerg Roedel wrote:
> > Looks like the DRM code uses the DMA API.
> 
> The proprietary drivers make problems so far. For the ATI one I am in
> contact with the developers to try to fix it. But I can't do anything
> about the other proprietary driver I am aware of :-(

I see no harm in "breaking" code which was already broken. If that's the
only reason we're enabling the graphics workaround, let's turn it off.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: IOMMU and graphics cards

[FUJITA Tomonori]

On Thu, 07 May 2009 11:22:01 +0100
David Woodhouse <dwmw2@infradead.org> wrote:

> On Thu, 2009-05-07 at 12:01 +0200, Joerg Roedel wrote:
> > > Looks like the DRM code uses the DMA API.
> > 
> > The proprietary drivers make problems so far. For the ATI one I am in
> > contact with the developers to try to fix it. But I can't do anything
> > about the other proprietary driver I am aware of :-(
> 
> I see no harm in "breaking" code which was already broken. If that's the
> only reason we're enabling the graphics workaround, let's turn it off.

I also prefer that approach. Adding a workaround for the 'bugs' of the
proprietary drivers doesn't sound a nice idea. Let them to fix the
bugs. It's not difficult at all to convert a driver to use the DMA
API.

Re: IOMMU and graphics cards

[Joerg Roedel]

On Thu, May 07, 2009 at 11:22:01AM +0100, David Woodhouse wrote:
> On Thu, 2009-05-07 at 12:01 +0200, Joerg Roedel wrote:
> > > Looks like the DRM code uses the DMA API.
> > 
> > The proprietary drivers make problems so far. For the ATI one I am in
> > contact with the developers to try to fix it. But I can't do anything
> > about the other proprietary driver I am aware of :-(
> 
> I see no harm in "breaking" code which was already broken. If that's the
> only reason we're enabling the graphics workaround, let's turn it off.

Well, it is not that simple. Having no workaround at all will not result
in working proprietary drivers. More likely is that distros just disable
the IOMMU because their customers want to use these drivers for working
3d acceleration.
As I wrote in the original mail I think we should just make sure that
these drivers still work with an IOMMU without disabling device
isolation.

Joerg

-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

Re: IOMMU and graphics cards

[FUJITA Tomonori]

On Thu, 7 May 2009 14:15:35 +0200
Joerg Roedel <joerg.roedel@amd.com> wrote:

> On Thu, May 07, 2009 at 11:22:01AM +0100, David Woodhouse wrote:
> > On Thu, 2009-05-07 at 12:01 +0200, Joerg Roedel wrote:
> > > > Looks like the DRM code uses the DMA API.
> > > 
> > > The proprietary drivers make problems so far. For the ATI one I am in
> > > contact with the developers to try to fix it. But I can't do anything
> > > about the other proprietary driver I am aware of :-(
> > 
> > I see no harm in "breaking" code which was already broken. If that's the
> > only reason we're enabling the graphics workaround, let's turn it off.
> 
> Well, it is not that simple. Having no workaround at all will not result
> in working proprietary drivers. More likely is that distros just disable
> the IOMMU because their customers want to use these drivers for working
> 3d acceleration.

Have you already discussed this issue with distribution people?

I'm not sure if they would just disable the IOMMU. They want it for
exotic virtualization features?


> As I wrote in the original mail I think we should just make sure that
> these drivers still work with an IOMMU without disabling device
> isolation.

Re: IOMMU and graphics cards

[Joerg Roedel]

On Thu, May 07, 2009 at 09:59:19PM +0900, FUJITA Tomonori wrote:
> 
> Have you already discussed this issue with distribution people?
> 
> I'm not sure if they would just disable the IOMMU. They want it for
> exotic virtualization features?

Not yet. But thats true for distributions using KVM as their
virtualization technology.

Joerg

-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

Re: IOMMU and graphics cards

[Christoph Hellwig]

On Thu, May 07, 2009 at 02:15:35PM +0200, Joerg Roedel wrote:
> Well, it is not that simple. Having no workaround at all will not result
> in working proprietary drivers.

Which could be considered at not too bad thing anyway.

> More likely is that distros just disable
> the IOMMU because their customers want to use these drivers for working
> 3d acceleration.

Maybe Ubuntu :)

Re: IOMMU and graphics cards

[David Miller]

From: David Woodhouse <dwmw2@infradead.org>
Date: Thu, 07 May 2009 11:22:01 +0100

> On Thu, 2009-05-07 at 12:01 +0200, Joerg Roedel wrote:
>> > Looks like the DRM code uses the DMA API.
>> 
>> The proprietary drivers make problems so far. For the ATI one I am in
>> contact with the developers to try to fix it. But I can't do anything
>> about the other proprietary driver I am aware of :-(
> 
> I see no harm in "breaking" code which was already broken. If that's the
> only reason we're enabling the graphics workaround, let's turn it off.

Seriously, this is an enormous price to pay just for one bad apple
proprietary driver.

I can't believe the level of concessions some people are suggesting.

Let's not make IOMMU's basically useless by working around one broken
driver.

Proper interfaces for what this driver is trying to do have existed in
the kernel for 10+ years.  There are no excuses, and it's time for
Nvidia to get with the game plan.

Re: IOMMU and graphics cards

[Joerg Roedel]

On Thu, May 07, 2009 at 11:36:32AM -0700, David Miller wrote:
> From: David Woodhouse <dwmw2@infradead.org>
> Date: Thu, 07 May 2009 11:22:01 +0100
> 
> > On Thu, 2009-05-07 at 12:01 +0200, Joerg Roedel wrote:
> >> > Looks like the DRM code uses the DMA API.
> >> 
> >> The proprietary drivers make problems so far. For the ATI one I am in
> >> contact with the developers to try to fix it. But I can't do anything
> >> about the other proprietary driver I am aware of :-(
> > 
> > I see no harm in "breaking" code which was already broken. If that's the
> > only reason we're enabling the graphics workaround, let's turn it off.
> 
> Seriously, this is an enormous price to pay just for one bad apple
> proprietary driver.
> 
> I can't believe the level of concessions some people are suggesting.

This concession is already in the VT-d code. And this concession is
_enabled_ in enterprise distribtions supporting VT-d. I don't like the
way it is implemented there and I definitly will not implement a similar
way for the AMD IOMMU. What I try to achieve is a consistent solution
for both IOMMUs. Having the broken drivers work on VT-d but not on
AMD IOMMU is clearly a bad solution.

The other way to achieve consistency is to remove the workaround from
VT-d code which significantly increases the chance to get the broken
stuff fixed. David?

> Let's not make IOMMU's basically useless by working around one broken
> driver.

Agreed.

> Proper interfaces for what this driver is trying to do have existed in
> the kernel for 10+ years.  There are no excuses, and it's time for
> Nvidia to get with the game plan.

Agreed.

Joerg

Re: IOMMU and graphics cards

[David Woodhouse]

On Fri, 2009-05-08 at 11:35 +0200, Joerg Roedel wrote:
> The other way to achieve consistency is to remove the workaround from
> VT-d code which significantly increases the chance to get the broken
> stuff fixed. David?

It's gone from 2.6.31-rc2, but even the in-tree drivers break, so I
think we have to put it back for 2.6.31 (I've sent Linus the pull
request already). 

I do intend to remove it again for 2.6.32 though, and I filed 
http://bugzilla.kernel.org/show_bug.cgi?id=13721

For now it's got a clearer description ("Workaround broken graphics
drivers") and defaults to 'n'.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: IOMMU and graphics cards

[Joerg Roedel]

On Mon, Jul 06, 2009 at 01:26:17PM +0100, David Woodhouse wrote:
> On Fri, 2009-05-08 at 11:35 +0200, Joerg Roedel wrote:
> > The other way to achieve consistency is to remove the workaround from
> > VT-d code which significantly increases the chance to get the broken
> > stuff fixed. David?
> 
> It's gone from 2.6.31-rc2, but even the in-tree drivers break, so I
> think we have to put it back for 2.6.31 (I've sent Linus the pull
> request already). 
> 
> I do intend to remove it again for 2.6.32 though, and I filed 
> http://bugzilla.kernel.org/show_bug.cgi?id=13721
> 
> For now it's got a clearer description ("Workaround broken graphics
> drivers") and defaults to 'n'.

Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU
for you? I'll may probably add a similar temporary workaround for AMD
IOMMU too...

	Joerg

-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

Re: IOMMU and graphics cards

[David Woodhouse]

On Mon, 2009-07-06 at 15:11 +0200, Joerg Roedel wrote:
> Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU
> for you? I'll may probably add a similar temporary workaround for AMD
> IOMMU too...

The Intel one definitely broke -- I don't know about the others. There
are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
which make it look like _all_ AGP drivers are broken.

I wouldn't bother adding the workaround -- as I said, I'm planning to
rip it out of 2.6.32 (and in linux-next as soon as it's reasonable to do
so). Let's just let them fix it.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: IOMMU and graphics cards

[Dave Airlie]

On Mon, 6 Jul 2009, David Woodhouse wrote:

> On Mon, 2009-07-06 at 15:11 +0200, Joerg Roedel wrote:
> > Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU
> > for you? I'll may probably add a similar temporary workaround for AMD
> > IOMMU too...
> 
> The Intel one definitely broke -- I don't know about the others. There
> are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
> which make it look like _all_ AGP drivers are broken.
> 
> I wouldn't bother adding the workaround -- as I said, I'm planning to
> rip it out of 2.6.32 (and in linux-next as soon as it's reasonable to do
> so). Let's just let them fix it.

cc'ing Eric,

My memory of this is graphics becomes totally useless and can be 10x-50x 
slower. I think ripping this out without the person doing the ripping 
taking responsiblity for doing speed regression testing is totally insane.

I personally have no IOMMU hw from Intel or AMD and nobody has seen it fit 
to supply me with any at any point in time, I'm not on the correct gravy 
train. So I suspect the people with the hw will have to do the work and 
the regression testing.

Dave.

Re: IOMMU and graphics cards

[Dave Airlie]

On 07/07/2009, at 7:35, Dave Airlie <airlied@linux.ie> wrote:

>
>
> On Mon, 6 Jul 2009, David Woodhouse wrote:
>
>> On Mon, 2009-07-06 at 15:11 +0200, Joerg Roedel wrote:
>>> Ok, cool, that sounds good. Which in-kernel DRM drivers break with  
>>> IOMMU
>>> for you? I'll may probably add a similar temporary workaround for  
>>> AMD
>>> IOMMU too...
>>
>> The Intel one definitely broke -- I don't know about the others.  
>> There
>> are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
>> which make it look like _all_ AGP drivers are broken.
>>
>> I wouldn't bother adding the workaround -- as I said, I'm planning to
>> rip it out of 2.6.32 (and in linux-next as soon as it's reasonable  
>> to do
>> so). Let's just let them fix it.
>
> cc'ing Eric,
>
> My memory of this is graphics becomes totally useless and can be  
> 10x-50x
> slower. I think ripping this out without the person doing the ripping
> taking responsiblity for doing speed regression testing is totally  
> insane.
>
> I personally have no IOMMU hw from Intel or AMD and nobody has seen  
> it fit
> to supply me with any at any point in time, I'm not on the correct  
> gravy
> train. So I suspect the people with the hw will have to do the work  
> and
> the regression testing.

Could you also enumerate any limitations of the IOMMUs on the amount  
of memory they can remap per device if any.

Dave

> Dave.
> --
> To unsubscribe from this list: send the line "unsubscribe linux- 
> kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: IOMMU and graphics cards

[Joerg Roedel]

On Tue, Jul 07, 2009 at 08:00:01AM +1000, Dave Airlie wrote:
> 
> Could you also enumerate any limitations of the IOMMUs on the amount
> of memory they can remap per device if any.

The AMD IOMMU driver in 2.6.31-rcX can map up to 4GB per domain. If the
gfx card has a domain of its own it can use all of the 4GB.
I am not 100% sure about VT-d, but as far as I understood the allocator
it doesn't have that 4GB limitation per domain.

	Joerg

-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

RE: IOMMU and graphics cards

[Duran, Leo]

On 07/07/2009, at 10:08 CST, Dave Airlie <airlied@linux.ie> wrote:
> Could you also enumerate any limitations of the IOMMUs
> on the amount of memory they can remap per device if any.

Here's the spec for the AND IOMMU:
http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/
34434.pdf

Below is a cut-paste of "Table 3: Device Table Entry Field Definitions".
As an example, paging mode 4 supports a 48-bit virtual address space
(well beyond 4GB's). In terms of physical address space, "Figure 7: I/O
Page Translation Entry (PTE)" shows supports for up-to a 52-bit page
address.

Leo.


11:9 Mode: paging mode. Specify how the IOMMU performs page translation
on behalf of the device. If
page translation is enabled, the mode specifies the depth of the
device's I/O page tables (1 to 6
levels).

000b Translation disabled (Access controlled by IR and IW bits)
001b 1 Level Page Table (provides a 21-bit device virtual address space)
010b 2 Level Page Table (provides a 30-bit device virtual address space)
011b 3 Level Page Table (provides a 39-bit device virtual address space)
100b 4 Level Page Table (provides a 48-bit device virtual address space)
101b 5 Level Page Table (provides a 57-bit device virtual address space)
110b 6 Level Page Table (provides a 64-bit device virtual address space)
111b Reserved
Note: the page table root pointer is ignored when Mode=000b and when
Mode=111b.
Note: Mode=111b is reported as an error when V=1 and TV=1.

RE: IOMMU and graphics cards

[Duran, Leo]

Here's the spec for the AND IOMMU (with braces):
<http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs
/34434.pdf>

HTH,
Leo.

RE: IOMMU and graphics cards

[David Woodhouse]

On Tue, 2009-07-07 at 10:33 -0500, Duran, Leo wrote:
> Here's the spec for the AND IOMMU (with braces):
> <http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs
> /34434.pdf>

It wasn't the lack of braces that was the problem -- it was the presence
of a spurious newline in the middle.

Here's a nickel, kid -- get yourself a proper mail program. One that can
cope with the 1980s invention known as WYSIWYG, perhaps?

http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/34434.pdf

-- 
dwmw2

Re: IOMMU and graphics cards

[Joerg Roedel]

On Mon, Jul 06, 2009 at 10:35:59PM +0100, Dave Airlie wrote:
> 
> 
> On Mon, 6 Jul 2009, David Woodhouse wrote:
> 
> > On Mon, 2009-07-06 at 15:11 +0200, Joerg Roedel wrote:
> > > Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU
> > > for you? I'll may probably add a similar temporary workaround for AMD
> > > IOMMU too...
> > 
> > The Intel one definitely broke -- I don't know about the others. There
> > are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
> > which make it look like _all_ AGP drivers are broken.
> > 
> > I wouldn't bother adding the workaround -- as I said, I'm planning to
> > rip it out of 2.6.32 (and in linux-next as soon as it's reasonable to do
> > so). Let's just let them fix it.
> 
> cc'ing Eric,
> 
> My memory of this is graphics becomes totally useless and can be 10x-50x 
> slower. I think ripping this out without the person doing the ripping 
> taking responsiblity for doing speed regression testing is totally insane.

Are you sure that using the dma-api has such an performance impact? I've
heard from other people that switching to dma-api with amd iommu had no
significant performance impact.

> I personally have no IOMMU hw from Intel or AMD and nobody has seen it fit 
> to supply me with any at any point in time, I'm not on the correct gravy 
> train. So I suspect the people with the hw will have to do the work and 
> the regression testing.

You just need to switch to dma-api. You don't need an iommu to test.
Most bugs in such code can also be found and eliminated using the
dma-api debugging interface.

	Joerg

-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

Re: IOMMU and graphics cards

[David Woodhouse]

On Tue, 2009-07-07 at 10:59 +0200, Joerg Roedel wrote:
> On Mon, Jul 06, 2009 at 10:35:59PM +0100, Dave Airlie wrote:
> > My memory of this is graphics becomes totally useless and can be 10x-50x 
> > slower. I think ripping this out without the person doing the ripping 
> > taking responsiblity for doing speed regression testing is totally insane.
> 
> Are you sure that using the dma-api has such an performance impact? I've
> heard from other people that switching to dma-api with amd iommu had no
> significant performance impact.

The DMA API on its own shouldn't hurt at all. and I'm not sure about the
AMD IOMMU.

It's the other IOMMU that hurts -- something about cache-incoherent page
tables and having to flush the cache for every change... especially on
an architecture which doesn't have any sane cache management (because it
was always designed for everything to be cache coherent, dammit!) so you
end up _invalidating_ the cache at the same time as you write it back!

Of course, it's a lot saner now that we don't individually flush the
cache after _every_ PTE we change, but do it in batches instead.
Multi-page mapping/unmapping is now a _lot_ faster than it used to be.

I'm also looking at mapping the page tables uncached, so we don't have
to do the cache flushes at all.

> > I personally have no IOMMU hw from Intel or AMD and nobody has seen it fit 
> > to supply me with any at any point in time, I'm not on the correct gravy 
> > train. So I suspect the people with the hw will have to do the work and 
> > the regression testing.
> 
> You just need to switch to dma-api. You don't need an iommu to test.
> Most bugs in such code can also be found and eliminated using the
> dma-api debugging interface.

Indeed. Just turn on CONFIG_DMA_API_DEBUG (that might have some
performance impact, but it's not huge).

-- 
dwmw2

Re: IOMMU and graphics cards

[David Woodhouse]

On Mon, 2009-07-06 at 22:35 +0100, Dave Airlie wrote:
> 
> On Mon, 6 Jul 2009, David Woodhouse wrote:
> 
> > On Mon, 2009-07-06 at 15:11 +0200, Joerg Roedel wrote:
> > > Ok, cool, that sounds good. Which in-kernel DRM drivers break with IOMMU
> > > for you? I'll may probably add a similar temporary workaround for AMD
> > > IOMMU too...
> > 
> > The Intel one definitely broke -- I don't know about the others. There
> > are some old patches at http://people.freedesktop.org/~zhen/agp-mm-*
> > which make it look like _all_ AGP drivers are broken.
> > 
> > I wouldn't bother adding the workaround -- as I said, I'm planning to
> > rip it out of 2.6.32 (and in linux-next as soon as it's reasonable to do
> > so). Let's just let them fix it.
> 
> cc'ing Eric,
> 
> My memory of this is graphics becomes totally useless and can be 10x-50x 
> slower. I think ripping this out without the person doing the ripping 
> taking responsiblity for doing speed regression testing is totally insane.

If the IOMMU is absent, disabled or in pass-through mode, then the
performance impact should be virtually zero.

It's the case where the IOMMU is in use that matters -- and the choice
there is between "slow" and "broken". I'll take "slow", please.

Having said that, I've done a bunch of performance work recently and
especially multi-page mappings are a _lot_ faster than they used to be.
I'll see if I can do more once I can test it with your actual usage
patterns.

We currently have an evil hack which uses a shitload of memory to set up
a full set of page tables to map all of physical memory, for every
graphics device at boot time whether we need it or not. The AMD folks
have so far resisted doing that, partly because of the amount of RAM it
would take. I'm getting poked from all sides to _remove_ our hack.

> I personally have no IOMMU hw from Intel or AMD and nobody has seen it fit 
> to supply me with any at any point in time, I'm not on the correct gravy 
> train. So I suspect the people with the hw will have to do the work and 
> the regression testing.

Anything with Cantiga chipset would suffice. Lenovo x200s, T400, etc...

> Could you also enumerate any limitations of the IOMMUs on the amount  
> of memory they can remap per device if any.

The Intel one basically has no relevant limit -- it's 54 bits of virtual
address space or something like that.

I think I saw something about the AMD one perhaps being limited to 4GiB
per device -- which makes 1:1 mapping of all memory a lot less feasible,
but I'll let Jörg answer that question definitively.

Not sure about other platforms off-hand.

-- 
dwmw2

Re: IOMMU and graphics cards

[FUJITA Tomonori]

On Thu, 7 May 2009 12:01:14 +0200
Joerg Roedel <joerg.roedel@amd.com> wrote:

> > Do you meant that AMD IOMMU code use the bitmap each device to manage
> > address space so enlarging the DMA address space wastes memory?
> 
> Its not only the bitmap. There are also the page tables that would waste
> memory if the aperture in the AMD IOMMU driver is enlarged to, say, 4GB.
> Most devices don't need such a large amount of DMA address space. As I
> found out VT-d does not have this problem because they implemented
> another allocator which can cover all possible memory.
> For the AMD IOMMU I currently implement a kind of a dynamically growing
> bitmap allocator to fix this problem.

Yeah, I know the deference. Well, I like the AMD IOMMU driver's
property that dma mapping doesn't fail in OOM but I understand why you
want such daynamically growing.


> > >  But unless this problem isn't solved the
> > > drivers won't be fixed, I guess.
> > > I guess the DRM code in the kernel may have the same problem with IOMMU
> > > enabled?
> > 
> > Looks like the DRM code uses the DMA API.
> 
> The proprietary drivers make problems so far. For the ATI one I am in
> contact with the developers to try to fix it. But I can't do anything
> about the other proprietary driver I am aware of :-(

I don't know anything about the graphic drivers but are there any
other proprietary drivers except for ATI and AMD? Fixing only two
drivers to make the majority happy?

Re: IOMMU and graphics cards

[Ingo Molnar]

* FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp> wrote:

> > The proprietary drivers make problems so far. For the ATI one I 
> > am in contact with the developers to try to fix it. But I can't 
> > do anything about the other proprietary driver I am aware of :-(
> 
> I don't know anything about the graphic drivers but are there any 
> other proprietary drivers except for ATI and AMD? Fixing only two 
> drivers to make the majority happy?

ATI ~== AMD these days. The problem is Nvidia cards stuck into AMD 
systems.

I guess refusing those DMA accesses (and printing something 
meaningful and relentlessly honest so that the user knows where the 
problem comes from) is the proper solution. There will be some 
end-user moaning, there might be distros turning off the IOMMU code 
summarily, but nvidia-in-AMD-box is an unfriendly alliance to begin 
with.

	Ingo

Re: IOMMU and graphics cards

[FUJITA Tomonori]

On Thu, 7 May 2009 13:01:35 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp> wrote:
> 
> > > The proprietary drivers make problems so far. For the ATI one I 
> > > am in contact with the developers to try to fix it. But I can't 
> > > do anything about the other proprietary driver I am aware of :-(
> > 
> > I don't know anything about the graphic drivers but are there any 
> > other proprietary drivers except for ATI and AMD? Fixing only two 
> > drivers to make the majority happy?
> 
> ATI ~== AMD these days. The problem is Nvidia cards stuck into AMD 
> systems.

Oops, I meant ATI and Nvidia.


> I guess refusing those DMA accesses (and printing something 
> meaningful and relentlessly honest so that the user knows where the 
> problem comes from) is the proper solution.

Refusing DMA means that we will break these broken drivers. That's
what David and I like. Yeah, telling users explicitly who is to blame
for the problem is even better.

> There will be some 
> end-user moaning, there might be distros turning off the IOMMU code 
> summarily, but nvidia-in-AMD-box is an unfriendly alliance to begin 
> with.

Re: IOMMU and graphics cards

[Ingo Molnar]

* FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp> wrote:

> On Thu, 7 May 2009 13:01:35 +0200
> Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp> wrote:
> > 
> > > > The proprietary drivers make problems so far. For the ATI one I 
> > > > am in contact with the developers to try to fix it. But I can't 
> > > > do anything about the other proprietary driver I am aware of :-(
> > > 
> > > I don't know anything about the graphic drivers but are there any 
> > > other proprietary drivers except for ATI and AMD? Fixing only two 
> > > drivers to make the majority happy?
> > 
> > ATI ~== AMD these days. The problem is Nvidia cards stuck into AMD 
> > systems.
> 
> Oops, I meant ATI and Nvidia.
> 
> 
> > I guess refusing those DMA accesses (and printing something 
> > meaningful and relentlessly honest so that the user knows where the 
> > problem comes from) is the proper solution.
> 
> Refusing DMA means that we will break these broken drivers. That's 
> what David and I like. Yeah, telling users explicitly who is to 
> blame for the problem is even better.

If it means a non-working Xorg, then there might be nothing on the 
screen to report - just a seemingly hung box and people will blame 
the kernel. It might be better to just turn off all things IOMMU at 
that point, but still allow things to continue.

... which might break good drivers that relied on the IOMMU sorting 
out 32-bit DMA space limitations.

So it would be nice to make the failure mode somehow nicer. I.e. 
test how this affects the nvidia driver. Is there something on the 
screen to see? Could we print an URL to the Noveau driver perhaps 
too?

A bad solution in general is to crash/hang a significant proportion 
of Linux boxes. That is a foot-in-own-mouth masochistic excercise 
mostly, it creates a stigma for the IOMMU code, not for nvidia. So 
the failure mode has to be well thought out.

	Ingo

Re: IOMMU and graphics cards

[Joerg Roedel]

On Thu, May 07, 2009 at 07:23:33PM +0900, FUJITA Tomonori wrote:
> On Thu, 7 May 2009 12:01:14 +0200
> Joerg Roedel <joerg.roedel@amd.com> wrote:
> 
> > > Do you meant that AMD IOMMU code use the bitmap each device to manage
> > > address space so enlarging the DMA address space wastes memory?
> > 
> > Its not only the bitmap. There are also the page tables that would waste
> > memory if the aperture in the AMD IOMMU driver is enlarged to, say, 4GB.
> > Most devices don't need such a large amount of DMA address space. As I
> > found out VT-d does not have this problem because they implemented
> > another allocator which can cover all possible memory.
> > For the AMD IOMMU I currently implement a kind of a dynamically growing
> > bitmap allocator to fix this problem.
> 
> Yeah, I know the deference. Well, I like the AMD IOMMU driver's
> property that dma mapping doesn't fail in OOM but I understand why you
> want such daynamically growing.

Well, I will try to minimize the impact. Current plan is to preallocate
bitmap and page tables for 128MB of DMA address space. Everything beyond
that will be allocated on demand.

> > > >  But unless this problem isn't solved the
> > > > drivers won't be fixed, I guess.
> > > > I guess the DRM code in the kernel may have the same problem with IOMMU
> > > > enabled?
> > > 
> > > Looks like the DRM code uses the DMA API.
> > 
> > The proprietary drivers make problems so far. For the ATI one I am in
> > contact with the developers to try to fix it. But I can't do anything
> > about the other proprietary driver I am aware of :-(
> 
> I don't know anything about the graphic drivers but are there any
> other proprietary drivers except for ATI and AMD? Fixing only two
> drivers to make the majority happy?

At least Nvidia comes to my mind. Not sure about other graphic card
vendors...

Joerg

-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

Re: IOMMU and graphics cards

[Ingo Molnar]

* Joerg Roedel <joerg.roedel@amd.com> wrote:

> On Thu, May 07, 2009 at 04:47:14PM +0900, FUJITA Tomonori wrote:
> > On Tue, 28 Apr 2009 18:04:48 +0200
> > Joerg Roedel <joerg.roedel@amd.com> wrote:
> > > 
> > > Yes, it is in effect for other devices. But since its a security feature
> > > it only makes sense if it covers all devices.
> > 
> > For me, IOMMUs are not about security but accessing all memory space
> > (there are still lots of devices that are not capable of 64-bit DMA)
> > but I agree that this workaround is bad.
> 
> Yeah true. Another major reason for having an IOMMU is to enlarge the
> physical address range devices can access.
> 
> > Do you meant that AMD IOMMU code use the bitmap each device to manage
> > address space so enlarging the DMA address space wastes memory?
> 
> Its not only the bitmap. There are also the page tables that would waste
> memory if the aperture in the AMD IOMMU driver is enlarged to, say, 4GB.
> Most devices don't need such a large amount of DMA address space. As I
> found out VT-d does not have this problem because they implemented
> another allocator which can cover all possible memory.
> For the AMD IOMMU I currently implement a kind of a dynamically growing
> bitmap allocator to fix this problem.
> 
> > >  But unless this problem isn't solved the
> > > drivers won't be fixed, I guess.
> > > I guess the DRM code in the kernel may have the same problem with IOMMU
> > > enabled?
> > 
> > Looks like the DRM code uses the DMA API.
> 
> The proprietary drivers make problems so far. For the ATI one I am 
> in contact with the developers to try to fix it. But I can't do 
> anything about the other proprietary driver I am aware of :-(

One option would be to enumerate nvidia vga device memory resources, 
print a nasty warning if they are accessed while not mapped by any 
in-kernel driver (such as Noveau) and then install a workaround for 
their resources and exclude them from IOMMU coverage.

	Ingo