Re: PCI-DMA: Out of IOMMU space on x86-64 (Athlon64x2), with solution

[Andi Kleen <ak@suse.de>]

> - We have in-driver pending stuff, so we can just retry the operation
>   later when some of that completes.
> - We are unlucky enough that someone else holds all the resources, we
>   have nothing to wait for.

I suspect the second is more common - typically the problem seems to happen
when people have multiple devices active that need the IOMMU in parallel.

> The first case is easy, just punt and retry when some of your io
> completes. The last case requires a way to wait on the iommu as you
> describe, which the driver needs to do somewhere safe.

Also where to put the wait queue? The IOMMU code only 
sees the bus devices not the queues and I'm not sure the low level
devices would be the right place to put it because it wouldn't handle
the case of a queue having multiple devices well and in general
would probably violate the layers.

Maybe just using a global one? The situation should be rare anyways.
Would just need a way to detect this case to avoid bouncing the cache lines
of the wait queue in the normal case. Perhaps a simple global counter
would be good enough for that.

e.g. you increase the counter and then the IOMMU code just does a wakeup
on a global waitqueue every time it frees space.

Hrm one problem I guess is that you need to make sure there are no 
races between detection of the low space condition and the increasing
of the counter, but some lazy locking and rechecking might be able 
to cure that.

-Andi