Re: PCI-DMA: Out of IOMMU space on x86-64 (Athlon64x2), with solution

[Jens Axboe <axboe@suse.de>]

On Thu, Mar 02 2006, Andi Kleen wrote:
> 
> > - We have in-driver pending stuff, so we can just retry the operation
> >   later when some of that completes.
> > - We are unlucky enough that someone else holds all the resources, we
> >   have nothing to wait for.
> 
> I suspect the second is more common - typically the problem seems to happen
> when people have multiple devices active that need the IOMMU in parallel.

Hmm I would have guessed the first is way more common, the device/driver
consuming lots of iommu space would be the most likely to run into
IOMMU-OOM.

> > The first case is easy, just punt and retry when some of your io
> > completes. The last case requires a way to wait on the iommu as you
> > describe, which the driver needs to do somewhere safe.
> 
> Also where to put the wait queue? The IOMMU code only 
> sees the bus devices not the queues and I'm not sure the low level
> devices would be the right place to put it because it wouldn't handle
> the case of a queue having multiple devices well and in general
> would probably violate the layers.
> 
> Maybe just using a global one? The situation should be rare anyways.
> Would just need a way to detect this case to avoid bouncing the cache lines
> of the wait queue in the normal case. Perhaps a simple global counter
> would be good enough for that.

I was thinking just a global one, we are in soft error handling anyways
so should be ok. I don't think you would need to dirty any global cache
line unless you actually need to wake waiters.

> e.g. you increase the counter and then the IOMMU code just does a wakeup
> on a global waitqueue every time it frees space.
> 
> Hrm one problem I guess is that you need to make sure there are no 
> races between detection of the low space condition and the increasing
> of the counter, but some lazy locking and rechecking might be able 
> to cure that.

I think so, yes.

-- 
Jens Axboe