Swiotlb

Swiotlb

[Kieran Mansley]

While writing a driver for a device doing lots of DMA I've hit an
"swiotlb_full()" problem.  This surprised me somewhat as I wouldn't have
expected to need the use of the software TLB - it's a 64 bit capable
device on a server with only 2 GB of RAM, and so I'd have expected to be
using a hardware TLB.  Is this a peculiarity of Xen, or should I be
right to be surprised?

I expect I can increase the size of the swiotlb to avoid this problem,
but thought I'd check that there isn't something more fundamentally
wrong first.

Thanks

Kieran

Re: Swiotlb

[Keir Fraser]

On 5/4/07 15:23, "Kieran Mansley" <kmansley@solarflare.com> wrote:

> While writing a driver for a device doing lots of DMA I've hit an
> "swiotlb_full()" problem.  This surprised me somewhat as I wouldn't have
> expected to need the use of the software TLB - it's a 64 bit capable
> device on a server with only 2 GB of RAM, and so I'd have expected to be
> using a hardware TLB.  Is this a peculiarity of Xen, or should I be
> right to be surprised?
> 
> I expect I can increase the size of the swiotlb to avoid this problem,
> but thought I'd check that there isn't something more fundamentally
> wrong first.

Well, I would agree that it sounds odd. You should definitely investigate --
you don't want to be invoking the swiotlb on the data path for a modern
high-speed device.

 -- Keir

Re: Swiotlb

[Kieran Mansley]

On Thu, 2007-04-05 at 16:10 +0100, Keir Fraser wrote:
> On 5/4/07 15:23, "Kieran Mansley" <kmansley@solarflare.com> wrote:
> > While writing a driver for a device doing lots of DMA I've hit an
> > "swiotlb_full()" problem.  This surprised me somewhat as I wouldn't have
> > expected to need the use of the software TLB - it's a 64 bit capable
> > device on a server with only 2 GB of RAM, and so I'd have expected to be
> > using a hardware TLB.  Is this a peculiarity of Xen, or should I be
> > right to be surprised?
> > 
> > I expect I can increase the size of the swiotlb to avoid this problem,
> > but thought I'd check that there isn't something more fundamentally
> > wrong first.
> 
> Well, I would agree that it sounds odd. You should definitely investigate --
> you don't want to be invoking the swiotlb on the data path for a modern
> high-speed device.

It's not quite on the data path - it's hitting this problem when
allocating buffers for network access, which typically will come from a
preallocated pool rather than being on demand, so perhaps it's not a big
problem.

I've had a quick look at what's going on.  The call into swiotlb comes
from dma_map_single() (which is I think the one defined in
arch/i386/kernel/pci-dma-xen.c).  This happens because the swiotlb
global is set to 1 in swiotlb_init().  It looks like this is unavoidable
as there's a comment there saying:
        /* Domain 0 always has a swiotlb. */

I could undefine CONFIG_SWIOTLB or set swiotlb_force to -1 and so avoid
the whole thing, but I'm not sure that would be a good thing - some
other drivers might depend on it.

I guess what I'm surprised about is that there's no check to see if
swiotlb is necessary on any of the above paths.  Perhaps I should be
calling something other than dma_map_single()?  Or perhaps this is just
the way it's supposed to be and I need to allocate a bigger swiotlb?
Any suggestions welcome.

Kieran

Re: Swiotlb

[Muli Ben-Yehuda]

On Tue, Apr 10, 2007 at 09:41:55AM +0100, Kieran Mansley wrote:

> It's not quite on the data path - it's hitting this problem when
> allocating buffers for network access, which typically will come
> from a preallocated pool rather than being on demand, so perhaps
> it's not a big problem.
> 
> I've had a quick look at what's going on.  The call into swiotlb comes
> from dma_map_single() (which is I think the one defined in
> arch/i386/kernel/pci-dma-xen.c).  This happens because the swiotlb
> global is set to 1 in swiotlb_init().  It looks like this is unavoidable
> as there's a comment there saying:
>         /* Domain 0 always has a swiotlb. */
> 
> I could undefine CONFIG_SWIOTLB or set swiotlb_force to -1 and so
> avoid the whole thing, but I'm not sure that would be a good thing -
> some other drivers might depend on it.

swiotlb is used to give you machine-contigous buffers if your buffers
aren't machine-contigous. Just because it's enabled doesn't mean it
will be used - there's a check whether it's needed in every map/alloc
call and if it isn't it's not used. If all your buffers are
machine-contigous you shouldn't ever need to use any space in the
swiotlb aperture. If they aren't, that's probably the real problem.

Cheers,
Muli

swiotlb

[Ky Srinivasan]

Given that the function __sync_single() can be called in an interrupt
context, why do we call a   potentially blocking function
(__copy_to_user) from within this function?

Regards,

K. Y

Re: swiotlb

[Don Dutile]

Ky,
You are correct. I received this as a bugzilla on FC-5.  Testing a 
solution now (replacing copy_to_user w/copy_to_user_inatomic).
The other problem with the call to copy_to_user is that a kmap_atomic() 
is done just before it as well, and the thread can't be blocked before 
the kunmap_atomic() is invoked (after the copy_to_user()).

Unfortunately, tomorrow is company holiday, so I won't complete the 
testing until Monday; need to get more memory to force swiotlb's 
sync_single() to do the copy (due to bounce IO).

- Don


Ky Srinivasan wrote:
> Given that the function __sync_single() can be called in an interrupt
> context, why do we call a   potentially blocking function
> (__copy_to_user) from within this function?
> 
> Regards,
> 
> K. Y
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: swiotlb

[Ky Srinivasan]

Thanks Don. I am seeing this problem on SLES10 as well. I will wait for
your fix.

K. Y 
 
>>> Don Dutile <ddutile@redhat.com> 04/13/06 9:51 pm >>> 
Ky,
You are correct. I received this as a bugzilla on FC- 5.  Testing a 
solution now (replacing copy_to_user w/copy_to_user_inatomic).
The other problem with the call to copy_to_user is that a kmap_atomic()

is done just before it as well, and the thread can't be blocked before

the kunmap_atomic() is invoked (after the copy_to_user()).

Unfortunately, tomorrow is company holiday, so I won't complete the 
testing until Monday; need to get more memory to force swiotlb's 
sync_single() to do the copy (due to bounce IO).

-  Don


Ky Srinivasan wrote:
> Given that the function __sync_single() can be called in an
interrupt
> context, why do we call a   potentially blocking function
> (__copy_to_user) from within this function?
> 
> Regards,
> 
> K. Y
> 
> 
> _______________________________________________
> Xen- devel mailing list
> Xen- devel@lists.xensource.com
> http://lists.xensource.com/xen- devel

_______________________________________________
Xen- devel mailing list
Xen- devel@lists.xensource.com
http://lists.xensource.com/xen- devel

Re: swiotlb

[Keir Fraser]

On 13 Apr 2006, at 21:29, Ky Srinivasan wrote:

> Given that the function __sync_single() can be called in an interrupt
> context, why do we call a   potentially blocking function
> (__copy_to_user) from within this function?

Good point. Would __copy_to_user_inatomic() be safe? It avoids 
might_sleep(). We know that the page fault itself (if any) will not 
block on I/O -- we only use __copy_to_user() to safely catch the fault 
and return.

  -- Keir

Re: swiotlb

[Ky Srinivasan]

That should do the trick. 

K. Y 
 
>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 04/14/06 2:50 am >>> 

On 13 Apr 2006, at 21:29, Ky Srinivasan wrote:

> Given that the function __sync_single() can be called in an
interrupt
> context, why do we call a   potentially blocking function
> (__copy_to_user) from within this function?

Good point. Would __copy_to_user_inatomic() be safe? It avoids 
might_sleep(). We know that the page fault itself (if any) will not 
block on I/O --  we only use __copy_to_user() to safely catch the fault

and return.

  --  Keir


_______________________________________________
Xen- devel mailing list
Xen- devel@lists.xensource.com
http://lists.xensource.com/xen- devel