[PATCH 0/3] myri10ge Large Receive Offload

[PATCH 0/3] myri10ge Large Receive Offload

[Brice Goglin]

This is a complete rework of the myri10ge receive path. The first
patch converts skb allocation to use physical pages. The second one
adds a software implementation of Large Receive Offload. The third
one updates the driver version to 1.1.0.

The complete driver code in our CVS actually also supports high-order
allocations instead of single physical pages since it significantly
increase the performance. Order=2 allows us to receive standard frames
at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2
Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot
about memory fragmentation if the performance is better.

But, since high-order allocations are generally considered a bad idea,
we do not include the relevant code in the following patch for inclusion
in Linux. Here, we simply pass order=0 to all page allocation routines.
If necessary, I could drop the remaining reference to high-order
(especially replace alloc_pages() with alloc_page()) but I'd rather
keep it as is.

If high-order allocations are ever considered OK under some circum-
stances, we could send an additional patch (a module parameter would
be used to switch from single physical pages to high-order pages).

Thanks,
Brice

Re: [PATCH 0/3] myri10ge Large Receive Offload

[Herbert Xu]

Brice Goglin <brice@myri.com> wrote:
> This is a complete rework of the myri10ge receive path. The first
> patch converts skb allocation to use physical pages. The second one
> adds a software implementation of Large Receive Offload. The third
> one updates the driver version to 1.1.0.

If this LRO is completely in software it might be time to consider
something more generic since this would be the second software LRO
implementation (the other being s2io).

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/3] myri10ge Large Receive Offload

[Jeff Garzik]

Brice Goglin wrote:
> This is a complete rework of the myri10ge receive path. The first
> patch converts skb allocation to use physical pages. The second one
> adds a software implementation of Large Receive Offload. The third
> one updates the driver version to 1.1.0.
> 
> The complete driver code in our CVS actually also supports high-order
> allocations instead of single physical pages since it significantly
> increase the performance. Order=2 allows us to receive standard frames
> at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2
> Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot
> about memory fragmentation if the performance is better.
> 
> But, since high-order allocations are generally considered a bad idea,
> we do not include the relevant code in the following patch for inclusion
> in Linux. Here, we simply pass order=0 to all page allocation routines.
> If necessary, I could drop the remaining reference to high-order
> (especially replace alloc_pages() with alloc_page()) but I'd rather
> keep it as is.
> 
> If high-order allocations are ever considered OK under some circum-
> stances, we could send an additional patch (a module parameter would
> be used to switch from single physical pages to high-order pages).

As Herbert's already done, I would rather let the net core people 
comment on this.  The code implementation doesn't look scary, but we may 
want to be smarter about this in the core net stack, rather than 
implementing it inside multiple drivers.

	Jeff

Re: [PATCH 0/3] myri10ge Large Receive Offload

[Brice Goglin]

Jeff Garzik a écrit :
> Brice Goglin wrote:
>> This is a complete rework of the myri10ge receive path. The first
>> patch converts skb allocation to use physical pages. The second one
>> adds a software implementation of Large Receive Offload. The third
>> one updates the driver version to 1.1.0.
>>
>> The complete driver code in our CVS actually also supports high-order
>> allocations instead of single physical pages since it significantly
>> increase the performance. Order=2 allows us to receive standard frames
>> at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2
>> Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot
>> about memory fragmentation if the performance is better.
>>
>> But, since high-order allocations are generally considered a bad idea,
>> we do not include the relevant code in the following patch for inclusion
>> in Linux. Here, we simply pass order=0 to all page allocation routines.
>> If necessary, I could drop the remaining reference to high-order
>> (especially replace alloc_pages() with alloc_page()) but I'd rather
>> keep it as is.
>>
>> If high-order allocations are ever considered OK under some circum-
>> stances, we could send an additional patch (a module parameter would
>> be used to switch from single physical pages to high-order pages).
>
> As Herbert's already done, I would rather let the net core people
> comment on this.  The code implementation doesn't look scary, but we
> may want to be smarter about this in the core net stack, rather than
> implementing it inside multiple drivers.

Ok, makes sense. We look forward to see this.

Could we get patch #1 merged anyway (page-based skb allocation)?

Any comments about what I was saying about high-order allocations above?

thanks,
Brice

Re: [PATCH 0/3] myri10ge Large Receive Offload

[Evgeniy Polyakov]

On Sat, Sep 30, 2006 at 12:16:44AM +0200, Brice Goglin (brice@myri.com) wrote:
> Jeff Garzik a écrit :
> > Brice Goglin wrote:
> >> This is a complete rework of the myri10ge receive path. The first
> >> patch converts skb allocation to use physical pages. The second one
> >> adds a software implementation of Large Receive Offload. The third
> >> one updates the driver version to 1.1.0.
> >>
> >> The complete driver code in our CVS actually also supports high-order
> >> allocations instead of single physical pages since it significantly
> >> increase the performance. Order=2 allows us to receive standard frames
> >> at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2
> >> Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot
> >> about memory fragmentation if the performance is better.
> >>
> >> But, since high-order allocations are generally considered a bad idea,
> >> we do not include the relevant code in the following patch for inclusion
> >> in Linux. Here, we simply pass order=0 to all page allocation routines.
> >> If necessary, I could drop the remaining reference to high-order
> >> (especially replace alloc_pages() with alloc_page()) but I'd rather
> >> keep it as is.
> >>
> >> If high-order allocations are ever considered OK under some circum-
> >> stances, we could send an additional patch (a module parameter would
> >> be used to switch from single physical pages to high-order pages).
> >
> > As Herbert's already done, I would rather let the net core people
> > comment on this.  The code implementation doesn't look scary, but we
> > may want to be smarter about this in the core net stack, rather than
> > implementing it inside multiple drivers.
> 
> Ok, makes sense. We look forward to see this.
> 
> Could we get patch #1 merged anyway (page-based skb allocation)?
> 
> Any comments about what I was saying about high-order allocations above?

It is quite strnage that you see very noticeble speed degradation after
switching from higher order to 0 order allocations, could specify where 
observed  bottleneck in network stack is?

> thanks,
> Brice

-- 
	Evgeniy Polyakov

Re: [PATCH 0/3] myri10ge Large Receive Offload

[Brice Goglin]

Brice Goglin a écrit :
> Could we get patch #1 merged anyway (page-based skb allocation)?
>   

Oops, actually, one hunk has to be dropped (it reverts const'ification
of ethtool_ops by mistake). If it is ok to merge the patch apart from
this hunk,  will resend an updated patch.

Brice

Re: [PATCH 0/3] myri10ge Large Receive Offload

[Brice Goglin]

Evgeniy Polyakov a écrit :
> On Sat, Sep 30, 2006 at 12:16:44AM +0200, Brice Goglin (brice@myri.com) wrote:
>   
>> Jeff Garzik a écrit :
>>     
>>> Brice Goglin wrote:
>>>       
>>>> The complete driver code in our CVS actually also supports high-order
>>>> allocations instead of single physical pages since it significantly
>>>> increase the performance. Order=2 allows us to receive standard frames
>>>> at line rate even on low-end hardware such as an AMD Athlon(tm) 64 X2
>>>> Dual Core Processor 3800+ (2.0GHz). Some customer might not care a lot
>>>> about memory fragmentation if the performance is better.
>>>>
>>>> But, since high-order allocations are generally considered a bad idea,
>>>> we do not include the relevant code in the following patch for inclusion
>>>> in Linux. Here, we simply pass order=0 to all page allocation routines.
>>>> If necessary, I could drop the remaining reference to high-order
>>>> (especially replace alloc_pages() with alloc_page()) but I'd rather
>>>> keep it as is.
>>>>
>>>> If high-order allocations are ever considered OK under some circum-
>>>> stances, we could send an additional patch (a module parameter would
>>>> be used to switch from single physical pages to high-order pages).
>>>>         
>> Any comments about what I was saying about high-order allocations above?
>>     
>
> It is quite strnage that you see very noticeble speed degradation after
> switching from higher order to 0 order allocations, could specify where 
> observed  bottleneck in network stack is?
>   

The bottleneck is not in the network stack, it is simply related to the
number of page allocations that are required. Since we store multiple
fragments in a same page, if MTU=1500, we need one 0-order allocation
every 2 fragments, while we need one 2-order allocation every 8
fragments. IIRC, we observed about 20% higher throughput on the receive
side when switching from order=0 to order=2 (7.5Gbit/s -> 9.3Gbit/s with
roughly the same CPU usage).

Brice