jumbo frames and memory fragmentation

jumbo frames and memory fragmentation

[Chris Friesen]

I'm running a system with multiple e1000 devices, using 9KB jumbo 
frames.  I'm running a modified 2.6.10 with e1000 driver 5.5.4-k2.

I'm a bit concerned about the behaviour of this driver with jumbo 
frames.  We ask for 9KB.  The driver then bumps that up to a 
power-of-two, so it calls dev_alloc_skb(16384).  That then bumps it up a 
bit to allow for its own overhead, so it appears that we end up asking 
for 32KB of physically contiguous memory for every packet coming in.  Ouch.

Add to that the fact that this version of the driver doesn't do 
copybreak, and it means that after we're up for a few days it starts 
complaining about not being able to allocate buffers.

Anyone have any suggestions on how to improve this?  Upgrading kernels 
isn't an option.  I could port back the copybreak stuff fairly easily.

Back in 2.4 some of the drivers used to retry buffer allocations using 
GFP_KERNEL once interrupts were reenabled.  I don't see many of them 
doing that anymore--would there be any benefit to that?

Thanks,

Chris

Re: jumbo frames and memory fragmentation

[Herbert Xu]

Chris Friesen <cfriesen@nortel.com> wrote:
> 
> Anyone have any suggestions on how to improve this?  Upgrading kernels 
> isn't an option.  I could port back the copybreak stuff fairly easily.

Either upgrade your kernel or backport the page-splitting code in the
current tree.  That's really the only sane solution for jumbo packets.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: jumbo frames and memory fragmentation

[Chris Friesen]

Herbert Xu wrote:

> Either upgrade your kernel or backport the page-splitting code in the
> current tree.  That's really the only sane solution for jumbo packets.

Looking at the page-splitting code, it says "82571 and greater support 
packet-split...".  We're running the 82546GB device.  Looks like it 
won't help me.

Chris

Re: jumbo frames and memory fragmentation

[Evgeniy Polyakov]

On Fri, Jun 30, 2006 at 11:53:36AM -0600, Chris Friesen (cfriesen@nortel.com) wrote:
> >Either upgrade your kernel or backport the page-splitting code in the
> >current tree.  That's really the only sane solution for jumbo packets.
> 
> Looking at the page-splitting code, it says "82571 and greater support 
> packet-split...".  We're running the 82546GB device.  Looks like it 
> won't help me.

It definitely will.
Packet split in hardware means separating data and headers into
different pages in different reads, while software page split means that 
skb has a list of fragments where part of the packet will be DMAed, so
jumbo frame will be converted into several pages.

-- 
	Evgeniy Polyakov

Re: jumbo frames and memory fragmentation

[Chris Friesen]

Evgeniy Polyakov wrote:

> It definitely will.
> Packet split in hardware means separating data and headers into
> different pages in different reads, while software page split means that 
> skb has a list of fragments where part of the packet will be DMAed, so
> jumbo frame will be converted into several pages.

Maybe I'm looking at the wrong code then.  Can you point me to where 
this "software page split" is handled?

Chris

Re: jumbo frames and memory fragmentation

[Jesse Brandeburg]

On 6/30/06, Chris Friesen <cfriesen@nortel.com> wrote:
> Evgeniy Polyakov wrote:
>
> > It definitely will.
> > Packet split in hardware means separating data and headers into
> > different pages in different reads, while software page split means that
> > skb has a list of fragments where part of the packet will be DMAed, so
> > jumbo frame will be converted into several pages.
>
> Maybe I'm looking at the wrong code then.  Can you point me to where
> this "software page split" is handled?

You're right, it isn't supported for your hardware.  to solve the
problem (I got this from mailing lists) you probably need to adjust
the /proc/sys/vm/min_free_kbytes

let us know if this helps for you.

Re: jumbo frames and memory fragmentation

[Evgeniy Polyakov]

On Fri, Jun 30, 2006 at 05:35:34PM -0600, Chris Friesen (cfriesen@nortel.com) wrote:
> Evgeniy Polyakov wrote:
> 
> >It definitely will.
> >Packet split in hardware means separating data and headers into
> >different pages in different reads, while software page split means that 
> >skb has a list of fragments where part of the packet will be DMAed, so
> >jumbo frame will be converted into several pages.
> 
> Maybe I'm looking at the wrong code then.  Can you point me to where 
> this "software page split" is handled?

Yes, you are correct, e1000 only supports receiving page split for 82571 and
greater adapters.

> Chris

-- 
	Evgeniy Polyakov

Re: jumbo frames and memory fragmentation

[Herbert Xu]

Chris Friesen <cfriesen@nortel.com> wrote:
> Herbert Xu wrote:
> 
>> Either upgrade your kernel or backport the page-splitting code in the
>> current tree.  That's really the only sane solution for jumbo packets.
> 
> Looking at the page-splitting code, it says "82571 and greater support 
> packet-split...".  We're running the 82546GB device.  Looks like it 
> won't help me.

Well, time to fork out for a new card then :)
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: jumbo frames and memory fragmentation

[Chris Friesen]

Herbert Xu wrote:
> Chris Friesen <cfriesen@nortel.com> wrote:

>>Looking at the page-splitting code, it says "82571 and greater support 
>>packet-split...".  We're running the 82546GB device.  Looks like it 
>>won't help me.
> 
> 
> Well, time to fork out for a new card then :)

I wish.  This is an embedded ATCA board.  I'm not going to get a respin 
just because it doesn't deal well with jumbo frames.

Chris