Socket Buffers and Memory Managment

Socket Buffers and Memory Managment

[vinay ravuri]

Hi,

I am fairly new to linux socket buffers and have the
following questions!

I am working with a custom ethernet MAC that does not
allow me to specify a particular memory location for
the h/w to DMA the packet into (Rx side).  Instead, it
has a pool of fixed size buffers with some h/w
specific headers around each buffer that are managed
by h/w and will pick a free buffer and DMA the packet.
 It appears dev_alloc_skb() actually allocates the
physical memory and doesn't allow the user to specify
the skb.data to something specific to what I want
which is a problem for me.  First is my assumption
correct that I am cannot pick an arbitrary skb.data
location in struct sk_buff?  I want to avoid copying
the dma'ed data into a new socket buffer as it is
expense.  Is there any ways around this problem?

Also, if the h/w gives me a single packet in multiple
locations (i.e. non-contiguous chunks of memory), can
socket buffers handle chains of buffers?  I am looking
for a facility like mbuf's in netbsd where one can
chain multiple buffers together to make construct a
single packet.

Please e-mail me responses to vinay_nyc@yahoo.com

Thanks,
Vinay


 
____________________________________________________________________________________
Bored stiff? Loosen up... 
Download and play hundreds of games for free on Yahoo! Games.
http://games.yahoo.com/games/front

Re: Socket Buffers and Memory Managment

[Stephen Hemminger]

On Tue, 17 Jul 2007 10:20:58 -0700 (PDT)
vinay ravuri <vinaynyc@yahoo.com> wrote:

> Hi,
> 
> I am fairly new to linux socket buffers and have the
> following questions!
> 
> I am working with a custom ethernet MAC that does not
> allow me to specify a particular memory location for
> the h/w to DMA the packet into (Rx side).  Instead, it
> has a pool of fixed size buffers with some h/w
> specific headers around each buffer that are managed
> by h/w and will pick a free buffer and DMA the packet.

Sounds like sucky hardware...
You need to copy to a newly allocated skb, see 8139too.c

>  It appears dev_alloc_skb() actually allocates the
> physical memory and doesn't allow the user to specify
> the skb.data to something specific to what I want
> which is a problem for me.  First is my assumption
> correct that I am cannot pick an arbitrary skb.data
> location in struct sk_buff?  I want to avoid copying
> the dma'ed data into a new socket buffer as it is
> expense.  Is there any ways around this problem?

You could play tricks with skb frags but it would be fragile
and not worth the trouble. The problem is that the receive
skb can stay in the system for a really long time (until the application
reads the data) so your fixed size buffer pool in hardware
would get exhausted.
 
> Also, if the h/w gives me a single packet in multiple
> locations (i.e. non-contiguous chunks of memory), can
> socket buffers handle chains of buffers?  I am looking
> for a facility like mbuf's in netbsd where one can
> chain multiple buffers together to make construct a
> single packet.

Yes, skb frag list could be used for that but you don't
want to do that. See above. Copy the data into an new
skb and reserve any necessary bytes so IP header is
aligned.  I.e. if using ethernet header (14 bytes), do
skb_reserve(skb, 2) before copying the data.

Re: Socket Buffers and Memory Managment

[David Miller]

From: Stephen Hemminger <shemminger@linux-foundation.org>
Date: Tue, 17 Jul 2007 20:41:29 +0100

> Sounds like sucky hardware...

Although expect more of the same in the future, not less.

Re: Socket Buffers and Memory Managment

[Roy Pledge]

Stephen Hemminger wrote:
> 
> You could play tricks with skb frags but it would be fragile
> and not worth the trouble. The problem is that the receive
> skb can stay in the system for a really long time (until the application
> reads the data) so your fixed size buffer pool in hardware
> would get exhausted.
>  
Perhaps you could elaborate on why this is considered fragile?  It seems to me 
that as long as proper page management is performed, this mechanism should be 
stable for processing non-contiguous receive buffers.  I agree that 
replenishment needs to be addressed, but I see that as an independent issue from 
using frag lists on receive.

>> Also, if the h/w gives me a single packet in multiple
>> locations (i.e. non-contiguous chunks of memory), can
>> socket buffers handle chains of buffers?  I am looking
>> for a facility like mbuf's in netbsd where one can
>> chain multiple buffers together to make construct a
>> single packet.
> 
> Yes, skb frag list could be used for that but you don't
> want to do that. See above. Copy the data into an new
> skb and reserve any necessary bytes so IP header is
> aligned.  I.e. if using ethernet header (14 bytes), do
> skb_reserve(skb, 2) before copying the data.

But isn't avoiding the copy the point of all this?  With some hardware models 
using skbs as receive buffers is inconvenient and can be wasteful if jumbo 
frames are required.  Wouldn't a mechanism where it is possible to create (or 
populate) an skb from data in a pre existing buffer without copying be more 
general and address more hardware models?  It seems to me that more and more 
hardware is supporting scatter/gather output for received data, but the page* 
model in the frag list can be restrictive in how data is placed before being 
passed into the stack.

Thanks

Roy

Re: Socket Buffers and Memory Managment

[Stephen Hemminger]

On Wed, 18 Jul 2007 13:13:21 -0400
Roy Pledge <Roy.Pledge@freescale.com> wrote:

> Stephen Hemminger wrote:
> > 
> > You could play tricks with skb frags but it would be fragile
> > and not worth the trouble. The problem is that the receive
> > skb can stay in the system for a really long time (until the application
> > reads the data) so your fixed size buffer pool in hardware
> > would get exhausted.
> >  
> Perhaps you could elaborate on why this is considered fragile?  It seems to me 
> that as long as proper page management is performed, this mechanism should be 
> stable for processing non-contiguous receive buffers.  I agree that 
> replenishment needs to be addressed, but I see that as an independent issue from 
> using frag lists on receive.

Imagine you had a 4MB receive area. If you used fraglist to point to
the data area, you would still be at risk of receive lock up when you have
all the receive area used up in skb's waiting in the receive queue of the
sockets. You might get it to work if the receive are was very large (ie much
larger than tcp_rmem), but at that point you might as well as be able
to access all of system memory.

Using a DMA I/O engine would help, but you probably won't have one available.

Re: Socket Buffers and Memory Managment

[vinay ravuri]

How about the following approach:

I allocate an skb of 0 bytes and replace data element
of skb struct (i.e. skb.data = addr_given_by_hw) when
the h/w interrupts me with a packet.  I register for a
destructor for this skb and when the kernel is ready
to free the skb, I make sure that my free is invoked -
Ofcourse this is assuming that their is a facility in
linux socket buffers to be able to do destructors.  Is
this approach a viable, if so, are any gottcha's?

-Vinay


--- Stephen Hemminger
<shemminger@linux-foundation.org> wrote:

> On Tue, 17 Jul 2007 10:20:58 -0700 (PDT)
> vinay ravuri <vinaynyc@yahoo.com> wrote:
> 
> > Hi,
> > 
> > I am fairly new to linux socket buffers and have
> the
> > following questions!
> > 
> > I am working with a custom ethernet MAC that does
> not
> > allow me to specify a particular memory location
> for
> > the h/w to DMA the packet into (Rx side). 
> Instead, it
> > has a pool of fixed size buffers with some h/w
> > specific headers around each buffer that are
> managed
> > by h/w and will pick a free buffer and DMA the
> packet.
> 
> Sounds like sucky hardware...
> You need to copy to a newly allocated skb, see
> 8139too.c
> 
> >  It appears dev_alloc_skb() actually allocates the
> > physical memory and doesn't allow the user to
> specify
> > the skb.data to something specific to what I want
> > which is a problem for me.  First is my assumption
> > correct that I am cannot pick an arbitrary
> skb.data
> > location in struct sk_buff?  I want to avoid
> copying
> > the dma'ed data into a new socket buffer as it is
> > expense.  Is there any ways around this problem?
> 
> You could play tricks with skb frags but it would be
> fragile
> and not worth the trouble. The problem is that the
> receive
> skb can stay in the system for a really long time
> (until the application
> reads the data) so your fixed size buffer pool in
> hardware
> would get exhausted.
>  
> > Also, if the h/w gives me a single packet in
> multiple
> > locations (i.e. non-contiguous chunks of memory),
> can
> > socket buffers handle chains of buffers?  I am
> looking
> > for a facility like mbuf's in netbsd where one can
> > chain multiple buffers together to make construct
> a
> > single packet.
> 
> Yes, skb frag list could be used for that but you
> don't
> want to do that. See above. Copy the data into an
> new
> skb and reserve any necessary bytes so IP header is
> aligned.  I.e. if using ethernet header (14 bytes),
> do
> skb_reserve(skb, 2) before copying the data.
> 



       
____________________________________________________________________________________
Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase.
http://farechase.yahoo.com/

Re: Socket Buffers and Memory Managment

[pradeep singh]

On 7/19/07, vinay ravuri <vinaynyc@yahoo.com> wrote:
> How about the following approach:
>
> I allocate an skb of 0 bytes and replace data element
> of skb struct (i.e. skb.data = addr_given_by_hw) when
> the h/w interrupts me with a packet.  I register for a
> destructor for this skb and when the kernel is ready
> to free the skb, I make sure that my free is invoked -
> Ofcourse this is assuming that their is a facility in
> linux socket buffers to be able to do destructors.  Is
> this approach a viable, if so, are any gottcha's?

I wonder if passing a zero size will work correctly with
alloc_skb/__alloc_skb as you still have just skb frag list to work
with in this case.
And as Stephen pointed out, if i understand correctly that would be
still a problem.

yes/no?

thanks
--pradeep
>
> -Vinay
>
>
> --- Stephen Hemminger
> <shemminger@linux-foundation.org> wrote:
>
> > On Tue, 17 Jul 2007 10:20:58 -0700 (PDT)
> > vinay ravuri <vinaynyc@yahoo.com> wrote:
> >
> > > Hi,
> > >
> > > I am fairly new to linux socket buffers and have
> > the
> > > following questions!
> > >
> > > I am working with a custom ethernet MAC that does
> > not
> > > allow me to specify a particular memory location
> > for
> > > the h/w to DMA the packet into (Rx side).
> > Instead, it
> > > has a pool of fixed size buffers with some h/w
> > > specific headers around each buffer that are
> > managed
> > > by h/w and will pick a free buffer and DMA the
> > packet.
> >
> > Sounds like sucky hardware...
> > You need to copy to a newly allocated skb, see
> > 8139too.c
> >
> > >  It appears dev_alloc_skb() actually allocates the
> > > physical memory and doesn't allow the user to
> > specify
> > > the skb.data to something specific to what I want
> > > which is a problem for me.  First is my assumption
> > > correct that I am cannot pick an arbitrary
> > skb.data
> > > location in struct sk_buff?  I want to avoid
> > copying
> > > the dma'ed data into a new socket buffer as it is
> > > expense.  Is there any ways around this problem?
> >
> > You could play tricks with skb frags but it would be
> > fragile
> > and not worth the trouble. The problem is that the
> > receive
> > skb can stay in the system for a really long time
> > (until the application
> > reads the data) so your fixed size buffer pool in
> > hardware
> > would get exhausted.
> >
> > > Also, if the h/w gives me a single packet in
> > multiple
> > > locations (i.e. non-contiguous chunks of memory),
> > can
> > > socket buffers handle chains of buffers?  I am
> > looking
> > > for a facility like mbuf's in netbsd where one can
> > > chain multiple buffers together to make construct
> > a
> > > single packet.
> >
> > Yes, skb frag list could be used for that but you
> > don't
> > want to do that. See above. Copy the data into an
> > new
> > skb and reserve any necessary bytes so IP header is
> > aligned.  I.e. if using ethernet header (14 bytes),
> > do
> > skb_reserve(skb, 2) before copying the data.
> >
>
>
>
>
> ____________________________________________________________________________________
> Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase.
> http://farechase.yahoo.com/
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Pradeep

Re: Socket Buffers and Memory Managment

[Stephen Hemminger]

On Wed, 18 Jul 2007 23:51:03 -0700 (PDT)
vinay ravuri <vinaynyc@yahoo.com> wrote:

> How about the following approach:
> 
> I allocate an skb of 0 bytes and replace data element
> of skb struct (i.e. skb.data = addr_given_by_hw) when
> the h/w interrupts me with a packet.  I register for a
> destructor for this skb and when the kernel is ready
> to free the skb, I make sure that my free is invoked -
> Ofcourse this is assuming that their is a facility in
> linux socket buffers to be able to do destructors.  Is
> this approach a viable, if so, are any gottcha's?
> 
> -Vinay

You need to use frag list for that since upper layers expect to
be able to use that data area for normal use, ie bridging/routing, etc.
Also access to data area would be non-cached so you want to make
sure it is only accessed once.

But how will you handle a slow receiver where all the skb's end up
staying queued. Won't you exhaust your packet memory.

Re: Socket Buffers and Memory Managment

[Evgeniy Polyakov]

Hi.

On Wed, Jul 18, 2007 at 11:51:03PM -0700, vinay ravuri (vinaynyc@yahoo.com) wrote:
> How about the following approach:
> 
> I allocate an skb of 0 bytes and replace data element
> of skb struct (i.e. skb.data = addr_given_by_hw) when
> the h/w interrupts me with a packet.  I register for a
> destructor for this skb and when the kernel is ready
> to free the skb, I make sure that my free is invoked -
> Ofcourse this is assuming that their is a facility in
> linux socket buffers to be able to do destructors.  Is
> this approach a viable, if so, are any gottcha's?

It will not work, since kfree_skb() eventually tries to free skb->head
into kmem cache, so you will need to hack kfree_skb() not to try to
release that data at all. Likely the best zero-copy approach in your
case is to use frag_list, but be ready that your netwrok will regularily
stall - when hardware buffers are all in use and not yet freed, you will
not be able to send/receive new packets, although amount of memory will
allow that.

> -Vinay

-- 
	Evgeniy Polyakov

Re: Socket Buffers and Memory Managment

[Andi Kleen]

Evgeniy Polyakov <johnpol@2ka.mipt.ru> writes:

> Hi.
> 
> On Wed, Jul 18, 2007 at 11:51:03PM -0700, vinay ravuri (vinaynyc@yahoo.com) wrote:
> > How about the following approach:
> > 
> > I allocate an skb of 0 bytes and replace data element
> > of skb struct (i.e. skb.data = addr_given_by_hw) when
> > the h/w interrupts me with a packet.  I register for a
> > destructor for this skb and when the kernel is ready
> > to free the skb, I make sure that my free is invoked -
> > Ofcourse this is assuming that their is a facility in
> > linux socket buffers to be able to do destructors.  Is
> > this approach a viable, if so, are any gottcha's?
> 
> It will not work, since kfree_skb() eventually tries to free skb->head
> into kmem cache,

And in addition if the skbuff is ever passed towards the socket
layer the destructor will be overwritten

-Andi