Non-linear SKBs

Non-linear SKBs

[Kristian Evensen]

Hello,

I have developed a small patch for the TCP code in 2.6.19 and it works 
flawlessly. A couple of days ago I decided to make it compatible with 
2.6.22.5 and have stumbled upon a problem I cannot solve.

In 2.6.19 it seems that all packets (at least the ones my patch work 
with) are linear, while they are non-linear in 2.6.22.5. I have searched 
through the code (focusing on tcp_sendmsg) to try to figure out what 
happens, but can't find any differences that would explain this. Does 
anyone know what might be the cause and if there is an easy way to 
return to linear skbs (unless that is totally stupid)? I would also like 
the benefits offered by the collapsing when retransmitting (which 
requires number of fragments to be 0).

Currently I us e the skb_linearize(skb) on all fragmentet packets, which 
is not nice :) Both kernels are vanilla kernels with the patch applied, 
and I used OpenSuse with 2.6.19 and Ubuntu with 2.6.22.5 (but I guess 
that shouldn't matter in this case).

The reason this is a problem is that I copy data between SKBs, and in 
2.6.19 I have used memcpy for this purpose. I have looked at the way the 
kernel copies data into a non-linear skb in the else-part of the large 
if-test in tcp_sendmsg and the skb_copy_bits-function, but I have to 
admit that I think the first one is a bit advanced and I don't quite 
understand how to use the last one. Does anyone have any easier to 
understand examples or explanations on how to copy data from one 
non-linear skb to another?

Thanks.

-Kristian

Re: Non-linear SKBs

[David Miller]

From: Kristian Evensen <kristrev@ifi.uio.no>
Date: Fri, 12 Oct 2007 00:54:37 +0200

> I have developed a small patch for the TCP code in 2.6.19 and it works 
> flawlessly. A couple of days ago I decided to make it compatible with 
> 2.6.22.5 and have stumbled upon a problem I cannot solve.
> 
> In 2.6.19 it seems that all packets (at least the ones my patch work 
> with) are linear, while they are non-linear in 2.6.22.5. I have searched 
> through the code (focusing on tcp_sendmsg) to try to figure out what 
> happens, but can't find any differences that would explain this. Does 
> anyone know what might be the cause and if there is an easy way to 
> return to linear skbs (unless that is totally stupid)? I would also like 
> the benefits offered by the collapsing when retransmitting (which 
> requires number of fragments to be 0).

If the underlying device can do scatter-gather and checksumming,
the TCP code builds outgoing packets by copying user date into
full system pages, and then attaching those pages into the SKB.
The protocol headers sit under the skb->data linear area, and
the user data mostly sits in the user pages under
skb_shinfo(skb)->frags[]

This increases the density of data packed into the memory allocated
compared to using skb->data for it.  It also enormously simplifies
the logic necessary to support TSO.

Re: Non-linear SKBs

[kristrev]

> If the underlying device can do scatter-gather and checksumming,
> the TCP code builds outgoing packets by copying user date into
> full system pages, and then attaching those pages into the SKB.
> The protocol headers sit under the skb->data linear area, and
> the user data mostly sits in the user pages under
> skb_shinfo(skb)->frags[]
>
> This increases the density of data packed into the memory allocated
> compared to using skb->data for it.  It also enormously simplifies
> the logic necessary to support TSO.

Thank you very much, I think I am starting to get it now and coming to
think of it this will make my patch much more elegant. I have spent the
day reading more code, and am wondering if something along the likes of
this piece of code will do what I want ("copy" the data from the next skb
in the retransmission queue into this skb):

//Do preliminary checks to see if the "new" packet will be within mss,
that this_skb->nr_frags + next_skb->nr_frags < MAX_SKB_FRAGS and so on

int i;
int this_frags = this_skb->nr_frags;

for(i=0; i<next_skb->nr_frags; i++)
        //Does the "copy"
        this_skb->frags[this_frags + i] = next_skb->frags[i];

this_skb->data_len += next_skb->data_len;
this_skb->truesize += next_skb->data_len;
this_skb->nr_frags += next_skb->nr_frags;

//Update TSO?

By the way, am I correct in my assumption that one SKB's frags is stored
linearly in the frags-array? Or have I made a horrible misunderstanding?
:)

One of the things that I have yet to understand is the frag_list in the
skb_shared_info-struct. Does this contain all skb's that "use" this frag
and works as a sort of referance counter (frag won't be removed until the
variable is NULL and I have to append this_skb to the list), or is it
something else?

Thanks again for all help.

-Kristian

Re: Non-linear SKBs

[Kristian Evensen]

Reading through my mail again, I see that I was a bit unclear. What I 
want to achieve is to share a frag between to skbs (where one has no 
earlier referance to it). Sorry.

kristrev@student.matnat.uio.no wrote:
>> If the underlying device can do scatter-gather and checksumming,
>> the TCP code builds outgoing packets by copying user date into
>> full system pages, and then attaching those pages into the SKB.
>> The protocol headers sit under the skb->data linear area, and
>> the user data mostly sits in the user pages under
>> skb_shinfo(skb)->frags[]
>>
>> This increases the density of data packed into the memory allocated
>> compared to using skb->data for it.  It also enormously simplifies
>> the logic necessary to support TSO.
>>     
>
> Thank you very much, I think I am starting to get it now and coming to
> think of it this will make my patch much more elegant. I have spent the
> day reading more code, and am wondering if something along the likes of
> this piece of code will do what I want ("copy" the data from the next skb
> in the retransmission queue into this skb):
>
> //Do preliminary checks to see if the "new" packet will be within mss,
> that this_skb->nr_frags + next_skb->nr_frags < MAX_SKB_FRAGS and so on
>
> int i;
> int this_frags = this_skb->nr_frags;
>
> for(i=0; i<next_skb->nr_frags; i++)
>         //Does the "copy"
>         this_skb->frags[this_frags + i] = next_skb->frags[i];
>
> this_skb->data_len += next_skb->data_len;
> this_skb->truesize += next_skb->data_len;
> this_skb->nr_frags += next_skb->nr_frags;
>
> //Update TSO?
>
> By the way, am I correct in my assumption that one SKB's frags is stored
> linearly in the frags-array? Or have I made a horrible misunderstanding?
> :)
>
> One of the things that I have yet to understand is the frag_list in the
> skb_shared_info-struct. Does this contain all skb's that "use" this frag
> and works as a sort of referance counter (frag won't be removed until the
> variable is NULL and I have to append this_skb to the list), or is it
> something else?
>
> Thanks again for all help.
>
> -Kristian
>
>