pre-fetching skb for delayed send

pre-fetching skb for delayed send

[Ben Greear]

Suppose one has a bridge-like thing that may store up packets
for a bit (100+ms) before sending them....

We notice that performance is very good if we can run with (near) zero
delay, but at higher delay, performance goes down significantly when attempting 10G
speeds.  I assume this is because we are sending skbs that are no longer in the cache...

So, is there an easy way to pre-fetch an skb and it's related pages?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: pre-fetching skb for delayed send

[Benjamin LaHaise]

On Fri, Jul 06, 2012 at 04:10:40PM -0700, Ben Greear wrote:
> So, is there an easy way to pre-fetch an skb and it's related pages?

One trick would be to store the pointers to the skbs in an array rather than 
relying on a list of skbs.  That would enable you to issue prefetches for 
multiple skbs from a single cache line of the array, rather than relying 
upon the slow chasing of pointers guaranteed to cache miss in a list.

		-ben
-- 
"Thought is the essence of where you are now."

Re: pre-fetching skb for delayed send

[Ben Greear]

On 07/06/2012 04:52 PM, Benjamin LaHaise wrote:
> On Fri, Jul 06, 2012 at 04:10:40PM -0700, Ben Greear wrote:
>> So, is there an easy way to pre-fetch an skb and it's related pages?
>
> One trick would be to store the pointers to the skbs in an array rather than
> relying on a list of skbs.  That would enable you to issue prefetches for
> multiple skbs from a single cache line of the array, rather than relying
> upon the slow chasing of pointers guaranteed to cache miss in a list.

Well, to start with..I at least know the next skb to transmit,
so I figured I'd prefetch it before starting tx of the current
skb.

My question is more basic though:  Given an skb, how do you prefetch
it...do you just prefetch the skb pointer, or do you need to dig into
the guts of the skb?

Thanks,
Ben

>
> 		-ben
>


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: pre-fetching skb for delayed send

[Benjamin LaHaise]

On Fri, Jul 06, 2012 at 09:15:29PM -0700, Ben Greear wrote:
> Well, to start with..I at least know the next skb to transmit,
> so I figured I'd prefetch it before starting tx of the current
> skb.

Prefetching data you're just about to immediately access doesn't actually 
help improve performance -- it's better to just access the data.  Prefetching 
subsequent skbs should be of more benefit.

> My question is more basic though:  Given an skb, how do you prefetch
> it...do you just prefetch the skb pointer, or do you need to dig into
> the guts of the skb?

See prefetch.h for details.  Just pass the pointer to the cacheline you want 
to trigger prefetch on to prefetch() or prefetchw(), or use prefetch_range() 
(probably useful for skbs given that they're larger than one cacheline).  
For an skb, you may have to prefetch the frag list as well.

		-ben
-- 
"Thought is the essence of where you are now."