Re: [PATCH net-next-2.6] net: Increase NET_SKB_PAD to 64 bytes

[Eric Dumazet <eric.dumazet@gmail.com>]

Le vendredi 07 mai 2010 à 12:09 +0200, Michal Simek a écrit :
> Eric Dumazet wrote:
> > Le vendredi 07 mai 2010 à 11:48 +0200, Michal Simek a écrit :
> >> David Miller wrote:
> >>> From: Michal Simek <monstr@monstr.eu>
> >>> Date: Fri, 07 May 2010 09:53:48 +0200
> >>>
> >>>> I will add this Microblaze patch to my repo for testing and anyway
> >>>> should go through my repo.
> >>> It's already in the net-next-2.6 tree.
> >> Anyway.
> >>
> >> I am ok with removing NET_IP_ALIGN because it is already defined in 
> >> skbuff.h to 2.
> >> But increasing NET_SKB_PAD to 64 caused that Microblaze extends skb 
> >> buffers for some bytes.
> >> I measured it by iperf and netperf and I see regression around 1-2Mbit/s 
> >> that's why I would like to ask you to revert this patch or keep at least 
> >> NET_SKB_PAD part.
> > 
> > Interesting.
> > 
> > Increasing NET_SKB_PAD to say 128 or 256 should not have performance
> > impact, but reserve a bit more ram. (truesize...)
> 
> yes. Microblaze is very sensitive on it. I have spent a couple of days 
> to find out why bandwidth is so bad and I found that truesize is the 
> problem. Whole driver allocated skb buffer to max mtu size (9000).
> 
> But if mtu is 1500 the driver still worked with max skb buffers size
> Please correct me if I am wrong but if truesize is setup to 9000 then 
> net code is working with that size even if mtu is 1500.
> The next thing is that cache operations take a lot of cpu cycles.

Allocating space <before> data doesnt mean we touch it. In fact we dont,
this is why I said increasing SKB_PAD from 32 to 64 should not make a
difference on perfs.

But yes, udp/tcp pacing is sensitive to truesize, since truesize is used
to limit socket receive buffer (bytes limit, not packets)

If you hit socket limit (or global limit), tcp will try to coalesce
buffers to reduce the 'truesize'. This involves very expensive
operations. UDP dont have such slow path, but might drop frames one
socket limit is reached.

So David patch (or mine if you prefer) only points an underlying problem
(truesize changed from 9056 to 9088, a very small increase...)

Using 9000 bytes buffers is very risky in the long term (if PAGE_SIZE is
4096), because after a while, your driver wont be able to allocate new
skbs.

> 
> I create a patch which allocated aligned skb buffers where size depends 
> on current mtu size and this change rapidly increase bandwidth for 
> mtu=1500 from 8Mb/s to 35Mb/s (depends on setting). There is no 
> difference if jumbo frames are used (I means for driver with/without my 
> patch).
> 
> > 
> > Investigation is needed. Maybe your NIC now allocates high order pages ?
> > 
> > What driver are you using ?
> > 
> 
> I am using non-mainline ll_temac driver.
> drivers/net/xilinx_lltemac/
> 
> http://developer.petalogix.com/git/gitweb.cgi?p=linux-2.6-microblaze.git;a=shortlog
> 
> Regards,
> Michal
> 
> 
> 
>