Re: Network performance degradation from 2.6.11.12 to 2.6.16.20

[Andi Kleen <ak@suse.de>]

"Vladimir B. Savkin" <master@sectorb.msk.ru> writes:

> On Mon, Sep 18, 2006 at 10:35:38AM +0200, Andi Kleen wrote:
> > > I just found out that TSC clocksource is not implemented on x86-64.
> > > Kernel version 2.6.18-rc7, is it true?
> > 
> > The x86-64 timer subsystems currently doesn't have clocksources
> > at all, but it supports TSC and some other timers.
> 

> until I hacked arch/i386/kernel/tsc.c

Then you don't use x86-64. 


> 
> > > I've also had experience of unsychronized TSC on dual-core Athlon,
> > > but it was cured by idle=poll.
> > 
> > You can use that, but it will make your system run quite hot 
> > and cost you a lot of powe^wmoney.
> 
> Here in Russia electric power is cheap compared with hardware upgrade.

It's not just electrical power - the hardware is more stressed and will
likely fail earlier too.  As a rule of thumb the hotter your hardware runs
the earlier it will fail.

> 
> > > It seems that dhcpd3 makes the box timestamping incoming packets,
> > > killing the performance. I think that combining router and DHCP server
> > > on a same box is a legitimate situation, isn't it?
> > 
> > Yes.  Good point. DHCP is broken and needs to be fixed. Can you
> > send a bug report to the DHCP maintainers? 
> > 
> > iirc the problem used to be that RAW sockets didn't do something
> > they need them to do. Maybe we can fix that now.
> 
> Will try some days later.
> 
> Oh, and pppoe-server uses some kind of packet socket too, doesn't it?

The problem is not really using a packet socket, but using the SIOCGSTAMP
ioctl on it. As soon as someone issues it the system will take accurate 
time stamps for each incoming packet until the respective socket is closed.

Quick fix is to change user space to use gettimeofday() when it reads
the packet instead.

For netdev: I'm more and more thinking we should just avoid the problem
completely and switch to "true end2end" timestamps. This means don't
time stamp when a packet is received, but only when it is delivered
to a socket. The timestamp at receiving is a lie anyways because
the network hardware can add an arbitary long delay before the driver interrupt
handler runs. Then the problem above would completely disappear. 
Comments? Opinions? 

-Andi