Re: 3.4.1 and 3.5-rc1 Packet lost at 250Mb/s

[Eric Dumazet <eric.dumazet@gmail.com>]

On Sun, 2012-10-07 at 21:18 +0200, Nieścierowicz Adam wrote:
> W dniu 06.07.2012 12:13, Eric Dumazet napisał(a):
> 
> > On Fri, 2012-07-06 at 11:47 +0200, Nieścierowicz Adam wrote:
> >
> >> Hello, Can I send something that will help determine the cause of 
> >> the
> >> problem? W dniu 08.06.2012 11:41, Eric Dumazet napisał(a):
> >>
> >>> On Fri, 2012-06-08 at 10:58 +0200, Nieścierowicz Adam wrote:
> >>>
> >>>> Hello, recently we changed on the router kernel from 2.6.38.1 to
> >>>> 3.4.1 and noticed 30% packet loss when traffic increases up to
> >>>> 250MB / s. Similar is for kernel 3.5-rc1 Here a link to ifstat
> >>>> http://wklej.org/id/767577/ [1] [2]
> >>> You should give as much as possible delails on your setup 
> >>> (hardware,
> >>> software) lspci cat /proc/cpuinfo cat /proc/interrupts ifconfig -a 
> >>> tc
> >>> -s -d qdisc dmesg netstat -s
> >> currently running on 2.6.38.1 and traffic is 100Mb / s lspci:
> >> http://wklej.org/id/769102/ [2] /proc/cpuinfo:
> >> http://wklej.org/id/769104/ [3] /proc/interrupts:
> >> http://wklej.org/id/769106/ [4] ifconfig -a:
> >> http://wklej.org/id/769108/ [5] tc -s -d qdisc:
> >> http://wklej.org/id/769109/ [6] dmesg: here are some logs from 
> >> iptables
> >> netstat -s: http://wklej.org/id/769110/ [7] lsmod:
> >> http://wklej.org/id/769117/ [8] /proc/net/softnet_stat:
> >> http://wklej.org/id/769116/ [9]
> >
> > Same infos of 3.5-rcX kernel would be nice.
> >
> > What NIC is eth0 ? (dmesg please)
> >
> > It seems all network traffic on 2.6.38 is handled by a single cpu 
> > (cpu0)
> >
> > (seen in /proc/interrupts)
> >
> > I suspect that with 3.4 or 3.5 kernels, traffic is handled by many 
> > cpus
> > and they hit false sharing and contention.
> >
> > You probably get better performance doing some affinity tuning :
> >
> > For example,
> > eth0 serviced by cpu0
> > eth2 serviced by cpu1
> > eth3 serviced by cpu2
> > eth5 serviced by cpu3
> >
> > and so on...
> >
> > check and/or set /proc/irq/${NUM}/smp_affinity
> 
> hello
> I would go back to an earlier thread.
> 
> Currently is installed kernel 3.6.0 and symptoms are the same
> 
> about configuration:
> 
> - affinity on
> 
> - lspci: http://wklej.org/id/843156/ [10]
> 
> - /proc/cpuinfo: http://wklej.org/id/843158/ [11]
> 
> - /proc/interrupts: http://wklej.org/id/843161/ [12]
> 
> - ifconfig -a: http://wklej.org/id/843162/ [13]
> 
> - tc -s -d qdisc: http://wklej.org/id/843164/ [14]
> 
> - dmesg: http://wklej.org/id/843166/ [15]
> 
> - lsmod: http://wklej.org/id/843167/ [16]
> 
> - /proc/net/softnet_stat: /proc/net/softnet_stat
> 
> attach something else?
> 
> Thanks

You should use RPS on eth2/eth3 because they are non multi queue.

Documentation/networking/scaling.txt should give you all the needed info