* 10GBE performance drop with net.ipv4.tcp_timestamps=0
@ 2012-06-19 21:08 Stefan Priebe
2012-06-19 21:31 ` Eric Dumazet
0 siblings, 1 reply; 17+ messages in thread
From: Stefan Priebe @ 2012-06-19 21:08 UTC (permalink / raw)
To: Linux Netdev List
Hello List,
i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe
other one whith RHEL 6 (2.6.32 kernel).
I noticed that setting
net.ipv4.tcp_timestamps=0
descreased the performance from 9,7 Full Duplex to 3-4Gb/s.
Is this bahviour fine? What should / could i tet?
Greets
Stefan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-19 21:08 10GBE performance drop with net.ipv4.tcp_timestamps=0 Stefan Priebe
@ 2012-06-19 21:31 ` Eric Dumazet
2012-06-20 7:00 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2012-06-19 21:31 UTC (permalink / raw)
To: Stefan Priebe; +Cc: Linux Netdev List
On Tue, 2012-06-19 at 23:08 +0200, Stefan Priebe wrote:
> Hello List,
>
> i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe
> other one whith RHEL 6 (2.6.32 kernel).
>
> I noticed that setting
> net.ipv4.tcp_timestamps=0
>
> descreased the performance from 9,7 Full Duplex to 3-4Gb/s.
>
> Is this bahviour fine? What should / could i tet?
>
Really, you should provide more input than that, if you really want us
to help.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-19 21:31 ` Eric Dumazet
@ 2012-06-20 7:00 ` Stefan Priebe - Profihost AG
2012-06-20 7:22 ` Eric Dumazet
0 siblings, 1 reply; 17+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-20 7:00 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Netdev List
Am 19.06.2012 23:31, schrieb Eric Dumazet:
> On Tue, 2012-06-19 at 23:08 +0200, Stefan Priebe wrote:
>> i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe
>> other one whith RHEL 6 (2.6.32 kernel).
>>
>> I noticed that setting
>> net.ipv4.tcp_timestamps=0
>>
>> descreased the performance from 9,7 Full Duplex to 3-4Gb/s.
>>
>> Is this bahviour fine? What should / could i tet?
>>
>
> Really, you should provide more input than that, if you really want us
> to help.
*arg* forgot to add the pastebin links. Sorry. Speed degraded in this
case from 9,88Gbit/s to 2,45Gbit/s. When i turn on timestamps it's
perfect again. Server A has 3.5.0-rc3 and server B has an RHEL6 2.6.32
kernel.
Before:
http://pastebin.com/raw.php?i=1gVraWVc
After:
http://pastebin.com/raw.php?i=NSh8Y29s
Thanks,
Stefan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 7:00 ` Stefan Priebe - Profihost AG
@ 2012-06-20 7:22 ` Eric Dumazet
2012-06-20 8:21 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2012-06-20 7:22 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Linux Netdev List
On Wed, 2012-06-20 at 09:00 +0200, Stefan Priebe - Profihost AG wrote:
> Am 19.06.2012 23:31, schrieb Eric Dumazet:
> > On Tue, 2012-06-19 at 23:08 +0200, Stefan Priebe wrote:
> >> i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe
> >> other one whith RHEL 6 (2.6.32 kernel).
> >>
> >> I noticed that setting
> >> net.ipv4.tcp_timestamps=0
> >>
> >> descreased the performance from 9,7 Full Duplex to 3-4Gb/s.
> >>
> >> Is this bahviour fine? What should / could i tet?
> >>
> >
> > Really, you should provide more input than that, if you really want us
> > to help.
>
> *arg* forgot to add the pastebin links. Sorry. Speed degraded in this
> case from 9,88Gbit/s to 2,45Gbit/s. When i turn on timestamps it's
> perfect again. Server A has 3.5.0-rc3 and server B has an RHEL6 2.6.32
> kernel.
>
> Before:
> http://pastebin.com/raw.php?i=1gVraWVc
>
> After:
> http://pastebin.com/raw.php?i=NSh8Y29s
You have a lot of packet losses
add "tc -s -d qdisc" , "ifconfig -a " and "ethtool -S ethX" outputs for
both servers
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 7:22 ` Eric Dumazet
@ 2012-06-20 8:21 ` Stefan Priebe - Profihost AG
2012-06-20 8:41 ` Eric Dumazet
0 siblings, 1 reply; 17+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-20 8:21 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Netdev List
Am 20.06.2012 09:22, schrieb Eric Dumazet:
> On Wed, 2012-06-20 at 09:00 +0200, Stefan Priebe - Profihost AG wrote:
>> Am 19.06.2012 23:31, schrieb Eric Dumazet:
>> Before:
>> http://pastebin.com/raw.php?i=1gVraWVc
>>
>> After:
>> http://pastebin.com/raw.php?i=NSh8Y29s
>
> You have a lot of packet losses
But this ONLY happens with tcp_timestamps=0.
> add "tc -s -d qdisc" , "ifconfig -a " and "ethtool -S ethX" outputs for
> both servers
eth2 is the 10Gb device on both systems
server a has kernel 3.5 server b has rhel 6 kernel
Server A:
# tc -s -d qdisc
RTNETLINK answers: Operation not supported
Dump terminated
Server B (eth2 is the 10GB/s device):
# tc -s -d qdisc
qdisc mq 0: dev eth0 root
Sent 55151 bytes 555 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc mq 0: dev eth2 root
Sent 38374148475 bytes 2774405 pkt (dropped 0, overlimits 0 requeues 4)
rate 0bit 0pps backlog 0b 0p requeues 4
ifconfig -a:
http://pastebin.com/raw.php?i=M3QHQjSU
ethtool -S:
http://pastebin.com/raw.php?i=Eap05xKc
Thanks again,
Stefan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 8:21 ` Stefan Priebe - Profihost AG
@ 2012-06-20 8:41 ` Eric Dumazet
2012-06-20 9:06 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2012-06-20 8:41 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Linux Netdev List
On Wed, 2012-06-20 at 10:21 +0200, Stefan Priebe - Profihost AG wrote:
> Am 20.06.2012 09:22, schrieb Eric Dumazet:
> > On Wed, 2012-06-20 at 09:00 +0200, Stefan Priebe - Profihost AG wrote:
> >> Am 19.06.2012 23:31, schrieb Eric Dumazet:
> >> Before:
> >> http://pastebin.com/raw.php?i=1gVraWVc
> >>
> >> After:
> >> http://pastebin.com/raw.php?i=NSh8Y29s
> >
> > You have a lot of packet losses
> But this ONLY happens with tcp_timestamps=0.
>
Yes, you already told that in subject line.
single tcp flow ?
You seem to have a switch or something that drops packets in this case.
You could try to rate limit to 9Gb/s and see if it is better.
Here, I roughly have same bandwidth with tcp_timestamps on or off, with
ixgbe cards and net-next kernels.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 8:41 ` Eric Dumazet
@ 2012-06-20 9:06 ` Stefan Priebe - Profihost AG
2012-06-20 9:12 ` Stefan Priebe - Profihost AG
2012-06-20 9:16 ` Eric Dumazet
0 siblings, 2 replies; 17+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-20 9:06 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Netdev List
Am 20.06.2012 10:41, schrieb Eric Dumazet:
> On Wed, 2012-06-20 at 10:21 +0200, Stefan Priebe - Profihost AG wrote:
>> Am 20.06.2012 09:22, schrieb Eric Dumazet:
> Yes, you already told that in subject line.
>
> single tcp flow ?
I use iperf - i think it uses just a single tcp flow. But i'm not sure.
> You seem to have a switch or something that drops packets in this case.
> You could try to rate limit to 9Gb/s and see if it is better.
Sadly i can't rate limit to 9Gbit/s on the switch.
> Here, I roughly have same bandwidth with tcp_timestamps on or off, with
> ixgbe cards and net-next kernels.
Mhm strange. Do you have any vague idea what could cause this? Any wrong
reordering of the packets without tcp_timestamps?
Stefan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 9:06 ` Stefan Priebe - Profihost AG
@ 2012-06-20 9:12 ` Stefan Priebe - Profihost AG
2012-06-20 9:17 ` Eric Dumazet
2012-06-20 9:16 ` Eric Dumazet
1 sibling, 1 reply; 17+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-20 9:12 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Netdev List
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
>> You seem to have a switch or something that drops packets in this case.
>> You could try to rate limit to 9Gb/s and see if it is better.
> Sadly i can't rate limit to 9Gbit/s on the switch.
>
>> Here, I roughly have same bandwidth with tcp_timestamps on or off, with
>> ixgbe cards and net-next kernels.
> Mhm strange. Do you have any vague idea what could cause this? Any wrong
> reordering of the packets without tcp_timestamps?
I've now done another test without the switch. So both systems where
direct attached and still the same. Without tcp_timstamps speed drops to
2-4Gbit/s.
Stefan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 9:12 ` Stefan Priebe - Profihost AG
@ 2012-06-20 9:17 ` Eric Dumazet
2012-06-20 9:25 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2012-06-20 9:17 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Linux Netdev List
On Wed, 2012-06-20 at 11:12 +0200, Stefan Priebe - Profihost AG wrote:
> Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
> >> You seem to have a switch or something that drops packets in this case.
> >> You could try to rate limit to 9Gb/s and see if it is better.
> > Sadly i can't rate limit to 9Gbit/s on the switch.
> >
> >> Here, I roughly have same bandwidth with tcp_timestamps on or off, with
> >> ixgbe cards and net-next kernels.
> > Mhm strange. Do you have any vague idea what could cause this? Any wrong
> > reordering of the packets without tcp_timestamps?
>
> I've now done another test without the switch. So both systems where
> direct attached and still the same. Without tcp_timstamps speed drops to
> 2-4Gbit/s.
>
> Stefan
If you exchange sender/receiver role between linux kernel versions, is
it the same problem ?
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 9:17 ` Eric Dumazet
@ 2012-06-20 9:25 ` Stefan Priebe - Profihost AG
2012-06-20 9:28 ` Eric Dumazet
0 siblings, 1 reply; 17+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-20 9:25 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Netdev List
Am 20.06.2012 11:17, schrieb Eric Dumazet:
> On Wed, 2012-06-20 at 11:12 +0200, Stefan Priebe - Profihost AG wrote:
>> Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
>>>> You seem to have a switch or something that drops packets in this case.
>>>> You could try to rate limit to 9Gb/s and see if it is better.
>>> Sadly i can't rate limit to 9Gbit/s on the switch.
> If you exchange sender/receiver role between linux kernel versions, is
> it the same problem ?
I'm testing in both directions. So both are sending and receiving.
I've now made tests with only one sending an the other receiving.
When server B is the sender i get 4Gbit/s. When server A is the sender i
get full 9,9Gbit/s.
Stefan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 9:25 ` Stefan Priebe - Profihost AG
@ 2012-06-20 9:28 ` Eric Dumazet
2012-06-20 9:33 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2012-06-20 9:28 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Linux Netdev List
On Wed, 2012-06-20 at 11:25 +0200, Stefan Priebe - Profihost AG wrote:
> Am 20.06.2012 11:17, schrieb Eric Dumazet:
> > On Wed, 2012-06-20 at 11:12 +0200, Stefan Priebe - Profihost AG wrote:
> >> Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
> >>>> You seem to have a switch or something that drops packets in this case.
> >>>> You could try to rate limit to 9Gb/s and see if it is better.
> >>> Sadly i can't rate limit to 9Gbit/s on the switch.
>
> > If you exchange sender/receiver role between linux kernel versions, is
> > it the same problem ?
> I'm testing in both directions. So both are sending and receiving.
>
> I've now made tests with only one sending an the other receiving.
>
> When server B is the sender i get 4Gbit/s. When server A is the sender i
> get full 9,9Gbit/s.
>
> Stefan
To rule out bad hardware, could you try a 3.5-rc3 kernel on B ?
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 9:28 ` Eric Dumazet
@ 2012-06-20 9:33 ` Stefan Priebe - Profihost AG
2012-06-20 9:47 ` Eric Dumazet
0 siblings, 1 reply; 17+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-20 9:33 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Netdev List
Am 20.06.2012 11:28, schrieb Eric Dumazet:
> On Wed, 2012-06-20 at 11:25 +0200, Stefan Priebe - Profihost AG wrote:
>> Am 20.06.2012 11:17, schrieb Eric Dumazet:
>>> On Wed, 2012-06-20 at 11:12 +0200, Stefan Priebe - Profihost AG wrote:
>>>> Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
>>>>>> You seem to have a switch or something that drops packets in this case.
>>>>>> You could try to rate limit to 9Gb/s and see if it is better.
>>>>> Sadly i can't rate limit to 9Gbit/s on the switch.
>>
>>> If you exchange sender/receiver role between linux kernel versions, is
>>> it the same problem ?
>> I'm testing in both directions. So both are sending and receiving.
>>
>> I've now made tests with only one sending an the other receiving.
>>
>> When server B is the sender i get 4Gbit/s. When server A is the sender i
>> get full 9,9Gbit/s.
>>
> To rule out bad hardware, could you try a 3.5-rc3 kernel on B ?
Sure. In that case i get 4Gbit/s in both variants. I also tried two
other different machines same result.
Stefan
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 9:33 ` Stefan Priebe - Profihost AG
@ 2012-06-20 9:47 ` Eric Dumazet
2012-06-20 9:50 ` Stefan Priebe - Profihost AG
0 siblings, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2012-06-20 9:47 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Linux Netdev List
On Wed, 2012-06-20 at 11:33 +0200, Stefan Priebe - Profihost AG wrote:
> Sure. In that case i get 4Gbit/s in both variants. I also tried two
> other different machines same result.
>
So 3.5 on receiver is the problem, it seems ?
And you checked all the stuff about irq affinities, i presume, since a
lot of things might have changed between 2.6.32 and 3.5 ?
cat /proc/interrupts
what kind of NIC it is ?
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 9:47 ` Eric Dumazet
@ 2012-06-20 9:50 ` Stefan Priebe - Profihost AG
2012-06-20 10:06 ` Eric Dumazet
0 siblings, 1 reply; 17+ messages in thread
From: Stefan Priebe - Profihost AG @ 2012-06-20 9:50 UTC (permalink / raw)
To: Eric Dumazet; +Cc: Linux Netdev List
Am 20.06.2012 11:47, schrieb Eric Dumazet:
> On Wed, 2012-06-20 at 11:33 +0200, Stefan Priebe - Profihost AG wrote:
>
>> Sure. In that case i get 4Gbit/s in both variants. I also tried two
>> other different machines same result.
>>
>
> So 3.5 on receiver is the problem, it seems ?
Yes.
> And you checked all the stuff about irq affinities, i presume, since a
> lot of things might have changed between 2.6.32 and 3.5 ?
It is a single core E5 Xeon - i've set the affinity like this:
eth2 mask=1 for /proc/irq/83/smp_affinity
eth2 mask=2 for /proc/irq/84/smp_affinity
eth2 mask=4 for /proc/irq/85/smp_affinity
eth2 mask=8 for /proc/irq/86/smp_affinity
eth2 mask=10 for /proc/irq/87/smp_affinity
eth2 mask=20 for /proc/irq/88/smp_affinity
eth2 mask=40 for /proc/irq/89/smp_affinity
eth2 mask=80 for /proc/irq/90/smp_affinity
> cat /proc/interrupts
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
0: 141 0 0 0 0 0
0 0 IO-APIC-edge timer
1: 1 8 0 0 0 0
0 0 IO-APIC-edge i8042
9: 0 0 0 0 0 0
0 0 IO-APIC-fasteoi acpi
12: 0 3 0 0 0 0
0 0 IO-APIC-edge i8042
14: 0 0 0 0 0 0
0 0 IO-APIC-edge ide0
15: 0 0 0 0 0 0
0 0 IO-APIC-edge ide1
16: 0 0 26 0 0 0
0 0 IO-APIC-fasteoi ehci_hcd:usb1
23: 0 0 30 0 0 0
0 0 IO-APIC-fasteoi ehci_hcd:usb2
64: 0 0 0 81979 0 0
0 0 PCI-MSI-edge ahci
65: 0 0 0 1 0 0
0 0 PCI-MSI-edge eth0
66: 0 0 0 0 1090 0
0 0 PCI-MSI-edge eth0-TxRx-0
67: 0 0 0 0 411 0
0 0 PCI-MSI-edge eth0-TxRx-1
68: 0 0 0 0 592 0
0 0 PCI-MSI-edge eth0-TxRx-2
69: 0 0 0 0 472 0
0 0 PCI-MSI-edge eth0-TxRx-3
70: 0 0 0 0 0 1196
0 0 PCI-MSI-edge eth0-TxRx-4
71: 0 0 0 0 0 374
0 0 PCI-MSI-edge eth0-TxRx-5
72: 0 0 0 0 0 405
0 0 PCI-MSI-edge eth0-TxRx-6
73: 0 0 0 0 0 468
0 0 PCI-MSI-edge eth0-TxRx-7
83: 31278 0 0 65 0 0
0 0 PCI-MSI-edge eth2-TxRx-0
84: 0 36311 0 0 61 0
0 0 PCI-MSI-edge eth2-TxRx-1
85: 0 0 46189 0 61 0
0 0 PCI-MSI-edge eth2-TxRx-2
86: 0 0 0 28712 67 0
0 0 PCI-MSI-edge eth2-TxRx-3
87: 0 0 0 0 28089 0
0 0 PCI-MSI-edge eth2-TxRx-4
88: 0 0 0 0 0 34982
0 0 PCI-MSI-edge eth2-TxRx-5
89: 0 0 0 0 0 61
32420 0 PCI-MSI-edge eth2-TxRx-6
90: 0 0 0 0 0 61
0 25922 PCI-MSI-edge eth2-TxRx-7
91: 0 0 0 0 0 3
0 0 PCI-MSI-edge eth2
NMI: 13 12 15 22 5 5
5 5 Non-maskable interrupts
LOC: 58919 61420 65519 82647 35519 40489
27141 30228 Local timer interrupts
SPU: 0 0 0 0 0 0
0 0 Spurious interrupts
PMI: 13 12 15 22 5 5
5 5 Performance monitoring interrupts
IWI: 0 0 0 0 0 0
0 0 IRQ work interrupts
RTR: 6 0 0 0 0 0
0 0 APIC ICR read retries
RES: 15116 4521 2418 1814 2375 1615
1488 1367 Rescheduling interrupts
CAL: 134 148 100 162 170 172
172 172 Function call interrupts
TLB: 422 486 415 483 460 460
476 398 TLB shootdowns
TRM: 0 0 0 0 0 0
0 0 Thermal event interrupts
THR: 0 0 0 0 0 0
0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0
0 0 Machine check exceptions
MCP: 4 4 4 4 4 4
4 4 Machine check polls
> what kind of NIC it is ?
# lspci | grep 10-Giga
06:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit
SFI/SFP+ Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit
SFI/SFP+ Network Connection (rev 01)
Stefan
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 9:50 ` Stefan Priebe - Profihost AG
@ 2012-06-20 10:06 ` Eric Dumazet
2012-06-20 11:08 ` Eric Dumazet
0 siblings, 1 reply; 17+ messages in thread
From: Eric Dumazet @ 2012-06-20 10:06 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Linux Netdev List
On Wed, 2012-06-20 at 11:50 +0200, Stefan Priebe - Profihost AG wrote:
> Am 20.06.2012 11:47, schrieb Eric Dumazet:
> > On Wed, 2012-06-20 at 11:33 +0200, Stefan Priebe - Profihost AG wrote:
> >
> >> Sure. In that case i get 4Gbit/s in both variants. I also tried two
> >> other different machines same result.
> >>
> >
> > So 3.5 on receiver is the problem, it seems ?
> Yes.
>
> > And you checked all the stuff about irq affinities, i presume, since a
> > lot of things might have changed between 2.6.32 and 3.5 ?
>
> It is a single core E5 Xeon - i've set the affinity like this:
And you still have the retransmits in "netstat -s" output ?
Might be a firmware or pci issue, I have same cards but no problem here.
Check LRO is on ?
ethtool -k eth2
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 10:06 ` Eric Dumazet
@ 2012-06-20 11:08 ` Eric Dumazet
0 siblings, 0 replies; 17+ messages in thread
From: Eric Dumazet @ 2012-06-20 11:08 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Linux Netdev List
On Wed, 2012-06-20 at 12:06 +0200, Eric Dumazet wrote:
> On Wed, 2012-06-20 at 11:50 +0200, Stefan Priebe - Profihost AG wrote:
> > Am 20.06.2012 11:47, schrieb Eric Dumazet:
> > > On Wed, 2012-06-20 at 11:33 +0200, Stefan Priebe - Profihost AG wrote:
> > >
> > >> Sure. In that case i get 4Gbit/s in both variants. I also tried two
> > >> other different machines same result.
> > >>
> > >
> > > So 3.5 on receiver is the problem, it seems ?
> > Yes.
> >
> > > And you checked all the stuff about irq affinities, i presume, since a
> > > lot of things might have changed between 2.6.32 and 3.5 ?
> >
> > It is a single core E5 Xeon - i've set the affinity like this:
>
> And you still have the retransmits in "netstat -s" output ?
>
> Might be a firmware or pci issue, I have same cards but no problem here.
>
> Check LRO is on ?
>
> ethtool -k eth2
>
Ah, your ethtool -S gives strange fdir_miss counts, you should ask Intel
guys help maybe...
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: 10GBE performance drop with net.ipv4.tcp_timestamps=0
2012-06-20 9:06 ` Stefan Priebe - Profihost AG
2012-06-20 9:12 ` Stefan Priebe - Profihost AG
@ 2012-06-20 9:16 ` Eric Dumazet
1 sibling, 0 replies; 17+ messages in thread
From: Eric Dumazet @ 2012-06-20 9:16 UTC (permalink / raw)
To: Stefan Priebe - Profihost AG; +Cc: Linux Netdev List
On Wed, 2012-06-20 at 11:06 +0200, Stefan Priebe - Profihost AG wrote:
> Am 20.06.2012 10:41, schrieb Eric Dumazet:
> > On Wed, 2012-06-20 at 10:21 +0200, Stefan Priebe - Profihost AG wrote:
> >> Am 20.06.2012 09:22, schrieb Eric Dumazet:
> > Yes, you already told that in subject line.
> >
> > single tcp flow ?
> I use iperf - i think it uses just a single tcp flow. But i'm not sure.
>
> > You seem to have a switch or something that drops packets in this case.
> > You could try to rate limit to 9Gb/s and see if it is better.
> Sadly i can't rate limit to 9Gbit/s on the switch.
I was suggesting rate limiting on your linux sender machine, just to
verify if its indeed a problem on the path.
Its a matter of a "tc qdisc ..." command ;)
Or maybe iperf has an option for rate limiting.
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2012-06-20 11:08 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-06-19 21:08 10GBE performance drop with net.ipv4.tcp_timestamps=0 Stefan Priebe
2012-06-19 21:31 ` Eric Dumazet
2012-06-20 7:00 ` Stefan Priebe - Profihost AG
2012-06-20 7:22 ` Eric Dumazet
2012-06-20 8:21 ` Stefan Priebe - Profihost AG
2012-06-20 8:41 ` Eric Dumazet
2012-06-20 9:06 ` Stefan Priebe - Profihost AG
2012-06-20 9:12 ` Stefan Priebe - Profihost AG
2012-06-20 9:17 ` Eric Dumazet
2012-06-20 9:25 ` Stefan Priebe - Profihost AG
2012-06-20 9:28 ` Eric Dumazet
2012-06-20 9:33 ` Stefan Priebe - Profihost AG
2012-06-20 9:47 ` Eric Dumazet
2012-06-20 9:50 ` Stefan Priebe - Profihost AG
2012-06-20 10:06 ` Eric Dumazet
2012-06-20 11:08 ` Eric Dumazet
2012-06-20 9:16 ` Eric Dumazet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).