* results with diff affinity settings using ixgbe
@ 2009-01-27 15:14 Paweł Staszewski
2009-01-27 15:53 ` Eric Dumazet
0 siblings, 1 reply; 2+ messages in thread
From: Paweł Staszewski @ 2009-01-27 15:14 UTC (permalink / raw)
To: Linux Network Development list
I make some tests with affinity on my machine with some traffic load.
If someone want look:
Traffic in all test was 350Mbit/s TX and the same on RX
RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu4
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.05 0.00 0.00 0.00 1.89 10.03
0.00 88.03 221511.70
Average: 0 0.40 0.00 0.00 0.00 0.00 0.00
0.00 99.60 0.70
Average: 1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.80
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.70
Average: 3 0.00 0.00 0.10 0.00 0.00 0.40
0.00 99.50 0.70
Average: 4 0.00 0.00 0.00 0.00 2.06 18.74
0.00 79.20 12870.50
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 1.30
Average: 6 0.00 0.00 0.00 0.00 2.65 57.49
0.00 39.86 59410.60
Average: 7 0.00 0.00 0.00 0.00 10.50 5.00
0.00 84.50 53977.90
RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu1
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.08 0.00 0.09 0.00 0.75 14.24
0.00 84.84 215755.20
Average: 0 0.50 0.00 0.00 0.00 0.00 0.00
0.00 99.50 1.30
Average: 1 0.00 0.00 0.53 0.00 2.34 36.91
0.00 60.21 9771.90
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 1.40
Average: 3 0.00 0.00 0.00 0.00 0.00 0.20
0.00 99.80 1.30
Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 1.30
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 1.80
Average: 6 0.00 0.00 0.20 0.00 2.65 78.51
0.00 18.64 54691.70
Average: 7 0.00 0.00 0.00 0.00 1.03 0.51
0.00 98.46 54060.60
All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu1
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.07 0.00 0.10 0.00 1.19 9.44
0.00 89.19 210155.50
Average: 0 0.60 0.00 0.00 0.00 0.00 0.00
0.00 99.40 0.60
Average: 1 0.00 0.00 0.41 0.00 2.87 16.27
0.00 80.45 13045.30
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.70
Average: 3 0.10 0.00 0.30 0.00 0.00 0.20
0.00 99.40 0.60
Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.70
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 1.10
Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.60
Average: 7 0.00 0.00 0.10 0.00 6.77 60.00
0.00 33.13 107242.50
All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu4
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.05 0.00 0.04 0.00 1.24 9.55
0.00 89.12 212186.90
Average: 0 0.40 0.00 0.00 0.00 0.00 0.00
0.00 99.60 2.20
Average: 1 0.10 0.00 0.00 0.00 0.00 0.00
0.00 99.90 2.30
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 2.20
Average: 3 0.00 0.00 0.00 0.00 0.00 0.30
0.00 99.70 2.20
Average: 4 0.00 0.00 0.00 0.00 1.03 14.68
0.00 84.28 11965.50
Average: 5 0.00 0.00 0.10 0.00 0.00 0.00
0.00 99.90 2.80
Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 2.30
Average: 7 0.00 0.00 0.20 0.00 8.97 62.69
0.00 28.13 110943.50
Affinity set to "ffffffff" / and intel e1000e is on cpu4
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.10 0.00 0.14 0.01 1.16 15.07
0.00 83.52 185043.10
Average: 0 0.00 0.00 0.00 0.10 2.90 36.90
0.00 60.10 9008.60
Average: 1 0.00 0.00 0.10 0.00 2.30 28.36
0.00 69.24 9009.10
Average: 2 0.80 0.00 0.40 0.00 0.50 13.76
0.00 84.54 9008.90
Average: 3 0.00 0.00 0.20 0.00 0.20 4.61
0.00 94.98 9008.90
Average: 4 0.00 0.00 0.10 0.00 2.35 19.55
0.00 77.99 14142.10
Average: 5 0.00 0.00 0.10 0.00 0.20 3.85
0.00 95.84 9009.10
Average: 6 0.00 0.00 0.10 0.00 0.40 4.22
0.00 95.28 9008.90
Average: 7 0.00 0.00 0.10 0.00 0.51 8.69
0.00 90.70 9009.00
And if i split more RX/TX queues to different CPU-s i have more load
done by:
CPU: Core 2, speed 2999.98 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
unit mask of 0x00 (Unhalted core cycles) count 100000
samples % image name app name
symbol name
458001 29.5565 vmlinux vmlinux
native_read_tsc
If all interrupts are on one CPU i have the best results
So for me the best is to put ixgbe work on one CPU or make affinity to
balance over all cpus
But something is wrong with my intel e1000e
because if i put intel e1000e with ixgbe on the same CPU system have
100% cpu load and all hangs.
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: results with diff affinity settings using ixgbe
2009-01-27 15:14 results with diff affinity settings using ixgbe Paweł Staszewski
@ 2009-01-27 15:53 ` Eric Dumazet
0 siblings, 0 replies; 2+ messages in thread
From: Eric Dumazet @ 2009-01-27 15:53 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: Linux Network Development list
Paweł Staszewski a écrit :
> I make some tests with affinity on my machine with some traffic load.
> If someone want look:
>
>
>
> Traffic in all test was 350Mbit/s TX and the same on RX
>
> RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu4
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.05 0.00 0.00 0.00 1.89 10.03
> 0.00 88.03 221511.70
> Average: 0 0.40 0.00 0.00 0.00 0.00 0.00
> 0.00 99.60 0.70
> Average: 1 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.80
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.70
> Average: 3 0.00 0.00 0.10 0.00 0.00 0.40
> 0.00 99.50 0.70
> Average: 4 0.00 0.00 0.00 0.00 2.06 18.74
> 0.00 79.20 12870.50
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.30
> Average: 6 0.00 0.00 0.00 0.00 2.65 57.49
> 0.00 39.86 59410.60
> Average: 7 0.00 0.00 0.00 0.00 10.50 5.00
> 0.00 84.50 53977.90
>
>
> RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu1
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.08 0.00 0.09 0.00 0.75 14.24
> 0.00 84.84 215755.20
> Average: 0 0.50 0.00 0.00 0.00 0.00 0.00
> 0.00 99.50 1.30
> Average: 1 0.00 0.00 0.53 0.00 2.34 36.91
> 0.00 60.21 9771.90
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.40
> Average: 3 0.00 0.00 0.00 0.00 0.00 0.20
> 0.00 99.80 1.30
> Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.30
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.80
> Average: 6 0.00 0.00 0.20 0.00 2.65 78.51
> 0.00 18.64 54691.70
> Average: 7 0.00 0.00 0.00 0.00 1.03 0.51
> 0.00 98.46 54060.60
>
>
> All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu1
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.07 0.00 0.10 0.00 1.19 9.44
> 0.00 89.19 210155.50
> Average: 0 0.60 0.00 0.00 0.00 0.00 0.00
> 0.00 99.40 0.60
> Average: 1 0.00 0.00 0.41 0.00 2.87 16.27
> 0.00 80.45 13045.30
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.70
> Average: 3 0.10 0.00 0.30 0.00 0.00 0.20
> 0.00 99.40 0.60
> Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.70
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.10
> Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.60
> Average: 7 0.00 0.00 0.10 0.00 6.77 60.00
> 0.00 33.13 107242.50
>
>
> All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu4
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.05 0.00 0.04 0.00 1.24 9.55
> 0.00 89.12 212186.90
> Average: 0 0.40 0.00 0.00 0.00 0.00 0.00
> 0.00 99.60 2.20
> Average: 1 0.10 0.00 0.00 0.00 0.00 0.00
> 0.00 99.90 2.30
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 2.20
> Average: 3 0.00 0.00 0.00 0.00 0.00 0.30
> 0.00 99.70 2.20
> Average: 4 0.00 0.00 0.00 0.00 1.03 14.68
> 0.00 84.28 11965.50
> Average: 5 0.00 0.00 0.10 0.00 0.00 0.00
> 0.00 99.90 2.80
> Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 2.30
> Average: 7 0.00 0.00 0.20 0.00 8.97 62.69
> 0.00 28.13 110943.50
>
>
> Affinity set to "ffffffff" / and intel e1000e is on cpu4
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.10 0.00 0.14 0.01 1.16 15.07
> 0.00 83.52 185043.10
> Average: 0 0.00 0.00 0.00 0.10 2.90 36.90
> 0.00 60.10 9008.60
> Average: 1 0.00 0.00 0.10 0.00 2.30 28.36
> 0.00 69.24 9009.10
> Average: 2 0.80 0.00 0.40 0.00 0.50 13.76
> 0.00 84.54 9008.90
> Average: 3 0.00 0.00 0.20 0.00 0.20 4.61
> 0.00 94.98 9008.90
> Average: 4 0.00 0.00 0.10 0.00 2.35 19.55
> 0.00 77.99 14142.10
> Average: 5 0.00 0.00 0.10 0.00 0.20 3.85
> 0.00 95.84 9009.10
> Average: 6 0.00 0.00 0.10 0.00 0.40 4.22
> 0.00 95.28 9008.90
> Average: 7 0.00 0.00 0.10 0.00 0.51 8.69
> 0.00 90.70 9009.00
>
> And if i split more RX/TX queues to different CPU-s i have more load
> done by:
> CPU: Core 2, speed 2999.98 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
> unit mask of 0x00 (Unhalted core cycles) count 100000
> samples % image name app name
> symbol name
> 458001 29.5565 vmlinux vmlinux
> native_read_tsc
>
> If all interrupts are on one CPU i have the best results
> So for me the best is to put ixgbe work on one CPU or make affinity to
> balance over all cpus
I am confused by all these numbers. What is your goal exactly ?
If one CPU is able to do the whole workload, it will be faster too.
All other cpus are idle and wont fight with cache lines.
>
> But something is wrong with my intel e1000e
> because if i put intel e1000e with ixgbe on the same CPU system have
> 100% cpu load and all hangs.
Try to reproduce the bug and submit it ?
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2009-01-27 15:53 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-01-27 15:14 results with diff affinity settings using ixgbe Paweł Staszewski
2009-01-27 15:53 ` Eric Dumazet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox