From: Eric Dumazet <dada1@cosmosbay.com>
To: "Paweł Staszewski" <pstaszewski@itcare.pl>
Cc: Linux Network Development list <netdev@vger.kernel.org>
Subject: Re: results with diff affinity settings using ixgbe
Date: Tue, 27 Jan 2009 16:53:24 +0100 [thread overview]
Message-ID: <497F2DF4.4000507@cosmosbay.com> (raw)
In-Reply-To: <497F24CE.6060201@itcare.pl>
Paweł Staszewski a écrit :
> I make some tests with affinity on my machine with some traffic load.
> If someone want look:
>
>
>
> Traffic in all test was 350Mbit/s TX and the same on RX
>
> RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu4
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.05 0.00 0.00 0.00 1.89 10.03
> 0.00 88.03 221511.70
> Average: 0 0.40 0.00 0.00 0.00 0.00 0.00
> 0.00 99.60 0.70
> Average: 1 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.80
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.70
> Average: 3 0.00 0.00 0.10 0.00 0.00 0.40
> 0.00 99.50 0.70
> Average: 4 0.00 0.00 0.00 0.00 2.06 18.74
> 0.00 79.20 12870.50
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.30
> Average: 6 0.00 0.00 0.00 0.00 2.65 57.49
> 0.00 39.86 59410.60
> Average: 7 0.00 0.00 0.00 0.00 10.50 5.00
> 0.00 84.50 53977.90
>
>
> RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu1
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.08 0.00 0.09 0.00 0.75 14.24
> 0.00 84.84 215755.20
> Average: 0 0.50 0.00 0.00 0.00 0.00 0.00
> 0.00 99.50 1.30
> Average: 1 0.00 0.00 0.53 0.00 2.34 36.91
> 0.00 60.21 9771.90
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.40
> Average: 3 0.00 0.00 0.00 0.00 0.00 0.20
> 0.00 99.80 1.30
> Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.30
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.80
> Average: 6 0.00 0.00 0.20 0.00 2.65 78.51
> 0.00 18.64 54691.70
> Average: 7 0.00 0.00 0.00 0.00 1.03 0.51
> 0.00 98.46 54060.60
>
>
> All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu1
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.07 0.00 0.10 0.00 1.19 9.44
> 0.00 89.19 210155.50
> Average: 0 0.60 0.00 0.00 0.00 0.00 0.00
> 0.00 99.40 0.60
> Average: 1 0.00 0.00 0.41 0.00 2.87 16.27
> 0.00 80.45 13045.30
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.70
> Average: 3 0.10 0.00 0.30 0.00 0.00 0.20
> 0.00 99.40 0.60
> Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.70
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 1.10
> Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 0.60
> Average: 7 0.00 0.00 0.10 0.00 6.77 60.00
> 0.00 33.13 107242.50
>
>
> All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu4
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.05 0.00 0.04 0.00 1.24 9.55
> 0.00 89.12 212186.90
> Average: 0 0.40 0.00 0.00 0.00 0.00 0.00
> 0.00 99.60 2.20
> Average: 1 0.10 0.00 0.00 0.00 0.00 0.00
> 0.00 99.90 2.30
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 2.20
> Average: 3 0.00 0.00 0.00 0.00 0.00 0.30
> 0.00 99.70 2.20
> Average: 4 0.00 0.00 0.00 0.00 1.03 14.68
> 0.00 84.28 11965.50
> Average: 5 0.00 0.00 0.10 0.00 0.00 0.00
> 0.00 99.90 2.80
> Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 100.00 2.30
> Average: 7 0.00 0.00 0.20 0.00 8.97 62.69
> 0.00 28.13 110943.50
>
>
> Affinity set to "ffffffff" / and intel e1000e is on cpu4
> Average: CPU %user %nice %sys %iowait %irq %soft
> %steal %idle intr/s
> Average: all 0.10 0.00 0.14 0.01 1.16 15.07
> 0.00 83.52 185043.10
> Average: 0 0.00 0.00 0.00 0.10 2.90 36.90
> 0.00 60.10 9008.60
> Average: 1 0.00 0.00 0.10 0.00 2.30 28.36
> 0.00 69.24 9009.10
> Average: 2 0.80 0.00 0.40 0.00 0.50 13.76
> 0.00 84.54 9008.90
> Average: 3 0.00 0.00 0.20 0.00 0.20 4.61
> 0.00 94.98 9008.90
> Average: 4 0.00 0.00 0.10 0.00 2.35 19.55
> 0.00 77.99 14142.10
> Average: 5 0.00 0.00 0.10 0.00 0.20 3.85
> 0.00 95.84 9009.10
> Average: 6 0.00 0.00 0.10 0.00 0.40 4.22
> 0.00 95.28 9008.90
> Average: 7 0.00 0.00 0.10 0.00 0.51 8.69
> 0.00 90.70 9009.00
>
> And if i split more RX/TX queues to different CPU-s i have more load
> done by:
> CPU: Core 2, speed 2999.98 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
> unit mask of 0x00 (Unhalted core cycles) count 100000
> samples % image name app name
> symbol name
> 458001 29.5565 vmlinux vmlinux
> native_read_tsc
>
> If all interrupts are on one CPU i have the best results
> So for me the best is to put ixgbe work on one CPU or make affinity to
> balance over all cpus
I am confused by all these numbers. What is your goal exactly ?
If one CPU is able to do the whole workload, it will be faster too.
All other cpus are idle and wont fight with cache lines.
>
> But something is wrong with my intel e1000e
> because if i put intel e1000e with ixgbe on the same CPU system have
> 100% cpu load and all hangs.
Try to reproduce the bug and submit it ?
prev parent reply other threads:[~2009-01-27 15:53 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-01-27 15:14 results with diff affinity settings using ixgbe Paweł Staszewski
2009-01-27 15:53 ` Eric Dumazet [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=497F2DF4.4000507@cosmosbay.com \
--to=dada1@cosmosbay.com \
--cc=netdev@vger.kernel.org \
--cc=pstaszewski@itcare.pl \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.