From: Jay Vosburgh <fubar@us.ibm.com>
To: Chris Caputo <ccaputo-dated-1268032962.0a05a1@alt.net>
Cc: bonding-devel@lists.sourceforge.net, netdev@vger.kernel.org
Subject: Re: bonding forwarding perf issues in 2.6.32.7 & 2.6.29.6
Date: Sat, 06 Feb 2010 08:36:51 -0800 [thread overview]
Message-ID: <1125.1265474211@death.nxdomain.ibm.com> (raw)
In-Reply-To: <Pine.LNX.4.64.1002060641380.14159@nacho.alt.net>
Chris Caputo <ccaputo@alt.net> wrote:
>Kernel 2.6.32.7 (and 2.6.32.5 & 2.6.29.6) on a 2x Intel Xeon E5420
>(Quad-Core 2.5Ghz), SuperMicro X7DBE+, 32GB (16 * 2GB) DDR2-667MHz.
>
>I have a router with a variety of e1000 and e1000e based interfaces.
>
>bond0 is a 2xGigE (82571EB) with two active slaves.
>
>bond1 has up to 3 slaves (2x 80003ES2LAN/82563, 82546EB).
>
>Both are configured with miimon=100, balance-xor, layer3+4.
>
>When bond1 has just a single active slave, outbound (and possibly inbound)
>forwarding performance on bond1 is better than when it has two or three
>active slaves. Ie., when I activate the second slave, by enabling the
>port on the switch it is connected to, forwarding performance drops
>dramatically across the full bond1.
What exactly do you mean by "forwarding performance drops
dramatically"? How are you measuring this?
Also, just to confirm, are the switch ports connected to the
respective bonds also grouped on the switch? The balance-xor mode is
meant to interop with an Etherchannel compatible switch port
aggregation.
>Locally originated packets do not seem to be harmed by the second GigE
>coming online. From what I have observed, the issue is with forwarding.
>The majority of the forwarding traffic is coming in on bond0 and egressing
>on bond1.
Perhaps it has something to do with forwarding causing LRO to be
disabled.
-J
>I have tried changing IRQ binding in a variety of ways (same CPU, same
>core, different cores, paired based on bond, irqbalance) and it hasn't
>helped.
>
>I have tried having one of bond1's GigEs be on a separate bus with a
>separate NIC, to no avail.
>
>Oprofiling (data below) does not reveal much time is being spent in the
>bonding driver. bond_start_xmit() is the peak for the bonding driver, at
>less than 1% regardless of how many interfaces are bound.
>
>Does anyone have any tips on how I should try to narrow down this further?
>
>Thanks,
>Chris
>
>---
>
>bond1 with just one 80003ES2LAN/82563 active:
>
>samples % image name app name symbol name
>114103 13.4161 vmlinux-2.6.32.7 vmlinux-2.6.32.7 ipt_do_table
>24447 2.8745 e1000e.ko e1000e.ko e1000_xmit_frame
>23687 2.7851 vmlinux-2.6.32.7 vmlinux-2.6.32.7 dev_queue_xmit
>19088 2.2444 e1000e.ko e1000e.ko e1000_clean_tx_irq
>18820 2.2128 vmlinux-2.6.32.7 vmlinux-2.6.32.7 skb_copy_bits
>16028 1.8846 vmlinux-2.6.32.7 vmlinux-2.6.32.7 skb_segment
>15013 1.7652 vmlinux-2.6.32.7 vmlinux-2.6.32.7 __slab_free
>14187 1.6681 vmlinux-2.6.32.7 vmlinux-2.6.32.7 __slab_alloc
>13649 1.6048 vmlinux-2.6.32.7 vmlinux-2.6.32.7 mwait_idle
>13177 1.5493 e1000e.ko e1000e.ko e1000_irq_enable
>13017 1.5305 bgpd bgpd bgp_process_announce_selected
>12242 1.4394 vmlinux-2.6.32.7 vmlinux-2.6.32.7 __alloc_skb
>11186 1.3152 vmlinux-2.6.32.7 vmlinux-2.6.32.7 ip_vs_in
>11054 1.2997 vmlinux-2.6.32.7 vmlinux-2.6.32.7 find_vma
>10861 1.2770 vmlinux-2.6.32.7 vmlinux-2.6.32.7 ip_rcv
>10724 1.2609 vmlinux-2.6.32.7 vmlinux-2.6.32.7 nf_iterate
>10659 1.2533 vmlinux-2.6.32.7 vmlinux-2.6.32.7 kmem_cache_alloc
>
>bond1 with a 80003ES2LAN/82563 and a 82546EB active:
>
>samples % image name app name symbol name
>36249 14.1261 vmlinux-2.6.32.7 vmlinux-2.6.32.7 ipt_do_table
>5985 2.3323 vmlinux-2.6.32.7 vmlinux-2.6.32.7 skb_copy_bits
>5731 2.2333 vmlinux-2.6.32.7 vmlinux-2.6.32.7 __slab_free
>5496 2.1418 e1000.ko e1000.ko e1000_clean
>5489 2.1390 vmlinux-2.6.32.7 vmlinux-2.6.32.7 dev_queue_xmit
>5247 2.0447 vmlinux-2.6.32.7 vmlinux-2.6.32.7 mwait_idle
>5090 1.9835 e1000e.ko e1000e.ko e1000_xmit_frame
>5025 1.9582 e1000e.ko e1000e.ko e1000_irq_enable
>4777 1.8616 vmlinux-2.6.32.7 vmlinux-2.6.32.7 __slab_alloc
>4714 1.8370 e1000e.ko e1000e.ko e1000_clean_tx_irq
>4102 1.5985 e1000.ko e1000.ko e1000_intr
>4004 1.5603 vmlinux-2.6.32.7 vmlinux-2.6.32.7 skb_segment
>3924 1.5292 e1000e.ko e1000e.ko e1000_intr_msi
>3867 1.5070 vmlinux-2.6.32.7 vmlinux-2.6.32.7 __alloc_skb
>3424 1.3343 e1000.ko e1000.ko e1000_xmit_frame
>3225 1.2568 vmlinux-2.6.32.7 vmlinux-2.6.32.7 find_vma
>3148 1.2268 vmlinux-2.6.32.7 vmlinux-2.6.32.7 kfree
>
>bond1 with 2x 80003ES2LAN/82563 active and a 82546EB active:
>
>samples % image name app name symbol name
>28124 14.5651 vmlinux-2.6.32.7 vmlinux-2.6.32.7 ipt_do_table
>5725 2.9649 e1000e.ko e1000e.ko e1000_irq_enable
>5077 2.6293 vmlinux-2.6.32.7 vmlinux-2.6.32.7 mwait_idle
>4374 2.2652 vmlinux-2.6.32.7 vmlinux-2.6.32.7 skb_copy_bits
>4277 2.2150 e1000e.ko e1000e.ko e1000_intr_msi
>4224 2.1876 e1000e.ko e1000e.ko e1000_xmit_frame
>3863 2.0006 vmlinux-2.6.32.7 vmlinux-2.6.32.7 __slab_free
>3826 1.9814 e1000e.ko e1000e.ko e1000_clean_tx_irq
>3682 1.9069 vmlinux-2.6.32.7 vmlinux-2.6.32.7 dev_queue_xmit
>3512 1.8188 vmlinux-2.6.32.7 vmlinux-2.6.32.7 __slab_alloc
>3191 1.6526 e1000.ko e1000.ko e1000_clean
>3042 1.5754 e1000.ko e1000.ko e1000_intr
>2540 1.3154 vmlinux-2.6.32.7 vmlinux-2.6.32.7 __alloc_skb
>2425 1.2559 vmlinux-2.6.32.7 vmlinux-2.6.32.7 ip_rcv
>2406 1.2460 vmlinux-2.6.32.7 vmlinux-2.6.32.7 skb_segment
>2333 1.2082 vmlinux-2.6.32.7 vmlinux-2.6.32.7 nf_iterate
>2329 1.2062 vmlinux-2.6.32.7 vmlinux-2.6.32.7 ip_vs_in
>--
>To unsubscribe from this list: send the line "unsubscribe netdev" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2010-02-06 16:36 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-02-06 7:22 bonding forwarding perf issues in 2.6.32.7 & 2.6.29.6 Chris Caputo
2010-02-06 16:36 ` Jay Vosburgh [this message]
2010-02-07 6:52 ` Chris Caputo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1125.1265474211@death.nxdomain.ibm.com \
--to=fubar@us.ibm.com \
--cc=bonding-devel@lists.sourceforge.net \
--cc=ccaputo-dated-1268032962.0a05a1@alt.net \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).