From: Mugunthan V N <mugunthanvnm@ti.com>
To: Florian Fainelli <f.fainelli@gmail.com>
Cc: Ben Hutchings <bhutchings@solarflare.com>,
netdev <netdev@vger.kernel.org>
Subject: Re: TI CPSW Ethernet Tx performance regression
Date: Tue, 4 Feb 2014 00:04:07 +0530 [thread overview]
Message-ID: <52EFE11F.5040605@ti.com> (raw)
In-Reply-To: <CAGVrzcbdDAAfFXjYc-ksxqxeJWeY_Jyh1DbwFiOW=p7WqRvzFQ@mail.gmail.com>
Hi
On Friday 17 January 2014 05:05 AM, Florian Fainelli wrote:
> Whenever I had bad TX performance with hardware, the culprit was that
> transmit buffers were not freed quickly enough so the transmit
> scheduler cannot push as many packets as expected. When this happens,
> the root cause for me was bad TX interrupt which messed up the TX flow
> control, but there are plenty other stuff that can go wrong.
>
> You could try to check a few things like TX interrupt rate for the
> same workload on both kernels, dump the queue usage every few seconds
> etc...
I did a further analysis using oprofile and found some more info. In
v3.2 kernel most of the time is spend in csum_partial_copy_from_user and
cpdma_chan_submit which are in the path of tx but the dump in v3.12 cpu
is held more in __do_softirq and __irq_put_desc_unlock. I think because
of this Tx performance is affected. Since __do_softirq is used to invode
NAPI, how to reduce its priority or is there any other code that I
should be looking into?
Pasting the O-Profile dump with iperf running in v3.2 and v3.12 kernel
v3.2:
====
samples % app name symbol name
33152 9.3792 vmlinux-3.2 csum_partial_copy_from_user
23960 6.7786 vmlinux-3.2 cpdma_chan_submit
19288 5.4569 vmlinux-3.2 __do_softirq
13425 3.7981 vmlinux-3.2 __irq_put_desc_unlock
11065 3.1305 vmlinux-3.2 tcp_packet
8458 2.3929 vmlinux-3.2 __cpdma_chan_free
8386 2.3725 vmlinux-3.2 cpdma_ctlr_int_ctrl
7316 2.0698 vmlinux-3.2 __cpdma_chan_process
5186 1.4672 vmlinux-3.2 tcp_transmit_skb
5118 1.4480 vmlinux-3.2 ipt_do_table
4954 1.4016 vmlinux-3.2 kfree
4857 1.3741 vmlinux-3.2 nf_iterate
4797 1.3571 vmlinux-3.2 tcp_ack
4511 1.2762 vmlinux-3.2 __kmalloc
4433 1.2542 vmlinux-3.2 v7_dma_inv_range
4393 1.2428 vmlinux-3.2 nf_conntrack_in
4069 1.1512 vmlinux-3.2 tcp_sendmsg
3607 1.0205 vmlinux-3.2 local_bh_enable
3148 0.8906 vmlinux-3.2 __memzero
3127 0.8847 vmlinux-3.2 csum_partial
2850 0.8063 vmlinux-3.2 __alloc_skb
2825 0.7992 vmlinux-3.2 ip_queue_xmit
2559 0.7240 vmlinux-3.2 tcp_write_xmit
2399 0.6787 vmlinux-3.2 clocksource_read_cycles
2091 0.5916 vmlinux-3.2 dev_hard_start_xmit
v3.12:
=====
samples % app name symbol name
9040 15.8034 vmlinux __do_softirq
6410 11.2057 vmlinux __irq_put_desc_unlock
3584 6.2654 vmlinux cpdma_chan_submit
3250 5.6815 vmlinux csum_partial_copy_from_user
3070 5.3669 vmlinux __cpdma_chan_process
2894 5.0592 vmlinux resend_irqs
2567 4.4875 vmlinux cpdma_ctlr_int_ctrl
2214 3.8704 vmlinux mod_timer
1922 3.3600 vmlinux lock_acquire
1402 2.4509 vmlinux __cpdma_chan_free
1063 1.8583 vmlinux local_bh_enable
783 1.3688 vmlinux cpdma_check_free_tx_desc
668 1.1678 vmlinux lock_is_held
610 1.0664 vmlinux __kmalloc_track_caller
584 1.0209 vmlinux lock_release
559 0.9772 vmlinux kmem_cache_alloc
557 0.9737 vmlinux kfree
460 0.8042 vmlinux tcp_transmit_skb
429 0.7500 vmlinux tcp_ack
418 0.7307 vmlinux tcp_sendmsg
378 0.6608 vmlinux kmem_cache_free
366 0.6398 vmlinux ip_queue_xmit
363 0.6346 vmlinux cache_alloc_refill
351 0.6136 vmlinux sub_preempt_count
347 0.6066 vmlinux napi_complete
335 0.5856 vmlinux __alloc_skb
311 0.5437 vmlinux ip_finish_output
next prev parent reply other threads:[~2014-02-03 18:34 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-01-15 12:48 TI CPSW Ethernet Tx performance regression Mugunthan V N
2014-01-15 17:54 ` Ben Hutchings
2014-01-15 21:21 ` Florian Fainelli
2014-01-16 6:07 ` Mugunthan V N
2014-01-16 23:35 ` Florian Fainelli
2014-02-03 18:34 ` Mugunthan V N [this message]
2014-02-03 19:24 ` Florian Fainelli
2014-02-04 8:46 ` Mugunthan V N
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=52EFE11F.5040605@ti.com \
--to=mugunthanvnm@ti.com \
--cc=bhutchings@solarflare.com \
--cc=f.fainelli@gmail.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.