From: Amir Vadai <amirv@mellanox.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Or Gerlitz <gerlitz.or@gmail.com>,
Alexei Starovoitov <ast@plumgrid.com>,
"David S. Miller" <davem@davemloft.net>,
Jesper Dangaard Brouer <brouer@redhat.com>,
Eric Dumazet <edumazet@google.com>,
John Fastabend <john.r.fastabend@intel.com>,
Linux Netdev List <netdev@vger.kernel.org>,
"Or Gerlitz" <or.gerlitz@gmail.com>, <amira@mellanox.com>,
<idos@mellanox.com>, Yevgeny Petrilin <yevgenyp@mellanox.com>,
<eyalpe@mellanox.com>
Subject: Re: [PATCH v2 net-next] mlx4: optimize xmit path
Date: Thu, 2 Oct 2014 15:45:23 +0300 [thread overview]
Message-ID: <542D48E3.2090403@mellanox.com> (raw)
In-Reply-To: <1412251620.16704.88.camel@edumazet-glaptop2.roam.corp.google.com>
On 10/2/2014 3:07 PM, Eric Dumazet wrote:
> On Thu, 2014-10-02 at 14:56 +0300, Amir Vadai wrote:
>> After making sure the sender thread and the TX completions are not on
>> the same CPU, I see the expected improvement. +0.5Mpps with tx
>> optimizations.
>
> I got 40% here.
>
> Hmm... both cpus are on the same socket, right ?
Yes. And on a NUMA node close to the NIC
>
> TX coalescing is properly setup ?
I will try to play with it, if it is too aggressive, the queue is stopped.
I will play again with it and the ring size to find a better spot.
>
> What interrupt rate do you get ?
~50K per second.
>
> You can take a look at where cycles are spent
>
> perf record -a -g sleep 5
> perf report
>
>
I will.
In case your curious, I have this on the CPU doing completions:
+ 44.86% swapper [kernel.kallsyms] [k] consume_skb
+ 21.38% swapper [kernel.kallsyms] [k] mlx4_en_poll_tx_cq
+ 6.81% swapper [kernel.kallsyms] [k] mlx4_en_free_tx_desc.isra.24
+ 6.07% swapper [kernel.kallsyms] [k] intel_idle
+ 2.98% swapper [kernel.kallsyms] [k] __dev_kfree_skb_any
+ 2.66% rngd rngd [.] 0x0000000000002749
+ 1.79% rngd [kernel.kallsyms] [k] consume_skb
+ 1.28% swapper [kernel.kallsyms] [k] irq_entries_start
+ 1.16% rngd [kernel.kallsyms] [k] mlx4_en_poll_tx_cq
+ 0.86% swapper [kernel.kallsyms] [k] swiotlb_unmap_page
+ 0.79% swapper [kernel.kallsyms] [k] unmap_single
+ 0.63% swapper [kernel.kallsyms] [k] irqtime_account_irq
+ 0.55% swapper [kernel.kallsyms] [k] mlx4_eq_int
+ 0.51% swapper [kernel.kallsyms] [k] eq_set_ci.isra.14
and this on the xmit thread:
+ 72.00% kpktgend_6 [kernel.kallsyms] [k] mlx4_en_xmit
+ 13.11% kpktgend_6 [kernel.kallsyms] [k] pktgen_thread_worker
+ 5.34% kpktgend_6 [kernel.kallsyms] [k] _raw_spin_lock
+ 3.80% kpktgend_6 [kernel.kallsyms] [k] swiotlb_map_page
+ 2.08% kpktgend_6 [kernel.kallsyms] [k] __iowrite64_copy
+ 1.21% kpktgend_6 [kernel.kallsyms] [k] skb_clone_tx_timestamp
+ 0.87% kpktgend_6 [kernel.kallsyms] [k] kthread_should_stop
+ 0.64% kpktgend_6 [kernel.kallsyms] [k] swiotlb_dma_mapping_error
+ 0.51% kpktgend_6 [kernel.kallsyms] [k] __local_bh_enable_ip
Thanks,
Amir
next prev parent reply other threads:[~2014-10-02 12:45 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-09-26 0:46 [RFC PATCH net-next] net: pktgen: packet bursting via skb->xmit_more Alexei Starovoitov
2014-09-26 1:20 ` Eric Dumazet
2014-09-26 7:42 ` Eric Dumazet
2014-09-26 15:44 ` Eric Dumazet
2014-09-26 15:59 ` Alexei Starovoitov
2014-09-26 16:06 ` Eric Dumazet
2014-09-27 20:43 ` Eric Dumazet
2014-09-27 20:55 ` Or Gerlitz
2014-09-27 21:30 ` Eric Dumazet
2014-09-27 22:56 ` [PATCH net-next] mlx4: optimize xmit path Eric Dumazet
2014-09-27 23:44 ` Hannes Frederic Sowa
2014-09-28 0:05 ` Eric Dumazet
2014-09-28 0:22 ` Hannes Frederic Sowa
2014-09-28 12:42 ` Eric Dumazet
2014-09-28 14:35 ` Or Gerlitz
2014-09-28 16:03 ` Eric Dumazet
2014-09-29 4:19 ` [PATCH v2 " Eric Dumazet
2014-09-30 12:01 ` Amir Vadai
2014-09-30 12:11 ` Eric Dumazet
2014-10-02 4:35 ` Eric Dumazet
2014-10-02 8:03 ` Amir Vadai
2014-10-02 8:29 ` Jesper Dangaard Brouer
2014-10-02 8:57 ` Amir Vadai
2014-10-02 11:45 ` Eric Dumazet
2014-10-02 11:56 ` Amir Vadai
2014-10-02 12:07 ` Eric Dumazet
2014-10-02 12:45 ` Amir Vadai [this message]
2014-09-26 8:05 ` [RFC PATCH net-next] net: pktgen: packet bursting via skb->xmit_more Jesper Dangaard Brouer
2014-09-27 20:59 ` Or Gerlitz
-- strict thread matches above, loose matches on Subject: below --
2014-09-29 17:46 [PATCH v2 net-next] mlx4: optimize xmit path Alexei Starovoitov
2014-09-29 18:08 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=542D48E3.2090403@mellanox.com \
--to=amirv@mellanox.com \
--cc=amira@mellanox.com \
--cc=ast@plumgrid.com \
--cc=brouer@redhat.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=eric.dumazet@gmail.com \
--cc=eyalpe@mellanox.com \
--cc=gerlitz.or@gmail.com \
--cc=idos@mellanox.com \
--cc=john.r.fastabend@intel.com \
--cc=netdev@vger.kernel.org \
--cc=or.gerlitz@gmail.com \
--cc=yevgenyp@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).