From: Amir Vadai <amirv@mellanox.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Or Gerlitz <gerlitz.or@gmail.com>,
Alexei Starovoitov <ast@plumgrid.com>,
"David S. Miller" <davem@davemloft.net>,
Jesper Dangaard Brouer <brouer@redhat.com>,
Eric Dumazet <edumazet@google.com>,
John Fastabend <john.r.fastabend@intel.com>,
Linux Netdev List <netdev@vger.kernel.org>,
"Or Gerlitz" <or.gerlitz@gmail.com>, <amira@mellanox.com>,
<idos@mellanox.com>, Yevgeny Petrilin <yevgenyp@mellanox.com>,
<eyalpe@mellanox.com>
Subject: Re: [PATCH v2 net-next] mlx4: optimize xmit path
Date: Thu, 2 Oct 2014 15:45:23 +0300 [thread overview]
Message-ID: <542D48E3.2090403@mellanox.com> (raw)
In-Reply-To: <1412251620.16704.88.camel@edumazet-glaptop2.roam.corp.google.com>
On 10/2/2014 3:07 PM, Eric Dumazet wrote:
> On Thu, 2014-10-02 at 14:56 +0300, Amir Vadai wrote:
>> After making sure the sender thread and the TX completions are not on
>> the same CPU, I see the expected improvement. +0.5Mpps with tx
>> optimizations.
>
> I got 40% here.
>
> Hmm... both cpus are on the same socket, right ?
Yes. And on a NUMA node close to the NIC
>
> TX coalescing is properly setup ?
I will try to play with it, if it is too aggressive, the queue is stopped.
I will play again with it and the ring size to find a better spot.
>
> What interrupt rate do you get ?
~50K per second.
>
> You can take a look at where cycles are spent
>
> perf record -a -g sleep 5
> perf report
>
>
I will.
In case your curious, I have this on the CPU doing completions:
+ 44.86% swapper [kernel.kallsyms] [k] consume_skb
+ 21.38% swapper [kernel.kallsyms] [k] mlx4_en_poll_tx_cq
+ 6.81% swapper [kernel.kallsyms] [k] mlx4_en_free_tx_desc.isra.24
+ 6.07% swapper [kernel.kallsyms] [k] intel_idle
+ 2.98% swapper [kernel.kallsyms] [k] __dev_kfree_skb_any
+ 2.66% rngd rngd [.] 0x0000000000002749
+ 1.79% rngd [kernel.kallsyms] [k] consume_skb
+ 1.28% swapper [kernel.kallsyms] [k] irq_entries_start
+ 1.16% rngd [kernel.kallsyms] [k] mlx4_en_poll_tx_cq
+ 0.86% swapper [kernel.kallsyms] [k] swiotlb_unmap_page
+ 0.79% swapper [kernel.kallsyms] [k] unmap_single
+ 0.63% swapper [kernel.kallsyms] [k] irqtime_account_irq
+ 0.55% swapper [kernel.kallsyms] [k] mlx4_eq_int
+ 0.51% swapper [kernel.kallsyms] [k] eq_set_ci.isra.14
and this on the xmit thread:
+ 72.00% kpktgend_6 [kernel.kallsyms] [k] mlx4_en_xmit
+ 13.11% kpktgend_6 [kernel.kallsyms] [k] pktgen_thread_worker
+ 5.34% kpktgend_6 [kernel.kallsyms] [k] _raw_spin_lock
+ 3.80% kpktgend_6 [kernel.kallsyms] [k] swiotlb_map_page
+ 2.08% kpktgend_6 [kernel.kallsyms] [k] __iowrite64_copy
+ 1.21% kpktgend_6 [kernel.kallsyms] [k] skb_clone_tx_timestamp
+ 0.87% kpktgend_6 [kernel.kallsyms] [k] kthread_should_stop
+ 0.64% kpktgend_6 [kernel.kallsyms] [k] swiotlb_dma_mapping_error
+ 0.51% kpktgend_6 [kernel.kallsyms] [k] __local_bh_enable_ip
Thanks,
Amir
next prev parent reply other threads:[~2014-10-02 12:45 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-09-26 0:46 [RFC PATCH net-next] net: pktgen: packet bursting via skb->xmit_more Alexei Starovoitov
2014-09-26 1:20 ` Eric Dumazet
2014-09-26 7:42 ` Eric Dumazet
2014-09-26 15:44 ` Eric Dumazet
2014-09-26 15:59 ` Alexei Starovoitov
2014-09-26 16:06 ` Eric Dumazet
2014-09-27 20:43 ` Eric Dumazet
2014-09-27 20:55 ` Or Gerlitz
2014-09-27 21:30 ` Eric Dumazet
2014-09-27 22:56 ` [PATCH net-next] mlx4: optimize xmit path Eric Dumazet
2014-09-27 23:44 ` Hannes Frederic Sowa
2014-09-28 0:05 ` Eric Dumazet
2014-09-28 0:22 ` Hannes Frederic Sowa
2014-09-28 12:42 ` Eric Dumazet
2014-09-28 14:35 ` Or Gerlitz
2014-09-28 16:03 ` Eric Dumazet
2014-09-29 4:19 ` [PATCH v2 " Eric Dumazet
2014-09-30 12:01 ` Amir Vadai
2014-09-30 12:11 ` Eric Dumazet
2014-10-02 4:35 ` Eric Dumazet
2014-10-02 8:03 ` Amir Vadai
2014-10-02 8:29 ` Jesper Dangaard Brouer
2014-10-02 8:57 ` Amir Vadai
2014-10-02 11:45 ` Eric Dumazet
2014-10-02 11:56 ` Amir Vadai
2014-10-02 12:07 ` Eric Dumazet
2014-10-02 12:45 ` Amir Vadai [this message]
2014-09-26 8:05 ` [RFC PATCH net-next] net: pktgen: packet bursting via skb->xmit_more Jesper Dangaard Brouer
2014-09-27 20:59 ` Or Gerlitz
-- strict thread matches above, loose matches on Subject: below --
2014-09-29 17:46 [PATCH v2 net-next] mlx4: optimize xmit path Alexei Starovoitov
2014-09-29 18:08 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=542D48E3.2090403@mellanox.com \
--to=amirv@mellanox.com \
--cc=amira@mellanox.com \
--cc=ast@plumgrid.com \
--cc=brouer@redhat.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=eric.dumazet@gmail.com \
--cc=eyalpe@mellanox.com \
--cc=gerlitz.or@gmail.com \
--cc=idos@mellanox.com \
--cc=john.r.fastabend@intel.com \
--cc=netdev@vger.kernel.org \
--cc=or.gerlitz@gmail.com \
--cc=yevgenyp@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.