From: Or Gerlitz <or.gerlitz@gmail.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: David Miller <davem@davemloft.net>,
Joseph Gasparakis <joseph.gasparakis@intel.com>,
Jerry Chu <hkchu@google.com>, Or Gerlitz <ogerlitz@mellanox.com>,
Eric Dumazet <edumazet@google.com>,
Alexei Starovoitov <ast@plumgrid.com>,
Pravin B Shelar <pshelar@nicira.com>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>
Subject: Re: vxlan/veth performance issues on net.git + latest kernels
Date: Wed, 4 Dec 2013 01:30:09 +0200 [thread overview]
Message-ID: <CAJZOPZKg0UMmdknLJCgNo3cTFmszASLPYYJnB7g9h4M64xMy8w@mail.gmail.com> (raw)
In-Reply-To: <CAJZOPZ+bkvg4G6qzY6YmMui9-KmGmpHyFCJUD2xsFLOXm12rgA@mail.gmail.com>
On Wed, Dec 4, 2013 at 1:10 AM, Or Gerlitz <or.gerlitz@gmail.com> wrote:
> On Tue, Dec 3, 2013 at 11:55 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>> On Tue, 2013-12-03 at 16:50 -0500, David Miller wrote:
>>
>>> At this point I would use drop monitor to figure out in what context
>>> packets are being dropped on the floor. There are scripts provided
>>> with the perf tool to utilize it.
>>
>> Most easy way is to do :
>>
>> perf record -e skb:kfree_skb -a -g sleep 10
>> perf report
>
> $ ./perf record -e skb:kfree_skb -g -a sleep 10
> $ ./perf report -i perf.data
>
>
> Samples: 883K of event 'skb:kfree_skb', Event count (approx.): 883406
> + 97.13% swapper [kernel.kallsyms] [k] net_tx_action
> + 1.53% iperf [kernel.kallsyms] [k] net_tx_action
> + 1.03% perf [kernel.kallsyms] [k] net_tx_action
> + 0.27% ksoftirqd/7 [kernel.kallsyms] [k] net_tx_action
> + 0.03% kworker/7:1 [kernel.kallsyms] [k] net_tx_action
> + 0.00% rpcbind [kernel.kallsyms] [k] net_tx_action
> + 0.00% swapper [kernel.kallsyms] [k] kfree_skb
> + 0.00% sleep [kernel.kallsyms] [k] net_tx_action
> + 0.00% hald-addon-acpi [kernel.kallsyms] [k] kfree_skb
> + 0.00% iperf [kernel.kallsyms] [k] kfree_skb
> + 0.00% perf [kernel.kallsyms] [k] kfree_skb
I added proper sorting (thanks Rick), here's the passive side report
- 99.99% [kernel.kallsyms] [k] net_tx_action
?
- net_tx_action
?
- 59.20% __libc_recv
?
100.00% 0
?
- 35.61% __write_nocancel
?
- 100.00% run_builtin
?
main
?
__libc_start_main
?
- 2.60% __poll
?
- 92.62% run_builtin
?
main
?
__libc_start_main
?
- 7.38% 0x7f2d4c4adfd4
?
0x626370523a32333a
?
- 1.86% cmd_record
?
run_builtin
?
main
?
__libc_start_main
?
- 0.01% [kernel.kallsyms] [k] kfree_skb
?
- kfree_skb
?
- 50.00% __connect_nocancel
?
0x64697063612f6e75
?
- 33.33% __libc_recv
?
0
?
- 16.67% __write_nocancel
?
run_builtin
?
main
?
__libc_start_main
and the active side report
100.00% [kernel.kallsyms] [k] net_tx_action
?
- net_tx_action
?
- 76.91% __write_nocancel
?
- 100.00% run_builtin
?
main
?
__libc_start_main
?
15.92% 0x37cc60e4ed
?
- 2.69% __poll
?
run_builtin
?
main
?
__libc_start_main
?
- 1.92% cmd_record
?
run_builtin
?
main
?
__libc_start_main
?
- 1.66% pthread_cond_signal@@GLIBC_2.
3.2
?
0x10000
?
- 0.52% perf_header__has_feat
?
- 73.28% run_builtin
?
main
?
__libc_start_main
?
- 26.72% cmd_record
?
run_builtin
?
main
?
__libc_start_main
?
- 0.00% [kernel.kallsyms] [k] kfree_skb
?
- kfree_skb
?
- 100.00% __GI___connect_internal
?
- 50.00% get_mapping
?
__nscd_get_map_ref
?
50.00% __nscd_open_socket
>
>
>
>>
>>
>>
next prev parent reply other threads:[~2013-12-03 23:30 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-12-03 15:05 vxlan/veth performance issues on net.git + latest kernels Or Gerlitz
2013-12-03 15:30 ` Eric Dumazet
2013-12-03 19:55 ` Or Gerlitz
2013-12-03 21:11 ` Joseph Gasparakis
2013-12-03 21:09 ` Or Gerlitz
2013-12-03 21:24 ` Eric Dumazet
2013-12-03 21:36 ` Or Gerlitz
2013-12-03 21:50 ` David Miller
2013-12-03 21:55 ` Eric Dumazet
2013-12-03 22:15 ` Or Gerlitz
2013-12-03 22:22 ` Or Gerlitz
2013-12-03 22:30 ` Hannes Frederic Sowa
2013-12-03 22:35 ` Or Gerlitz
2013-12-03 22:39 ` Hannes Frederic Sowa
2013-12-03 23:10 ` Or Gerlitz
2013-12-03 23:30 ` Or Gerlitz [this message]
2013-12-03 23:49 ` Hannes Frederic Sowa
2013-12-03 23:59 ` Eric Dumazet
2013-12-04 0:26 ` Alexei Starovoitov
2013-12-04 0:36 ` Eric Dumazet
2013-12-04 0:55 ` Alexei Starovoitov
2013-12-04 1:23 ` Eric Dumazet
2013-12-04 1:59 ` Alexei Starovoitov
2013-12-06 9:06 ` Or Gerlitz
2013-12-06 13:36 ` Eric Dumazet
2013-12-07 21:20 ` Or Gerlitz
2013-12-08 12:09 ` Or Gerlitz
2013-12-04 6:39 ` David Miller
2013-12-04 17:40 ` Eric Dumazet
2013-12-05 12:45 ` [PATCH net-next] net: introduce dev_consume_skb_any() Eric Dumazet
2013-12-05 14:13 ` Hannes Frederic Sowa
2013-12-05 14:45 ` Eric Dumazet
2013-12-05 15:05 ` Eric Dumazet
2013-12-05 15:44 ` Hannes Frederic Sowa
2013-12-05 16:38 ` Eric Dumazet
2013-12-05 16:54 ` Hannes Frederic Sowa
2013-12-06 20:24 ` David Miller
2013-12-03 23:13 ` vxlan/veth performance issues on net.git + latest kernels Joseph Gasparakis
2013-12-03 23:09 ` Or Gerlitz
2013-12-04 0:35 ` Joseph Gasparakis
2013-12-04 0:34 ` Alexei Starovoitov
2013-12-04 1:29 ` Joseph Gasparakis
2013-12-04 1:18 ` Eric Dumazet
2013-12-04 0:44 ` Joseph Gasparakis
2013-12-04 8:35 ` Or Gerlitz
2013-12-04 9:24 ` Joseph Gasparakis
2013-12-04 9:41 ` Or Gerlitz
2013-12-04 15:20 ` Or Gerlitz
[not found] ` <52A197DF.5010806@mellanox.com>
2013-12-06 9:30 ` Or Gerlitz
2013-12-08 12:43 ` Mike Rapoport
2013-12-08 13:07 ` Or Gerlitz
2013-12-08 14:30 ` Mike Rapoport
2013-12-08 20:50 ` Eric Dumazet
2013-12-08 21:36 ` Eric Dumazet
2013-12-06 10:30 ` Joseph Gasparakis
2013-12-07 21:27 ` Or Gerlitz
2013-12-08 18:08 ` Joseph Gasparakis
2013-12-08 20:12 ` Or Gerlitz
2013-12-08 15:21 ` Or Gerlitz
2013-12-03 17:12 ` Eric Dumazet
2013-12-03 19:50 ` Or Gerlitz
2013-12-03 20:19 ` John Fastabend
2013-12-03 21:12 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAJZOPZKg0UMmdknLJCgNo3cTFmszASLPYYJnB7g9h4M64xMy8w@mail.gmail.com \
--to=or.gerlitz@gmail.com \
--cc=ast@plumgrid.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=eric.dumazet@gmail.com \
--cc=hkchu@google.com \
--cc=joseph.gasparakis@intel.com \
--cc=netdev@vger.kernel.org \
--cc=ogerlitz@mellanox.com \
--cc=pshelar@nicira.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).