* Tools for sampling ethtool --statistics
@ 2016-01-20 9:13 Jesper Dangaard Brouer
2016-01-20 22:34 ` Or Gerlitz
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Jesper Dangaard Brouer @ 2016-01-20 9:13 UTC (permalink / raw)
To: netdev@vger.kernel.org; +Cc: brouer, Or Gerlitz
Hi All,
I wrote a small tool[1] to extract ethtool --statistics|-S, sample and
present it in a more human readable manor. You might also find this
useful...
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
The reason I wrote this script is that the mlx5 driver (Mellanox 100 Gbit/s)
does not report correct ifconfig stats. It reports the HW counters,
not what the network stack is able to handle. I/we obviously need the
stack number when working on improving the stack.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer
Output examples:
pktgen (1 core) sending ~20Mpps
Show adapter mlx5p2 statistics (ONLY that changed!)
Ethtool(mlx5p2) stat: 19431694 ( 19,431,694) <= frames_rx /sec
Ethtool(mlx5p2) stat: 1243722215 ( 1,243,722,215) <= in_octets /sec
Ethtool(mlx5p2) stat: 19433161 ( 19,433,161) <= in_ucast_pkts /sec
Ethtool(mlx5p2) stat: 1243746943 ( 1,243,746,943) <= octets /sec
Ethtool(mlx5p2) stat: 1243629263 ( 1,243,629,263) <= octets_received /sec
Ethtool(mlx5p2) stat: 19433654 ( 19,433,654) <= p64octets /sec
Ethtool(mlx5p2) stat: 19433556 ( 19,433,556) <= pkts /sec
Ethtool(mlx5p2) stat: 1815546 ( 1,815,546) <= rx3_packets /sec
Ethtool(mlx5p2) stat: 1165854821 ( 1,165,854,821) <= rx_bytes /sec
Ethtool(mlx5p2) stat: 19430906 ( 19,430,906) <= rx_csum_good /sec
Ethtool(mlx5p2) stat: 19430906 ( 19,430,906) <= rx_packets /sec
Ethtool(mlx5p2) stat: 1165854821 ( 1,165,854,821) <= rx_unicast_bytes /sec
Ethtool(mlx5p2) stat: 19430906 ( 19,430,906) <= rx_unicast_packets /sec
pktgen (4 cores) sending ~50Mpps
Show adapter mlx5p2 statistics (ONLY that changed!)
Ethtool(mlx5p2) stat: 25710443 ( 25,710,443) <= drop_events /sec
Ethtool(mlx5p2) stat: 51861645 ( 51,861,645) <= frames_rx /sec
Ethtool(mlx5p2) stat: 25710502 ( 25,710,502) <= in_discards /sec
Ethtool(mlx5p2) stat: 3319134161 ( 3,319,134,161) <= in_octets /sec
Ethtool(mlx5p2) stat: 26150970 ( 26,150,970) <= in_ucast_pkts /sec
Ethtool(mlx5p2) stat: 3319128053 ( 3,319,128,053) <= octets /sec
Ethtool(mlx5p2) stat: 3319145319 ( 3,319,145,319) <= octets_received /sec
Ethtool(mlx5p2) stat: 51861369 ( 51,861,369) <= p64octets /sec
Ethtool(mlx5p2) stat: 51861375 ( 51,861,375) <= pkts /sec
Ethtool(mlx5p2) stat: 1383475 ( 1,383,475) <= rx3_packets /sec
Ethtool(mlx5p2) stat: 1569059171 ( 1,569,059,171) <= rx_bytes /sec
Ethtool(mlx5p2) stat: 26150988 ( 26,150,988) <= rx_csum_good /sec
Ethtool(mlx5p2) stat: 26150988 ( 26,150,988) <= rx_packets /sec
Ethtool(mlx5p2) stat: 1569059171 ( 1,569,059,171) <= rx_unicast_bytes /sec
Ethtool(mlx5p2) stat: 26150988 ( 26,150,988) <= rx_unicast_packets /sec
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Tools for sampling ethtool --statistics
2016-01-20 9:13 Tools for sampling ethtool --statistics Jesper Dangaard Brouer
@ 2016-01-20 22:34 ` Or Gerlitz
2016-01-20 22:53 ` John Fastabend
2016-01-21 10:35 ` Jesper Dangaard Brouer
[not found] ` <CAKHjkj=+zaYXvHQWkWuU2LLVRxT_V7Pvmd3QqoGUdOL=HDjsRA@mail.gmail.com>
2016-02-24 15:21 ` Or Gerlitz
2 siblings, 2 replies; 6+ messages in thread
From: Or Gerlitz @ 2016-01-20 22:34 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: netdev@vger.kernel.org, Or Gerlitz
On Wed, Jan 20, 2016 at 11:13 AM, Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
> Hi All,
>
> I wrote a small tool[1] to extract ethtool --statistics|-S, sample and
> present it in a more human readable manor. You might also find this
> useful...
>
> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>
> The reason I wrote this script is that the mlx5 driver (Mellanox 100 Gbit/s)
> does not report correct ifconfig stats. It reports the HW counters,
> not what the network stack is able to handle. I/we obviously need the
> stack number when working on improving the stack.
correct && we should be fixing that ... still it's worthwhile to be
able and see per ring stats, specially when you're dealing/optimizing
in multi-stream env.
> Output examples:
>
> pktgen (1 core) sending ~20Mpps
>
> Show adapter mlx5p2 statistics (ONLY that changed!)
> Ethtool(mlx5p2) stat: 19431694 ( 19,431,694) <= frames_rx /sec
> Ethtool(mlx5p2) stat: 1243722215 ( 1,243,722,215) <= in_octets /sec
> Ethtool(mlx5p2) stat: 19433161 ( 19,433,161) <= in_ucast_pkts /sec
> Ethtool(mlx5p2) stat: 1243746943 ( 1,243,746,943) <= octets /sec
> Ethtool(mlx5p2) stat: 1243629263 ( 1,243,629,263) <= octets_received /sec
> Ethtool(mlx5p2) stat: 19433654 ( 19,433,654) <= p64octets /sec
> Ethtool(mlx5p2) stat: 19433556 ( 19,433,556) <= pkts /sec
> Ethtool(mlx5p2) stat: 1815546 ( 1,815,546) <= rx3_packets /sec
> Ethtool(mlx5p2) stat: 1165854821 ( 1,165,854,821) <= rx_bytes /sec
> Ethtool(mlx5p2) stat: 19430906 ( 19,430,906) <= rx_csum_good /sec
> Ethtool(mlx5p2) stat: 19430906 ( 19,430,906) <= rx_packets /sec
> Ethtool(mlx5p2) stat: 1165854821 ( 1,165,854,821) <= rx_unicast_bytes /sec
> Ethtool(mlx5p2) stat: 19430906 ( 19,430,906) <= rx_unicast_packets /sec
was expecting to see tx_XXX stats, something wrong here?
> pktgen (4 cores) sending ~50Mpps
[...]
The 20M for single ring/core and 50M pps for four streams are really
cool results. Do the scripts to get ~such results exist in your
network testing github repo? last time I looked I wasn't sure if the
multi-stream case is supported.
Could you post the two commands you used for these results along with
minimal spec of the system (CPU type/freq, special .config directives,
anything else)?
Or.
> Show adapter mlx5p2 statistics (ONLY that changed!)
> Ethtool(mlx5p2) stat: 25710443 ( 25,710,443) <= drop_events /sec
> Ethtool(mlx5p2) stat: 51861645 ( 51,861,645) <= frames_rx /sec
> Ethtool(mlx5p2) stat: 25710502 ( 25,710,502) <= in_discards /sec
> Ethtool(mlx5p2) stat: 3319134161 ( 3,319,134,161) <= in_octets /sec
> Ethtool(mlx5p2) stat: 26150970 ( 26,150,970) <= in_ucast_pkts /sec
> Ethtool(mlx5p2) stat: 3319128053 ( 3,319,128,053) <= octets /sec
> Ethtool(mlx5p2) stat: 3319145319 ( 3,319,145,319) <= octets_received /sec
> Ethtool(mlx5p2) stat: 51861369 ( 51,861,369) <= p64octets /sec
> Ethtool(mlx5p2) stat: 51861375 ( 51,861,375) <= pkts /sec
> Ethtool(mlx5p2) stat: 1383475 ( 1,383,475) <= rx3_packets /sec
> Ethtool(mlx5p2) stat: 1569059171 ( 1,569,059,171) <= rx_bytes /sec
> Ethtool(mlx5p2) stat: 26150988 ( 26,150,988) <= rx_csum_good /sec
> Ethtool(mlx5p2) stat: 26150988 ( 26,150,988) <= rx_packets /sec
> Ethtool(mlx5p2) stat: 1569059171 ( 1,569,059,171) <= rx_unicast_bytes /sec
> Ethtool(mlx5p2) stat: 26150988 ( 26,150,988) <= rx_unicast_packets /sec
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Tools for sampling ethtool --statistics
2016-01-20 22:34 ` Or Gerlitz
@ 2016-01-20 22:53 ` John Fastabend
2016-01-21 10:35 ` Jesper Dangaard Brouer
1 sibling, 0 replies; 6+ messages in thread
From: John Fastabend @ 2016-01-20 22:53 UTC (permalink / raw)
To: Or Gerlitz, Jesper Dangaard Brouer; +Cc: netdev@vger.kernel.org, Or Gerlitz
[...]
>
>> pktgen (4 cores) sending ~50Mpps
> [...]
>
> The 20M for single ring/core and 50M pps for four streams are really
> cool results. Do the scripts to get ~such results exist in your
> network testing github repo? last time I looked I wasn't sure if the
> multi-stream case is supported.
>
> Could you post the two commands you used for these results along with
> minimal spec of the system (CPU type/freq, special .config directives,
> anything else)?
>
Take a look at
./samples/pktgen/pktgen_sample02_multiqueue.sh
./samples/pktgen/pktgen_sample03_burst_single_flow.sh
that should get you there.
.John
> Or.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Tools for sampling ethtool --statistics
2016-01-20 22:34 ` Or Gerlitz
2016-01-20 22:53 ` John Fastabend
@ 2016-01-21 10:35 ` Jesper Dangaard Brouer
1 sibling, 0 replies; 6+ messages in thread
From: Jesper Dangaard Brouer @ 2016-01-21 10:35 UTC (permalink / raw)
To: Or Gerlitz; +Cc: netdev@vger.kernel.org, Or Gerlitz, brouer
On Thu, 21 Jan 2016 00:34:16 +0200
Or Gerlitz <gerlitz.or@gmail.com> wrote:
> On Wed, Jan 20, 2016 at 11:13 AM, Jesper Dangaard Brouer
> <brouer@redhat.com> wrote:
> > Hi All,
> >
> > I wrote a small tool[1] to extract ethtool --statistics|-S, sample and
> > present it in a more human readable manor. You might also find this
> > useful...
> >
> > https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
> >
> > The reason I wrote this script is that the mlx5 driver (Mellanox 100 Gbit/s)
> > does not report correct ifconfig stats. It reports the HW counters,
> > not what the network stack is able to handle. I/we obviously need the
> > stack number when working on improving the stack.
>
> correct && we should be fixing that ...
Yes, this need to be fixed. I'm currently playing around in that code,
and will fix it if you don't ;-)... (although, to be realistic, I will
not have time until after netdev 1.1. to submit upstream patches).
> still it's worthwhile to be
> able and see per ring stats, specially when you're dealing/optimizing
> in multi-stream env.
It is actually also useful for detecting other situation, which are
visible via obscure stats that we usually don't look at. I've been
postponing writing this script for years, and in the end it only took
me 2 hours to write (and 1 hour fixups the next day).
> > Output examples:
> >
> > pktgen (1 core) sending ~20Mpps
> >
> > Show adapter mlx5p2 statistics (ONLY that changed!)
> > Ethtool(mlx5p2) stat: 19431694 ( 19,431,694) <= frames_rx /sec
> > Ethtool(mlx5p2) stat: 1243722215 ( 1,243,722,215) <= in_octets /sec
> > Ethtool(mlx5p2) stat: 19433161 ( 19,433,161) <= in_ucast_pkts /sec
> > Ethtool(mlx5p2) stat: 1243746943 ( 1,243,746,943) <= octets /sec
> > Ethtool(mlx5p2) stat: 1243629263 ( 1,243,629,263) <= octets_received /sec
> > Ethtool(mlx5p2) stat: 19433654 ( 19,433,654) <= p64octets /sec
> > Ethtool(mlx5p2) stat: 19433556 ( 19,433,556) <= pkts /sec
> > Ethtool(mlx5p2) stat: 1815546 ( 1,815,546) <= rx3_packets /sec
> > Ethtool(mlx5p2) stat: 1165854821 ( 1,165,854,821) <= rx_bytes /sec
> > Ethtool(mlx5p2) stat: 19430906 ( 19,430,906) <= rx_csum_good /sec
> > Ethtool(mlx5p2) stat: 19430906 ( 19,430,906) <= rx_packets /sec
> > Ethtool(mlx5p2) stat: 1165854821 ( 1,165,854,821) <= rx_unicast_bytes /sec
> > Ethtool(mlx5p2) stat: 19430906 ( 19,430,906) <= rx_unicast_packets /sec
>
> was expecting to see tx_XXX stats, something wrong here?
This was a RX only test (with another machine generating packets with
pktgen).
TX_xxx stats does show up when I run netperf. And the single netperf
results are around 50Gbit/s (if IRQ RX process runs on another CPU that
netperf process, I guess this was Eric point earlier), if IRQ and
netperf runs on same CPU, perf drops to 23-26Gbit/s. Running multiple
(super_)netperf the result max out at 64Gbit/s.
> > pktgen (4 cores) sending ~50Mpps
> [...]
>
> The 20M for single ring/core and 50M pps for four streams are really
> cool results.
Yes, these 100G NICs are definitely the fastest HW I currently have!
Thanks BTW ;-)
> Do the scripts to get ~such results exist in your
> network testing github repo? last time I looked I wasn't sure if the
> multi-stream case is supported.
>
> Could you post the two commands you used for these results along with
> minimal spec of the system (CPU type/freq, special .config directives,
> anything else)?
Two mlx5 100G NICs
- firmware-version: 12.12.1240
- Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s)
and 100GbE; dual-port QSFP28; PCIe3.0 x16; ROHS R6
- PSID: MT_2190110032
The sender/generator machine:
- CPU: i7-4790K CPU @ 4.00GHz
The receiver machine:
- CPU: i7-6700K CPU @ 4.00GHz ("skylake")
Pktgen command [1] or [2]:
./pktgen_sample03_burst_single_flow.sh -i mlx5p2 -d 192.168.100.66 -m e4:1d:2d:af:60:65 -c1 -t 4
For sending multi-threads use parameter "-t X".
> > Show adapter mlx5p2 statistics (ONLY that changed!)
> > Ethtool(mlx5p2) stat: 25710443 ( 25,710,443) <= drop_events /sec
> > Ethtool(mlx5p2) stat: 51861645 ( 51,861,645) <= frames_rx /sec
> > Ethtool(mlx5p2) stat: 25710502 ( 25,710,502) <= in_discards /sec
Notice I had to pressure it more that 26 Mpps before in_discards and
drop_events show up.
> > Ethtool(mlx5p2) stat: 3319134161 ( 3,319,134,161) <= in_octets /sec
> > Ethtool(mlx5p2) stat: 26150970 ( 26,150,970) <= in_ucast_pkts /sec
> > Ethtool(mlx5p2) stat: 3319128053 ( 3,319,128,053) <= octets /sec
> > Ethtool(mlx5p2) stat: 3319145319 ( 3,319,145,319) <= octets_received /sec
> > Ethtool(mlx5p2) stat: 51861369 ( 51,861,369) <= p64octets /sec
> > Ethtool(mlx5p2) stat: 51861375 ( 51,861,375) <= pkts /sec
> > Ethtool(mlx5p2) stat: 1383475 ( 1,383,475) <= rx3_packets /sec
The rx3_packets were reduced a little (1.8Mpps -> 1.4Mpps), once
in_discards starts...
> > Ethtool(mlx5p2) stat: 1569059171 ( 1,569,059,171) <= rx_bytes /sec
> > Ethtool(mlx5p2) stat: 26150988 ( 26,150,988) <= rx_csum_good /sec
> > Ethtool(mlx5p2) stat: 26150988 ( 26,150,988) <= rx_packets /sec
> > Ethtool(mlx5p2) stat: 1569059171 ( 1,569,059,171) <= rx_unicast_bytes /sec
> > Ethtool(mlx5p2) stat: 26150988 ( 26,150,988) <= rx_unicast_packets /sec
[1] https://github.com/netoptimizer/network-testing/tree/master/pktgen
[2] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/samples/pktgen
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Tools for sampling ethtool --statistics
[not found] ` <CAKHjkj=+zaYXvHQWkWuU2LLVRxT_V7Pvmd3QqoGUdOL=HDjsRA@mail.gmail.com>
@ 2016-02-21 9:40 ` Rana Shahout
0 siblings, 0 replies; 6+ messages in thread
From: Rana Shahout @ 2016-02-21 9:40 UTC (permalink / raw)
To: brouer, netdev, ogerlitz
> Hi All,
>
> I wrote a small tool[1] to extract ethtool --statistics|-S, sample and
> present it in a more human readable manor. You might also find this
> useful...
>
> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>
> The reason I wrote this script is that the mlx5 driver (Mellanox 100 Gbit/s)
> does not report correct ifconfig stats. It reports the HW counters,
> not what the network stack is able to handle. I/we obviously need the
> stack number when working on improving the stack.
mlx5 drive does not report correct counters also in ethtool statistics
(ethtool -S).
ethtool statistics reports also the HW counters, so all counters
except counters per channel are wrong.
We are working on fix. It will be available soon.
>
> --
> Best regards,
> Jesper Dangaard Brouer
> MSc.CS, Principal Kernel Engineer at Red Hat
> Author of http://www.iptv-analyzer.org
> LinkedIn: http://www.linkedin.com/in/brouer
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Tools for sampling ethtool --statistics
2016-01-20 9:13 Tools for sampling ethtool --statistics Jesper Dangaard Brouer
2016-01-20 22:34 ` Or Gerlitz
[not found] ` <CAKHjkj=+zaYXvHQWkWuU2LLVRxT_V7Pvmd3QqoGUdOL=HDjsRA@mail.gmail.com>
@ 2016-02-24 15:21 ` Or Gerlitz
2 siblings, 0 replies; 6+ messages in thread
From: Or Gerlitz @ 2016-02-24 15:21 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: netdev@vger.kernel.org
On Wed, Jan 20, 2016 at 11:13 AM, Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
> I wrote a small tool[1] to extract ethtool --statistics|-S, sample and
> present it in a more human readable manor. You might also find this useful...
[..]
> The reason I wrote this script is that the mlx5 driver (Mellanox 100 Gbit/s)
> does not report correct ifconfig stats. It reports the HW counters,
> not what the network stack is able to handle. I/we obviously need the
> stack number when working on improving the stack.
Hi Jesper,
So... we finally fixed that, see
https://patchwork.ozlabs.org/patch/587298/
https://patchwork.ozlabs.org/patch/587296/
Or.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-02-24 15:21 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-01-20 9:13 Tools for sampling ethtool --statistics Jesper Dangaard Brouer
2016-01-20 22:34 ` Or Gerlitz
2016-01-20 22:53 ` John Fastabend
2016-01-21 10:35 ` Jesper Dangaard Brouer
[not found] ` <CAKHjkj=+zaYXvHQWkWuU2LLVRxT_V7Pvmd3QqoGUdOL=HDjsRA@mail.gmail.com>
2016-02-21 9:40 ` Rana Shahout
2016-02-24 15:21 ` Or Gerlitz
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).