* Re: AF_VSOCK unimplemented sockopts
From: Stefan Hajnoczi @ 2017-08-15 8:36 UTC (permalink / raw)
To: Jorgen S. Hansen; +Cc: netdev@vger.kernel.org
In-Reply-To: <9D465B5B-5A07-4250-8AB3-DEC80CCC45B7@vmware.com>
[-- Attachment #1: Type: text/plain, Size: 1983 bytes --]
On Fri, Aug 11, 2017 at 09:23:17AM +0000, Jorgen S. Hansen wrote:
> > On Aug 3, 2017, at 12:41 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >
> > Hi Jorgen,
> > There are 3 sockopts defined in include/uapi/linux/vm_sockets.h that are
> > currently not implemented in net/vmw_vsock/af_vsock.c:
> >
> > * SO_VM_SOCKETS_PEER_HOST_VM_ID
> > * SO_VM_SOCKETS_TRUSTED
> > * SO_VM_SOCKETS_NONBLOCK_TXRX
> >
> > I noticed this because SO_VM_SOCKETS_TRUSTED is interesting for
> > virtio-vsock. Services listening on AF_VSOCK inside the guest may not
> > want arbitrary unprivileged host processes to connect. Instead of
> > inventing a new solution I wanted to look into SO_VM_SOCKETS_TRUSTED but
> > found it is not implemented in linux.git.
> >
> > What is the status of these sockets?
>
> These options were only implemented for ESX host endpoints, so were never part of the Linux host side support. It looks like they could have been omitted from vm_sockets.h, when the initial upstreaming was performed.
>
> On ESX, the equivalent of SO_VM_SOCKETS_TRUSTED, is used for retrieving the value of s->trusted of a VMCI socket. It cannot be used to mark a socket as trusted. On Linux, trusted is tied to the CAP_NET_ADMIN capability of the socket creator. VMCI based vSockets will per default only allow host side sockets that are trusted, or are created by the same user as the VM, to communicate with a given VM. This is achieved by per default creating VMs with the VMCI privilege flag VMCI_PRIVILEGE_FLAG_RESTRICTED. It is possible to create a VM that isn’t restricted, in which case any host process will be able to communicate with the VM.
>
> So it should be straight forward to implement the getsockopt part of SO_VM_SOCKETS_TRUSTED, since it just needs to return s->trusted.
Currently virtio-vsock does not implement the 'restricted' mode but I'm
evaluating using it by default for stronger security. Thanks for your
response!
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]
^ permalink raw reply
* (unknown),
From: ccc @ 2017-08-15 8:46 UTC (permalink / raw)
To: netdev
[-- Attachment #1: 765228233.zip --]
[-- Type: application/zip, Size: 3033 bytes --]
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Paweł Staszewski @ 2017-08-15 9:11 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
In-Reply-To: <1502759849.4936.46.camel@edumazet-glaptop3.roam.corp.google.com>
Hi
Yes it helped - now there is almost no difference when using vlans or not:
10.5Mpps - with vlan
11Mpps - without vlan
W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
> On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
>
>> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
> Something like :
>
> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
> index 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7 100644
> --- a/net/8021q/vlan_netlink.c
> +++ b/net/8021q/vlan_netlink.c
> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net, struct net_device *dev,
> vlan->vlan_proto = proto;
> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
> vlan->real_dev = real_dev;
> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
> vlan->flags = VLAN_FLAG_REORDER_HDR;
>
> err = vlan_check_real_dev(real_dev, vlan->vlan_proto, vlan->vlan_id);
>
>
>
>
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Paweł Staszewski @ 2017-08-15 9:19 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
In-Reply-To: <4b1efff7-4f91-fd78-beb8-2c7ebcf18895@itcare.pl>
With hack:
14.44% [kernel] [k] do_raw_spin_lock
8.30% [kernel] [k] page_frag_free
7.06% [mlx5_core] [k] mlx5e_xmit
5.97% [kernel] [k] acpi_processor_ffh_cstate_enter
5.73% [kernel] [k] fib_table_lookup
4.81% [mlx5_core] [k] mlx5e_poll_tx_cq
4.51% [mlx5_core] [k] skb_from_cqe.isra.32
3.81% [kernel] [k] virt_to_head_page
2.45% [kernel] [k] __dev_queue_xmit
1.84% [kernel] [k] ipt_do_table
1.77% [kernel] [k] napi_consume_skb
1.62% [kernel] [k] __build_skb
1.46% [kernel] [k] netif_skb_features
1.43% [kernel] [k] __netif_receive_skb_core
1.41% [kernel] [k] ip_rcv
1.08% [kernel] [k] dev_hard_start_xmit
1.02% [kernel] [k] build_skb
1.00% [mlx5_core] [k] mlx5_cqwq_get_cqe
0.96% [kernel] [k] ip_route_input_rcu
0.95% [kernel] [k] ip_forward
0.89% [kernel] [k] ip_finish_output2
0.89% [kernel] [k] kmem_cache_alloc
0.78% [kernel] [k] __local_bh_enable_ip
0.76% [kernel] [k] udp_v4_early_demux
0.75% [kernel] [k] compound_head
0.75% [kernel] [k] __netdev_pick_tx
0.73% [kernel] [k] sch_direct_xmit
0.65% [kernel] [k] irq_entries_start
0.63% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
0.61% [kernel] [k] netdev_pick_tx
0.61% [kernel] [k] validate_xmit_skb
0.55% [kernel] [k] skb_network_protocol
0.53% [mlx5_core] [k] mlx5e_rx_cache_get
0.53% [mlx5_core] [k] mlx5e_build_rx_skb
0.51% [kernel] [k] ip_rcv_finish
0.50% [kernel] [k] eth_header
0.50% [kernel] [k] fib_validate_source
0.50% [mlx5_core] [k] mlx5e_handle_rx_cqe
0.48% [mlx5_core] [k] eq_update_ci
0.47% [kernel] [k] kmem_cache_free_bulk
0.44% [kernel] [k] deliver_ptype_list_skb
0.43% [kernel] [k] skb_release_data
0.42% [kernel] [k] cpuidle_enter_state
0.40% [kernel] [k] virt_to_head_page
0.39% [kernel] [k] vlan_dev_hard_start_xmit
0.39% [kernel] [k] neigh_connected_output
0.38% [kernel] [k] eth_type_vlan
0.35% [mlx5_core] [k] mlx5e_alloc_rx_wqe
0.32% [kernel] [k] nf_hook_slow
0.32% [kernel] [k] swiotlb_map_page
0.31% [kernel] [k] ip_finish_output
0.29% [kernel] [k] ip_output
0.28% [kernel] [k] skb_free_head
0.25% [kernel] [k] netif_receive_skb_internal
0.25% [kernel] [k] __jhash_nwords
Without hack:
14.25% [kernel] [k] dst_release
14.17% [kernel] [k] skb_dst_force
13.41% [kernel] [k] rt_cache_valid
11.47% [kernel] [k] ip_finish_output2
7.01% [kernel] [k] do_raw_spin_lock
5.07% [kernel] [k] page_frag_free
3.47% [mlx5_core] [k] mlx5e_xmit
2.88% [kernel] [k] fib_table_lookup
2.43% [mlx5_core] [k] skb_from_cqe.isra.32
1.97% [kernel] [k] virt_to_head_page
1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
0.93% [kernel] [k] __dev_queue_xmit
0.87% [kernel] [k] __build_skb
0.84% [kernel] [k] ipt_do_table
0.79% [kernel] [k] ip_rcv
0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
0.78% [kernel] [k] netif_skb_features
0.73% [kernel] [k] __netif_receive_skb_core
0.52% [kernel] [k] dev_hard_start_xmit
0.52% [kernel] [k] build_skb
0.51% [kernel] [k] ip_route_input_rcu
0.50% [kernel] [k] skb_unref
0.49% [kernel] [k] ip_forward
0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
0.44% [kernel] [k] udp_v4_early_demux
0.41% [kernel] [k] napi_consume_skb
0.40% [kernel] [k] __local_bh_enable_ip
0.39% [kernel] [k] ip_rcv_finish
0.39% [kernel] [k] kmem_cache_alloc
0.38% [kernel] [k] sch_direct_xmit
0.33% [kernel] [k] validate_xmit_skb
0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
0.29% [kernel] [k] netdev_pick_tx
0.28% [mlx5_core] [k] mlx5e_build_rx_skb
0.27% [kernel] [k] deliver_ptype_list_skb
0.26% [kernel] [k] fib_validate_source
0.26% [mlx5_core] [k] mlx5e_napi_poll
0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
0.26% [mlx5_core] [k] mlx5e_rx_cache_get
0.25% [kernel] [k] eth_header
0.23% [kernel] [k] skb_network_protocol
0.20% [kernel] [k] nf_hook_slow
0.20% [kernel] [k] vlan_passthru_hard_header
0.20% [kernel] [k] vlan_dev_hard_start_xmit
0.19% [kernel] [k] swiotlb_map_page
0.18% [kernel] [k] compound_head
0.18% [kernel] [k] neigh_connected_output
0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
0.18% [kernel] [k] ip_output
0.17% [kernel] [k] prefetch_freepointer.isra.70
0.17% [kernel] [k] __slab_free
0.16% [kernel] [k] eth_type_vlan
0.16% [kernel] [k] ip_finish_output
0.15% [kernel] [k] kmem_cache_free_bulk
0.14% [kernel] [k] netif_receive_skb_internal
W dniu 2017-08-15 o 11:11, Paweł Staszewski pisze:
> Hi
>
>
> Yes it helped - now there is almost no difference when using vlans or
> not:
>
> 10.5Mpps - with vlan
>
> 11Mpps - without vlan
>
>
>
>
> W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
>> On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
>>
>>> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
>> Something like :
>>
>> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
>> index
>> 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
>> 100644
>> --- a/net/8021q/vlan_netlink.c
>> +++ b/net/8021q/vlan_netlink.c
>> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
>> struct net_device *dev,
>> vlan->vlan_proto = proto;
>> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
>> vlan->real_dev = real_dev;
>> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
>> vlan->flags = VLAN_FLAG_REORDER_HDR;
>> err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
>> vlan->vlan_id);
>>
>>
>>
>>
>
>
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Jesper Dangaard Brouer @ 2017-08-15 9:23 UTC (permalink / raw)
To: Paweł Staszewski
Cc: brouer, Linux Kernel Network Developers, Alexander Duyck,
Saeed Mahameed, Tariq Toukan
In-Reply-To: <1ff1b747-758e-afdd-9376-80ff3bd8a6d5@itcare.pl>
On Tue, 15 Aug 2017 02:38:56 +0200
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
> > On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> >
> >> To show some difference below comparision vlan/no-vlan traffic
> >>
> >> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
> > I'm trying to reproduce in my testlab (with ixgbe). I do see, a
> > performance reduction of about 10-19% when I forward out a VLAN
> > interface. This is larger than I expected, but still lower than what
> > you reported 30-40% slowdown.
> >
> > [...]
> Ok mellanox afrrived (MT27700 - mlnx5 driver)
> And to compare melannox with vlans and without: 33% performance
> degradation (less than with ixgbe where i reach ~40% with same settings)
>
> Mellanox without TX traffix on vlan:
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;16;64;11089305;709715520;8871553;567779392
> 1;16;64;11096292;710162688;11095566;710116224
> 2;16;64;11095770;710129280;11096799;710195136
> 3;16;64;11097199;710220736;11097702;710252928
> 4;16;64;11080984;567081856;11079662;709098368
> 5;16;64;11077696;708972544;11077039;708930496
> 6;16;64;11082991;709311424;8864802;567347328
> 7;16;64;11089596;709734144;8870927;709789184
> 8;16;64;11094043;710018752;11095391;710105024
>
> Mellanox with TX traffic on vlan:
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;16;64;7369914;471674496;7370281;471697980
> 1;16;64;7368896;471609408;7368043;471554752
> 2;16;64;7367577;471524864;7367759;471536576
> 3;16;64;7368744;377305344;7369391;471641024
> 4;16;64;7366824;471476736;7364330;471237120
> 5;16;64;7368352;471574528;7367239;471503296
> 6;16;64;7367459;471517376;7367806;471539584
> 7;16;64;7367190;471500160;7367988;471551232
> 8;16;64;7368023;471553472;7368076;471556864
I wonder if the drivers page recycler is active/working or not, and if
the situation is different between VLAN vs no-vlan (given
page_frag_free is so high in you perf top). The Mellanox drivers
fortunately have a stats counter to tell us this explicitly (which the
ixgbe driver doesn't).
You can use my ethtool_stats.pl script watch these stats:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
(Hint perl dependency: dnf install perl-Time-HiRes)
> ethtool settings for both tests:
> ifc='enp175s0f0 enp175s0f1'
> for i in $ifc
> do
> ip link set up dev $i
> ethtool -A $i autoneg off rx off tx off
> ethtool -G $i rx 128 tx 256
The ring queue size recommendations, might be different for the mlx5
driver (Cc'ing Mellanox maintainers).
> ip link set $i txqueuelen 1000
> ethtool -C $i rx-usecs 25
> ethtool -L $i combined 16
> ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
> tx-nocache-copy off ntuple on
> ethtool -N $i rx-flow-hash udp4 sdfn
> done
Thanks for being explicit about what you setup is :-)
> and perf top:
> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> 14.25% [kernel] [k] dst_release
> 14.17% [kernel] [k] skb_dst_force
> 13.41% [kernel] [k] rt_cache_valid
> 11.47% [kernel] [k] ip_finish_output2
> 7.01% [kernel] [k] do_raw_spin_lock
> 5.07% [kernel] [k] page_frag_free
> 3.47% [mlx5_core] [k] mlx5e_xmit
> 2.88% [kernel] [k] fib_table_lookup
> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
> 1.97% [kernel] [k] virt_to_head_page
> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
> 0.93% [kernel] [k] __dev_queue_xmit
> 0.87% [kernel] [k] __build_skb
> 0.84% [kernel] [k] ipt_do_table
> 0.79% [kernel] [k] ip_rcv
> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
> 0.78% [kernel] [k] netif_skb_features
> 0.73% [kernel] [k] __netif_receive_skb_core
> 0.52% [kernel] [k] dev_hard_start_xmit
> 0.52% [kernel] [k] build_skb
> 0.51% [kernel] [k] ip_route_input_rcu
> 0.50% [kernel] [k] skb_unref
> 0.49% [kernel] [k] ip_forward
> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
> 0.44% [kernel] [k] udp_v4_early_demux
> 0.41% [kernel] [k] napi_consume_skb
> 0.40% [kernel] [k] __local_bh_enable_ip
> 0.39% [kernel] [k] ip_rcv_finish
> 0.39% [kernel] [k] kmem_cache_alloc
> 0.38% [kernel] [k] sch_direct_xmit
> 0.33% [kernel] [k] validate_xmit_skb
> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
> 0.29% [kernel] [k] netdev_pick_tx
> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
> 0.27% [kernel] [k] deliver_ptype_list_skb
> 0.26% [kernel] [k] fib_validate_source
> 0.26% [mlx5_core] [k] mlx5e_napi_poll
> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
> 0.25% [kernel] [k] eth_header
> 0.23% [kernel] [k] skb_network_protocol
> 0.20% [kernel] [k] nf_hook_slow
> 0.20% [kernel] [k] vlan_passthru_hard_header
> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
> 0.19% [kernel] [k] swiotlb_map_page
> 0.18% [kernel] [k] compound_head
> 0.18% [kernel] [k] neigh_connected_output
> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
> 0.18% [kernel] [k] ip_output
> 0.17% [kernel] [k] prefetch_freepointer.isra.70
> 0.17% [kernel] [k] __slab_free
> 0.16% [kernel] [k] eth_type_vlan
> 0.16% [kernel] [k] ip_finish_output
> 0.15% [kernel] [k] kmem_cache_free_bulk
> 0.14% [kernel] [k] netif_receive_skb_internal
>
>
>
>
> wondering why this:
> 1.97% [kernel] [k] virt_to_head_page
> is in top...
This is related to the page_frag_free() call, but it is weird that it
shows up because it is suppose to be inlined (it is explicitly marked
inline in include/linux/mm.h).
> >>>>> perf top:
> >>>>>
> >>>>> PerfTop: 77835 irqs/sec kernel:99.7%
> >>>>> ---------------------------------------------
> >>>>>
> >>>>> 16.32% [kernel] [k] skb_dst_force
> >>>>> 16.30% [kernel] [k] dst_release
> >>>>> 15.11% [kernel] [k] rt_cache_valid
> >>>>> 12.62% [kernel] [k] ipv4_mtu
> >>>> It seems a little strange that these 4 functions are on the top
> > I don't see these in my test.
> >
> >>>>
> >>>>> 5.60% [kernel] [k] do_raw_spin_lock
> >>>> Why is calling/taking this lock? (Use perf call-graph recording).
> >>> can be hard to paste it here:)
> >>> attached file
> > The attached was very big. Please don't attach so big file on mailing
> > lists. Next time plase share them via e.g. pastebin. The output was a
> > capture from your terminal, which made the output more difficult to
> > read. Hint: You can/could use perf --stdio and place it in a file
> > instead.
> >
> > The output (extracted below) didn't show who called 'do_raw_spin_lock',
> > BUT it showed another interesting thing. The kernel code
> > __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> > as it will first call skb_dst_force() and then skb_dst_drop() when the
> > packet is transmitted on a VLAN.
> >
> > static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> > {
> > [...]
> > /* If device/qdisc don't need skb->dst, release it right now while
> > * its hot in this cpu cache.
> > */
> > if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> > skb_dst_drop(skb);
> > else
> > skb_dst_force(skb);
> >
> >
> >
> > Extracted part of attached perf output:
> >
> > --5.37%--ip_rcv_finish
> > |
> > |--4.02%--ip_forward
> > | |
> > | --3.92%--ip_forward_finish
> > | |
> > | --3.91%--ip_output
> > | |
> > | --3.90%--ip_finish_output
> > | |
> > | --3.88%--ip_finish_output2
> > | |
> > | --2.77%--neigh_connected_output
> > | |
> > | --2.74%--dev_queue_xmit
> > | |
> > | --2.73%--__dev_queue_xmit
> > | |
> > | |--1.66%--dev_hard_start_xmit
> > | | |
> > | | --1.64%--vlan_dev_hard_start_xmit
> > | | |
> > | | --1.63%--dev_queue_xmit
> > | | |
> > | | --1.62%--__dev_queue_xmit
> > | | |
> > | | |--0.99%--skb_dst_drop.isra.77
> > | | | |
> > | | | --0.99%--dst_release
> > | | |
> > | | --0.55%--sch_direct_xmit
> > | |
> > | --0.99%--skb_dst_force
> > |
> > --1.29%--ip_route_input_noref
> > |
> > --1.29%--ip_route_input_rcu
> > |
> > --1.05%--rt_cache_valid
> >
>
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Paweł Staszewski @ 2017-08-15 9:30 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan
In-Reply-To: <20170815112307.2dd366fe@redhat.com>
W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
> On Tue, 15 Aug 2017 02:38:56 +0200
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
>>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>>
>>>> To show some difference below comparision vlan/no-vlan traffic
>>>>
>>>> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
>>> I'm trying to reproduce in my testlab (with ixgbe). I do see, a
>>> performance reduction of about 10-19% when I forward out a VLAN
>>> interface. This is larger than I expected, but still lower than what
>>> you reported 30-40% slowdown.
>>>
>>> [...]
>> Ok mellanox afrrived (MT27700 - mlnx5 driver)
>> And to compare melannox with vlans and without: 33% performance
>> degradation (less than with ixgbe where i reach ~40% with same settings)
>>
>> Mellanox without TX traffix on vlan:
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;16;64;11089305;709715520;8871553;567779392
>> 1;16;64;11096292;710162688;11095566;710116224
>> 2;16;64;11095770;710129280;11096799;710195136
>> 3;16;64;11097199;710220736;11097702;710252928
>> 4;16;64;11080984;567081856;11079662;709098368
>> 5;16;64;11077696;708972544;11077039;708930496
>> 6;16;64;11082991;709311424;8864802;567347328
>> 7;16;64;11089596;709734144;8870927;709789184
>> 8;16;64;11094043;710018752;11095391;710105024
>>
>> Mellanox with TX traffic on vlan:
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;16;64;7369914;471674496;7370281;471697980
>> 1;16;64;7368896;471609408;7368043;471554752
>> 2;16;64;7367577;471524864;7367759;471536576
>> 3;16;64;7368744;377305344;7369391;471641024
>> 4;16;64;7366824;471476736;7364330;471237120
>> 5;16;64;7368352;471574528;7367239;471503296
>> 6;16;64;7367459;471517376;7367806;471539584
>> 7;16;64;7367190;471500160;7367988;471551232
>> 8;16;64;7368023;471553472;7368076;471556864
> I wonder if the drivers page recycler is active/working or not, and if
> the situation is different between VLAN vs no-vlan (given
> page_frag_free is so high in you perf top). The Mellanox drivers
> fortunately have a stats counter to tell us this explicitly (which the
> ixgbe driver doesn't).
>
> You can use my ethtool_stats.pl script watch these stats:
> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
> (Hint perl dependency: dnf install perl-Time-HiRes)
For RX NIC:
Show adapter(s) (enp175s0f0) statistics (ONLY that changed!)
Ethtool(enp175s0f0) stat: 78380071 ( 78,380,071) <= rx0_bytes /sec
Ethtool(enp175s0f0) stat: 230978 ( 230,978) <=
rx0_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <=
rx0_csum_complete /sec
Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_packets /sec
Ethtool(enp175s0f0) stat: 921614 ( 921,614) <=
rx0_page_reuse /sec
Ethtool(enp175s0f0) stat: 78956591 ( 78,956,591) <= rx1_bytes /sec
Ethtool(enp175s0f0) stat: 233343 ( 233,343) <=
rx1_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <=
rx1_csum_complete /sec
Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_packets /sec
Ethtool(enp175s0f0) stat: 927793 ( 927,793) <=
rx1_page_reuse /sec
Ethtool(enp175s0f0) stat: 79677124 ( 79,677,124) <= rx2_bytes /sec
Ethtool(enp175s0f0) stat: 233735 ( 233,735) <=
rx2_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <=
rx2_csum_complete /sec
Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_packets /sec
Ethtool(enp175s0f0) stat: 937989 ( 937,989) <=
rx2_page_reuse /sec
Ethtool(enp175s0f0) stat: 78392893 ( 78,392,893) <= rx3_bytes /sec
Ethtool(enp175s0f0) stat: 230311 ( 230,311) <=
rx3_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <=
rx3_csum_complete /sec
Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_packets /sec
Ethtool(enp175s0f0) stat: 922513 ( 922,513) <=
rx3_page_reuse /sec
Ethtool(enp175s0f0) stat: 65165583 ( 65,165,583) <= rx4_bytes /sec
Ethtool(enp175s0f0) stat: 191969 ( 191,969) <=
rx4_cache_reuse /sec
Ethtool(enp175s0f0) stat: 958317 ( 958,317) <=
rx4_csum_complete /sec
Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_packets /sec
Ethtool(enp175s0f0) stat: 766332 ( 766,332) <=
rx4_page_reuse /sec
Ethtool(enp175s0f0) stat: 66920721 ( 66,920,721) <= rx5_bytes /sec
Ethtool(enp175s0f0) stat: 197150 ( 197,150) <=
rx5_cache_reuse /sec
Ethtool(enp175s0f0) stat: 984128 ( 984,128) <=
rx5_csum_complete /sec
Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_packets /sec
Ethtool(enp175s0f0) stat: 786978 ( 786,978) <=
rx5_page_reuse /sec
Ethtool(enp175s0f0) stat: 79076984 ( 79,076,984) <= rx6_bytes /sec
Ethtool(enp175s0f0) stat: 233735 ( 233,735) <=
rx6_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <=
rx6_csum_complete /sec
Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_packets /sec
Ethtool(enp175s0f0) stat: 929163 ( 929,163) <=
rx6_page_reuse /sec
Ethtool(enp175s0f0) stat: 78660672 ( 78,660,672) <= rx7_bytes /sec
Ethtool(enp175s0f0) stat: 230413 ( 230,413) <=
rx7_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <=
rx7_csum_complete /sec
Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_packets /sec
Ethtool(enp175s0f0) stat: 926376 ( 926,376) <=
rx7_page_reuse /sec
Ethtool(enp175s0f0) stat: 10674565 ( 10,674,565) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp175s0f0) stat: 605241031 ( 605,241,031) <= rx_bytes /sec
Ethtool(enp175s0f0) stat: 768585608 ( 768,585,608) <= rx_bytes_phy
/sec
Ethtool(enp175s0f0) stat: 1781569 ( 1,781,569) <=
rx_cache_reuse /sec
Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <=
rx_csum_complete /sec
Ethtool(enp175s0f0) stat: 1773785 ( 1,773,785) <=
rx_out_of_buffer /sec
Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_packets /sec
Ethtool(enp175s0f0) stat: 10674799 ( 10,674,799) <=
rx_packets_phy /sec
Ethtool(enp175s0f0) stat: 7118993 ( 7,118,993) <=
rx_page_reuse /sec
Ethtool(enp175s0f0) stat: 768565744 ( 768,565,744) <=
rx_prio0_bytes /sec
Ethtool(enp175s0f0) stat: 10674522 ( 10,674,522) <=
rx_prio0_packets /sec
Ethtool(enp175s0f0) stat: 725871089 ( 725,871,089) <=
rx_vport_unicast_bytes /sec
Ethtool(enp175s0f0) stat: 10674575 ( 10,674,575) <=
rx_vport_unicast_packets /sec
For TX nic with vlan:
Show adapter(s) (enp175s0f1) statistics (ONLY that changed!)
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_bytes_phy
/sec
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_multicast_phy /sec
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_packets_phy /sec
Ethtool(enp175s0f1) stat: 71 ( 71) <=
rx_prio0_bytes /sec
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_prio0_packets /sec
Ethtool(enp175s0f1) stat: 67 ( 67) <=
rx_vport_multicast_bytes /sec
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_vport_multicast_packets /sec
Ethtool(enp175s0f1) stat: 64955114 ( 64,955,114) <= tx0_bytes /sec
Ethtool(enp175s0f1) stat: 955222 ( 955,222) <=
tx0_csum_none /sec
Ethtool(enp175s0f1) stat: 26489 ( 26,489) <= tx0_nop /sec
Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_packets /sec
Ethtool(enp175s0f1) stat: 66799214 ( 66,799,214) <= tx1_bytes /sec
Ethtool(enp175s0f1) stat: 982341 ( 982,341) <=
tx1_csum_none /sec
Ethtool(enp175s0f1) stat: 27225 ( 27,225) <= tx1_nop /sec
Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_packets /sec
Ethtool(enp175s0f1) stat: 78650421 ( 78,650,421) <= tx2_bytes /sec
Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <=
tx2_csum_none /sec
Ethtool(enp175s0f1) stat: 32059 ( 32,059) <= tx2_nop /sec
Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_packets /sec
Ethtool(enp175s0f1) stat: 78186849 ( 78,186,849) <= tx3_bytes /sec
Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <=
tx3_csum_none /sec
Ethtool(enp175s0f1) stat: 31879 ( 31,879) <= tx3_nop /sec
Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_packets /sec
Ethtool(enp175s0f1) stat: 234 ( 234) <=
tx3_xmit_more /sec
Ethtool(enp175s0f1) stat: 78466099 ( 78,466,099) <= tx4_bytes /sec
Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <=
tx4_csum_none /sec
Ethtool(enp175s0f1) stat: 31990 ( 31,990) <= tx4_nop /sec
Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_packets /sec
Ethtool(enp175s0f1) stat: 78765724 ( 78,765,724) <= tx5_bytes /sec
Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <=
tx5_csum_none /sec
Ethtool(enp175s0f1) stat: 32115 ( 32,115) <= tx5_nop /sec
Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_packets /sec
Ethtool(enp175s0f1) stat: 264 ( 264) <=
tx5_xmit_more /sec
Ethtool(enp175s0f1) stat: 79669524 ( 79,669,524) <= tx6_bytes /sec
Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <=
tx6_csum_none /sec
Ethtool(enp175s0f1) stat: 32490 ( 32,490) <= tx6_nop /sec
Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_packets /sec
Ethtool(enp175s0f1) stat: 79389329 ( 79,389,329) <= tx7_bytes /sec
Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <=
tx7_csum_none /sec
Ethtool(enp175s0f1) stat: 32365 ( 32,365) <= tx7_nop /sec
Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_packets /sec
Ethtool(enp175s0f1) stat: 604885175 ( 604,885,175) <= tx_bytes /sec
Ethtool(enp175s0f1) stat: 676059749 ( 676,059,749) <= tx_bytes_phy
/sec
Ethtool(enp175s0f1) stat: 8895370 ( 8,895,370) <= tx_packets /sec
Ethtool(enp175s0f1) stat: 8895522 ( 8,895,522) <=
tx_packets_phy /sec
Ethtool(enp175s0f1) stat: 676063067 ( 676,063,067) <=
tx_prio0_bytes /sec
Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <=
tx_prio0_packets /sec
Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <=
tx_vport_unicast_bytes /sec
Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <=
tx_vport_unicast_packets /sec
Ethtool(enp175s0f1) stat: 498 ( 498) <= tx_xmit_more
/sec
>
>> ethtool settings for both tests:
>> ifc='enp175s0f0 enp175s0f1'
>> for i in $ifc
>> do
>> ip link set up dev $i
>> ethtool -A $i autoneg off rx off tx off
>> ethtool -G $i rx 128 tx 256
> The ring queue size recommendations, might be different for the mlx5
> driver (Cc'ing Mellanox maintainers).
>
>
>> ip link set $i txqueuelen 1000
>> ethtool -C $i rx-usecs 25
>> ethtool -L $i combined 16
>> ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
>> tx-nocache-copy off ntuple on
>> ethtool -N $i rx-flow-hash udp4 sdfn
>> done
> Thanks for being explicit about what you setup is :-)
>
>> and perf top:
>> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>> cycles], (all, 56 CPUs)
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> 14.25% [kernel] [k] dst_release
>> 14.17% [kernel] [k] skb_dst_force
>> 13.41% [kernel] [k] rt_cache_valid
>> 11.47% [kernel] [k] ip_finish_output2
>> 7.01% [kernel] [k] do_raw_spin_lock
>> 5.07% [kernel] [k] page_frag_free
>> 3.47% [mlx5_core] [k] mlx5e_xmit
>> 2.88% [kernel] [k] fib_table_lookup
>> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
>> 1.97% [kernel] [k] virt_to_head_page
>> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
>> 0.93% [kernel] [k] __dev_queue_xmit
>> 0.87% [kernel] [k] __build_skb
>> 0.84% [kernel] [k] ipt_do_table
>> 0.79% [kernel] [k] ip_rcv
>> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
>> 0.78% [kernel] [k] netif_skb_features
>> 0.73% [kernel] [k] __netif_receive_skb_core
>> 0.52% [kernel] [k] dev_hard_start_xmit
>> 0.52% [kernel] [k] build_skb
>> 0.51% [kernel] [k] ip_route_input_rcu
>> 0.50% [kernel] [k] skb_unref
>> 0.49% [kernel] [k] ip_forward
>> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
>> 0.44% [kernel] [k] udp_v4_early_demux
>> 0.41% [kernel] [k] napi_consume_skb
>> 0.40% [kernel] [k] __local_bh_enable_ip
>> 0.39% [kernel] [k] ip_rcv_finish
>> 0.39% [kernel] [k] kmem_cache_alloc
>> 0.38% [kernel] [k] sch_direct_xmit
>> 0.33% [kernel] [k] validate_xmit_skb
>> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
>> 0.29% [kernel] [k] netdev_pick_tx
>> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
>> 0.27% [kernel] [k] deliver_ptype_list_skb
>> 0.26% [kernel] [k] fib_validate_source
>> 0.26% [mlx5_core] [k] mlx5e_napi_poll
>> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
>> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
>> 0.25% [kernel] [k] eth_header
>> 0.23% [kernel] [k] skb_network_protocol
>> 0.20% [kernel] [k] nf_hook_slow
>> 0.20% [kernel] [k] vlan_passthru_hard_header
>> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
>> 0.19% [kernel] [k] swiotlb_map_page
>> 0.18% [kernel] [k] compound_head
>> 0.18% [kernel] [k] neigh_connected_output
>> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
>> 0.18% [kernel] [k] ip_output
>> 0.17% [kernel] [k] prefetch_freepointer.isra.70
>> 0.17% [kernel] [k] __slab_free
>> 0.16% [kernel] [k] eth_type_vlan
>> 0.16% [kernel] [k] ip_finish_output
>> 0.15% [kernel] [k] kmem_cache_free_bulk
>> 0.14% [kernel] [k] netif_receive_skb_internal
>>
>>
>>
>>
>> wondering why this:
>> 1.97% [kernel] [k] virt_to_head_page
>> is in top...
> This is related to the page_frag_free() call, but it is weird that it
> shows up because it is suppose to be inlined (it is explicitly marked
> inline in include/linux/mm.h).
>
>
>>>>>>> perf top:
>>>>>>>
>>>>>>> PerfTop: 77835 irqs/sec kernel:99.7%
>>>>>>> ---------------------------------------------
>>>>>>>
>>>>>>> 16.32% [kernel] [k] skb_dst_force
>>>>>>> 16.30% [kernel] [k] dst_release
>>>>>>> 15.11% [kernel] [k] rt_cache_valid
>>>>>>> 12.62% [kernel] [k] ipv4_mtu
>>>>>> It seems a little strange that these 4 functions are on the top
>>> I don't see these in my test.
>>>
>>>>>>
>>>>>>> 5.60% [kernel] [k] do_raw_spin_lock
>>>>>> Why is calling/taking this lock? (Use perf call-graph recording).
>>>>> can be hard to paste it here:)
>>>>> attached file
>>> The attached was very big. Please don't attach so big file on mailing
>>> lists. Next time plase share them via e.g. pastebin. The output was a
>>> capture from your terminal, which made the output more difficult to
>>> read. Hint: You can/could use perf --stdio and place it in a file
>>> instead.
>>>
>>> The output (extracted below) didn't show who called 'do_raw_spin_lock',
>>> BUT it showed another interesting thing. The kernel code
>>> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
>>> as it will first call skb_dst_force() and then skb_dst_drop() when the
>>> packet is transmitted on a VLAN.
>>>
>>> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
>>> {
>>> [...]
>>> /* If device/qdisc don't need skb->dst, release it right now while
>>> * its hot in this cpu cache.
>>> */
>>> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
>>> skb_dst_drop(skb);
>>> else
>>> skb_dst_force(skb);
>>>
>>>
>>>
>>> Extracted part of attached perf output:
>>>
>>> --5.37%--ip_rcv_finish
>>> |
>>> |--4.02%--ip_forward
>>> | |
>>> | --3.92%--ip_forward_finish
>>> | |
>>> | --3.91%--ip_output
>>> | |
>>> | --3.90%--ip_finish_output
>>> | |
>>> | --3.88%--ip_finish_output2
>>> | |
>>> | --2.77%--neigh_connected_output
>>> | |
>>> | --2.74%--dev_queue_xmit
>>> | |
>>> | --2.73%--__dev_queue_xmit
>>> | |
>>> | |--1.66%--dev_hard_start_xmit
>>> | | |
>>> | | --1.64%--vlan_dev_hard_start_xmit
>>> | | |
>>> | | --1.63%--dev_queue_xmit
>>> | | |
>>> | | --1.62%--__dev_queue_xmit
>>> | | |
>>> | | |--0.99%--skb_dst_drop.isra.77
>>> | | | |
>>> | | | --0.99%--dst_release
>>> | | |
>>> | | --0.55%--sch_direct_xmit
>>> | |
>>> | --0.99%--skb_dst_force
>>> |
>>> --1.29%--ip_route_input_noref
>>> |
>>> --1.29%--ip_route_input_rcu
>>> |
>>> --1.05%--rt_cache_valid
>>>
>
>
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Jesper Dangaard Brouer @ 2017-08-15 9:35 UTC (permalink / raw)
To: Paolo Abeni
Cc: Paweł Staszewski, Linux Kernel Network Developers,
Alexander Duyck, Eric Dumazet, brouer
In-Reply-To: <1502729870.8411.63.camel@redhat.com>
On Mon, 14 Aug 2017 18:57:50 +0200
Paolo Abeni <pabeni@redhat.com> wrote:
> On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
> > The output (extracted below) didn't show who called 'do_raw_spin_lock',
> > BUT it showed another interesting thing. The kernel code
> > __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> > as it will first call skb_dst_force() and then skb_dst_drop() when the
> > packet is transmitted on a VLAN.
> >
> > static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> > {
> > [...]
> > /* If device/qdisc don't need skb->dst, release it right now while
> > * its hot in this cpu cache.
> > */
> > if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> > skb_dst_drop(skb);
> > else
> > skb_dst_force(skb);
>
> I think that the high impact of the above code in this specific test is
> mostly due to the following:
>
> - ingress packets with different RSS rx hash lands on different CPUs
> - but they use the same dst entry, since the destination IPs belong to
> the same subnet
> - the dst refcnt cacheline is contented between all the CPUs
Good point and explanation Paolo :-)
I changed my pktgen setup to be closer to Pawel's to provoke this
situation some more, and I get closer to provoke this although not as
clearly as Pawel.
A perf diff does show, that the overhead in the VLAN cause originates
from the routing "dst_release" code. Diff Baseline==non-vlan case.
[jbrouer@canyon ~]$ sudo ~/perf diff
# Event 'cycles'
#
# Baseline Delta Abs Shared Object Symbol
# ........ ......... ................ .........................................
#
3.23% +4.32% [kernel.vmlinux] [k] __dev_queue_xmit
+3.43% [kernel.vmlinux] [k] dst_release
13.54% -3.17% [kernel.vmlinux] [k] fib_table_lookup
9.33% -2.73% [kernel.vmlinux] [k] _raw_spin_lock
7.91% -1.75% [ixgbe] [k] ixgbe_poll
+1.64% [8021q] [k] vlan_dev_hard_start_xmit
7.23% -1.26% [ixgbe] [k] ixgbe_xmit_frame_ring
3.34% -1.10% [kernel.vmlinux] [k] eth_type_trans
5.20% +0.97% [kernel.vmlinux] [k] ip_route_input_rcu
1.13% +0.95% [kernel.vmlinux] [k] ip_rcv_finish
2.49% -0.82% [kernel.vmlinux] [k] ip_forward
3.05% -0.80% [kernel.vmlinux] [k] __build_skb
0.44% +0.74% [kernel.vmlinux] [k] __netif_receive_skb
+0.71% [kernel.vmlinux] [k] neigh_connected_output
1.70% +0.68% [kernel.vmlinux] [k] validate_xmit_skb
1.42% +0.67% [kernel.vmlinux] [k] dev_hard_start_xmit
0.49% +0.66% [kernel.vmlinux] [k] netif_receive_skb_internal
+0.62% [kernel.vmlinux] [k] eth_header
+0.57% [ixgbe] [k] ixgbe_tx_ctxtdesc
1.19% -0.55% [kernel.vmlinux] [k] __netdev_pick_tx
2.54% -0.48% [kernel.vmlinux] [k] fib_validate_source
2.83% +0.46% [kernel.vmlinux] [k] ip_finish_output2
1.45% +0.45% [kernel.vmlinux] [k] netif_skb_features
1.66% -0.45% [kernel.vmlinux] [k] napi_gro_receive
0.90% -0.40% [kernel.vmlinux] [k] validate_xmit_skb_list
1.45% -0.39% [kernel.vmlinux] [k] ip_finish_output
+0.36% [8021q] [k] vlan_passthru_hard_header
1.28% -0.33% [kernel.vmlinux] [k] netdev_pick_tx
> Perhaps we can inprove the situation setting the IFF_XMIT_DST_RELEASE
> flag for vlan if the underlaying device does not have (relevant)
> classifier attached? (and clearing it as needed)
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* Re: [PATCH 1/2] mpls: add handlers
From: David Lamparter @ 2017-08-15 9:37 UTC (permalink / raw)
To: Roopa Prabhu; +Cc: Amine Kherbouche, David Lamparter, netdev@vger.kernel.org
In-Reply-To: <CAJieiUijsyWjzNL=oFaGxtuos367v0VrpSaEqwHvVacrk5hcTA@mail.gmail.com>
On Sat, Aug 12, 2017 at 08:29:18PM -0700, Roopa Prabhu wrote:
> On Sat, Aug 12, 2017 at 6:35 AM, Amine Kherbouche
> <amine.kherbouche@6wind.com> wrote:
> >
> >
> > On 11/08/2017 16:37, Roopa Prabhu wrote:
> >>
> >> On Fri, Aug 11, 2017 at 5:34 AM, David Lamparter <equinox@diac24.net>
> >> wrote:
> >>>
> >>> On Thu, Aug 10, 2017 at 10:28:36PM +0200, Amine Kherbouche wrote:
> >>>>
> >>>> Mpls handler allows creation/deletion of mpls routes without using
> >>>> rtnetlink. When an incoming mpls packet matches this route, the saved
> >>>> function handler is called.
> >>>
> >>> Since I originally authored this patch, I have come to believe that it
> >>> might be unneccessarily complicated. It is unlikely that a lot of
> >>> different "handlers" will exist here; the only things I can think of
> >>> are VPLS support and BIER-MPLS multicast replication. I'm not saying
> >>> it's a bad idea, but, well, this was in the README that I gave to 6WIND
> >>> with this code:
> >>>
> >>> ...
> >>
> >> yes, I would also prefer just exporting the functions and calling
> >> them directly instead of adding a
> >> handler layer. We can move to that later if it becomes necessary.
> >
> > I understand that the handler layer is an overhead (as said by David's
> > note), and I agree with the solution for exporting the mpls functions that
> > allows route creation/deletion, but how about forwarding the right mpls
> > packet to the right vpls device with the device ptr? I don't see
> > another way.
>
>
> hmm...ok, so you are adding a mpls route to get into vpls_rcv and you
> want this route to carry the vpls_rcv information. Ideally if you knew
> the route is pointing to a vpls device kind, you can directly call
> vpls_rcv.
> But, am not sure if a route is necessary here either.
>
> It just seems like the vpls device information is duplicated in the
> mpls route per vpls dev. Wondering if we can skip the route part and
> always do a lookup on vpls-id/label in mpls_rcv to send it to a
> vpls_rcv if there is a match. This will be the l2 handler for mpls.
I think the reverse is the better option, removing the vpls device
information and just going with the route table. My approach to this
would be to add a new netlink route attribute "RTA_VPLS" which
identifies the vpls device, is stored in the route table, and provides
the device ptr needed here.
(The control word config should also be on the route.)
My reason for thinking this is that the VPLS code needs exactly the same
information as does a normal MPLS route: it attaches to an incoming
label (decapsulating packets instead of forwarding them), and for TX it
does the same operation of looking up a nexthop (possibly with ECMP
support) and adding a label stack. The code should, in fact, probably
reuse the TX path.
This also fits both an 1:1 and 1:n model pretty well. Creating a VPLS
head-end netdevice doesn't even need any config. It'd just work like:
- ip link add name vpls123 type vpls
- ip -f mpls route add \
1234 \ # incoming label for decap
vpls vpls123 \ # new attr: VPLS device
as 2345 via inet 10.0.0.1 dev eth0 # outgoing label for encap
For a 1:n model, one would simply add multiple routes on the same vpls
device.
-David
^ permalink raw reply
* Re: [PATCH] netfilter: fix indent on in statements
From: Sergei Shtylyov @ 2017-08-15 9:45 UTC (permalink / raw)
To: Colin King, Pablo Neira Ayuso, Jozsef Kadlecsik, Florian Westphal,
Stephen Hemminger, David S . Miller, netfilter-devel, coreteam,
bridge, netdev
Cc: kernel-janitors, linux-kernel
In-Reply-To: <20170815065025.18471-1-colin.king@canonical.com>
Hello!
On 8/15/2017 9:50 AM, Colin King wrote:
> From: Colin Ian King <colin.king@canonical.com>
>
> The returns on some if statements are not indented correctly,
s/in/if/ in the subject?
> add in the missing tab.
>
> Signed-off-by: Colin Ian King <colin.king@canonical.com>
[...]
MBR, Sergei
^ permalink raw reply
* Re: general protection fault in fib_dump_info
From: Eric Dumazet @ 2017-08-15 9:49 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: idaifish, David Miller, Alexey Kuznetsov, Hideaki YOSHIFUJI,
netdev, syzkaller
In-Reply-To: <CACT4Y+bnZxac87Bz6VYLhAzpn5JG-bmm7kzPefoYzDP7z9rftw@mail.gmail.com>
On Tue, 2017-08-15 at 08:51 +0200, Dmitry Vyukov wrote:
> Eric, what's the "David Miller net tree"? Is it
> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git? I
> don't see 2c87d63ac853550e734edfd45e1be5e5aa44fbcc there.
> https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git is
> what we are testing and the last commit on which we saw this bug is:
>
> commit cb44a8606f063099991242ada4b8fa2fb26769b3
> Merge: 54161ed4eede a656d34a6e5a
> Author: David S. Miller
> Date: Mon Aug 14 11:18:16 2017 -0700
> Merge branch 'mlnx-i2c'
>
Extracted from Documentation/networking/netdev-FAQ.txt
A: There are always two trees (git repositories) in play. Both are driven
by David Miller, the main network maintainer. There is the "net" tree,
and the "net-next" tree. As you can probably guess from the names, the
net tree is for fixes to existing code already in the mainline tree from
Linus, and net-next is where the new code goes for the future release.
You can find the trees here:
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git
Hope this helps.
^ permalink raw reply
* [PATCH][V2] netfilter: fix indent on if statements
From: Colin King @ 2017-08-15 9:50 UTC (permalink / raw)
To: Pablo Neira Ayuso, Jozsef Kadlecsik, Florian Westphal,
Stephen Hemminger, David S . Miller, netfilter-devel, coreteam,
bridge, netdev
Cc: kernel-janitors, linux-kernel
From: Colin Ian King <colin.king@canonical.com>
The returns on some if statements are not indented correctly,
add in the missing tab.
Signed-off-by: Colin Ian King <colin.king@canonical.com>
---
net/bridge/netfilter/ebt_ip.c | 4 ++--
net/bridge/netfilter/ebt_ip6.c | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/net/bridge/netfilter/ebt_ip.c b/net/bridge/netfilter/ebt_ip.c
index d06968bdf5ec..2b46c50abce0 100644
--- a/net/bridge/netfilter/ebt_ip.c
+++ b/net/bridge/netfilter/ebt_ip.c
@@ -64,14 +64,14 @@ ebt_ip_mt(const struct sk_buff *skb, struct xt_action_param *par)
if (NF_INVF(info, EBT_IP_DPORT,
dst < info->dport[0] ||
dst > info->dport[1]))
- return false;
+ return false;
}
if (info->bitmask & EBT_IP_SPORT) {
u32 src = ntohs(pptr->src);
if (NF_INVF(info, EBT_IP_SPORT,
src < info->sport[0] ||
src > info->sport[1]))
- return false;
+ return false;
}
}
return true;
diff --git a/net/bridge/netfilter/ebt_ip6.c b/net/bridge/netfilter/ebt_ip6.c
index 4617491be41e..2a5a52a53ec4 100644
--- a/net/bridge/netfilter/ebt_ip6.c
+++ b/net/bridge/netfilter/ebt_ip6.c
@@ -89,7 +89,7 @@ ebt_ip6_mt(const struct sk_buff *skb, struct xt_action_param *par)
if (NF_INVF(info, EBT_IP6_SPORT,
src < info->sport[0] ||
src > info->sport[1]))
- return false;
+ return false;
}
if ((info->bitmask & EBT_IP6_ICMP6) &&
NF_INVF(info, EBT_IP6_ICMP6,
--
2.11.0
^ permalink raw reply related
* Re: [PATCH] netfilter: fix indent on in statements
From: Colin Ian King @ 2017-08-15 9:51 UTC (permalink / raw)
To: Sergei Shtylyov, Pablo Neira Ayuso, Jozsef Kadlecsik,
Florian Westphal, Stephen Hemminger, David S . Miller,
netfilter-devel, coreteam, bridge, netdev
Cc: kernel-janitors, linux-kernel
In-Reply-To: <6b38b83c-21b3-5cec-6145-cdbb54a63dd2@cogentembedded.com>
On 15/08/17 10:45, Sergei Shtylyov wrote:
> Hello!
>
> On 8/15/2017 9:50 AM, Colin King wrote:
>
>> From: Colin Ian King <colin.king@canonical.com>
>>
>> The returns on some if statements are not indented correctly,
>
> s/in/if/ in the subject?
Doh, fix resent.
>
>> add in the missing tab.
>>
>> Signed-off-by: Colin Ian King <colin.king@canonical.com>
> [...]
>
> MBR, Sergei
> --
> To unsubscribe from this list: send the line "unsubscribe
> kernel-janitors" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Jesper Dangaard Brouer @ 2017-08-15 9:57 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan, brouer
In-Reply-To: <271f731c-080f-8ccb-cd08-f0ec1dfb0c51@itcare.pl>
On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
> > On Tue, 15 Aug 2017 02:38:56 +0200
> > Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> >
> >> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
> >>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> >>>
> >>>> To show some difference below comparision vlan/no-vlan traffic
> >>>>
> >>>> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
> >>> I'm trying to reproduce in my testlab (with ixgbe). I do see, a
> >>> performance reduction of about 10-19% when I forward out a VLAN
> >>> interface. This is larger than I expected, but still lower than what
> >>> you reported 30-40% slowdown.
> >>>
> >>> [...]
> >> Ok mellanox afrrived (MT27700 - mlnx5 driver)
> >> And to compare melannox with vlans and without: 33% performance
> >> degradation (less than with ixgbe where i reach ~40% with same settings)
> >>
> >> Mellanox without TX traffix on vlan:
> >> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> >> 0;16;64;11089305;709715520;8871553;567779392
> >> 1;16;64;11096292;710162688;11095566;710116224
> >> 2;16;64;11095770;710129280;11096799;710195136
> >> 3;16;64;11097199;710220736;11097702;710252928
> >> 4;16;64;11080984;567081856;11079662;709098368
> >> 5;16;64;11077696;708972544;11077039;708930496
> >> 6;16;64;11082991;709311424;8864802;567347328
> >> 7;16;64;11089596;709734144;8870927;709789184
> >> 8;16;64;11094043;710018752;11095391;710105024
> >>
> >> Mellanox with TX traffic on vlan:
> >> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> >> 0;16;64;7369914;471674496;7370281;471697980
> >> 1;16;64;7368896;471609408;7368043;471554752
> >> 2;16;64;7367577;471524864;7367759;471536576
> >> 3;16;64;7368744;377305344;7369391;471641024
> >> 4;16;64;7366824;471476736;7364330;471237120
> >> 5;16;64;7368352;471574528;7367239;471503296
> >> 6;16;64;7367459;471517376;7367806;471539584
> >> 7;16;64;7367190;471500160;7367988;471551232
> >> 8;16;64;7368023;471553472;7368076;471556864
> > I wonder if the drivers page recycler is active/working or not, and if
> > the situation is different between VLAN vs no-vlan (given
> > page_frag_free is so high in you perf top). The Mellanox drivers
> > fortunately have a stats counter to tell us this explicitly (which the
> > ixgbe driver doesn't).
> >
> > You can use my ethtool_stats.pl script watch these stats:
> > https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
> > (Hint perl dependency: dnf install perl-Time-HiRes)
> For RX NIC:
> Show adapter(s) (enp175s0f0) statistics (ONLY that changed!)
> Ethtool(enp175s0f0) stat: 78380071 ( 78,380,071) <= rx0_bytes /sec
> Ethtool(enp175s0f0) stat: 230978 ( 230,978) <= rx0_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_packets /sec
> Ethtool(enp175s0f0) stat: 921614 ( 921,614) <= rx0_page_reuse /sec
> Ethtool(enp175s0f0) stat: 78956591 ( 78,956,591) <= rx1_bytes /sec
> Ethtool(enp175s0f0) stat: 233343 ( 233,343) <= rx1_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_packets /sec
> Ethtool(enp175s0f0) stat: 927793 ( 927,793) <= rx1_page_reuse /sec
> Ethtool(enp175s0f0) stat: 79677124 ( 79,677,124) <= rx2_bytes /sec
> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <= rx2_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_packets /sec
> Ethtool(enp175s0f0) stat: 937989 ( 937,989) <= rx2_page_reuse /sec
> Ethtool(enp175s0f0) stat: 78392893 ( 78,392,893) <= rx3_bytes /sec
> Ethtool(enp175s0f0) stat: 230311 ( 230,311) <= rx3_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_packets /sec
> Ethtool(enp175s0f0) stat: 922513 ( 922,513) <= rx3_page_reuse /sec
> Ethtool(enp175s0f0) stat: 65165583 ( 65,165,583) <= rx4_bytes /sec
> Ethtool(enp175s0f0) stat: 191969 ( 191,969) <= rx4_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_csum_complete /sec
> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_packets /sec
> Ethtool(enp175s0f0) stat: 766332 ( 766,332) <= rx4_page_reuse /sec
> Ethtool(enp175s0f0) stat: 66920721 ( 66,920,721) <= rx5_bytes /sec
> Ethtool(enp175s0f0) stat: 197150 ( 197,150) <= rx5_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_csum_complete /sec
> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_packets /sec
> Ethtool(enp175s0f0) stat: 786978 ( 786,978) <= rx5_page_reuse /sec
> Ethtool(enp175s0f0) stat: 79076984 ( 79,076,984) <= rx6_bytes /sec
> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <= rx6_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_packets /sec
> Ethtool(enp175s0f0) stat: 929163 ( 929,163) <= rx6_page_reuse /sec
> Ethtool(enp175s0f0) stat: 78660672 ( 78,660,672) <= rx7_bytes /sec
> Ethtool(enp175s0f0) stat: 230413 ( 230,413) <= rx7_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_packets /sec
> Ethtool(enp175s0f0) stat: 926376 ( 926,376) <= rx7_page_reuse /sec
> Ethtool(enp175s0f0) stat: 10674565 ( 10,674,565) <= rx_65_to_127_bytes_phy /sec
> Ethtool(enp175s0f0) stat: 605241031 ( 605,241,031) <= rx_bytes /sec
> Ethtool(enp175s0f0) stat: 768585608 ( 768,585,608) <= rx_bytes_phy /sec
> Ethtool(enp175s0f0) stat: 1781569 ( 1,781,569) <= rx_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1773785 ( 1,773,785) <= rx_out_of_buffer /sec
> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_packets /sec
> Ethtool(enp175s0f0) stat: 10674799 ( 10,674,799) <= rx_packets_phy /sec
> Ethtool(enp175s0f0) stat: 7118993 ( 7,118,993) <= rx_page_reuse /sec
> Ethtool(enp175s0f0) stat: 768565744 ( 768,565,744) <= rx_prio0_bytes /sec
> Ethtool(enp175s0f0) stat: 10674522 ( 10,674,522) <= rx_prio0_packets /sec
> Ethtool(enp175s0f0) stat: 725871089 ( 725,871,089) <= rx_vport_unicast_bytes /sec
> Ethtool(enp175s0f0) stat: 10674575 ( 10,674,575) <= rx_vport_unicast_packets /sec
It looks like the mlx5 page recycle mechanism works:
230413 ( 230,413) <= rx7_cache_reuse /sec
+ 926376 ( 926,376) <= rx7_page_reuse /sec
=1156789 (230413+926376)
-1156775 ( 1,156,775) <= rx7_packets /sec
= 14
You can also determine this as there are no counters for:
rx_cache_full or
rx_cache_empty or
rx1_cache_empty
rx1_cache_busy
> For TX nic with vlan:
> Show adapter(s) (enp175s0f1) statistics (ONLY that changed!)
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_65_to_127_bytes_phy /sec
> Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_bytes_phy /sec
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_multicast_phy /sec
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_packets_phy /sec
> Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_prio0_bytes /sec
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_prio0_packets /sec
> Ethtool(enp175s0f1) stat: 67 ( 67) <= rx_vport_multicast_bytes /sec
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_vport_multicast_packets /sec
> Ethtool(enp175s0f1) stat: 64955114 ( 64,955,114) <= tx0_bytes /sec
> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_csum_none /sec
> Ethtool(enp175s0f1) stat: 26489 ( 26,489) <= tx0_nop /sec
> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_packets /sec
> Ethtool(enp175s0f1) stat: 66799214 ( 66,799,214) <= tx1_bytes /sec
> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_csum_none /sec
> Ethtool(enp175s0f1) stat: 27225 ( 27,225) <= tx1_nop /sec
> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_packets /sec
> Ethtool(enp175s0f1) stat: 78650421 ( 78,650,421) <= tx2_bytes /sec
> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_csum_none /sec
> Ethtool(enp175s0f1) stat: 32059 ( 32,059) <= tx2_nop /sec
> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_packets /sec
> Ethtool(enp175s0f1) stat: 78186849 ( 78,186,849) <= tx3_bytes /sec
> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_csum_none /sec
> Ethtool(enp175s0f1) stat: 31879 ( 31,879) <= tx3_nop /sec
> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_packets /sec
> Ethtool(enp175s0f1) stat: 234 ( 234) <= tx3_xmit_more /sec
> Ethtool(enp175s0f1) stat: 78466099 ( 78,466,099) <= tx4_bytes /sec
> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_csum_none /sec
> Ethtool(enp175s0f1) stat: 31990 ( 31,990) <= tx4_nop /sec
> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_packets /sec
> Ethtool(enp175s0f1) stat: 78765724 ( 78,765,724) <= tx5_bytes /sec
> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_csum_none /sec
> Ethtool(enp175s0f1) stat: 32115 ( 32,115) <= tx5_nop /sec
> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_packets /sec
> Ethtool(enp175s0f1) stat: 264 ( 264) <= tx5_xmit_more /sec
> Ethtool(enp175s0f1) stat: 79669524 ( 79,669,524) <= tx6_bytes /sec
> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_csum_none /sec
> Ethtool(enp175s0f1) stat: 32490 ( 32,490) <= tx6_nop /sec
> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_packets /sec
> Ethtool(enp175s0f1) stat: 79389329 ( 79,389,329) <= tx7_bytes /sec
> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_csum_none /sec
> Ethtool(enp175s0f1) stat: 32365 ( 32,365) <= tx7_nop /sec
> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_packets /sec
> Ethtool(enp175s0f1) stat: 604885175 ( 604,885,175) <= tx_bytes /sec
> Ethtool(enp175s0f1) stat: 676059749 ( 676,059,749) <= tx_bytes_phy /sec
> Ethtool(enp175s0f1) stat: 8895370 ( 8,895,370) <= tx_packets /sec
> Ethtool(enp175s0f1) stat: 8895522 ( 8,895,522) <= tx_packets_phy /sec
> Ethtool(enp175s0f1) stat: 676063067 ( 676,063,067) <= tx_prio0_bytes /sec
> Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <= tx_prio0_packets /sec
> Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <= tx_vport_unicast_bytes /sec
> Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <= tx_vport_unicast_packets /sec
> Ethtool(enp175s0f1) stat: 498 ( 498) <= tx_xmit_more /sec
We are seeing some xmit_more, this is interesting. Have you noticed,
if (in the VLAN case) there is a queue in the qdisc layer?
Simply inspect with: tc -s qdisc show dev ixgbe2
> >
> >> ethtool settings for both tests:
> >> ifc='enp175s0f0 enp175s0f1'
> >> for i in $ifc
> >> do
> >> ip link set up dev $i
> >> ethtool -A $i autoneg off rx off tx off
> >> ethtool -G $i rx 128 tx 256
> > The ring queue size recommendations, might be different for the mlx5
> > driver (Cc'ing Mellanox maintainers).
> >
> >
> >> ip link set $i txqueuelen 1000
> >> ethtool -C $i rx-usecs 25
> >> ethtool -L $i combined 16
> >> ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
> >> tx-nocache-copy off ntuple on
> >> ethtool -N $i rx-flow-hash udp4 sdfn
> >> done
> > Thanks for being explicit about what you setup is :-)
> >
> >> and perf top:
> >> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
> >> cycles], (all, 56 CPUs)
> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >>
> >> 14.25% [kernel] [k] dst_release
> >> 14.17% [kernel] [k] skb_dst_force
> >> 13.41% [kernel] [k] rt_cache_valid
> >> 11.47% [kernel] [k] ip_finish_output2
> >> 7.01% [kernel] [k] do_raw_spin_lock
> >> 5.07% [kernel] [k] page_frag_free
> >> 3.47% [mlx5_core] [k] mlx5e_xmit
> >> 2.88% [kernel] [k] fib_table_lookup
> >> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
> >> 1.97% [kernel] [k] virt_to_head_page
> >> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
> >> 0.93% [kernel] [k] __dev_queue_xmit
> >> 0.87% [kernel] [k] __build_skb
> >> 0.84% [kernel] [k] ipt_do_table
> >> 0.79% [kernel] [k] ip_rcv
> >> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
> >> 0.78% [kernel] [k] netif_skb_features
> >> 0.73% [kernel] [k] __netif_receive_skb_core
> >> 0.52% [kernel] [k] dev_hard_start_xmit
> >> 0.52% [kernel] [k] build_skb
> >> 0.51% [kernel] [k] ip_route_input_rcu
> >> 0.50% [kernel] [k] skb_unref
> >> 0.49% [kernel] [k] ip_forward
> >> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
> >> 0.44% [kernel] [k] udp_v4_early_demux
> >> 0.41% [kernel] [k] napi_consume_skb
> >> 0.40% [kernel] [k] __local_bh_enable_ip
> >> 0.39% [kernel] [k] ip_rcv_finish
> >> 0.39% [kernel] [k] kmem_cache_alloc
> >> 0.38% [kernel] [k] sch_direct_xmit
> >> 0.33% [kernel] [k] validate_xmit_skb
> >> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
> >> 0.29% [kernel] [k] netdev_pick_tx
> >> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
> >> 0.27% [kernel] [k] deliver_ptype_list_skb
> >> 0.26% [kernel] [k] fib_validate_source
> >> 0.26% [mlx5_core] [k] mlx5e_napi_poll
> >> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
> >> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
> >> 0.25% [kernel] [k] eth_header
> >> 0.23% [kernel] [k] skb_network_protocol
> >> 0.20% [kernel] [k] nf_hook_slow
> >> 0.20% [kernel] [k] vlan_passthru_hard_header
> >> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
> >> 0.19% [kernel] [k] swiotlb_map_page
> >> 0.18% [kernel] [k] compound_head
> >> 0.18% [kernel] [k] neigh_connected_output
> >> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
> >> 0.18% [kernel] [k] ip_output
> >> 0.17% [kernel] [k] prefetch_freepointer.isra.70
> >> 0.17% [kernel] [k] __slab_free
> >> 0.16% [kernel] [k] eth_type_vlan
> >> 0.16% [kernel] [k] ip_finish_output
> >> 0.15% [kernel] [k] kmem_cache_free_bulk
> >> 0.14% [kernel] [k] netif_receive_skb_internal
> >>
> >>
> >>
> >>
> >> wondering why this:
> >> 1.97% [kernel] [k] virt_to_head_page
> >> is in top...
> > This is related to the page_frag_free() call, but it is weird that it
> > shows up because it is suppose to be inlined (it is explicitly marked
> > inline in include/linux/mm.h).
> >
> >
> >>>>>>> perf top:
> >>>>>>>
> >>>>>>> PerfTop: 77835 irqs/sec kernel:99.7%
> >>>>>>> ---------------------------------------------
> >>>>>>>
> >>>>>>> 16.32% [kernel] [k] skb_dst_force
> >>>>>>> 16.30% [kernel] [k] dst_release
> >>>>>>> 15.11% [kernel] [k] rt_cache_valid
> >>>>>>> 12.62% [kernel] [k] ipv4_mtu
> >>>>>> It seems a little strange that these 4 functions are on the top
> >>> I don't see these in my test.
> >>>
> >>>>>>
> >>>>>>> 5.60% [kernel] [k] do_raw_spin_lock
> >>>>>> Why is calling/taking this lock? (Use perf call-graph recording).
> >>>>> can be hard to paste it here:)
> >>>>> attached file
> >>> The attached was very big. Please don't attach so big file on mailing
> >>> lists. Next time plase share them via e.g. pastebin. The output was a
> >>> capture from your terminal, which made the output more difficult to
> >>> read. Hint: You can/could use perf --stdio and place it in a file
> >>> instead.
> >>>
> >>> The output (extracted below) didn't show who called 'do_raw_spin_lock',
> >>> BUT it showed another interesting thing. The kernel code
> >>> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> >>> as it will first call skb_dst_force() and then skb_dst_drop() when the
> >>> packet is transmitted on a VLAN.
> >>>
> >>> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> >>> {
> >>> [...]
> >>> /* If device/qdisc don't need skb->dst, release it right now while
> >>> * its hot in this cpu cache.
> >>> */
> >>> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> >>> skb_dst_drop(skb);
> >>> else
> >>> skb_dst_force(skb);
> >>>
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Paweł Staszewski @ 2017-08-15 10:02 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan
In-Reply-To: <20170815115756.7ef50c23@redhat.com>
W dniu 2017-08-15 o 11:57, Jesper Dangaard Brouer pisze:
> On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
>>> On Tue, 15 Aug 2017 02:38:56 +0200
>>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>>
>>>> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
>>>>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>>>>
>>>>>> To show some difference below comparision vlan/no-vlan traffic
>>>>>>
>>>>>> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
>>>>> I'm trying to reproduce in my testlab (with ixgbe). I do see, a
>>>>> performance reduction of about 10-19% when I forward out a VLAN
>>>>> interface. This is larger than I expected, but still lower than what
>>>>> you reported 30-40% slowdown.
>>>>>
>>>>> [...]
>>>> Ok mellanox afrrived (MT27700 - mlnx5 driver)
>>>> And to compare melannox with vlans and without: 33% performance
>>>> degradation (less than with ixgbe where i reach ~40% with same settings)
>>>>
>>>> Mellanox without TX traffix on vlan:
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;16;64;11089305;709715520;8871553;567779392
>>>> 1;16;64;11096292;710162688;11095566;710116224
>>>> 2;16;64;11095770;710129280;11096799;710195136
>>>> 3;16;64;11097199;710220736;11097702;710252928
>>>> 4;16;64;11080984;567081856;11079662;709098368
>>>> 5;16;64;11077696;708972544;11077039;708930496
>>>> 6;16;64;11082991;709311424;8864802;567347328
>>>> 7;16;64;11089596;709734144;8870927;709789184
>>>> 8;16;64;11094043;710018752;11095391;710105024
>>>>
>>>> Mellanox with TX traffic on vlan:
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;16;64;7369914;471674496;7370281;471697980
>>>> 1;16;64;7368896;471609408;7368043;471554752
>>>> 2;16;64;7367577;471524864;7367759;471536576
>>>> 3;16;64;7368744;377305344;7369391;471641024
>>>> 4;16;64;7366824;471476736;7364330;471237120
>>>> 5;16;64;7368352;471574528;7367239;471503296
>>>> 6;16;64;7367459;471517376;7367806;471539584
>>>> 7;16;64;7367190;471500160;7367988;471551232
>>>> 8;16;64;7368023;471553472;7368076;471556864
>>> I wonder if the drivers page recycler is active/working or not, and if
>>> the situation is different between VLAN vs no-vlan (given
>>> page_frag_free is so high in you perf top). The Mellanox drivers
>>> fortunately have a stats counter to tell us this explicitly (which the
>>> ixgbe driver doesn't).
>>>
>>> You can use my ethtool_stats.pl script watch these stats:
>>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>> (Hint perl dependency: dnf install perl-Time-HiRes)
>> For RX NIC:
>> Show adapter(s) (enp175s0f0) statistics (ONLY that changed!)
>> Ethtool(enp175s0f0) stat: 78380071 ( 78,380,071) <= rx0_bytes /sec
>> Ethtool(enp175s0f0) stat: 230978 ( 230,978) <= rx0_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_packets /sec
>> Ethtool(enp175s0f0) stat: 921614 ( 921,614) <= rx0_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 78956591 ( 78,956,591) <= rx1_bytes /sec
>> Ethtool(enp175s0f0) stat: 233343 ( 233,343) <= rx1_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_packets /sec
>> Ethtool(enp175s0f0) stat: 927793 ( 927,793) <= rx1_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 79677124 ( 79,677,124) <= rx2_bytes /sec
>> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <= rx2_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_packets /sec
>> Ethtool(enp175s0f0) stat: 937989 ( 937,989) <= rx2_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 78392893 ( 78,392,893) <= rx3_bytes /sec
>> Ethtool(enp175s0f0) stat: 230311 ( 230,311) <= rx3_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_packets /sec
>> Ethtool(enp175s0f0) stat: 922513 ( 922,513) <= rx3_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 65165583 ( 65,165,583) <= rx4_bytes /sec
>> Ethtool(enp175s0f0) stat: 191969 ( 191,969) <= rx4_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_packets /sec
>> Ethtool(enp175s0f0) stat: 766332 ( 766,332) <= rx4_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 66920721 ( 66,920,721) <= rx5_bytes /sec
>> Ethtool(enp175s0f0) stat: 197150 ( 197,150) <= rx5_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_packets /sec
>> Ethtool(enp175s0f0) stat: 786978 ( 786,978) <= rx5_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 79076984 ( 79,076,984) <= rx6_bytes /sec
>> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <= rx6_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_packets /sec
>> Ethtool(enp175s0f0) stat: 929163 ( 929,163) <= rx6_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 78660672 ( 78,660,672) <= rx7_bytes /sec
>> Ethtool(enp175s0f0) stat: 230413 ( 230,413) <= rx7_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_packets /sec
>> Ethtool(enp175s0f0) stat: 926376 ( 926,376) <= rx7_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 10674565 ( 10,674,565) <= rx_65_to_127_bytes_phy /sec
>> Ethtool(enp175s0f0) stat: 605241031 ( 605,241,031) <= rx_bytes /sec
>> Ethtool(enp175s0f0) stat: 768585608 ( 768,585,608) <= rx_bytes_phy /sec
>> Ethtool(enp175s0f0) stat: 1781569 ( 1,781,569) <= rx_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1773785 ( 1,773,785) <= rx_out_of_buffer /sec
>> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_packets /sec
>> Ethtool(enp175s0f0) stat: 10674799 ( 10,674,799) <= rx_packets_phy /sec
>> Ethtool(enp175s0f0) stat: 7118993 ( 7,118,993) <= rx_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 768565744 ( 768,565,744) <= rx_prio0_bytes /sec
>> Ethtool(enp175s0f0) stat: 10674522 ( 10,674,522) <= rx_prio0_packets /sec
>> Ethtool(enp175s0f0) stat: 725871089 ( 725,871,089) <= rx_vport_unicast_bytes /sec
>> Ethtool(enp175s0f0) stat: 10674575 ( 10,674,575) <= rx_vport_unicast_packets /sec
>
> It looks like the mlx5 page recycle mechanism works:
>
> 230413 ( 230,413) <= rx7_cache_reuse /sec
> + 926376 ( 926,376) <= rx7_page_reuse /sec
> =1156789 (230413+926376)
> -1156775 ( 1,156,775) <= rx7_packets /sec
> = 14
>
> You can also determine this as there are no counters for:
> rx_cache_full or
> rx_cache_empty or
> rx1_cache_empty
> rx1_cache_busy
>
>
>> For TX nic with vlan:
>> Show adapter(s) (enp175s0f1) statistics (ONLY that changed!)
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_65_to_127_bytes_phy /sec
>> Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_bytes_phy /sec
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_multicast_phy /sec
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_packets_phy /sec
>> Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_prio0_bytes /sec
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_prio0_packets /sec
>> Ethtool(enp175s0f1) stat: 67 ( 67) <= rx_vport_multicast_bytes /sec
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_vport_multicast_packets /sec
>> Ethtool(enp175s0f1) stat: 64955114 ( 64,955,114) <= tx0_bytes /sec
>> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_csum_none /sec
>> Ethtool(enp175s0f1) stat: 26489 ( 26,489) <= tx0_nop /sec
>> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_packets /sec
>> Ethtool(enp175s0f1) stat: 66799214 ( 66,799,214) <= tx1_bytes /sec
>> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_csum_none /sec
>> Ethtool(enp175s0f1) stat: 27225 ( 27,225) <= tx1_nop /sec
>> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_packets /sec
>> Ethtool(enp175s0f1) stat: 78650421 ( 78,650,421) <= tx2_bytes /sec
>> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_csum_none /sec
>> Ethtool(enp175s0f1) stat: 32059 ( 32,059) <= tx2_nop /sec
>> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_packets /sec
>> Ethtool(enp175s0f1) stat: 78186849 ( 78,186,849) <= tx3_bytes /sec
>> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_csum_none /sec
>> Ethtool(enp175s0f1) stat: 31879 ( 31,879) <= tx3_nop /sec
>> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_packets /sec
>> Ethtool(enp175s0f1) stat: 234 ( 234) <= tx3_xmit_more /sec
>> Ethtool(enp175s0f1) stat: 78466099 ( 78,466,099) <= tx4_bytes /sec
>> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_csum_none /sec
>> Ethtool(enp175s0f1) stat: 31990 ( 31,990) <= tx4_nop /sec
>> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_packets /sec
>> Ethtool(enp175s0f1) stat: 78765724 ( 78,765,724) <= tx5_bytes /sec
>> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_csum_none /sec
>> Ethtool(enp175s0f1) stat: 32115 ( 32,115) <= tx5_nop /sec
>> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_packets /sec
>> Ethtool(enp175s0f1) stat: 264 ( 264) <= tx5_xmit_more /sec
>> Ethtool(enp175s0f1) stat: 79669524 ( 79,669,524) <= tx6_bytes /sec
>> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_csum_none /sec
>> Ethtool(enp175s0f1) stat: 32490 ( 32,490) <= tx6_nop /sec
>> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_packets /sec
>> Ethtool(enp175s0f1) stat: 79389329 ( 79,389,329) <= tx7_bytes /sec
>> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_csum_none /sec
>> Ethtool(enp175s0f1) stat: 32365 ( 32,365) <= tx7_nop /sec
>> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_packets /sec
>> Ethtool(enp175s0f1) stat: 604885175 ( 604,885,175) <= tx_bytes /sec
>> Ethtool(enp175s0f1) stat: 676059749 ( 676,059,749) <= tx_bytes_phy /sec
>> Ethtool(enp175s0f1) stat: 8895370 ( 8,895,370) <= tx_packets /sec
>> Ethtool(enp175s0f1) stat: 8895522 ( 8,895,522) <= tx_packets_phy /sec
>> Ethtool(enp175s0f1) stat: 676063067 ( 676,063,067) <= tx_prio0_bytes /sec
>> Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <= tx_prio0_packets /sec
>> Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <= tx_vport_unicast_bytes /sec
>> Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <= tx_vport_unicast_packets /sec
>> Ethtool(enp175s0f1) stat: 498 ( 498) <= tx_xmit_more /sec
> We are seeing some xmit_more, this is interesting. Have you noticed,
> if (in the VLAN case) there is a queue in the qdisc layer?
>
> Simply inspect with: tc -s qdisc show dev ixgbe2
for vlan no qdisc:
tc -s -d qdisc show dev vlan1000
qdisc noqueue 0: root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
physical interface mq attached with pfifo_fast:
tc -s -d qdisc show dev enp175s0f1
qdisc mq 0: root
Sent 1397200697212 bytes 3965888669 pkt (dropped 78065663, overlimits
0 requeues 629868)
backlog 0b 0p requeues 629868
qdisc pfifo_fast 0: parent :38 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :37 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :36 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :35 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :34 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :33 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :32 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :31 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :30 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :29 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :28 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :27 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :26 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :25 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :24 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :23 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :22 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :21 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :20 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :19 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :18 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 176 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :17 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :16 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :15 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :14 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :13 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :12 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :11 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 18534143578 bytes 289595993 pkt (dropped 0, overlimits 0 requeues
1191)
backlog 0b 0p requeues 1191
qdisc pfifo_fast 0: parent :f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 17022213952 bytes 265972093 pkt (dropped 0, overlimits 0 requeues
1126)
backlog 0b 0p requeues 1126
qdisc pfifo_fast 0: parent :e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 19610732552 bytes 306417696 pkt (dropped 0, overlimits 0 requeues
147499)
backlog 0b 0p requeues 147499
qdisc pfifo_fast 0: parent :d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 18153006206 bytes 283640721 pkt (dropped 0, overlimits 0 requeues
171576)
backlog 0b 0p requeues 171576
qdisc pfifo_fast 0: parent :c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 20642461846 bytes 322538466 pkt (dropped 0, overlimits 0 requeues
67124)
backlog 0b 0p requeues 67124
qdisc pfifo_fast 0: parent :b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 19343910144 bytes 302248596 pkt (dropped 0, overlimits 0 requeues
224627)
backlog 0b 0p requeues 224627
qdisc pfifo_fast 0: parent :a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 20735716968 bytes 323995576 pkt (dropped 0, overlimits 0 requeues
1451)
backlog 0b 0p requeues 1451
qdisc pfifo_fast 0: parent :9 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 19371526272 bytes 302680098 pkt (dropped 0, overlimits 0 requeues
2244)
backlog 0b 0p requeues 2244
qdisc pfifo_fast 0: parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 162202525454 bytes 2444195441 pkt (dropped 31716535, overlimits 0
requeues 2456)
backlog 0b 0p requeues 2456
qdisc pfifo_fast 0: parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 163308408548 bytes 2461561759 pkt (dropped 0, overlimits 0
requeues 1454)
backlog 0b 0p requeues 1454
qdisc pfifo_fast 0: parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 164214830346 bytes 2475734074 pkt (dropped 19272, overlimits 0
requeues 2216)
backlog 0b 0p requeues 2216
qdisc pfifo_fast 0: parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 160398101830 bytes 2417635038 pkt (dropped 0, overlimits 0
requeues 1509)
backlog 0b 0p requeues 1509
qdisc pfifo_fast 0: parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 161760263056 bytes 2437965677 pkt (dropped 0, overlimits 0
requeues 1442)
backlog 0b 0p requeues 1442
qdisc pfifo_fast 0: parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 160300203040 bytes 2415206290 pkt (dropped 0, overlimits 0
requeues 1615)
backlog 0b 0p requeues 1615
qdisc pfifo_fast 0: parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 140144336184 bytes 2114089739 pkt (dropped 0, overlimits 0
requeues 1177)
backlog 0b 0p requeues 1177
qdisc pfifo_fast 0: parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 131458317060 bytes 1982280594 pkt (dropped 46329856, overlimits 0
requeues 1161)
backlog 0b 0p requeues 1161
>
>
>>>
>>>> ethtool settings for both tests:
>>>> ifc='enp175s0f0 enp175s0f1'
>>>> for i in $ifc
>>>> do
>>>> ip link set up dev $i
>>>> ethtool -A $i autoneg off rx off tx off
>>>> ethtool -G $i rx 128 tx 256
>>> The ring queue size recommendations, might be different for the mlx5
>>> driver (Cc'ing Mellanox maintainers).
>>>
>>>
>>>> ip link set $i txqueuelen 1000
>>>> ethtool -C $i rx-usecs 25
>>>> ethtool -L $i combined 16
>>>> ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
>>>> tx-nocache-copy off ntuple on
>>>> ethtool -N $i rx-flow-hash udp4 sdfn
>>>> done
>>> Thanks for being explicit about what you setup is :-)
>>>
>>>> and perf top:
>>>> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>>>> cycles], (all, 56 CPUs)
>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> 14.25% [kernel] [k] dst_release
>>>> 14.17% [kernel] [k] skb_dst_force
>>>> 13.41% [kernel] [k] rt_cache_valid
>>>> 11.47% [kernel] [k] ip_finish_output2
>>>> 7.01% [kernel] [k] do_raw_spin_lock
>>>> 5.07% [kernel] [k] page_frag_free
>>>> 3.47% [mlx5_core] [k] mlx5e_xmit
>>>> 2.88% [kernel] [k] fib_table_lookup
>>>> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
>>>> 1.97% [kernel] [k] virt_to_head_page
>>>> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
>>>> 0.93% [kernel] [k] __dev_queue_xmit
>>>> 0.87% [kernel] [k] __build_skb
>>>> 0.84% [kernel] [k] ipt_do_table
>>>> 0.79% [kernel] [k] ip_rcv
>>>> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
>>>> 0.78% [kernel] [k] netif_skb_features
>>>> 0.73% [kernel] [k] __netif_receive_skb_core
>>>> 0.52% [kernel] [k] dev_hard_start_xmit
>>>> 0.52% [kernel] [k] build_skb
>>>> 0.51% [kernel] [k] ip_route_input_rcu
>>>> 0.50% [kernel] [k] skb_unref
>>>> 0.49% [kernel] [k] ip_forward
>>>> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
>>>> 0.44% [kernel] [k] udp_v4_early_demux
>>>> 0.41% [kernel] [k] napi_consume_skb
>>>> 0.40% [kernel] [k] __local_bh_enable_ip
>>>> 0.39% [kernel] [k] ip_rcv_finish
>>>> 0.39% [kernel] [k] kmem_cache_alloc
>>>> 0.38% [kernel] [k] sch_direct_xmit
>>>> 0.33% [kernel] [k] validate_xmit_skb
>>>> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
>>>> 0.29% [kernel] [k] netdev_pick_tx
>>>> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
>>>> 0.27% [kernel] [k] deliver_ptype_list_skb
>>>> 0.26% [kernel] [k] fib_validate_source
>>>> 0.26% [mlx5_core] [k] mlx5e_napi_poll
>>>> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
>>>> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
>>>> 0.25% [kernel] [k] eth_header
>>>> 0.23% [kernel] [k] skb_network_protocol
>>>> 0.20% [kernel] [k] nf_hook_slow
>>>> 0.20% [kernel] [k] vlan_passthru_hard_header
>>>> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
>>>> 0.19% [kernel] [k] swiotlb_map_page
>>>> 0.18% [kernel] [k] compound_head
>>>> 0.18% [kernel] [k] neigh_connected_output
>>>> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
>>>> 0.18% [kernel] [k] ip_output
>>>> 0.17% [kernel] [k] prefetch_freepointer.isra.70
>>>> 0.17% [kernel] [k] __slab_free
>>>> 0.16% [kernel] [k] eth_type_vlan
>>>> 0.16% [kernel] [k] ip_finish_output
>>>> 0.15% [kernel] [k] kmem_cache_free_bulk
>>>> 0.14% [kernel] [k] netif_receive_skb_internal
>>>>
>>>>
>>>>
>>>>
>>>> wondering why this:
>>>> 1.97% [kernel] [k] virt_to_head_page
>>>> is in top...
>>> This is related to the page_frag_free() call, but it is weird that it
>>> shows up because it is suppose to be inlined (it is explicitly marked
>>> inline in include/linux/mm.h).
>>>
>>>
>>>>>>>>> perf top:
>>>>>>>>>
>>>>>>>>> PerfTop: 77835 irqs/sec kernel:99.7%
>>>>>>>>> ---------------------------------------------
>>>>>>>>>
>>>>>>>>> 16.32% [kernel] [k] skb_dst_force
>>>>>>>>> 16.30% [kernel] [k] dst_release
>>>>>>>>> 15.11% [kernel] [k] rt_cache_valid
>>>>>>>>> 12.62% [kernel] [k] ipv4_mtu
>>>>>>>> It seems a little strange that these 4 functions are on the top
>>>>> I don't see these in my test.
>>>>>
>>>>>>>>
>>>>>>>>> 5.60% [kernel] [k] do_raw_spin_lock
>>>>>>>> Why is calling/taking this lock? (Use perf call-graph recording).
>>>>>>> can be hard to paste it here:)
>>>>>>> attached file
>>>>> The attached was very big. Please don't attach so big file on mailing
>>>>> lists. Next time plase share them via e.g. pastebin. The output was a
>>>>> capture from your terminal, which made the output more difficult to
>>>>> read. Hint: You can/could use perf --stdio and place it in a file
>>>>> instead.
>>>>>
>>>>> The output (extracted below) didn't show who called 'do_raw_spin_lock',
>>>>> BUT it showed another interesting thing. The kernel code
>>>>> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
>>>>> as it will first call skb_dst_force() and then skb_dst_drop() when the
>>>>> packet is transmitted on a VLAN.
>>>>>
>>>>> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
>>>>> {
>>>>> [...]
>>>>> /* If device/qdisc don't need skb->dst, release it right now while
>>>>> * its hot in this cpu cache.
>>>>> */
>>>>> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
>>>>> skb_dst_drop(skb);
>>>>> else
>>>>> skb_dst_force(skb);
>>>>>
>
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Jesper Dangaard Brouer @ 2017-08-15 10:05 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Eric Dumazet, Paolo Abeni, Linux Kernel Network Developers,
Alexander Duyck, brouer
In-Reply-To: <4b1efff7-4f91-fd78-beb8-2c7ebcf18895@itcare.pl>
On Tue, 15 Aug 2017 11:11:57 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> Yes it helped - now there is almost no difference when using vlans or not:
>
> 10.5Mpps - with vlan
>
> 11Mpps - without vlan
Great! - it seems like we have pinpointed the root-cause. It also
demonstrate how big the benefit is of Eric commit (thanks!):
https://git.kernel.org/torvalds/c/93f154b594fe
> W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
> > On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
> >
> >> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
> > Something like :
> >
> > diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
> > index 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7 100644
> > --- a/net/8021q/vlan_netlink.c
> > +++ b/net/8021q/vlan_netlink.c
> > @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net, struct net_device *dev,
> > vlan->vlan_proto = proto;
> > vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
> > vlan->real_dev = real_dev;
> > + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
> > vlan->flags = VLAN_FLAG_REORDER_HDR;
> >
> > err = vlan_check_real_dev(real_dev, vlan->vlan_proto, vlan->vlan_id);
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Paweł Staszewski @ 2017-08-15 10:05 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan
In-Reply-To: <fee2de62-c781-4ceb-8e97-be2b07eb1ca1@itcare.pl>
W dniu 2017-08-15 o 12:02, Paweł Staszewski pisze:
>
>
> W dniu 2017-08-15 o 11:57, Jesper Dangaard Brouer pisze:
>> On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
>>>> On Tue, 15 Aug 2017 02:38:56 +0200
>>>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>>>> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
>>>>>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski
>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>> To show some difference below comparision vlan/no-vlan traffic
>>>>>>>
>>>>>>> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
>>>>>> I'm trying to reproduce in my testlab (with ixgbe). I do see, a
>>>>>> performance reduction of about 10-19% when I forward out a VLAN
>>>>>> interface. This is larger than I expected, but still lower than
>>>>>> what
>>>>>> you reported 30-40% slowdown.
>>>>>>
>>>>>> [...]
>>>>> Ok mellanox afrrived (MT27700 - mlnx5 driver)
>>>>> And to compare melannox with vlans and without: 33% performance
>>>>> degradation (less than with ixgbe where i reach ~40% with same
>>>>> settings)
>>>>>
>>>>> Mellanox without TX traffix on vlan:
>>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>>> 0;16;64;11089305;709715520;8871553;567779392
>>>>> 1;16;64;11096292;710162688;11095566;710116224
>>>>> 2;16;64;11095770;710129280;11096799;710195136
>>>>> 3;16;64;11097199;710220736;11097702;710252928
>>>>> 4;16;64;11080984;567081856;11079662;709098368
>>>>> 5;16;64;11077696;708972544;11077039;708930496
>>>>> 6;16;64;11082991;709311424;8864802;567347328
>>>>> 7;16;64;11089596;709734144;8870927;709789184
>>>>> 8;16;64;11094043;710018752;11095391;710105024
>>>>>
>>>>> Mellanox with TX traffic on vlan:
>>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>>> 0;16;64;7369914;471674496;7370281;471697980
>>>>> 1;16;64;7368896;471609408;7368043;471554752
>>>>> 2;16;64;7367577;471524864;7367759;471536576
>>>>> 3;16;64;7368744;377305344;7369391;471641024
>>>>> 4;16;64;7366824;471476736;7364330;471237120
>>>>> 5;16;64;7368352;471574528;7367239;471503296
>>>>> 6;16;64;7367459;471517376;7367806;471539584
>>>>> 7;16;64;7367190;471500160;7367988;471551232
>>>>> 8;16;64;7368023;471553472;7368076;471556864
>>>> I wonder if the drivers page recycler is active/working or not, and if
>>>> the situation is different between VLAN vs no-vlan (given
>>>> page_frag_free is so high in you perf top). The Mellanox drivers
>>>> fortunately have a stats counter to tell us this explicitly (which the
>>>> ixgbe driver doesn't).
>>>>
>>>> You can use my ethtool_stats.pl script watch these stats:
>>>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>>> (Hint perl dependency: dnf install perl-Time-HiRes)
>>> For RX NIC:
>>> Show adapter(s) (enp175s0f0) statistics (ONLY that changed!)
>>> Ethtool(enp175s0f0) stat: 78380071 ( 78,380,071) <=
>>> rx0_bytes /sec
>>> Ethtool(enp175s0f0) stat: 230978 ( 230,978) <=
>>> rx0_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <=
>>> rx0_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <=
>>> rx0_packets /sec
>>> Ethtool(enp175s0f0) stat: 921614 ( 921,614) <=
>>> rx0_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 78956591 ( 78,956,591) <=
>>> rx1_bytes /sec
>>> Ethtool(enp175s0f0) stat: 233343 ( 233,343) <=
>>> rx1_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <=
>>> rx1_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <=
>>> rx1_packets /sec
>>> Ethtool(enp175s0f0) stat: 927793 ( 927,793) <=
>>> rx1_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 79677124 ( 79,677,124) <=
>>> rx2_bytes /sec
>>> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <=
>>> rx2_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <=
>>> rx2_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <=
>>> rx2_packets /sec
>>> Ethtool(enp175s0f0) stat: 937989 ( 937,989) <=
>>> rx2_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 78392893 ( 78,392,893) <=
>>> rx3_bytes /sec
>>> Ethtool(enp175s0f0) stat: 230311 ( 230,311) <=
>>> rx3_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <=
>>> rx3_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <=
>>> rx3_packets /sec
>>> Ethtool(enp175s0f0) stat: 922513 ( 922,513) <=
>>> rx3_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 65165583 ( 65,165,583) <=
>>> rx4_bytes /sec
>>> Ethtool(enp175s0f0) stat: 191969 ( 191,969) <=
>>> rx4_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <=
>>> rx4_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <=
>>> rx4_packets /sec
>>> Ethtool(enp175s0f0) stat: 766332 ( 766,332) <=
>>> rx4_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 66920721 ( 66,920,721) <=
>>> rx5_bytes /sec
>>> Ethtool(enp175s0f0) stat: 197150 ( 197,150) <=
>>> rx5_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <=
>>> rx5_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <=
>>> rx5_packets /sec
>>> Ethtool(enp175s0f0) stat: 786978 ( 786,978) <=
>>> rx5_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 79076984 ( 79,076,984) <=
>>> rx6_bytes /sec
>>> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <=
>>> rx6_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <=
>>> rx6_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <=
>>> rx6_packets /sec
>>> Ethtool(enp175s0f0) stat: 929163 ( 929,163) <=
>>> rx6_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 78660672 ( 78,660,672) <=
>>> rx7_bytes /sec
>>> Ethtool(enp175s0f0) stat: 230413 ( 230,413) <=
>>> rx7_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <=
>>> rx7_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <=
>>> rx7_packets /sec
>>> Ethtool(enp175s0f0) stat: 926376 ( 926,376) <=
>>> rx7_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 10674565 ( 10,674,565) <=
>>> rx_65_to_127_bytes_phy /sec
>>> Ethtool(enp175s0f0) stat: 605241031 ( 605,241,031) <= rx_bytes
>>> /sec
>>> Ethtool(enp175s0f0) stat: 768585608 ( 768,585,608) <=
>>> rx_bytes_phy /sec
>>> Ethtool(enp175s0f0) stat: 1781569 ( 1,781,569) <=
>>> rx_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <=
>>> rx_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1773785 ( 1,773,785) <=
>>> rx_out_of_buffer /sec
>>> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <=
>>> rx_packets /sec
>>> Ethtool(enp175s0f0) stat: 10674799 ( 10,674,799) <=
>>> rx_packets_phy /sec
>>> Ethtool(enp175s0f0) stat: 7118993 ( 7,118,993) <=
>>> rx_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 768565744 ( 768,565,744) <=
>>> rx_prio0_bytes /sec
>>> Ethtool(enp175s0f0) stat: 10674522 ( 10,674,522) <=
>>> rx_prio0_packets /sec
>>> Ethtool(enp175s0f0) stat: 725871089 ( 725,871,089) <=
>>> rx_vport_unicast_bytes /sec
>>> Ethtool(enp175s0f0) stat: 10674575 ( 10,674,575) <=
>>> rx_vport_unicast_packets /sec
>>
>> It looks like the mlx5 page recycle mechanism works:
>>
>> 230413 ( 230,413) <= rx7_cache_reuse /sec
>> + 926376 ( 926,376) <= rx7_page_reuse /sec
>> =1156789 (230413+926376)
>> -1156775 ( 1,156,775) <= rx7_packets /sec
>> = 14
>>
>> You can also determine this as there are no counters for:
>> rx_cache_full or
>> rx_cache_empty or
>> rx1_cache_empty
>> rx1_cache_busy
>>
>>> For TX nic with vlan:
>>> Show adapter(s) (enp175s0f1) statistics (ONLY that changed!)
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_65_to_127_bytes_phy /sec
>>> Ethtool(enp175s0f1) stat: 71 ( 71) <=
>>> rx_bytes_phy /sec
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_multicast_phy /sec
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_packets_phy /sec
>>> Ethtool(enp175s0f1) stat: 71 ( 71) <=
>>> rx_prio0_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_prio0_packets /sec
>>> Ethtool(enp175s0f1) stat: 67 ( 67) <=
>>> rx_vport_multicast_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_vport_multicast_packets /sec
>>> Ethtool(enp175s0f1) stat: 64955114 ( 64,955,114) <=
>>> tx0_bytes /sec
>>> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <=
>>> tx0_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 26489 ( 26,489) <= tx0_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <=
>>> tx0_packets /sec
>>> Ethtool(enp175s0f1) stat: 66799214 ( 66,799,214) <=
>>> tx1_bytes /sec
>>> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <=
>>> tx1_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 27225 ( 27,225) <= tx1_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <=
>>> tx1_packets /sec
>>> Ethtool(enp175s0f1) stat: 78650421 ( 78,650,421) <=
>>> tx2_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <=
>>> tx2_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 32059 ( 32,059) <= tx2_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <=
>>> tx2_packets /sec
>>> Ethtool(enp175s0f1) stat: 78186849 ( 78,186,849) <=
>>> tx3_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <=
>>> tx3_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 31879 ( 31,879) <= tx3_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <=
>>> tx3_packets /sec
>>> Ethtool(enp175s0f1) stat: 234 ( 234) <=
>>> tx3_xmit_more /sec
>>> Ethtool(enp175s0f1) stat: 78466099 ( 78,466,099) <=
>>> tx4_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <=
>>> tx4_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 31990 ( 31,990) <= tx4_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <=
>>> tx4_packets /sec
>>> Ethtool(enp175s0f1) stat: 78765724 ( 78,765,724) <=
>>> tx5_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <=
>>> tx5_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 32115 ( 32,115) <= tx5_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <=
>>> tx5_packets /sec
>>> Ethtool(enp175s0f1) stat: 264 ( 264) <=
>>> tx5_xmit_more /sec
>>> Ethtool(enp175s0f1) stat: 79669524 ( 79,669,524) <=
>>> tx6_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <=
>>> tx6_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 32490 ( 32,490) <= tx6_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <=
>>> tx6_packets /sec
>>> Ethtool(enp175s0f1) stat: 79389329 ( 79,389,329) <=
>>> tx7_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <=
>>> tx7_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 32365 ( 32,365) <= tx7_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <=
>>> tx7_packets /sec
>>> Ethtool(enp175s0f1) stat: 604885175 ( 604,885,175) <= tx_bytes
>>> /sec
>>> Ethtool(enp175s0f1) stat: 676059749 ( 676,059,749) <=
>>> tx_bytes_phy /sec
>>> Ethtool(enp175s0f1) stat: 8895370 ( 8,895,370) <=
>>> tx_packets /sec
>>> Ethtool(enp175s0f1) stat: 8895522 ( 8,895,522) <=
>>> tx_packets_phy /sec
>>> Ethtool(enp175s0f1) stat: 676063067 ( 676,063,067) <=
>>> tx_prio0_bytes /sec
>>> Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <=
>>> tx_prio0_packets /sec
>>> Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <=
>>> tx_vport_unicast_bytes /sec
>>> Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <=
>>> tx_vport_unicast_packets /sec
>>> Ethtool(enp175s0f1) stat: 498 ( 498) <=
>>> tx_xmit_more /sec
>> We are seeing some xmit_more, this is interesting. Have you noticed,
>> if (in the VLAN case) there is a queue in the qdisc layer?
>>
>> Simply inspect with: tc -s qdisc show dev ixgbe2
> for vlan no qdisc:
> tc -s -d qdisc show dev vlan1000
>
> qdisc noqueue 0: root refcnt 2
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>
> physical interface mq attached with pfifo_fast:
>
> tc -s -d qdisc show dev enp175s0f1
> qdisc mq 0: root
> Sent 1397200697212 bytes 3965888669 pkt (dropped 78065663, overlimits
> 0 requeues 629868)
> backlog 0b 0p requeues 629868
> qdisc pfifo_fast 0: parent :38 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :37 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :36 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :35 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :34 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :33 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :32 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :31 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :30 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :29 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :28 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :27 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :26 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :25 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :24 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :23 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :22 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :21 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :20 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :19 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :18 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 176 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :17 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :16 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :15 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :14 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :13 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :12 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :11 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 18534143578 bytes 289595993 pkt (dropped 0, overlimits 0
> requeues 1191)
> backlog 0b 0p requeues 1191
> qdisc pfifo_fast 0: parent :f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 17022213952 bytes 265972093 pkt (dropped 0, overlimits 0
> requeues 1126)
> backlog 0b 0p requeues 1126
> qdisc pfifo_fast 0: parent :e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 19610732552 bytes 306417696 pkt (dropped 0, overlimits 0
> requeues 147499)
> backlog 0b 0p requeues 147499
> qdisc pfifo_fast 0: parent :d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 18153006206 bytes 283640721 pkt (dropped 0, overlimits 0
> requeues 171576)
> backlog 0b 0p requeues 171576
> qdisc pfifo_fast 0: parent :c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 20642461846 bytes 322538466 pkt (dropped 0, overlimits 0
> requeues 67124)
> backlog 0b 0p requeues 67124
> qdisc pfifo_fast 0: parent :b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 19343910144 bytes 302248596 pkt (dropped 0, overlimits 0
> requeues 224627)
> backlog 0b 0p requeues 224627
> qdisc pfifo_fast 0: parent :a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 20735716968 bytes 323995576 pkt (dropped 0, overlimits 0
> requeues 1451)
> backlog 0b 0p requeues 1451
> qdisc pfifo_fast 0: parent :9 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 19371526272 bytes 302680098 pkt (dropped 0, overlimits 0
> requeues 2244)
> backlog 0b 0p requeues 2244
> qdisc pfifo_fast 0: parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 162202525454 bytes 2444195441 pkt (dropped 31716535, overlimits
> 0 requeues 2456)
> backlog 0b 0p requeues 2456
> qdisc pfifo_fast 0: parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 163308408548 bytes 2461561759 pkt (dropped 0, overlimits 0
> requeues 1454)
> backlog 0b 0p requeues 1454
> qdisc pfifo_fast 0: parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 164214830346 bytes 2475734074 pkt (dropped 19272, overlimits 0
> requeues 2216)
> backlog 0b 0p requeues 2216
> qdisc pfifo_fast 0: parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 160398101830 bytes 2417635038 pkt (dropped 0, overlimits 0
> requeues 1509)
> backlog 0b 0p requeues 1509
> qdisc pfifo_fast 0: parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 161760263056 bytes 2437965677 pkt (dropped 0, overlimits 0
> requeues 1442)
> backlog 0b 0p requeues 1442
> qdisc pfifo_fast 0: parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 160300203040 bytes 2415206290 pkt (dropped 0, overlimits 0
> requeues 1615)
> backlog 0b 0p requeues 1615
> qdisc pfifo_fast 0: parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 140144336184 bytes 2114089739 pkt (dropped 0, overlimits 0
> requeues 1177)
> backlog 0b 0p requeues 1177
> qdisc pfifo_fast 0: parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 131458317060 bytes 1982280594 pkt (dropped 46329856, overlimits
> 0 requeues 1161)
> backlog 0b 0p requeues 1161
>
>
just see that after changing RSS on nics did't deleted qdisc and added
again:
Here situation with qdisc del / add
tc -s -d qdisc show dev enp175s0f1
qdisc mq 1: root
Sent 43738523966 bytes 683414438 pkt (dropped 0, overlimits 0 requeues
1886)
backlog 0b 0p requeues 1886
qdisc pfifo_fast 0: parent 1:10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2585011904 bytes 40390811 pkt (dropped 0, overlimits 0 requeues 110)
backlog 0b 0p requeues 110
qdisc pfifo_fast 0: parent 1:f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2602068416 bytes 40657319 pkt (dropped 0, overlimits 0 requeues 121)
backlog 0b 0p requeues 121
qdisc pfifo_fast 0: parent 1:e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2793524544 bytes 43648821 pkt (dropped 0, overlimits 0 requeues 12)
backlog 0b 0p requeues 12
qdisc pfifo_fast 0: parent 1:d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2792512768 bytes 43633012 pkt (dropped 0, overlimits 0 requeues 8)
backlog 0b 0p requeues 8
qdisc pfifo_fast 0: parent 1:c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789822080 bytes 43590970 pkt (dropped 0, overlimits 0 requeues 8)
backlog 0b 0p requeues 8
qdisc pfifo_fast 0: parent 1:b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789904640 bytes 43592260 pkt (dropped 0, overlimits 0 requeues 9)
backlog 0b 0p requeues 9
qdisc pfifo_fast 0: parent 1:a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2793279146 bytes 43644987 pkt (dropped 0, overlimits 0 requeues 192)
backlog 0b 0p requeues 192
qdisc pfifo_fast 0: parent 1:9 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789506688 bytes 43586042 pkt (dropped 0, overlimits 0 requeues 171)
backlog 0b 0p requeues 171
qdisc pfifo_fast 0: parent 1:8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789948928 bytes 43592952 pkt (dropped 0, overlimits 0 requeues 179)
backlog 0b 0p requeues 179
qdisc pfifo_fast 0: parent 1:7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2790191552 bytes 43596743 pkt (dropped 0, overlimits 0 requeues 177)
backlog 0b 0p requeues 177
qdisc pfifo_fast 0: parent 1:6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789649280 bytes 43588270 pkt (dropped 0, overlimits 0 requeues 182)
backlog 0b 0p requeues 182
qdisc pfifo_fast 0: parent 1:5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789654612 bytes 43588354 pkt (dropped 0, overlimits 0 requeues 151)
backlog 0b 0p requeues 151
qdisc pfifo_fast 0: parent 1:4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789409280 bytes 43584520 pkt (dropped 0, overlimits 0 requeues 168)
backlog 0b 0p requeues 168
qdisc pfifo_fast 0: parent 1:3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2792827648 bytes 43637932 pkt (dropped 0, overlimits 0 requeues 165)
backlog 0b 0p requeues 165
qdisc pfifo_fast 0: parent 1:2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2572823936 bytes 40200374 pkt (dropped 0, overlimits 0 requeues 119)
backlog 0b 0p requeues 119
qdisc pfifo_fast 0: parent 1:1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2488427264 bytes 38881676 pkt (dropped 0, overlimits 0 requeues 114)
backlog 0b 0p requeues 114
>
>>
>>
>>>>> ethtool settings for both tests:
>>>>> ifc='enp175s0f0 enp175s0f1'
>>>>> for i in $ifc
>>>>> do
>>>>> ip link set up dev $i
>>>>> ethtool -A $i autoneg off rx off tx off
>>>>> ethtool -G $i rx 128 tx 256
>>>> The ring queue size recommendations, might be different for the mlx5
>>>> driver (Cc'ing Mellanox maintainers).
>>>>
>>>>> ip link set $i txqueuelen 1000
>>>>> ethtool -C $i rx-usecs 25
>>>>> ethtool -L $i combined 16
>>>>> ethtool -K $i gro off tso off gso off sg on
>>>>> l2-fwd-offload off
>>>>> tx-nocache-copy off ntuple on
>>>>> ethtool -N $i rx-flow-hash udp4 sdfn
>>>>> done
>>>> Thanks for being explicit about what you setup is :-)
>>>>> and perf top:
>>>>> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>>>>> cycles], (all, 56 CPUs)
>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> 14.25% [kernel] [k] dst_release
>>>>> 14.17% [kernel] [k] skb_dst_force
>>>>> 13.41% [kernel] [k] rt_cache_valid
>>>>> 11.47% [kernel] [k] ip_finish_output2
>>>>> 7.01% [kernel] [k] do_raw_spin_lock
>>>>> 5.07% [kernel] [k] page_frag_free
>>>>> 3.47% [mlx5_core] [k] mlx5e_xmit
>>>>> 2.88% [kernel] [k] fib_table_lookup
>>>>> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
>>>>> 1.97% [kernel] [k] virt_to_head_page
>>>>> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
>>>>> 0.93% [kernel] [k] __dev_queue_xmit
>>>>> 0.87% [kernel] [k] __build_skb
>>>>> 0.84% [kernel] [k] ipt_do_table
>>>>> 0.79% [kernel] [k] ip_rcv
>>>>> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
>>>>> 0.78% [kernel] [k] netif_skb_features
>>>>> 0.73% [kernel] [k] __netif_receive_skb_core
>>>>> 0.52% [kernel] [k] dev_hard_start_xmit
>>>>> 0.52% [kernel] [k] build_skb
>>>>> 0.51% [kernel] [k] ip_route_input_rcu
>>>>> 0.50% [kernel] [k] skb_unref
>>>>> 0.49% [kernel] [k] ip_forward
>>>>> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
>>>>> 0.44% [kernel] [k] udp_v4_early_demux
>>>>> 0.41% [kernel] [k] napi_consume_skb
>>>>> 0.40% [kernel] [k] __local_bh_enable_ip
>>>>> 0.39% [kernel] [k] ip_rcv_finish
>>>>> 0.39% [kernel] [k] kmem_cache_alloc
>>>>> 0.38% [kernel] [k] sch_direct_xmit
>>>>> 0.33% [kernel] [k] validate_xmit_skb
>>>>> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
>>>>> 0.29% [kernel] [k] netdev_pick_tx
>>>>> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
>>>>> 0.27% [kernel] [k] deliver_ptype_list_skb
>>>>> 0.26% [kernel] [k] fib_validate_source
>>>>> 0.26% [mlx5_core] [k] mlx5e_napi_poll
>>>>> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
>>>>> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
>>>>> 0.25% [kernel] [k] eth_header
>>>>> 0.23% [kernel] [k] skb_network_protocol
>>>>> 0.20% [kernel] [k] nf_hook_slow
>>>>> 0.20% [kernel] [k] vlan_passthru_hard_header
>>>>> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
>>>>> 0.19% [kernel] [k] swiotlb_map_page
>>>>> 0.18% [kernel] [k] compound_head
>>>>> 0.18% [kernel] [k] neigh_connected_output
>>>>> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
>>>>> 0.18% [kernel] [k] ip_output
>>>>> 0.17% [kernel] [k] prefetch_freepointer.isra.70
>>>>> 0.17% [kernel] [k] __slab_free
>>>>> 0.16% [kernel] [k] eth_type_vlan
>>>>> 0.16% [kernel] [k] ip_finish_output
>>>>> 0.15% [kernel] [k] kmem_cache_free_bulk
>>>>> 0.14% [kernel] [k] netif_receive_skb_internal
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> wondering why this:
>>>>> 1.97% [kernel] [k] virt_to_head_page
>>>>> is in top...
>>>> This is related to the page_frag_free() call, but it is weird that it
>>>> shows up because it is suppose to be inlined (it is explicitly marked
>>>> inline in include/linux/mm.h).
>>>>
>>>>>>>>>> perf top:
>>>>>>>>>>
>>>>>>>>>> PerfTop: 77835 irqs/sec kernel:99.7%
>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>
>>>>>>>>>> 16.32% [kernel] [k] skb_dst_force
>>>>>>>>>> 16.30% [kernel] [k] dst_release
>>>>>>>>>> 15.11% [kernel] [k] rt_cache_valid
>>>>>>>>>> 12.62% [kernel] [k] ipv4_mtu
>>>>>>>>> It seems a little strange that these 4 functions are on the top
>>>>>> I don't see these in my test.
>>>>>>>>>> 5.60% [kernel] [k] do_raw_spin_lock
>>>>>>>>> Why is calling/taking this lock? (Use perf call-graph recording).
>>>>>>>> can be hard to paste it here:)
>>>>>>>> attached file
>>>>>> The attached was very big. Please don't attach so big file on
>>>>>> mailing
>>>>>> lists. Next time plase share them via e.g. pastebin. The output
>>>>>> was a
>>>>>> capture from your terminal, which made the output more difficult to
>>>>>> read. Hint: You can/could use perf --stdio and place it in a file
>>>>>> instead.
>>>>>>
>>>>>> The output (extracted below) didn't show who called
>>>>>> 'do_raw_spin_lock',
>>>>>> BUT it showed another interesting thing. The kernel code
>>>>>> __dev_queue_xmit() in might create route dst-cache problem for
>>>>>> itself(?),
>>>>>> as it will first call skb_dst_force() and then skb_dst_drop()
>>>>>> when the
>>>>>> packet is transmitted on a VLAN.
>>>>>>
>>>>>> static int __dev_queue_xmit(struct sk_buff *skb, void
>>>>>> *accel_priv)
>>>>>> {
>>>>>> [...]
>>>>>> /* If device/qdisc don't need skb->dst, release it right now
>>>>>> while
>>>>>> * its hot in this cpu cache.
>>>>>> */
>>>>>> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
>>>>>> skb_dst_drop(skb);
>>>>>> else
>>>>>> skb_dst_force(skb);
>>>>>>
>>
>
>
^ permalink raw reply
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
From: Jesper Dangaard Brouer @ 2017-08-15 10:28 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan, brouer
In-Reply-To: <3d59ff3b-246b-2f46-2351-31c2d58ca6fe@itcare.pl>
On Tue, 15 Aug 2017 12:05:37 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 2017-08-15 o 12:02, Paweł Staszewski pisze:
> > W dniu 2017-08-15 o 11:57, Jesper Dangaard Brouer pisze:
> >> On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski
> >> <pstaszewski@itcare.pl> wrote:
> >>> W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
> >>>> On Tue, 15 Aug 2017 02:38:56 +0200
> >>>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> >>>>> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
> >>>>>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski
> >>>>>> <pstaszewski@itcare.pl> wrote:
[... cut ...]
> >>> Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <= tx_prio0_packets /sec
> >>> Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <= tx_vport_unicast_bytes /sec
> >>> Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <= tx_vport_unicast_packets /sec
> >>> Ethtool(enp175s0f1) stat: 498 ( 498) <= tx_xmit_more /sec
> >>
> >> We are seeing some xmit_more, this is interesting. Have you noticed,
> >> if (in the VLAN case) there is a queue in the qdisc layer?
> >>
> >> Simply inspect with: tc -s qdisc show dev ixgbe2
[...]
> > physical interface mq attached with pfifo_fast:
> >
> > tc -s -d qdisc show dev enp175s0f1
> > qdisc mq 0: root
> > Sent 1397200697212 bytes 3965888669 pkt (dropped 78065663, overlimits 0 requeues 629868)
> > backlog 0b 0p requeues 629868
> > qdisc pfifo_fast 0: parent :38 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
> > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> > backlog 0b 0p requeues 0
> > qdisc pfifo_fast 0: parent :37 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
> > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> > backlog 0b 0p requeues 0
[...]
So, it doesn't look like there is any backlog queue. Although, this
can be difficult to measure/see this way (as the kernel empty the queue
quickly via bulk deq), also given the small amount of xmit_more which
indicate that the queue was likely very small.
There is a "dropped" counter, which indicate that you likely had a
setup (earlier) where you managed to overflow the qdisc queues.
> just see that after changing RSS on nics did't deleted qdisc and added
> again:
> Here situation with qdisc del / add
> tc -s -d qdisc show dev enp175s0f1
> qdisc mq 1: root
> Sent 43738523966 bytes 683414438 pkt (dropped 0, overlimits 0 requeues 1886)
> backlog 0b 0p requeues 1886
> qdisc pfifo_fast 0: parent 1:10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
> Sent 2585011904 bytes 40390811 pkt (dropped 0, overlimits 0 requeues 110)
> backlog 0b 0p requeues 110
> qdisc pfifo_fast 0: parent 1:f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
> Sent 2602068416 bytes 40657319 pkt (dropped 0, overlimits 0 requeues 121)
> backlog 0b 0p requeues 121
[...]
Exactly as you indicated above, these "dropped" stats came from another
(earlier) test case. (Great that you caught this yourself)
While trying to reproduce you case, I also managed to cause a situation
with qdisc overload. This caused some weird behavior, where I saw
RX=8Mpps and TX only 4Mpps. (I didn't figure out the exact tuning that
caused this, and cannot reproduce it now).
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply
* [PATCH] i40e{,vf}: Fix out-of-bound cpumask read in IRQ affinity handler
From: Stefano Brivio @ 2017-08-15 10:30 UTC (permalink / raw)
To: Jeff Kirsher, netdev, intel-wired-lan
Cc: David S . Miller, Alan Brady, Stefan Assmann
The cpumask used in i40e{,vf}_irq_affinity_notify() is allocated
by irq_affinity_notify() with alloc_cpumask_var(), which doesn't
allocate NR_CPUS bits, but only nr_cpumask_bits bits. If we just
dereference it, we'll read way more than what is allocated, e.g.
1024 bytes vs. 8 bytes allocated on x86_64 machine with 24 CPUs.
Use cpumask_copy() instead. A comprehensive explanation is given
in the comments about cpumask_var_t, in include/linux/cpumask.h.
KASAN reports:
[ 25.242312] BUG: KASAN: slab-out-of-bounds in i40e_irq_affinity_notify+0x30/0x50 [i40e] at addr ffff880462eea960
[ 25.242315] Read of size 1024 by task kworker/2:1/170
[ 25.242322] CPU: 2 PID: 170 Comm: kworker/2:1 Not tainted 4.11.0-22.el7a.x86_64 #1
[ 25.242325] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 05/06/2015
[ 25.242336] Workqueue: events irq_affinity_notify
[ 25.242340] Call Trace:
[ 25.242350] dump_stack+0x63/0x8d
[ 25.242358] kasan_object_err+0x21/0x70
[ 25.242364] kasan_report+0x288/0x540
[ 25.242397] ? i40e_irq_affinity_notify+0x30/0x50 [i40e]
[ 25.242403] check_memory_region+0x13c/0x1a0
[ 25.242408] __asan_loadN+0xf/0x20
[ 25.242440] i40e_irq_affinity_notify+0x30/0x50 [i40e]
[ 25.242446] irq_affinity_notify+0x1b4/0x230
[ 25.242452] ? irq_set_affinity_notifier+0x130/0x130
[ 25.242457] ? kasan_slab_free+0x89/0xc0
[ 25.242466] process_one_work+0x32f/0x6f0
[ 25.242472] worker_thread+0x89/0x770
[ 25.242481] ? pci_mmcfg_check_reserved+0xc0/0xc0
[ 25.242488] kthread+0x18c/0x1e0
[ 25.242493] ? process_one_work+0x6f0/0x6f0
[ 25.242499] ? kthread_create_on_node+0xc0/0xc0
[ 25.242506] ret_from_fork+0x2c/0x40
[ 25.242511] Object at ffff880462eea960, in cache kmalloc-8 size: 8
[ 25.242513] Allocated:
[ 25.242514] PID = 170
[ 25.242522] save_stack_trace+0x1b/0x20
[ 25.242529] save_stack+0x46/0xd0
[ 25.242533] kasan_kmalloc+0xad/0xe0
[ 25.242537] __kmalloc_node+0x12c/0x2b0
[ 25.242542] alloc_cpumask_var_node+0x3c/0x60
[ 25.242546] alloc_cpumask_var+0xe/0x10
[ 25.242550] irq_affinity_notify+0x94/0x230
[ 25.242555] process_one_work+0x32f/0x6f0
[ 25.242559] worker_thread+0x89/0x770
[ 25.242564] kthread+0x18c/0x1e0
[ 25.242568] ret_from_fork+0x2c/0x40
[ 25.242569] Freed:
[ 25.242570] PID = 0
[ 25.242572] (stack is not available)
[ 25.242573] Memory state around the buggy address:
[ 25.242578] ffff880462eea800: fc fc 00 fc fc 00 fc fc 00 fc fc 00 fc fc fb fc
[ 25.242582] ffff880462eea880: fc fb fc fc fb fc fc 00 fc fc 00 fc fc 00 fc fc
[ 25.242586] >ffff880462eea900: 00 fc fc 00 fc fc 00 fc fc fb fc fc 00 fc fc fc
[ 25.242588] ^
[ 25.242592] ffff880462eea980: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 25.242596] ffff880462eeaa00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 25.242597] ==================================================================
Fixes: 96db776a3682 ("i40e/i40evf: fix interrupt affinity bug")
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
---
This should be considered for -stable, back to 4.10.
drivers/net/ethernet/intel/i40e/i40e_main.c | 2 +-
drivers/net/ethernet/intel/i40evf/i40evf_main.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c
index 2db93d3f6d23..c0e42d162c7c 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_main.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
@@ -3495,7 +3495,7 @@ static void i40e_irq_affinity_notify(struct irq_affinity_notify *notify,
struct i40e_q_vector *q_vector =
container_of(notify, struct i40e_q_vector, affinity_notify);
- q_vector->affinity_mask = *mask;
+ cpumask_copy(&q_vector->affinity_mask, mask);
}
/**
diff --git a/drivers/net/ethernet/intel/i40evf/i40evf_main.c b/drivers/net/ethernet/intel/i40evf/i40evf_main.c
index 7c213a347909..a4b60367ecce 100644
--- a/drivers/net/ethernet/intel/i40evf/i40evf_main.c
+++ b/drivers/net/ethernet/intel/i40evf/i40evf_main.c
@@ -520,7 +520,7 @@ static void i40evf_irq_affinity_notify(struct irq_affinity_notify *notify,
struct i40e_q_vector *q_vector =
container_of(notify, struct i40e_q_vector, affinity_notify);
- q_vector->affinity_mask = *mask;
+ cpumask_copy(&q_vector->affinity_mask, mask);
}
/**
--
2.9.4
^ permalink raw reply related
* Re: [PATCH] i40e{,vf}: Fix out-of-bound cpumask read in IRQ affinity handler
From: Stefano Brivio @ 2017-08-15 10:33 UTC (permalink / raw)
To: Jeff Kirsher, netdev, intel-wired-lan
Cc: David S . Miller, Alan Brady, Stefan Assmann
In-Reply-To: <ae9c9586f61e914dc1c6fe2e6ac1fb2bf07283bc.1502792828.git.sbrivio@redhat.com>
On Tue, 15 Aug 2017 12:30:14 +0200
Stefano Brivio <sbrivio@redhat.com> wrote:
> The cpumask used in i40e{,vf}_irq_affinity_notify() is allocated
> by irq_affinity_notify() with alloc_cpumask_var(), which doesn't
> allocate NR_CPUS bits, but only nr_cpumask_bits bits. If we just
> dereference it, we'll read way more than what is allocated, e.g.
> 1024 bytes vs. 8 bytes allocated on x86_64 machine with 24 CPUs.
Sorry, just two minutes after sending this I noticed Juergen submitted
the same fixes on Saturday:
From: Juergen Gross <jgross@suse.com>
To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org
Cc: jeffrey.t.kirsher@intel.com, Juergen Gross <jgross@suse.com>, stable@vger.kernel.org
Subject: [PATCH] net/i40e: use cpumask_copy() for assigning cpumask
Date: Sat, 12 Aug 2017 18:09:46 +0200
Please discard.
^ permalink raw reply
* RE: [PATCH] New Chapter on CodingStyle .
From: David Laight @ 2017-08-15 10:42 UTC (permalink / raw)
To: 'Jonathan Corbet', Corcodel Marian
Cc: netdev@vger.kernel.org, linux-doc@vger.kernel.org,
linux-kernel@vger.kernel.org, Realtek linux nic maintainers
In-Reply-To: <20170812085439.573c6c55@lwn.net>
From: Jonathan Corbet
> Sent: 12 August 2017 15:55
...
> > + Chapter 20: Put values on initialisers without exception
> > +
> > +When declaring variables on functions must put values:
>
> Thanks for sending a patch for the kernel's documentation.
> Unfortunately, I can't accept this patch for a couple of reasons:
...
> - The coding style document is there to describe the community's
> standards for kernel code. It is *not* a mechanism for imposing new
> standards. If you really think that the kernel community should adopt
> this rule, you will need to argue for it on the mailing lists. I will
> say, though, that I do not expect that this effort would be successful.
I'd even go as far as suggesting almost the opposite.
Declarations should only have initialisers if the value is constant.
David
^ permalink raw reply
* [PATCH v2 net-next 3/3] tls: Use correct sk->sk_prot for IPV6
From: Ilya Lesokhin @ 2017-08-15 11:08 UTC (permalink / raw)
To: netdev, davem; +Cc: davejwatson, aviadye, Ilya Lesokhin, Boris Pismenny
In-Reply-To: <1502795320-22538-1-git-send-email-ilyal@mellanox.com>
The tls ulp overrides sk->prot with a new tls specific proto structs.
The tls specific structs were previously based on the ipv4 specific
tcp_prot sturct.
As a result, attaching the tls ulp to an ipv6 tcp socket replaced
some ipv6 callback with the ipv4 equivalents.
This patch adds ipv6 tls proto structs and uses them when
attached to ipv6 sockets.
Signed-off-by: Boris Pismenny <borisp@mellanox.com>
Signed-off-by: Ilya Lesokhin <ilyal@mellanox.com>
---
net/tls/Kconfig | 1 +
net/tls/tls_main.c | 50 ++++++++++++++++++++++++++++++++++++++------------
2 files changed, 39 insertions(+), 12 deletions(-)
diff --git a/net/tls/Kconfig b/net/tls/Kconfig
index eb58303..7e9cf8b 100644
--- a/net/tls/Kconfig
+++ b/net/tls/Kconfig
@@ -7,6 +7,7 @@ config TLS
select CRYPTO
select CRYPTO_AES
select CRYPTO_GCM
+ select IPV6
default n
---help---
Enable kernel support for TLS protocol. This allows symmetric
diff --git a/net/tls/tls_main.c b/net/tls/tls_main.c
index 60aff60..9caad11 100644
--- a/net/tls/tls_main.c
+++ b/net/tls/tls_main.c
@@ -40,13 +40,25 @@
#include <linux/sched/signal.h>
#include <net/tls.h>
+#include <net/transp_v6.h>
MODULE_AUTHOR("Mellanox Technologies");
MODULE_DESCRIPTION("Transport Layer Security Support");
MODULE_LICENSE("Dual BSD/GPL");
-static struct proto tls_base_prot;
-static struct proto tls_sw_prot;
+enum {
+ TLSV4,
+ TLSV6,
+ TLS_NUM_PROTS,
+};
+
+enum {
+ TLS_BASE_TX,
+ TLS_SW_TX,
+ TLS_NUM_CONFIG,
+};
+
+static struct proto tls_prots[TLS_NUM_PROTS][TLS_NUM_CONFIG];
int wait_on_pending_writer(struct sock *sk, long *timeo)
{
@@ -342,6 +354,7 @@ static int do_tls_setsockopt_tx(struct sock *sk, char __user *optval,
struct tls_context *ctx = tls_get_ctx(sk);
struct proto *prot = NULL;
int rc = 0;
+ int ip_ver = sk->sk_family == AF_INET6 ? TLSV6 : TLSV4;
if (!optval || (optlen < sizeof(*crypto_info))) {
rc = -EINVAL;
@@ -396,7 +409,7 @@ static int do_tls_setsockopt_tx(struct sock *sk, char __user *optval,
/* currently SW is default, we will have ethtool in future */
rc = tls_set_sw_offload(sk, ctx);
- prot = &tls_sw_prot;
+ prot = &tls_prots[ip_ver][TLS_SW_TX];
if (rc)
goto err_crypto_info;
@@ -443,6 +456,12 @@ static int tls_init(struct sock *sk)
struct inet_connection_sock *icsk = inet_csk(sk);
struct tls_context *ctx;
int rc = 0;
+ int ip_ver = TLSV4;
+
+ if (sk->sk_prot == &tcpv6_prot)
+ ip_ver = TLSV6;
+ else if (sk->sk_prot != &tcp_prot)
+ return -EINVAL;
/* allocate tls context */
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
@@ -453,7 +472,8 @@ static int tls_init(struct sock *sk)
icsk->icsk_ulp_data = ctx;
ctx->setsockopt = sk->sk_prot->setsockopt;
ctx->getsockopt = sk->sk_prot->getsockopt;
- sk->sk_prot = &tls_base_prot;
+
+ sk->sk_prot = &tls_prots[ip_ver][TLS_BASE_TX];
out:
return rc;
}
@@ -464,16 +484,22 @@ static int tls_init(struct sock *sk)
.init = tls_init,
};
+static void build_protos(struct proto *prot, struct proto *base)
+{
+ prot[TLS_BASE_TX] = *base;
+ prot[TLS_BASE_TX].setsockopt = tls_setsockopt;
+ prot[TLS_BASE_TX].getsockopt = tls_getsockopt;
+
+ prot[TLS_SW_TX] = prot[TLS_BASE_TX];
+ prot[TLS_SW_TX].close = tls_sk_proto_close;
+ prot[TLS_SW_TX].sendmsg = tls_sw_sendmsg;
+ prot[TLS_SW_TX].sendpage = tls_sw_sendpage;
+}
+
static int __init tls_register(void)
{
- tls_base_prot = tcp_prot;
- tls_base_prot.setsockopt = tls_setsockopt;
- tls_base_prot.getsockopt = tls_getsockopt;
-
- tls_sw_prot = tls_base_prot;
- tls_sw_prot.sendmsg = tls_sw_sendmsg;
- tls_sw_prot.sendpage = tls_sw_sendpage;
- tls_sw_prot.close = tls_sk_proto_close;
+ build_protos(tls_prots[TLSV4], &tcp_prot);
+ build_protos(tls_prots[TLSV6], &tcpv6_prot);
tcp_register_ulp(&tcp_tls_ulp_ops);
--
1.8.3.1
^ permalink raw reply related
* [PATCH v2 net-next 0/3] Use correct sk->sk_prot for IPV6
From: Ilya Lesokhin @ 2017-08-15 11:08 UTC (permalink / raw)
To: netdev, davem; +Cc: davejwatson, aviadye, Ilya Lesokhin
The tls ulp overrides sk->prot with a new tls specific proto structs.
The tls specific structs were previously based on the ipv4 specific
tcp_prot sturct.
As a result, attaching the tls ulp to an ipv6 tcp socket replaced
some ipv6 callback with the ipv4 equivalents.
This patch adds ipv6 tls proto structs and uses them when
attached to ipv6 sockets.
Changes since v1:
- TLS now dependes on IPV6
This fixes complication issues when TLS is built-in and IPV6 is a module.
The downside should be small as it is unlikely that there are kernel TLS
users who can't afford to include IPV6 in thier kernel.
- tls_init now checks sk->sk_prot directly
This is somewhat safer then checking indirectly through sk->sk_family
Ilya Lesokhin (3):
ipv6: Prevent unexpected sk->sk_prot changes
net: Export tcpv6_prot
tls: Use correct sk->sk_prot for IPV6
net/ipv6/ipv6_sockglue.c | 12 ++++++++++++
net/ipv6/tcp_ipv6.c | 1 +
net/tls/Kconfig | 1 +
net/tls/tls_main.c | 50 ++++++++++++++++++++++++++++++++++++------------
4 files changed, 52 insertions(+), 12 deletions(-)
--
1.8.3.1
^ permalink raw reply
* [PATCH v2 net-next 2/3] net: Export tcpv6_prot
From: Ilya Lesokhin @ 2017-08-15 11:08 UTC (permalink / raw)
To: netdev, davem; +Cc: davejwatson, aviadye, Ilya Lesokhin, Boris Pismenny
In-Reply-To: <1502795320-22538-1-git-send-email-ilyal@mellanox.com>
Want to be able to use these in TLS.
Signed-off-by: Boris Pismenny <borisp@mellanox.com>
---
net/ipv6/tcp_ipv6.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 2521690..ef8d5b4 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1944,6 +1944,7 @@ struct proto tcpv6_prot = {
#endif
.diag_destroy = tcp_abort,
};
+EXPORT_SYMBOL_GPL(tcpv6_prot);
static struct inet6_protocol tcpv6_protocol = {
.early_demux = tcp_v6_early_demux,
--
1.8.3.1
^ permalink raw reply related
* [PATCH v2 net-next 1/3] ipv6: Prevent unexpected sk->sk_prot changes
From: Ilya Lesokhin @ 2017-08-15 11:08 UTC (permalink / raw)
To: netdev, davem; +Cc: davejwatson, aviadye, Ilya Lesokhin, Boris Pismenny
In-Reply-To: <1502795320-22538-1-git-send-email-ilyal@mellanox.com>
With this patch IPV6 code ensure that only sockets with the
expected sk->sk_prot are converted to IPV4.
Signed-off-by: Boris Pismenny <borisp@mellanox.com>
---
net/ipv6/ipv6_sockglue.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/net/ipv6/ipv6_sockglue.c b/net/ipv6/ipv6_sockglue.c
index 02d795f..318cd344 100644
--- a/net/ipv6/ipv6_sockglue.c
+++ b/net/ipv6/ipv6_sockglue.c
@@ -174,6 +174,7 @@ static int do_ipv6_setsockopt(struct sock *sk, int level, int optname,
if (val == PF_INET) {
struct ipv6_txoptions *opt;
struct sk_buff *pktopt;
+ struct proto *expected_prot;
if (sk->sk_type == SOCK_RAW)
break;
@@ -199,6 +200,17 @@ static int do_ipv6_setsockopt(struct sock *sk, int level, int optname,
break;
}
+ if (sk->sk_protocol == IPPROTO_TCP &&
+ sk->sk_prot != &tcpv6_prot)
+ break;
+
+ expected_prot = &udpv6_prot;
+ if (sk->sk_protocol == IPPROTO_UDPLITE)
+ expected_prot = &udplitev6_prot;
+
+ if (sk->sk_prot != expected_prot)
+ break;
+
fl6_free_socklist(sk);
__ipv6_sock_mc_close(sk);
--
1.8.3.1
^ permalink raw reply related
* [PATCH net] ipv6: fix NULL dereference in ip6_route_dev_notify()
From: Eric Dumazet @ 2017-08-15 11:09 UTC (permalink / raw)
To: David Miller; +Cc: netdev
From: Eric Dumazet <edumazet@google.com>
Based on a syzkaller report [1], I found that a per cpu allocation
failure in snmp6_alloc_dev() would then lead to NULL dereference in
ip6_route_dev_notify().
It seems this is a very old bug, thus no Fixes tag in this submission.
Let's add in6_dev_put_clear() helper, as we will probably use
it elsewhere (once available/present in net-next)
[1]
kasan: CONFIG_KASAN_INLINE enabled
kasan: GPF could be caused by NULL-ptr deref or user memory access
general protection fault: 0000 [#1] SMP KASAN
Dumping ftrace buffer:
(ftrace buffer empty)
Modules linked in:
CPU: 1 PID: 17294 Comm: syz-executor6 Not tainted 4.13.0-rc2+ #10
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
task: ffff88019f456680 task.stack: ffff8801c6e58000
RIP: 0010:__read_once_size include/linux/compiler.h:250 [inline]
RIP: 0010:atomic_read arch/x86/include/asm/atomic.h:26 [inline]
RIP: 0010:refcount_sub_and_test+0x7d/0x1b0 lib/refcount.c:178
RSP: 0018:ffff8801c6e5f1b0 EFLAGS: 00010202
RAX: 0000000000000037 RBX: dffffc0000000000 RCX: ffffc90005d25000
RDX: ffff8801c6e5f218 RSI: ffffffff82342bbf RDI: 0000000000000001
RBP: ffff8801c6e5f240 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 1ffff10038dcbe37
R13: 0000000000000006 R14: 0000000000000001 R15: 00000000000001b8
FS: 00007f21e0429700(0000) GS:ffff8801dc100000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001ddbc22000 CR3: 00000001d632b000 CR4: 00000000001426e0
DR0: 0000000020000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Call Trace:
refcount_dec_and_test+0x1a/0x20 lib/refcount.c:211
in6_dev_put include/net/addrconf.h:335 [inline]
ip6_route_dev_notify+0x1c9/0x4a0 net/ipv6/route.c:3732
notifier_call_chain+0x136/0x2c0 kernel/notifier.c:93
__raw_notifier_call_chain kernel/notifier.c:394 [inline]
raw_notifier_call_chain+0x2d/0x40 kernel/notifier.c:401
call_netdevice_notifiers_info+0x51/0x90 net/core/dev.c:1678
call_netdevice_notifiers net/core/dev.c:1694 [inline]
rollback_registered_many+0x91c/0xe80 net/core/dev.c:7107
rollback_registered+0x1be/0x3c0 net/core/dev.c:7149
register_netdevice+0xbcd/0xee0 net/core/dev.c:7587
register_netdev+0x1a/0x30 net/core/dev.c:7669
loopback_net_init+0x76/0x160 drivers/net/loopback.c:214
ops_init+0x10a/0x570 net/core/net_namespace.c:118
setup_net+0x313/0x710 net/core/net_namespace.c:294
copy_net_ns+0x27c/0x580 net/core/net_namespace.c:418
create_new_namespaces+0x425/0x880 kernel/nsproxy.c:107
unshare_nsproxy_namespaces+0xae/0x1e0 kernel/nsproxy.c:206
SYSC_unshare kernel/fork.c:2347 [inline]
SyS_unshare+0x653/0xfa0 kernel/fork.c:2297
entry_SYSCALL_64_fastpath+0x1f/0xbe
RIP: 0033:0x4512c9
RSP: 002b:00007f21e0428c08 EFLAGS: 00000216 ORIG_RAX: 0000000000000110
RAX: ffffffffffffffda RBX: 0000000000718150 RCX: 00000000004512c9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000062020200
RBP: 0000000000000086 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000216 R12: 00000000004b973d
R13: 00000000ffffffff R14: 000000002001d000 R15: 00000000000002dd
Code: 50 2b 34 82 c7 00 f1 f1 f1 f1 c7 40 04 04 f2 f2 f2 c7 40 08 f3 f3
f3 f3 e8 a1 43 39 ff 4c 89 f8 48 8b 95 70 ff ff ff 48 c1 e8 03 <0f> b6
0c 18 4c 89 f8 83 e0 07 83 c0 03 38 c8 7c 08 84 c9 0f 85
RIP: __read_once_size include/linux/compiler.h:250 [inline] RSP:
ffff8801c6e5f1b0
RIP: atomic_read arch/x86/include/asm/atomic.h:26 [inline] RSP:
ffff8801c6e5f1b0
RIP: refcount_sub_and_test+0x7d/0x1b0 lib/refcount.c:178 RSP:
ffff8801c6e5f1b0
---[ end trace e441d046c6410d31 ]---
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reported-by: Dmitry Vyukov <dvyukov@google.com>
---
include/net/addrconf.h | 10 ++++++++++
net/ipv6/route.c | 6 +++---
2 files changed, 13 insertions(+), 3 deletions(-)
diff --git a/include/net/addrconf.h b/include/net/addrconf.h
index 6df79e96a780..f44ff2476758 100644
--- a/include/net/addrconf.h
+++ b/include/net/addrconf.h
@@ -336,6 +336,16 @@ static inline void in6_dev_put(struct inet6_dev *idev)
in6_dev_finish_destroy(idev);
}
+static inline void in6_dev_put_clear(struct inet6_dev **pidev)
+{
+ struct inet6_dev *idev = *pidev;
+
+ if (idev) {
+ in6_dev_put(idev);
+ *pidev = NULL;
+ }
+}
+
static inline void __in6_dev_put(struct inet6_dev *idev)
{
refcount_dec(&idev->refcnt);
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 99d4727f2b18..94d6a13d47f0 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -3721,10 +3721,10 @@ static int ip6_route_dev_notify(struct notifier_block *this,
/* NETDEV_UNREGISTER could be fired for multiple times by
* netdev_wait_allrefs(). Make sure we only call this once.
*/
- in6_dev_put(net->ipv6.ip6_null_entry->rt6i_idev);
+ in6_dev_put_clear(&net->ipv6.ip6_null_entry->rt6i_idev);
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
- in6_dev_put(net->ipv6.ip6_prohibit_entry->rt6i_idev);
- in6_dev_put(net->ipv6.ip6_blk_hole_entry->rt6i_idev);
+ in6_dev_put_clear(&net->ipv6.ip6_prohibit_entry->rt6i_idev);
+ in6_dev_put_clear(&net->ipv6.ip6_blk_hole_entry->rt6i_idev);
#endif
}
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox