Performance problem with bond interface

Performance problem with bond interface

[Venkat Venkatsubra]

We see a performance problem when the slaves of the bond 
don't support checksum offload features. What we see is
tcp_sendmsg's skb_add_data_nocache ending up not using the
csum_and_copy_from_user which would have computed the
checksum while copying from user buffer to kernel buffer.
Instead it computes later in dev_hard_start_xmit when it
figures out the slave doesn't support checksum offload and
ends up expensive . 

The bonding interface's "features" has  NETIF_F_HW_CSUM 
(or NETIF_F_NO_CSUM in 2.6.39) set which makes the 
stack think checksum need not be computed in software.
/*
* Check whether we can use HW checksum.
*/
if (sk->sk_route_caps & NETIF_F_ALL_CSUM)
     skb->ip_summed = CHECKSUM_PARTIAL;

But later in dev_hard_start_xmit it finds out the slave does not
support checksumming and decides to compute in software.

/* If packet is not checksummed and device does not
* support checksumming for this protocol, complete
* checksumming here.
*/
if (skb->ip_summed == CHECKSUM_PARTIAL) {
         skb_set_transport_header(skb,
              skb_checksum_start_offset(skb));
        if (!(features & NETIF_F_ALL_CSUM) &&
             skb_checksum_help(skb))
               goto out_kfree_skb;
}

We see this problem after this commit:
commit 1742f183fc218798dab6fcf0ded25b6608fc0a48
Author: MichaÅ<82> MirosÅ<82>aw <mirq-linux@rere.qmqm.pl>
Date:   Fri Apr 22 06:31:16 2011 +0000

    net: fix netdev_increment_features()

    Simplify and fix netdev_increment_features() to conform to what is
    stated in netdevice.h comments about NETIF_F_ONE_FOR_ALL.
    Include FCoE segmentation and VLAN-challedged flags in computation.

    Signed-off-by: MichaÅ<82> MirosÅ<82>aw <mirq-linux@rere.qmqm.pl>
    Signed-off-by: David S. Miller <davem@davemloft.net>

Prior to that the below code in netdev_increment_features was helping in
turning off NETIF_F_NO_CSUM on bond when the slaves don't support it: 
/* If device needs checksumming, downgrade to it. */
if (all & NETIF_F_NO_CSUM && !(one & NETIF_F_NO_CSUM))
        all ^= NETIF_F_NO_CSUM | (one & NETIF_F_ALL_CSUM);

The slaves are Mellanox IB adapters. This is on x86_64 platform.

Please let us know if you need any additional information.

Thanks.

Venkat

Re: Performance problem with bond interface

[Eric Dumazet]

On Mon, 2014-04-21 at 07:23 -0700, Venkat Venkatsubra wrote:
> We see a performance problem when the slaves of the bond 
> don't support checksum offload features. What we see is
> tcp_sendmsg's skb_add_data_nocache ending up not using the
> csum_and_copy_from_user which would have computed the
> checksum while copying from user buffer to kernel buffer.
> Instead it computes later in dev_hard_start_xmit when it
> figures out the slave doesn't support checksum offload and
> ends up expensive . 
> 
> The bonding interface's "features" has  NETIF_F_HW_CSUM 
> (or NETIF_F_NO_CSUM in 2.6.39) set which makes the 
> stack think checksum need not be computed in software.
> /*
> * Check whether we can use HW checksum.
> */
> if (sk->sk_route_caps & NETIF_F_ALL_CSUM)
>      skb->ip_summed = CHECKSUM_PARTIAL;
> 
> But later in dev_hard_start_xmit it finds out the slave does not
> support checksumming and decides to compute in software.
> 
> /* If packet is not checksummed and device does not
> * support checksumming for this protocol, complete
> * checksumming here.
> */
> if (skb->ip_summed == CHECKSUM_PARTIAL) {
>          skb_set_transport_header(skb,
>               skb_checksum_start_offset(skb));
>         if (!(features & NETIF_F_ALL_CSUM) &&
>              skb_checksum_help(skb))
>                goto out_kfree_skb;
> }
> 
> We see this problem after this commit:
> commit 1742f183fc218798dab6fcf0ded25b6608fc0a48
> Author: MichaÅ<82> MirosÅ<82>aw <mirq-linux@rere.qmqm.pl>
> Date:   Fri Apr 22 06:31:16 2011 +0000
> 
>     net: fix netdev_increment_features()
> 
>     Simplify and fix netdev_increment_features() to conform to what is
>     stated in netdevice.h comments about NETIF_F_ONE_FOR_ALL.
>     Include FCoE segmentation and VLAN-challedged flags in computation.
> 
>     Signed-off-by: MichaÅ<82> MirosÅ<82>aw <mirq-linux@rere.qmqm.pl>
>     Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> Prior to that the below code in netdev_increment_features was helping in
> turning off NETIF_F_NO_CSUM on bond when the slaves don't support it: 
> /* If device needs checksumming, downgrade to it. */
> if (all & NETIF_F_NO_CSUM && !(one & NETIF_F_NO_CSUM))
>         all ^= NETIF_F_NO_CSUM | (one & NETIF_F_ALL_CSUM);
> 
> The slaves are Mellanox IB adapters. This is on x86_64 platform.
> 
> Please let us know if you need any additional information.

Please CC patch author (I did), instead of sending this to hundred of
people (linux-kernel ??? netdev is more appropriate...)

Do these NIC really not support TX checksum ?

You did not provide kernel version you use.

Please also provide : (using a recent ethtool to get extended offload
info)

ethtool -k bond0 # or the bonding device name

ethtool -k eth1  # or the slave name

ethtool -i eth1

Thanks !

RE: Performance problem with bond interface

[Venkat Venkatsubra]

Hi Eric,
> 
> Do these NIC really not support TX checksum ?
> 

In IPoIB connected (IB RC) mode the TX checksum is not supported.
In datagram mode it is supported.

> You did not provide kernel version you use.

So far we have tested against 2.6.39 and 3.8.13.
 
> 
> Please also provide : (using a recent ethtool to get extended offload
> info)
> 
> ethtool -k bond0 # or the bonding device name
> 
# ethtool -k bond0 | more
Features for bond0:
rx-checksumming: off [fixed]
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off [requested on]
        tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp-ecn-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: on
vlan-challenged: on [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: on [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]

> ethtool -k eth1  # or the slave name
> 
# ethtool -k ib0 | more
Features for ib0:
rx-checksumming: on
tx-checksumming: off
        tx-checksum-ipv4: off [requested on]
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off [requested on]
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: on [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]

> ethtool -i eth1
> 
# ethtool -i ib0
driver: ipoib
version:
firmware-version:
bus-info:
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

2nd slave:
-----------
# ethtool -k ib1 | more
Features for ib1:
rx-checksumming: on
tx-checksumming: off
        tx-checksum-ipv4: off [requested on]
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off [requested on]
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: on [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]

# ethtool -i ib1
driver: ipoib
version:
firmware-version:
bus-info:
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

> Thanks !
> 
> 

Thanks for looking at it.

Venkat

RE: Performance problem with bond interface

[Venkat Venkatsubra]

What is the purpose of initializing "features"
with NETIF_F_HW_CSUM in bond_setup() ?

[drivers/net/bonding/bond_main.c] bond-setup():
bond_dev->hw_features &= ~(NETIF_F_ALL_CSUM & ~NETIF_F_HW_CSUM);
bond_dev->features |= bond_dev->hw_features;

If netdev_increment_features() adds features in
incremental fashion then shouldn't bond let the
features such as NETIF_F_HW_CSUM be added when
netdev_increment_features() is called for each slave
in bond_compute_features() if the slave supports it ?

Thanks.

Venkat