Re: A question about FAILOVER event in RoCE LAG

[Mark Bloch <mbloch@nvidia.com>]

On 26/02/2023 14:53, Leon Romanovsky wrote:
> +Mark
> 
> On Fri, Feb 24, 2023 at 11:14:47AM +0000, huangjunxian (C) wrote:
>> Hi folks!
>>
>> We've been working on LAG in hns RoCE driver, and we notice that when a FAILOVER event
>> occurs in active-backup mode, all GIDs of the RDMA bond device are deleted and new GIDs
>> are added, triggered by the event handler listed below.
>>
>> So, when a FAILOVER event occurs on a RDMA bond device with running traffic, does it make
>> sense that the traffic is terminated since its GIDs are deleted?

Yep, please read the original commit message:

commit 238fdf48f2b54a01cedb5774c3a1e81c94e1a3a0
Author: Matan Barak <matanb@mellanox.com>
Date:   Thu Jul 30 18:33:27 2015 +0300

    IB/core: Add RoCE table bonding support

    Handling bonding and other devices require us to all all GIDs of the
    net-devices which are upper-devices of the RoCE port related
    net-device.

    Active-backup configurations imposes even more challenges as the
    default GID should only be set on the active devices (this is
    necessary as otherwise the same MAC could be used for several
    slaves and thus several slaves will have identical GIDs).

    Managing these configurations are done by listening to:
    (a) NETDEV_CHANGEUPPER event
            (1) if a related net-device is linked, delete all inactive
                slaves default GIDs and add the upper device GIDs.
            (2) if a related net-device is unlinked, delete all upper GIDs
                and add the default GIDs.
    (b) NETDEV_BONDING_FAILOVER:
            (1) delete the bond GIDs from inactive slaves
            (2) delete the inactive slave's default GIDs
            (3) Add the bond GIDs to the active slave.

    Signed-off-by: Matan Barak <matanb@mellanox.com>
    Signed-off-by: Doug Ledford <dledford@redhat.com>

and please read: https://wiki.linuxfoundation.org/networking/bonding
especially the section that explains some of the restrictions of
active-backup mode.

Mark

>>
>> The FAILOVER event handler mentioned above:
>> static int netdevice_event(struct notifier_block *this, unsigned long event, void *ptr)
>> {
>>          ......
>>          static const struct netdev_event_work_cmd bonding_event_ips_del_cmd = {
>>                   .cb = del_netdev_upper_ips, .filter = upper_device_filter};
>>          ......
>>          switch (event) {
>>          ......
>>          case NETDEV_BONDING_FAILOVER:
>>                   cmds[0] = bonding_event_ips_del_cmd;
>>                   /* Add default GIDs of the bond device */
>>                   cmds[1] = bonding_default_add_cmd;
>>                   /* Add IP based GIDs of the bond device */
>>                   cmds[2] = add_cmd_upper_ips;
>>                   break;
>>          ......
>>          }
>>          ......
>> }
>>
>> Thanks,
>> Junxian