KSZ9477 HSR Offloading

KSZ9477 HSR Offloading

[Frieder Schrempf]

Hi,

I'm trying out HSR support on KSZ9477 with v6.12. My setup looks like this:

+-------------+         +-------------+
|             |         |             |
|   Node A    |         |   Node D    |
|             |         |             |
|             |         |             |
| LAN1   LAN2 |         | LAN1   LAN2 |
+--+-------+--+         +--+------+---+
   |       |               |      |
   |       +---------------+      |
   |                              |
   |       +---------------+      |
   |       |               |      |
+--+-------+--+         +--+------+---+
| LAN1   LAN2 |         | LAN1   LAN2 |
|             |         |             |
|             |         |             |
|   Node B    |         |   Node C    |
|             |         |             |
+-------------+         +-------------+

On each device the LAN1 and LAN2 are added as HSR slaves. Then I try to
do ping tests between each of the HSR interfaces.

The result is that I can reach the neighboring nodes just fine, but I
can't reach the remote node that needs packages to be forwarded through
the other nodes. For example I can't ping from node A to C.

I've tried to disable HW offloading in the driver and then everything
starts working.

Is this a problem with HW offloading in the KSZ driver, or am I missing
something essential?

Thanks!
Frieder

Re: KSZ9477 HSR Offloading

[Andrew Lunn]

On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf wrote:
> Hi,
> 
> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks like this:
> 
> +-------------+         +-------------+
> |             |         |             |
> |   Node A    |         |   Node D    |
> |             |         |             |
> |             |         |             |
> | LAN1   LAN2 |         | LAN1   LAN2 |
> +--+-------+--+         +--+------+---+
>    |       |               |      |
>    |       +---------------+      |
>    |                              |
>    |       +---------------+      |
>    |       |               |      |
> +--+-------+--+         +--+------+---+
> | LAN1   LAN2 |         | LAN1   LAN2 |
> |             |         |             |
> |             |         |             |
> |   Node B    |         |   Node C    |
> |             |         |             |
> +-------------+         +-------------+
> 
> On each device the LAN1 and LAN2 are added as HSR slaves. Then I try to
> do ping tests between each of the HSR interfaces.
> 
> The result is that I can reach the neighboring nodes just fine, but I
> can't reach the remote node that needs packages to be forwarded through
> the other nodes. For example I can't ping from node A to C.
> 
> I've tried to disable HW offloading in the driver and then everything
> starts working.
> 
> Is this a problem with HW offloading in the KSZ driver, or am I missing
> something essential?

How are IP addresses configured? I assume you have a bridge, LAN1 and
LAN2 are members of the bridge, and the IP address is on the bridge
interface?

	Andrew

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

Hi Andrew,

On 28.01.25 6:51 PM, Andrew Lunn wrote:
> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf wrote:
>> Hi,
>>
>> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks like this:
>>
>> +-------------+         +-------------+
>> |             |         |             |
>> |   Node A    |         |   Node D    |
>> |             |         |             |
>> |             |         |             |
>> | LAN1   LAN2 |         | LAN1   LAN2 |
>> +--+-------+--+         +--+------+---+
>>    |       |               |      |
>>    |       +---------------+      |
>>    |                              |
>>    |       +---------------+      |
>>    |       |               |      |
>> +--+-------+--+         +--+------+---+
>> | LAN1   LAN2 |         | LAN1   LAN2 |
>> |             |         |             |
>> |             |         |             |
>> |   Node B    |         |   Node C    |
>> |             |         |             |
>> +-------------+         +-------------+
>>
>> On each device the LAN1 and LAN2 are added as HSR slaves. Then I try to
>> do ping tests between each of the HSR interfaces.
>>
>> The result is that I can reach the neighboring nodes just fine, but I
>> can't reach the remote node that needs packages to be forwarded through
>> the other nodes. For example I can't ping from node A to C.
>>
>> I've tried to disable HW offloading in the driver and then everything
>> starts working.
>>
>> Is this a problem with HW offloading in the KSZ driver, or am I missing
>> something essential?
> 
> How are IP addresses configured? I assume you have a bridge, LAN1 and
> LAN2 are members of the bridge, and the IP address is on the bridge
> interface?

I have a HSR interface on each node that covers LAN1 and LAN2 as slaves
and the IP addresses are on those HSR interfaces. For node A:

ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision 45
version 1
ip addr add 172.20.1.1/24 dev hsr

The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24 and
172.20.1.4/24 respectively.

Then on node A, I'm doing:

ping 172.20.1.2 # neighboring node B works
ping 172.20.1.4 # neighboring node D works
ping 172.20.1.3 # remote node C works only if I disable offloading

Thanks
Frieder

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

On 29.01.25 8:24 AM, Frieder Schrempf wrote:
> Hi Andrew,
> 
> On 28.01.25 6:51 PM, Andrew Lunn wrote:
>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf wrote:
>>> Hi,
>>>
>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks like this:
>>>
>>> +-------------+         +-------------+
>>> |             |         |             |
>>> |   Node A    |         |   Node D    |
>>> |             |         |             |
>>> |             |         |             |
>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>> +--+-------+--+         +--+------+---+
>>>    |       |               |      |
>>>    |       +---------------+      |
>>>    |                              |
>>>    |       +---------------+      |
>>>    |       |               |      |
>>> +--+-------+--+         +--+------+---+
>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>> |             |         |             |
>>> |             |         |             |
>>> |   Node B    |         |   Node C    |
>>> |             |         |             |
>>> +-------------+         +-------------+
>>>
>>> On each device the LAN1 and LAN2 are added as HSR slaves. Then I try to
>>> do ping tests between each of the HSR interfaces.
>>>
>>> The result is that I can reach the neighboring nodes just fine, but I
>>> can't reach the remote node that needs packages to be forwarded through
>>> the other nodes. For example I can't ping from node A to C.
>>>
>>> I've tried to disable HW offloading in the driver and then everything
>>> starts working.
>>>
>>> Is this a problem with HW offloading in the KSZ driver, or am I missing
>>> something essential?
>>
>> How are IP addresses configured? I assume you have a bridge, LAN1 and
>> LAN2 are members of the bridge, and the IP address is on the bridge
>> interface?
> 
> I have a HSR interface on each node that covers LAN1 and LAN2 as slaves
> and the IP addresses are on those HSR interfaces. For node A:
> 
> ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision 45
> version 1
> ip addr add 172.20.1.1/24 dev hsr
> 
> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24 and
> 172.20.1.4/24 respectively.
> 
> Then on node A, I'm doing:
> 
> ping 172.20.1.2 # neighboring node B works
> ping 172.20.1.4 # neighboring node D works
> ping 172.20.1.3 # remote node C works only if I disable offloading

BTW, it's enough to disable the offloading of the forwarding for HSR
frames to make it work.

--- a/drivers/net/dsa/microchip/ksz9477.c
+++ b/drivers/net/dsa/microchip/ksz9477.c
@@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct ksz_device
*dev, int port, u32 val)
  * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as HSR frames
  * can be forwarded in the switch fabric between HSR ports.
  */
-#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
NETIF_F_HW_HSR_FWD)
+#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)

 void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
net_device *hsr)
 {
@@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch *ds, int
port, struct net_device *hsr)
        /* Program which port(s) shall support HSR */
        ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port), BIT(port));

-       /* Forward frames between HSR ports (i.e. bridge together HSR
ports) */
-       if (dev->hsr_ports) {
-               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
-                       hsr_ports |= BIT(hsr_dp->index);
-
-               hsr_ports |= BIT(dsa_upstream_port(ds, port));
-               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
-                       ksz9477_cfg_port_member(dev, hsr_dp->index,
hsr_ports);
-       }
-
        if (!dev->hsr_ports) {
                /* Enable discarding of received HSR frames */
                ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);

Re: KSZ9477 HSR Offloading

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 4812 bytes --]

Hi Frieder,

> On 29.01.25 8:24 AM, Frieder Schrempf wrote:
> > Hi Andrew,
> > 
> > On 28.01.25 6:51 PM, Andrew Lunn wrote:  
> >> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf wrote:  
> >>> Hi,
> >>>
> >>> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks
> >>> like this:
> >>>
> >>> +-------------+         +-------------+
> >>> |             |         |             |
> >>> |   Node A    |         |   Node D    |
> >>> |             |         |             |
> >>> |             |         |             |
> >>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>> +--+-------+--+         +--+------+---+
> >>>    |       |               |      |
> >>>    |       +---------------+      |
> >>>    |                              |
> >>>    |       +---------------+      |
> >>>    |       |               |      |
> >>> +--+-------+--+         +--+------+---+
> >>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>> |             |         |             |
> >>> |             |         |             |
> >>> |   Node B    |         |   Node C    |
> >>> |             |         |             |
> >>> +-------------+         +-------------+
> >>>
> >>> On each device the LAN1 and LAN2 are added as HSR slaves. Then I
> >>> try to do ping tests between each of the HSR interfaces.
> >>>
> >>> The result is that I can reach the neighboring nodes just fine,
> >>> but I can't reach the remote node that needs packages to be
> >>> forwarded through the other nodes. For example I can't ping from
> >>> node A to C.
> >>>
> >>> I've tried to disable HW offloading in the driver and then
> >>> everything starts working.
> >>>
> >>> Is this a problem with HW offloading in the KSZ driver, or am I
> >>> missing something essential?  

Thanks for looking and testing such large scale setup.

> >>
> >> How are IP addresses configured? I assume you have a bridge, LAN1
> >> and LAN2 are members of the bridge, and the IP address is on the
> >> bridge interface?  
> > 
> > I have a HSR interface on each node that covers LAN1 and LAN2 as
> > slaves and the IP addresses are on those HSR interfaces. For node A:
> > 
> > ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision 45
> > version 1
> > ip addr add 172.20.1.1/24 dev hsr
> > 
> > The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24 and
> > 172.20.1.4/24 respectively.
> > 
> > Then on node A, I'm doing:
> > 
> > ping 172.20.1.2 # neighboring node B works
> > ping 172.20.1.4 # neighboring node D works
> > ping 172.20.1.3 # remote node C works only if I disable offloading  
> 
> BTW, it's enough to disable the offloading of the forwarding for HSR
> frames to make it work.
> 
> --- a/drivers/net/dsa/microchip/ksz9477.c
> +++ b/drivers/net/dsa/microchip/ksz9477.c
> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct ksz_device
> *dev, int port, u32 val)
>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as HSR
> frames
>   * can be forwarded in the switch fabric between HSR ports.
>   */
> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
> NETIF_F_HW_HSR_FWD)
> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
> 
>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
> net_device *hsr)
>  {
> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch *ds,
> int port, struct net_device *hsr)
>         /* Program which port(s) shall support HSR */
>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port), BIT(port));
> 
> -       /* Forward frames between HSR ports (i.e. bridge together HSR
> ports) */
> -       if (dev->hsr_ports) {
> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> -                       hsr_ports |= BIT(hsr_dp->index);
> -
> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> -                       ksz9477_cfg_port_member(dev, hsr_dp->index,
> hsr_ports);
> -       }
> -
>         if (!dev->hsr_ports) {
>                 /* Enable discarding of received HSR frames */
>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);

This means that KSZ9477 forwarding is dropping frames when HW
acceleration is used (for non "neighbour" nodes).

On my setup I only had 2 KSZ9477 devel boards.

And as you wrote - the SW based one works, so extending
https://elixir.bootlin.com/linux/v6.12-rc2/source/tools/testing/selftests/net/hsr

would not help in this case.


Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

Hi Lukasz,

On 29.01.25 12:17 PM, Lukasz Majewski wrote:
> Hi Frieder,
> 
>> On 29.01.25 8:24 AM, Frieder Schrempf wrote:
>>> Hi Andrew,
>>>
>>> On 28.01.25 6:51 PM, Andrew Lunn wrote:  
>>>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf wrote:  
>>>>> Hi,
>>>>>
>>>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks
>>>>> like this:
>>>>>
>>>>> +-------------+         +-------------+
>>>>> |             |         |             |
>>>>> |   Node A    |         |   Node D    |
>>>>> |             |         |             |
>>>>> |             |         |             |
>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>> +--+-------+--+         +--+------+---+
>>>>>    |       |               |      |
>>>>>    |       +---------------+      |
>>>>>    |                              |
>>>>>    |       +---------------+      |
>>>>>    |       |               |      |
>>>>> +--+-------+--+         +--+------+---+
>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>> |             |         |             |
>>>>> |             |         |             |
>>>>> |   Node B    |         |   Node C    |
>>>>> |             |         |             |
>>>>> +-------------+         +-------------+
>>>>>
>>>>> On each device the LAN1 and LAN2 are added as HSR slaves. Then I
>>>>> try to do ping tests between each of the HSR interfaces.
>>>>>
>>>>> The result is that I can reach the neighboring nodes just fine,
>>>>> but I can't reach the remote node that needs packages to be
>>>>> forwarded through the other nodes. For example I can't ping from
>>>>> node A to C.
>>>>>
>>>>> I've tried to disable HW offloading in the driver and then
>>>>> everything starts working.
>>>>>
>>>>> Is this a problem with HW offloading in the KSZ driver, or am I
>>>>> missing something essential?  
> 
> Thanks for looking and testing such large scale setup.
> 
>>>>
>>>> How are IP addresses configured? I assume you have a bridge, LAN1
>>>> and LAN2 are members of the bridge, and the IP address is on the
>>>> bridge interface?  
>>>
>>> I have a HSR interface on each node that covers LAN1 and LAN2 as
>>> slaves and the IP addresses are on those HSR interfaces. For node A:
>>>
>>> ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision 45
>>> version 1
>>> ip addr add 172.20.1.1/24 dev hsr
>>>
>>> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24 and
>>> 172.20.1.4/24 respectively.
>>>
>>> Then on node A, I'm doing:
>>>
>>> ping 172.20.1.2 # neighboring node B works
>>> ping 172.20.1.4 # neighboring node D works
>>> ping 172.20.1.3 # remote node C works only if I disable offloading  
>>
>> BTW, it's enough to disable the offloading of the forwarding for HSR
>> frames to make it work.
>>
>> --- a/drivers/net/dsa/microchip/ksz9477.c
>> +++ b/drivers/net/dsa/microchip/ksz9477.c
>> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct ksz_device
>> *dev, int port, u32 val)
>>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as HSR
>> frames
>>   * can be forwarded in the switch fabric between HSR ports.
>>   */
>> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
>> NETIF_F_HW_HSR_FWD)
>> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
>>
>>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
>> net_device *hsr)
>>  {
>> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch *ds,
>> int port, struct net_device *hsr)
>>         /* Program which port(s) shall support HSR */
>>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port), BIT(port));
>>
>> -       /* Forward frames between HSR ports (i.e. bridge together HSR
>> ports) */
>> -       if (dev->hsr_ports) {
>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>> -                       hsr_ports |= BIT(hsr_dp->index);
>> -
>> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>> -                       ksz9477_cfg_port_member(dev, hsr_dp->index,
>> hsr_ports);
>> -       }
>> -
>>         if (!dev->hsr_ports) {
>>                 /* Enable discarding of received HSR frames */
>>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);
> 
> This means that KSZ9477 forwarding is dropping frames when HW
> acceleration is used (for non "neighbour" nodes).
> 
> On my setup I only had 2 KSZ9477 devel boards.
> 
> And as you wrote - the SW based one works, so extending
> https://elixir.bootlin.com/linux/v6.12-rc2/source/tools/testing/selftests/net/hsr
> 
> would not help in this case.

I see. With two boards you can't test the accelerated forwarding. So how
did you test the forwarding at all? Or are you telling me, that this was
added to the driver without prior testing (which seems a bit bold and
unusual)?

Anyway, do you have any suggestions for debugging this? Unfortunately I
know almost nothing about this topic. But I can offer to test on my
setup, at least for now. I don't know how long I will still have access
to the hardware.

If we can't find a proper solution in the long run, I will probably send
a patch to disable the accelerated forwarding to at least make HSR work
by default.

Thanks
Frieder

Re: KSZ9477 HSR Offloading

[Andrew Lunn]

> Anyway, do you have any suggestions for debugging this?

Could you dump the fdb tables. `bridge fdb show`. Has the local device
learned the MAC addresses of the other devices? I'm making a big
assumption here, that HSR makes use of the FDB to decide the egress
interface. If there is no entry in the table, the hardware could be
dropping the frame. Rather than drop, it should probably forward to
the CPU, and then software HSR will determine the egrees
interface. Once the peer has replied, the FDB should get populated,
and then hardware should be able to do all the work without Linux?

	Andrew

Re: KSZ9477 HSR Offloading

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 6965 bytes --]

Hi Frieder,

> Hi Lukasz,
> 
> On 29.01.25 12:17 PM, Lukasz Majewski wrote:
> > Hi Frieder,
> >   
> >> On 29.01.25 8:24 AM, Frieder Schrempf wrote:  
> >>> Hi Andrew,
> >>>
> >>> On 28.01.25 6:51 PM, Andrew Lunn wrote:    
> >>>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf
> >>>> wrote:    
> >>>>> Hi,
> >>>>>
> >>>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks
> >>>>> like this:
> >>>>>
> >>>>> +-------------+         +-------------+
> >>>>> |             |         |             |
> >>>>> |   Node A    |         |   Node D    |
> >>>>> |             |         |             |
> >>>>> |             |         |             |
> >>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>>>> +--+-------+--+         +--+------+---+
> >>>>>    |       |               |      |
> >>>>>    |       +---------------+      |
> >>>>>    |                              |
> >>>>>    |       +---------------+      |
> >>>>>    |       |               |      |
> >>>>> +--+-------+--+         +--+------+---+
> >>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>>>> |             |         |             |
> >>>>> |             |         |             |
> >>>>> |   Node B    |         |   Node C    |
> >>>>> |             |         |             |
> >>>>> +-------------+         +-------------+
> >>>>>
> >>>>> On each device the LAN1 and LAN2 are added as HSR slaves. Then I
> >>>>> try to do ping tests between each of the HSR interfaces.
> >>>>>
> >>>>> The result is that I can reach the neighboring nodes just fine,
> >>>>> but I can't reach the remote node that needs packages to be
> >>>>> forwarded through the other nodes. For example I can't ping from
> >>>>> node A to C.
> >>>>>
> >>>>> I've tried to disable HW offloading in the driver and then
> >>>>> everything starts working.
> >>>>>
> >>>>> Is this a problem with HW offloading in the KSZ driver, or am I
> >>>>> missing something essential?    
> > 
> > Thanks for looking and testing such large scale setup.
> >   
> >>>>
> >>>> How are IP addresses configured? I assume you have a bridge, LAN1
> >>>> and LAN2 are members of the bridge, and the IP address is on the
> >>>> bridge interface?    
> >>>
> >>> I have a HSR interface on each node that covers LAN1 and LAN2 as
> >>> slaves and the IP addresses are on those HSR interfaces. For node
> >>> A:
> >>>
> >>> ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision
> >>> 45 version 1
> >>> ip addr add 172.20.1.1/24 dev hsr
> >>>
> >>> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24
> >>> and 172.20.1.4/24 respectively.
> >>>
> >>> Then on node A, I'm doing:
> >>>
> >>> ping 172.20.1.2 # neighboring node B works
> >>> ping 172.20.1.4 # neighboring node D works
> >>> ping 172.20.1.3 # remote node C works only if I disable
> >>> offloading    
> >>
> >> BTW, it's enough to disable the offloading of the forwarding for
> >> HSR frames to make it work.
> >>
> >> --- a/drivers/net/dsa/microchip/ksz9477.c
> >> +++ b/drivers/net/dsa/microchip/ksz9477.c
> >> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct ksz_device
> >> *dev, int port, u32 val)
> >>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as
> >> HSR frames
> >>   * can be forwarded in the switch fabric between HSR ports.
> >>   */
> >> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
> >> NETIF_F_HW_HSR_FWD)
> >> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
> >>
> >>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
> >> net_device *hsr)
> >>  {
> >> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch *ds,
> >> int port, struct net_device *hsr)
> >>         /* Program which port(s) shall support HSR */
> >>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port), BIT(port));
> >>
> >> -       /* Forward frames between HSR ports (i.e. bridge together
> >> HSR ports) */
> >> -       if (dev->hsr_ports) {
> >> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> >> -                       hsr_ports |= BIT(hsr_dp->index);
> >> -
> >> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
> >> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> >> -                       ksz9477_cfg_port_member(dev, hsr_dp->index,
> >> hsr_ports);
> >> -       }
> >> -
> >>         if (!dev->hsr_ports) {
> >>                 /* Enable discarding of received HSR frames */
> >>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);  
> > 
> > This means that KSZ9477 forwarding is dropping frames when HW
> > acceleration is used (for non "neighbour" nodes).
> > 
> > On my setup I only had 2 KSZ9477 devel boards.
> > 
> > And as you wrote - the SW based one works, so extending
> > https://elixir.bootlin.com/linux/v6.12-rc2/source/tools/testing/selftests/net/hsr
> > 
> > would not help in this case.  
> 
> I see. With two boards you can't test the accelerated forwarding. So
> how did you test the forwarding at all? Or are you telling me, that
> this was added to the driver without prior testing (which seems a bit
> bold and unusual)?

The packet forwarding is for generating two frames copies on two HSR
coupled ports on a single KSZ9477:

https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ApplicationNotes/ApplicationNotes/AN3474-KSZ9477-High-Availability-Seamless-Redundancy-Application-Note-00003474A.pdf

The KSZ9477 chip also supports RX packet duplication removal, but
cannot guarantee 100% success (so as a fallback it is done in SW).

The infrastructure from:
https://elixir.bootlin.com/linux/v6.13/source/tools/testing/selftests/net/hsr/hsr_redbox.sh#L50

is enough to test HW accelerated forwarding (of KSZ9477) from NS1 and
NS2.

> 
> Anyway, do you have any suggestions for debugging this? Unfortunately
> I know almost nothing about this topic. But I can offer to test on my
> setup, at least for now. I don't know how long I will still have
> access to the hardware.

For some reason only frames to neighbours are delivered.

So those are removed at some point (either in KSZ9477 HW or in HSR
driver itself).

Do you have some dumps from tshark/wireshark to share?

> 
> If we can't find a proper solution in the long run, I will probably
> send a patch to disable the accelerated forwarding to at least make
> HSR work by default.

As I've noted above - the HW accelerated forwarding is in the KSZ9477
chip.

The code which you uncomment, is following what I've understood from
the standard (and maybe the bug is somewhere there).

> 
> Thanks
> Frieder




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

On 29.01.25 2:58 PM, Lukasz Majewski wrote:
> Hi Frieder,
> 
>> Hi Lukasz,
>>
>> On 29.01.25 12:17 PM, Lukasz Majewski wrote:
>>> Hi Frieder,
>>>   
>>>> On 29.01.25 8:24 AM, Frieder Schrempf wrote:  
>>>>> Hi Andrew,
>>>>>
>>>>> On 28.01.25 6:51 PM, Andrew Lunn wrote:    
>>>>>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf
>>>>>> wrote:    
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks
>>>>>>> like this:
>>>>>>>
>>>>>>> +-------------+         +-------------+
>>>>>>> |             |         |             |
>>>>>>> |   Node A    |         |   Node D    |
>>>>>>> |             |         |             |
>>>>>>> |             |         |             |
>>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>>>> +--+-------+--+         +--+------+---+
>>>>>>>    |       |               |      |
>>>>>>>    |       +---------------+      |
>>>>>>>    |                              |
>>>>>>>    |       +---------------+      |
>>>>>>>    |       |               |      |
>>>>>>> +--+-------+--+         +--+------+---+
>>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>>>> |             |         |             |
>>>>>>> |             |         |             |
>>>>>>> |   Node B    |         |   Node C    |
>>>>>>> |             |         |             |
>>>>>>> +-------------+         +-------------+
>>>>>>>
>>>>>>> On each device the LAN1 and LAN2 are added as HSR slaves. Then I
>>>>>>> try to do ping tests between each of the HSR interfaces.
>>>>>>>
>>>>>>> The result is that I can reach the neighboring nodes just fine,
>>>>>>> but I can't reach the remote node that needs packages to be
>>>>>>> forwarded through the other nodes. For example I can't ping from
>>>>>>> node A to C.
>>>>>>>
>>>>>>> I've tried to disable HW offloading in the driver and then
>>>>>>> everything starts working.
>>>>>>>
>>>>>>> Is this a problem with HW offloading in the KSZ driver, or am I
>>>>>>> missing something essential?    
>>>
>>> Thanks for looking and testing such large scale setup.
>>>   
>>>>>>
>>>>>> How are IP addresses configured? I assume you have a bridge, LAN1
>>>>>> and LAN2 are members of the bridge, and the IP address is on the
>>>>>> bridge interface?    
>>>>>
>>>>> I have a HSR interface on each node that covers LAN1 and LAN2 as
>>>>> slaves and the IP addresses are on those HSR interfaces. For node
>>>>> A:
>>>>>
>>>>> ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision
>>>>> 45 version 1
>>>>> ip addr add 172.20.1.1/24 dev hsr
>>>>>
>>>>> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24
>>>>> and 172.20.1.4/24 respectively.
>>>>>
>>>>> Then on node A, I'm doing:
>>>>>
>>>>> ping 172.20.1.2 # neighboring node B works
>>>>> ping 172.20.1.4 # neighboring node D works
>>>>> ping 172.20.1.3 # remote node C works only if I disable
>>>>> offloading    
>>>>
>>>> BTW, it's enough to disable the offloading of the forwarding for
>>>> HSR frames to make it work.
>>>>
>>>> --- a/drivers/net/dsa/microchip/ksz9477.c
>>>> +++ b/drivers/net/dsa/microchip/ksz9477.c
>>>> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct ksz_device
>>>> *dev, int port, u32 val)
>>>>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as
>>>> HSR frames
>>>>   * can be forwarded in the switch fabric between HSR ports.
>>>>   */
>>>> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
>>>> NETIF_F_HW_HSR_FWD)
>>>> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
>>>>
>>>>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
>>>> net_device *hsr)
>>>>  {
>>>> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch *ds,
>>>> int port, struct net_device *hsr)
>>>>         /* Program which port(s) shall support HSR */
>>>>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port), BIT(port));
>>>>
>>>> -       /* Forward frames between HSR ports (i.e. bridge together
>>>> HSR ports) */
>>>> -       if (dev->hsr_ports) {
>>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>>>> -                       hsr_ports |= BIT(hsr_dp->index);
>>>> -
>>>> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
>>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>>>> -                       ksz9477_cfg_port_member(dev, hsr_dp->index,
>>>> hsr_ports);
>>>> -       }
>>>> -
>>>>         if (!dev->hsr_ports) {
>>>>                 /* Enable discarding of received HSR frames */
>>>>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);  
>>>
>>> This means that KSZ9477 forwarding is dropping frames when HW
>>> acceleration is used (for non "neighbour" nodes).
>>>
>>> On my setup I only had 2 KSZ9477 devel boards.
>>>
>>> And as you wrote - the SW based one works, so extending
>>> https://elixir.bootlin.com/linux/v6.12-rc2/source/tools/testing/selftests/net/hsr
>>>
>>> would not help in this case.  
>>
>> I see. With two boards you can't test the accelerated forwarding. So
>> how did you test the forwarding at all? Or are you telling me, that
>> this was added to the driver without prior testing (which seems a bit
>> bold and unusual)?
> 
> The packet forwarding is for generating two frames copies on two HSR
> coupled ports on a single KSZ9477:

Isn't that what duplication aka NETIF_F_HW_HSR_DUP is for?

> 
> https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ApplicationNotes/ApplicationNotes/AN3474-KSZ9477-High-Availability-Seamless-Redundancy-Application-Note-00003474A.pdf
> 
> The KSZ9477 chip also supports RX packet duplication removal, but
> cannot guarantee 100% success (so as a fallback it is done in SW).
> 
> The infrastructure from:
> https://elixir.bootlin.com/linux/v6.13/source/tools/testing/selftests/net/hsr/hsr_redbox.sh#L50
> 
> is enough to test HW accelerated forwarding (of KSZ9477) from NS1 and
> NS2.

I'm not really sure if I get it. In this setup NS1 and NS2 are connected
via HSR link (two physical links). On one side packets are sent
duplicated on both physical ports. On the receiving side the duplication
is removed and one packet is forwarded to the CPU.

Where is forwarding involved here? Isn't forwarding only for cases with
one intermediate node between the sending and receiving node?

> 
>>
>> Anyway, do you have any suggestions for debugging this? Unfortunately
>> I know almost nothing about this topic. But I can offer to test on my
>> setup, at least for now. I don't know how long I will still have
>> access to the hardware.
> 
> For some reason only frames to neighbours are delivered.
> 
> So those are removed at some point (either in KSZ9477 HW or in HSR
> driver itself).
> 
> Do you have some dumps from tshark/wireshark to share?
> 
>>
>> If we can't find a proper solution in the long run, I will probably
>> send a patch to disable the accelerated forwarding to at least make
>> HSR work by default.
> 
> As I've noted above - the HW accelerated forwarding is in the KSZ9477
> chip.

Yeah, but if the HW accelerated forwarding doesn't work it would be
better to use no acceleration and have it work in SW at least by
default, right?

> 
> The code which you uncomment, is following what I've understood from
> the standard (and maybe the bug is somewhere there).

Ok, thanks for explaining. I will see if I can find some time to gather
some more information on the problem.

RE: KSZ9477 HSR Offloading

[Woojung.Huh]

Hi Frieder,

Can you please create a ticket at Microchip's site and share it with me?
https://microchipsupport.force.com/s/supportservice

Best regards,
Woojung

> -----Original Message-----
> From: Frieder Schrempf <frieder.schrempf@kontron.de>
> Sent: Wednesday, January 29, 2025 12:08 PM
> To: Lukasz Majewski <lukma@denx.de>
> Cc: Andrew Lunn <andrew@lunn.ch>; netdev@vger.kernel.org
> Subject: Re: KSZ9477 HSR Offloading
> 
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the
> content is safe
> 
> On 29.01.25 2:58 PM, Lukasz Majewski wrote:
> > Hi Frieder,
> >
> >> Hi Lukasz,
> >>
> >> On 29.01.25 12:17 PM, Lukasz Majewski wrote:
> >>> Hi Frieder,
> >>>
> >>>> On 29.01.25 8:24 AM, Frieder Schrempf wrote:
> >>>>> Hi Andrew,
> >>>>>
> >>>>> On 28.01.25 6:51 PM, Andrew Lunn wrote:
> >>>>>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf
> >>>>>> wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup looks
> >>>>>>> like this:
> >>>>>>>
> >>>>>>> +-------------+         +-------------+
> >>>>>>> |             |         |             |
> >>>>>>> |   Node A    |         |   Node D    |
> >>>>>>> |             |         |             |
> >>>>>>> |             |         |             |
> >>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>>>>>> +--+-------+--+         +--+------+---+
> >>>>>>>    |       |               |      |
> >>>>>>>    |       +---------------+      |
> >>>>>>>    |                              |
> >>>>>>>    |       +---------------+      |
> >>>>>>>    |       |               |      |
> >>>>>>> +--+-------+--+         +--+------+---+
> >>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>>>>>> |             |         |             |
> >>>>>>> |             |         |             |
> >>>>>>> |   Node B    |         |   Node C    |
> >>>>>>> |             |         |             |
> >>>>>>> +-------------+         +-------------+
> >>>>>>>
> >>>>>>> On each device the LAN1 and LAN2 are added as HSR slaves. Then I
> >>>>>>> try to do ping tests between each of the HSR interfaces.
> >>>>>>>
> >>>>>>> The result is that I can reach the neighboring nodes just fine,
> >>>>>>> but I can't reach the remote node that needs packages to be
> >>>>>>> forwarded through the other nodes. For example I can't ping from
> >>>>>>> node A to C.
> >>>>>>>
> >>>>>>> I've tried to disable HW offloading in the driver and then
> >>>>>>> everything starts working.
> >>>>>>>
> >>>>>>> Is this a problem with HW offloading in the KSZ driver, or am I
> >>>>>>> missing something essential?
> >>>
> >>> Thanks for looking and testing such large scale setup.
> >>>
> >>>>>>
> >>>>>> How are IP addresses configured? I assume you have a bridge, LAN1
> >>>>>> and LAN2 are members of the bridge, and the IP address is on the
> >>>>>> bridge interface?
> >>>>>
> >>>>> I have a HSR interface on each node that covers LAN1 and LAN2 as
> >>>>> slaves and the IP addresses are on those HSR interfaces. For node
> >>>>> A:
> >>>>>
> >>>>> ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision
> >>>>> 45 version 1
> >>>>> ip addr add 172.20.1.1/24 dev hsr
> >>>>>
> >>>>> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24
> >>>>> and 172.20.1.4/24 respectively.
> >>>>>
> >>>>> Then on node A, I'm doing:
> >>>>>
> >>>>> ping 172.20.1.2 # neighboring node B works
> >>>>> ping 172.20.1.4 # neighboring node D works
> >>>>> ping 172.20.1.3 # remote node C works only if I disable
> >>>>> offloading
> >>>>
> >>>> BTW, it's enough to disable the offloading of the forwarding for
> >>>> HSR frames to make it work.
> >>>>
> >>>> --- a/drivers/net/dsa/microchip/ksz9477.c
> >>>> +++ b/drivers/net/dsa/microchip/ksz9477.c
> >>>> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct ksz_device
> >>>> *dev, int port, u32 val)
> >>>>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as
> >>>> HSR frames
> >>>>   * can be forwarded in the switch fabric between HSR ports.
> >>>>   */
> >>>> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
> >>>> NETIF_F_HW_HSR_FWD)
> >>>> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
> >>>>
> >>>>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
> >>>> net_device *hsr)
> >>>>  {
> >>>> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch *ds,
> >>>> int port, struct net_device *hsr)
> >>>>         /* Program which port(s) shall support HSR */
> >>>>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port), BIT(port));
> >>>>
> >>>> -       /* Forward frames between HSR ports (i.e. bridge together
> >>>> HSR ports) */
> >>>> -       if (dev->hsr_ports) {
> >>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> >>>> -                       hsr_ports |= BIT(hsr_dp->index);
> >>>> -
> >>>> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
> >>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> >>>> -                       ksz9477_cfg_port_member(dev, hsr_dp->index,
> >>>> hsr_ports);
> >>>> -       }
> >>>> -
> >>>>         if (!dev->hsr_ports) {
> >>>>                 /* Enable discarding of received HSR frames */
> >>>>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);
> >>>
> >>> This means that KSZ9477 forwarding is dropping frames when HW
> >>> acceleration is used (for non "neighbour" nodes).
> >>>
> >>> On my setup I only had 2 KSZ9477 devel boards.
> >>>
> >>> And as you wrote - the SW based one works, so extending
> >>> https://elixir.bootlin.com/linux/v6.12-
> rc2/source/tools/testing/selftests/net/hsr
> >>>
> >>> would not help in this case.
> >>
> >> I see. With two boards you can't test the accelerated forwarding. So
> >> how did you test the forwarding at all? Or are you telling me, that
> >> this was added to the driver without prior testing (which seems a bit
> >> bold and unusual)?
> >
> > The packet forwarding is for generating two frames copies on two HSR
> > coupled ports on a single KSZ9477:
> 
> Isn't that what duplication aka NETIF_F_HW_HSR_DUP is for?
> 
> >
> >
> https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ApplicationNo
> tes/ApplicationNotes/AN3474-KSZ9477-High-Availability-Seamless-Redundancy-
> Application-Note-00003474A.pdf
> >
> > The KSZ9477 chip also supports RX packet duplication removal, but
> > cannot guarantee 100% success (so as a fallback it is done in SW).
> >
> > The infrastructure from:
> >
> https://elixir.bootlin.com/linux/v6.13/source/tools/testing/selftests/net/hs
> r/hsr_redbox.sh#L50
> >
> > is enough to test HW accelerated forwarding (of KSZ9477) from NS1 and
> > NS2.
> 
> I'm not really sure if I get it. In this setup NS1 and NS2 are connected
> via HSR link (two physical links). On one side packets are sent
> duplicated on both physical ports. On the receiving side the duplication
> is removed and one packet is forwarded to the CPU.
> 
> Where is forwarding involved here? Isn't forwarding only for cases with
> one intermediate node between the sending and receiving node?
> 
> >
> >>
> >> Anyway, do you have any suggestions for debugging this? Unfortunately
> >> I know almost nothing about this topic. But I can offer to test on my
> >> setup, at least for now. I don't know how long I will still have
> >> access to the hardware.
> >
> > For some reason only frames to neighbours are delivered.
> >
> > So those are removed at some point (either in KSZ9477 HW or in HSR
> > driver itself).
> >
> > Do you have some dumps from tshark/wireshark to share?
> >
> >>
> >> If we can't find a proper solution in the long run, I will probably
> >> send a patch to disable the accelerated forwarding to at least make
> >> HSR work by default.
> >
> > As I've noted above - the HW accelerated forwarding is in the KSZ9477
> > chip.
> 
> Yeah, but if the HW accelerated forwarding doesn't work it would be
> better to use no acceleration and have it work in SW at least by
> default, right?
> 
> >
> > The code which you uncomment, is following what I've understood from
> > the standard (and maybe the bug is somewhere there).
> 
> Ok, thanks for explaining. I will see if I can find some time to gather
> some more information on the problem.

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

Hi Woojung,

On 29.01.25 7:57 PM, Woojung.Huh@microchip.com wrote:
> [Sie erhalten nicht häufig E-Mails von woojung.huh@microchip.com. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]
> 
> Hi Frieder,
> 
> Can you please create a ticket at Microchip's site and share it with me?

Sure, here is the link:
https://microchip.my.site.com/s/case/500V400000KQi1tIAD/

Thanks
Frieder

RE: KSZ9477 HSR Offloading

[Woojung.Huh]

HI Frieder,

Thanks for the link. I reminded the support team this ticket.
Please wait response in the ticket. Hope we can get the solution for you.

Thanks.
Woojung

> -----Original Message-----
> From: Frieder Schrempf <frieder.schrempf@kontron.de>
> Sent: Thursday, January 30, 2025 3:44 AM
> To: Woojung Huh - C21699 <Woojung.Huh@microchip.com>
> Cc: andrew@lunn.ch; netdev@vger.kernel.org; lukma@denx.de; Tristram Ha -
> C24268 <Tristram.Ha@microchip.com>
> Subject: Re: KSZ9477 HSR Offloading
> 
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the
> content is safe
> 
> Hi Woojung,
> 
> On 29.01.25 7:57 PM, Woojung.Huh@microchip.com wrote:
> > [Sie erhalten nicht häufig E-Mails von woojung.huh@microchip.com. Weitere
> Informationen, warum dies wichtig ist, finden Sie unter
> https://aka.ms/LearnAboutSenderIdentification ]
> >
> > Hi Frieder,
> >
> > Can you please create a ticket at Microchip's site and share it with me?
> 
> Sure, here is the link:
> https://microchip.my.site.com/s/case/500V400000KQi1tIAD/
> 
> Thanks
> Frieder

Re: KSZ9477 HSR Offloading

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

Hi Woojung,

> HI Frieder,
> 
> Thanks for the link. I reminded the support team this ticket.
> Please wait response in the ticket. Hope we can get the solution for
> you.
> 
> Thanks.
> Woojung
> 
> > -----Original Message-----
> > From: Frieder Schrempf <frieder.schrempf@kontron.de>
> > Sent: Thursday, January 30, 2025 3:44 AM
> > To: Woojung Huh - C21699 <Woojung.Huh@microchip.com>
> > Cc: andrew@lunn.ch; netdev@vger.kernel.org; lukma@denx.de; Tristram
> > Ha - C24268 <Tristram.Ha@microchip.com>
> > Subject: Re: KSZ9477 HSR Offloading
> > 
> > EXTERNAL EMAIL: Do not click links or open attachments unless you
> > know the content is safe
> > 
> > Hi Woojung,
> > 
> > On 29.01.25 7:57 PM, Woojung.Huh@microchip.com wrote:  
> > > [Sie erhalten nicht häufig E-Mails von woojung.huh@microchip.com.
> > > Weitere  
> > Informationen, warum dies wichtig ist, finden Sie unter
> > https://aka.ms/LearnAboutSenderIdentification ]  
> > >
> > > Hi Frieder,
> > >
> > > Can you please create a ticket at Microchip's site and share it
> > > with me?  
> > 
> > Sure, here is the link:
> > https://microchip.my.site.com/s/case/500V400000KQi1tIAD/

Is the link correct?

When I login into microchip.my.site.com I don't see this "case" created
for KSZ9477.

> > 
> > Thanks
> > Frieder  




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

RE: KSZ9477 HSR Offloading

[Woojung.Huh]

Hi Lukasz,

> > > Sure, here is the link:
> > > https://microchip.my.site.com/s/case/500V400000KQi1tIAD/
> 
> Is the link correct?
> 
> When I login into microchip.my.site.com I don't see this "case" created
> for KSZ9477.
> 

I heard that your ticket was assigned. Let me check and get back to you.

Thanks
Woojung

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

On 03.02.25 10:31 AM, Lukasz Majewski wrote:
> Hi Woojung,
> 
>> HI Frieder,
>>
>> Thanks for the link. I reminded the support team this ticket.
>> Please wait response in the ticket. Hope we can get the solution for
>> you.
>>
>> Thanks.
>> Woojung
>>
>>> -----Original Message-----
>>> From: Frieder Schrempf <frieder.schrempf@kontron.de>
>>> Sent: Thursday, January 30, 2025 3:44 AM
>>> To: Woojung Huh - C21699 <Woojung.Huh@microchip.com>
>>> Cc: andrew@lunn.ch; netdev@vger.kernel.org; lukma@denx.de; Tristram
>>> Ha - C24268 <Tristram.Ha@microchip.com>
>>> Subject: Re: KSZ9477 HSR Offloading
>>>
>>> EXTERNAL EMAIL: Do not click links or open attachments unless you
>>> know the content is safe
>>>
>>> Hi Woojung,
>>>
>>> On 29.01.25 7:57 PM, Woojung.Huh@microchip.com wrote:  
>>>> [Sie erhalten nicht häufig E-Mails von woojung.huh@microchip.com.
>>>> Weitere  
>>> Informationen, warum dies wichtig ist, finden Sie unter
>>> https://aka.ms/LearnAboutSenderIdentification ]  
>>>>
>>>> Hi Frieder,
>>>>
>>>> Can you please create a ticket at Microchip's site and share it
>>>> with me?  
>>>
>>> Sure, here is the link:
>>> https://microchip.my.site.com/s/case/500V400000KQi1tIAD/
> 
> Is the link correct?
> 
> When I login into microchip.my.site.com I don't see this "case" created
> for KSZ9477.

The link works for me. Maybe you can't see the ticket as you have no
permissions? I added you as "team member" to the case with your Denx
email address. If you already have an account on a different email, let
me know and I can change the address.

RE: KSZ9477 HSR Offloading

[Woojung.Huh]

Hi Lukasz & Frieder,

Oops! My bad. I confused that Lukasz was filed a case originally. Monday brain-freeze. :(

Yes, it is not a public link and per-user case. So, only Frieder can see it.
It may be able for you when Frieder adds you as a team. (Not tested personally though)

Thanks
Woojung

> -----Original Message-----
> From: Frieder Schrempf <frieder.schrempf@kontron.de>
> Sent: Monday, February 3, 2025 9:48 AM
> To: Lukasz Majewski <lukma@denx.de>; Woojung Huh - C21699
> <Woojung.Huh@microchip.com>
> Cc: andrew@lunn.ch; netdev@vger.kernel.org; Tristram Ha - C24268
> <Tristram.Ha@microchip.com>
> Subject: Re: KSZ9477 HSR Offloading
> 
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the
> content is safe
> 
> On 03.02.25 10:31 AM, Lukasz Majewski wrote:
> > Hi Woojung,
> >
> >> HI Frieder,
> >>
> >> Thanks for the link. I reminded the support team this ticket.
> >> Please wait response in the ticket. Hope we can get the solution for
> >> you.
> >>
> >> Thanks.
> >> Woojung
> >>
> >>> -----Original Message-----
> >>> From: Frieder Schrempf <frieder.schrempf@kontron.de>
> >>> Sent: Thursday, January 30, 2025 3:44 AM
> >>> To: Woojung Huh - C21699 <Woojung.Huh@microchip.com>
> >>> Cc: andrew@lunn.ch; netdev@vger.kernel.org; lukma@denx.de; Tristram
> >>> Ha - C24268 <Tristram.Ha@microchip.com>
> >>> Subject: Re: KSZ9477 HSR Offloading
> >>>
> >>> EXTERNAL EMAIL: Do not click links or open attachments unless you
> >>> know the content is safe
> >>>
> >>> Hi Woojung,
> >>>
> >>> On 29.01.25 7:57 PM, Woojung.Huh@microchip.com wrote:
> >>>> [Sie erhalten nicht häufig E-Mails von woojung.huh@microchip.com.
> >>>> Weitere
> >>> Informationen, warum dies wichtig ist, finden Sie unter
> >>> https://aka.ms/LearnAboutSenderIdentification ]
> >>>>
> >>>> Hi Frieder,
> >>>>
> >>>> Can you please create a ticket at Microchip's site and share it
> >>>> with me?
> >>>
> >>> Sure, here is the link:
> >>> https://microchip.my.site.com/s/case/500V400000KQi1tIAD/
> >
> > Is the link correct?
> >
> > When I login into microchip.my.site.com I don't see this "case" created
> > for KSZ9477.
> 
> The link works for me. Maybe you can't see the ticket as you have no
> permissions? I added you as "team member" to the case with your Denx
> email address. If you already have an account on a different email, let
> me know and I can change the address.

Re: KSZ9477 HSR Offloading

[Jakub Kicinski]

On Mon, 3 Feb 2025 14:58:12 +0000 Woojung.Huh@microchip.com wrote:
> Hi Lukasz & Frieder,
> 
> Oops! My bad. I confused that Lukasz was filed a case originally. Monday brain-freeze. :(
> 
> Yes, it is not a public link and per-user case. So, only Frieder can see it.
> It may be able for you when Frieder adds you as a team. (Not tested personally though)

Woojung Huh, please make sure the mailing list is informed about 
the outcomes. Taking discussion off list to a closed ticketing 
system is against community rules. See below, thanks.

Quoting documentation:

  Open development
  ----------------
  
  Discussions about user reported issues, and development of new code
  should be conducted in a manner typical for the larger subsystem.
  It is common for development within a single company to be conducted
  behind closed doors. However, development and discussions initiated
  by community members must not be redirected from public to closed forums
  or to private email conversations. Reasonable exceptions to this guidance
  include discussions about security related issues.
  
See: https://www.kernel.org/doc/html/next/maintainer/feature-and-driver-maintainers.html#open-development

RE: KSZ9477 HSR Offloading

[Woojung.Huh]

Hi Jakub,

> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Monday, February 3, 2025 6:04 PM
> To: Woojung Huh - C21699 <Woojung.Huh@microchip.com>
> Cc: frieder.schrempf@kontron.de; lukma@denx.de; andrew@lunn.ch;
> netdev@vger.kernel.org; Tristram Ha - C24268 <Tristram.Ha@microchip.com>
> Subject: Re: KSZ9477 HSR Offloading
> 
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the
> content is safe
> 
> On Mon, 3 Feb 2025 14:58:12 +0000 Woojung.Huh@microchip.com wrote:
> > Hi Lukasz & Frieder,
> >
> > Oops! My bad. I confused that Lukasz was filed a case originally. Monday
> brain-freeze. :(
> >
> > Yes, it is not a public link and per-user case. So, only Frieder can see
> it.
> > It may be able for you when Frieder adds you as a team. (Not tested
> personally though)
> 
> Woojung Huh, please make sure the mailing list is informed about
> the outcomes. Taking discussion off list to a closed ticketing
> system is against community rules. See below, thanks.
> 
> Quoting documentation:
> 
>   Open development
>   ----------------
> 
>   Discussions about user reported issues, and development of new code
>   should be conducted in a manner typical for the larger subsystem.
>   It is common for development within a single company to be conducted
>   behind closed doors. However, development and discussions initiated
>   by community members must not be redirected from public to closed forums
>   or to private email conversations. Reasonable exceptions to this guidance
>   include discussions about security related issues.
> 
> See: https://www.kernel.org/doc/html/next/maintainer/feature-and-driver-
> maintainers.html#open-development

Learn new thing today. Didn't know this. Definitely I will share it
when this work is done. My intention was for easier work for request than
having me as an middleman for the issue.

Best regards,
Woojung

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

Am 04.02.25 um 15:55 schrieb Woojung.Huh@microchip.com:
> Hi Jakub,
> 
>> -----Original Message-----
>> From: Jakub Kicinski <kuba@kernel.org>
>> Sent: Monday, February 3, 2025 6:04 PM
>> To: Woojung Huh - C21699 <Woojung.Huh@microchip.com>
>> Cc: frieder.schrempf@kontron.de; lukma@denx.de; andrew@lunn.ch;
>> netdev@vger.kernel.org; Tristram Ha - C24268 <Tristram.Ha@microchip.com>
>> Subject: Re: KSZ9477 HSR Offloading
>>
>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the
>> content is safe
>>
>> On Mon, 3 Feb 2025 14:58:12 +0000 Woojung.Huh@microchip.com wrote:
>>> Hi Lukasz & Frieder,
>>>
>>> Oops! My bad. I confused that Lukasz was filed a case originally. Monday
>> brain-freeze. :(
>>>
>>> Yes, it is not a public link and per-user case. So, only Frieder can see
>> it.
>>> It may be able for you when Frieder adds you as a team. (Not tested
>> personally though)
>>
>> Woojung Huh, please make sure the mailing list is informed about
>> the outcomes. Taking discussion off list to a closed ticketing
>> system is against community rules. See below, thanks.
>>
>> Quoting documentation:
>>
>>   Open development
>>   ----------------
>>
>>   Discussions about user reported issues, and development of new code
>>   should be conducted in a manner typical for the larger subsystem.
>>   It is common for development within a single company to be conducted
>>   behind closed doors. However, development and discussions initiated
>>   by community members must not be redirected from public to closed forums
>>   or to private email conversations. Reasonable exceptions to this guidance
>>   include discussions about security related issues.
>>
>> See: https://www.kernel.org/doc/html/next/maintainer/feature-and-driver-maintainers.html#open-development
> 
> Learn new thing today. Didn't know this. Definitely I will share it
> when this work is done. My intention was for easier work for request than
> having me as an middleman for the issue.

Here is a follow-up for this thread. I was busy elsewhere, didn't have
access to the hardware and failed to respond to the comments from
Microchip support team provided in their internal ticket system.

As a summary, the Microchip support couldn't reproduce my issue on their
side and asked for further information.

With the hardware now back on my table I was able to do some further
investigations and found out that this is caused by a misconfiguration
on my side, that doesn't get handled/prevented by the kernel.

The HW forwarding between HSR ports is configured in ksz9477_hsr_join()
at the time of creating the HSR interface by calling
ksz9477_cfg_port_member().

In my case I enabled the ports **after** that which caused the
forwarding to be disabled again as ksz9477_cfg_port_member() gets called
with the default configuration.

If I reorder my commands everything seems to work fine even with
NETIF_F_HW_HSR_FWD enabled.

I wonder if the kernel should handle this case and prevent the
forwarding configuration to be disabled if HSR is configured? I'll have
a look if I can come up with a patch for this.

Re: KSZ9477 HSR Offloading

[Łukasz Majewski]

Hi Frieder,

> Am 04.02.25 um 15:55 schrieb Woojung.Huh@microchip.com:
> > Hi Jakub,
> >   
> >> -----Original Message-----
> >> From: Jakub Kicinski <kuba@kernel.org>
> >> Sent: Monday, February 3, 2025 6:04 PM
> >> To: Woojung Huh - C21699 <Woojung.Huh@microchip.com>
> >> Cc: frieder.schrempf@kontron.de; lukma@denx.de; andrew@lunn.ch;
> >> netdev@vger.kernel.org; Tristram Ha - C24268
> >> <Tristram.Ha@microchip.com> Subject: Re: KSZ9477 HSR Offloading
> >>
> >> EXTERNAL EMAIL: Do not click links or open attachments unless you
> >> know the content is safe
> >>
> >> On Mon, 3 Feb 2025 14:58:12 +0000 Woojung.Huh@microchip.com wrote:
> >>  
> >>> Hi Lukasz & Frieder,
> >>>
> >>> Oops! My bad. I confused that Lukasz was filed a case originally.
> >>> Monday  
> >> brain-freeze. :(  
> >>>
> >>> Yes, it is not a public link and per-user case. So, only Frieder
> >>> can see  
> >> it.  
> >>> It may be able for you when Frieder adds you as a team. (Not
> >>> tested  
> >> personally though)
> >>
> >> Woojung Huh, please make sure the mailing list is informed about
> >> the outcomes. Taking discussion off list to a closed ticketing
> >> system is against community rules. See below, thanks.
> >>
> >> Quoting documentation:
> >>
> >>   Open development
> >>   ----------------
> >>
> >>   Discussions about user reported issues, and development of new
> >> code should be conducted in a manner typical for the larger
> >> subsystem. It is common for development within a single company to
> >> be conducted behind closed doors. However, development and
> >> discussions initiated by community members must not be redirected
> >> from public to closed forums or to private email conversations.
> >> Reasonable exceptions to this guidance include discussions about
> >> security related issues.
> >>
> >> See:
> >> https://www.kernel.org/doc/html/next/maintainer/feature-and-driver-maintainers.html#open-development
> >>  
> > 
> > Learn new thing today. Didn't know this. Definitely I will share it
> > when this work is done. My intention was for easier work for
> > request than having me as an middleman for the issue.  
> 
> Here is a follow-up for this thread. I was busy elsewhere, didn't have
> access to the hardware and failed to respond to the comments from
> Microchip support team provided in their internal ticket system.
> 
> As a summary, the Microchip support couldn't reproduce my issue on
> their side and asked for further information.
> 
> With the hardware now back on my table I was able to do some further
> investigations and found out that this is caused by a misconfiguration
> on my side, that doesn't get handled/prevented by the kernel.

If I remember correctly from the ticket - there was also an issue with
the size of MTU for HSR packets.

Am I correct?

> 
> The HW forwarding between HSR ports is configured in
> ksz9477_hsr_join() at the time of creating the HSR interface by
> calling ksz9477_cfg_port_member().
> 
> In my case I enabled the ports **after** that which caused the
> forwarding to be disabled again as ksz9477_cfg_port_member() gets
> called with the default configuration.
> 
> If I reorder my commands everything seems to work fine even with
> NETIF_F_HW_HSR_FWD enabled.
> 
> I wonder if the kernel should handle this case and prevent the
> forwarding configuration to be disabled if HSR is configured? I'll
> have a look if I can come up with a patch for this.
> 

Thanks  for looking for this issue. I'm looking forward for your
patches.

-- 
Best regards,

Lukasz Majewski

--
Nabla Software Engineering GmbH
HRB 40522 Augsburg
Phone: +49 821 45592596
E-Mail: office@nabladev.com
Geschftsfhrer : Stefano Babic

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

Hi Lukasz,

Am 13.08.25 um 15:54 schrieb Łukasz Majewski:
> [Sie erhalten nicht h?ufig E-Mails von lukma@nabladev.com. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]
> 
> Hi Frieder,
> 
>> Am 04.02.25 um 15:55 schrieb Woojung.Huh@microchip.com:
>>> Hi Jakub,
>>>
>>>> -----Original Message-----
>>>> From: Jakub Kicinski <kuba@kernel.org>
>>>> Sent: Monday, February 3, 2025 6:04 PM
>>>> To: Woojung Huh - C21699 <Woojung.Huh@microchip.com>
>>>> Cc: frieder.schrempf@kontron.de; lukma@denx.de; andrew@lunn.ch;
>>>> netdev@vger.kernel.org; Tristram Ha - C24268
>>>> <Tristram.Ha@microchip.com> Subject: Re: KSZ9477 HSR Offloading
>>>>
>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you
>>>> know the content is safe
>>>>
>>>> On Mon, 3 Feb 2025 14:58:12 +0000 Woojung.Huh@microchip.com wrote:
>>>>
>>>>> Hi Lukasz & Frieder,
>>>>>
>>>>> Oops! My bad. I confused that Lukasz was filed a case originally.
>>>>> Monday
>>>> brain-freeze. :(
>>>>>
>>>>> Yes, it is not a public link and per-user case. So, only Frieder
>>>>> can see
>>>> it.
>>>>> It may be able for you when Frieder adds you as a team. (Not
>>>>> tested
>>>> personally though)
>>>>
>>>> Woojung Huh, please make sure the mailing list is informed about
>>>> the outcomes. Taking discussion off list to a closed ticketing
>>>> system is against community rules. See below, thanks.
>>>>
>>>> Quoting documentation:
>>>>
>>>>   Open development
>>>>   ----------------
>>>>
>>>>   Discussions about user reported issues, and development of new
>>>> code should be conducted in a manner typical for the larger
>>>> subsystem. It is common for development within a single company to
>>>> be conducted behind closed doors. However, development and
>>>> discussions initiated by community members must not be redirected
>>>> from public to closed forums or to private email conversations.
>>>> Reasonable exceptions to this guidance include discussions about
>>>> security related issues.
>>>>
>>>> See:
>>>> https://www.kernel.org/doc/html/next/maintainer/feature-and-driver-maintainers.html#open-development
>>>>
>>>
>>> Learn new thing today. Didn't know this. Definitely I will share it
>>> when this work is done. My intention was for easier work for
>>> request than having me as an middleman for the issue.
>>
>> Here is a follow-up for this thread. I was busy elsewhere, didn't have
>> access to the hardware and failed to respond to the comments from
>> Microchip support team provided in their internal ticket system.
>>
>> As a summary, the Microchip support couldn't reproduce my issue on
>> their side and asked for further information.
>>
>> With the hardware now back on my table I was able to do some further
>> investigations and found out that this is caused by a misconfiguration
>> on my side, that doesn't get handled/prevented by the kernel.
> 
> If I remember correctly from the ticket - there was also an issue with
> the size of MTU for HSR packets.
> 
> Am I correct?

The Microchip support mentioned the following:

###
Note the default MTU of hsr0 is (1500 - 6) = 1494. When other HSR node
is used the MTU needs to match. The HSR driver code checks the MTU of
the underneath device to see if it should be lowered. Even that device
can send more than 1506 bytes the value still is fixed at 1500 and then
decreased by 6. In my opinion that should be changed to start 1506 and
then lower to 1500 if the lower device cannot send that. The RedBox
operation also will be impacted if the MTU 1494 is used as the devices
behind the Redbox all uses MTU 1500
###

So far I didn't run into any issues related to MTU, but I might stumble
upon this in the future.

> 
>>
>> The HW forwarding between HSR ports is configured in
>> ksz9477_hsr_join() at the time of creating the HSR interface by
>> calling ksz9477_cfg_port_member().
>>
>> In my case I enabled the ports **after** that which caused the
>> forwarding to be disabled again as ksz9477_cfg_port_member() gets
>> called with the default configuration.
>>
>> If I reorder my commands everything seems to work fine even with
>> NETIF_F_HW_HSR_FWD enabled.
>>
>> I wonder if the kernel should handle this case and prevent the
>> forwarding configuration to be disabled if HSR is configured? I'll
>> have a look if I can come up with a patch for this.
>>
> 
> Thanks  for looking for this issue. I'm looking forward for your
> patches.

I came up with [1] which fixes the issue that I was seeing. I hope that
this goes into the right direction.

Thanks
Frieder

[1]
https://patchwork.kernel.org/project/netdevbpf/patch/20250813152615.856532-1-frieder@fris.de/

Re: KSZ9477 HSR Offloading

[Andrew Lunn]

On Wed, Aug 13, 2025 at 03:43:14PM +0200, Frieder Schrempf wrote:
> Am 04.02.25 um 15:55 schrieb Woojung.Huh@microchip.com:
> > Hi Jakub,
> > 
> >> -----Original Message-----
> >> From: Jakub Kicinski <kuba@kernel.org>
> >> Sent: Monday, February 3, 2025 6:04 PM
> >> To: Woojung Huh - C21699 <Woojung.Huh@microchip.com>
> >> Cc: frieder.schrempf@kontron.de; lukma@denx.de; andrew@lunn.ch;
> >> netdev@vger.kernel.org; Tristram Ha - C24268 <Tristram.Ha@microchip.com>
> >> Subject: Re: KSZ9477 HSR Offloading
> >>
> >> EXTERNAL EMAIL: Do not click links or open attachments unless you know the
> >> content is safe
> >>
> >> On Mon, 3 Feb 2025 14:58:12 +0000 Woojung.Huh@microchip.com wrote:
> >>> Hi Lukasz & Frieder,
> >>>
> >>> Oops! My bad. I confused that Lukasz was filed a case originally. Monday
> >> brain-freeze. :(
> >>>
> >>> Yes, it is not a public link and per-user case. So, only Frieder can see
> >> it.
> >>> It may be able for you when Frieder adds you as a team. (Not tested
> >> personally though)
> >>
> >> Woojung Huh, please make sure the mailing list is informed about
> >> the outcomes. Taking discussion off list to a closed ticketing
> >> system is against community rules. See below, thanks.
> >>
> >> Quoting documentation:
> >>
> >>   Open development
> >>   ----------------
> >>
> >>   Discussions about user reported issues, and development of new code
> >>   should be conducted in a manner typical for the larger subsystem.
> >>   It is common for development within a single company to be conducted
> >>   behind closed doors. However, development and discussions initiated
> >>   by community members must not be redirected from public to closed forums
> >>   or to private email conversations. Reasonable exceptions to this guidance
> >>   include discussions about security related issues.
> >>
> >> See: https://www.kernel.org/doc/html/next/maintainer/feature-and-driver-maintainers.html#open-development
> > 
> > Learn new thing today. Didn't know this. Definitely I will share it
> > when this work is done. My intention was for easier work for request than
> The HW forwarding between HSR ports is configured in ksz9477_hsr_join()
> at the time of creating the HSR interface by calling
> ksz9477_cfg_port_member().
> 
> In my case I enabled the ports **after** that which caused the
> forwarding to be disabled again as ksz9477_cfg_port_member() gets called
> with the default configuration.
> 
> If I reorder my commands everything seems to work fine even with
> NETIF_F_HW_HSR_FWD enabled.
> 
> I wonder if the kernel should handle this case and prevent the
> forwarding configuration to be disabled if HSR is configured? I'll have
> a look if I can come up with a patch for this.

If you don't offload it, but do it in software, does the order matter?

As a user, i should not need to care if offload is used or not. It
should be transparent. Which means any order that works with pure
software should also work with offloading. At least, it should not
break. If it fails to offload, and uses software, that is fine. Not
optimal, but O.K.

	Andrew

Re: KSZ9477 HSR Offloading

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 9099 bytes --]

Hi Frieder,

> On 29.01.25 2:58 PM, Lukasz Majewski wrote:
> > Hi Frieder,
> >   
> >> Hi Lukasz,
> >>
> >> On 29.01.25 12:17 PM, Lukasz Majewski wrote:  
> >>> Hi Frieder,
> >>>     
> >>>> On 29.01.25 8:24 AM, Frieder Schrempf wrote:    
> >>>>> Hi Andrew,
> >>>>>
> >>>>> On 28.01.25 6:51 PM, Andrew Lunn wrote:      
> >>>>>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf
> >>>>>> wrote:      
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup
> >>>>>>> looks like this:
> >>>>>>>
> >>>>>>> +-------------+         +-------------+
> >>>>>>> |             |         |             |
> >>>>>>> |   Node A    |         |   Node D    |
> >>>>>>> |             |         |             |
> >>>>>>> |             |         |             |
> >>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>>>>>> +--+-------+--+         +--+------+---+
> >>>>>>>    |       |               |      |
> >>>>>>>    |       +---------------+      |
> >>>>>>>    |                              |
> >>>>>>>    |       +---------------+      |
> >>>>>>>    |       |               |      |
> >>>>>>> +--+-------+--+         +--+------+---+
> >>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
> >>>>>>> |             |         |             |
> >>>>>>> |             |         |             |
> >>>>>>> |   Node B    |         |   Node C    |
> >>>>>>> |             |         |             |
> >>>>>>> +-------------+         +-------------+
> >>>>>>>
> >>>>>>> On each device the LAN1 and LAN2 are added as HSR slaves.
> >>>>>>> Then I try to do ping tests between each of the HSR
> >>>>>>> interfaces.
> >>>>>>>
> >>>>>>> The result is that I can reach the neighboring nodes just
> >>>>>>> fine, but I can't reach the remote node that needs packages
> >>>>>>> to be forwarded through the other nodes. For example I can't
> >>>>>>> ping from node A to C.
> >>>>>>>
> >>>>>>> I've tried to disable HW offloading in the driver and then
> >>>>>>> everything starts working.
> >>>>>>>
> >>>>>>> Is this a problem with HW offloading in the KSZ driver, or am
> >>>>>>> I missing something essential?      
> >>>
> >>> Thanks for looking and testing such large scale setup.
> >>>     
> >>>>>>
> >>>>>> How are IP addresses configured? I assume you have a bridge,
> >>>>>> LAN1 and LAN2 are members of the bridge, and the IP address is
> >>>>>> on the bridge interface?      
> >>>>>
> >>>>> I have a HSR interface on each node that covers LAN1 and LAN2 as
> >>>>> slaves and the IP addresses are on those HSR interfaces. For
> >>>>> node A:
> >>>>>
> >>>>> ip link add name hsr type hsr slave1 lan1 slave2 lan2
> >>>>> supervision 45 version 1
> >>>>> ip addr add 172.20.1.1/24 dev hsr
> >>>>>
> >>>>> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24
> >>>>> and 172.20.1.4/24 respectively.
> >>>>>
> >>>>> Then on node A, I'm doing:
> >>>>>
> >>>>> ping 172.20.1.2 # neighboring node B works
> >>>>> ping 172.20.1.4 # neighboring node D works
> >>>>> ping 172.20.1.3 # remote node C works only if I disable
> >>>>> offloading      
> >>>>
> >>>> BTW, it's enough to disable the offloading of the forwarding for
> >>>> HSR frames to make it work.
> >>>>
> >>>> --- a/drivers/net/dsa/microchip/ksz9477.c
> >>>> +++ b/drivers/net/dsa/microchip/ksz9477.c
> >>>> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct
> >>>> ksz_device *dev, int port, u32 val)
> >>>>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as
> >>>> HSR frames
> >>>>   * can be forwarded in the switch fabric between HSR ports.
> >>>>   */
> >>>> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
> >>>> NETIF_F_HW_HSR_FWD)
> >>>> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
> >>>>
> >>>>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
> >>>> net_device *hsr)
> >>>>  {
> >>>> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch
> >>>> *ds, int port, struct net_device *hsr)
> >>>>         /* Program which port(s) shall support HSR */
> >>>>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port),
> >>>> BIT(port));
> >>>>
> >>>> -       /* Forward frames between HSR ports (i.e. bridge together
> >>>> HSR ports) */
> >>>> -       if (dev->hsr_ports) {
> >>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> >>>> -                       hsr_ports |= BIT(hsr_dp->index);
> >>>> -
> >>>> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
> >>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
> >>>> -                       ksz9477_cfg_port_member(dev,
> >>>> hsr_dp->index, hsr_ports);
> >>>> -       }
> >>>> -
> >>>>         if (!dev->hsr_ports) {
> >>>>                 /* Enable discarding of received HSR frames */
> >>>>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);    
> >>>
> >>> This means that KSZ9477 forwarding is dropping frames when HW
> >>> acceleration is used (for non "neighbour" nodes).
> >>>
> >>> On my setup I only had 2 KSZ9477 devel boards.
> >>>
> >>> And as you wrote - the SW based one works, so extending
> >>> https://elixir.bootlin.com/linux/v6.12-rc2/source/tools/testing/selftests/net/hsr
> >>>
> >>> would not help in this case.    
> >>
> >> I see. With two boards you can't test the accelerated forwarding.
> >> So how did you test the forwarding at all? Or are you telling me,
> >> that this was added to the driver without prior testing (which
> >> seems a bit bold and unusual)?  
> > 
> > The packet forwarding is for generating two frames copies on two HSR
> > coupled ports on a single KSZ9477:  
> 
> Isn't that what duplication aka NETIF_F_HW_HSR_DUP is for?

As I mentioned - the NETIF_F_HW_HSR_DUP is to remove duplicated frames.

NETIF_F_HW_HSR_FWD is to in-hw generate frame copy for HSR port to be
sent:
https://elixir.bootlin.com/linux/v6.13/source/drivers/net/dsa/microchip/ksz9477.c#L1252

> 
> > 
> > https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ApplicationNotes/ApplicationNotes/AN3474-KSZ9477-High-Availability-Seamless-Redundancy-Application-Note-00003474A.pdf
> > 
> > The KSZ9477 chip also supports RX packet duplication removal, but
> > cannot guarantee 100% success (so as a fallback it is done in SW).
> > 
> > The infrastructure from:
> > https://elixir.bootlin.com/linux/v6.13/source/tools/testing/selftests/net/hsr/hsr_redbox.sh#L50
> > 
> > is enough to test HW accelerated forwarding (of KSZ9477) from NS1
> > and NS2.  
> 
> I'm not really sure if I get it. In this setup NS1 and NS2 are
> connected via HSR link (two physical links). On one side packets are
> sent duplicated on both physical ports. On the receiving side the
> duplication is removed and one packet is forwarded to the CPU.
> 
> Where is forwarding involved here? 

In-HW forwarding is when KSZ9477 duplicates frame to be send on second
HSR aware port.

(only 2 of them can be coupled to have in-hw support for duplication
and forwarding. Creating more hsr "interfaces" would just use SW).

> Isn't forwarding only for cases
> with one intermediate node between the sending and receiving node?

This kind of "forwarding" is done in software in the hsr driver.

> 
> >   
> >>
> >> Anyway, do you have any suggestions for debugging this?
> >> Unfortunately I know almost nothing about this topic. But I can
> >> offer to test on my setup, at least for now. I don't know how long
> >> I will still have access to the hardware.  
> > 
> > For some reason only frames to neighbours are delivered.
> > 
> > So those are removed at some point (either in KSZ9477 HW or in HSR
> > driver itself).
> > 
> > Do you have some dumps from tshark/wireshark to share?
> >   
> >>
> >> If we can't find a proper solution in the long run, I will probably
> >> send a patch to disable the accelerated forwarding to at least make
> >> HSR work by default.  
> > 
> > As I've noted above - the HW accelerated forwarding is in the
> > KSZ9477 chip.  
> 
> Yeah, but if the HW accelerated forwarding doesn't work

The "forwarding" in KSZ9477 IC works OK, as frames are duplicated (i.e.
forwarded) to both HSR coupled ports.

The problem is with dropping frames travelling in connected KSZ9477
devices.

> it would be
> better to use no acceleration and have it work in SW at least by
> default, right?

IMHO, it would be best to fix the code.

> 
> > 
> > The code which you uncomment, is following what I've understood from
> > the standard (and maybe the bug is somewhere there).  
> 
> Ok, thanks for explaining. I will see if I can find some time to
> gather some more information on the problem.

That would be very helpful. Thanks in advance for it.


Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Erika Unter
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

Hi Lukasz,

On 29.01.25 11:52 PM, Lukasz Majewski wrote:
> Hi Frieder,
> 
>> On 29.01.25 2:58 PM, Lukasz Majewski wrote:
>>> Hi Frieder,
>>>   
>>>> Hi Lukasz,
>>>>
>>>> On 29.01.25 12:17 PM, Lukasz Majewski wrote:  
>>>>> Hi Frieder,
>>>>>     
>>>>>> On 29.01.25 8:24 AM, Frieder Schrempf wrote:    
>>>>>>> Hi Andrew,
>>>>>>>
>>>>>>> On 28.01.25 6:51 PM, Andrew Lunn wrote:      
>>>>>>>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf
>>>>>>>> wrote:      
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup
>>>>>>>>> looks like this:
>>>>>>>>>
>>>>>>>>> +-------------+         +-------------+
>>>>>>>>> |             |         |             |
>>>>>>>>> |   Node A    |         |   Node D    |
>>>>>>>>> |             |         |             |
>>>>>>>>> |             |         |             |
>>>>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>>>>>> +--+-------+--+         +--+------+---+
>>>>>>>>>    |       |               |      |
>>>>>>>>>    |       +---------------+      |
>>>>>>>>>    |                              |
>>>>>>>>>    |       +---------------+      |
>>>>>>>>>    |       |               |      |
>>>>>>>>> +--+-------+--+         +--+------+---+
>>>>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>>>>>> |             |         |             |
>>>>>>>>> |             |         |             |
>>>>>>>>> |   Node B    |         |   Node C    |
>>>>>>>>> |             |         |             |
>>>>>>>>> +-------------+         +-------------+
>>>>>>>>>
>>>>>>>>> On each device the LAN1 and LAN2 are added as HSR slaves.
>>>>>>>>> Then I try to do ping tests between each of the HSR
>>>>>>>>> interfaces.
>>>>>>>>>
>>>>>>>>> The result is that I can reach the neighboring nodes just
>>>>>>>>> fine, but I can't reach the remote node that needs packages
>>>>>>>>> to be forwarded through the other nodes. For example I can't
>>>>>>>>> ping from node A to C.
>>>>>>>>>
>>>>>>>>> I've tried to disable HW offloading in the driver and then
>>>>>>>>> everything starts working.
>>>>>>>>>
>>>>>>>>> Is this a problem with HW offloading in the KSZ driver, or am
>>>>>>>>> I missing something essential?      
>>>>>
>>>>> Thanks for looking and testing such large scale setup.
>>>>>     
>>>>>>>>
>>>>>>>> How are IP addresses configured? I assume you have a bridge,
>>>>>>>> LAN1 and LAN2 are members of the bridge, and the IP address is
>>>>>>>> on the bridge interface?      
>>>>>>>
>>>>>>> I have a HSR interface on each node that covers LAN1 and LAN2 as
>>>>>>> slaves and the IP addresses are on those HSR interfaces. For
>>>>>>> node A:
>>>>>>>
>>>>>>> ip link add name hsr type hsr slave1 lan1 slave2 lan2
>>>>>>> supervision 45 version 1
>>>>>>> ip addr add 172.20.1.1/24 dev hsr
>>>>>>>
>>>>>>> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24
>>>>>>> and 172.20.1.4/24 respectively.
>>>>>>>
>>>>>>> Then on node A, I'm doing:
>>>>>>>
>>>>>>> ping 172.20.1.2 # neighboring node B works
>>>>>>> ping 172.20.1.4 # neighboring node D works
>>>>>>> ping 172.20.1.3 # remote node C works only if I disable
>>>>>>> offloading      
>>>>>>
>>>>>> BTW, it's enough to disable the offloading of the forwarding for
>>>>>> HSR frames to make it work.
>>>>>>
>>>>>> --- a/drivers/net/dsa/microchip/ksz9477.c
>>>>>> +++ b/drivers/net/dsa/microchip/ksz9477.c
>>>>>> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct
>>>>>> ksz_device *dev, int port, u32 val)
>>>>>>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as
>>>>>> HSR frames
>>>>>>   * can be forwarded in the switch fabric between HSR ports.
>>>>>>   */
>>>>>> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
>>>>>> NETIF_F_HW_HSR_FWD)
>>>>>> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
>>>>>>
>>>>>>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
>>>>>> net_device *hsr)
>>>>>>  {
>>>>>> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch
>>>>>> *ds, int port, struct net_device *hsr)
>>>>>>         /* Program which port(s) shall support HSR */
>>>>>>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port),
>>>>>> BIT(port));
>>>>>>
>>>>>> -       /* Forward frames between HSR ports (i.e. bridge together
>>>>>> HSR ports) */
>>>>>> -       if (dev->hsr_ports) {
>>>>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>>>>>> -                       hsr_ports |= BIT(hsr_dp->index);
>>>>>> -
>>>>>> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
>>>>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>>>>>> -                       ksz9477_cfg_port_member(dev,
>>>>>> hsr_dp->index, hsr_ports);
>>>>>> -       }
>>>>>> -
>>>>>>         if (!dev->hsr_ports) {
>>>>>>                 /* Enable discarding of received HSR frames */
>>>>>>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);    
>>>>>
>>>>> This means that KSZ9477 forwarding is dropping frames when HW
>>>>> acceleration is used (for non "neighbour" nodes).
>>>>>
>>>>> On my setup I only had 2 KSZ9477 devel boards.
>>>>>
>>>>> And as you wrote - the SW based one works, so extending
>>>>> https://elixir.bootlin.com/linux/v6.12-rc2/source/tools/testing/selftests/net/hsr
>>>>>
>>>>> would not help in this case.    
>>>>
>>>> I see. With two boards you can't test the accelerated forwarding.
>>>> So how did you test the forwarding at all? Or are you telling me,
>>>> that this was added to the driver without prior testing (which
>>>> seems a bit bold and unusual)?  
>>>
>>> The packet forwarding is for generating two frames copies on two HSR
>>> coupled ports on a single KSZ9477:  
>>
>> Isn't that what duplication aka NETIF_F_HW_HSR_DUP is for?
> 
> As I mentioned - the NETIF_F_HW_HSR_DUP is to remove duplicated frames.
> 
> NETIF_F_HW_HSR_FWD is to in-hw generate frame copy for HSR port to be
> sent:
> https://elixir.bootlin.com/linux/v6.13/source/drivers/net/dsa/microchip/ksz9477.c#L1252

Ok, so you are using a different definition for the "forwarding". What
puzzles me is the explanation for the HSR feature flags here [1]. They
seem to suggest the following, which differs from your explanation:

Forwarding (aka NETIF_F_HW_HSR_FWD):

"Forwarding involves automatically forwarding between redundant ports in
an HSR."

This sounds more like the "forwarding" of a HSR frame within the Ring,
between two HSR ports, that I was thinking of.

Duplication (aka NETIF_F_HW_HSR_DUP):

"Duplication involves the switch automatically sending a single frame
from the CPU port to both redundant ports."

This is contradictory to what you wrote above and sounds more
reasonable. This is what you instead call forwarding above.

Are you mixing things up, here? Am I completely on the wrong track? I'm
just trying to understand the basics here.

>>
>>>
>>> https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ApplicationNotes/ApplicationNotes/AN3474-KSZ9477-High-Availability-Seamless-Redundancy-Application-Note-00003474A.pdf
>>>
>>> The KSZ9477 chip also supports RX packet duplication removal, but
>>> cannot guarantee 100% success (so as a fallback it is done in SW).
>>>
>>> The infrastructure from:
>>> https://elixir.bootlin.com/linux/v6.13/source/tools/testing/selftests/net/hsr/hsr_redbox.sh#L50
>>>
>>> is enough to test HW accelerated forwarding (of KSZ9477) from NS1
>>> and NS2.  
>>
>> I'm not really sure if I get it. In this setup NS1 and NS2 are
>> connected via HSR link (two physical links). On one side packets are
>> sent duplicated on both physical ports. On the receiving side the
>> duplication is removed and one packet is forwarded to the CPU.
>>
>> Where is forwarding involved here? 
> 
> In-HW forwarding is when KSZ9477 duplicates frame to be send on second
> HSR aware port.
> 
> (only 2 of them can be coupled to have in-hw support for duplication
> and forwarding. Creating more hsr "interfaces" would just use SW).
> 
>> Isn't forwarding only for cases
>> with one intermediate node between the sending and receiving node?
> 
> This kind of "forwarding" is done in software in the hsr driver.

But according to the official explanation of the flags [1] this kind of
forwarding is exactly what NETIF_F_HW_HSR_FWD seems to be about.

> 
>>
>>>   
>>>>
>>>> Anyway, do you have any suggestions for debugging this?
>>>> Unfortunately I know almost nothing about this topic. But I can
>>>> offer to test on my setup, at least for now. I don't know how long
>>>> I will still have access to the hardware.  
>>>
>>> For some reason only frames to neighbours are delivered.
>>>
>>> So those are removed at some point (either in KSZ9477 HW or in HSR
>>> driver itself).
>>>
>>> Do you have some dumps from tshark/wireshark to share?
>>>   
>>>>
>>>> If we can't find a proper solution in the long run, I will probably
>>>> send a patch to disable the accelerated forwarding to at least make
>>>> HSR work by default.  
>>>
>>> As I've noted above - the HW accelerated forwarding is in the
>>> KSZ9477 chip.  
>>
>> Yeah, but if the HW accelerated forwarding doesn't work
> 
> The "forwarding" in KSZ9477 IC works OK, as frames are duplicated (i.e.
> forwarded) to both HSR coupled ports.

No, I don't think duplication and forwarding refer to the same thing
here. At least it doesn't make sense for me.

> The problem is with dropping frames travelling in connected KSZ9477
> devices.

I'm not really sure.

> 
>> it would be
>> better to use no acceleration and have it work in SW at least by
>> default, right?
> 
> IMHO, it would be best to fix the code.

Agreed.

>>
>>>
>>> The code which you uncomment, is following what I've understood from
>>> the standard (and maybe the bug is somewhere there).  
>>
>> Ok, thanks for explaining. I will see if I can find some time to
>> gather some more information on the problem.
> 
> That would be very helpful. Thanks in advance for it.
One more information I just found out: I can leave ksz9477_hsr_join()
untouched and only remove the feature flags from
KSZ9477_SUPPORTED_HSR_FEATURES to make things work.

Broken:

#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
NETIF_F_HW_HSR_FWD)

Works:

#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)

Works:

#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_FWD)

Works:

#define KSZ9477_SUPPORTED_HSR_FEATURES (0)

Thanks
Frieder

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit?id=dcf0cd1cc58b8e88793ad6531db9b3a47324ca09

Re: KSZ9477 HSR Offloading

[Frieder Schrempf]

On 30.01.25 11:19 AM, Frieder Schrempf wrote:
> Hi Lukasz,
> 
> On 29.01.25 11:52 PM, Lukasz Majewski wrote:
>> Hi Frieder,
>>
>>> On 29.01.25 2:58 PM, Lukasz Majewski wrote:
>>>> Hi Frieder,
>>>>   
>>>>> Hi Lukasz,
>>>>>
>>>>> On 29.01.25 12:17 PM, Lukasz Majewski wrote:  
>>>>>> Hi Frieder,
>>>>>>     
>>>>>>> On 29.01.25 8:24 AM, Frieder Schrempf wrote:    
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> On 28.01.25 6:51 PM, Andrew Lunn wrote:      
>>>>>>>>> On Tue, Jan 28, 2025 at 05:14:46PM +0100, Frieder Schrempf
>>>>>>>>> wrote:      
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'm trying out HSR support on KSZ9477 with v6.12. My setup
>>>>>>>>>> looks like this:
>>>>>>>>>>
>>>>>>>>>> +-------------+         +-------------+
>>>>>>>>>> |             |         |             |
>>>>>>>>>> |   Node A    |         |   Node D    |
>>>>>>>>>> |             |         |             |
>>>>>>>>>> |             |         |             |
>>>>>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>>>>>>> +--+-------+--+         +--+------+---+
>>>>>>>>>>    |       |               |      |
>>>>>>>>>>    |       +---------------+      |
>>>>>>>>>>    |                              |
>>>>>>>>>>    |       +---------------+      |
>>>>>>>>>>    |       |               |      |
>>>>>>>>>> +--+-------+--+         +--+------+---+
>>>>>>>>>> | LAN1   LAN2 |         | LAN1   LAN2 |
>>>>>>>>>> |             |         |             |
>>>>>>>>>> |             |         |             |
>>>>>>>>>> |   Node B    |         |   Node C    |
>>>>>>>>>> |             |         |             |
>>>>>>>>>> +-------------+         +-------------+
>>>>>>>>>>
>>>>>>>>>> On each device the LAN1 and LAN2 are added as HSR slaves.
>>>>>>>>>> Then I try to do ping tests between each of the HSR
>>>>>>>>>> interfaces.
>>>>>>>>>>
>>>>>>>>>> The result is that I can reach the neighboring nodes just
>>>>>>>>>> fine, but I can't reach the remote node that needs packages
>>>>>>>>>> to be forwarded through the other nodes. For example I can't
>>>>>>>>>> ping from node A to C.
>>>>>>>>>>
>>>>>>>>>> I've tried to disable HW offloading in the driver and then
>>>>>>>>>> everything starts working.
>>>>>>>>>>
>>>>>>>>>> Is this a problem with HW offloading in the KSZ driver, or am
>>>>>>>>>> I missing something essential?      
>>>>>>
>>>>>> Thanks for looking and testing such large scale setup.
>>>>>>     
>>>>>>>>>
>>>>>>>>> How are IP addresses configured? I assume you have a bridge,
>>>>>>>>> LAN1 and LAN2 are members of the bridge, and the IP address is
>>>>>>>>> on the bridge interface?      
>>>>>>>>
>>>>>>>> I have a HSR interface on each node that covers LAN1 and LAN2 as
>>>>>>>> slaves and the IP addresses are on those HSR interfaces. For
>>>>>>>> node A:
>>>>>>>>
>>>>>>>> ip link add name hsr type hsr slave1 lan1 slave2 lan2
>>>>>>>> supervision 45 version 1
>>>>>>>> ip addr add 172.20.1.1/24 dev hsr
>>>>>>>>
>>>>>>>> The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24
>>>>>>>> and 172.20.1.4/24 respectively.
>>>>>>>>
>>>>>>>> Then on node A, I'm doing:
>>>>>>>>
>>>>>>>> ping 172.20.1.2 # neighboring node B works
>>>>>>>> ping 172.20.1.4 # neighboring node D works
>>>>>>>> ping 172.20.1.3 # remote node C works only if I disable
>>>>>>>> offloading      
>>>>>>>
>>>>>>> BTW, it's enough to disable the offloading of the forwarding for
>>>>>>> HSR frames to make it work.
>>>>>>>
>>>>>>> --- a/drivers/net/dsa/microchip/ksz9477.c
>>>>>>> +++ b/drivers/net/dsa/microchip/ksz9477.c
>>>>>>> @@ -1267,7 +1267,7 @@ int ksz9477_tc_cbs_set_cinc(struct
>>>>>>> ksz_device *dev, int port, u32 val)
>>>>>>>   * Moreover, the NETIF_F_HW_HSR_FWD feature is also enabled, as
>>>>>>> HSR frames
>>>>>>>   * can be forwarded in the switch fabric between HSR ports.
>>>>>>>   */
>>>>>>> -#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
>>>>>>> NETIF_F_HW_HSR_FWD)
>>>>>>> +#define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
>>>>>>>
>>>>>>>  void ksz9477_hsr_join(struct dsa_switch *ds, int port, struct
>>>>>>> net_device *hsr)
>>>>>>>  {
>>>>>>> @@ -1279,16 +1279,6 @@ void ksz9477_hsr_join(struct dsa_switch
>>>>>>> *ds, int port, struct net_device *hsr)
>>>>>>>         /* Program which port(s) shall support HSR */
>>>>>>>         ksz_rmw32(dev, REG_HSR_PORT_MAP__4, BIT(port),
>>>>>>> BIT(port));
>>>>>>>
>>>>>>> -       /* Forward frames between HSR ports (i.e. bridge together
>>>>>>> HSR ports) */
>>>>>>> -       if (dev->hsr_ports) {
>>>>>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>>>>>>> -                       hsr_ports |= BIT(hsr_dp->index);
>>>>>>> -
>>>>>>> -               hsr_ports |= BIT(dsa_upstream_port(ds, port));
>>>>>>> -               dsa_hsr_foreach_port(hsr_dp, ds, hsr)
>>>>>>> -                       ksz9477_cfg_port_member(dev,
>>>>>>> hsr_dp->index, hsr_ports);
>>>>>>> -       }
>>>>>>> -
>>>>>>>         if (!dev->hsr_ports) {
>>>>>>>                 /* Enable discarding of received HSR frames */
>>>>>>>                 ksz_read8(dev, REG_HSR_ALU_CTRL_0__1, &data);    
>>>>>>
>>>>>> This means that KSZ9477 forwarding is dropping frames when HW
>>>>>> acceleration is used (for non "neighbour" nodes).
>>>>>>
>>>>>> On my setup I only had 2 KSZ9477 devel boards.
>>>>>>
>>>>>> And as you wrote - the SW based one works, so extending
>>>>>> https://elixir.bootlin.com/linux/v6.12-rc2/source/tools/testing/selftests/net/hsr
>>>>>>
>>>>>> would not help in this case.    
>>>>>
>>>>> I see. With two boards you can't test the accelerated forwarding.
>>>>> So how did you test the forwarding at all? Or are you telling me,
>>>>> that this was added to the driver without prior testing (which
>>>>> seems a bit bold and unusual)?  
>>>>
>>>> The packet forwarding is for generating two frames copies on two HSR
>>>> coupled ports on a single KSZ9477:  
>>>
>>> Isn't that what duplication aka NETIF_F_HW_HSR_DUP is for?
>>
>> As I mentioned - the NETIF_F_HW_HSR_DUP is to remove duplicated frames.
>>
>> NETIF_F_HW_HSR_FWD is to in-hw generate frame copy for HSR port to be
>> sent:
>> https://elixir.bootlin.com/linux/v6.13/source/drivers/net/dsa/microchip/ksz9477.c#L1252
> 
> Ok, so you are using a different definition for the "forwarding". What
> puzzles me is the explanation for the HSR feature flags here [1]. They
> seem to suggest the following, which differs from your explanation:
> 
> Forwarding (aka NETIF_F_HW_HSR_FWD):
> 
> "Forwarding involves automatically forwarding between redundant ports in
> an HSR."
> 
> This sounds more like the "forwarding" of a HSR frame within the Ring,
> between two HSR ports, that I was thinking of.
> 
> Duplication (aka NETIF_F_HW_HSR_DUP):
> 
> "Duplication involves the switch automatically sending a single frame
> from the CPU port to both redundant ports."
> 
> This is contradictory to what you wrote above and sounds more
> reasonable. This is what you instead call forwarding above.
> 
> Are you mixing things up, here? Am I completely on the wrong track? I'm
> just trying to understand the basics here.
> 
>>>
>>>>
>>>> https://ww1.microchip.com/downloads/aemDocuments/documents/UNG/ApplicationNotes/ApplicationNotes/AN3474-KSZ9477-High-Availability-Seamless-Redundancy-Application-Note-00003474A.pdf
>>>>
>>>> The KSZ9477 chip also supports RX packet duplication removal, but
>>>> cannot guarantee 100% success (so as a fallback it is done in SW).
>>>>
>>>> The infrastructure from:
>>>> https://elixir.bootlin.com/linux/v6.13/source/tools/testing/selftests/net/hsr/hsr_redbox.sh#L50
>>>>
>>>> is enough to test HW accelerated forwarding (of KSZ9477) from NS1
>>>> and NS2.  
>>>
>>> I'm not really sure if I get it. In this setup NS1 and NS2 are
>>> connected via HSR link (two physical links). On one side packets are
>>> sent duplicated on both physical ports. On the receiving side the
>>> duplication is removed and one packet is forwarded to the CPU.
>>>
>>> Where is forwarding involved here? 
>>
>> In-HW forwarding is when KSZ9477 duplicates frame to be send on second
>> HSR aware port.
>>
>> (only 2 of them can be coupled to have in-hw support for duplication
>> and forwarding. Creating more hsr "interfaces" would just use SW).
>>
>>> Isn't forwarding only for cases
>>> with one intermediate node between the sending and receiving node?
>>
>> This kind of "forwarding" is done in software in the hsr driver.
> 
> But according to the official explanation of the flags [1] this kind of
> forwarding is exactly what NETIF_F_HW_HSR_FWD seems to be about.
> 
>>
>>>
>>>>   
>>>>>
>>>>> Anyway, do you have any suggestions for debugging this?
>>>>> Unfortunately I know almost nothing about this topic. But I can
>>>>> offer to test on my setup, at least for now. I don't know how long
>>>>> I will still have access to the hardware.  
>>>>
>>>> For some reason only frames to neighbours are delivered.
>>>>
>>>> So those are removed at some point (either in KSZ9477 HW or in HSR
>>>> driver itself).
>>>>
>>>> Do you have some dumps from tshark/wireshark to share?
>>>>   
>>>>>
>>>>> If we can't find a proper solution in the long run, I will probably
>>>>> send a patch to disable the accelerated forwarding to at least make
>>>>> HSR work by default.  
>>>>
>>>> As I've noted above - the HW accelerated forwarding is in the
>>>> KSZ9477 chip.  
>>>
>>> Yeah, but if the HW accelerated forwarding doesn't work
>>
>> The "forwarding" in KSZ9477 IC works OK, as frames are duplicated (i.e.
>> forwarded) to both HSR coupled ports.
> 
> No, I don't think duplication and forwarding refer to the same thing
> here. At least it doesn't make sense for me.
> 
>> The problem is with dropping frames travelling in connected KSZ9477
>> devices.
> 
> I'm not really sure.
> 
>>
>>> it would be
>>> better to use no acceleration and have it work in SW at least by
>>> default, right?
>>
>> IMHO, it would be best to fix the code.
> 
> Agreed.
> 
>>>
>>>>
>>>> The code which you uncomment, is following what I've understood from
>>>> the standard (and maybe the bug is somewhere there).  
>>>
>>> Ok, thanks for explaining. I will see if I can find some time to
>>> gather some more information on the problem.
>>
>> That would be very helpful. Thanks in advance for it.
> One more information I just found out: I can leave ksz9477_hsr_join()
> untouched and only remove the feature flags from
> KSZ9477_SUPPORTED_HSR_FEATURES to make things work.
> 
> Broken:
> 
> #define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
> NETIF_F_HW_HSR_FWD)
> 
> Works:
> 
> #define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
> 
> Works:
> 
> #define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_FWD)
> 
> Works:
> 
> #define KSZ9477_SUPPORTED_HSR_FEATURES (0)

Please take this with a grain of salt. I've just seen, that the behavior
is not 100% consistent.

I'm pretty sure that things work reliably with the diff I posted before
as I have tested this case more thoroughly.

Re: KSZ9477 HSR Offloading

[Andrew Lunn]

> One more information I just found out: I can leave ksz9477_hsr_join()
> untouched and only remove the feature flags from
> KSZ9477_SUPPORTED_HSR_FEATURES to make things work.
> 
> Broken:
> 
> #define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP |
> NETIF_F_HW_HSR_FWD)
> 
> Works:
> 
> #define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_DUP)
> 
> Works:
> 
> #define KSZ9477_SUPPORTED_HSR_FEATURES (NETIF_F_HW_HSR_FWD)
> 
> Works:
> 
> #define KSZ9477_SUPPORTED_HSR_FEATURES (0)

It would be good to define "works". Are the packets getting delivered
to Linux and then the software HSR bridge in Linux is emitting the
packets back out the interfaces? Or is it all happening in hardware
and Linux never sees the packet.

Ideally we want the hardware to be doing all the work, in the "Works"
condition.

	Andrew

Re: KSZ9477 HSR Offloading

[Andrew Lunn]

> >> How are IP addresses configured? I assume you have a bridge, LAN1 and
> >> LAN2 are members of the bridge, and the IP address is on the bridge
> >> interface?
> > 
> > I have a HSR interface on each node that covers LAN1 and LAN2 as slaves
> > and the IP addresses are on those HSR interfaces. For node A:
> > 
> > ip link add name hsr type hsr slave1 lan1 slave2 lan2 supervision 45
> > version 1
> > ip addr add 172.20.1.1/24 dev hsr
> > 
> > The other nodes have the addresses 172.20.1.2/24, 172.20.1.3/24 and
> > 172.20.1.4/24 respectively.
> > 
> > Then on node A, I'm doing:
> > 
> > ping 172.20.1.2 # neighboring node B works
> > ping 172.20.1.4 # neighboring node D works
> > ping 172.20.1.3 # remote node C works only if I disable offloading
> 
> BTW, it's enough to disable the offloading of the forwarding for HSR
> frames to make it work.

HSR is not my area of expertise, but that makes sense, HSR is a L2
concept, so you need something like a bridge, which the hsr device is,
and it is where you put the IP address.

Hopefully Lukasz can help you.

	Andrew