[BUG] mlxsw: Driver/Firmware disagreement about max headroom size

[BUG] mlxsw: Driver/Firmware disagreement about max headroom size

[Keno Fischer]

Hello mlxsw maintainers,

I don't think this bug is particularly consequential, but there appears to be a
disagreement between the driver and the firmware regarding the proper max
headroom size. On my SN3800, I see the following:

 ```
$ tc qdisc replace dev swp1 root handle 1: prio bands 8 priomap 0 1 2 3 4 5 6 7
$ tc qdisc replace dev swp2 root handle 1: prio bands 8 priomap 0 1 2 3 4 5 6 7
$ dcb buffer set dev swp1 \
      buffer-size all:0 0:1219K \
      prio-buffer 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
Attribute write: No buffer space available
$ dcb buffer set dev swp2 \
      buffer-size all:0 0:1219K \
      prio-buffer 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
[78711.680961] mlxsw_spectrum2 0000:07:00.0: EMAD reg access failed
(tid=43b7c51f003ef26e,reg_id=500c(pbmc),type=write,status=7(bad
parameter))
Attribute write: I/O error
```

I don't know why this differs between the two switch ports, but since
the driver goes through a lot of effort to calculate the max value
and return a nice error (as shown in the first command), it seems
like a bug that it wouldn't do so on the second port.

As an aside, as a novice to this area, it wasn't clear to me from the
docs (https://github.com/Mellanox/mlxsw/wiki/Quality-of-Service),
whether the headroom buffer is a separate piece of memory or
whether it steals part of the shared buffer memory space, which
is what prompted me to just try various values here. It also wasn't
clear to me whether the headroom config did anything for lossy PGs.
Hope my naive view can help improve the page :).

Thanks,
Keno

Re: [BUG] mlxsw: Driver/Firmware disagreement about max headroom size

[Petr Machata]

Keno Fischer <keno@juliahub.com> writes:

> Hello mlxsw maintainers,
>
> I don't think this bug is particularly consequential, but there appears to be a
> disagreement between the driver and the firmware regarding the proper max
> headroom size. On my SN3800, I see the following:
>
>  ```
> $ tc qdisc replace dev swp1 root handle 1: prio bands 8 priomap 0 1 2 3 4 5 6 7
> $ tc qdisc replace dev swp2 root handle 1: prio bands 8 priomap 0 1 2 3 4 5 6 7
> $ dcb buffer set dev swp1 \
>       buffer-size all:0 0:1219K \
>       prio-buffer 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
> Attribute write: No buffer space available
> $ dcb buffer set dev swp2 \
>       buffer-size all:0 0:1219K \
>       prio-buffer 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
> [78711.680961] mlxsw_spectrum2 0000:07:00.0: EMAD reg access failed
> (tid=43b7c51f003ef26e,reg_id=500c(pbmc),type=write,status=7(bad
> parameter))
> Attribute write: I/O error
> ```
>
> I don't know why this differs between the two switch ports, but since
> the driver goes through a lot of effort to calculate the max value
> and return a nice error (as shown in the first command), it seems
> like a bug that it wouldn't do so on the second port.

The driver is making the assumption that the headroom is allocated per
port, but in fact it's allocated per a higher-level "port cluster".
There's a separate HW resource to query how many ports are shared like
this, and then the ports are allocated into clusters linearly by their
local_port number.

I have something sketched up that might as well work, but I need to test
it a bit and see if it survives regression.

> As an aside, as a novice to this area, it wasn't clear to me from the
> docs (https://github.com/Mellanox/mlxsw/wiki/Quality-of-Service),
> whether the headroom buffer is a separate piece of memory or
> whether it steals part of the shared buffer memory space, which

My understanding is that the headroom space that is configured at a PG
cannot be used for other PG's and other ports' traffic. In that sense it
subtracts from the shared buffer -- when there is traffic from other
ports that is considered for admission into the shared buffer, it will
not be admitted if the only available space is in "foreign" headrooms.

> is what prompted me to just try various values here. It also wasn't
> clear to me whether the headroom config did anything for lossy PGs.
> Hope my naive view can help improve the page :).

It's been a while since I touched this stuff, so maybe I'm missing
something, but I think that configuring anything above MTU is only
really relevant for lossless flows. One other use case from the top of
my head is egress mirror, but the driver should account for that
automatically.