b43: multiple parens bugs and double CORDIC_FIXED

[Alessio Ferri <alessio.ferri@mythread.it>]

I developed a tool to capture io traces of proprietary wl driver by
hooking on its io accessors and then started to check mainline against
the captured trace, i already spotted 3 bugs:

1) in b43_nphy_rev3_rssi_cal(struct b43_wldev *dev)

abs should accept only offset[j] and not offset[j] + 4, immediate result
is failure to reproduce the write 0x3e for the same reads

records #3723-#3740, from "PHY.WR   addr=0x01b8 val=0x003f"

2) In b43_nphy_load_samples()

shift has more priority than and, so samples[i].i & 0x3FF << 10 masks
with 0x3FF << 10 instead of shifting the masked value into place. The
in-phase component of a sample sits in the low ten bits, so it is
masked to zero. brcmsmac packs the same table as ((buf[t].i & 0x3ff) <<
10) | (buf[t].q & 0x3ff), in wlc_phy_loadsampletable_nphy()

3) In b43_nphy_gen_load_samples()

cordic_calc_iq() call CORDIC_FIXED(theta) as the first op, so the
pre-scaling of rot by 2^16 is wrong. For reference brcmsmac computes
the step as ((f_kHz * 36) / phy_bw) / 100 into an s32 and passes it in
unscaled. The wrong scaling and the double CORDIC_FIXED() result in
wrong values.

with both fixes it reproduce correctly:

records #8638-#8959, from "TBL.WR   id=0x0011 off=0x0000 len=160"

Bonus: with the trace I also found that I introduced too many values
for the table r2057_rev8_init, while wl write only a subsection of them:
the ones marked with an extra flag in the raw data.

I'll send patches in a few days once i verified them on hardware, do
you see any problems with the capture & compare approach?


LINK:
https://github.com/aleferri/b43-6362-wip/raw/refs/heads/main/router-data/dsl-3580l/opinit-ch1-ch6-bw20.decoded