ath12k: handling of HE and EHT capabilities

[Alexander Wilhelm <alexander.wilhelm@westermo.com>]

Hello devs,

I’m currently trying to fix the HE and EHT capabilities handling on the
big‑endian platform. Unfortunately, I don’t fully understand how exactly
these capabilities are supposed to be defined.

For example, I use the `iw` tool to display the capabilities and their
descriptions. The code for that has the following function prototypes:

    * void print_ht_capability(__u16 cap);
    * void print_vht_info(__u32 capa, const __u8 *mcs);
    * static void __print_he_capa(const __u16 *mac_cap,
                                  const __u16 *phy_cap,
                                  const __u16 *mcs_set, size_t mcs_len,
                                  const __u8 *ppet, int ppet_len,
                                  bool indent);
    * static void __print_eht_capa(int band,
                                   const __u8 *mac_cap,
                                   const __u32 *phy_cap,
                                   const __u8 *mcs_set, size_t mcs_len,
                                   const __u8 *ppet, size_t ppet_len,
                                   const __u16 *he_phy_cap,
                                   bool indent);

For HE capabilites in 6 GHz band I couldn't find the respective function.
Then I looked into `include/net/cfg80211.h` and examined the structures
that define the capability data types.

    struct ieee80211_sta_ht_cap {
        u16 cap; /* use IEEE80211_HT_CAP_ */
        bool ht_supported;
        u8 ampdu_factor;
        u8 ampdu_density;
        struct ieee80211_mcs_info mcs;
    };

    struct ieee80211_sta_vht_cap {
        bool vht_supported;
        u32 cap; /* use IEEE80211_VHT_CAP_ */
        struct ieee80211_vht_mcs_info vht_mcs;
    };

The structs for HT and VHT use `u16` and `u32` data types for the `cap`
variable, matching what `iw` does. That part is consistent.

    struct ieee80211_he_cap_elem {
        u8 mac_cap_info[6];
        u8 phy_cap_info[11];
    } __packed;

    struct ieee80211_he_6ghz_capa {
        /* uses IEEE80211_HE_6GHZ_CAP_* below */
        __le16 capa; }
    __packed;

However, for HE the types differ from the `iw` implementation. Here, `u8`
arrays are used instead of `u16` for MAC and PHY capabilities. The 6 GHz
capabilities use `u16`, which is also different.

    struct ieee80211_eht_cap_elem_fixed {
        u8 mac_cap_info[2];
        u8 phy_cap_info[9];
    } __packed;

For EHT, `u8` arrays are also used for both MAC and PHY caps, instead of
`u32` for the PHY caps as in the `iw` implementation.

The current `ath12k` implementation always uses `u32` values, which does
not work on big‑endian platforms:

    ath12k_pci 0001:01:00.0: BAR 0: assigned [mem 0xc00000000-0xc001fffff 64bit]
    ath12k_pci 0001:01:00.0: MSI vectors: 1
    ath12k_pci 0001:01:00.0: Hardware name: qcn9274 hw2.0
    ath12k_pci 0001:01:00.0: qmi dma allocation failed (29360128 B type 1), will try later with small size
    ath12k_pci 0001:01:00.0: memory type 10 not supported
    ath12k_pci 0001:01:00.0: chip_id 0x0 chip_family 0xb board_id 0x1005 soc_id 0x401a2200
    ath12k_pci 0001:01:00.0: fw_version 0x111300d6 fw_build_timestamp 2024-08-06 08:43 fw_build_id QC_IMAGE_VERSION_STRING=WLAN.WBE.1.1.1-00214-QCAHKSWPL_SILICONZ-1
    ath12k_pci 0001:01:00.0: leaving PCI ASPM disabled to avoid MHI M2 problems
    ath12k_pci 0001:01:00.0: Invalid module id 2
    ath12k_pci 0001:01:00.0: failed to parse tlv -22
    ath12k_pci 0001:01:00.0: ieee80211 registration failed: -22
    ath12k_pci 0001:01:00.0: failed register the radio with mac80211: -22
    ath12k_pci 0001:01:00.0: failed to create pdev core: -22
    ath12k_pci 0001:01:00.0: qmi failed set mode request, mode: 4, err = -110
    ath12k_pci 0001:01:00.0: qmi failed to send wlan mode off

I want to address and fix this issue. However, I cannot apply the “never
break the userspace” rule here, as it seems, it is already broken. Can
someone help clarify which datatypes are supposed to be used? Once that is
clear, I can fix the `ath12k` implementation.


Best regards
Alexander Wilhelm