RE: rtw88: 8822CE passes corrupted A-MSDUs as hw-decrypted, poisoning CCMP replay counter

[Ping-Ke Shih <pkshih@realtek.com>]

Bob Beckett <bob.beckett@collabora.com> wrote:
> Hi Ping-Ke,
> 
> We are investigating a Wi-Fi failure on the Steam Deck LCD (RTL8822CE,
> rtw88) and would appreciate your input; details below.
> 
> Summary
> =======
> 
> On RTL8822CE (PCIe, rtw88), during heavy 5 GHz VHT downlink from one AP
> model,

Since The RTL8822CE works long time in field, is the symptom only on
specific AP with heavy load?

If you change to another AP or lower load (e.g. 100Mbps), does problem
still present?

> the driver occasionally hands mac80211 a frame with these
> properties:
> - the RX descriptor looks normal: CRC OK, ICV OK, AES, not
>   software-decrypted;

Luka's patch can fix partial, so is it still normal?

> - the 802.11 header is correct;
> - the bytes from the CCMP header onwards are garbage.

Do you have an air capture to compare with? I'd like to bisect the cause
if AP TX weird packets? or receive side process it wrongly?

> 
> The AP's own PN sequence is consistent before and after the bad frame,
> so the corruption happens on the receive side. 

Maybe this is a hint to make a patch to defer (drop) reporting weird packet,
as Luka has fixed 3 of 4 problems. 

> Public reports of the same symptom with this AP (Steam Deck LCD +
> Freebox Revolution, 5 GHz, drops when a download starts, fixed by
> disabling 802.11ac on the AP, works under Windows) go back to at least
> September 2024.

With disabling 802.11ac, how about the change of traffic? become not a heavy
load? Can you test with 802.11ac enabled but lower load as disabling 80211.ac?


> 
> Evidence 3: other RX-path anomalies in the same run
> ===================================================
> 
> - 33 s before the first jump, during the download and with no other
>   activity, one descriptor claims pkt_len 11619 (above the 11454
>   maximum), and rtw_rx_query_rx_desc() returns -EINVAL twice in the same
>   second. These are garbage descriptors, which the rtw-next
>   RX-descriptor validation correctly rejected. This was never seen in a
>   21-minute ~540 Mbit/s test against a different AP.

If we enlarge RTK_PCI_RX_BUF_SIZE (e.g. +1000 bytes), would it help?
But it is still weird to me why the pkt_len 11619 can present. 

For different AP, it might not send a specific size of packets, so it
never happens. No idea. 

> 
> Where the corruption appears to come from
> =========================================
> 
> Host DMA bookkeeping is consistent. rtw88's rtw_pci_dma_check() compares
> the RX tag in each buffer descriptor with the expected sequence and logs
> "pci bus timeout, check dma status" on a mismatch (an unconditional
> dev_warn). That warning appears zero times in all three runs, 

Good to know this. 

> so every
> buffer descriptor had been written when the driver consumed it. The
> start of each bad buffer is also fresh: the RX descriptor's sequence
> number (w2[11:0]) matches the 802.11 header's.
> 
> The corruption boundary does not line up with PCIe writes. The device
> has MaxPayload 128 bytes, and on this host nearly every RX buffer goes
> through SWIOTLB bounce slots, which start 2 KB-aligned because rtw88
> sets no min_align_mask (see below). So DMA writes split at 128-byte
> boundaries of the buffer. The good data ends 82 or 86 bytes into the
> buffer, which is not such a boundary.

What is the problem? Cache-line size? or device DMA corruption over boundary?

As Luka's suggestion, it is worth to try IOMMU. 

> Ruled out
> =========
> 
> - Oversized MPDUs (new_len > RTK_PCI_RX_BUF_SIZE): the bad frames are
>   3092 bytes, and A-MSDUs from this AP top out at about 9.7 KB.

The bad frames are never over RTK_PCI_RX_BUF_SIZE, right?

> - ASPM / deep LPS: jumps 3 and 4 happened seconds after reloading with
>   disable_aspm=1 disable_lps_deep=1.

With disable_aspm=1, you should turn off/on (cold reboot) platform to
take effect. 

> - Power save (iw ... power_save off) and beamformee (support_bf=0 +
>   reconnect): no effect.
> - Driver consuming a ring entry before its buffer descriptor was
>   written: no RX-tag mismatch ("pci bus timeout") in any run.
> - The AP sending a bad PN: the PN sequences on both sides of each bad
>   frame are consecutive.

Can you confirm this by an air capture?

> - The RX-descriptor validation added in rtw-next: it does not reject
>   these frames, because their descriptors are well-formed.
> 
> Questions for Realtek
> =====================
> 
> 1. Is there a known condition on 8822C (or 8822B/8821C) where a
>    hardware-decrypted MPDU or A-MSDU is delivered with a valid RX
>    descriptor (CRC32 0, ICV_ERR 0, SWDEC 0, enc AES) and a valid 802.11
>    header, but an invalid CCMP header and payload? Does the RX
>    descriptor or drv_info have any bit that flags it? For example
>    "payload invalid", "decrypt engine bypassed or aborted", or a
>    per-packet RX FIFO overflow.

As I know, no. 

> 2. What does the vendor/Windows driver check before trusting a
>    hardware-decrypted frame, and does it handle HISR1 RXFOVW / HISR0
>    RDU? rtw88 masks and ignores both.

Why do you think rtw88 needs to handle them?

RXFOVW and RDU are to tell driver the info, but data should be handled
by hardware properly -- with some missing data, but should not receive 
corrupted data to driver.

> 3. Are there RX FIFO, RX DMA, security-engine or RX aggregation settings
>    in the Windows driver for 8822CE that differ from rtw88's and matter
>    for high-rate VHT A-MSDU reception (80 MHz, 2SS MCS 8-9, ~3 KB
>    A-MSDUs at ~20k/s)?

I'm not very sure they are identical (but most similar). You can check
association frames by air sniffer to see their difference.

> 4. For completeness: on these chips can RXBD_IDX or the
>    buffer-descriptor RX tag ever be updated before the packet data DMA
>    has completed, as described for WiFi 7 in rtw89 0bc7d1d4e63c?

The RX tag in rtw88 located on memory allocated by dma_alloc_coherent(), and
the RX tag must be set after packet data MAC completed. 

> 5. rtw_rx_fill_rx_status() sets RX_FLAG_DECRYPTED whenever !swdec &&
>    enc_type != NONE, ignoring ICV_ERR. rtw89 sets RX_FLAG_FAILED_FCS_CRC
>    and not DECRYPTED on icv_err; mt76 marks such frames monitor-only;
>    rtlwifi drops them. rtw88 does not set BIT_AICV in RCR, so such
>    frames are presumably dropped in hardware, and these frames had
>    ICV_ERR clear anyway. Would you take a patch aligning rtw88 with
>    rtw89 regardless?

I suppose Luka's patch did them.

> Appendix: raw dumps (run C, MACs redacted)

If you have an air capture by another platform, it'd be helpful. 


In my perspective, we need an air capture to bisect which part causes
the garbage (or not properly decrypted). The packets with abnormal PN
not handled by Luka's patch need additional workaround to drop it.

Ping-Ke