[PATCH rtw-next] wifi: rtw89: do BIP in software when not in station mode

[V schurink <spam@acio.nl>]

rtw89_cam_sec_key_add() installs the AES-CMAC key (BIP, 802.11w PMF)
in hardware. rtw89_core_tx_update_sec_key() has no case for
WLAN_CIPHER_SUITE_AES_CMAC, and the driver does not set
IEEE80211_KEY_FLAG_GENERATE_MMIE. Thus, an AP with PMF sends each
protected group-addressed management frame without an MMIE, for
example a broadcast deauthentication. For each frame, the driver logs
"key cipher not supported 1027078". PMF clients discard these frames.

An AP sends these frames. Thus, return -EOPNOTSUPP for AES-CMAC when
the interface is not a station, so that mac80211 does BIP in software.
A station only receives these frames, so it keeps the IGTK in
hardware. This does not change WoWLAN, which uses the IGTK of a
station.

BIP in hardware does not work for an AP on RTL8922A. The test used
IEEE80211_KEY_FLAG_GENERATE_MMIE and RTW89_SEC_KEY_TYPE_BIP_CCMP128 in
the TX descriptor. The client did not accept the protected broadcast
deauthentication.

Tested on RTL8912AU (RTL8922A, USB) in AP mode with hostapd 2.10,
WPA2-PSK and SAE, ieee80211w=1. The client accepts the protected
broadcast deauthentication, and the driver does not log the warning.

Fixes: 940cd99625de ("wifi: rtw89: wow: support 802.11w PMF IGTK rekey")
Signed-off-by: V schurink <spam@acio.nl>
---
 drivers/net/wireless/realtek/rtw89/cam.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/net/wireless/realtek/rtw89/cam.c b/drivers/net/wireless/realtek/rtw89/cam.c
index 22f21a0..58a7e8f 100644
--- a/drivers/net/wireless/realtek/rtw89/cam.c
+++ b/drivers/net/wireless/realtek/rtw89/cam.c
@@ -514,6 +514,12 @@ int rtw89_cam_sec_key_add(struct rtw89_dev *rtwdev,
 		ext_key = true;
 		break;
 	case WLAN_CIPHER_SUITE_AES_CMAC:
+		/* TX does not support BIP. An AP sends group-addressed robust
+		 * management frames, so let mac80211 do BIP in software there.
+		 * A station only receives them; keep its IGTK in hardware.
+		 */
+		if (vif->type != NL80211_IFTYPE_STATION)
+			return -EOPNOTSUPP;
 		hw_key_type = RTW89_SEC_KEY_TYPE_BIP_CCMP128;
 		break;
 	default:
-- 
2.53.0
---
Also discussed in the out-of-tree repository:
https://github.com/morrownr/rtw89/pull/135