brcmfmac: regression using AP mode

[Stefan Wahren <stefan.wahren@i2se.com>]

Hi guys,

today i wanted to setup an access point on my Raspberry Pi 3 A+ (BCM43455). Unfortunately the hostapd 2.4 shipped with Raspbian failed with recent Linux kernel:

rfkill: initial event: idx=0 type=2 op=0 soft=0 hard=0
rfkill: initial event: idx=1 type=1 op=0 soft=0 hard=0
nl80211: Using driver-based roaming
nl80211: TDLS supported
nl80211: Supported cipher 00-0f-ac:1
nl80211: Supported cipher 00-0f-ac:5
nl80211: Supported cipher 00-0f-ac:2
nl80211: Supported cipher 00-0f-ac:4
nl80211: Supported cipher 00-0f-ac:6
nl80211: Using driver-based off-channel TX
nl80211: Supported vendor command: vendor_id=0x1018 subcmd=1
nl80211: Use separate P2P group interface (driver advertised support)
nl80211: Enable multi-channel concurrent (driver advertised support)
nl80211: use P2P_DEVICE support
nl80211: Disable use_monitor with device_ap_sme since no monitor mode support detected
nl80211: interface wlan0 in phy phy0
nl80211: Set mode ifindex 3 iftype 3 (AP)
nl80211: Setup AP(wlan0) - device_ap_sme=1 use_monitor=0
nl80211: Subscribe to mgmt frames with AP handle 0x114350 (device SME)
nl80211: Register frame type=0xd0 (WLAN_FC_STYPE_ACTION) nl_handle=0x114350 match=
nl80211: Register frame command failed (type=208): ret=-22 (Invalid argument)
nl80211: Register frame match - hexdump(len=0): [NULL]
nl80211: Could not configure driver mode
nl80211: deinit ifname=wlan0 disabled_11b_rates=0
nl80211: Remove monitor interface: refcount=0
nl80211: Remove beacon (ifindex=3)
netlink: Operstate: ifindex=3 linkmode=0 (kernel-control), operstate=6 (IF_OPER_UP)
nl80211: Set mode ifindex 3 iftype 2 (STATION)
nl80211: Teardown AP(wlan0) - device_ap_sme=1 use_monitor=0
nl80211 driver initialization failed.

I was able to bisect this issue down to this commit:

1204aa17f3b4 ("brcmfmac: set WIPHY_FLAG_HAVE_AP_SME flag")

After reverting this commit hostapd works as expected.

Best regards
Stefan