Performance of Reassociation

Performance of Reassociation

[Jesuiter, Henry]

Hello, 

we are using the rtw88 driver (with kernel 5.4) together with a RTL8822ce chip (Firmware 9.9.15, WOW 9.9.4, H2C Version 15) on a PCI-Express card. We stumbled about long roaming durations and investigated a little into this issue, by using different hardware:
    * Freescale i.MX6/ARMv7
    * Intel Core i5-7300) 

and different distributions:
   * Yocto Hardknott/3.3.6, Kernel 5.4
   * Buildroot, Kernel 6.6
   * Ubuntu 20.04, Kernel 5.15
   * Ubuntu 24.04, Kernel 6.8 

and found roaming times from 450ms (Ubuntu 24.04, Intel) up to 900ms (Yocto 3.3.6, FreeScale). 

Since we are used to much shorter roaming times (about 200ms even on the Freescale), we did some research and found that the driver is taking most time on two occasions: 
1. Reinitializing the MAC (that was shutdown completely on the previous disconnect)
2. Recalibration of the PHY (maybe due to the MAC shutdown too)

The following is a (shortened) trace of a roaming between two APs on our device. Please note: this log is taken with debug mask 0xffffffff in /sys/module/rtw88_core/parameters/debug_mask - so there might be some timing impact just due to the output, but we see similar results by setting debug mask to 0x0. We also added some output for function tracing. 

1. disconnect and shutdown the  MAC after leaving the old AP
    [  561.500691] wlan0: disconnect from AP <old_ap> for new auth to <new_ap>
    [  561.532634] rtw_8822ce 0000:01:00.0: send H2C content 00000001 00000000
    [  561.532656] rtw_8822ce 0000:01:00.0: sta <old_ssid> with macid 0 left
    [  561.557185] rtw_8822ce 0000:01:00.0: this vif is not mu bfee
1.1 MAC shutdown happens here
    [  561.557589] rtw_8822ce 0000:01:00.0: rtw_mac_power_switch() 

2. Start authentication to the new AP
    [  561.557691] wlan0: authenticate with <new_ssid>
2.1 Reinitialize the hardware 
    [  561.577099] rtw_8822ce 0000:01:00.0:  rtw_power_on()
    [  561.577109] rtw_8822ce 0000:01:00.0:  rtw_hci_setup()
    [  561.577128] rtw_8822ce 0000:01:00.0:  rtw_mac_power_on()
    [  561.577262] rtw_8822ce 0000:01:00.0:  rtw_mac_power_switch()
    [  561.580561] rtw_8822ce 0000:01:00.0:  rtw_wait_firmware_completion()
    [  561.584826] rtw_8822ce 0000:01:00.0:  rtw_download_firmware()
=> about 40ms here
    [  561.614635] rtw_8822ce 0000:01:00.0: rtw_hci_setup()
    [  561.614657] rtw_8822ce 0000:01:00.0: rtw_mac_init()
=> about 250ms here
    [  561.876080] rtw_8822ce 0000:01:00.0: rtw_hci_start()
    ...
2.2 PHY calibration
    [  561.877140] rtw_8822ce 0000:01:00.0: [RFK] WiFi / BT RFK handshake start!!
    ...
    [  561.896456] rtw_8822ce 0000:01:00.0: [RFK] WiFi / BT RFK handshake finish!!
    [  561.936393] rtw_8822ce 0000:01:00.0: [DPK] s0 dpk start
    [  562.056272] rtw_8822ce 0000:01:00.0: [DPK] s0 dpk finish
    [  562.056280] rtw_8822ce 0000:01:00.0: [DPK] s1 dpk start
    [  562.176275] rtw_8822ce 0000:01:00.0: [DPK] s1 dpk finish
=> about additional 300ms until here (that's the phy calibration stuff)
    ...
    [  562.297750] wlan0: send auth to 1c:28:af:40:fc:70 (try 1/3)

So, we are wondering if this long duration times for reassociation are hardware-related (maybe even someone uses the same chip and does not experience these delays)? 

On the software side, is there some optimization potential known in the software/driver that is just not realized yet? We would be happy to realize/implement those potentials. To be more specific: We are looking for a way that the driver does not need to reinitialize/restart the hardware while roaming in order to reduce that part of the roaming delay specifically (and maybe skip/shorten the phy_calibration part). Any ideas are welcome. We are also happy to provide any additional logs that might be needed (please specify) and to answer any additional question you might have. 

Viele Grüße / Best regards
Henry Jesuiter

Teamleiter Firmware / Team Leader Firmware
E-Mail: henry.jesuiter@softing.com

Softing Automotive Electronics GmbH
Richard-Reitzner-Allee 6 – 85540 Haar - Germany
Fax +49 89 456 56 -499 – www.automotive.softing.com
Sitz: Haar bei München, Amtsgericht München, HRB 184930
Geschäftsführer: René Schneider, Oliver Fieth, Dr. Wolfgang Trier

RE: Performance of Reassociation

[Ping-Ke Shih]

Jesuiter, Henry <henry.jesuiter@softing.com> wrote:
> 
> we are using the rtw88 driver (with kernel 5.4) together with a RTL8822ce chip (Firmware 9.9.15, WOW 9.9.4,
> H2C Version 15) on a PCI-Express card. We stumbled about long roaming durations and investigated a little
> into this issue, by using different hardware:
>     * Freescale i.MX6/ARMv7
>     * Intel Core i5-7300)
> 
> and different distributions:
>    * Yocto Hardknott/3.3.6, Kernel 5.4
>    * Buildroot, Kernel 6.6
>    * Ubuntu 20.04, Kernel 5.15
>    * Ubuntu 24.04, Kernel 6.8
> 
> and found roaming times from 450ms (Ubuntu 24.04, Intel) up to 900ms (Yocto 3.3.6, FreeScale).

Does it mean 450ms on kernel 5.15, but 900ms on kernel 5.4 for RTL8822ce?

> 
> Since we are used to much shorter roaming times (about 200ms even on the Freescale), 

What is the WiFi card you are used to measure this roaming time?


> 2. Start authentication to the new AP
>     [  561.557691] wlan0: authenticate with <new_ssid>
> 2.1 Reinitialize the hardware
>     [  561.577099] rtw_8822ce 0000:01:00.0:  rtw_power_on()
>     [  561.577109] rtw_8822ce 0000:01:00.0:  rtw_hci_setup()
>     [  561.577128] rtw_8822ce 0000:01:00.0:  rtw_mac_power_on()
>     [  561.577262] rtw_8822ce 0000:01:00.0:  rtw_mac_power_switch()
>     [  561.580561] rtw_8822ce 0000:01:00.0:  rtw_wait_firmware_completion()
>     [  561.584826] rtw_8822ce 0000:01:00.0:  rtw_download_firmware()
> => about 40ms here
>     [  561.614635] rtw_8822ce 0000:01:00.0: rtw_hci_setup()
>     [  561.614657] rtw_8822ce 0000:01:00.0: rtw_mac_init()
> => about 250ms here
>     [  561.876080] rtw_8822ce 0000:01:00.0: rtw_hci_start()
>     ...

I measure rtw_leave_ips() which is to power on wifi card, and the cost is
about 200ms in my x86 NB.

> 2.2 PHY calibration
>     [  561.877140] rtw_8822ce 0000:01:00.0: [RFK] WiFi / BT RFK handshake start!!
>     ...
>     [  561.896456] rtw_8822ce 0000:01:00.0: [RFK] WiFi / BT RFK handshake finish!!
>     [  561.936393] rtw_8822ce 0000:01:00.0: [DPK] s0 dpk start
>     [  562.056272] rtw_8822ce 0000:01:00.0: [DPK] s0 dpk finish
>     [  562.056280] rtw_8822ce 0000:01:00.0: [DPK] s1 dpk start
>     [  562.176275] rtw_8822ce 0000:01:00.0: [DPK] s1 dpk finish
> => about additional 300ms until here (that's the phy calibration stuff)

I measure rtw_chip_prepare_tx() which is to do phy calibration. The cost is
about 190ms on 2GHz and 5GHz channels. 

> 
> So, we are wondering if this long duration times for reassociation are hardware-related (maybe even someone
> uses the same chip and does not experience these delays)?
> 
> On the software side, is there some optimization potential known in the software/driver that is just not
> realized yet? We would be happy to realize/implement those potentials. To be more specific: We are looking
> for a way that the driver does not need to reinitialize/restart the hardware while roaming in order to reduce
> that part of the roaming delay specifically (and maybe skip/shorten the phy_calibration part). Any ideas
> are welcome. We are also happy to provide any additional logs that might be needed (please specify) and
> to answer any additional question you might have.

The costs I posted are hardware-related. Ignore IEEE80211_CONF_IDLE to avoid calling
rtw_enter_ips()/rtw_leave_ips(), saving 200ms to power on hardware. 

For PHY calibration, I think this is strictly necessary to get good performance.
One way may be deferred the calibration after getting connected, but 4 way and 
DHCP (right after getting connected) may not work well result from bad RF signal
quality.

AW: Performance of Reassociation

[Jesuiter, Henry]

Hello Ping-Ke Shih,
 
thank you very much for your answer. Please find my comments below.
 
Ping-Ke Shih <pkshih@realtek.com> wrote:
> Does it mean 450ms on kernel 5.15, but 900ms on kernel 5.4 for RTL8822ce?
[Jesuiter, Henry]
Actually it's the following:
  * about 450ms for Kernel 6.8 (Ubuntu 24.04), i5-7300U
  * about 600ms for Kernel 6.6 (Buildroot), COMi.MX 6
  * about 600ms for Kernel 5.15 (Ubuntu 20.04), i5-7300U
  * about 700ms - 900ms for Kernel 5.4 (Yocto 3.3.6), COMi.MX 6
 
For comparison:
  * roaming times of about 100ms on an ATH9k module (on an older HW revision), Kernel 5.4, Yocto 3.3.6, COMi.MX 6
 
> What is the WiFi card you are used to measure this roaming time?
[Jesuiter, Henry ]
We are using an Emwicon WMX6218 (WiFi 5, 802.11ac 2x2 MU-MIMO).
 
> I measure rtw_leave_ips() which is to power on wifi card, and the cost is about 200ms in my x86 NB.
[Jesuiter, Henry]
Thanks for the measurement. So it's quite the same as on our x86 NB.
 
> I measure rtw_chip_prepare_tx() which is to do phy calibration. The cost is about 190ms on 2GHz and 5GHz channels.
[Jesuiter, Henry]
This is the same function that is called on roaming, and we are experiencing similar results here. Thanks for the effort.
 
> The costs I posted are hardware-related. Ignore IEEE80211_CONF_IDLE to avoid calling rtw_enter_ips()/rtw_leave_ips(), saving 200ms to power on hardware.
[Jesuiter, Henry]
We thought about that too, but we see no easy way to avoid the power down beforehand, since we are not able to distinguish a power down due to roaming from a power down due to other reasons. So - since the chip is powered down - we can't  just skip the power up here. Any ideas are welcome 😉.
 
> For PHY calibration, I think this is strictly necessary to get good performance.
> One way may be deferred the calibration after getting connected, but 4 way and DHCP (right after getting connected) may not work well result from bad RF signal quality.
[Jesuiter, Henry]
Thanks for the consideration, we have similar concerns - especially regarding the EAPOL stuff.
 
One more question. Is there a way to use 802.11r (fast roaming) with the mainline driver?
 
Kind regards
Henry Jesuiter
 

RE: Performance of Reassociation

[Ping-Ke Shih]

Jesuiter, Henry <henry.jesuiter@softing.com> wrote:
> Ping-Ke Shih <pkshih@realtek.com> wrote:
> > Does it mean 450ms on kernel 5.15, but 900ms on kernel 5.4 for RTL8822ce?
> [Jesuiter, Henry]
> Actually it's the following:
>   * about 450ms for Kernel 6.8 (Ubuntu 24.04), i5-7300U
>   * about 600ms for Kernel 6.6 (Buildroot), COMi.MX 6
>   * about 600ms for Kernel 5.15 (Ubuntu 20.04), i5-7300U
>   * about 700ms - 900ms for Kernel 5.4 (Yocto 3.3.6), COMi.MX 6

No idea why they are different since RTL8822CE doesn't have much change for a long time.

> > I measure rtw_chip_prepare_tx() which is to do phy calibration. The cost is about 190ms on 2GHz and 5GHz
> channels.
> [Jesuiter, Henry]
> This is the same function that is called on roaming, and we are experiencing similar results here. Thanks
> for the effort.
> 
> > The costs I posted are hardware-related. Ignore IEEE80211_CONF_IDLE to avoid calling
> rtw_enter_ips()/rtw_leave_ips(), saving 200ms to power on hardware.
> [Jesuiter, Henry]
> We thought about that too, but we see no easy way to avoid the power down beforehand, since we are not able
> to distinguish a power down due to roaming from a power down due to other reasons. So - since the chip is
> powered down - we can't  just skip the power up here. Any ideas are welcome 😉.

Maybe you can start a delyed_work with 1 second delay when entering to
rtw_enter_ips() but not actually enter. For normal use case, after 1
second, call rtw_enter_ips() to power down. For roaming case, it must
ask to power on by rtw_leave_ips() immediately, so cancel the delayed_work
and no actually power off/on in this case.

1 second is an example, maybe need more time. Fine tune the value by
your experiment result.

> 
> One more question. Is there a way to use 802.11r (fast roaming) with the mainline driver?

As I know, 802.11r is mainly implemented in wpa_supplicant. I don't remember 
drivers need to implement special handles for that. 

AW: Performance of Reassociation

[Jesuiter, Henry]

Ping-Ke Shih <pkshih@realtek.com> wrote:

> Maybe you can start a delyed_work with 1 second delay when entering to
> rtw_enter_ips() but not actually enter. For normal use case, after 1
> second, call rtw_enter_ips() to power down. For roaming case, it must
> ask to power on by rtw_leave_ips() immediately, so cancel the delayed_work
> and no actually power off/on in this case.

This is a very interesting idea. We will try this for sure! Thank you very much. 

>
> One more question. Is there a way to use 802.11r (fast roaming) with the mainline driver?

As I know, 802.11r is mainly implemented in wpa_supplicant. I don't remember
drivers need to implement special handles for that.

Ah, ok. Thanks for that advice. 

Kind regards
Henry Jesuiter