* help troubleshooting low throughput @ 2014-05-22 9:39 Tim Harvey 2014-05-22 9:46 ` Tim Harvey 0 siblings, 1 reply; 7+ messages in thread From: Tim Harvey @ 2014-05-22 9:39 UTC (permalink / raw) To: ath10k@lists.infradead.org Greetings, I could use some help troubleshooting a low throughput issue. I'm currently using the following: - UNEX DAXA-O1 11ac/n/a 3x3 MIMO qca988x hw2.0 http://www.unex.com.tw/product/daxa-o1 - 80MHz channel w/o local interference - ath10k git 0dbbb028a7c461777bf4a0d53780e539e6f40e14 (May 16) - up-to-date git of hostapd/wpa_supplicant/iw - fw 10.1.467.2-1 api 2 htt 2.1 - infrastructure mode using http://wireless.kernel.org/en/users/Drivers/ath10k/configuration#Full_hostapd_configuration I'm using iperf for throughput tests and getting no more than 220mbps best case, typically more like 120mbps. The rx bitrate bounces around MCS 5 to 8 and shows 3 spatial streams so I would be expecting a much higher throughput. The cards are in boards with a quad-core ARM 1GHz Cortex-A9 CPU and there is no indication the system is bottle-necked. There are no other kernel modules loaded other thank ath10k_pci/ath10k_core/ath and debugging is disabled. Using tcpdump/wireshark to inspect the radiotap headers I only see packets with 'Antenna: 0' - Does this field indicate what transmitter the pkt was received on and indicate I'm only receiving from a single transmitter? I've noticed 'iw phy0 info' shows 'Available Antennas: TX 0 RX 0' and any attempt to set the antenna mask with 'iw phy set antenna' fails with an 'Operation not supported (-95)' - does this indicate an issue with ath10k, iw, or perhaps the card's EEPROM? I've also noticed that 'iw wlan0 station dump' statistics show in the rx bitrate stat that sometimes SGI is flagged and sometimes its not - should I expect this to change like this? I was under the impression that was a static configuration. Thanks, Tim _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: help troubleshooting low throughput 2014-05-22 9:39 help troubleshooting low throughput Tim Harvey @ 2014-05-22 9:46 ` Tim Harvey 2014-05-22 10:08 ` Michal Kazior 0 siblings, 1 reply; 7+ messages in thread From: Tim Harvey @ 2014-05-22 9:46 UTC (permalink / raw) To: ath10k@lists.infradead.org On Thu, May 22, 2014 at 2:39 AM, Tim Harvey <tharvey@gateworks.com> wrote: > Greetings, > > I could use some help troubleshooting a low throughput issue. I'm > currently using the following: > - UNEX DAXA-O1 11ac/n/a 3x3 MIMO qca988x hw2.0 > http://www.unex.com.tw/product/daxa-o1 > - 80MHz channel w/o local interference > - ath10k git 0dbbb028a7c461777bf4a0d53780e539e6f40e14 (May 16) > - up-to-date git of hostapd/wpa_supplicant/iw > - fw 10.1.467.2-1 api 2 htt 2.1 > - infrastructure mode using > http://wireless.kernel.org/en/users/Drivers/ath10k/configuration#Full_hostapd_configuration > > I'm using iperf for throughput tests and getting no more than 220mbps > best case, typically more like 120mbps. The rx bitrate bounces around > MCS 5 to 8 and shows 3 spatial streams so I would be expecting a much > higher throughput. The cards are in boards with a quad-core ARM 1GHz > Cortex-A9 CPU and there is no indication the system is bottle-necked. > There are no other kernel modules loaded other thank > ath10k_pci/ath10k_core/ath and debugging is disabled. > > Using tcpdump/wireshark to inspect the radiotap headers I only see > packets with 'Antenna: 0' - Does this field indicate what transmitter > the pkt was received on and indicate I'm only receiving from a single > transmitter? > > I've noticed 'iw phy0 info' shows 'Available Antennas: TX 0 RX 0' and > any attempt to set the antenna mask with 'iw phy set antenna' fails > with an 'Operation not supported (-95)' - does this indicate an issue > with ath10k, iw, or perhaps the card's EEPROM? > > I've also noticed that 'iw wlan0 station dump' statistics show in the > rx bitrate stat that sometimes SGI is flagged and sometimes its not - > should I expect this to change like this? I was under the impression > that was a static configuration. > > Thanks, > > Tim I neglected to mention that on whichever end is running the iperf server (receiving) I do see periodic 'failed to pop...' warnings, a pair of them a couple times a minute. I'm not quite clear if this is a hardware issue or something else: [ 3763.131280] ath10k: failed to pop amsdu from htt rx ring -22 [ 3769.081093] ath10k: failed to pop chained msdus, dropping [ 3769.086575] ath10k: failed to pop amsdu from htt rx ring -22 [ 3781.092383] ath10k: failed to pop chained msdus, dropping [ 3781.097869] ath10k: failed to pop amsdu from htt rx ring -22 [ 3784.367225] ath10k: failed to pop chained msdus, dropping [ 3784.372735] ath10k: failed to pop amsdu from htt rx ring -22 [ 3809.501484] ath10k: failed to pop chained msdus, dropping [ 3809.506993] ath10k: failed to pop amsdu from htt rx ring -22 [ 3821.723404] ath10k: failed to pop chained msdus, dropping [ 3821.728914] ath10k: failed to pop amsdu from htt rx ring -22 [ 3832.067136] ath10k: failed to pop chained msdus, dropping [ 3832.072690] ath10k: failed to pop amsdu from htt rx ring -22 [ 3833.632859] ath10k: failed to pop chained msdus, dropping [ 3833.638341] ath10k: failed to pop amsdu from htt rx ring -22 [ 3837.940414] ath10k: failed to pop chained msdus, dropping [ 3837.945982] ath10k: failed to pop amsdu from htt rx ring -22 [ 3844.100514] ath10k: failed to pop chained msdus, dropping Tim _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: help troubleshooting low throughput 2014-05-22 9:46 ` Tim Harvey @ 2014-05-22 10:08 ` Michal Kazior 2014-05-22 18:37 ` Tim Harvey 0 siblings, 1 reply; 7+ messages in thread From: Michal Kazior @ 2014-05-22 10:08 UTC (permalink / raw) To: Tim Harvey; +Cc: ath10k@lists.infradead.org On 22 May 2014 11:46, Tim Harvey <tharvey@gateworks.com> wrote: > On Thu, May 22, 2014 at 2:39 AM, Tim Harvey <tharvey@gateworks.com> wrote: >> Greetings, >> >> I could use some help troubleshooting a low throughput issue. I'm >> currently using the following: >> - UNEX DAXA-O1 11ac/n/a 3x3 MIMO qca988x hw2.0 >> http://www.unex.com.tw/product/daxa-o1 >> - 80MHz channel w/o local interference >> - ath10k git 0dbbb028a7c461777bf4a0d53780e539e6f40e14 (May 16) >> - up-to-date git of hostapd/wpa_supplicant/iw >> - fw 10.1.467.2-1 api 2 htt 2.1 >> - infrastructure mode using >> http://wireless.kernel.org/en/users/Drivers/ath10k/configuration#Full_hostapd_configuration Did you just copy&paste the example config file (and updated interface=) or did you do something extra? Typically the most important option is vht_capab MAX-A-MPDU-LEN-EXPx (x=0 or 7 depending on hostapd version). Without this aggregation is crippled and it yields pretty poor throughput. What about the client? What is it? Is it connected in VHT80 mode? >> I'm using iperf for throughput tests and getting no more than 220mbps >> best case, typically more like 120mbps. The rx bitrate bounces around >> MCS 5 to 8 and shows 3 spatial streams so I would be expecting a much >> higher throughput. The cards are in boards with a quad-core ARM 1GHz >> Cortex-A9 CPU and there is no indication the system is bottle-necked. >> There are no other kernel modules loaded other thank >> ath10k_pci/ath10k_core/ath and debugging is disabled. Currently ath10k doesn't really scale much with number of CPUs. There are basically two tasklets that could split the work just a little bit, but this requires interrupt spreading. From what I know some ARM chips can't do that so ath10k ends up using only single CPU all the time. 1GHz of an A9 should still be enough to get you 500mbps+ though. Did you run TCP and/or UDP tests? What direction did you test (station->ap / ap->station)? >> Using tcpdump/wireshark to inspect the radiotap headers I only see >> packets with 'Antenna: 0' - Does this field indicate what transmitter >> the pkt was received on and indicate I'm only receiving from a single >> transmitter? Antenna callbacks aren't implemented in master branch yet. You can check ath-next-test for that or cherry picks Ben's patches. You can verify max number of spatial streams with iw by looking at `HT TX/RX MCS rate indexes supported:` and/or VHT TX/RX MCS sets. >> I've noticed 'iw phy0 info' shows 'Available Antennas: TX 0 RX 0' and >> any attempt to set the antenna mask with 'iw phy set antenna' fails >> with an 'Operation not supported (-95)' - does this indicate an issue >> with ath10k, iw, or perhaps the card's EEPROM? It would be more helpful if you could provide `iw list` output instead next time. >> I've also noticed that 'iw wlan0 station dump' statistics show in the >> rx bitrate stat that sometimes SGI is flagged and sometimes its not - >> should I expect this to change like this? I was under the impression >> that was a static configuration. I think this ath10k's hw rate control is free to pick SGI/LGI (assuming target station supports SGI at all). [...] > I neglected to mention that on whichever end is running the iperf > server (receiving) I do see periodic 'failed to pop...' warnings, a > pair of them a couple times a minute. I'm not quite clear if this is a > hardware issue or something else: > > [ 3763.131280] ath10k: failed to pop amsdu from htt rx ring -22 > [ 3769.081093] ath10k: failed to pop chained msdus, dropping > [ 3769.086575] ath10k: failed to pop amsdu from htt rx ring -22 > [ 3781.092383] ath10k: failed to pop chained msdus, dropping > [ 3781.097869] ath10k: failed to pop amsdu from htt rx ring -22 > [ 3784.367225] ath10k: failed to pop chained msdus, dropping > [ 3784.372735] ath10k: failed to pop amsdu from htt rx ring -22 > [ 3809.501484] ath10k: failed to pop chained msdus, dropping > [ 3809.506993] ath10k: failed to pop amsdu from htt rx ring -22 > [ 3821.723404] ath10k: failed to pop chained msdus, dropping > [ 3821.728914] ath10k: failed to pop amsdu from htt rx ring -22 > [ 3832.067136] ath10k: failed to pop chained msdus, dropping > [ 3832.072690] ath10k: failed to pop amsdu from htt rx ring -22 > [ 3833.632859] ath10k: failed to pop chained msdus, dropping > [ 3833.638341] ath10k: failed to pop amsdu from htt rx ring -22 > [ 3837.940414] ath10k: failed to pop chained msdus, dropping > [ 3837.945982] ath10k: failed to pop amsdu from htt rx ring -22 > [ 3844.100514] ath10k: failed to pop chained msdus, dropping You don't have to worry about these messages. This was recently introduced by one of my patches to further analyze a bug that Avery was seeing (kernel panic due to skb_push). I plan on making a patch to clean this up. Anyway, thanks for letting know :-) Michał _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: help troubleshooting low throughput 2014-05-22 10:08 ` Michal Kazior @ 2014-05-22 18:37 ` Tim Harvey 2014-05-23 7:42 ` Michal Kazior 0 siblings, 1 reply; 7+ messages in thread From: Tim Harvey @ 2014-05-22 18:37 UTC (permalink / raw) To: Michal Kazior; +Cc: ath10k@lists.infradead.org On Thu, May 22, 2014 at 3:08 AM, Michal Kazior <michal.kazior@tieto.com> wrote: > On 22 May 2014 11:46, Tim Harvey <tharvey@gateworks.com> wrote: >> On Thu, May 22, 2014 at 2:39 AM, Tim Harvey <tharvey@gateworks.com> wrote: >>> Greetings, >>> >>> I could use some help troubleshooting a low throughput issue. I'm >>> currently using the following: >>> - UNEX DAXA-O1 11ac/n/a 3x3 MIMO qca988x hw2.0 >>> http://www.unex.com.tw/product/daxa-o1 >>> - 80MHz channel w/o local interference >>> - ath10k git 0dbbb028a7c461777bf4a0d53780e539e6f40e14 (May 16) >>> - up-to-date git of hostapd/wpa_supplicant/iw >>> - fw 10.1.467.2-1 api 2 htt 2.1 >>> - infrastructure mode using >>> http://wireless.kernel.org/en/users/Drivers/ath10k/configuration#Full_hostapd_configuration > > Did you just copy&paste the example config file (and updated > interface=) or did you do something extra? Hi Michal, I disabled bridge mode, DFS, wpa, wps and added 'vht_oper_centr_freq_seg0_idx=42' which appears to be something new that is required or hostapd bails out: ### hostapd configuration file ctrl_interface=/var/run/hostapd interface=wlan0 driver=nl80211 #bridge=br-lan ### IEEE 802.11 ssid=timtest hw_mode=a channel=36 max_num_sta=128 auth_algs=1 disassoc_low_ack=1 ### DFS #ieee80211h=1 #ieee80211d=1 #country_code=FR ieee80211h=0 ieee80211d=0 ### IEEE 802.11n ieee80211n=1 ht_capab=[HT40+][LDPC][SHORT-GI-20][SHORT-GI-40][TX-STBC][RX-STBC1][DSSS_CCK-40] ### IEEE 802.11ac ieee80211ac=1 vht_oper_chwidth=1 vht_oper_centr_freq_seg0_idx=42 vht_capab=[MAX-MPDU-11454][RXLDPC][SHORT-GI-80][TX-STBC-2BY1][RX-STBC-1][MAX-A-MPDU-LEN-EXP7][RX-ANTENNA-PATTERN][TX-ANTENNA-PATTERN] ### WPA/IEEE 802.11i #wpa=2 #wpa_key_mgmt=WPA-PSK #wpa_passphrase=12345678 #wpa_pairwise=CCMP wpa=0 ### Wi-Fi Protected Setup (WPS) #wps_state=2 #ap_setup_locked=0 #wps_pin_requests=/var/run/hostapd_wps_pin_requests #device_name=QCA Access Point #manufacturer=Qualcomm Atheros #device_type=6-0050F204-1 #config_methods=virtual_push_button physical_push_button label keypad virtual_display #pbc_in_m1=1 #ap_pin=12345670 #upnp_iface=br-lan #eap_server=1 ### hostapd event logger configuration logger_syslog=127 logger_syslog_level=2 logger_stdout=127 logger_stdout_level=2 ### WMM wmm_enabled=1 uapsd_advertisement_enabled=1 wmm_ac_bk_cwmin=4 wmm_ac_bk_cwmax=10 wmm_ac_bk_aifs=7 wmm_ac_bk_txop_limit=0 wmm_ac_bk_acm=0 wmm_ac_be_aifs=3 wmm_ac_be_cwmin=4 wmm_ac_be_cwmax=10 wmm_ac_be_txop_limit=0 wmm_ac_be_acm=0 wmm_ac_vi_aifs=2 wmm_ac_vi_cwmin=3 wmm_ac_vi_cwmax=4 wmm_ac_vi_txop_limit=94 wmm_ac_vi_acm=0 wmm_ac_vo_aifs=2 wmm_ac_vo_cwmin=2 wmm_ac_vo_cwmax=3 wmm_ac_vo_txop_limit=47 wmm_ac_vo_acm=0 ### TX queue parameters tx_queue_data3_aifs=7 tx_queue_data3_cwmin=15 tx_queue_data3_cwmax=1023 tx_queue_data3_burst=0 tx_queue_data2_aifs=3 tx_queue_data2_cwmin=15 tx_queue_data2_cwmax=63 tx_queue_data2_burst=0 tx_queue_data1_aifs=1 tx_queue_data1_cwmin=7 tx_queue_data1_cwmax=15 tx_queue_data1_burst=3.0 tx_queue_data0_aifs=1 tx_queue_data0_cwmin=3 tx_queue_data0_cwmax=7 tx_queue_data0_burst=1.5 > > Typically the most important option is vht_capab MAX-A-MPDU-LEN-EXPx > (x=0 or 7 depending on hostapd version). Without this aggregation is > crippled and it yields pretty poor throughput. good to know - got that. hostapd is git 4e0a94b7dc76db58cddbbcfe0be0bfef547f6dd0 Fri May 16 built with following config: CONFIG_DRIVER_HOSTAP=y CONFIG_DRIVER_WIRED=y CONFIG_DRIVER_NL80211=y CONFIG_LIBNL32=y CONFIG_IAPP=y CONFIG_RSN_PREAUTH=y CONFIG_PEERKEY=y CONFIG_IEEE80211W=y CONFIG_EAP=y CONFIG_EAP_MD5=y CONFIG_EAP_TLS=y CONFIG_EAP_MSCHAPV2=y CONFIG_EAP_PEAP=y CONFIG_EAP_GTC=y CONFIG_EAP_TTLS=y CONFIG_WPS=y CONFIG_PKCS12=y CONFIG_RADIUS_SERVER=y CONFIG_IPV6=y CONFIG_DRIVER_RADIUS_ACL=y CONFIG_IEEE80211N=y CONFIG_IEEE80211AC=y CONFIG_ACS=y root@ap-99:~# hostapd -v hostapd v2.2-devel User space daemon for IEEE 802.11 AP management, IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator Copyright (c) 2002-2014, Jouni Malinen <j@w1.fi> and contributors > > What about the client? What is it? Is it connected in VHT80 mode? identical to the ap, but running wpa_supplicant from same git rev as hostapd with: network={ scan_ssid=1 ssid="timtest" key_mgmt=NONE } Its showing 80Mhz MCS 5 (between 5 and 8) root@sta-97:~# iw wlan0 station dump Station 60:02:b4:9d:99:7f (on wlan0) inactive time: 590 ms rx bytes: 160004 rx packets: 1824 tx bytes: 9832 tx packets: 87 tx retries: 0 tx failed: 0 signal: -53 dBm signal avg: -52 dBm tx bitrate: 6.0 MBit/s rx bitrate: 975.0 MBit/s VHT-MCS 7 80MHz short GI VHT-NSS 3 authorized: yes authenticated: yes preamble: long WMM/WME: yes MFP: no TDLS peer: no ap is showing 80MHz width between MCS 5 and MCS 8: root@ap-99:~# iw wlan0 station dump Station 60:02:b4:9d:99:62 (on wlan0) inactive time: 0 ms rx bytes: 275591916 rx packets: 182178 tx bytes: 4394890 tx packets: 50807 tx retries: 0 tx failed: 0 signal: -60 dBm signal avg: -60 dBm tx bitrate: 6.0 MBit/s rx bitrate: 702.0 MBit/s VHT-MCS 5 80MHz VHT-NSS 3 authorized: yes authenticated: yes preamble: long WMM/WME: yes MFP: no TDLS peer: no > > >>> I'm using iperf for throughput tests and getting no more than 220mbps >>> best case, typically more like 120mbps. The rx bitrate bounces around >>> MCS 5 to 8 and shows 3 spatial streams so I would be expecting a much >>> higher throughput. The cards are in boards with a quad-core ARM 1GHz >>> Cortex-A9 CPU and there is no indication the system is bottle-necked. >>> There are no other kernel modules loaded other thank >>> ath10k_pci/ath10k_core/ath and debugging is disabled. > > Currently ath10k doesn't really scale much with number of CPUs. There > are basically two tasklets that could split the work just a little > bit, but this requires interrupt spreading. From what I know some ARM > chips can't do that so ath10k ends up using only single CPU all the > time. 1GHz of an A9 should still be enough to get you 500mbps+ though. interesting. I see [ath10k_wq] in ps, what is the other task? ath10k will just register 1 interrupt for PCI, how would you spread that if only 1 ath10k device is in the system? I would agree that a 1GHz CortexA9 should be able to do well. The top application shows that only CPU0 is being utilized and never more than 25% or so (softirq mostly) and mostly idle. So I don't think this is any sort of CPU bottleneck. > > Did you run TCP and/or UDP tests? What direction did you test > (station->ap / ap->station)? both - the best throughput I see is appx 220mbps TCP and 260mbps UDP and this is consistent in both directions. > > >>> Using tcpdump/wireshark to inspect the radiotap headers I only see >>> packets with 'Antenna: 0' - Does this field indicate what transmitter >>> the pkt was received on and indicate I'm only receiving from a single >>> transmitter? > > Antenna callbacks aren't implemented in master branch yet. You can > check ath-next-test for that or cherry picks Ben's patches. ok - I will do that. Does this field indicate which antenna the frame was sent out or received from? > > You can verify max number of spatial streams with iw by looking at `HT > TX/RX MCS rate indexes supported:` and/or VHT TX/RX MCS sets. > These look fine (see below) but is there a way to actually prove (ie by radiotap) that frames come in/out of multiple antennas? root@ap-99:~# iw phy0 info Wiphy phy0 max # scan SSIDs: 16 max scan IEs length: 199 bytes Retry short limit: 7 Retry long limit: 4 Coverage class: 0 (up to 0m) Device supports AP-side u-APSD. Supported Ciphers: * WEP40 (00-0f-ac:1) * WEP104 (00-0f-ac:5) * TKIP (00-0f-ac:2) * CCMP (00-0f-ac:4) * CMAC (00-0f-ac:6) Available Antennas: TX 0 RX 0 Supported interface modes: * IBSS * managed * AP * AP/VLAN * monitor * P2P-client * P2P-GO Band 2: Capabilities: 0x19e3 RX LDPC HT20/HT40 Static SM Power Save RX HT20 SGI RX HT40 SGI TX STBC RX STBC 1-stream Max AMSDU length: 7935 bytes DSSS/CCK HT40 Maximum RX AMPDU length 65535 bytes (exponent: 0x003) Minimum RX AMPDU time spacing: 8 usec (0x06) HT TX/RX MCS rate indexes supported: 0-23 VHT Capabilities (0x338001b2): Max MPDU length: 11454 Supported Channel Width: neither 160 nor 80+80 RX LDPC short GI (80 MHz) TX STBC RX antenna pattern consistency TX antenna pattern consistency VHT RX MCS set: 1 streams: MCS 0-9 2 streams: MCS 0-9 3 streams: MCS 0-9 4 streams: not supported 5 streams: not supported 6 streams: not supported 7 streams: not supported 8 streams: not supported VHT RX highest supported: 0 Mbps VHT TX MCS set: 1 streams: MCS 0-9 2 streams: MCS 0-9 3 streams: MCS 0-9 4 streams: not supported 5 streams: not supported 6 streams: not supported 7 streams: not supported 8 streams: not supported VHT TX highest supported: 0 Mbps Bitrates (non-HT): * 6.0 Mbps * 9.0 Mbps * 12.0 Mbps * 18.0 Mbps * 24.0 Mbps * 36.0 Mbps * 48.0 Mbps * 54.0 Mbps Frequencies: Frequencies: * 5180 MHz [36] (17.0 dBm) * 5200 MHz [40] (17.0 dBm) * 5220 MHz [44] (17.0 dBm) * 5240 MHz [48] (17.0 dBm) * 5260 MHz [52] (23.0 dBm) (no IR, radar detection) DFS state: usable (for 4294712 sec) * 5280 MHz [56] (23.0 dBm) (no IR, radar detection) DFS state: usable (for 4294712 sec) * 5300 MHz [60] (23.0 dBm) (no IR, radar detection) DFS state: usable (for 4294712 sec) * 5320 MHz [64] (23.0 dBm) (no IR, radar detection) DFS state: usable (for 4294712 sec) * 5500 MHz [100] (disabled) * 5520 MHz [104] (disabled) * 5540 MHz [108] (disabled) * 5560 MHz [112] (disabled) * 5580 MHz [116] (disabled) * 5600 MHz [120] (disabled) * 5620 MHz [124] (disabled) * 5640 MHz [128] (disabled) * 5660 MHz [132] (disabled) * 5680 MHz [136] (disabled) * 5700 MHz [140] (disabled) * 5745 MHz [149] (30.0 dBm) * 5765 MHz [153] (30.0 dBm) * 5785 MHz [157] (30.0 dBm) * 5805 MHz [161] (30.0 dBm) * 5825 MHz [165] (30.0 dBm) Supported commands: * new_interface * set_interface * new_key * start_ap * new_station * new_mpath * set_mesh_config * set_bss * authenticate * associate * deauthenticate * disassociate * join_ibss * join_mesh * remain_on_channel * set_tx_bitrate_mask * frame * frame_wait_cancel * set_wiphy_netns * set_channel * set_wds_peer * probe_client * set_noack_map * register_beacons * start_p2p_device * set_mcast_rate * Unknown command (104) * connect * disconnect Supported TX frame types: * IBSS: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0 * managed: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0 * AP: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0 * AP/VLAN: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0 * mesh point: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0 * P2P-client: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0 * P2P-GO: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0 * P2P-device: 0x00 0x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xa0 0xb0 0xc0 0xd0 0xe0 0xf0 Supported RX frame types: * IBSS: 0x40 0xb0 0xc0 0xd0 * managed: 0x40 0xd0 * AP: 0x00 0x20 0x40 0xa0 0xb0 0xc0 0xd0 * AP/VLAN: 0x00 0x20 0x40 0xa0 0xb0 0xc0 0xd0 * mesh point: 0xb0 0xc0 0xd0 * P2P-client: 0x40 0xd0 * P2P-GO: 0x00 0x20 0x40 0xa0 0xb0 0xc0 0xd0 * P2P-device: 0x40 0xd0 software interface modes (can always be added): * AP/VLAN * monitor valid interface combinations: * #{ AP } <= 8, total <= 8, #channels <= 1, STA/AP BI must match, radar detect widths: { 20 MHz (no HT), 20 MHz, 40 MHz, 80 MHz } HT Capability overrides: * MCS: ff ff ff ff ff ff ff ff ff ff * maximum A-MSDU length * supported channel width * short GI for 40 MHz * max A-MPDU length exponent * min MPDU start spacing Device supports TX status socket option. Device supports HT-IBSS. Device supports scan flush. > >>> I've noticed 'iw phy0 info' shows 'Available Antennas: TX 0 RX 0' and >>> any attempt to set the antenna mask with 'iw phy set antenna' fails >>> with an 'Operation not supported (-95)' - does this indicate an issue >>> with ath10k, iw, or perhaps the card's EEPROM? > > It would be more helpful if you could provide `iw list` output instead > next time. right - I've put it above this time. > > >>> I've also noticed that 'iw wlan0 station dump' statistics show in the >>> rx bitrate stat that sometimes SGI is flagged and sometimes its not - >>> should I expect this to change like this? I was under the impression >>> that was a static configuration. > > I think this ath10k's hw rate control is free to pick SGI/LGI > (assuming target station supports SGI at all). I always assumed a user would want to force one way or another - does the rate control engine try to optimize by using short guard-band interval (if supported) and then back-off to long if it detects lots of retries? > > > [...] > >> I neglected to mention that on whichever end is running the iperf >> server (receiving) I do see periodic 'failed to pop...' warnings, a >> pair of them a couple times a minute. I'm not quite clear if this is a >> hardware issue or something else: >> >> [ 3763.131280] ath10k: failed to pop amsdu from htt rx ring -22 >> [ 3769.081093] ath10k: failed to pop chained msdus, dropping >> [ 3769.086575] ath10k: failed to pop amsdu from htt rx ring -22 >> [ 3781.092383] ath10k: failed to pop chained msdus, dropping >> [ 3781.097869] ath10k: failed to pop amsdu from htt rx ring -22 >> [ 3784.367225] ath10k: failed to pop chained msdus, dropping >> [ 3784.372735] ath10k: failed to pop amsdu from htt rx ring -22 >> [ 3809.501484] ath10k: failed to pop chained msdus, dropping >> [ 3809.506993] ath10k: failed to pop amsdu from htt rx ring -22 >> [ 3821.723404] ath10k: failed to pop chained msdus, dropping >> [ 3821.728914] ath10k: failed to pop amsdu from htt rx ring -22 >> [ 3832.067136] ath10k: failed to pop chained msdus, dropping >> [ 3832.072690] ath10k: failed to pop amsdu from htt rx ring -22 >> [ 3833.632859] ath10k: failed to pop chained msdus, dropping >> [ 3833.638341] ath10k: failed to pop amsdu from htt rx ring -22 >> [ 3837.940414] ath10k: failed to pop chained msdus, dropping >> [ 3837.945982] ath10k: failed to pop amsdu from htt rx ring -22 >> [ 3844.100514] ath10k: failed to pop chained msdus, dropping > > You don't have to worry about these messages. > > This was recently introduced by one of my patches to further analyze a > bug that Avery was seeing (kernel panic due to skb_push). I plan on > making a patch to clean this up. > > Anyway, thanks for letting know :-) ok - I saw the patch but from the commit message I through it was indicating these were failures. Thanks for the explanation. > > > Michał Thanks for helping! Tim _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: help troubleshooting low throughput 2014-05-22 18:37 ` Tim Harvey @ 2014-05-23 7:42 ` Michal Kazior 2014-05-24 5:32 ` Tim Harvey 0 siblings, 1 reply; 7+ messages in thread From: Michal Kazior @ 2014-05-23 7:42 UTC (permalink / raw) To: Tim Harvey; +Cc: ath10k@lists.infradead.org On 22 May 2014 20:37, Tim Harvey <tharvey@gateworks.com> wrote: > On Thu, May 22, 2014 at 3:08 AM, Michal Kazior <michal.kazior@tieto.com> wrote: >> On 22 May 2014 11:46, Tim Harvey <tharvey@gateworks.com> wrote: >>> On Thu, May 22, 2014 at 2:39 AM, Tim Harvey <tharvey@gateworks.com> wrote: >>>> Greetings, >>>> >>>> I could use some help troubleshooting a low throughput issue. I'm >>>> currently using the following: >>>> - UNEX DAXA-O1 11ac/n/a 3x3 MIMO qca988x hw2.0 >>>> http://www.unex.com.tw/product/daxa-o1 >>>> - 80MHz channel w/o local interference >>>> - ath10k git 0dbbb028a7c461777bf4a0d53780e539e6f40e14 (May 16) >>>> - up-to-date git of hostapd/wpa_supplicant/iw >>>> - fw 10.1.467.2-1 api 2 htt 2.1 >>>> - infrastructure mode using >>>> http://wireless.kernel.org/en/users/Drivers/ath10k/configuration#Full_hostapd_configuration >> >> Did you just copy&paste the example config file (and updated >> interface=) or did you do something extra? > > Hi Michal, > > I disabled bridge mode, DFS, wpa, wps and added > 'vht_oper_centr_freq_seg0_idx=42' which appears to be something new > that is required or hostapd bails out: VHT requires a generic center frequency (or rather channel number in hostapd) to be provided. Since you have channel=36 then the center frequency for 80MHz bandwidth is 36+6 = 42. > ### hostapd configuration file [..] I'd try simply: > ht_capab=[HT40+][SHORT-GI-20][SHORT-GI-40] > vht_capab=[MAX-MPDU-11454][SHORT-GI-80][MAX-A-MPDU-LEN-EXP7] Otherwise it looks fine to me. [...] > Its showing 80Mhz MCS 5 (between 5 and 8) > > root@sta-97:~# iw wlan0 station dump > Station 60:02:b4:9d:99:7f (on wlan0) > inactive time: 590 ms > rx bytes: 160004 > rx packets: 1824 > tx bytes: 9832 > tx packets: 87 > tx retries: 0 > tx failed: 0 > signal: -53 dBm > signal avg: -52 dBm > tx bitrate: 6.0 MBit/s > rx bitrate: 975.0 MBit/s VHT-MCS 7 80MHz short GI VHT-NSS 3 > authorized: yes > authenticated: yes > preamble: long > WMM/WME: yes > MFP: no > TDLS peer: no > > ap is showing 80MHz width between MCS 5 and MCS 8: > > root@ap-99:~# iw wlan0 station dump > Station 60:02:b4:9d:99:62 (on wlan0) > inactive time: 0 ms > rx bytes: 275591916 > rx packets: 182178 > tx bytes: 4394890 > tx packets: 50807 > tx retries: 0 > tx failed: 0 > signal: -60 dBm > signal avg: -60 dBm > tx bitrate: 6.0 MBit/s > rx bitrate: 702.0 MBit/s VHT-MCS 5 80MHz VHT-NSS 3 > authorized: yes > authenticated: yes > preamble: long > WMM/WME: yes > MFP: no > TDLS peer: no This looks good. So rate control is doing rather fine. 3 spatial streams (VHT-NSS 3) are at work. >>>> I'm using iperf for throughput tests and getting no more than 220mbps >>>> best case, typically more like 120mbps. The rx bitrate bounces around >>>> MCS 5 to 8 and shows 3 spatial streams so I would be expecting a much >>>> higher throughput. The cards are in boards with a quad-core ARM 1GHz >>>> Cortex-A9 CPU and there is no indication the system is bottle-necked. >>>> There are no other kernel modules loaded other thank >>>> ath10k_pci/ath10k_core/ath and debugging is disabled. >> >> Currently ath10k doesn't really scale much with number of CPUs. There >> are basically two tasklets that could split the work just a little >> bit, but this requires interrupt spreading. From what I know some ARM >> chips can't do that so ath10k ends up using only single CPU all the >> time. 1GHz of an A9 should still be enough to get you 500mbps+ though. > > interesting. I see [ath10k_wq] in ps, what is the other task? ath10k > will just register 1 interrupt for PCI, how would you spread that if > only 1 ath10k device is in the system? ath10k_wq is the workqueue. It is not related to tasklets at all. Even if you have a single interrupt your controller may spread interrupts across sockets/cores/threads. So one time device issues an interrupt CPU0 gets interrupted and another time CPU1 gets interrupted. > I would agree that a 1GHz CortexA9 should be able to do well. The top > application shows that only CPU0 is being utilized and never more than > 25% or so (softirq mostly) and mostly idle. So I don't think this is > any sort of CPU bottleneck. The 25% sound fishy considering you have quad-core CPU. I'm not really sure if top (or top you use for that matter) reports percent wrt to a single CPU or globally. I would certainly investigate this. I recall vmstat sums everything up, i.e. if it says "25% sys" then it means "25% of your entrie CPU set is doing sys, regardless which core it is". >> Did you run TCP and/or UDP tests? What direction did you test >> (station->ap / ap->station)? > > both - the best throughput I see is appx 220mbps TCP and 260mbps UDP > and this is consistent in both directions. Did you try using the -P switch to send parallel streams? E.g. -u -b 100M -P5 for UDP? Also, now that I think about you don't have a bridge. This means your AP system has to perform a lot more packet mangling which I guess can be pretty taxing for the A9. >>>> Using tcpdump/wireshark to inspect the radiotap headers I only see >>>> packets with 'Antenna: 0' - Does this field indicate what transmitter >>>> the pkt was received on and indicate I'm only receiving from a single >>>> transmitter? >> >> Antenna callbacks aren't implemented in master branch yet. You can >> check ath-next-test for that or cherry picks Ben's patches. > > ok - I will do that. Does this field indicate which antenna the frame > was sent out or received from? I'm not familiar with this stuff, but I guess so. >> You can verify max number of spatial streams with iw by looking at `HT >> TX/RX MCS rate indexes supported:` and/or VHT TX/RX MCS sets. >> > > These look fine (see below) but is there a way to actually prove (ie > by radiotap) that frames come in/out of multiple antennas? > [...] > VHT RX MCS set: > 1 streams: MCS 0-9 > 2 streams: MCS 0-9 > 3 streams: MCS 0-9 > 4 streams: not supported > 5 streams: not supported > 6 streams: not supported > 7 streams: not supported > 8 streams: not supported > VHT RX highest supported: 0 Mbps > VHT TX MCS set: > 1 streams: MCS 0-9 > 2 streams: MCS 0-9 > 3 streams: MCS 0-9 > 4 streams: not supported > 5 streams: not supported > 6 streams: not supported > 7 streams: not supported > 8 streams: not supported [...] Device capabilities look fine (iw station dump you posted above already verified we're good though). >>>> I've also noticed that 'iw wlan0 station dump' statistics show in the >>>> rx bitrate stat that sometimes SGI is flagged and sometimes its not - >>>> should I expect this to change like this? I was under the impression >>>> that was a static configuration. >> >> I think this ath10k's hw rate control is free to pick SGI/LGI >> (assuming target station supports SGI at all). > > I always assumed a user would want to force one way or another - does > the rate control engine try to optimize by using short guard-band > interval (if supported) and then back-off to long if it detects lots > of retries? Beats me. At best you can force a single tx bitrate mcs and force it to LGI/SGI (firmware interface limitation). But that's it I guess. >>> I neglected to mention that on whichever end is running the iperf >>> server (receiving) I do see periodic 'failed to pop...' warnings, a >>> pair of them a couple times a minute. I'm not quite clear if this is a >>> hardware issue or something else: >>> >>> [ 3763.131280] ath10k: failed to pop amsdu from htt rx ring -22 >>> [ 3769.081093] ath10k: failed to pop chained msdus, dropping >>> [ 3769.086575] ath10k: failed to pop amsdu from htt rx ring -22 >>> [ 3781.092383] ath10k: failed to pop chained msdus, dropping >>> [ 3781.097869] ath10k: failed to pop amsdu from htt rx ring -22 >>> [ 3784.367225] ath10k: failed to pop chained msdus, dropping >>> [ 3784.372735] ath10k: failed to pop amsdu from htt rx ring -22 >>> [ 3809.501484] ath10k: failed to pop chained msdus, dropping >>> [ 3809.506993] ath10k: failed to pop amsdu from htt rx ring -22 >>> [ 3821.723404] ath10k: failed to pop chained msdus, dropping >>> [ 3821.728914] ath10k: failed to pop amsdu from htt rx ring -22 >>> [ 3832.067136] ath10k: failed to pop chained msdus, dropping >>> [ 3832.072690] ath10k: failed to pop amsdu from htt rx ring -22 >>> [ 3833.632859] ath10k: failed to pop chained msdus, dropping >>> [ 3833.638341] ath10k: failed to pop amsdu from htt rx ring -22 >>> [ 3837.940414] ath10k: failed to pop chained msdus, dropping >>> [ 3837.945982] ath10k: failed to pop amsdu from htt rx ring -22 >>> [ 3844.100514] ath10k: failed to pop chained msdus, dropping >> >> You don't have to worry about these messages. >> >> This was recently introduced by one of my patches to further analyze a >> bug that Avery was seeing (kernel panic due to skb_push). I plan on >> making a patch to clean this up. >> >> Anyway, thanks for letting know :-) > > ok - I saw the patch but from the commit message I through it was > indicating these were failures. Thanks for the explanation. Yeah. I originally thought these shouldn't pop up until Really Bad Things [tm] happen but I got proved wrong. Michał _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: help troubleshooting low throughput 2014-05-23 7:42 ` Michal Kazior @ 2014-05-24 5:32 ` Tim Harvey 2014-05-25 7:32 ` Kalle Valo 0 siblings, 1 reply; 7+ messages in thread From: Tim Harvey @ 2014-05-24 5:32 UTC (permalink / raw) To: Michal Kazior; +Cc: ath10k@lists.infradead.org On Fri, May 23, 2014 at 12:42 AM, Michal Kazior <michal.kazior@tieto.com> wrote: > On 22 May 2014 20:37, Tim Harvey <tharvey@gateworks.com> wrote: >> On Thu, May 22, 2014 at 3:08 AM, Michal Kazior <michal.kazior@tieto.com> wrote: >>> On 22 May 2014 11:46, Tim Harvey <tharvey@gateworks.com> wrote: >>>> On Thu, May 22, 2014 at 2:39 AM, Tim Harvey <tharvey@gateworks.com> wrote: >>>>> Greetings, >>>>> >>>>> I could use some help troubleshooting a low throughput issue. I'm >>>>> currently using the following: >>>>> - UNEX DAXA-O1 11ac/n/a 3x3 MIMO qca988x hw2.0 >>>>> http://www.unex.com.tw/product/daxa-o1 >>>>> - 80MHz channel w/o local interference >>>>> - ath10k git 0dbbb028a7c461777bf4a0d53780e539e6f40e14 (May 16) >>>>> - up-to-date git of hostapd/wpa_supplicant/iw >>>>> - fw 10.1.467.2-1 api 2 htt 2.1 >>>>> - infrastructure mode using >>>>> http://wireless.kernel.org/en/users/Drivers/ath10k/configuration#Full_hostapd_configuration >>> >>> Did you just copy&paste the example config file (and updated >>> interface=) or did you do something extra? >> >> Hi Michal, >> >> I disabled bridge mode, DFS, wpa, wps and added >> 'vht_oper_centr_freq_seg0_idx=42' which appears to be something new >> that is required or hostapd bails out: > > VHT requires a generic center frequency (or rather channel number in > hostapd) to be provided. Since you have channel=36 then the center > frequency for 80MHz bandwidth is 36+6 = 42. > > >> ### hostapd configuration file > [..] > > I'd try simply: > >> ht_capab=[HT40+][SHORT-GI-20][SHORT-GI-40] >> vht_capab=[MAX-MPDU-11454][SHORT-GI-80][MAX-A-MPDU-LEN-EXP7] > > Otherwise it looks fine to me. > > > [...] > >> Its showing 80Mhz MCS 5 (between 5 and 8) >> >> root@sta-97:~# iw wlan0 station dump >> Station 60:02:b4:9d:99:7f (on wlan0) >> inactive time: 590 ms >> rx bytes: 160004 >> rx packets: 1824 >> tx bytes: 9832 >> tx packets: 87 >> tx retries: 0 >> tx failed: 0 >> signal: -53 dBm >> signal avg: -52 dBm >> tx bitrate: 6.0 MBit/s >> rx bitrate: 975.0 MBit/s VHT-MCS 7 80MHz short GI VHT-NSS 3 >> authorized: yes >> authenticated: yes >> preamble: long >> WMM/WME: yes >> MFP: no >> TDLS peer: no >> >> ap is showing 80MHz width between MCS 5 and MCS 8: >> >> root@ap-99:~# iw wlan0 station dump >> Station 60:02:b4:9d:99:62 (on wlan0) >> inactive time: 0 ms >> rx bytes: 275591916 >> rx packets: 182178 >> tx bytes: 4394890 >> tx packets: 50807 >> tx retries: 0 >> tx failed: 0 >> signal: -60 dBm >> signal avg: -60 dBm >> tx bitrate: 6.0 MBit/s >> rx bitrate: 702.0 MBit/s VHT-MCS 5 80MHz VHT-NSS 3 >> authorized: yes >> authenticated: yes >> preamble: long >> WMM/WME: yes >> MFP: no >> TDLS peer: no > > This looks good. So rate control is doing rather fine. 3 spatial > streams (VHT-NSS 3) are at work. > > >>>>> I'm using iperf for throughput tests and getting no more than 220mbps >>>>> best case, typically more like 120mbps. The rx bitrate bounces around >>>>> MCS 5 to 8 and shows 3 spatial streams so I would be expecting a much >>>>> higher throughput. The cards are in boards with a quad-core ARM 1GHz >>>>> Cortex-A9 CPU and there is no indication the system is bottle-necked. >>>>> There are no other kernel modules loaded other thank >>>>> ath10k_pci/ath10k_core/ath and debugging is disabled. >>> >>> Currently ath10k doesn't really scale much with number of CPUs. There >>> are basically two tasklets that could split the work just a little >>> bit, but this requires interrupt spreading. From what I know some ARM >>> chips can't do that so ath10k ends up using only single CPU all the >>> time. 1GHz of an A9 should still be enough to get you 500mbps+ though. >> >> interesting. I see [ath10k_wq] in ps, what is the other task? ath10k >> will just register 1 interrupt for PCI, how would you spread that if >> only 1 ath10k device is in the system? > > ath10k_wq is the workqueue. It is not related to tasklets at all. > > Even if you have a single interrupt your controller may spread > interrupts across sockets/cores/threads. So one time device issues an > interrupt CPU0 gets interrupted and another time CPU1 gets > interrupted. > > >> I would agree that a 1GHz CortexA9 should be able to do well. The top >> application shows that only CPU0 is being utilized and never more than >> 25% or so (softirq mostly) and mostly idle. So I don't think this is >> any sort of CPU bottleneck. > > The 25% sound fishy considering you have quad-core CPU. I'm not really > sure if top (or top you use for that matter) reports percent wrt to a > single CPU or globally. I would certainly investigate this. I recall > vmstat sums everything up, i.e. if it says "25% sys" then it means > "25% of your entrie CPU set is doing sys, regardless which core it > is". > > >>> Did you run TCP and/or UDP tests? What direction did you test >>> (station->ap / ap->station)? >> >> both - the best throughput I see is appx 220mbps TCP and 260mbps UDP >> and this is consistent in both directions. > > Did you try using the -P switch to send parallel streams? E.g. -u -b > 100M -P5 for UDP? > > Also, now that I think about you don't have a bridge. This means your > AP system has to perform a lot more packet mangling which I guess can > be pretty taxing for the A9. > yep - turns out it is a cpu bottleneck. I'm not sure what I was looking at when I checked the performance before but now I see that I am hitting 100% utilization. Putting the AP in a bridge and moving iperf off of it, brings up my TCP bandwidth to 240mbps (at which point the STA running iperf is 100% pegged) and 463mbps UDP (at which point the AP was 100% pegged). Trying to move iperf off the STA to try to see what I could get TCP bandwidth up to, I found that putting the STA in a bridge of the (10.1...) firmware crashes which is the subject of another current thread (http://lists.infradead.org/pipermail/ath10k/2014-May/002042.html), so I'll respond about that there. thanks for the help Michal! Tim _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: help troubleshooting low throughput 2014-05-24 5:32 ` Tim Harvey @ 2014-05-25 7:32 ` Kalle Valo 0 siblings, 0 replies; 7+ messages in thread From: Kalle Valo @ 2014-05-25 7:32 UTC (permalink / raw) To: Tim Harvey; +Cc: Michal Kazior, ath10k@lists.infradead.org Tim Harvey <tharvey@gateworks.com> writes: >> Also, now that I think about you don't have a bridge. This means your >> AP system has to perform a lot more packet mangling which I guess can >> be pretty taxing for the A9. >> > > yep - turns out it is a cpu bottleneck. I'm not sure what I was > looking at when I checked the performance before but now I see that I > am hitting 100% utilization. > > Putting the AP in a bridge and moving iperf off of it, brings up my > TCP bandwidth to 240mbps (at which point the STA running iperf is 100% > pegged) and 463mbps UDP (at which point the AP was 100% pegged). Did you check that you had all kernel debug options disabled, including ath10k debug options? Kernel debugging code can significantly decrease maximum throughput. -- Kalle Valo _______________________________________________ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2014-05-25 7:33 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-05-22 9:39 help troubleshooting low throughput Tim Harvey 2014-05-22 9:46 ` Tim Harvey 2014-05-22 10:08 ` Michal Kazior 2014-05-22 18:37 ` Tim Harvey 2014-05-23 7:42 ` Michal Kazior 2014-05-24 5:32 ` Tim Harvey 2014-05-25 7:32 ` Kalle Valo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.