Question about starting up an AP

Question about starting up an AP

[Joshua Smith]

[-- Attachment #1: Type: text/plain, Size: 3448 bytes --]

I'm building an AP using the latest CW code with a 2.6.27.8 kernel, and a Linsys/Cisco WMP660N PCI adapter.  I'm having a problem with the boot-up sequence.  Sometimes, when I start hostapd, I get this error in the system log:

 kernel: phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy, aborting.

and hostapd craps out.  If I try to run hostapd again, the kernel dies.  (Crash details attached.)

(Other times, the boot and startup sequence works just fine.)

So I'd like to know if there is a way (in user space) for me to poll whether I'm going to get that error (which I presume means something in the card is not quite ready yet), and wait to start hostapd after that condition clears.

Note that I am doing an "iw wlan0 scan" before this, to see if I need to jump to a different channel, so I'm already talking to the card successfully.  I presume that means the firmware is already loaded, so I'm not really sure what is "busy".

Please reply directly, as I am not subscribed to this list.

Thanks in advance!!!!

-Joshua

BUG: unable to handle kernel NULL pointer dereference at 00000010
IP: [<f8d2d901>] :rt2800pci:rt2800pci_clear_entry+0x1/0x40
*pde = 00000000 
Oops: 0000 [#1] SMP 
Modules linked in: vga2usb usbvideo videodev v4l1_compat snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss pcmcia agpgart ppdev lp parport_pc parport pcspkr psmouse arc4 ecb rt2800pci rt2800lib crc_ccitt rt2x00pci rt2x00lib led_class compat_firmware_class mac80211 cfg80211 rfkill_backport ftdi_sio compat usbserial pcmcia_core eeprom_93cx6 sg snd_hda_intel e1000e snd_pcm snd_timer snd_page_alloc snd_hwdep snd soundcore evdev fuse aufs squashfs sqlzma unlzma [last unloaded: rsrc_nonstatic]

Pid: 5995, comm: hostapd Not tainted (2.6.27.8 #1)
EIP: 0060:[<f8d2d901>] EFLAGS: 00210246 CPU: 3
EIP is at rt2800pci_clear_entry+0x1/0x40 [rt2800pci]
EAX: 00000000 EBX: f698863c ECX: 00200296 EDX: f8d2dee0
ESI: f6988600 EDI: f5b6f000 EBP: 00000000 ESP: f6d75e4c
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process hostapd (pid: 5995, ti=f6d74000 task=f6ce2300 task.ti=f6d74000)
Stack: f698863c fa00eaec 00000000 f5b6f000 00000000 f7b67000 f5b6e280 fa00c629 
       f5b6f000 00000000 fa00ca3d f7b67480 00000001 fa177d4c 01b6e890 f7b67000 
       00000000 f7b67000 00000001 00001003 00001002 c066c366 f7b67000 c0668ad0 
Call Trace:
 [<fa00eaec>] rt2x00queue_init_queues+0x5c/0x90 [rt2x00lib]
 [<fa00c629>] rt2x00lib_enable_radio+0x29/0xa0 [rt2x00lib]
 [<fa00ca3d>] rt2x00lib_start+0x5d/0xd0 [rt2x00lib]
 [<fa177d4c>] ieee80211_do_open+0x21c/0x510 [mac80211]
 [<c066c366>] dev_open+0x56/0xb0
 [<c0668ad0>] dev_set_rx_mode+0x20/0x40
 [<c066a67f>] dev_change_flags+0x7f/0x190
 [<c06b1495>] devinet_ioctl+0x515/0x690
 [<c0668d24>] __dev_get_by_name+0x74/0x90
 [<c065d3f0>] sock_ioctl+0xd0/0x240
 [<c065d320>] sock_ioctl+0x0/0x240
 [<c018179b>] vfs_ioctl+0x2b/0x90
 [<c0181a5b>] do_vfs_ioctl+0x25b/0x2a0
 [<c0181af6>] sys_ioctl+0x56/0x70
 [<c0103262>] syscall_call+0x7/0xb
 [<c0700000>] add_card+0xad0/0xba0
 =======================
Code: 83 78 08 0e 74 14 8b 02 8b 48 04 85 c9 0f 99 c0 0f b6 c0 c3 8d b6 00 00 00 00 8b 02 8b 40 04 85 c0 0f 99 c0 0f b6 c0 c3 66 90 53 <8b> 48 10 8b 58 08 8b 40 04 83 78 08 0e 74 15 8b 11 83 c2 04 8b 
EIP: [<f8d2d901>] rt2800pci_clear_entry+0x1/0x40 [rt2800pci] SS:ESP 0068:f6d75e4c
---[ end trace cff9a5c094bb8837 ]---



[-- Attachment #2: crashLogs.ZIP --]
[-- Type: application/zip, Size: 19188 bytes --]

Re: Question about starting up an AP

[Helmut Schaa]

Hi Joshua,

Am Freitag 24 September 2010 schrieb Joshua Smith:
> BUG: unable to handle kernel NULL pointer dereference at 00000010
> IP: [<f8d2d901>] :rt2800pci:rt2800pci_clear_entry+0x1/0x40
> *pde = 00000000 
> Oops: 0000 [#1] SMP 
> Modules linked in: vga2usb usbvideo videodev v4l1_compat snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss pcmcia agpgart ppdev lp parport_pc parport pcspkr psmouse arc4 ecb rt2800pci rt2800lib crc_ccitt rt2x00pci rt2x00lib led_class compat_firmware_class mac80211 cfg80211 rfkill_backport ftdi_sio compat usbserial pcmcia_core eeprom_93cx6 sg snd_hda_intel e1000e snd_pcm snd_timer snd_page_alloc snd_hwdep snd soundcore evdev fuse aufs squashfs sqlzma unlzma [last unloaded: rsrc_nonstatic]
> 
> Pid: 5995, comm: hostapd Not tainted (2.6.27.8 #1)
> EIP: 0060:[<f8d2d901>] EFLAGS: 00210246 CPU: 3
> EIP is at rt2800pci_clear_entry+0x1/0x40 [rt2800pci]
> EAX: 00000000 EBX: f698863c ECX: 00200296 EDX: f8d2dee0
> ESI: f6988600 EDI: f5b6f000 EBP: 00000000 ESP: f6d75e4c
>  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
> Process hostapd (pid: 5995, ti=f6d74000 task=f6ce2300 task.ti=f6d74000)
> Stack: f698863c fa00eaec 00000000 f5b6f000 00000000 f7b67000 f5b6e280 fa00c629 
>        f5b6f000 00000000 fa00ca3d f7b67480 00000001 fa177d4c 01b6e890 f7b67000 
>        00000000 f7b67000 00000001 00001003 00001002 c066c366 f7b67000 c0668ad0 
> Call Trace:
>  [<fa00eaec>] rt2x00queue_init_queues+0x5c/0x90 [rt2x00lib]
>  [<fa00c629>] rt2x00lib_enable_radio+0x29/0xa0 [rt2x00lib]
>  [<fa00ca3d>] rt2x00lib_start+0x5d/0xd0 [rt2x00lib]
>  [<fa177d4c>] ieee80211_do_open+0x21c/0x510 [mac80211]
>  [<c066c366>] dev_open+0x56/0xb0
>  [<c0668ad0>] dev_set_rx_mode+0x20/0x40
>  [<c066a67f>] dev_change_flags+0x7f/0x190
>  [<c06b1495>] devinet_ioctl+0x515/0x690
>  [<c0668d24>] __dev_get_by_name+0x74/0x90
>  [<c065d3f0>] sock_ioctl+0xd0/0x240
>  [<c065d320>] sock_ioctl+0x0/0x240
>  [<c018179b>] vfs_ioctl+0x2b/0x90
>  [<c0181a5b>] do_vfs_ioctl+0x25b/0x2a0
>  [<c0181af6>] sys_ioctl+0x56/0x70
>  [<c0103262>] syscall_call+0x7/0xb
>  [<c0700000>] add_card+0xad0/0xba0
>  =======================
> Code: 83 78 08 0e 74 14 8b 02 8b 48 04 85 c9 0f 99 c0 0f b6 c0 c3 8d b6 00 00 00 00 8b 02 8b 40 04 85 c0 0f 99 c0 0f b6 c0 c3 66 90 53 <8b> 48 10 8b 58 08 8b 40 04 83 78 08 0e 74 15 8b 11 83 c2 04 8b 
> EIP: [<f8d2d901>] rt2800pci_clear_entry+0x1/0x40 [rt2800pci] SS:ESP 0068:f6d75e4c
> ---[ end trace cff9a5c094bb8837 ]---

Could you please try this patch against the crash (just a shot in the dark)?

Signed-off-by: Helmut Schaa <helmut.schaa@googlemail.com>
---
 drivers/net/wireless/rt2x00/rt2x00dev.c |    4 +---
 1 files changed, 1 insertions(+), 3 deletions(-)

diff --git a/drivers/net/wireless/rt2x00/rt2x00dev.c b/drivers/net/wireless/rt2x00/rt2x00dev.c
index 053fdd3..193fada 100644
--- a/drivers/net/wireless/rt2x00/rt2x00dev.c
+++ b/drivers/net/wireless/rt2x00/rt2x00dev.c
@@ -909,10 +909,8 @@ int rt2x00lib_start(struct rt2x00_dev *rt2x00dev)
 
 	/* Enable the radio */
 	retval = rt2x00lib_enable_radio(rt2x00dev);
-	if (retval) {
-		rt2x00queue_uninitialize(rt2x00dev);
+	if (retval)
 		return retval;
-	}
 
 	set_bit(DEVICE_STATE_STARTED, &rt2x00dev->flags);
  
-- 
1.7.1

Re: Question about starting up an AP

[Helmut Schaa]

Hi Joshua,

Am Freitag 24 September 2010 schrieb Joshua Smith:
> I'm building an AP using the latest CW code with a 2.6.27.8 kernel, and a
> Linsys/Cisco WMP660N PCI adapter.  I'm having a problem with the boot-up
> sequence.  Sometimes, when I start hostapd, I get this error in the system
> log:
> 
>  kernel: phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy, aborting.
> 
> and hostapd craps out.  If I try to run hostapd again, the kernel dies. 

Hmm, the legacy driver waits much longer (100 * 1ms) for DMA being ready then
we do in rt2800 (5 * 1ms). Not sure if that will make a difference for you
but please try this patch:

>From d9792259cfc5253725f3855bb2cb9b4acadec103 Mon Sep 17 00:00:00 2001
From: Helmut Schaa <helmut.schaa@googlemail.com>
Date: Sat, 25 Sep 2010 10:32:21 +0200
Subject: [PATCH] rt2x00: increase wait time for WPDMA being ready


Signed-off-by: Helmut Schaa <helmut.schaa@googlemail.com>
---
 drivers/net/wireless/rt2x00/rt2800lib.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/net/wireless/rt2x00/rt2800lib.c b/drivers/net/wireless/rt2x00/rt2800lib.c
index c7076de..058de10 100644
--- a/drivers/net/wireless/rt2x00/rt2800lib.c
+++ b/drivers/net/wireless/rt2x00/rt2800lib.c
@@ -277,7 +277,7 @@ int rt2800_wait_wpdma_ready(struct rt2x00_dev *rt2x00dev)
 	unsigned int i;
 	u32 reg;
 
-	for (i = 0; i < REGISTER_BUSY_COUNT; i++) {
+	for (i = 0; i < 100; i++) {
 		rt2800_register_read(rt2x00dev, WPDMA_GLO_CFG, &reg);
 		if (!rt2x00_get_field32(reg, WPDMA_GLO_CFG_TX_DMA_BUSY) &&
 		    !rt2x00_get_field32(reg, WPDMA_GLO_CFG_RX_DMA_BUSY))
-- 
1.7.1

Re: Question about starting up an AP

[Helmut Schaa]

Hi,

Am Montag 27 September 2010 schrieb Joshua Smith:
> The change to the REGISTER_BUSY_COUNT did not eliminate the issue. 
> The device still sometimes exhibits this behavior.  (It may be exhibiting
> it less often, but the occurrences are so random, that it's hard for me to
> know for sure.)

Yeah, as I said, just a shot in the dark.

> However, the other fix DID indeed eliminate the kernel crash when I tried
> to restart hostapd, and it also fixed a crash that I experienced when I
> tried to rmmod the rt2800pci module.  So you should definitely mainline
> that patch.

Sure.

> Now that I can retry hostapd and rmmod the modules, I have found that there
> is apparently no software workaround for the case where the DMA busy
> condition does not pass. 

Hmm, maybe there is but we have't found it yet ;)

> I have tried restarting hostapd, and I have tried rmmod'ding all the
> associated modules and modprobing them back in.  Nothing clears the
> condition except a reboot.

Too bad.

> Any other ideas of what I could do to reset the hardware?  I'd love to find
> a "soft" fix for this, rather than having to reboot to clear the condition.

Nothing obviouis at least. So, no without further debugging I don't know what's
going on in your case. Would be interesting if others experience the same
problem.

Helmut

Re: Question about starting up an AP

[Joshua Smith]

Just to be specific, here is the problem I'm having:

Sometimes my Linksys/Cisco WMP600N adapter will not initialize in AP mode.  Prior to going into AP mode, I do 'iw wlan0 scan' and get a bunch of info, so the hardware is working up to that point.  When I try to start hostapd, I get this error in dmesg:

phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy, aborting.
phy0 -> rt2800pci_set_device_state: Error - Device failed to enter state 4 (-5).

(note that I also get this message:
phy0 -> rt2800pci_mcu_status: Error - MCU request failed, no response from hardware
but I get that when everything works, as well.)

After the WPMDA TX/RX error occurs I usually cannot use the hardware at all.  Even a simple 'ifconfig wlan0 up' will result in the same error.  On rare occasions, the hardware will work if I simply retry hostapd.  But usually I have to reboot to get the hardware working again.

Helmut provided me with a patch to change the timeout for DMA to become ready; this did not solve the problem.

Restarting the kernel modules like this:

rmmod rt2800pci rt2800lib rt2x00pci rt2x00lib compat_firmware_class mac80211 cfg80211 compat
modprobe rt2800pci

also has no effect.  I'm open to suggestions for other ways I might be able to kick the pci card without rebooting.

Note that this exact software/hardware sometimes works, and sometimes fails.  I have reproduced this problem on multiple WiFi adapters (same make/model), so I am confident it is not an isolated hardware failure.  lspci -vv shows this data about the adapter:

03:01.0 Network controller: RaLink RT2800 802.11n PCI
	Subsystem: Linksys Unknown device 0067
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32 (500ns min, 1000ns max), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 22
	Region 0: Memory at f3000000 (32-bit, non-prefetchable) [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: rt2800pci
	Kernel modules: rt2800pci, rt2860sta

I am running compat-wireless 2010-09-20 with a 2.6.27.8 kernel (slax 6.0.9-based distro)

In the "probably not related, but just in case" category: I have another problem with these cards and hostapd where occasionally they will send a beacon but not respond to clients trying to join the network.  However, stopping hostapd and restarting it clears this condition.  (Now that I can successfully restart hostapd, thanks to a patch from Helmut!)  In one case, restarting hostapd resulted in the WPDMA TX/RX error, and then I had to reboot.

I'd be happy to try any wild ideas anyone has on this.

Please reply to me directly, as I am not subscribed to this list.

-Joshua

On Sep 27, 2010, at 1:56 PM, Helmut Schaa wrote:

> Hi,
> 
> Am Montag 27 September 2010 schrieb Joshua Smith:
>> The change to the REGISTER_BUSY_COUNT did not eliminate the issue. 
>> The device still sometimes exhibits this behavior.  (It may be exhibiting
>> it less often, but the occurrences are so random, that it's hard for me to
>> know for sure.)
> 
> Yeah, as I said, just a shot in the dark.
> 
>> However, the other fix DID indeed eliminate the kernel crash when I tried
>> to restart hostapd, and it also fixed a crash that I experienced when I
>> tried to rmmod the rt2800pci module.  So you should definitely mainline
>> that patch.
> 
> Sure.
> 
>> Now that I can retry hostapd and rmmod the modules, I have found that there
>> is apparently no software workaround for the case where the DMA busy
>> condition does not pass. 
> 
> Hmm, maybe there is but we have't found it yet ;)
> 
>> I have tried restarting hostapd, and I have tried rmmod'ding all the
>> associated modules and modprobing them back in.  Nothing clears the
>> condition except a reboot.
> 
> Too bad.
> 
>> Any other ideas of what I could do to reset the hardware?  I'd love to find
>> a "soft" fix for this, rather than having to reboot to clear the condition.
> 
> Nothing obviouis at least. So, no without further debugging I don't know what's
> going on in your case. Would be interesting if others experience the same
> problem.
> 
> Helmut

Re: Question about starting up an AP

[Helmut Schaa]

Am Montag 27 September 2010 schrieb Joshua Smith: 
> I have another problem with these cards and hostapd where occasionally they
> will send a beacon but not respond to clients trying to join the network. 
> However, stopping hostapd and restarting it clears this condition.

Most likely a stuck tx queue. Do you have 11n enabled?

A fix to make the tx queue not stuck anymore (at least on rt3052 SoC) is
already in rt2x00 git [1] and [2]. However, I'm not sure if this is the same
issue you're facing.

Helmut

[1] http://git.kernel.org/?p=linux/kernel/git/ivd/rt2x00.git;a=commit;h=4758f8c3686911ecff11f39508a60327d42893df
[2] http://git.kernel.org/?p=linux/kernel/git/ivd/rt2x00.git;a=commit;h=0402943d65e6832ba1cd88621d79c0c83a23ddeb

Re: Question about starting up an AP

[Joshua Smith]

On Sep 27, 2010, at 3:43 PM, Helmut Schaa wrote:

> Am Montag 27 September 2010 schrieb Joshua Smith: 
>> I have another problem with these cards and hostapd where occasionally they
>> will send a beacon but not respond to clients trying to join the network. 
>> However, stopping hostapd and restarting it clears this condition.
> 
> Most likely a stuck tx queue. Do you have 11n enabled?

No.  My hostapd has:

hw_mode=a

Is that consistent or inconsistent with your stuck tx queue theory?

> 
> A fix to make the tx queue not stuck anymore (at least on rt3052 SoC) is
> already in rt2x00 git [1] and [2]. However, I'm not sure if this is the same
> issue you're facing.

Would things "already in rt2x00 git" be in the compat-wireless bleeding edge tarball?

> 
> Helmut
> 
> [1] http://git.kernel.org/?p=linux/kernel/git/ivd/rt2x00.git;a=commit;h=4758f8c3686911ecff11f39508a60327d42893df
> [2] http://git.kernel.org/?p=linux/kernel/git/ivd/rt2x00.git;a=commit;h=0402943d65e6832ba1cd88621d79c0c83a23ddeb

Re: Question about starting up an AP

[Helmut Schaa]

Am Montag 27 September 2010 schrieb Joshua Smith:
> 
> On Sep 27, 2010, at 3:43 PM, Helmut Schaa wrote:
> 
> > Am Montag 27 September 2010 schrieb Joshua Smith: 
> >> I have another problem with these cards and hostapd where occasionally they
> >> will send a beacon but not respond to clients trying to join the network. 
> >> However, stopping hostapd and restarting it clears this condition.
> > 
> > Most likely a stuck tx queue. Do you have 11n enabled?
> 
> No.  My hostapd has:
> 
> hw_mode=a
> 
> Is that consistent or inconsistent with your stuck tx queue theory?

With 11n it is more likely to occur. So, it might be the same issue.

> > A fix to make the tx queue not stuck anymore (at least on rt3052 SoC) is
> > already in rt2x00 git [1] and [2]. However, I'm not sure if this is the same
> > issue you're facing.
> 
> Would things "already in rt2x00 git" be in the compat-wireless bleeding edge tarball?

Nope. But it means they are on their way into wireless-testing and thus into
compat-wireless.

Helmut

Re: Question about starting up an AP

[Joshua Smith]

OK, I have a working system now.  It's not pretty, but it's better than where I was.

- Using compat-wireless-2010-09-20, with the patch "rt2x00: Fix oops caused by error path in rt2x00lib_start"
  Compiled with:
CONFIG_RT2X00_DEBUG=y
CONFIG_RT2X00_LIB_DEBUGFS=y
  and all the other DEBUGFS also set to y

- Using a 2.6.27.8 kernel compiled with debug_fs support

When my system comes up, I try to start hostapd.  This may succeed, or it may fail because of the error:

phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy, aborting.
phy0 -> rt2800pci_set_device_state: Error - Device failed to enter state 4 (-5).

If the error occurs, try starting hostapd one more time.  (This hardly ever works.)  If it fails again, reboot.

If we got past hostapd starting, then we need to see whether the card is deaf.  Do this by polling
/sys/kernel/debug/ieee80211/phy0/rt2800pci/queue/queue
for a while, to see if the RX queue counts ever get over zero.

If they do, we're good to go.

If not, kick the card this way:

echo 1 > /sys/kernel/debug/ieee80211/phy0/reset

Now check to see if that hit the WPDMA TX/RX busy error.  If it did, reboot.  If not, assume that restarting the card fixed the RX queue (it always seems to), and we're good to go.

For your amusement and horror, I've pasted in the bash script below.

At this point, if I could eliminate the WPDMA TX/RX busy error, I'd be able to eliminate all the reboots, and make a more sane system.  I'm still open to any and all suggestions on how to clear that condition!

-Joshua

…hideous bash script...

  ifconfig wlan0 down

  sleep 1
  if ! hostapd -B /etc/hostapd.conf
  then
    echo "Problem starting Wi-Fi hardware.  Retrying..."
    sleep 3
    if ! hostapd -B /etc/hostapd.conf
    then
      echo -e "***\n*** Unable to initialize Wi-Fi hardware\n***\n***\n*** Restarting...\n***"
      cleanreboot
      sleep 60
    fi
  fi
  sleep 1

  mount -t debugfs none /sys/kernel/debug/
  if [ -e /sys/kernel/debug/ieee80211/phy0/rt2800pci/ ]
  then
    echo "Testing Wi-Fi hardware..."
    RETRY=1
    while [ $RETRY -lt 20 ]
    do
      if [ `cat /sys/kernel/debug/ieee80211/phy0/rt2800pci/queue/queue | egrep -c '^14.0.24.24.0.0.0$'` -eq 0 ]
      then
        break
      fi
      echo -n '.'
      sleep 1
      RETRY=$((RETRY+1))
    done
    if [ $RETRY -eq 20 ]
    then
      echo "failed.  Resetting..."
      echo 1 > /sys/kernel/debug/ieee80211/phy0/reset
      sleep 1
      if [ `dmesg | tail -n 30 | grep -c "rt2800_wait_wpdma_ready"` -gt 0 ]
      then
        echo -e "***\n*** Unable to reset Wi-Fi hardware\n***\n***\n*** Restarting...\n***"
        cleanreboot
        sleep 60
      fi
    else
     echo "Passed!"
    fi
  fi

Re: Question about starting up an AP

[Joshua Smith]

I think I have a clue about the WPDMA TX/RX busy bug.  I was having trouble using the latest compat-wireless with USB dongle I have (ID AB50:2019 rt73usb), so I tried rolling Helmut's "oops" fix into:

compat-wireless-2010-07-15

which I have been using successfully for the last few months.  Lo and behold, this version does *NOT* exhibit the TX/RX busy error.  So whatever is causing that problem was introduced between 7/15/2010 and 9/20/2010.

This leaves me très happy, since with the "oops" fix, I can restart hostapd at will, and the card never crashes.  Woo hoo!

-Joshua

On Sep 29, 2010, at 2:21 PM, Joshua Smith wrote:

> OK, I have a working system now.  It's not pretty, but it's better than where I was.
> 
> - Using compat-wireless-2010-09-20, with the patch "rt2x00: Fix oops caused by error path in rt2x00lib_start"
>  Compiled with:
> CONFIG_RT2X00_DEBUG=y
> CONFIG_RT2X00_LIB_DEBUGFS=y
>  and all the other DEBUGFS also set to y
> 
> - Using a 2.6.27.8 kernel compiled with debug_fs support
> 
> When my system comes up, I try to start hostapd.  This may succeed, or it may fail because of the error:
> 
> phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy, aborting.
> phy0 -> rt2800pci_set_device_state: Error - Device failed to enter state 4 (-5).
> 
> If the error occurs, try starting hostapd one more time.  (This hardly ever works.)  If it fails again, reboot.
> 
> If we got past hostapd starting, then we need to see whether the card is deaf.  Do this by polling
> /sys/kernel/debug/ieee80211/phy0/rt2800pci/queue/queue
> for a while, to see if the RX queue counts ever get over zero.
> 
> If they do, we're good to go.
> 
> If not, kick the card this way:
> 
> echo 1 > /sys/kernel/debug/ieee80211/phy0/reset
> 
> Now check to see if that hit the WPDMA TX/RX busy error.  If it did, reboot.  If not, assume that restarting the card fixed the RX queue (it always seems to), and we're good to go.
> 
> For your amusement and horror, I've pasted in the bash script below.
> 
> At this point, if I could eliminate the WPDMA TX/RX busy error, I'd be able to eliminate all the reboots, and make a more sane system.  I'm still open to any and all suggestions on how to clear that condition!
> 
> -Joshua
> 
> …hideous bash script...
> 
>  ifconfig wlan0 down
> 
>  sleep 1
>  if ! hostapd -B /etc/hostapd.conf
>  then
>    echo "Problem starting Wi-Fi hardware.  Retrying..."
>    sleep 3
>    if ! hostapd -B /etc/hostapd.conf
>    then
>      echo -e "***\n*** Unable to initialize Wi-Fi hardware\n***\n***\n*** Restarting...\n***"
>      cleanreboot
>      sleep 60
>    fi
>  fi
>  sleep 1
> 
>  mount -t debugfs none /sys/kernel/debug/
>  if [ -e /sys/kernel/debug/ieee80211/phy0/rt2800pci/ ]
>  then
>    echo "Testing Wi-Fi hardware..."
>    RETRY=1
>    while [ $RETRY -lt 20 ]
>    do
>      if [ `cat /sys/kernel/debug/ieee80211/phy0/rt2800pci/queue/queue | egrep -c '^14.0.24.24.0.0.0$'` -eq 0 ]
>      then
>        break
>      fi
>      echo -n '.'
>      sleep 1
>      RETRY=$((RETRY+1))
>    done
>    if [ $RETRY -eq 20 ]
>    then
>      echo "failed.  Resetting..."
>      echo 1 > /sys/kernel/debug/ieee80211/phy0/reset
>      sleep 1
>      if [ `dmesg | tail -n 30 | grep -c "rt2800_wait_wpdma_ready"` -gt 0 ]
>      then
>        echo -e "***\n*** Unable to reset Wi-Fi hardware\n***\n***\n*** Restarting...\n***"
>        cleanreboot
>        sleep 60
>      fi
>    else
>     echo "Passed!"
>    fi
>  fi
> 

Re: Question about starting up an AP

[Helmut Schaa]

Am Donnerstag 30 September 2010 schrieb Joshua Smith:
> I think I have a clue about the WPDMA TX/RX busy bug.  I was having trouble
> using the latest compat-wireless with USB dongle I have (ID AB50:2019
> rt73usb), so I tried rolling Helmut's "oops" fix into:
> 
> compat-wireless-2010-07-15
> 
> which I have been using successfully for the last few months.  Lo and behold,
> this version does *NOT* exhibit the TX/RX busy error.  So whatever is causing
> that problem was introduced between 7/15/2010 and 9/20/2010.

Thanks for the analysis Joshua! Since this is a regression I guess we
should try to find out quickly which commit introduced it (and fix it up
of course) ...

Helmut

Re: Question about starting up an AP

[Helmut Schaa]

Hi Joshua,

Am Donnerstag 30 September 2010 schrieb Helmut Schaa:
> Am Donnerstag 30 September 2010 schrieb Joshua Smith:
> > I think I have a clue about the WPDMA TX/RX busy bug.  I was having trouble
> > using the latest compat-wireless with USB dongle I have (ID AB50:2019
> > rt73usb), so I tried rolling Helmut's "oops" fix into:
> > 
> > compat-wireless-2010-07-15
> > 
> > which I have been using successfully for the last few months.  Lo and behold,
> > this version does *NOT* exhibit the TX/RX busy error.  So whatever is causing
> > that problem was introduced between 7/15/2010 and 9/20/2010.
> 
> Thanks for the analysis Joshua! Since this is a regression I guess we
> should try to find out quickly which commit introduced it (and fix it up
> of course) ...

Unfortunately during that time a lot of rt2x00 patches went into
wireless-testing. However, there's one suspicious commit that touches the
relevant code: "rt2x00: Merge rt2800{pci/usb} radio enabling/disabling code
to rt2800lib".

This is again a shot in the dark but worth a try. Could you please revert this
commit [1] from the newer compat-wireless tarball and try if it fixes the "DMA
Busy" issue?

Thanks Joshua,
Helmut

[1] http://git.kernel.org/?p=linux/kernel/git/linville/wireless-testing.git;a=commit;h=b9a07ae9d9e09662013992088fd58ffbcb2f9a30

Re: Question about starting up an AP

[Joshua Smith]

OK, I finally got a chance to run the test.  Before doing the revert, I tried just using the current drivers, but with a ridiculously large number of retries.

Lo and behold: no error.  So I think this is the reason the old drivers were working for me, not anything that happened between July and September.  When I went back to the old drivers, I had made this change, and they worked.  I figured it was because I was using the old drivers, but now I realize it is because of this change.

Also, I can see the behavior pretty clearly.  Sometimes when I start up hostapd, it starts right up, and sometimes it takes about a second, and then starts up.  So it looks like this retry number just needs to be really, really big.  (Note that with this change in the old drivers, on very rare occasions, I still see the "busy, aborting" error.  So perhaps it should be even bigger than 1000?)

Not sure how you want to proceed with this, but this is the code that makes my AP work:

 nl compat-wireless-2010-09-20/drivers/net/wireless/rt2x00/rt2800lib.c | grep -C 10 1000/

   225	int rt2800_wait_wpdma_ready(struct rt2x00_dev *rt2x00dev)
   226	{
   227		unsigned int i;
   228		u32 reg;
       
   229		for (i = 0; i < 1000/*REGISTER_BUSY_COUNT*/; i++) {
   230			rt2800_register_read(rt2x00dev, WPDMA_GLO_CFG, &reg);
   231			if (!rt2x00_get_field32(reg, WPDMA_GLO_CFG_TX_DMA_BUSY) &&
   232			    !rt2x00_get_field32(reg, WPDMA_GLO_CFG_RX_DMA_BUSY))
   233				return 0;
       
   234			msleep(1);
   235		}
       
   236		ERROR(rt2x00dev, "WPDMA TX/RX busy, aborting.\n");
   237		return -EACCES;


On Oct 1, 2010, at 1:42 AM, Helmut Schaa wrote:

> Hi Joshua,
> 
> Am Donnerstag 30 September 2010 schrieb Helmut Schaa:
>> Am Donnerstag 30 September 2010 schrieb Joshua Smith:
>>> I think I have a clue about the WPDMA TX/RX busy bug.  I was having trouble
>>> using the latest compat-wireless with USB dongle I have (ID AB50:2019
>>> rt73usb), so I tried rolling Helmut's "oops" fix into:
>>> 
>>> compat-wireless-2010-07-15
>>> 
>>> which I have been using successfully for the last few months.  Lo and behold,
>>> this version does *NOT* exhibit the TX/RX busy error.  So whatever is causing
>>> that problem was introduced between 7/15/2010 and 9/20/2010.
>> 
>> Thanks for the analysis Joshua! Since this is a regression I guess we
>> should try to find out quickly which commit introduced it (and fix it up
>> of course) ...
> 
> Unfortunately during that time a lot of rt2x00 patches went into
> wireless-testing. However, there's one suspicious commit that touches the
> relevant code: "rt2x00: Merge rt2800{pci/usb} radio enabling/disabling code
> to rt2800lib".
> 
> This is again a shot in the dark but worth a try. Could you please revert this
> commit [1] from the newer compat-wireless tarball and try if it fixes the "DMA
> Busy" issue?
> 
> Thanks Joshua,
> Helmut
> 
> [1] http://git.kernel.org/?p=linux/kernel/git/linville/wireless-testing.git;a=commit;h=b9a07ae9d9e09662013992088fd58ffbcb2f9a30

Re: Question about starting up an AP

[Helmut Schaa]

Am Donnerstag 14 Oktober 2010 schrieb Joshua Smith:
> OK, I finally got a chance to run the test.  Before doing the revert, I
> tried just using the current drivers, but with a ridiculously large
> number of retries.

Thanks for testing.

> Lo and behold: no error.  So I think this is the reason the old drivers
> were working for me, not anything that happened between July and September. 
> When I went back to the old drivers, I had made this change, and they
> worked.  I figured it was because I was using the old drivers, but now I
> realize it is because of this change.
> 
> Also, I can see the behavior pretty clearly.  Sometimes when I start up
> hostapd, it starts right up, and sometimes it takes about a second, and then
> starts up.  So it looks like this retry number just needs to be really,
> really big.  (Note that with this change in the old drivers, on very rare
> occasions, I still see the "busy, aborting" error.  So perhaps it should
> be even bigger than 1000?)
> 
> Not sure how you want to proceed with this, but this is the code that makes my AP work:
> 
>  nl compat-wireless-2010-09-20/drivers/net/wireless/rt2x00/rt2800lib.c | grep -C 10 1000/
> 
>    225	int rt2800_wait_wpdma_ready(struct rt2x00_dev *rt2x00dev)
>    226	{
>    227		unsigned int i;
>    228		u32 reg;
>        
>    229		for (i = 0; i < 1000/*REGISTER_BUSY_COUNT*/; i++) {
>    230			rt2800_register_read(rt2x00dev, WPDMA_GLO_CFG, &reg);
>    231			if (!rt2x00_get_field32(reg, WPDMA_GLO_CFG_TX_DMA_BUSY) &&
>    232			    !rt2x00_get_field32(reg, WPDMA_GLO_CFG_RX_DMA_BUSY))
>    233				return 0;
>        
>    234			msleep(1);
>    235		}
>        
>    236		ERROR(rt2x00dev, "WPDMA TX/RX busy, aborting.\n");
>    237		return -EACCES;

I'll let Ivo decide on this. At least if we really want to wait more then one
second for DMA to be ready we should increase the msleep(1) to msleep(10) and
keep the maximum number of retries at 100.

Joshua, did you ever ran the ralink legacy driver? Because that one also waits
a maximum of 100ms and thus would also fail sometimes ...

Helmut

Re: Question about starting up an AP

[Joshua Smith]

The legacy drivers don't work with iw, hostapd, etc., so I really couldn't test with those.

On Oct 15, 2010, at 5:46 AM, Helmut Schaa wrote:

> Am Donnerstag 14 Oktober 2010 schrieb Joshua Smith:
>> OK, I finally got a chance to run the test.  Before doing the revert, I
>> tried just using the current drivers, but with a ridiculously large
>> number of retries.
> 
> Thanks for testing.
> 
>> Lo and behold: no error.  So I think this is the reason the old drivers
>> were working for me, not anything that happened between July and September. 
>> When I went back to the old drivers, I had made this change, and they
>> worked.  I figured it was because I was using the old drivers, but now I
>> realize it is because of this change.
>> 
>> Also, I can see the behavior pretty clearly.  Sometimes when I start up
>> hostapd, it starts right up, and sometimes it takes about a second, and then
>> starts up.  So it looks like this retry number just needs to be really,
>> really big.  (Note that with this change in the old drivers, on very rare
>> occasions, I still see the "busy, aborting" error.  So perhaps it should
>> be even bigger than 1000?)
>> 
>> Not sure how you want to proceed with this, but this is the code that makes my AP work:
>> 
>> nl compat-wireless-2010-09-20/drivers/net/wireless/rt2x00/rt2800lib.c | grep -C 10 1000/
>> 
>>   225	int rt2800_wait_wpdma_ready(struct rt2x00_dev *rt2x00dev)
>>   226	{
>>   227		unsigned int i;
>>   228		u32 reg;
>> 
>>   229		for (i = 0; i < 1000/*REGISTER_BUSY_COUNT*/; i++) {
>>   230			rt2800_register_read(rt2x00dev, WPDMA_GLO_CFG, &reg);
>>   231			if (!rt2x00_get_field32(reg, WPDMA_GLO_CFG_TX_DMA_BUSY) &&
>>   232			    !rt2x00_get_field32(reg, WPDMA_GLO_CFG_RX_DMA_BUSY))
>>   233				return 0;
>> 
>>   234			msleep(1);
>>   235		}
>> 
>>   236		ERROR(rt2x00dev, "WPDMA TX/RX busy, aborting.\n");
>>   237		return -EACCES;
> 
> I'll let Ivo decide on this. At least if we really want to wait more then one
> second for DMA to be ready we should increase the msleep(1) to msleep(10) and
> keep the maximum number of retries at 100.
> 
> Joshua, did you ever ran the ralink legacy driver? Because that one also waits
> a maximum of 100ms and thus would also fail sometimes ...
> 
> Helmut

Re: Question about starting up an AP

[Helmut Schaa]

Am Freitag 15 Oktober 2010 schrieb Helmut Schaa:
> Am Donnerstag 14 Oktober 2010 schrieb Joshua Smith:
> > Not sure how you want to proceed with this, but this is the code that makes my AP work:
> > 
> >  nl compat-wireless-2010-09-20/drivers/net/wireless/rt2x00/rt2800lib.c | grep -C 10 1000/
> > 
> >    225	int rt2800_wait_wpdma_ready(struct rt2x00_dev *rt2x00dev)
> >    226	{
> >    227		unsigned int i;
> >    228		u32 reg;
> >        
> >    229		for (i = 0; i < 1000/*REGISTER_BUSY_COUNT*/; i++) {
> >    230			rt2800_register_read(rt2x00dev, WPDMA_GLO_CFG, &reg);
> >    231			if (!rt2x00_get_field32(reg, WPDMA_GLO_CFG_TX_DMA_BUSY) &&
> >    232			    !rt2x00_get_field32(reg, WPDMA_GLO_CFG_RX_DMA_BUSY))
> >    233				return 0;
> >        
> >    234			msleep(1);
> >    235		}
> >        
> >    236		ERROR(rt2x00dev, "WPDMA TX/RX busy, aborting.\n");
> >    237		return -EACCES;
> 
> I'll let Ivo decide on this. At least if we really want to wait more then one
> second for DMA to be ready we should increase the msleep(1) to msleep(10) and
> keep the maximum number of retries at 100.

Ping.

Ivo, do we want to add the excessive waiting (up to one second) for DMA on
rt2800 as it fixes Joshuas device? At the moment we're waiting 100ms.

Helmut

Re: Question about starting up an AP

[Johannes Berg]

On Tue, 2010-10-26 at 11:35 +0200, Helmut Schaa wrote:

> > I'll let Ivo decide on this. At least if we really want to wait more then one
> > second for DMA to be ready we should increase the msleep(1) to msleep(10) and
> > keep the maximum number of retries at 100.

> Ivo, do we want to add the excessive waiting (up to one second) for DMA on
> rt2800 as it fixes Joshuas device? At the moment we're waiting 100ms.

I don't usually chime in on other drivers but is there any reason not
to? Properly functioning devices obviously reply within 100ms, so they
wouldn't change behaviour, hopefully broken ones that _never_ work are
rare, and the remaining ones like Joshua's would be fixed ...

johannes

Re: Question about starting up an AP

[Helmut Schaa]

Am Dienstag 26 Oktober 2010 schrieb Johannes Berg:
> On Tue, 2010-10-26 at 11:35 +0200, Helmut Schaa wrote:
> 
> > > I'll let Ivo decide on this. At least if we really want to wait more then one
> > > second for DMA to be ready we should increase the msleep(1) to msleep(10) and
> > > keep the maximum number of retries at 100.
> 
> > Ivo, do we want to add the excessive waiting (up to one second) for DMA on
> > rt2800 as it fixes Joshuas device? At the moment we're waiting 100ms.
> 
> I don't usually chime in on other drivers but is there any reason not
> to? Properly functioning devices obviously reply within 100ms, so they
> wouldn't change behaviour, hopefully broken ones that _never_ work are
> rare, and the remaining ones like Joshua's would be fixed ...

True, nevertheless I wanted to get Ivos ack before going ahead.

Helmut

Re: Question about starting up an AP

[Ivo Van Doorn]

On Tue, Oct 26, 2010 at 12:04 PM, Helmut Schaa
<helmut.schaa@googlemail.com> wrote:
> Am Dienstag 26 Oktober 2010 schrieb Johannes Berg:
>> On Tue, 2010-10-26 at 11:35 +0200, Helmut Schaa wrote:
>>
>> > > I'll let Ivo decide on this. At least if we really want to wait more then one
>> > > second for DMA to be ready we should increase the msleep(1) to msleep(10) and
>> > > keep the maximum number of retries at 100.
>>
>> > Ivo, do we want to add the excessive waiting (up to one second) for DMA on
>> > rt2800 as it fixes Joshuas device? At the moment we're waiting 100ms.
>>
>> I don't usually chime in on other drivers but is there any reason not
>> to? Properly functioning devices obviously reply within 100ms, so they
>> wouldn't change behaviour, hopefully broken ones that _never_ work are
>> rare, and the remaining ones like Joshua's would be fixed ...
>
> True, nevertheless I wanted to get Ivos ack before going ahead.

yeah, I don't mind waiting for a full second, on my machine a normal
ifconfig up takes already ages for some odd reason, so another second wouldn't
matter if that means that more devices can function. :)

Ivo

Re: Question about starting up an AP

[Helmut Schaa]

Am Montag 27 September 2010 schrieb Joshua Smith:
> Sometimes my Linksys/Cisco WMP600N adapter will not initialize in AP mode. 
> Prior to going into AP mode, I do 'iw wlan0 scan' and get a bunch of info,
> so the hardware is working up to that point.  When I try to start hostapd,
> I get this error in dmesg:
> 
> phy0 -> rt2800_wait_wpdma_ready: Error - WPDMA TX/RX busy, aborting.
> phy0 -> rt2800pci_set_device_state: Error - Device failed to enter state 4 (-5).

Since you've scanned already before, I guess some DMA registers aren't reset
correctly when taking the interface down.

> (note that I also get this message:
> phy0 -> rt2800pci_mcu_status: Error - MCU request failed, no response from hardware
> but I get that when everything works, as well.)

Yeah, that shouldn't be related.

Helmut