bcm43xx driver unstable behaviour (and linux wireless is junk btw)

bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Benjamin Herrenschmidt]

Hi folks !

So this is 2.6.18 + Larry "fix" (though I've seen this problem before,
it seems using WPA just make it happen more often).

This is also a 4318, so the link is pretty weak due to the Tx Power
problem and I suspects it makes the driver problems more visible...

So basically, I "lose" the link every few minutes for a minute or so, I
suspect it's related to wpa_supplicant vs. the ack losses due to the
4318 Tx Power problems. That alone would be ok though, if the driver
wasn't totally stuck after a while. (Similar problem to after
sleep/wakeup, looks like nothign goes through).

When it goes bunk, it looks like that in the logs:

Sep 24 12:24:18 localhost kernel: [  285.686826] SoftMAC: Sent Authentication Request to 00:0f:66:52:4b:60.
Sep 24 12:24:18 localhost kernel: [  285.686976] SoftMAC: generic IE set to <....>
Sep 24 12:24:18 localhost kernel: [  285.686999] SoftMAC: Already associating or associated to 00:0f:66:52:4b:60
Sep 24 12:24:28 localhost kernel: [  295.687229] SoftMAC: Start scanning with channel: 1
Sep 24 12:24:28 localhost kernel: [  295.687240] SoftMAC: Scanning 14 channels
Sep 24 12:24:29 localhost kernel: [  296.027053] SoftMAC: Scanning finished
Sep 24 12:24:29 localhost kernel: [  296.035267] SoftMAC: generic IE set to <....>
Sep 24 12:24:29 localhost kernel: [  296.035310] SoftMAC: Already associating or associated to 00:0f:66:52:4b:60
Sep 24 12:24:31 localhost kernel: [  297.690969] SoftMAC: Sent Authentication Request to 00:0f:66:52:4b:60.
Sep 24 12:24:39 localhost kernel: [  306.039210] SoftMAC: Start scanning with channel: 1
Sep 24 12:24:39 localhost kernel: [  306.039222] SoftMAC: Scanning 14 channels
Sep 24 12:24:39 localhost kernel: [  306.375046] SoftMAC: Scanning finished
Sep 24 12:24:39 localhost kernel: [  306.383018] SoftMAC: generic IE set to <....>
Sep 24 12:24:39 localhost kernel: [  306.383075] SoftMAC: Already associating or associated to 00:0f:66:52:4b:60
Sep 24 12:24:42 localhost kernel: [  309.695021] SoftMAC: Sent Authentication Request to 00:0f:66:52:4b:60.
Sep 24 12:24:49 localhost kernel: [  316.387211] SoftMAC: Start scanning with channel: 1

 etc...

Then, if you rmmod, you get back a prompt, and about a second later, the kernel blows up. At this point,
I've always been in X and it's too dead to dump anything into the disk logs so I don't know what
the precise crash is, but it looks to me like the driver is not properly removing some timer
or something there.

Note that it also goes "bunk" on sleep/wakeup, and sometimes ifdown/ifup... in general, it's fragile and
just 'loses it' in which case the only way to get it back is to rmmod/insmod.

Doesn't help me to have my prism54 not working with WPA (apparently, the driver looks like it handles hostap
ioctls but it doesn't agree on the ioctl numbers, among others, with whatever wpa_supplicant sends when
configured to wpa mode... somebody knows if that driver is maintained ?)

So at this point I have a choice between two wireless devices that don't work (and none of them is less than
a couple years old). Looks like the linux wireless situation isn't getting any better since last KS.

Oh and I don't care about "it works in dscape stack" sort of crap I regulary get. I want something that
works with upstream kernels. That isn't that much to ask... or is it ?

Ben, back to ethernet cables.

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Daniel Drake]

Benjamin Herrenschmidt wrote:
> Oh and I don't care about "it works in dscape stack" sort of crap I regulary get. I want something that
> works with upstream kernels. That isn't that much to ask... or is it ?

wpa_supplicant triggers races in softmac relatively easily, which are 
hard to fix properly. At least for me, motivation to work on this stuff 
is low given the potentially impending merge of devicescape, and every 
time I do spend some time investigating I just get even more frustrated 
at how difficult WE is to implement *properly* for "non-hardmac" 
drivers. We really have a need for a configuration system designed 
around 802.11.

I agree, the stuff in mainline should be fixed, but at least personally 
I am finding it harder and harder to justify working on softmac.

Daniel

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Benjamin Herrenschmidt]

On Sat, 2006-09-23 at 23:45 -0400, Daniel Drake wrote:
> Benjamin Herrenschmidt wrote:
> > Oh and I don't care about "it works in dscape stack" sort of crap I regulary get. I want something that
> > works with upstream kernels. That isn't that much to ask... or is it ?
> 
> wpa_supplicant triggers races in softmac relatively easily, which are 
> hard to fix properly. At least for me, motivation to work on this stuff 
> is low given the potentially impending merge of devicescape, and every 
> time I do spend some time investigating I just get even more frustrated 
> at how difficult WE is to implement *properly* for "non-hardmac" 
> drivers. We really have a need for a configuration system designed 
> around 802.11.
> 
> I agree, the stuff in mainline should be fixed, but at least personally 
> I am finding it harder and harder to justify working on softmac.

So what are the chances of getting this dscape stack merged, let's
say... for 2.6.19 ? Or we'll get yet another full release with barely
working wireless ?

Cheers,
Ben.

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Michael Buesch]

On Sunday 24 September 2006 08:18, Benjamin Herrenschmidt wrote:
> On Sat, 2006-09-23 at 23:45 -0400, Daniel Drake wrote:
> > Benjamin Herrenschmidt wrote:
> > > Oh and I don't care about "it works in dscape stack" sort of crap I regulary get. I want something that
> > > works with upstream kernels. That isn't that much to ask... or is it ?
> > 
> > wpa_supplicant triggers races in softmac relatively easily, which are 
> > hard to fix properly. At least for me, motivation to work on this stuff 
> > is low given the potentially impending merge of devicescape, and every 
> > time I do spend some time investigating I just get even more frustrated 
> > at how difficult WE is to implement *properly* for "non-hardmac" 
> > drivers. We really have a need for a configuration system designed 
> > around 802.11.
> > 
> > I agree, the stuff in mainline should be fixed, but at least personally 
> > I am finding it harder and harder to justify working on softmac.
> 
> So what are the chances of getting this dscape stack merged, let's
> say... for 2.6.19 ? Or we'll get yet another full release with barely
> working wireless ?

I don't think it's possible to happen for 2.6.19.
Ealiest 2.6.20 but likely one or two releases later (me thinks).

-- 
Greetings Michael.

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Benjamin Herrenschmidt]

> > So what are the chances of getting this dscape stack merged, let's
> > say... for 2.6.19 ? Or we'll get yet another full release with barely
> > working wireless ?
> 
> I don't think it's possible to happen for 2.6.19.
> Ealiest 2.6.20 but likely one or two releases later (me thinks).

Which means we'll still have nothing near decent wireless for what ... 6
monthes min ?

Ben.

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Michael Buesch]

On Sunday 24 September 2006 10:12, Benjamin Herrenschmidt wrote:
> > > So what are the chances of getting this dscape stack merged, let's
> > > say... for 2.6.19 ? Or we'll get yet another full release with barely
> > > working wireless ?
> > 
> > I don't think it's possible to happen for 2.6.19.
> > Ealiest 2.6.20 but likely one or two releases later (me thinks).
> 
> Which means we'll still have nothing near decent wireless for what ... 6
> monthes min ?

Well. Works For Me (tm).
If there is some bug for you in current mainline, it needs to
be fixed. But I can't fix something I am not able to reproduce and
don't know what happens.

-- 
Greetings Michael.

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Benjamin Herrenschmidt]

On Sun, 2006-09-24 at 10:43 +0200, Michael Buesch wrote:
> On Sunday 24 September 2006 10:12, Benjamin Herrenschmidt wrote:
> > > > So what are the chances of getting this dscape stack merged, let's
> > > > say... for 2.6.19 ? Or we'll get yet another full release with barely
> > > > working wireless ?
> > > 
> > > I don't think it's possible to happen for 2.6.19.
> > > Ealiest 2.6.20 but likely one or two releases later (me thinks).
> > 
> > Which means we'll still have nothing near decent wireless for what ... 6
> > monthes min ?
> 
> Well. Works For Me (tm).
> If there is some bug for you in current mainline, it needs to
> be fixed. But I can't fix something I am not able to reproduce and
> don't know what happens.

Well, there is definitely an issue with softmac going nuts, possibly
related to the chip stopping to transmit, I'm not certain at this point.
It -seems- to be better with Larry's latest patch (the big one) so we'll
see. Then, there is a problem when that happens an d your rmmod,
something kicks in after the driver is gone and crashes the kernel... If
I ever get into the situation where I suspect the rmmod will trigger
that again, I'll try to get into a text VT first so I can capture an
oops...

Ben.
 

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Daniel Drake]

Michael Buesch wrote:
> Well. Works For Me (tm).
> If there is some bug for you in current mainline, it needs to
> be fixed. But I can't fix something I am not able to reproduce and
> don't know what happens.

Take a look at the logs in Ben's original mail. I've seen this a lot 
myself and am fairly sure that the problem is softmac's handling of a 
SIWESSID *immediately* followed by a SIWAP call. This is what 
wpa_supplicant does, and the timing screws us over.

You can see in the logs that the driver starts authenticating after the 
first ioctl comes in, but then starts scanning (in preparation for 
authentication, again) as soon as the 2nd call comes in immediately 
after. As the device is busy scanning other channels it misses the 
authentication response, and this goes round in circles.

Now, Jose recently bolted on a few more lock-like flags onto the whole 
auth+assoc procedure, which has certainly helped, but the races do still 
exist, and I don't think that approach is practical: there are simply 
too many points in the sequence where softmac could be 'interrupted' by 
another ESSID/AP call.

But, I'd be absolutely delighted if I'm missing something and you can 
fix it :)

Daniel

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Michael Buesch]

On Sunday 24 September 2006 17:34, Daniel Drake wrote:
> Michael Buesch wrote:
> > Well. Works For Me (tm).
> > If there is some bug for you in current mainline, it needs to
> > be fixed. But I can't fix something I am not able to reproduce and
> > don't know what happens.
> 
> Take a look at the logs in Ben's original mail. I've seen this a lot 
> myself and am fairly sure that the problem is softmac's handling of a 
> SIWESSID *immediately* followed by a SIWAP call. This is what 
> wpa_supplicant does, and the timing screws us over.
> 
> You can see in the logs that the driver starts authenticating after the 
> first ioctl comes in, but then starts scanning (in preparation for 
> authentication, again) as soon as the 2nd call comes in immediately 
> after. As the device is busy scanning other channels it misses the 
> authentication response, and this goes round in circles.
> 
> Now, Jose recently bolted on a few more lock-like flags onto the whole 
> auth+assoc procedure, which has certainly helped, but the races do still 
> exist, and I don't think that approach is practical: there are simply 
> too many points in the sequence where softmac could be 'interrupted' by 
> another ESSID/AP call.
> 
> But, I'd be absolutely delighted if I'm missing something and you can 
> fix it :)

Ok, thanks for the explaination. I will look into the issue.
I remember seeing racing wext calls, too.

-- 
Greetings Michael.

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Johannes Berg]

On Sun, 2006-09-24 at 16:18 +1000, Benjamin Herrenschmidt wrote:

> So what are the chances of getting this dscape stack merged, let's
> say... for 2.6.19 ?

0

>  Or we'll get yet another full release with barely
> working wireless ?

yes. Many locking issues in d80211 to still sort out. Basically, there
are next to no useful locks in it and most data structures are not
protected at all.

johannes

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Benjamin Herrenschmidt]

> yes. Many locking issues in d80211 to still sort out. Basically, there
> are next to no useful locks in it and most data structures are not
> protected at all.

Doh ! Scary... locking is hard ... if the stuff was written without
locking in mind in the first place, it might end up being a nightmare...

Ben.

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Johannes Berg]

On Mon, 2006-09-25 at 19:58 +1000, Benjamin Herrenschmidt wrote:

> Doh ! Scary... locking is hard ... if the stuff was written without
> locking in mind in the first place, it might end up being a nightmare...

Yeah, I've actually suggested "rewriting" it from its own pieces :) But
Jiri says he understands it well enough now to not have to do that.

johannes

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Dan Williams]

On Sun, 2006-09-24 at 12:43 +1000, Benjamin Herrenschmidt wrote:
> Hi folks !
> 
> So this is 2.6.18 + Larry "fix" (though I've seen this problem before,
> it seems using WPA just make it happen more often).
> 
> This is also a 4318, so the link is pretty weak due to the Tx Power
> problem and I suspects it makes the driver problems more visible...
> 
> So basically, I "lose" the link every few minutes for a minute or so, I
> suspect it's related to wpa_supplicant vs. the ack losses due to the
> 4318 Tx Power problems. That alone would be ok though, if the driver
> wasn't totally stuck after a while. (Similar problem to after
> sleep/wakeup, looks like nothign goes through).
> 
> When it goes bunk, it looks like that in the logs:
> 
> Sep 24 12:24:18 localhost kernel: [  285.686826] SoftMAC: Sent Authentication Request to 00:0f:66:52:4b:60.
> Sep 24 12:24:18 localhost kernel: [  285.686976] SoftMAC: generic IE set to <....>
> Sep 24 12:24:18 localhost kernel: [  285.686999] SoftMAC: Already associating or associated to 00:0f:66:52:4b:60
> Sep 24 12:24:28 localhost kernel: [  295.687229] SoftMAC: Start scanning with channel: 1
> Sep 24 12:24:28 localhost kernel: [  295.687240] SoftMAC: Scanning 14 channels
> Sep 24 12:24:29 localhost kernel: [  296.027053] SoftMAC: Scanning finished
> Sep 24 12:24:29 localhost kernel: [  296.035267] SoftMAC: generic IE set to <....>
> Sep 24 12:24:29 localhost kernel: [  296.035310] SoftMAC: Already associating or associated to 00:0f:66:52:4b:60
> Sep 24 12:24:31 localhost kernel: [  297.690969] SoftMAC: Sent Authentication Request to 00:0f:66:52:4b:60.
> Sep 24 12:24:39 localhost kernel: [  306.039210] SoftMAC: Start scanning with channel: 1
> Sep 24 12:24:39 localhost kernel: [  306.039222] SoftMAC: Scanning 14 channels
> Sep 24 12:24:39 localhost kernel: [  306.375046] SoftMAC: Scanning finished
> Sep 24 12:24:39 localhost kernel: [  306.383018] SoftMAC: generic IE set to <....>
> Sep 24 12:24:39 localhost kernel: [  306.383075] SoftMAC: Already associating or associated to 00:0f:66:52:4b:60
> Sep 24 12:24:42 localhost kernel: [  309.695021] SoftMAC: Sent Authentication Request to 00:0f:66:52:4b:60.
> Sep 24 12:24:49 localhost kernel: [  316.387211] SoftMAC: Start scanning with channel: 1
> 
>  etc...
> 
> Then, if you rmmod, you get back a prompt, and about a second later, the kernel blows up. At this point,
> I've always been in X and it's too dead to dump anything into the disk logs so I don't know what
> the precise crash is, but it looks to me like the driver is not properly removing some timer
> or something there.
> 
> Note that it also goes "bunk" on sleep/wakeup, and sometimes ifdown/ifup... in general, it's fragile and
> just 'loses it' in which case the only way to get it back is to rmmod/insmod.
> 
> Doesn't help me to have my prism54 not working with WPA (apparently, the driver looks like it handles hostap
> ioctls but it doesn't agree on the ioctl numbers, among others, with whatever wpa_supplicant sends when
> configured to wpa mode... somebody knows if that driver is maintained ?)

prism54 fullmac, right?

Try using -Dwext; the prism54 wpa_supplicant driver is a dead-end and I
added WE-19 commands to it a bit ago anyway.  Oddly enough, I couldn't
seem to get the driver to work reliably for me using straight WEP
either, let alone WPA.  It's pretty unmaintained at the moment.

Dan

> So at this point I have a choice between two wireless devices that don't work (and none of them is less than
> a couple years old). Looks like the linux wireless situation isn't getting any better since last KS.
> 
> Oh and I don't care about "it works in dscape stack" sort of crap I regulary get. I want something that
> works with upstream kernels. That isn't that much to ask... or is it ?
> 
> Ben, back to ethernet cables.
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Benjamin Herrenschmidt]

> prism54 fullmac, right?

Yes.

> Try using -Dwext; the prism54 wpa_supplicant driver is a dead-end and I
> added WE-19 commands to it a bit ago anyway.  Oddly enough, I couldn't
> seem to get the driver to work reliably for me using straight WEP
> either, let alone WPA.  It's pretty unmaintained at the moment.

Ok, well, I tried with wext but iirc, I got rejected ioctls' from
wpa_supplicant. I'll try again later. Thanks.

Ben.

Re: bcm43xx driver unstable behaviour (and linux wireless is junk btw)

[Dan Williams]

On Sun, 2006-09-24 at 16:13 +1000, Benjamin Herrenschmidt wrote:
> > prism54 fullmac, right?
> 
> Yes.
> 
> > Try using -Dwext; the prism54 wpa_supplicant driver is a dead-end and I
> > added WE-19 commands to it a bit ago anyway.  Oddly enough, I couldn't
> > seem to get the driver to work reliably for me using straight WEP
> > either, let alone WPA.  It's pretty unmaintained at the moment.
> 
> Ok, well, I tried with wext but iirc, I got rejected ioctls' from
> wpa_supplicant. I'll try again later. Thanks.

Ok, the changes appear to be in Linville's wireless-2.6 git tree, but
not yet in the netdev git tree.

Dan