off- & multi-channel operation

off- & multi-channel operation

[Johannes Berg]

All,

I'm thinking about all the off-channel bugs and multi-channel operation.

We currently have at least one rather important bug: Almost every call
to ieee80211_tx_skb()/ieee80211_xmit() is wrong because it assumes we
are on the operating channel. This leads to bugs such as
https://bugzilla.redhat.com/attachment.cgi?id=530264
(https://bugzilla.redhat.com/show_bug.cgi?id=749125) where I'd say we
are sending a frame for connection monitoring while doing some work on
another channel.

The other thing is more of a design issue: we have multiple ways to
leave the operating channel: scanning & work. Soon we'll have to deal
with multi-channel *operation* so we'll have multiple operating
channels. I don't even want to have to handle that in mac80211, and I
think we'll have to push it into devices (or drivers in some cases). In
my opinion, these two ways should be consolidated.

Also, as devices (or drivers) get more complex, we find ourselves
further and further away from a model where mac80211 is truly fully
controlling the device. Given the differences in hardware we support
now, I think that it obvious now that we need drivers to be smarter in
many cases.

How do we want to handle these things? I'm sure I want the drivers to
handle multi-channel operation fairly transparently, with multiple
(hardware) queues (in the driver), so mac80211 doesn't have to
start/stop the queues continuously and can just transmit on that
interface, the frame might only go out a bit later.

But then what about scanning? Can we, and should we, force such drivers
to implement hw_scan? I think maybe we should, and maybe drivers such as
brcmsmac & ath9k can share a common scan implementation they call into?

Authenticating & associating seems fairly easy, we create a new vif and
set it to the right channel and then do everything there -- simpler than
what we have now -- but what about FT-OTA? What if the device supports
only two channels and is a P2P GO on channel 1 and now wants to roam on
the infrastructure connection, say from channel 36 to channel 11? How
can we handle that?

Has anyone else started thinking about this?

johannes

Re: off- & multi-channel operation

[Ben Greear]

On 11/09/2011 05:08 AM, Johannes Berg wrote:
> All,
>
> I'm thinking about all the off-channel bugs and multi-channel operation.
>
> We currently have at least one rather important bug: Almost every call
> to ieee80211_tx_skb()/ieee80211_xmit() is wrong because it assumes we
> are on the operating channel. This leads to bugs such as
> https://bugzilla.redhat.com/attachment.cgi?id=530264
> (https://bugzilla.redhat.com/show_bug.cgi?id=749125) where I'd say we
> are sending a frame for connection monitoring while doing some work on
> another channel.
>
> The other thing is more of a design issue: we have multiple ways to
> leave the operating channel: scanning&  work. Soon we'll have to deal
> with multi-channel *operation* so we'll have multiple operating
> channels. I don't even want to have to handle that in mac80211, and I
> think we'll have to push it into devices (or drivers in some cases). In
> my opinion, these two ways should be consolidated.

Seems like you could move the scanning into the work logic.  That might
be a good first step to simplify the off-channel stuff.  And, once you get
the off-channel stuff working better, it would seem to me that just using
that for your off-channel interfaces might be better than trying to re-write
everything and force a bunch of new functionality down into drivers.

For drivers/NICS that can handle off-channel stuff themselves, hopefully
you can add some new API or new arguments to existing API to help
leverage that.

> Also, as devices (or drivers) get more complex, we find ourselves
> further and further away from a model where mac80211 is truly fully
> controlling the device. Given the differences in hardware we support
> now, I think that it obvious now that we need drivers to be smarter in
> many cases.
>
> How do we want to handle these things? I'm sure I want the drivers to
> handle multi-channel operation fairly transparently, with multiple
> (hardware) queues (in the driver), so mac80211 doesn't have to
> start/stop the queues continuously and can just transmit on that
> interface, the frame might only go out a bit later.

This sounds like a very big change.  I think it will somehow have
to be done in stages, maintaining support for the existing API
for a while so that drivers can be migrated to the new
framework at leisure.

> But then what about scanning? Can we, and should we, force such drivers
> to implement hw_scan? I think maybe we should, and maybe drivers such as
> brcmsmac&  ath9k can share a common scan implementation they call into?

Last thing we need is each driver having it's own complicated off-channel
scan logic.  It's bad enough having it in one place in mac80211.

> Authenticating&  associating seems fairly easy, we create a new vif and
> set it to the right channel and then do everything there -- simpler than
> what we have now -- but what about FT-OTA? What if the device supports
> only two channels and is a P2P GO on channel 1 and now wants to roam on
> the infrastructure connection, say from channel 36 to channel 11? How
> can we handle that?

With just an improved off-channel work logic, and as much smarts as possible
to keep un-necessary channel flops and power-saving changes (and driver resets
in general) from happening?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: off- & multi-channel operation

[Johannes Berg]

On Wed, 2011-11-09 at 09:48 -0800, Ben Greear wrote:

> > The other thing is more of a design issue: we have multiple ways to
> > leave the operating channel: scanning&  work. Soon we'll have to deal
> > with multi-channel *operation* so we'll have multiple operating
> > channels. I don't even want to have to handle that in mac80211, and I
> > think we'll have to push it into devices (or drivers in some cases). In
> > my opinion, these two ways should be consolidated.
> 
> Seems like you could move the scanning into the work logic.  That might
> be a good first step to simplify the off-channel stuff.  And, once you get
> the off-channel stuff working better, it would seem to me that just using
> that for your off-channel interfaces might be better than trying to re-write
> everything and force a bunch of new functionality down into drivers.

Yeah, I had that thought a long time ago, it just never happened :-)

However, it's not really going to work well for multi-channel capable
drivers: say the driver is continually switching between channels 1 and
11 for multi-channel operation, but now you want to do a p2p listen
phase on channel 6. That needs to interact with the driver since it has
to stop switching, and it'd be much easier to have the driver manage
that interaction.

> For drivers/NICS that can handle off-channel stuff themselves, hopefully
> you can add some new API or new arguments to existing API to help
> leverage that.

We have some APIs already, but they aren't actually used for things like
auth/assoc etc. and are optional at this point so we can't start using
them either.

> > Also, as devices (or drivers) get more complex, we find ourselves
> > further and further away from a model where mac80211 is truly fully
> > controlling the device. Given the differences in hardware we support
> > now, I think that it obvious now that we need drivers to be smarter in
> > many cases.
> >
> > How do we want to handle these things? I'm sure I want the drivers to
> > handle multi-channel operation fairly transparently, with multiple
> > (hardware) queues (in the driver), so mac80211 doesn't have to
> > start/stop the queues continuously and can just transmit on that
> > interface, the frame might only go out a bit later.
> 
> This sounds like a very big change.  I think it will somehow have
> to be done in stages, maintaining support for the existing API
> for a while so that drivers can be migrated to the new
> framework at leisure.

I agree, and in fact I think support for devices that can only handle a
single channel (or where nobody can be bothered to write driver support)
needs to be done in a very easy way, possibly completely integrated in
mac80211.

> > But then what about scanning? Can we, and should we, force such drivers
> > to implement hw_scan? I think maybe we should, and maybe drivers such as
> > brcmsmac&  ath9k can share a common scan implementation they call into?
> 
> Last thing we need is each driver having it's own complicated off-channel
> scan logic.  It's bad enough having it in one place in mac80211.

Well, so ... each driver will have to have its own channel switching
logic for implementing multi-channel operation, and off-channel logic
for scanning sorta falls into that. The scan logic could be similar to
now, but I think essentially "remain_on_channel" will have to be
supported by drivers.

> > Authenticating&  associating seems fairly easy, we create a new vif and
> > set it to the right channel and then do everything there -- simpler than
> > what we have now -- but what about FT-OTA? What if the device supports
> > only two channels and is a P2P GO on channel 1 and now wants to roam on
> > the infrastructure connection, say from channel 36 to channel 11? How
> > can we handle that?
> 
> With just an improved off-channel work logic, and as much smarts as possible
> to keep un-necessary channel flops and power-saving changes (and driver resets
> in general) from happening?

Thing is that this doesn't really work too well even today with a lot of
devices we support, e.g. iwlwifi or even iwlegacy isn't really built to
do this. You're mostly looking at it from an ath9k perspective I suppose
where all of this is really quite simple :-)

johannes

Re: off- & multi-channel operation

[Ben Greear]

On 11/09/2011 10:15 AM, Johannes Berg wrote:
> On Wed, 2011-11-09 at 09:48 -0800, Ben Greear wrote:
>
>>> The other thing is more of a design issue: we have multiple ways to
>>> leave the operating channel: scanning&   work. Soon we'll have to deal
>>> with multi-channel *operation* so we'll have multiple operating
>>> channels. I don't even want to have to handle that in mac80211, and I
>>> think we'll have to push it into devices (or drivers in some cases). In
>>> my opinion, these two ways should be consolidated.
>>
>> Seems like you could move the scanning into the work logic.  That might
>> be a good first step to simplify the off-channel stuff.  And, once you get
>> the off-channel stuff working better, it would seem to me that just using
>> that for your off-channel interfaces might be better than trying to re-write
>> everything and force a bunch of new functionality down into drivers.
>
> Yeah, I had that thought a long time ago, it just never happened :-)

Well, now that you've got the state machine back to 'simple', maybe
it will be easier :P

> However, it's not really going to work well for multi-channel capable
> drivers: say the driver is continually switching between channels 1 and
> 11 for multi-channel operation, but now you want to do a p2p listen
> phase on channel 6. That needs to interact with the driver since it has
> to stop switching, and it'd be much easier to have the driver manage
> that interaction.

If mac80211 is controlling the channel switching with off-channel work
items, can't it just keep the nic on channel 6 doing the p2p listen
for the proper amount of time?  Why does the driver have to make
the decision?

If you expect the driver might have pkts queued for multiple channels,
and that is why it would be switching, then you could just have a mac80211
call that says 'stay on this channel (6) until I tell you otherwise'.
The driver could then stop it's automated switching.  That would seem
fairly easy to implement.  Worst case, you flush all buffers and
then it should certainly stop switching?  (The hard part would seem to be making the
driver able to deal with multiple xmit queues and doing automated
switching in the first place.  It certainly never worked well when
ath9k tried to do something similar with virtual phys)

>> For drivers/NICS that can handle off-channel stuff themselves, hopefully
>> you can add some new API or new arguments to existing API to help
>> leverage that.
>
> We have some APIs already, but they aren't actually used for things like
> auth/assoc etc. and are optional at this point so we can't start using
> them either.

You could start using them on hardware/drivers that supports the options, and
keep fallback code for others?

>>> But then what about scanning? Can we, and should we, force such drivers
>>> to implement hw_scan? I think maybe we should, and maybe drivers such as
>>> brcmsmac&   ath9k can share a common scan implementation they call into?
>>
>> Last thing we need is each driver having it's own complicated off-channel
>> scan logic.  It's bad enough having it in one place in mac80211.
>
> Well, so ... each driver will have to have its own channel switching
> logic for implementing multi-channel operation, and off-channel logic
> for scanning sorta falls into that. The scan logic could be similar to
> now, but I think essentially "remain_on_channel" will have to be
> supported by drivers.

I think mac80211 should have a good scan implementation, and just
let drivers without hw-scan use it.  The remain-on-channel could be
part of the work/scan logic in mac80211.  Sounds like you need
remain-on-channel work logic to make p2p work anyway?

>>> Authenticating&   associating seems fairly easy, we create a new vif and
>>> set it to the right channel and then do everything there -- simpler than
>>> what we have now -- but what about FT-OTA? What if the device supports
>>> only two channels and is a P2P GO on channel 1 and now wants to roam on
>>> the infrastructure connection, say from channel 36 to channel 11? How
>>> can we handle that?
>>
>> With just an improved off-channel work logic, and as much smarts as possible
>> to keep un-necessary channel flops and power-saving changes (and driver resets
>> in general) from happening?
>
> Thing is that this doesn't really work too well even today with a lot of
> devices we support, e.g. iwlwifi or even iwlegacy isn't really built to
> do this. You're mostly looking at it from an ath9k perspective I suppose
> where all of this is really quite simple :-)

If you can put the scan state machine into the work logic,
the work logic will get much better testing, and hopefully
that will simplify the on-off channel logic (just having no
scan_channel to worry about would simplify on/off channel
logic that I wrote (and which was just removed).

The only driver I have any idea about is ath9k:  What does iw* do
that makes this difficult?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: off- & multi-channel operation

[Johannes Berg]

On Wed, 2011-11-09 at 11:54 -0800, Ben Greear wrote:

> > However, it's not really going to work well for multi-channel capable
> > drivers: say the driver is continually switching between channels 1 and
> > 11 for multi-channel operation, but now you want to do a p2p listen
> > phase on channel 6. That needs to interact with the driver since it has
> > to stop switching, and it'd be much easier to have the driver manage
> > that interaction.
> 
> If mac80211 is controlling the channel switching with off-channel work
> items, can't it just keep the nic on channel 6 doing the p2p listen
> for the proper amount of time?  Why does the driver have to make
> the decision?

mac80211 isn't going to be controlling the switching between 1 and 11 at
all though. And there will be drivers like iwlwifi and wl12xx that want
to control the "stay on channel 6" as well.

> If you expect the driver might have pkts queued for multiple channels,
> and that is why it would be switching, then you could just have a mac80211
> call that says 'stay on this channel (6) until I tell you otherwise'.
> The driver could then stop it's automated switching.  That would seem
> fairly easy to implement.

Yes, that's what we'll probably need. Actually we already have that in a
way with the remain_on_channel, but it's not used for work/scan/etc.
today, only for p2p.

> Worst case, you flush all buffers and
> then it should certainly stop switching?  (The hard part would seem to be making the
> driver able to deal with multiple xmit queues and doing automated
> switching in the first place.  It certainly never worked well when
> ath9k tried to do something similar with virtual phys)

At least for iwlwifi I expect we'd have multiple hardware queues with
the uCode handling the switching.

> > Thing is that this doesn't really work too well even today with a lot of
> > devices we support, e.g. iwlwifi or even iwlegacy isn't really built to
> > do this. You're mostly looking at it from an ath9k perspective I suppose
> > where all of this is really quite simple :-)
> 
> If you can put the scan state machine into the work logic,
> the work logic will get much better testing, and hopefully
> that will simplify the on-off channel logic (just having no
> scan_channel to worry about would simplify on/off channel
> logic that I wrote (and which was just removed).
> 
> The only driver I have any idea about is ath9k:  What does iw* do
> that makes this difficult?

Uh, that would be a really really long explanation so I think for now
you'll just have to take my word for it. It's just the way the firmware
is designed.

johannes

Re: off- & multi-channel operation

[Johannes Berg]

On Wed, 2011-11-09 at 14:08 +0100, Johannes Berg wrote:

> How do we want to handle these things? I'm sure I want the drivers to
> handle multi-channel operation fairly transparently, with multiple
> (hardware) queues (in the driver), so mac80211 doesn't have to
> start/stop the queues continuously and can just transmit on that
> interface, the frame might only go out a bit later.

Hmm, off-channel TDLS is also an interesting case ...

Maybe we need to have a channel for each station? And get timing info
from the device somehow for all the negotiation? Obviously stations
connected to an AP would all have the same channel etc...

johannes

Re: off- & multi-channel operation

[Stanislaw Gruszka]

On Wed, Nov 09, 2011 at 02:08:54PM +0100, Johannes Berg wrote:
> How do we want to handle these things? I'm sure I want the drivers to
> handle multi-channel operation fairly transparently, with multiple
> (hardware) queues (in the driver), so mac80211 doesn't have to
> start/stop the queues continuously and can just transmit on that
> interface, the frame might only go out a bit later.

I do not quite understand how separate tx queues could help here, because
we not only have to transmit off-channel but also receive data. For me
simple solution would be adding new functions on top of
ieee80211_tx_skb, which will just queue skb's if we are not on operating
channel at present, and send them lately when we come back.

Or perhaps you are talking about separate tx and rx queues, that could
eventually work, but driver still should know how long to stay off-channel
to receive all needed frames. 

Stanislaw

Re: off- & multi-channel operation

[Johannes Berg]

On Fri, 2011-11-11 at 14:16 +0100, Stanislaw Gruszka wrote:
> On Wed, Nov 09, 2011 at 02:08:54PM +0100, Johannes Berg wrote:
> > How do we want to handle these things? I'm sure I want the drivers to
> > handle multi-channel operation fairly transparently, with multiple
> > (hardware) queues (in the driver), so mac80211 doesn't have to
> > start/stop the queues continuously and can just transmit on that
> > interface, the frame might only go out a bit later.
> 
> I do not quite understand how separate tx queues could help here, because
> we not only have to transmit off-channel but also receive data. For me
> simple solution would be adding new functions on top of
> ieee80211_tx_skb, which will just queue skb's if we are not on operating
> channel at present, and send them lately when we come back.
> 
> Or perhaps you are talking about separate tx and rx queues, that could
> eventually work, but driver still should know how long to stay off-channel
> to receive all needed frames. 

Well, so usually you'd schedule NoA, or use null data frames to not
receive, etc. Using multiple TX queues would help schedule the TX,
scheduling the RX is a different issue.

johannes

Re: off- & multi-channel operation

[Ben Greear]

On 11/11/2011 02:13 AM, Johannes Berg wrote:
> On Wed, 2011-11-09 at 11:54 -0800, Ben Greear wrote:
>
>>> However, it's not really going to work well for multi-channel capable
>>> drivers: say the driver is continually switching between channels 1 and
>>> 11 for multi-channel operation, but now you want to do a p2p listen
>>> phase on channel 6. That needs to interact with the driver since it has
>>> to stop switching, and it'd be much easier to have the driver manage
>>> that interaction.
>>
>> If mac80211 is controlling the channel switching with off-channel work
>> items, can't it just keep the nic on channel 6 doing the p2p listen
>> for the proper amount of time?  Why does the driver have to make
>> the decision?
>
> mac80211 isn't going to be controlling the switching between 1 and 11 at
> all though. And there will be drivers like iwlwifi and wl12xx that want
> to control the "stay on channel 6" as well.

But at least for ath9k, it certainly *could* do this.  If other NICs
cannot handle this for whatever reason, then they can be handled some
other way.  It would seem to me that mac80211 is going to have a better
picture on what needs doing than the driver (it can see software queues,
all network-devices, the upper network stack if needed, etc).

>> If you expect the driver might have pkts queued for multiple channels,
>> and that is why it would be switching, then you could just have a mac80211
>> call that says 'stay on this channel (6) until I tell you otherwise'.
>> The driver could then stop it's automated switching.  That would seem
>> fairly easy to implement.
>
> Yes, that's what we'll probably need. Actually we already have that in a
> way with the remain_on_channel, but it's not used for work/scan/etc.
> today, only for p2p.

So, I'm suggesting we put this functionality into the work logic.  It's
already doing half-assed on/off channel stuff, as is scan, and it seems
as p2p or other off-channel stations needs as well.  Lets put it all in one
place, and then optimize it to work well.  We could batch work-items for
the same channel together, for instance, and handle the on/off channel
switching so that the stations get enough time to do useful work on their
channel.

>> Worst case, you flush all buffers and
>> then it should certainly stop switching?  (The hard part would seem to be making the
>> driver able to deal with multiple xmit queues and doing automated
>> switching in the first place.  It certainly never worked well when
>> ath9k tried to do something similar with virtual phys)
>
> At least for iwlwifi I expect we'd have multiple hardware queues with
> the uCode handling the switching.

So if the NIC can do everything, then maybe send work-items to it and let
it do the scheduling?  Like:  'spend max of 200ms on this channel and send/rcv as
many pkts as possible', or 'scan this channel for 20ms', or 'do-full-hw-scan'.
If the NIC wants to hop channels on it's own, the off-channel work items could
be considered suggestions and let the NIC schedule how it wants.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: off- & multi-channel operation

[Johannes Berg]

On Fri, 2011-11-11 at 09:54 -0800, Ben Greear wrote:

> > mac80211 isn't going to be controlling the switching between 1 and 11 at
> > all though. And there will be drivers like iwlwifi and wl12xx that want
> > to control the "stay on channel 6" as well.
> 
> But at least for ath9k, it certainly *could* do this.  If other NICs
> cannot handle this for whatever reason, then they can be handled some
> other way.  It would seem to me that mac80211 is going to have a better
> picture on what needs doing than the driver (it can see software queues,
> all network-devices, the upper network stack if needed, etc).

It could, but it won't be efficient. mac80211 mostly does everything in
workqueues, and this is rather timing sensitive so you want to do it
closer to the hardware, possibly even in the interrupt handler that
tells you to switch (you could program the timers in the hardware for
that)

> >> If you expect the driver might have pkts queued for multiple channels,
> >> and that is why it would be switching, then you could just have a mac80211
> >> call that says 'stay on this channel (6) until I tell you otherwise'.
> >> The driver could then stop it's automated switching.  That would seem
> >> fairly easy to implement.
> >
> > Yes, that's what we'll probably need. Actually we already have that in a
> > way with the remain_on_channel, but it's not used for work/scan/etc.
> > today, only for p2p.
> 
> So, I'm suggesting we put this functionality into the work logic.  It's
> already doing half-assed on/off channel stuff, as is scan, and it seems
> as p2p or other off-channel stations needs as well.  Lets put it all in one
> place, and then optimize it to work well.  We could batch work-items for
> the same channel together, for instance, and handle the on/off channel
> switching so that the stations get enough time to do useful work on their
> channel.

That works mostly, but not for actual operation on multiple channels.

> >> Worst case, you flush all buffers and
> >> then it should certainly stop switching?  (The hard part would seem to be making the
> >> driver able to deal with multiple xmit queues and doing automated
> >> switching in the first place.  It certainly never worked well when
> >> ath9k tried to do something similar with virtual phys)
> >
> > At least for iwlwifi I expect we'd have multiple hardware queues with
> > the uCode handling the switching.
> 
> So if the NIC can do everything, then maybe send work-items to it and let
> it do the scheduling?  Like:  'spend max of 200ms on this channel and send/rcv as
> many pkts as possible', or 'scan this channel for 20ms', or 'do-full-hw-scan'.
> If the NIC wants to hop channels on it's own, the off-channel work items could
> be considered suggestions and let the NIC schedule how it wants.

That's kinda how it works now, but not all NICs can even handle that.
Think wl1251 or iwlegacy for example, it'll never support all of these
features of course but it also doesn't deal well with the channel
switching work items we have today.

johannes

Re: off- & multi-channel operation

[Adrian Chadd]

I'm looking at this in FreeBSD's net80211, primarily to be able to
bring sanity to tx/rx aggregation handling so things like active, high
throughput traffic doesn't get annoyed when off-channel mode occurs.

So when you decide to start scanning, there may be frames already in
the TX queue. If you simply called "start scanning now!", hardware
like say ath9k would have to do this (and i haven't looked to see if
it does, I bet it does. :-)

* the TX queue would have to be paused in its entirety, so the higher
layers both stop sending frames, _and_ any existing call to
if_transmit/if_start (and the linux equivalents) have entirely
completed;
* that doesn't guarantee the hardware-queued TXes have completed, so
you either have to wait for those to complete before you start
off-channel operation, or ..
* .. you tell the driver "i'm about to start scanning, please get ready";
* where the driver (eg ath9k) would then stop RX/TX DMA, with frames
still in the RX completion queue and TX pending/completion state;
* .. which have to be handled at some point;
* .. since the hardware may hvae multiple TX queues, but you typically
use the same for on-channel and off-channel, you have to then handle
those frames then, OR set them aside and handle them once you go
on-channel again;
* then you do off-channel work, whilst all non-on-channel mgmt frame
TX is done (although god knows how I'm going to make sure that
off-channel mgmt frames (eg probes) go out, but on-channel mgmt frames
(eg BAR) are properly paused :) ;
* once the scan is completed, handle whatever off-channel stuff is
pending in RX, cancel (or handle?) TX to the off-channel stuff;
* .. then go back to on-channel,
* .. then restart TX/RX where you left off, including handle any and
all pending frames.

FreeBSD's net80211 has some logic already to inform the driver about
off-channel scanning (ie, ic scan_start and scan_end methods) along
with a way for firmware devices to take charge of scanning and replace
the net80211 default software scanning mode. The trick is what I've
covered above - making sure that absolutely, positively all the cases
are handled and you're not TX'ing on the wrong channel or dropping
frames which cause aggregation session hangs.

You can't just stuff it into a separate hardware TX queue right now
with ath9k unless you start populating ath9k with lots of special
queue "flavour" handling. You still have to cleanly stop/start DMA,
handle frame completion for ampdu session continuity (or send BAR
frames as needed), etc.

I do think you need the stack to be able to do the frame buffering;
the driver could exert a little more control so it's not forced to
simply drop all the frames on the floor, but be given time to complete
what it's doing before scanning begins.


Adrian

Re: off- & multi-channel operation

[Johannes Berg]

On Sat, 2011-11-12 at 14:46 -0800, Adrian Chadd wrote:
> I'm looking at this in FreeBSD's net80211, primarily to be able to
> bring sanity to tx/rx aggregation handling so things like active, high
> throughput traffic doesn't get annoyed when off-channel mode occurs.
> 
> So when you decide to start scanning, there may be frames already in
> the TX queue. If you simply called "start scanning now!", hardware
> like say ath9k would have to do this (and i haven't looked to see if
> it does, I bet it does. :-)
> 
> * the TX queue would have to be paused in its entirety, so the higher
> layers both stop sending frames, _and_ any existing call to
> if_transmit/if_start (and the linux equivalents) have entirely
> completed;
> * that doesn't guarantee the hardware-queued TXes have completed, so
> you either have to wait for those to complete before you start
> off-channel operation, or ..
> * .. you tell the driver "i'm about to start scanning, please get ready";
> * where the driver (eg ath9k) would then stop RX/TX DMA, with frames
> still in the RX completion queue and TX pending/completion state;

I have no idea what ath9k does, but we do tell it about start/stop
scanning and flush queues explicitly (the latter may not be the best
thing to do with multi-channel)

> * .. which have to be handled at some point;
> * .. since the hardware may hvae multiple TX queues, but you typically
> use the same for on-channel and off-channel, you have to then handle
> those frames then, OR set them aside and handle them once you go
> on-channel again;

I'm thinking that since the hardware has enough queues, you might use a
set per channel to switch more quickly, but that's up to the driver
implementation :-)

[snip]

> I do think you need the stack to be able to do the frame buffering;
> the driver could exert a little more control so it's not forced to
> simply drop all the frames on the floor, but be given time to complete
> what it's doing before scanning begins.

Yes, well, the driver can always start & stop queues, and for software
scan we do already stop queues (not for hardware scan for obvious
reasons).

johannes

Re: off- & multi-channel operation

[Johannes Berg]

So I spent a bit more time thinking about this, and am growing more and
more unhappy with the current state of mac80211. Why? For example the
fact that we've been adding workarounds left & right for a while, just
two recent examples:
* my tx_sync
* dummy AP station

At the same time, we haven't really defined the operation sequences we
have (or want to have) and a lot of the operations are getting hard to
understand, e.g. bss_info_changed() with all its different parameters.
Take
https://github.com/TI-OpenLink/wl12xx/commit/ed280a5d3b3141a6de8836993e851469cb4a8982 for example, we never even defined in what order stations should be added and thus never checked it either.

Similarly, devices are starting to differ more and more. It's not longer
the clear separation between full-MAC (cfg80211 driver) and soft-MAC
(mac80211 driver) -- even "soft-MAC" devices are acquiring features like
rate control (wl12xx, mwl8k, ath9k_htc, iwlwifi somewhat), probe
response offload (wl12xx) and various other offloads. (And realistically
that separation was never really quite as black & white)

I'm slowly coming to the conclusion that we need to give drivers more
structured information and move further away mac80211 fully controlling
all operations. Especially as we go into multi-channel use, this becomes
virtually impossible. Some of what I'm saying/thinking now is obviously
influenced by our (Intel's) hardware/firmware design, but I'm trying not
to get too entrenched in that.


One of the major points for me is roaming -- our older devices aren't
really able to deal with authenticating before breaking the connection,
and this is requiring a lot of workarounds in the drivers and I think it
also causes a few bugs. OTOH, wpa_supplicant expects to be able to auth
before it disconnects and we explicitly maintain multiple
authentications in cfg80211 -- maybe the latter was a mistake (yes,
mine, but I guess I have to learn)?

With our newer devices we could send the authentication in a
remain-on-channel phase, and I think for TI the situation is similar,
but with older devices the simple channel switching to auth with a new
AP and then switching back is definitely a source of some issues, e.g.
not draining aggregation queues properly since aggregation state is lost
in the device when we do this. Yes -- this can probably be worked around
in the driver, but is it worth it?

The biggest issue here is that we need wpa_s to do the authentication
handshake, especially with things like FT (OTA) or SAE.

I'm unsure how to handle this -- thoughts anyone?

johannes 

Re: off- & multi-channel operation

[Johannes Berg]

Another thought that crossed my mind is how to handle off-channel TDLS.
I'm sure it'll become a requirement fairly soon, and I think we'll want
to design it into the stack.

I wish I could gather a few people into a room and sit there for a few
days brainstorming how it should look like (and how to implement
backward compatibility!) .... :-)

Anyway so my thought about off-channel TDLS was -- should we maybe
assign a channel to each station? Obviously each station associated to
an AP would have the same channel. That might make channel switching
(CSA) a little interesting, but hey ...

In our device we'll probably not handle it this way, but it seems like
the best abstraction?

Also I think we should probably have some explicit channel handling. I'm
not thinking about scanning/off-channel operations but about actual use
of the channel. There could be some weight requests for the different
channels depending on airtime needs, but ultimately the driver will
probably have to make the timing decisions and will communicate that up
to make negotiation between devices possible for this. NoA might be
handled in the driver right now, but I think TDLS negotiation for
channels would not be, and there you need timings...

johannes

Re: off- & multi-channel operation

[Adrian Chadd]

I'm sorry I haven't responded to this. I'm busy moving country :-)

Let's talk about this in a couple weeks. I know that FreeBSD's
net80211 code allows drivers to only override what methods they
require net80211 to provide. So drivers can do their own MLME, for
example, but still throw some control frame handling at
net80211/hostapd. Or use the net80211 rate control API whilst doing
mostly everything else in firmware (eg if_iwn.)

I'm also looking at how to handle off-channel operation when doing
active 802.11n (and not, but software aggregation does tickle things)
traffic. Flushing the actively queued traffic whilst doing off-channel
work is likely sub-optimal. I have no idea if that's what mac80211
does.

Anyway, I'll be back doing active hacking on this stuff in two or
three weeks. I bet we could save time by solving the same problem
together rather than solving it individually and differently.

(At that stage I may pick your brain about how P2P is supposed to
integrate correctly with off-channel operation..)


Adrian