wireless notes / pre d80211 merge

wireless notes / pre d80211 merge

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 12703 bytes --]

0. Introduction

Hi,

As promised on IRC, here are some thoughts on what d80211 is currently
doing and what should in my opinion be changed. 

I'm writing this now mostly because I think that we have userspace
visible issues to sort out *before* we can land in -mm or even mainline
with d80211.

This discussion is split up as follows:

 (1) master netdev
 (2) skb path during rx
 (3) skb path during tx
 (4) "wiphy" concept
 (5) so it works, what's wrong?
 (6) d80211 as a protocol?
 (7) d80211 without master netdev?
 (8) what about a combination, or other solutions?
 (9) conclusion


(1) through (4) are just a recap of what we currently have to illustrate
some things for later. Also serves as a starting point for those not as
familiar. Could of course contain errors, I hope not too many.

(5) discusses what I think is wrong, while (6) through (8) discuss
possible scenarios I came up with to fix it.

Note that all these things are long-term. Probably longer than we can
wait with a merge, however we should settle on how the userspace
interface will look like before the merge.


1. master netdev

Currently, we have the 'master' netdev wmasterN which is created as
native 802.11 device but is essentially useless. It is exported to
userspace but only supports wireless extensions and, depending on what
the drivers do, ethtool ops. It isn't really useful for network
functionality although outgoing frames can be seen on it. See section
(3) for why.

Internally, the master netdev is also a virtual access point mode
device, but this is only relevant in the RX path I think.

Additionally, the master netdev represents the underlying 802.11
hardware when it comes to qdisc manipulation.



2. skb path during rx

When a frame is received, it is put into an skb by the lowlevel driver
and handed to d80211 by way of __ieee80211_rx() [1]. They then travel
through the whole d80211 (pre) rx handlers and finally show up as 802.3
frames in the appropriate virtual device. Note that these frames are
never associated with the master netdev, hence won't show up when you
run tcpdump or similar on it.

The whole rx handler thing could (and should!) be cleaned up quite a lot
and can be made more efficient too, but it is purely internal right now
so not a big deal. Food for another d80211 note.

[1] why we didn't make a static inline ieee80211_rx() that calls an
exported __ieee80211_rx() until we get rid of the other ieee80211 is
beyond me. Would have been good but I guess we can also just convert all
the drivers when we change the name again.


3. skb path during tx

This is more complicated. When a frame is created in one of the virtual
interfaces, it first goes through through conversion from 802.3 into
802.11, some tx control is added on and the frame is queued to the
master netdev. This is why we see outgoing frames in tcpdump.

After getting queued to the master netdev, the frame goes through the
qdiscs and some more info is tacked on into skb->cb by the 802.11 qdisc.
Afterwards, if the frame is not dropped, it shows up in
ieee80211_master_start_xmit where skb->cb is copied onto the stack and
cleared afterwards. Then, the frame goes through all the tx handlers
including fragmentation and encryption and is finally given to the
lowlevel driver via the hardware description's tx() call.



4. "wiphy" concept

Straying a bit from the discussion of frames and similar, let me
describe the "wiphy" concept we currently have. Currently, in sysfs we
have class/ieee80211/phy%d/ and below that a wealth of information not
only about various hardware related things but also, for example, a list
of all stations associated to any virtual access points intermixed with
those 'stations' that we are associated to on any virtual interface.
Also, we here find 'add_iface' and 'remove_iface', knobs to create and
destroy virtual interfaces.

The second wiphy concept is currently present in cfg80211 which
currently requires this concept in the userspace interface and should
effectively replace the 'add_iface' and 'remove_iface' hooks and make
them more generically available for non-d80211 drivers. I was thinking
of ipw2200 when I wrote it which allows adding a monitor device
(currently through some config option I think).



5. so it works, what's wrong?

Pretty broad question I asked myself here, but let me try to answer it
anyway.

For one, I think that having the master device in its current form is
useless. It sees outgoing frames that are to be sent to the hardware for
transmission, and in that way represents the actual underlying hardware.
However, it never sees incoming frames, so there again it doesn't
represent the hardware. Not seeing incoming frames makes it useless, and
even through it sees outgoing frames you cannot send frames to it.

Also, if the master netdev is thought of as representing the hardware
(which I don't think it fully does), it collides with the notion of the
'wiphy' as described above.

The master netdev and wiphy create two orthogonal interfaces to
userspace that nonetheless pretty much represent the same thing---the
underlying hardware.

As said previously, the master netdev represents the hardware when it
comes to qdisc manipulation, but the wiphy represents the hardware when
it comes to manipulation of virtual interfaces. I believe that this is
fundamentally wrong because they both manipulate operation of the
underlying hardware.

I know I actually strengthened the wiphy concept by adding it to
cfg80211, but to my defense at that point I was still pretty sure we
would be getting rid of the master device in the future. This seems to
have been naive, but see below.

This is my biggest problem with d80211 at the moment. All the internal
inefficiencies can be cleaned up without any externally visible impact.


6. d80211 as a protocol?

Some proof-of-concept patches I posted converted d80211 to a protocol.
They contained technical flaws and aren't usable at all, but do
illustrate the concept. What I did was

 - instead of having drivers create a wireless device specially, they
   simply register a netdev with 802.11 arphrd and assign a new net_dev
   member ieee80211_hw describing the hardware functionality and
   containing function pointers for example for changing channels. This
   can be thought of as similar to hardware vlan acceleration
 - d80211 hooks into the netdev notifier chain and creates appropriate
   state for itself when a native 802.11 device with hw description is
   registered

RX path:
 - drivers, instead of calling __ieee80211_rx, put the RX information
   into skb->cb (which is shared between drivers and d80211 in this
   scenario) and then netif_rx() this skb which gets a native 802.11
   type.
 - d80211 picks up this skb through a packet_type that it registered and
   injects it into its processing, ending up in some virtual device or
   scan results etc.

TX path:
 - an skb that is injected into some virtual device which is controlled
   by d80211 is transformed into whatever state it needs and subjected
   to the various tx handlers. Transmission control information is
   embedded into skb->cb for the device driver, and the frame is
   dev_queue_xmit()ed to the hardware device (former master device).
 - the SKB is subjected to qdiscs etc.
 - drivers receive this skb and transmit it to the hardware device
 - because wireless requires transmit status, when the hardware is done
   with the skb drivers don't simply free it but instead hand it to
   ieee80211_tx_status() which updates state as necessary and then frees
   the skb. Status information is given as a second parameter to that
   function along with the skb as currently.

This seems quite elegant, but there are a couple of problems with it:
 - skb->cb is sort of abused, and might be too small in the future. If
   it becomes too small, we could put a pointer into skb->cb instead and
   do lifetime management with an skb desctructor.
 - because of the transmit status requirement, skbs need to be 'given
   back' to d80211


Problems that have solutions:
 - the qdiscs can drop fragments of a frame which is useless, hence
   fragmentation (and consequently also encryption) should probably be
   moved to the qdisc, need to make sure that the 802.11 qdisc is
   attached first on the device and can't be removed.

Further comments I have received include the fact that again, the master
netdev is not very useful because there is no application that can live
with sending raw 802.11 frames without controlling transmission
parameters like transmission rate etc. [2]

From a programming point of view, this is also quite a logical
separation because drivers get to fully control the netdev they
register, and the generic 802.11 code mostly is not concerned with it
(except for the qdisc issue).

Also something that has been proposed is to add the transmission control
information as a header to a newly invented protocol. This seems nice at
first, but if d80211 were to be a protocol then it would also mean that
d80211 would have to serialise the internal transmission control
structure and the driver deserialise it again. Also, adding new items to
the structure becomes impossible (unless the parser we require is
complicated...)

Also, in this scenario the ieee80211 class would go away, probably in
favour of
 (1) exporting most of the things via cfg80211
 (2) putting sta information structs into sysfs where they belong --
     with the virtual device they are associated with
 (3) adding other hw information in sysfs as attributes to the driver's
     netdev

There's probably more, and rx path cleanup is one of the things this
would require. Comments welcome :)

[2] I actually have one such application internally using my previous
nl80211 work but really not filling in any of the transmit information.



7. d80211 without master netdev?

This is another possibility that I explored but got quite some headwind
from a few people :)

Effectively, this would empower the wiphy concept more. As described
previously, the master netdev is currently mostly just used for the
qdisc it can have associated with it, and thus used from userspace for
qdisc manipulation. In order to remove the master netdev completely it
would be necessary to abstract the skb queueing out from netdevs and
create some new structure responsible for it which would be embedded in
both a netdev and a wiphy structure. Then, the qdisc userspace interface
would have to be able to identify both the containers (netdev via
ifindex, wiphy probably by wiphyidx) and operate accordingly.

This actually requires some major surgery so I don't expect it to be
done before a merge. But if this should be the desired outcome, then I
propose to cut userspace breakage to a minimum: only the tc operations.
This means that the master netdev should be removed as far as possible
and the only thing left on it should be the qdisc interface. This means
drivers should no longer be able to access the netdev. [3]

I'm not sure this is completely feasible. Thinking about LLTX more it
would probably require moving that to the new queue structure, which
means that either netdev registration would have to copy it there or all
regular network drivers be modified.

[3] It was said that some drivers set LLTX in the flags. I'm not sure
this is safe because it doesn't just affect the driver but also d80211
which might not be able to cope with it.



8. what about a combination, or other solutions?

Of course combinations are possible. One thing that immediately comes to
mind is just migrating over all of the current wiphy functionality over
to the master netdev in order to unify the two userspace interfaces.
This would be visible in sysfs as well as require changes to
cfg80211[4].

Of course, the master netdev would still not see incoming frames because
they are handled by __ieee80211_rx().

I guess many more combinations are possible :)


[4] actually, in cfg80211 we'd just remove the wiphy attribute and the
change the internals a bit, it should already accept the master dev in
most cases


9. conclusion

Of course, this just represents my own opinion. I think that although
the skb->cb is sort of abused, a protocol makes most sense.

I'd like to see this resolved with a sane userspace API that doesn't
consist of multiple disjoint things representing the same hardware.

I hope it isn't all too incoherent :)

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: wireless notes / pre d80211 merge

[Jiri Benc]

On Tue, 14 Nov 2006 23:19:57 +0100, Johannes Berg wrote:
> 1. master netdev
> 
> Currently, we have the 'master' netdev wmasterN which is created as
> native 802.11 device but is essentially useless. It is exported to
> userspace but only supports wireless extensions and, depending on what
> the drivers do, ethtool ops. It isn't really useful for network
> functionality although outgoing frames can be seen on it. See section
> (3) for why.

I wouldn't say "useless". It's actually a hack (and you yourself described
in (7) why it is currently necessary). We'll need to live with it or
cripple the stack to support only very basic features or rewrite the Linux
networking core. Choose your own favorite solution :-)

> [1] why we didn't make a static inline ieee80211_rx() that calls an
> exported __ieee80211_rx() until we get rid of the other ieee80211 is
> beyond me. Would have been good but I guess we can also just convert all
> the drivers when we change the name again.

Yes, it's just temporary and it doesn't matter.

> 3. skb path during tx
> 
> This is more complicated. When a frame is created in one of the virtual
> interfaces, it first goes through through conversion from 802.3 into
> 802.11, some tx control is added on and the frame is queued to the
> master netdev. This is why we see outgoing frames in tcpdump.
> 
> After getting queued to the master netdev, the frame goes through the
> qdiscs and some more info is tacked on into skb->cb by the 802.11 qdisc.
> Afterwards, if the frame is not dropped, it shows up in
> ieee80211_master_start_xmit where skb->cb is copied onto the stack and
> cleared afterwards. Then, the frame goes through all the tx handlers
> including fragmentation and encryption and is finally given to the
> lowlevel driver via the hardware description's tx() call.

During the "tx handlers" phase that "copy of skb->cb" is extended quite a
lot with other information determined in the tx handlers. The result is a
structure that doesn't fit into cb anymore. This is a reason for not using
cb to pass tx information to drivers.

> 4. "wiphy" concept
> 
> Straying a bit from the discussion of frames and similar, let me
> describe the "wiphy" concept we currently have. Currently, in sysfs we
> have class/ieee80211/phy%d/ and below that a wealth of information not
> only about various hardware related things but also, for example, a list
> of all stations associated to any virtual access points intermixed with
> those 'stations' that we are associated to on any virtual interface.
> Also, we here find 'add_iface' and 'remove_iface', knobs to create and
> destroy virtual interfaces.
> 
> The second wiphy concept is currently present in cfg80211 which
> currently requires this concept in the userspace interface and should
> effectively replace the 'add_iface' and 'remove_iface' hooks and make
> them more generically available for non-d80211 drivers. I was thinking
> of ipw2200 when I wrote it which allows adding a monitor device
> (currently through some config option I think).

Not a big problem to combine these two "wiphys" into just one.

> 5. so it works, what's wrong?
> 
> Pretty broad question I asked myself here, but let me try to answer it
> anyway.
> 
> For one, I think that having the master device in its current form is
> useless. It sees outgoing frames that are to be sent to the hardware for
> transmission, and in that way represents the actual underlying hardware.
> However, it never sees incoming frames, so there again it doesn't
> represent the hardware. Not seeing incoming frames makes it useless, and
> even through it sees outgoing frames you cannot send frames to it.
> 
> Also, if the master netdev is thought of as representing the hardware
> (which I don't think it fully does), it collides with the notion of the
> 'wiphy' as described above.

Please note that master interface has been always considered as a hack. Of
course, this perception can be changed if it's desirable.

Personally, I tried to make this interface uninteresting for users as much
as possible - so we can tell ordinary users: "That's just an annoying
internal thing, don't touch it and don't care about it." Of course,
advanced users need to care (because of qdiscs). This is not optimal
(although I'm not sure there exists an optimal solution here) and I'm not
strongly attached to this.

> The master netdev and wiphy create two orthogonal interfaces to
> userspace that nonetheless pretty much represent the same thing---the
> underlying hardware.

Yup.

> As said previously, the master netdev represents the hardware when it
> comes to qdisc manipulation, but the wiphy represents the hardware when
> it comes to manipulation of virtual interfaces. I believe that this is
> fundamentally wrong because they both manipulate operation of the
> underlying hardware.

There is one thing you haven't mentioned and which was probably the key
reason for not believing this concept was wrong: we want to make the
transition to the new stack as smooth to the users as possible. Users are
not used to use the master interface. But since we're going to have a new
set of userspace tools anyway, this point is no longer valid.

> I know I actually strengthened the wiphy concept by adding it to
> cfg80211, but to my defense at that point I was still pretty sure we
> would be getting rid of the master device in the future. This seems to
> have been naive, but see below.
> 
> This is my biggest problem with d80211 at the moment. All the internal
> inefficiencies can be cleaned up without any externally visible impact.

If this is your biggest problem with d80211 you're really happy :-) Try to
fix some locking issue ;-)

> 6. d80211 as a protocol?
> 
> Some proof-of-concept patches I posted converted d80211 to a protocol.

A full conversion to 802.11 protocol should do much more things and is not
possible until we rewrite bridging. You converted the master interface
only. That's not what I would call converting d80211 to a 802.11 protocol.

> This seems quite elegant, but there are a couple of problems with it:
>  - skb->cb is sort of abused, and might be too small in the future. If

This is a blocker issue for me.

>    it becomes too small, we could put a pointer into skb->cb instead and
>    do lifetime management with an skb desctructor.

Not very effective but will work.

>  - because of the transmit status requirement, skbs need to be 'given
>    back' to d80211
> 
> Problems that have solutions:
>  - the qdiscs can drop fragments of a frame which is useless, hence
>    fragmentation (and consequently also encryption) should probably be
>    moved to the qdisc, need to make sure that the 802.11 qdisc is
>    attached first on the device and can't be removed.

Just for clarification: The problem here is that you need to ensure
fragments are not reordered. This mean you cannot do fragmentation (and
hence encryption) in virtual interfaces, i.e. before you enqueue the frame
into the master interface. Because 802.11 qdisc is the last one seeing the
frame before it reaches a hardware, we can move fragmentation and
encryption there. Actually, we have to do that if we want to convert master
interface to the 802.11 protocol.

> Also something that has been proposed is to add the transmission control
> information as a header to a newly invented protocol. This seems nice at
> first, but if d80211 were to be a protocol then it would also mean that
> d80211 would have to serialise the internal transmission control
> structure and the driver deserialise it again. Also, adding new items to
> the structure becomes impossible (unless the parser we require is
> complicated...)

Doesn't sound like a good solution to me.

> 9. conclusion

As I said when discussing your patch for passing ieee80211_hw to drivers,
I don't much care. I see pros and cons for both ways (the current one with
master interface perceived as a necessary evil, perhaps even strengthened
by passing ieee80211_hw to drivers, or the proposed one with drivers caring
of master netdev). If you feel strongly biased towards the second one, feel
free to send patches (but please not just one big patch this time :-)).

Thanks,

 Jiri

-- 
Jiri Benc
SUSE Labs

RE: wireless notes / pre d80211 merge

[Simon Barber]

Hi Jiri,

I disagree that the master device is a hack - I also disagree that we
should use 802.11 format frames anywhere but internally inside the
802.11 stack. The 802.11 specification does not use 802.11 format frames
to communicate with the upper layers - it uses almost exactly the same
interface as 802.3 ethernet does. This is a 2 address interface. I
believe that using the standard 2 address format to talk with upper
layers matches well with the spec and is the right way to do things. If
you put 802.11 frame format directly into the L3 protocols then you make
all these protocols depend on the MAC - since they will now implement
part of the MAC inside the L3 protocol. I don't think this is the right
thing to do. I believe that the mapping of 3 and 4 address frames to a
particular virtual 2 address port is the right way to do this mapping.

It is normal to have a single interface to represent the physical
hardware network interface. This is exactly the pattern that many other
physical network devices use. Things like sniffing should go on on this
physical interface - and the changes that johannes proposes are exactly
the right things to enable this.

I would go further - and perhaps move some of the meta-data that is in
the skb->cb into a d80211 specific hardware header. This would allow
sniffers to directly attach to the master device and both send and
receive frames complete with the meta data. It would also reduce the
amount of cb space we need. Virtual devices could be created in order to
use sniffers that use a different hardware header format (with a small
performance penalty when using those).

Simon
 

-----Original Message-----
From: Jiri Benc [mailto:jbenc@suse.cz] 
Sent: Tuesday, November 14, 2006 6:11 PM
To: Johannes Berg
Cc: netdev; Jeff Garzik; John W. Linville; Simon Barber; Michael Buesch;
Ivo van Doorn; Michael Wu; Jouni Malinen; Daniel Drake; Hong Liu; Luis
R. Rodriguez; James Ketrenos; David Kimdon; Udayan Singh
Subject: Re: wireless notes / pre d80211 merge

On Tue, 14 Nov 2006 23:19:57 +0100, Johannes Berg wrote:
> 1. master netdev
> 
> Currently, we have the 'master' netdev wmasterN which is created as 
> native 802.11 device but is essentially useless. It is exported to 
> userspace but only supports wireless extensions and, depending on what

> the drivers do, ethtool ops. It isn't really useful for network 
> functionality although outgoing frames can be seen on it. See section
> (3) for why.

I wouldn't say "useless". It's actually a hack (and you yourself
described in (7) why it is currently necessary). We'll need to live with
it or cripple the stack to support only very basic features or rewrite
the Linux networking core. Choose your own favorite solution :-)

> [1] why we didn't make a static inline ieee80211_rx() that calls an 
> exported __ieee80211_rx() until we get rid of the other ieee80211 is 
> beyond me. Would have been good but I guess we can also just convert 
> all the drivers when we change the name again.

Yes, it's just temporary and it doesn't matter.

> 3. skb path during tx
> 
> This is more complicated. When a frame is created in one of the 
> virtual interfaces, it first goes through through conversion from 
> 802.3 into 802.11, some tx control is added on and the frame is queued

> to the master netdev. This is why we see outgoing frames in tcpdump.
> 
> After getting queued to the master netdev, the frame goes through the 
> qdiscs and some more info is tacked on into skb->cb by the 802.11
qdisc.
> Afterwards, if the frame is not dropped, it shows up in 
> ieee80211_master_start_xmit where skb->cb is copied onto the stack and

> cleared afterwards. Then, the frame goes through all the tx handlers 
> including fragmentation and encryption and is finally given to the 
> lowlevel driver via the hardware description's tx() call.

During the "tx handlers" phase that "copy of skb->cb" is extended quite
a lot with other information determined in the tx handlers. The result
is a structure that doesn't fit into cb anymore. This is a reason for
not using cb to pass tx information to drivers.

> 4. "wiphy" concept
> 
> Straying a bit from the discussion of frames and similar, let me 
> describe the "wiphy" concept we currently have. Currently, in sysfs we

> have class/ieee80211/phy%d/ and below that a wealth of information not

> only about various hardware related things but also, for example, a 
> list of all stations associated to any virtual access points 
> intermixed with those 'stations' that we are associated to on any
virtual interface.
> Also, we here find 'add_iface' and 'remove_iface', knobs to create and

> destroy virtual interfaces.
> 
> The second wiphy concept is currently present in cfg80211 which 
> currently requires this concept in the userspace interface and should 
> effectively replace the 'add_iface' and 'remove_iface' hooks and make 
> them more generically available for non-d80211 drivers. I was thinking

> of ipw2200 when I wrote it which allows adding a monitor device 
> (currently through some config option I think).

Not a big problem to combine these two "wiphys" into just one.

> 5. so it works, what's wrong?
> 
> Pretty broad question I asked myself here, but let me try to answer it

> anyway.
> 
> For one, I think that having the master device in its current form is 
> useless. It sees outgoing frames that are to be sent to the hardware 
> for transmission, and in that way represents the actual underlying
hardware.
> However, it never sees incoming frames, so there again it doesn't 
> represent the hardware. Not seeing incoming frames makes it useless, 
> and even through it sees outgoing frames you cannot send frames to it.
> 
> Also, if the master netdev is thought of as representing the hardware 
> (which I don't think it fully does), it collides with the notion of 
> the 'wiphy' as described above.

Please note that master interface has been always considered as a hack.
Of course, this perception can be changed if it's desirable.

Personally, I tried to make this interface uninteresting for users as
much as possible - so we can tell ordinary users: "That's just an
annoying internal thing, don't touch it and don't care about it." Of
course, advanced users need to care (because of qdiscs). This is not
optimal (although I'm not sure there exists an optimal solution here)
and I'm not strongly attached to this.

> The master netdev and wiphy create two orthogonal interfaces to 
> userspace that nonetheless pretty much represent the same thing---the 
> underlying hardware.

Yup.

> As said previously, the master netdev represents the hardware when it 
> comes to qdisc manipulation, but the wiphy represents the hardware 
> when it comes to manipulation of virtual interfaces. I believe that 
> this is fundamentally wrong because they both manipulate operation of 
> the underlying hardware.

There is one thing you haven't mentioned and which was probably the key
reason for not believing this concept was wrong: we want to make the
transition to the new stack as smooth to the users as possible. Users
are not used to use the master interface. But since we're going to have
a new set of userspace tools anyway, this point is no longer valid.

> I know I actually strengthened the wiphy concept by adding it to 
> cfg80211, but to my defense at that point I was still pretty sure we 
> would be getting rid of the master device in the future. This seems to

> have been naive, but see below.
> 
> This is my biggest problem with d80211 at the moment. All the internal

> inefficiencies can be cleaned up without any externally visible
impact.

If this is your biggest problem with d80211 you're really happy :-) Try
to fix some locking issue ;-)

> 6. d80211 as a protocol?
> 
> Some proof-of-concept patches I posted converted d80211 to a protocol.

A full conversion to 802.11 protocol should do much more things and is
not possible until we rewrite bridging. You converted the master
interface only. That's not what I would call converting d80211 to a
802.11 protocol.

> This seems quite elegant, but there are a couple of problems with it:
>  - skb->cb is sort of abused, and might be too small in the future. If

This is a blocker issue for me.

>    it becomes too small, we could put a pointer into skb->cb instead
and
>    do lifetime management with an skb desctructor.

Not very effective but will work.

>  - because of the transmit status requirement, skbs need to be 'given
>    back' to d80211
> 
> Problems that have solutions:
>  - the qdiscs can drop fragments of a frame which is useless, hence
>    fragmentation (and consequently also encryption) should probably be
>    moved to the qdisc, need to make sure that the 802.11 qdisc is
>    attached first on the device and can't be removed.

Just for clarification: The problem here is that you need to ensure
fragments are not reordered. This mean you cannot do fragmentation (and
hence encryption) in virtual interfaces, i.e. before you enqueue the
frame into the master interface. Because 802.11 qdisc is the last one
seeing the frame before it reaches a hardware, we can move fragmentation
and encryption there. Actually, we have to do that if we want to convert
master interface to the 802.11 protocol.

> Also something that has been proposed is to add the transmission 
> control information as a header to a newly invented protocol. This 
> seems nice at first, but if d80211 were to be a protocol then it would

> also mean that
> d80211 would have to serialise the internal transmission control 
> structure and the driver deserialise it again. Also, adding new items 
> to the structure becomes impossible (unless the parser we require is
> complicated...)

Doesn't sound like a good solution to me.

> 9. conclusion

As I said when discussing your patch for passing ieee80211_hw to
drivers, I don't much care. I see pros and cons for both ways (the
current one with master interface perceived as a necessary evil, perhaps
even strengthened by passing ieee80211_hw to drivers, or the proposed
one with drivers caring of master netdev). If you feel strongly biased
towards the second one, feel free to send patches (but please not just
one big patch this time :-)).

Thanks,

 Jiri

--
Jiri Benc
SUSE Labs

Re: wireless notes / pre d80211 merge

[Jiri Benc]

On Tue, 14 Nov 2006 18:10:50 -0800, Simon Barber wrote:
> I disagree that the master device is a hack

It's a clever hack but still a hack. The Linux network interfaces
wasn't designed for the sort of things we do with master interface.
Actually, the networking core doesn't support the concept of "wiphys"
at all. Yes, Jouni found a really clever way around this. But the most
correct solution is different.

In particular:
- The operations that make sense for user to perform on wiphy/master
  interface are different than the ones on virtual interfaces. It isn't
  this way now.
- Why should master interface appear in ifconfig and similar tools the
  same way as "virtual" interfaces? If you want to know how users are
  confused by this, just search some forums.
- Does attaching of qdiscs to virtual interfaces make sense?
- Etc.

To me, the master interface in its current shape is a nice workaround
of kernel limitations, but still not a clean solution.

> I also disagree that we
> should use 802.11 format frames anywhere but internally inside the
> 802.11 stack. The 802.11 specification does not use 802.11 format frames
> to communicate with the upper layers - it uses almost exactly the same
> interface as 802.3 ethernet does. This is a 2 address interface. I
> believe that using the standard 2 address format to talk with upper
> layers matches well with the spec and is the right way to do things.

This is a different thing. See the difference between SOCK_RAW and
SOCK_DGRAM. The interface you are talking about is SOCK_DGRAM. Except
it supports only one address. The correct solution would be extending
sockaddr_ll (that's not possible, I know) or introduce a new
SOCK_SOMETHING type (not possible either, at least in a short term).

> If
> you put 802.11 frame format directly into the L3 protocols then you make
> all these protocols depend on the MAC - since they will now implement
> part of the MAC inside the L3 protocol. I don't think this is the right
> thing to do. I believe that the mapping of 3 and 4 address frames to a
> particular virtual 2 address port is the right way to do this mapping.

For some kind of applications SOCK_RAW on 802.11 interfaces makes
sense. (Actually, that's not so simple as they need to specify other
things like tx rate but that's an another story.)

> It is normal to have a single interface to represent the physical
> hardware network interface.

Except that all other hardware doesn't need things that we need. They
need just one virtual interface - so their master and virtual interface
is merged. This is net_device. We need to split it into two things.

> This is exactly the pattern that many other
> physical network devices use. Things like sniffing should go on on this
> physical interface - and the changes that johannes proposes are exactly
> the right things to enable this.
> 
> I would go further - and perhaps move some of the meta-data that is in
> the skb->cb into a d80211 specific hardware header. This would allow
> sniffers to directly attach to the master device and both send and
> receive frames complete with the meta data. It would also reduce the
> amount of cb space we need. Virtual devices could be created in order to
> use sniffers that use a different hardware header format (with a small
> performance penalty when using those).

This seems as a really bad solution to me. I fully agree with Johannes
regarding this.

Thanks,

 Jiri

-- 
Jiri Benc
SUSE Labs

Re: wireless notes / pre d80211 merge

[Johannes Berg]

On Wed, 2006-11-15 at 10:43 +0100, Jiri Benc wrote:

> In particular:
> - The operations that make sense for user to perform on wiphy/master
>   interface are different than the ones on virtual interfaces. It isn't
>   this way now.

Which particular operations do you have in mind? Granted, you won't ever
be able to specify the master device as the default route for IP, but
hey. I'd almost bet there are other devices that you can't do this with.

> - Why should master interface appear in ifconfig and similar tools the
>   same way as "virtual" interfaces? If you want to know how users are
>   confused by this, just search some forums.

Because it manages the qdiscs and frames actually show up there. Even
now, you can there see what you're transmitting :)

> - Does attaching of qdiscs to virtual interfaces make sense?

Yes, I do think so. Why not?

> This is a different thing. See the difference between SOCK_RAW and
> SOCK_DGRAM. The interface you are talking about is SOCK_DGRAM. Except
> it supports only one address. The correct solution would be extending
> sockaddr_ll (that's not possible, I know) or introduce a new
> SOCK_SOMETHING type (not possible either, at least in a short term).

I still haven't quite understood this argument. Could you elaborate?

> Except that all other hardware doesn't need things that we need. They
> need just one virtual interface - so their master and virtual interface
> is merged. This is net_device. We need to split it into two things.

That's not really true, if you have a 802.1q network then the raw netdev
is useless without vlan virtuals above unless you unpack the vlan on the
switch giving you a non-802.1q network again.

johannes

Re: wireless notes / pre d80211 merge

[Jeff Garzik]

Jiri Benc wrote:
> On Tue, 14 Nov 2006 18:10:50 -0800, Simon Barber wrote:
>> I disagree that the master device is a hack
> 
> It's a clever hack but still a hack. The Linux network interfaces
> wasn't designed for the sort of things we do with master interface.
> Actually, the networking core doesn't support the concept of "wiphys"
> at all. Yes, Jouni found a really clever way around this. But the most
> correct solution is different.

Agreed.

	Jeff

RE: wireless notes / pre d80211 merge

[Johannes Berg]

> I would go further - and perhaps move some of the meta-data that is in
> the skb->cb into a d80211 specific hardware header. This would allow
> sniffers to directly attach to the master device and both send and
> receive frames complete with the meta data. It would also reduce the
> amount of cb space we need. Virtual devices could be created in order to
> use sniffers that use a different hardware header format (with a small
> performance penalty when using those).

I'm not sure this is a good idea, it locks us in to some format because
user-space uses it and we'd need to serialise to that format to be
extensible.

On the other hand, we could fix some part of the structure to be
user-space visible, make it a real struct that just lives before the
802.11 header and don't serialise like it would be necessary with
something that is extensible. On RX, we could do the same.

For TX, this would allow applications to control whatever we put into
that struct (I would not, for example, allow tx status notifications to
ever go over a netdev as we do now), while for RX it would allow apps to
see almost everything.

It might complicate code a bit because different things would come from
different places (some things from a special header and some from
skb->cb), but I don't think it would be bad.

Other than this, I agree with you.

johannes

Re: wireless notes / pre d80211 merge

[Jeff Garzik]

Simon Barber wrote:
> I disagree that the master device is a hack - I also disagree that we
> should use 802.11 format frames anywhere but internally inside the
> 802.11 stack. The 802.11 specification does not use 802.11 format frames
> to communicate with the upper layers - it uses almost exactly the same
> interface as 802.3 ethernet does. This is a 2 address interface. I

The 802.11 specification doesn't communicate with any layer :)  The 
Linux kernel code in question does that.

In Linux it is generally bad form to emulate something (802.3 ethernet, 
in this case).  Overhead increases, performance decreases, and there are 
inevitably emulation bugs.

802.11 /is not ethernet/, and you are already breaking assumptions by 
pretending that it is.


> It is normal to have a single interface to represent the physical
> hardware network interface. This is exactly the pattern that many other
> physical network devices use. Things like sniffing should go on on this
> physical interface - and the changes that johannes proposes are exactly
> the right things to enable this.

Sure... when it behaves like a normal interface.  Stuff like "TX goes 
through the interface, but not RX" is definitely /not/ normal.

	Jeff

Re: wireless notes / pre d80211 merge

[Johannes Berg]

On Wed, 2006-11-15 at 01:11 +0100, Jiri Benc wrote:

> I wouldn't say "useless". It's actually a hack (and you yourself described
> in (7) why it is currently necessary). We'll need to live with it or
> cripple the stack to support only very basic features or rewrite the Linux
> networking core. Choose your own favorite solution :-)

I'd be fine with living with what you think of as a "hack" for a while
if the medium- to long-term plan is to rewrite the networking core to
get rid of it. I'm just not sure it makes sense.

> During the "tx handlers" phase that "copy of skb->cb" is extended quite a
> lot with other information determined in the tx handlers. The result is a
> structure that doesn't fit into cb anymore. This is a reason for not using
> cb to pass tx information to drivers.

Actually, no, it isn't :) The part that is passed to drivers does fit,
only the internal stuff doesn't.

> > Straying a bit from the discussion of frames and similar, let me
> > describe the "wiphy" concept we currently have. Currently, in sysfs we
> > have class/ieee80211/phy%d/ and below that a wealth of information not
> > only about various hardware related things but also, for example, a list
> > of all stations associated to any virtual access points intermixed with
> > those 'stations' that we are associated to on any virtual interface.
> > Also, we here find 'add_iface' and 'remove_iface', knobs to create and
> > destroy virtual interfaces.
> > 
> > The second wiphy concept is currently present in cfg80211 which
> > currently requires this concept in the userspace interface and should
> > effectively replace the 'add_iface' and 'remove_iface' hooks and make
> > them more generically available for non-d80211 drivers. I was thinking
> > of ipw2200 when I wrote it which allows adding a monitor device
> > (currently through some config option I think).
> 
> Not a big problem to combine these two "wiphys" into just one.

Right, that's no biggie. I was just trying to describe what we currently
have an forgot to mention it.

> Please note that master interface has been always considered as a hack. Of
> course, this perception can be changed if it's desirable.

Has it? Simon seems to disagree ;)

> > The master netdev and wiphy create two orthogonal interfaces to
> > userspace that nonetheless pretty much represent the same thing---the
> > underlying hardware.
> 
> Yup.

You just say it like that. Aren't you unhappy about that at all?

> There is one thing you haven't mentioned and which was probably the key
> reason for not believing this concept was wrong: we want to make the
> transition to the new stack as smooth to the users as possible. Users are
> not used to use the master interface. But since we're going to have a new
> set of userspace tools anyway, this point is no longer valid.

No, wait! We're transitioning from something that only supports one
interface to something that supports many virtual ones. As long as we
can represent the semantics of the previous single one as the default
virtual one, we're fine. Which is actually very possible and cfg80211's
WE compat code tries to show that.

I don't think the master interface matters here at all. It's just an
additional netdev that users previously didn't have, but so what?

> If this is your biggest problem with d80211 you're really happy :-) Try to
> fix some locking issue ;-)

Let me qualify to externally visible problems :)
I'm not too concerned about the internal stuff. That can be cleaned up
with some effort, but the externally visible stuff can't actually just
be cleaned up without thought and a plan.

> A full conversion to 802.11 protocol should do much more things and is not
> possible until we rewrite bridging. You converted the master interface
> only. That's not what I would call converting d80211 to a 802.11 protocol.

I've said this before but let me repeat.

There are two ends to d80211

top:    virtual interfaces, most likely 802.3 for at least a while if
        not forever
bottom: communication with hardware, qdiscs, etc.

I've never wanted to change the top part (so far anyway :) ). It's
orthogonal. And whether d80211 behaves as an 802.11 protocol is
something that's visible only in the RX path really, struct packet_type
and associated things.

> > This seems quite elegant, but there are a couple of problems with it:
> >  - skb->cb is sort of abused, and might be too small in the future. If
> 
> This is a blocker issue for me.
> 
> >    it becomes too small, we could put a pointer into skb->cb instead and
> >    do lifetime management with an skb desctructor.
> 
> Not very effective but will work.

Not really less effective than copying it around all the time as we do
now and then kmalloc()ing it in some circumstances.

johannes

Re: wireless notes / pre d80211 merge

[Jiri Benc]

On Wed, 15 Nov 2006 10:16:18 +0100, Johannes Berg wrote:
> > During the "tx handlers" phase that "copy of skb->cb" is extended quite a
> > lot with other information determined in the tx handlers. The result is a
> > structure that doesn't fit into cb anymore. This is a reason for not using
> > cb to pass tx information to drivers.
> 
> Actually, no, it isn't :) The part that is passed to drivers does fit,
> only the internal stuff doesn't.

Sorry, I was wrong here. I rechecked it now and starting from the last
diet (made by me long time ago, grr) it fits.

> > Please note that master interface has been always considered as a hack. Of
> > course, this perception can be changed if it's desirable.
> 
> Has it? Simon seems to disagree ;)

And I disagree with Simon :-)

> > > The master netdev and wiphy create two orthogonal interfaces to
> > > userspace that nonetheless pretty much represent the same thing---the
> > > underlying hardware.
> > 
> > Yup.
> 
> You just say it like that. Aren't you unhappy about that at all?

I wasn't in times when there was no cfg80211. But now... yes, you're
right. The sysfs ieee80211 class exports just some interesting
informations (we already agreed that add_interface and remove_interface
would go away). Feel free to move it wherever you think is appropriate.

> I don't think the master interface matters here at all. It's just an
> additional netdev that users previously didn't have, but so what?

As I said, my pre-cfg80211 arguments are not valid now.

> Let me qualify to externally visible problems :)

Okay :-)

> I've never wanted to change the top part (so far anyway :) ). It's
> orthogonal. And whether d80211 behaves as an 802.11 protocol is
> something that's visible only in the RX path really, struct packet_type
> and associated things.

I just said it's a confusing term, nothing against the idea.

> > >    it becomes too small, we could put a pointer into skb->cb instead and
> > >    do lifetime management with an skb desctructor.
> > 
> > Not very effective but will work.
> 
> Not really less effective than copying it around all the time as we do
> now and then kmalloc()ing it in some circumstances.

That's true.

Thanks,

 Jiri

-- 
Jiri Benc
SUSE Labs

Re: wireless notes / pre d80211 merge

[Johannes Berg]

On Wed, 2006-11-15 at 11:05 +0100, Jiri Benc wrote:

> I wasn't in times when there was no cfg80211. But now... yes, you're
> right. The sysfs ieee80211 class exports just some interesting
> informations (we already agreed that add_interface and remove_interface
> would go away). Feel free to move it wherever you think is appropriate.

Even with cfg80211 we have the wiphy concept embedded in there. Not much
difference because we still have to do tc on wmasterN and "wiconf" on
wiphyN. So that doesn't really change anything. Now, we could of course
drop the wiphy concept from cfg80211 easily and require doing all these
things on wmasterN. That's the option I considered in section (8). I
don't like it too much, but I think I could live with it.

johannes