multicasting netlink messages to groups > 31 from userspace

multicasting netlink messages to groups > 31 from userspace

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1098 bytes --]

Hey,

Looking through the code that uses NL_NONROOT_SEND I just realised that
it's impossible to send multicast messages from userspace to multicast
groups with IDs higher than 31. That's not really good given that
everywhere else we handle multicast groups up to 2^32-1 :/

Unfortunately, I haven't found any good way to fix this; in fact the
only way to fix it backward-compatibly I could come up with so far is to
extend struct sockaddr_nl by "__u32 nl_group;" and use that in
netlink_sendmsg instead of nl_groups when msg_namelen is large enough
and I'm not even sure that checking msg_namelen is enough, apps could
give you a length much longer than the actual structure... Maybe
additionally require that nl_groups is set to ~0 or something. The same
should be done for netlink_connect() then.

It would of course be possible to add a new sockopt
NETLINK_{SET,GET}_DST_GROUP, but that prevents you from sending messages
to multiple high groups with sendmsg(), and I think having a working
sendmsg() is desirable as well.

Does anybody have any better ideas?

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: multicasting netlink messages to groups > 31 from userspace

[Evgeniy Polyakov]

On Tue, Jul 03, 2007 at 09:51:25PM +0200, Johannes Berg (johannes@sipsolutions.net) wrote:
> Does anybody have any better ideas?

Hi Johannes.

Why don't you want to reserve a set of bits in group number, which means 
'allow to work with unpriveledged users', that bits should not cross existing 
users (hardcoded in various files in kernel), all new code can use that
groups.
This effectively limits number of groups, but hell, number of them was so
small just a year or two ago and no one complained (although that was a
reason for connector and gennetlink to appear).

I've read a thread about problems limiting 2^32 group space, while no
user has even more than 10...

> johannes



-- 
	Evgeniy Polyakov

Re: multicasting netlink messages to groups > 31 from userspace

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1610 bytes --]

On Wed, 2007-07-04 at 16:04 +0400, Evgeniy Polyakov wrote:
> On Tue, Jul 03, 2007 at 09:51:25PM +0200, Johannes Berg (johannes@sipsolutions.net) wrote:
> > Does anybody have any better ideas?

> Why don't you want to reserve a set of bits in group number, which means 
> 'allow to work with unpriveledged users', that bits should not cross existing 
> users (hardcoded in various files in kernel), all new code can use that
> groups.

Uh wait, we're confusing two issues here. This idea actually makes good
sense, although it'd have to be two bits (allow unpriv users to recv,
allow unpriv users to send), limiting the groups to 2^30-1. Fine with
me.

However, the actual issue I wanted to address with this mail is that we
cannot have userspace send to groups > 32. However, your idea above can
equally well apply here; we could say some bit in the nl_groups field is
reserved and that if that bit is set the other bits form the group
number to send to instead of a bitfield where the highest group is sent
to. That should work fine, both limits the number of available groups to
2^30 respectively 2^31 but that's not an issue.

The thing with actually trying to fix this issue now is that when we
have generic netlink and multicast group registrations, we'll have
userspace programs sending to a multicast group using nl_groups =
(1<<group.id) and then group.id grows > 32 if a whole bunch of other
groups were registered before, and it all messes up. Hence we actually
have to address this issue well before userspace starts using the
generic netlink multicast API.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: multicasting netlink messages to groups > 31 from userspace

[Patrick McHardy]

Johannes Berg wrote:
> Hey,
> 
> Looking through the code that uses NL_NONROOT_SEND I just realised that
> it's impossible to send multicast messages from userspace to multicast
> groups with IDs higher than 31. That's not really good given that
> everywhere else we handle multicast groups up to 2^32-1 :/


Why do you want to send to a multicast group from userspace?

Re: multicasting netlink messages to groups > 31 from userspace

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1379 bytes --]

On Wed, 2007-07-04 at 16:12 +0200, Patrick McHardy wrote:
> Johannes Berg wrote:
> > Hey,
> > 
> > Looking through the code that uses NL_NONROOT_SEND I just realised that
> > it's impossible to send multicast messages from userspace to multicast
> > groups with IDs higher than 31. That's not really good given that
> > everywhere else we handle multicast groups up to 2^32-1 :/
> 
> 
> Why do you want to send to a multicast group from userspace?

Why not, what's wrong with that?

Actually, I think I mentioned this earlier, I was thinking about doing
wireless configuration as a group where both the kernel and possibly a
userspace process listen on that multicast group and processes that want
to configure a device just send to that group. Then the kernel ignores
the message if a userspace process is handling the specific device
completely. For example changing the BSSID: if the kernel is doing MLME
then it changes the BSSID, but if a userspace process is doing it then
the kernel doesn't do anything since BSSID changing is a pure MLME
function, but for consistency it'd be nice if both could be done the
same way, hence a multicast group.

This was actually suggested by Herbert since it's easy to find out if
that multicast group has a listener and not so easy if a special generic
netlink socket in userspace is (still) open.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: multicasting netlink messages to groups > 31 from userspace

[Patrick McHardy]

Johannes Berg wrote:
> On Wed, 2007-07-04 at 16:12 +0200, Patrick McHardy wrote:
> 
>>Johannes Berg wrote:
>>
>>>Hey,
>>>
>>>Looking through the code that uses NL_NONROOT_SEND I just realised that
>>>it's impossible to send multicast messages from userspace to multicast
>>>groups with IDs higher than 31. That's not really good given that
>>>everywhere else we handle multicast groups up to 2^32-1 :/
>>
>>
>>Why do you want to send to a multicast group from userspace?
> 
> 
> Why not, what's wrong with that?


The kernel doesn't have any multicast listeners (yet).

> Actually, I think I mentioned this earlier, I was thinking about doing
> wireless configuration as a group where both the kernel and possibly a
> userspace process listen on that multicast group and processes that want
> to configure a device just send to that group. Then the kernel ignores
> the message if a userspace process is handling the specific device
> completely. For example changing the BSSID: if the kernel is doing MLME
> then it changes the BSSID, but if a userspace process is doing it then
> the kernel doesn't do anything since BSSID changing is a pure MLME
> function, but for consistency it'd be nice if both could be done the
> same way, hence a multicast group.
> 
> This was actually suggested by Herbert since it's easy to find out if
> that multicast group has a listener and not so easy if a special generic
> netlink socket in userspace is (still) open.


I wonder if thats really a good idea to use multicast for device
configuration. Unicast transmissions from userspace to kernel
are reliable when you don't use MSG_DONTWAIT. For multicasts
doing the same would mean blocking on each receiver when the
receive queue is full.

Re: multicasting netlink messages to groups > 31 from userspace

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

On Wed, 2007-07-04 at 16:30 +0200, Patrick McHardy wrote:

[...]
> The kernel doesn't have any multicast listeners (yet).

Right.

> I wonder if thats really a good idea to use multicast for device
> configuration. Unicast transmissions from userspace to kernel
> are reliable when you don't use MSG_DONTWAIT. For multicasts
> doing the same would mean blocking on each receiver when the
> receive queue is full.

That's a good point actually. Are transmissions from userspace to
userspace also reliable in that case?

In fact, we haven't quite decided yet whether we want all configuration
to be via netlink. I strongly prefer this, but other people say that
userspace<->userspace communication is nothing that nl80211 should be
involved in.

What I initially wanted was to publish something like "Device wlan0 is
managed by netlink pid 12345" (or 0 if the kernel is doing it) and then
configuration would be "send to <number the kernel told you>". However,
that got me into problems when trying to determine "is the process
owning netlink pid 12345 still alive" at which point Herbert suggested
the use of multicast groups (and doing the exclusion "only one manager
per netdev" in userspace instead of the kernel).

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: multicasting netlink messages to groups > 31 from userspace

[Patrick McHardy]

Johannes Berg wrote:
> On Wed, 2007-07-04 at 16:30 +0200, Patrick McHardy wrote:
> 
>>I wonder if thats really a good idea to use multicast for device
>>configuration. Unicast transmissions from userspace to kernel
>>are reliable when you don't use MSG_DONTWAIT. For multicasts
>>doing the same would mean blocking on each receiver when the
>>receive queue is full.
> 
> 
> That's a good point actually. Are transmissions from userspace to
> userspace also reliable in that case?


Yes, although in both cases you have no guarantee how long its
going to take, someone else could be flooding the receive queue.
For userspace this is more likely to be a real problem though
since the kernel will keep processing the queue as long as packets
are in it, while userspace could be scheduled away.

> In fact, we haven't quite decided yet whether we want all configuration
> to be via netlink. I strongly prefer this, but other people say that
> userspace<->userspace communication is nothing that nl80211 should be
> involved in.


I'm not so sure myself whether netlink is really a good idea for
userspace<->userspace communication because of the above reason.
IIRC Herbert had the same doubts some time ago, I wonder what
made him change his mind.

> What I initially wanted was to publish something like "Device wlan0 is
> managed by netlink pid 12345" (or 0 if the kernel is doing it) and then
> configuration would be "send to <number the kernel told you>". However,
> that got me into problems when trying to determine "is the process
> owning netlink pid 12345 still alive" at which point Herbert suggested
> the use of multicast groups (and doing the exclusion "only one manager
> per netdev" in userspace instead of the kernel).


There is a notifier for userspace unicast socket releases, would adding
another one for multicast groups help?

Re: multicasting netlink messages to groups > 31 from userspace

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

On Wed, 2007-07-04 at 16:48 +0200, Patrick McHardy wrote:

> Yes, although in both cases you have no guarantee how long its
> going to take, someone else could be flooding the receive queue.
> For userspace this is more likely to be a real problem though
> since the kernel will keep processing the queue as long as packets
> are in it, while userspace could be scheduled away.

Right, but in the case of wireless you'll have different problems if
that happens, namely your wireless card won't be reassociating etc. I
don't think it'll be a problem in practice.

> I'm not so sure myself whether netlink is really a good idea for
> userspace<->userspace communication because of the above reason.
> IIRC Herbert had the same doubts some time ago, I wonder what
> made him change his mind.

Hm. The reason I wanted it initially is that this way we can guarantee
that userspace programs work in either case and also that we have better
control over the various APIs.

> There is a notifier for userspace unicast socket releases, would adding
> another one for multicast groups help?

Huh I think that notifier is enough in fact. It'll be called if a
userspace process closes a socket or such, right? Might get a lot of
events for generic netlink but that should be acceptable since it'd only
need to check .pid to start with.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: multicasting netlink messages to groups > 31 from userspace

[Patrick McHardy]

Johannes Berg wrote:
> On Wed, 2007-07-04 at 16:48 +0200, Patrick McHardy wrote:
> 
> 
>>Yes, although in both cases you have no guarantee how long its
>>going to take, someone else could be flooding the receive queue.
>>For userspace this is more likely to be a real problem though
>>since the kernel will keep processing the queue as long as packets
>>are in it, while userspace could be scheduled away.
> 
> 
> Right, but in the case of wireless you'll have different problems if
> that happens, namely your wireless card won't be reassociating etc. I
> don't think it'll be a problem in practice.


Not by itself probably but a user could DoS your wireless connectivity
this way.

>>I'm not so sure myself whether netlink is really a good idea for
>>userspace<->userspace communication because of the above reason.
>>IIRC Herbert had the same doubts some time ago, I wonder what
>>made him change his mind.
> 
> 
> Hm. The reason I wanted it initially is that this way we can guarantee
> that userspace programs work in either case and also that we have better
> control over the various APIs.
> 
> 
>>There is a notifier for userspace unicast socket releases, would adding
>>another one for multicast groups help?
> 
> 
> Huh I think that notifier is enough in fact. It'll be called if a
> userspace process closes a socket or such, right? Might get a lot of
> events for generic netlink but that should be acceptable since it'd only
> need to check .pid to start with.


I'm not sure, it would probably also have to be called when userspace
unsubscribes from a group, no?

Re: multicasting netlink messages to groups > 31 from userspace

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

On Wed, 2007-07-04 at 17:00 +0200, Patrick McHardy wrote:

> Not by itself probably but a user could DoS your wireless connectivity
> this way.

Hmm, if anything then not the connectivity but rather the MLME i.e. it
won't do roaming properly maybe. Maybe we should then have a way to say
that somebody only wants to receive messages from privileged users, that
way the MLME process wouldn't be affected.

> I'm not sure, it would probably also have to be called when userspace
> unsubscribes from a group, no?

No actually having that hook means I can do without multicast groups by
just publishing that "send configuration requests to pid N" where N is
either 0 (in-kernel MLME) or something else (userspace MLME); I just
didn't know the hook existed.

Not that we shouldn't fix the multicast issues anyway though the one
from this particular thread wouldn't be too interesting right now until
somebody needs userspace to multicast; the other three patches still
should be done now before there are more generic netlink multicast
users.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: multicasting netlink messages to groups > 31 from userspace

[Patrick McHardy]

Johannes Berg wrote:
> On Wed, 2007-07-04 at 17:00 +0200, Patrick McHardy wrote:
> 
> 
>>Not by itself probably but a user could DoS your wireless connectivity
>>this way.
> 
> 
> Hmm, if anything then not the connectivity but rather the MLME i.e. it
> won't do roaming properly maybe. Maybe we should then have a way to say
> that somebody only wants to receive messages from privileged users, that
> way the MLME process wouldn't be affected.


Earlier filtering makes sense, especially for userspace. The other
part exceeds my wireless knowledge :)

>>I'm not sure, it would probably also have to be called when userspace
>>unsubscribes from a group, no?
> 
> 
> No actually having that hook means I can do without multicast groups by
> just publishing that "send configuration requests to pid N" where N is
> either 0 (in-kernel MLME) or something else (userspace MLME); I just
> didn't know the hook existed.


Even better. Publishing that to userspace is going to be racy though.

> Not that we shouldn't fix the multicast issues anyway though the one
> from this particular thread wouldn't be too interesting right now until
> somebody needs userspace to multicast; the other three patches still
> should be done now before there are more generic netlink multicast
> users.


The genetlink stuff defintely makes sense. I'll have a closer look at
your patches this weekend.

Re: multicasting netlink messages to groups > 31 from userspace

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1265 bytes --]

On Thu, 2007-07-05 at 15:37 +0200, Patrick McHardy wrote:

> Earlier filtering makes sense, especially for userspace. The other
> part exceeds my wireless knowledge :)

No worries. I'll see if I can come up with a way to do earlier
filtering, but it's not actually required for the functionality. And the
discussion in the wireless realm still isn't over.

> > No actually having that hook means I can do without multicast groups by
> > just publishing that "send configuration requests to pid N" where N is
> > either 0 (in-kernel MLME) or something else (userspace MLME); I just
> > didn't know the hook existed.
> 
> 
> Even better. Publishing that to userspace is going to be racy though.

I could instead have userspace always send message to 0 and just
redirect them to the userspace process if present; will have to look
into it. That wouldn't be racy at least, but I think this race is going
to be the least of your worries (if the userspace MLME just died then
you have to completely re-do all configuration anyway, and if it just
started the same)

> The genetlink stuff defintely makes sense. I'll have a closer look at
> your patches this weekend.

Thanks. I know Zhang Rui will definitely appreciate that too :)

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: multicasting netlink messages to groups > 31 from userspace

[jamal]

On Fri, 2007-06-07 at 12:53 +0200, Johannes Berg wrote:


> > The genetlink stuff defintely makes sense. I'll have a closer look at
> > your patches this weekend.
> 
> Thanks. I know Zhang Rui will definitely appreciate that too :)

Patrick - please add an ACK from me when you review. The patches
look fine to me.

cheers,
jamal

Re: multicasting netlink messages to groups > 31 from userspace

[jamal]

This email captures the essence of the thread, so let me start here. 
I dont know if i read well enough all the details, but i think i have a
good grasp of the discusion.

To just pick on mentioned issues on the thread which i picked up:
- i think it is fairly usable by netlink to be used as an IPC. The fact
you can do mcast is a huge advantage. i.e i should be able to send to
kernel/user and a mcast group to which a "netlink sniffer" is connected
for debugging etc.
The DoS issue is applicable IMO to any IPC. i.e
if i have access to sending to you, i can send you many messages and
fill up your unix socket rcvq etc.
- On reliability: This is something you will have to deal with yourself
in your application.
For example you could have netlink ACKs sent back to you by all
receivers to the mcast grp (just like the kernel does when you unicast
and ask for acknowledgement).
- In addition to the netlink sock open/close that Patrick mentioned, you
can listen to generic netlink events on the accounting/task netlink and
find out when processes are created/destroyed if that is useful. I
couldnt tell why you needed to know this...

On Tue, 2007-03-07 at 21:51 +0200, Johannes Berg wrote:
> Hey,
> 
> Looking through the code that uses NL_NONROOT_SEND I just realised that
> it's impossible to send multicast messages from userspace to multicast
> groups with IDs higher than 31. That's not really good given that
> everywhere else we handle multicast groups up to 2^32-1 :/

Yes, this is TheWayItHasBeen(tm).
Patrick had a patch a while back to allow for subscribing to higher
groups because bind() has a similar issue in that it allows you only to
subscribe to the first 32. So it is logical to fix connect, and sendmsg
in a similar fashion.

> Unfortunately, I haven't found any good way to fix this; in fact the
> only way to fix it backward-compatibly I could come up with so far is to
> extend struct sockaddr_nl by "__u32 nl_group;" and use that in
> netlink_sendmsg instead of nl_groups when msg_namelen is large enough
> and I'm not even sure that checking msg_namelen is enough, apps could
> give you a length much longer than the actual structure... Maybe
> additionally require that nl_groups is set to ~0 or something. 

We had long chats when Patricks patch went in on this specific issue. I
cant remember the outcome - i remember vaguely that something was going
to break if you muck with sockaddr_nl

> The same
> should be done for netlink_connect() then.

Fixing connect IMO would be a good first start because if you do
connect() to the right group(s) before sendsmg() you could send
to the correct group.

> It would of course be possible to add a new sockopt
> NETLINK_{SET,GET}_DST_GROUP, 

This maybe your best first starting option IMO. It will be in similar
spirit to Patricks bind() fix.

> but that prevents you from sending messages
> to multiple high groups with sendmsg(), 

Iirc, i dont think we can do that anymore. But we can introduce a new
scheme to allow for this. For example, could you not use CMSG to specify
an explicit list of groups on sendmsg? IIRC, Patrick used CMSG to report
on rcv which group a msg was received on. 

cheers,
jamal

Re: multicasting netlink messages to groups > 31 from userspace

[Patrick McHardy]

jamal wrote:
>
>>Looking through the code that uses NL_NONROOT_SEND I just realised that
>>it's impossible to send multicast messages from userspace to multicast
>>groups with IDs higher than 31. That's not really good given that
>>everywhere else we handle multicast groups up to 2^32-1 :/
> 
> 
> Yes, this is TheWayItHasBeen(tm).
> Patrick had a patch a while back to allow for subscribing to higher
> groups because bind() has a similar issue in that it allows you only to
> subscribe to the first 32. So it is logical to fix connect, and sendmsg
> in a similar fashion.


For sendmsg I'd suggest to use cmsgs as in recvmsg().

Re: multicasting netlink messages to groups > 31 from userspace

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

On Thu, 2007-07-05 at 09:53 -0400, jamal wrote:
> This email captures the essence of the thread, so let me start here. 
> I dont know if i read well enough all the details, but i think i have a
> good grasp of the discusion.

:)

> The DoS issue is applicable IMO to any IPC. i.e
> if i have access to sending to you, i can send you many messages and
> fill up your unix socket rcvq etc.

Yeah, good point.

> - In addition to the netlink sock open/close that Patrick mentioned, you
> can listen to generic netlink events on the accounting/task netlink and
> find out when processes are created/destroyed if that is useful. I
> couldnt tell why you needed to know this...

No, I need to know if a specific socket that I was using in the kernel
to communicate with a special userspace process is still open, but the
notifier seems to address that. I haven't tested it yet.

> Yes, this is TheWayItHasBeen(tm).
> Patrick had a patch a while back to allow for subscribing to higher
> groups because bind() has a similar issue in that it allows you only to
> subscribe to the first 32. So it is logical to fix connect, and sendmsg
> in a similar fashion.

Yeah, that's the sock opt.

> Fixing connect IMO would be a good first start because if you do
> connect() to the right group(s) before sendsmg() you could send
> to the correct group.
> 
> > It would of course be possible to add a new sockopt
> > NETLINK_{SET,GET}_DST_GROUP, 
> 
> This maybe your best first starting option IMO. It will be in similar
> spirit to Patricks bind() fix.

Right.

> Iirc, i dont think we can do that anymore. But we can introduce a new
> scheme to allow for this. For example, could you not use CMSG to specify
> an explicit list of groups on sendmsg? IIRC, Patrick used CMSG to report
> on rcv which group a msg was received on. 

I'll have to look into it. Patrick suggested this as well in the reply,
but I think it's not going to be too important to me right now,
depending on the outcome in the wireless discussion.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]