multicast, interfaces, kernel 3.0+...

multicast, interfaces, kernel 3.0+...

[Michael Tokarev]

Hello.

We found some, well, interesting behavour of kernels
3.0 and later, while 2.6.32 (previous long-stable
series) worked fine.  I'm not sure when it "broke",
since this is a production machine and we've difficult
time diagnosing it, and the app causing it is, well,
large.

The short story.  A big java app uses multicast group
to register one component and find it later.

The machine in question has 3 active network interfaces:
usual lo, eth0, and virtual (tap, pointopoint) tinc.
Tinc interface is marked as "multicast off".

When the app starts on 2.6.32 kernel, netstat -g shows
that multicast group on 2 interfaces: lo and eth0, but
not on tinc, which is sort of expected:

$ netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              4      228.5.6.7
lo              1      all-systems.mcast.net
eth0            4      228.5.6.7
eth0            1      all-systems.mcast.net
tinc            1      all-systems.mcast.net


But when the same app (actually the same userspace) is
booted on the same machine but on 3.0+ kernel, the same
multicast group is registered also on 2 interfaces, but
this time these are lo (as before) and tinc, but not eth0:

$ netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              4      228.5.6.7
lo              1      all-systems.mcast.net
eth0            1      all-systems.mcast.net
tinc            4      228.5.6.7
tinc            1      all-systems.mcast.net

Now, on 3.0+ kernel, parts of this app can't find each
other.  The "client" tries to send a datagram packet
to this address, 228.5.6.7, but receives no reply.

On 2.6.32 kernel, when eth0 is used instead of tinc,
it all works as expected.

Now, my knowlege of this multicast stuff is very limited
(reading about it now), so I don't really know what it
all means.  At least the fact that it somehow registers
tinc (which is multicast-off!) is already somewhat strange.
I tried removing this multicast setting from this iface,
but that didn't help.  I also tried enabling multicast on
lo (which was disabled!) and disabling it on others, but
that didn't help either.

According to strace, the app does not try to change iface
group membership, it does bind of a udp socket to 0.0.0.0:port,
and uses SOL_IP, IP_ADD_MEMBERSHIP to add this socket to a
multicast group.

Note: there's just ONE machine involved, and two applications
running on it.

Why with 3.0+, the non-multicast "tinc" interface is shown
as a member of 228.5.6.7 group, but not eth0 which actually
*is* multicast?

For the record, this "big java app" is Oracle reports server.
I've no idea why they use multicast to find two components
of one thing running on the same machine, and does not provide
any usable unicast solution...

Thanks!

/mjt

Re: multicast, interfaces, kernel 3.0+...

[Michael Tokarev]

On 21.09.2012 22:46, Michael Tokarev wrote:
> Hello.
> 
> We found some, well, interesting behavour of kernels
> 3.0 and later, while 2.6.32 (previous long-stable
> series) worked fine.  I'm not sure when it "broke",
> since this is a production machine and we've difficult
> time diagnosing it, and the app causing it is, well,
> large.
> 
> The short story.  A big java app uses multicast group
> to register one component and find it later.
> 
> The machine in question has 3 active network interfaces:
> usual lo, eth0, and virtual (tap, pointopoint) tinc.
> Tinc interface is marked as "multicast off".
> 
> When the app starts on 2.6.32 kernel, netstat -g shows
> that multicast group on 2 interfaces: lo and eth0, but
> not on tinc, which is sort of expected:
> 
> $ netstat -g
> IPv6/IPv4 Group Memberships
> Interface       RefCnt Group
> --------------- ------ ---------------------
> lo              4      228.5.6.7
> lo              1      all-systems.mcast.net
> eth0            4      228.5.6.7
> eth0            1      all-systems.mcast.net
> tinc            1      all-systems.mcast.net
> 
> 
> But when the same app (actually the same userspace) is
> booted on the same machine but on 3.0+ kernel, the same
> multicast group is registered also on 2 interfaces, but
> this time these are lo (as before) and tinc, but not eth0:
> 
> $ netstat -g
> IPv6/IPv4 Group Memberships
> Interface       RefCnt Group
> --------------- ------ ---------------------
> lo              4      228.5.6.7
> lo              1      all-systems.mcast.net
> eth0            1      all-systems.mcast.net
> tinc            4      228.5.6.7
> tinc            1      all-systems.mcast.net
> 
> Now, on 3.0+ kernel, parts of this app can't find each
> other.  The "client" tries to send a datagram packet
> to this address, 228.5.6.7, but receives no reply.
> 
> On 2.6.32 kernel, when eth0 is used instead of tinc,
> it all works as expected.

Now this is interesting, questionable, and is a change
in behavour, albiet, well, again, questionable ;)

I looked at straces, and found this.

The app looks at all interfaces on the host, and for
each interface found, it calls IP_ADD_MEMBERSHIP.

But.

On this machine, for years, we had the same address on
eth0 and on tinc interfaces (that's long story).

Now, the difference in behavour between 3.0+ and 2.6.32
is that for this one IP address, corresponding IP_ADD_MEMBERSHIP
call on one kernel adds one iface to the group, while on
another kernel it is another iface.  That's the whole
difference.

Why I said it is a "questionable question".  The IP_ADD_MEMBERSHIP
interface is apparently misdefined, because it accepts an
IP address of an interface, instead of an ifindex, or
ifname, or something like this, since there's no, obviously,
1:1 correspondence between ifaces and addresses, an iface
can have no addresses assotiated with it, or two ifaces can
share one IP address like in my case.  But the "questionable"
part is the "usualness" of this setup I have here, with two
ifaces having the same IP address.

I've no idea why the app does this thing to start with,
why it can't use wildcard address with IP_ADD_MEMBERSHIP,
or why it messes with that stuff at all.  It is a different
question.

So, should IP_ADD_MEMBERSHIP use some more iface-centric
interface, instead of relying on IP addresses?  And why
3.0+ changed order here?

Thanks!

/mjt

Re: multicast, interfaces, kernel 3.0+...

[David Miller]

From: Michael Tokarev <mjt@tls.msk.ru>
Date: Sat, 22 Sep 2012 08:21:52 +0400

> The IP_ADD_MEMBERSHIP interface is apparently misdefined, because it
> accepts an IP address of an interface, instead of an ifindex, or
> ifname, or something like this, since there's no, obviously, 1:1
> correspondence between ifaces and addresses, an iface can have no
> addresses assotiated with it, or two ifaces can share one IP address
> like in my case.  But the "questionable" part is the "usualness" of
> this setup I have here, with two ifaces having the same IP address.

Can you at least look at the API specification for IP_ADD_MEMBERSHIP
before making such claims?

The IP_ADD_MEMBERSHIP interface allows for the specification of a
specific interface, the structure you pass into IP_ADD_MEMBERSHIP has
an ->imr_ifindex field and this is the first key the call uses
to pick a device.

If you do not specify an explicit ifindex, and leave it at zero which
I bet your application is doing, it picks the first address which has
the specified address.

As you have discovered, just specifying the address can cause unwanted
effects when multiple devices have the same IP address.  Because the
order of network devices in the system is never, and has never, been
guaranteed.

So the selection in this situation is essentially random because
you haven't given the kernel enough information to choose things
the way that you want it to.

Re: multicast, interfaces, kernel 3.0+...

[Michael Tokarev]

On 22.09.2012 08:31, David Miller wrote:
> From: Michael Tokarev <mjt@tls.msk.ru>
> Date: Sat, 22 Sep 2012 08:21:52 +0400
> 
>> The IP_ADD_MEMBERSHIP interface is apparently misdefined, because it
>> accepts an IP address of an interface, instead of an ifindex, or
>> ifname, or something like this, since there's no, obviously, 1:1
>> correspondence between ifaces and addresses, an iface can have no
>> addresses assotiated with it, or two ifaces can share one IP address
>> like in my case.  But the "questionable" part is the "usualness" of
>> this setup I have here, with two ifaces having the same IP address.
> 
> Can you at least look at the API specification for IP_ADD_MEMBERSHIP
> before making such claims?

As I mentioned in previous email, I haven't dealt with multicast before,
so obviously I tried my best to learn before making any claims at all.
And the fine manual, http://tldp.org/HOWTO/Multicast-HOWTO-6.html ,
says:

 6.4 IP_ADD_MEMBERSHIP.

 The ip_mreq structure (taken from /usr/include/linux/in.h) has the following members:

    struct ip_mreq
    {
            struct in_addr imr_multiaddr;   /* IP multicast address of group */
            struct in_addr imr_interface;   /* local IP address of interface */
    };

 ...
 setsockopt (socket, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq));

Yes I probably should have read the mentioned header and the manpage too.
But the app does this:

 setsockopt(567, SOL_IP, IP_ADD_MEMBERSHIP, "\344\5\6\7\nM\7Z", 8) = 0
 setsockopt(567, SOL_IP, IP_ADD_MEMBERSHIP, "\344\5\6\7\nM\7Z", 8) = -1 EADDRINUSE (Address already in use)

so apparently sizeof(mreq) is 8 bytes for it, and I just didn't think there
may be additional fields in "real life".  That was puzzling, so I asked.
(This is most likely a generic java interface to this facility, not
linux-specific).

As you can see, I at least tried.  It wasn't apparently successful, but
that isn't entirely my fault either, -- the "canonical" howto does not
mention that there might be more members in this structure.. ;)

> The IP_ADD_MEMBERSHIP interface allows for the specification of a
> specific interface, the structure you pass into IP_ADD_MEMBERSHIP has
> an ->imr_ifindex field and this is the first key the call uses
> to pick a device.

Ok, after you mentioned this, I looked at the other sources and indeed
it has.  I stand corrected, and my questions answered.

> If you do not specify an explicit ifindex, and leave it at zero which
> I bet your application is doing, it picks the first address which has
> the specified address.
> 
> As you have discovered, just specifying the address can cause unwanted
> effects when multiple devices have the same IP address.  Because the
> order of network devices in the system is never, and has never, been
> guaranteed.
> 
> So the selection in this situation is essentially random because
> you haven't given the kernel enough information to choose things
> the way that you want it to.

Yes, that's what I thought too, it was just puzzling with the missing
bits of info.  And yes I remember when order of addresses changed
in various places (routing table was one example) and people started
complaining, even when order had never been deterministic.

I'm not complaining, I'm just asking.  And you answered my questions
perfectly, thank you!

/mjt

Re: multicast, interfaces, kernel 3.0+...

[David Miller]

From: Michael Tokarev <mjt@tls.msk.ru>
Date: Sat, 22 Sep 2012 08:43:07 +0400

> And the fine manual, http://tldp.org/HOWTO/Multicast-HOWTO-6.html ,

You're reading a document that's 14 years old.

Re: multicast, interfaces, kernel 3.0+...

[David Miller]

BTW, your site must be in a huge number of anti SPAM databases.

Because every time a mail is sent in this thread, as vger.kernel.org
postmaster I see hundreds of subscribers bounce.

It only happens for postings where your email address appears.

So you are basically invisible to much of the world, just FYI.