Resend: SIOCGIFBRDADDR?

Resend: SIOCGIFBRDADDR?

[Alan Robertson]

I sent this earlier, but I got no reply, and can't seem to find it in the 
archives.  If you reply to this, could you please copy both Kevin Dwyer and 
myself on the reply?  Many thanks!

Hi,

Some folks using the linux-ha (heartbeat) software have been running into
some network-related problems, and Alan Cox suggested that you folks could
help us out.

One of the ways you can configure linux-ha protocols is to send broadcast
packets.  In this case, our software needs to get the broadcast address from
the OS, so we can proceed.  We currently use the SIOCGIFBRDADDR ioctl to do
this for us.  This has worked pretty well for us for quite some time, but
recently some users have run into some problems.

It appears that if one configures an interface using ifconfig, then the
SIOCGIFBRDADDR ioctl works quite nicely to determine the broadcast address
of an interface.

However, if one uses the iproute2 tools to configure the interface, then it
appears that the SIOCGIFBRDADDR ioctl returns bogus results on that interface.

Is this supposed to be the case?

Is there a different method we're supposed to use instead of SIOCGIFBRDADDR?


	Thanks!

	-- Alan Robertson
	   alanr@unix.sh
	   http://linux-ha.org/

Re: Resend: SIOCGIFBRDADDR?

[kuznet]

Hello!

> Is this supposed to be the case?

You forgot to say what kind of "bogus result" is obtained.

Alexey

Re: Resend: SIOCGIFBRDADDR?

[Kevin Dwyer]

On Thu, 15 Aug 2002, kuznet@ms2.inr.ac.ru grunted something like:

> You forgot to say what kind of "bogus result" is obtained.

Since I'm the one who has primarily been chasing this problem, I'll
explain.

(cut from my original email...)

Specifically, it uses iproute to add aliases to a device, like eth0 in
this example:

ip -f inet addr add 10.5.5.1 dev eth0 scope link
ip -f inet addr add 10.5.5.2 dev eth0 scope link
ip -f inet addr add 10.5.5.3 dev eth0 scope link
ip -f inet addr add 10.5.5.4 dev eth0 scope link
ip -f inet addr add 10.5.5.5 dev eth0 scope link

That's all fine, except that it causes really weird stuff to happen to the
output of ifconfig.  For one, the inet addr: field will change once in a
while if I make a modification with ifconfig.  That doesn't affect
heartbeat, but the Bcast: field is wrongly reported as 0.0.0.0.  Now, I
know for a fact that I set it correctly when I bring up the interface,
prior to using the ip route commands.  Once they've been applied, it goes
crazy.  So my question: Does anyone know if there's an ip route command to
make the Bcast field show up as the correct value, or is there a better
way for heartbeat to get the broadcast address?

(and from a followup email...)

# ip addr show eth0 primary scope global

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:c0:95:e1:bc:64 brd ff:ff:ff:ff:ff:ff
    inet 10.5.5.8/24 brd 10.5.5.255 scope global eth0

That command gives me the correct broadcast address in the brd field.

Correct meaning, the one we're looking for, and would otherwise have
gotten if I hadn't used the before-mentioned iproute commands.

Any help is appreciated.

Thanks.

-[ kevin@pheared.net                 devel.pheared.net ]-
-[ Rather be forgotten, than remembered for giving in. ]-
-[ ZZ = g ^ (xb * xa) mod p      g = h^{(p-1)/q} mod p ]-

Re: Resend: SIOCGIFBRDADDR?

[kuznet]

Hello!

> crazy.  So my question: Does anyone know if there's an ip route command to
> make the Bcast field show up as the correct value,

Yes, that one which is described in manual (and equivalent to corresponding
ifconfig)

ifconfig eth0:tra-ta-ta 10.x.y.z etc.


>				      or is there a better
> way for heartbeat to get the broadcast address?

Well, yes. Initialize ifreq with right address, for instance.
Probably, your program has the same problem, which is in net-tools package.


Or this:

> That command gives me the correct broadcast address in the brd field.

Alexey

Re: Resend: SIOCGIFBRDADDR?

[Kevin Dwyer]

On Thu, 15 Aug 2002, kuznet@ms2.inr.ac.ru grunted something like:

Thanks for the response...

> > crazy.  So my question: Does anyone know if there's an ip route command to
> > make the Bcast field show up as the correct value,
>
> Yes, that one which is described in manual (and equivalent to corresponding
> ifconfig)
>
> ifconfig eth0:tra-ta-ta 10.x.y.z etc.

Perhaps I should clarify a bit further.
When we are starting heartbeat, the interface is setup thusly:
(I've simplified this to make it easier to read)

ifconfig eth0 10.5.5.8 broadcast 10.5.5.255 netmask 255.255.255.0
ip -f inet addr add 10.5.5.1 dev eth0 scope link
ip -f inet addr add 10.5.5.2 dev eth0 scope link

After executing those commands, we see the following:

ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:C0:95:E1:BC:64
          inet addr:10.5.5.1  Bcast:0.0.0.0  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
<snip>

(Note: the inet addr, Bcast, and Mask fields are all incorrectly reporting
       ifconfig's initial settings, which is why heartbeat is also
       confused.)

2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:c0:95:e1:bc:64 brd ff:ff:ff:ff:ff:ff
    inet 10.5.5.1/32 scope link eth0
    inet 10.5.5.2/32 scope link eth0
    inet 10.5.5.8/24 brd 10.5.5.255 scope global eth0


So we have eth0 setup as 10.5.5.8, and two aliases setup with iproute at
10.5.5.1 and 10.5.5.2.  They are not however aliases like eth0:1, eth0:2,
etc.  But, iproute has setup the kernel to reply to ARPs for those IPs.

So, heartbeat needs to know which interface to use for sending broadcast
packets to another node.  I've defined eth0 as said interface, and it uses
eth0 as the ifreq for the ioctl call.  However, the ioctl call returns
0.0.0.0, when in fact I need 10.5.5.255.


> >				      or is there a better
> > way for heartbeat to get the broadcast address?
>
> Well, yes. Initialize ifreq with right address, for instance.
> Probably, your program has the same problem, which is in net-tools package.

See above.. the problem is that using eth0 is the right address.

> Or this:
>
> > That command gives me the correct broadcast address in the brd field.

The only problem I see with emulating iproute's commands is that you'll
need netlink (is this correct?) so it won't work for everyone until
everyone starts adding that to their kernel config.  (A possibly
reasonable request, if this is indeed the best way to attack the problem.
This is partially why we're asking for commentary/help.)


-[ kevin@pheared.net                 devel.pheared.net ]-
-[ Rather be forgotten, than remembered for giving in. ]-
-[ ZZ = g ^ (xb * xa) mod p      g = h^{(p-1)/q} mod p ]-

Re: Resend: SIOCGIFBRDADDR?

[Matthew G. Marsh]

On Thu, 15 Aug 2002, Kevin Dwyer wrote:

> On Thu, 15 Aug 2002, kuznet@ms2.inr.ac.ru grunted something like:
>
> Thanks for the response...
>
> > > crazy.  So my question: Does anyone know if there's an ip route command to
> > > make the Bcast field show up as the correct value,
> >
> > Yes, that one which is described in manual (and equivalent to corresponding
> > ifconfig)
> >
> > ifconfig eth0:tra-ta-ta 10.x.y.z etc.
>
> Perhaps I should clarify a bit further.
> When we are starting heartbeat, the interface is setup thusly:
> (I've simplified this to make it easier to read)
>
> ifconfig eth0 10.5.5.8 broadcast 10.5.5.255 netmask 255.255.255.0
> ip -f inet addr add 10.5.5.1 dev eth0 scope link
> ip -f inet addr add 10.5.5.2 dev eth0 scope link

Why do you use the 'scope link' statements?

And as far as your original question try the following:

ip -f inet addr add 10.5.5.8/24 dev eth0 brd +
ip -f inet addr add 10.5.5.1 dev eth0 scope link brd 10.5.5.255
ip -f inet addr add 10.5.5.2 dev eth0 scope link brd 10.5.5.255

And then you will have the output of 'ip ad li dev eth0' as:

3: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:c0:9f:0c:5f:db brd ff:ff:ff:ff:ff:ff
    inet 10.5.5.1/32 brd 10.5.5.255 scope link eth0
    inet 10.5.5.2/32 brd 10.5.5.255 scope link eth0
    inet 10.5.5.8/24 brd 10.5.5.255 scope global eth0

And also you will then have 'ifconfig eth0':

eth0      Link encap:Ethernet  HWaddr 00:C0:9F:0C:5F:DB
          inet addr:10.5.5.2  Bcast:10.5.5.255  Mask:255.255.255.255
          UP BROADCAST MULTICAST  MTU:1500  Metric:1

Work for you?

[snip]

> -[ kevin@pheared.net                 devel.pheared.net ]-
> -[ Rather be forgotten, than remembered for giving in. ]-
> -[ ZZ = g ^ (xb * xa) mod p      g = h^{(p-1)/q} mod p ]-

And if I am way off base here due to jumping in just ignore me.

--------------------------------------------------
Matthew G. Marsh,  President
Paktronix Systems LLC
1506 North 59th Street
Omaha  NE  68104
Phone: (402) 932-7250 x101
Email: mgm@paktronix.com
WWW:  http://www.paktronix.com
--------------------------------------------------

Re: Resend: SIOCGIFBRDADDR?

[Kevin Dwyer]

On Sun, 18 Aug 2002, Matthew G. Marsh grunted something like:

> > ifconfig eth0 10.5.5.8 broadcast 10.5.5.255 netmask 255.255.255.0
> > ip -f inet addr add 10.5.5.1 dev eth0 scope link
> > ip -f inet addr add 10.5.5.2 dev eth0 scope link
>
> Why do you use the 'scope link' statements?

Why do I use the 'scope link' argument, or the statement as a whole?
Those two statements give the same effect as aliases (more or less)
without having to ifconfig eth0:X for each X.  My understanding was the
'scope link' part was useful in keeping the effects of the command local
to the device.  However, I also did not generate these commands on my own:
fwbuilder is doing that.  We're trying to get a setup that can be easily
managed by other people who may not have the same experience with
command-line tools and such.

> And as far as your original question try the following:
>
> ip -f inet addr add 10.5.5.8/24 dev eth0 brd +
> ip -f inet addr add 10.5.5.1 dev eth0 scope link brd 10.5.5.255
> ip -f inet addr add 10.5.5.2 dev eth0 scope link brd 10.5.5.255

Aha!  At the suggestion of the iproute2 documentation, and another person
on the linux-ha list, I tried using 'brd +' on the aliased IPs, but
apparently we were reading the docs wrong.  Setting it explicitly works!

> And also you will then have 'ifconfig eth0':
>
> eth0      Link encap:Ethernet  HWaddr 00:C0:9F:0C:5F:DB
>           inet addr:10.5.5.2  Bcast:10.5.5.255  Mask:255.255.255.255
>           UP BROADCAST MULTICAST  MTU:1500  Metric:1
>
> Work for you?

Exactly what we need.  Now the ioctl can pull the broadcast address.

> And if I am way off base here due to jumping in just ignore me.

Not at all, thanks for the suggestion.  It seems now that I need to try to
encourage the fwbuilder folks to specify the broadcast address on each
alias.  At the very least, I should be able to whip out a patch to do
it, and hopefully they'll accept it.


/* kevin@pheared.net               http://devel.pheared.net/ */
/* Network Security Engineer       http://pheared.net/~kevin */
/* Sabotage will set us free.   Throw a rock in the machine. */

Re: Resend: SIOCGIFBRDADDR?

[Matthew G. Marsh]

On Sun, 18 Aug 2002, Kevin Dwyer wrote:

> On Sun, 18 Aug 2002, Matthew G. Marsh grunted something like:
> > Why do you use the 'scope link' statements?
>
> Why do I use the 'scope link' argument, or the statement as a whole?
> Those two statements give the same effect as aliases (more or less)
> without having to ifconfig eth0:X for each X.  My understanding was the
> 'scope link' part was useful in keeping the effects of the command local
> to the device.  However, I also did not generate these commands on my own:
> fwbuilder is doing that.  We're trying to get a setup that can be easily
> managed by other people who may not have the same experience with
> command-line tools and such.

Ah OK. I was wondering because if you leave off the 'scope link' then you
get inheritance of the broadcast for secondary ip addrs.

> > And as far as your original question try the following:
> >
> > ip -f inet addr add 10.5.5.8/24 dev eth0 brd +
> > ip -f inet addr add 10.5.5.1 dev eth0 scope link brd 10.5.5.255
> > ip -f inet addr add 10.5.5.2 dev eth0 scope link brd 10.5.5.255
>
> Aha!  At the suggestion of the iproute2 documentation, and another person
> on the linux-ha list, I tried using 'brd +' on the aliased IPs, but
> apparently we were reading the docs wrong.  Setting it explicitly works!

Yep - that is the 'scope link' override that basically tells the system to
ignore any inheritance and treat the address as a special local case. Note
that the order in which the addresses are applied is critical. For
example if you do:

ip addr add 10.5.5.1/24 dev eth0 scope link brd +

Then you get the appropriate broadcast address for the /24 netmask. BUT
then you cannot add in a global address in the same network as in:

ip addr add 10.5.5.8/24 dev eth0 scope global brd 10.5.5.255
RTNETLINK answers: Invalid argument

You have to change the netmask at this point (think scope~=mask).
Essentially you cannot have a primary and secondary address on a single
device exist with different scope. Think of IPv6 scope and it becomes a
little clearer where the addressing conflicts.

Your best bet when in doubt is to always specify the actual broadcast
address you require. The 'brd +' is merly a shortcut to specify the
broadcast address associated with the specified CIDR mask. In fact you can
have way interesting amounts of fun by specifying alternate broadcasts
from CIDR masks as in:

ip addr add 10.5.5.1/24 dev eth0 brd 10.5.255.255

Using this on a "Class B" network allow you to "be seen" by other devices
but only speak to/from devices within your CIDR scope (hint: look at the
output of 'ip ro li tab local') as well as really fsck routers and ARP
tables for the network... ;-}

[snip]

> > And if I am way off base here due to jumping in just ignore me.
>
> Not at all, thanks for the suggestion.  It seems now that I need to try to
> encourage the fwbuilder folks to specify the broadcast address on each
> alias.  At the very least, I should be able to whip out a patch to do
> it, and hopefully they'll accept it.
>
> /* kevin@pheared.net               http://devel.pheared.net/ */
> /* Network Security Engineer       http://pheared.net/~kevin */
> /* Sabotage will set us free.   Throw a rock in the machine. */

--------------------------------------------------
Matthew G. Marsh,  President
Paktronix Systems LLC
1506 North 59th Street
Omaha  NE  68104
Phone: (402) 932-7250 x101
Email: mgm@paktronix.com
WWW:  http://www.paktronix.com
--------------------------------------------------