Fwd: 2.6.6: IPv6 initialisation bug

Fwd: 2.6.6: IPv6 initialisation bug

[Russell King]

Ok, I've just tried 2.6.7 out on my root-NFS'd firewall with IPv6 built
in, and it doesn't work because of the problem I described below.

Unfortunately, it's impossible to add the missing "local" routes
using /sbin/ip:

ip add route local fe80::a00:2bff:fe95:1d7b via :: dev eth0

results in an "unreachable" route being added for that address rather
than a local route.

What's the solution?

Is there a good reason why IPv6 uses the loopback device for local
routes?

(I'm copying lkml this time since afaik netdev ignored the previous
message.)

----- Forwarded message from Russell King <rmk@arm.linux.org.uk> -----
Date: Tue, 18 May 2004 16:46:44 +0100
From: Russell King <rmk@arm.linux.org.uk>
To: netdev@oss.sgi.com
Subject: 2.6.6: IPv6 initialisation bug

Hi,

I think I've found an IPv6 initialisation bug which occurs when IPv6
is modular, and you insert this module when eth0 is up and running,
but lo may be down.

IPv6 appears to create routes for addresses which are defined as
"local" (eg, link local, addresses assigned to the host etc) using
the loopback device.

However, if the ipv6 module is loaded when lo is down and some other
interface is up (eg, in the case of a root-NFS box), then things fall
apart.

bash-2.04# ip -6 addr
1: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qlen 1000
    inet6 fec0::1:a00:2bff:fe00:193/64 scope site dynamic
       valid_lft 2591630sec preferred_lft 604430sec
    inet6 2002:xxxx:xxxx:xxxx:a00:2bff:fe00:193/64 scope global dynamic
       valid_lft 2591630sec preferred_lft 604430sec
    inet6 fe80::a00:2bff:fe00:193/64 scope link
       valid_lft forever preferred_lft forever
2: lo: <LOOPBACK,UP> mtu 16436
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
bash-2.04# ip -6 route show table all
local ::1 via :: dev lo  proto none  metric 0  mtu 16436 advmss 16376
unreachable ::/96 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable ::ffff:0.0.0.0/96 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:a00::/24 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:7f00::/24 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:a9fe::/32 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:ac10::/28 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:c0a8::/32 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
2002:xxxx:xxxx:xxxx::/64 dev eth0  proto kernel  metric 256  expires 2591712sec mtu 1500 advmss 1440
unreachable 2002:e000::/19 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 3ffe:ffff::/32 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
fe80::/64 dev eth0  metric 256  mtu 1500 advmss 1440
fec0:0:0:1::/64 dev eth0  proto kernel  metric 256  expires 2591712sec mtu 1500 advmss 1440
ff02::1 via ff02::1 dev eth0  metric 0
    cache  mtu 1500 advmss 1440
ff00::/8 dev eth0  metric 256  mtu 1500 advmss 1440
unreachable default dev lo  proto none  metric -1  error -101

As you can see, we're missing the local routes for all our addresses
against "eth0" - because we tried to add them to the IPv6 routing
table when "lo" was down.

The result is rather distasteful on the network - we start hammering
the local segment with neighbour solicitations for our link local and
global addresses, as well as sending neighbour solicitations for other
nodes addresses.  We receive neighbour advertisment replies, but because
we believe we don't own our own address, we forward them back out the
same interface triggering yet more neighbour solicitations for our own
addresses.

So... what's the solution for nodes running off root-NFS where "lo"
will always be brought up _after_ some other interface?

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

----- End forwarded message -----

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: 2.6.6: IPv6 initialisation bug

[YOSHIFUJI Hideaki / 吉藤英明]

In article <20040628010200.A15067@flint.arm.linux.org.uk> (at Mon, 28 Jun 2004 01:02:01 +0100), Russell King <rmk+lkml@arm.linux.org.uk> says:

> Ok, I've just tried 2.6.7 out on my root-NFS'd firewall with IPv6 built
> in, and it doesn't work because of the problem I described below.
:
> What's the solution?

Bring lo up before bring others up.
What does prevent you from doing this?
(Do we need some bits to do this automatically?)


> Is there a good reason why IPv6 uses the loopback device for local
> routes?

IPv6 creates kernel routes for local addresses on lo to receive
packets for local address.

Well, someone probably wants to have
static local routes on ethX + temprary (cache) local routes on lo
(as IPv4 does; correct me if I'm wrong.)

But this won't work because IPv6 does DAD when we make some interface up.
We need lo anyway.

--yoshfuji

Re: 2.6.6: IPv6 initialisation bug

[Russell King]

On Tue, Jun 29, 2004 at 02:06:27AM +0900, YOSHIFUJI Hideaki / ?$B5HF#1QL@?(B wrote:
> In article <20040628010200.A15067@flint.arm.linux.org.uk> (at Mon, 28 Jun 2004 01:02:01 +0100), Russell King <rmk+lkml@arm.linux.org.uk> says:
> 
> > Ok, I've just tried 2.6.7 out on my root-NFS'd firewall with IPv6 built
> > in, and it doesn't work because of the problem I described below.
> :
> > What's the solution?
> 
> Bring lo up before bring others up.
> What does prevent you from doing this?
> (Do we need some bits to do this automatically?)

When you use root-NFS, the kernel itself brings up the interfaces,
and IPv6 immediately comes in and tries to configure itself to them,
trying to create the routes.

Unfortunately, the kernel doesn't bring up lo first because it
doesn't know to do that.

> > Is there a good reason why IPv6 uses the loopback device for local
> > routes?
> 
> IPv6 creates kernel routes for local addresses on lo to receive
> packets for local address.
> 
> Well, someone probably wants to have
> static local routes on ethX + temprary (cache) local routes on lo
> (as IPv4 does; correct me if I'm wrong.)

This would be preferable I think - it certainly would stop systems
exploding when you take lo down for whatever reason, even temporarily.
Currently, if you do that, all your IPv6 local routes die off and
you're left trying to forward your own local address out various
itnerfaces.

> But this won't work because IPv6 does DAD when we make some interface up.
> We need lo anyway.

Sorry, I don't understand how this has a bearing on which device the
local routes are attached to.

How come IPv4 can be happy having local routes attached to the
individual interface, yet IPv6 can't be?

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: 2.6.6: IPv6 initialisation bug

[YOSHIFUJI Hideaki / 吉藤英明]

In article <20040628184758.C9214@flint.arm.linux.org.uk> (at Mon, 28 Jun 2004 18:47:58 +0100), Russell King <rmk+lkml@arm.linux.org.uk> says:

> On Tue, Jun 29, 2004 at 02:06:27AM +0900, YOSHIFUJI Hideaki / ?$B5HF#1QL@?(B wrote:
> > In article <20040628010200.A15067@flint.arm.linux.org.uk> (at Mon, 28 Jun 2004 01:02:01 +0100), Russell King <rmk+lkml@arm.linux.org.uk> says:
> > 
> > > Ok, I've just tried 2.6.7 out on my root-NFS'd firewall with IPv6 built
> > > in, and it doesn't work because of the problem I described below.
> > :
> > > What's the solution?
> > 
> > Bring lo up before bring others up.
> > What does prevent you from doing this?
> > (Do we need some bits to do this automatically?)
> 
> When you use root-NFS, the kernel itself brings up the interfaces,
> and IPv6 immediately comes in and tries to configure itself to them,
> trying to create the routes.
> 
> Unfortunately, the kernel doesn't bring up lo first because it
> doesn't know to do that.

Okay, would you try the following patch, please?


D: Bring loopback device up first

Signed-Off-By: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>

===== net/ipv4/ipconfig.c 1.38 vs edited =====
--- 1.38/net/ipv4/ipconfig.c	2004-06-23 09:06:18 +09:00
+++ edited/net/ipv4/ipconfig.c	2004-06-29 09:53:36 +09:00
@@ -183,7 +183,14 @@
 
 	last = &ic_first_dev;
 	rtnl_shlock();
+
+	/* bring loopback device up first */
+	if (dev_change_flags(&loopback_dev, loopback_dev.flags | IFF_UP) < 0)
+		printk(KERN_ERR "IP-Config: Failed to open %s\n", loopback_dev.name);
+
 	for (dev = dev_base; dev; dev = dev->next) {
+		if (dev == &loopback_dev)
+			continue;
 		if (user_dev_name[0] ? !strcmp(dev->name, user_dev_name) :
 		    (!(dev->flags & IFF_LOOPBACK) &&
 		     (dev->flags & (IFF_POINTOPOINT|IFF_BROADCAST)) &&

-- 
Hideaki YOSHIFUJI @ USAGI Project <yoshfuji@linux-ipv6.org>
GPG FP: 9022 65EB 1ECF 3AD1 0BDF  80D8 4807 F894 E062 0EEA

Re: 2.6.6: IPv6 initialisation bug

[Russell King]

On Tue, Jun 29, 2004 at 09:59:03AM +0900, YOSHIFUJI Hideaki / ?$B5HF#1QL@?(B wrote:
> In article <20040628184758.C9214@flint.arm.linux.org.uk> (at Mon, 28 Jun 2004 18:47:58 +0100), Russell King <rmk+lkml@arm.linux.org.uk> says:
> > On Tue, Jun 29, 2004 at 02:06:27AM +0900, YOSHIFUJI Hideaki / ?$B5HF#1QL@?(B wrote:
> > > In article <20040628010200.A15067@flint.arm.linux.org.uk> (at Mon, 28 Jun 2004 01:02:01 +0100), Russell King <rmk+lkml@arm.linux.org.uk> says:
> > > 
> > > > Ok, I've just tried 2.6.7 out on my root-NFS'd firewall with IPv6 built
> > > > in, and it doesn't work because of the problem I described below.
> > > :
> > > > What's the solution?
> > > 
> > > Bring lo up before bring others up.
> > > What does prevent you from doing this?
> > > (Do we need some bits to do this automatically?)
> > 
> > When you use root-NFS, the kernel itself brings up the interfaces,
> > and IPv6 immediately comes in and tries to configure itself to them,
> > trying to create the routes.
> > 
> > Unfortunately, the kernel doesn't bring up lo first because it
> > doesn't know to do that.
> 
> Okay, would you try the following patch, please?

This does appear to fix the problem.

> D: Bring loopback device up first
> 
> Signed-Off-By: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>
> 
> ===== net/ipv4/ipconfig.c 1.38 vs edited =====
> --- 1.38/net/ipv4/ipconfig.c	2004-06-23 09:06:18 +09:00
> +++ edited/net/ipv4/ipconfig.c	2004-06-29 09:53:36 +09:00
> @@ -183,7 +183,14 @@
>  
>  	last = &ic_first_dev;
>  	rtnl_shlock();
> +
> +	/* bring loopback device up first */
> +	if (dev_change_flags(&loopback_dev, loopback_dev.flags | IFF_UP) < 0)
> +		printk(KERN_ERR "IP-Config: Failed to open %s\n", loopback_dev.name);
> +
>  	for (dev = dev_base; dev; dev = dev->next) {
> +		if (dev == &loopback_dev)
> +			continue;
>  		if (user_dev_name[0] ? !strcmp(dev->name, user_dev_name) :
>  		    (!(dev->flags & IFF_LOOPBACK) &&
>  		     (dev->flags & (IFF_POINTOPOINT|IFF_BROADCAST)) &&
> 
> -- 
> Hideaki YOSHIFUJI @ USAGI Project <yoshfuji@linux-ipv6.org>
> GPG FP: 9022 65EB 1ECF 3AD1 0BDF  80D8 4807 F894 E062 0EEA

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: 2.6.6: IPv6 initialisation bug

[YOSHIFUJI Hideaki / 吉藤英明]

In article <20040704170259.A16596@flint.arm.linux.org.uk> (at Sun, 4 Jul 2004 17:02:59 +0100), Russell King <rmk+lkml@arm.linux.org.uk> says:

> On Tue, Jun 29, 2004 at 09:59:03AM +0900, YOSHIFUJI Hideaki / ?$B5HF#1QL@?(B wrote:
> > In article <20040628184758.C9214@flint.arm.linux.org.uk> (at Mon, 28 Jun 2004 18:47:58 +0100), Russell King <rmk+lkml@arm.linux.org.uk> says:
> > > On Tue, Jun 29, 2004 at 02:06:27AM +0900, YOSHIFUJI Hideaki / ?$B5HF#1QL@?(B wrote:
> > > > In article <20040628010200.A15067@flint.arm.linux.org.uk> (at Mon, 28 Jun 2004 01:02:01 +0100), Russell King <rmk+lkml@arm.linux.org.uk> says:
> > > > 
> > > > > Ok, I've just tried 2.6.7 out on my root-NFS'd firewall with IPv6 built
> > > > > in, and it doesn't work because of the problem I described below.
> > > > :
> > > > > What's the solution?
> > > > 
> > > > Bring lo up before bring others up.
> > > > What does prevent you from doing this?
> > > > (Do we need some bits to do this automatically?)
> > > 
> > > When you use root-NFS, the kernel itself brings up the interfaces,
> > > and IPv6 immediately comes in and tries to configure itself to them,
> > > trying to create the routes.
> > > 
> > > Unfortunately, the kernel doesn't bring up lo first because it
> > > doesn't know to do that.
> > 
> > Okay, would you try the following patch, please?
> 
> This does appear to fix the problem.

Thanks.

David, please apply.

---------------
[IPV6] Bring lo up before setting other interface up.

IPv6 was not configured appropriately without lo.
Noticed by / tested by Russell King <rmk+lkml@arm.linux.org.uk>.

Signed-off-by: Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>

===== net/ipv4/ipconfig.c 1.39 vs edited =====
--- 1.39/net/ipv4/ipconfig.c	2004-07-01 07:26:09 +09:00
+++ edited/net/ipv4/ipconfig.c	2004-07-05 12:22:27 +09:00
@@ -183,7 +183,14 @@
 
 	last = &ic_first_dev;
 	rtnl_shlock();
+
+	/* bring loopback device up first */
+	if (dev_change_flags(&loopback_dev, loopback_dev.flags | IFF_UP) < 0)
+		printk(KERN_ERR "IP-Config: Failed to open %s\n", loopback_dev.name);
+
 	for (dev = dev_base; dev; dev = dev->next) {
+		if (dev == &loopback_dev)
+			continue;
 		if (user_dev_name[0] ? !strcmp(dev->name, user_dev_name) :
 		    (!(dev->flags & IFF_LOOPBACK) &&
 		     (dev->flags & (IFF_POINTOPOINT|IFF_BROADCAST)) &&


-- 
Hideaki YOSHIFUJI @ USAGI Project <yoshfuji@linux-ipv6.org>
GPG FP: 9022 65EB 1ECF 3AD1 0BDF  80D8 4807 F894 E062 0EEA

Re: 2.6.6: IPv6 initialisation bug

[David S. Miller]

On Mon, 05 Jul 2004 12:33:21 +0900 (JST)
YOSHIFUJI Hideaki / ^[$B5HF#1QL@^[(B <yoshfuji@linux-ipv6.org> wrote:

> David, please apply.
 ...
> [IPV6] Bring lo up before setting other interface up.

Applied.

2.6.6: IPv6 initialisation bug

[Russell King]

Hi,

I think I've found an IPv6 initialisation bug which occurs when IPv6
is modular, and you insert this module when eth0 is up and running,
but lo may be down.

IPv6 appears to create routes for addresses which are defined as
"local" (eg, link local, addresses assigned to the host etc) using
the loopback device.

However, if the ipv6 module is loaded when lo is down and some other
interface is up (eg, in the case of a root-NFS box), then things fall
apart.

bash-2.04# ip -6 addr
1: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qlen 1000
    inet6 fec0::1:a00:2bff:fe00:193/64 scope site dynamic
       valid_lft 2591630sec preferred_lft 604430sec
    inet6 2002:xxxx:xxxx:xxxx:a00:2bff:fe00:193/64 scope global dynamic
       valid_lft 2591630sec preferred_lft 604430sec
    inet6 fe80::a00:2bff:fe00:193/64 scope link
       valid_lft forever preferred_lft forever
2: lo: <LOOPBACK,UP> mtu 16436
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
bash-2.04# ip -6 route show table all
local ::1 via :: dev lo  proto none  metric 0  mtu 16436 advmss 16376
unreachable ::/96 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable ::ffff:0.0.0.0/96 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:a00::/24 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:7f00::/24 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:a9fe::/32 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:ac10::/28 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 2002:c0a8::/32 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
2002:xxxx:xxxx:xxxx::/64 dev eth0  proto kernel  metric 256  expires 2591712sec mtu 1500 advmss 1440
unreachable 2002:e000::/19 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
unreachable 3ffe:ffff::/32 dev lo  metric 1024  error -101 mtu 16436 advmss 16376
fe80::/64 dev eth0  metric 256  mtu 1500 advmss 1440
fec0:0:0:1::/64 dev eth0  proto kernel  metric 256  expires 2591712sec mtu 1500 advmss 1440
ff02::1 via ff02::1 dev eth0  metric 0
    cache  mtu 1500 advmss 1440
ff00::/8 dev eth0  metric 256  mtu 1500 advmss 1440
unreachable default dev lo  proto none  metric -1  error -101

As you can see, we're missing the local routes for all our addresses
against "eth0" - because we tried to add them to the IPv6 routing
table when "lo" was down.

The result is rather distasteful on the network - we start hammering
the local segment with neighbour solicitations for our link local and
global addresses, as well as sending neighbour solicitations for other
nodes addresses.  We receive neighbour advertisment replies, but because
we believe we don't own our own address, we forward them back out the
same interface triggering yet more neighbour solicitations for our own
addresses.

So... what's the solution for nodes running off root-NFS where "lo"
will always be brought up _after_ some other interface?

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core