Routing tables associated with VLANs dissappear when parent ethX down/up

Routing tables associated with VLANs dissappear when parent ethX down/up

[Ben Greear]

Hello!

Found something strange today.

Suppose I have eth2 with VLAN 2 (eth2.2) associated with it.  I
add a routing table that is associated with eth2.2 (table 76 in my case).


Now, I run:
ifconfig eth2 down
ifconfig eth2 up

All of the routes in routing table 76 are now gone!  I can
understand how it might be useful to remove the routes from
the eth2 table (as is also done), but it seems pretty extreme
to muck with the VLAN device's tables as well.


[root@lanforge-33-46 local]# ip route show table 76
27.1.1.0/24 via 27.1.1.2 dev eth2.2
default via 27.1.1.1 dev eth2.2
[root@lanforge-33-46 local]# ifconfig eth2.2
eth2.2    Link encap:Ethernet  HWaddr 00:03:2D:08:33:47
           inet addr:27.1.1.2  Bcast:27.1.1.255  Mask:255.255.255.0
           UP BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[root@lanforge-33-46 local]# ifconfig eth2 down;ifconfig eth2 up
[root@lanforge-33-46 local]# ip route show table 76



-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Routing tables associated with VLANs dissappear when parent ethX down/up

[Ben Greear]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]

For consideration, this patch seems to work for me.  I'm not sure
why we ever listed to these events.  I've only tested on a NIC that
doesn't support hw-accel at the moment..will test with e1000 later.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


[-- Attachment #2: vlan.patch --]
[-- Type: text/x-patch, Size: 823 bytes --]

diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c
index c4209c8..acbf0ff 100644
--- a/net/8021q/vlan.c
+++ b/net/8021q/vlan.c
@@ -615,6 +615,11 @@ static int vlan_device_event(struct notifier_block *unused, unsigned long event,
 		}
 		break;
 
+#if 0
+		/* Don't propagate management state from base dev to VLANs.  If you do this,
+		 * then if you 'ifconfig eth0 down; ifconfig eth0 up', you also lose all the
+		 * routes for eth0.* VLANs. --Ben
+		 */
 	case NETDEV_DOWN:
 		/* Put all VLANs for this dev in the down state too.  */
 		for (i = 0; i < VLAN_GROUP_ARRAY_LEN; i++) {
@@ -644,6 +649,7 @@ static int vlan_device_event(struct notifier_block *unused, unsigned long event,
 			dev_change_flags(vlandev, flgs | IFF_UP);
 		}
 		break;
+#endif
 		
 	case NETDEV_UNREGISTER:
 		/* Delete all VLANs for this dev. */

Re: Routing tables associated with VLANs dissappear when parent ethX down/up

[Stephen Hemminger]

On Wed, 21 Nov 2007 11:51:43 -0800
Ben Greear <greearb@candelatech.com> wrote:

> For consideration, this patch seems to work for me.  I'm not sure
> why we ever listed to these events.  I've only tested on a NIC that
> doesn't support hw-accel at the moment..will test with e1000 later.
> 
> Thanks,
> Ben
> 

But then if you are doing bonding or bridging of vlan's 
and you bring down the root network device, the upper layer is not
notified (for failover).

-- 
Stephen Hemminger <shemminger@linux-foundation.org>

Re: Routing tables associated with VLANs dissappear when parent ethX down/up

[Patrick McHardy]

Stephen Hemminger wrote:
> On Wed, 21 Nov 2007 11:51:43 -0800
> Ben Greear <greearb@candelatech.com> wrote:
>
>   
>> For consideration, this patch seems to work for me.  I'm not sure
>> why we ever listed to these events.  I've only tested on a NIC that
>> doesn't support hw-accel at the moment..will test with e1000 later.
>>
>> Thanks,
>> Ben
>>
>>     
>
> But then if you are doing bonding or bridging of vlan's 
> and you bring down the root network device, the upper layer is not
> notified (for failover).
>   

operstate should be enough for this I guess. Ben, what does iproute show
for the vlan device when the lower device is down?

Re: Routing tables associated with VLANs dissappear when parent ethX down/up

[Ben Greear]

Patrick McHardy wrote:
> Stephen Hemminger wrote:
>> On Wed, 21 Nov 2007 11:51:43 -0800
>> Ben Greear <greearb@candelatech.com> wrote:
>>
>>  
>>> For consideration, this patch seems to work for me.  I'm not sure
>>> why we ever listed to these events.  I've only tested on a NIC that
>>> doesn't support hw-accel at the moment..will test with e1000 later.
>>>
>>> Thanks,
>>> Ben
>>>
>>>     
>>
>> But then if you are doing bonding or bridging of vlan's and you bring 
>> down the root network device, the upper layer is not
>> notified (for failover).
>>   
> 
> operstate should be enough for this I guess. Ben, what does iproute show
> for the vlan device when the lower device is down?

It looks like it knows, assuming M-DOWN is useful information.
Eth2 is un-plugged, by the way.

[root@lanforge-33-46 ~]# ifconfig eth2 up
[root@lanforge-33-46 ~]# ip link show eth2.2
125: eth2.2@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
     link/ether 00:03:2d:08:33:47 brd ff:ff:ff:ff:ff:ff
[root@lanforge-33-46 ~]# ifconfig eth2 down
[root@lanforge-33-46 ~]# ip link show eth2.2
125: eth2.2@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue
     link/ether 00:03:2d:08:33:47 brd ff:ff:ff:ff:ff:ff
[root@lanforge-33-46 ~]#

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Routing tables associated with VLANs dissappear when parent ethX down/up

[Patrick McHardy]

Ben Greear wrote:
> Patrick McHardy wrote:
>> Stephen Hemminger wrote:
>>>
>>> But then if you are doing bonding or bridging of vlan's and you 
>>> bring down the root network device, the upper layer is not
>>> notified (for failover).
>>>   
>>
>> operstate should be enough for this I guess. Ben, what does iproute show
>> for the vlan device when the lower device is down?
>
> It looks like it knows, assuming M-DOWN is useful information.
> Eth2 is un-plugged, by the way.
>
> [root@lanforge-33-46 ~]# ifconfig eth2 up
> [root@lanforge-33-46 ~]# ip link show eth2.2
> 125: eth2.2@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc 
> noqueue
>     link/ether 00:03:2d:08:33:47 brd ff:ff:ff:ff:ff:ff
> [root@lanforge-33-46 ~]# ifconfig eth2 down
> [root@lanforge-33-46 ~]# ip link show eth2.2
> 125: eth2.2@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 
> qdisc noqueue
>     link/ether 00:03:2d:08:33:47 brd ff:ff:ff:ff:ff:ff
> [root@lanforge-33-46 ~]# 

That comes from iproute itself, but the missing LOWER-UP flag
indicates it and that should be enough for bridging and bonding.
I'm unsure about this though since its still a big difference in
userspace visible behaviour, people might just as well manually
configure failover once routing disappears or the device goes down,
or just have routing fall through to different routes. All this
wouldn't work anymore.

Maybe we can make this optional somehow without too much uglyness?

Re: Routing tables associated with VLANs dissappear when parent ethX down/up

[Ben Greear]

Patrick McHardy wrote:
> Ben Greear wrote:
>> Patrick McHardy wrote:
>>> Stephen Hemminger wrote:
>>>>
>>>> But then if you are doing bonding or bridging of vlan's and you 
>>>> bring down the root network device, the upper layer is not
>>>> notified (for failover).
>>>>   
>>>
>>> operstate should be enough for this I guess. Ben, what does iproute show
>>> for the vlan device when the lower device is down?
>>
>> It looks like it knows, assuming M-DOWN is useful information.
>> Eth2 is un-plugged, by the way.
>>
>> [root@lanforge-33-46 ~]# ifconfig eth2 up
>> [root@lanforge-33-46 ~]# ip link show eth2.2
>> 125: eth2.2@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc 
>> noqueue
>>     link/ether 00:03:2d:08:33:47 brd ff:ff:ff:ff:ff:ff
>> [root@lanforge-33-46 ~]# ifconfig eth2 down
>> [root@lanforge-33-46 ~]# ip link show eth2.2
>> 125: eth2.2@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 
>> qdisc noqueue
>>     link/ether 00:03:2d:08:33:47 brd ff:ff:ff:ff:ff:ff
>> [root@lanforge-33-46 ~]# 
> 
> That comes from iproute itself, but the missing LOWER-UP flag
> indicates it and that should be enough for bridging and bonding.
> I'm unsure about this though since its still a big difference in
> userspace visible behaviour, people might just as well manually
> configure failover once routing disappears or the device goes down,
> or just have routing fall through to different routes. All this
> wouldn't work anymore.
> 
> Maybe we can make this optional somehow without too much uglyness?

I'm fine with that..we can just add a new vlan-device flag similar to the
reorder-header flag.

With the current code, on 'UP' of the underlying
code, all of the VLANs will also go UP, even if the user had previously
put them DOWN.  That seems like it could be quite dangerous/unexpected
to me..but I guess it's required if we are going to automatically DOWN them...

One other thought:  Maybe we could tell a small lie and say that we have
NO-CARRIER on the VLAN when the underlying device is down OR has no carrier?

That way we keep normal link up/down semantics w/out having to change the
admin state of the VLANs...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Routing tables associated with VLANs dissappear when parent ethX down/up

[Patrick McHardy]

Ben Greear wrote:
> Patrick McHardy wrote:
>>
>> That comes from iproute itself, but the missing LOWER-UP flag
>> indicates it and that should be enough for bridging and bonding.
>> I'm unsure about this though since its still a big difference in
>> userspace visible behaviour, people might just as well manually
>> configure failover once routing disappears or the device goes down,
>> or just have routing fall through to different routes. All this
>> wouldn't work anymore.
>>
>> Maybe we can make this optional somehow without too much uglyness?
>
> I'm fine with that..we can just add a new vlan-device flag similar to the
> reorder-header flag.

An alternative to this would be something like Julian Anastasov static 
routes
patch. Not sure if it has ever been considered for merging, but its a 
cleaner
way than doing per-device hacks.

http://www.ssi.bg/~ja/

>
> With the current code, on 'UP' of the underlying
> code, all of the VLANs will also go UP, even if the user had previously
> put them DOWN.  That seems like it could be quite dangerous/unexpected
> to me..but I guess it's required if we are going to automatically DOWN 
> them...

Yeah, I too never liked this behaviour.

>
> One other thought:  Maybe we could tell a small lie and say that we have
> NO-CARRIER on the VLAN when the underlying device is down OR has no 
> carrier?
>
> That way we keep normal link up/down semantics w/out having to change the
> admin state of the VLANs...

Thats pretty much what the operstate is doing, it should go to
IF_OPER_LOWERLAYERDOWN when the lower device is down. But as I
said above, people could actually rely on routes disappearing.