[LARTC] Question about limited primary addresses

[LARTC] Question about limited primary addresses

[Patrick Turley]

I just realized that I hadn't actually asked a question when I posted 
this before. Let me try again...

I am trying to configure a Linux box with all possible VLANs (4094 of 
them), and a subnet on each VLAN. Creating the VLANs isn't a problem. 
But, when I try to use "ip addr add ..." commands to assign an IP 
address to each VLAN interface, I get to do about 280 of them before all 
the interfaces on the box become unresponsive. I've also tried to assign 
the same IP addresses all to eth0, and I get the same result.

The problem seems to hinge on the distinction between primary and 
secondary addresses.

If you're reading the LARTC list, then you've probably had the need 
(e.g., load testing) to assign thousands of IP addresses to a single 
interface. However, it's usually the case that one is assigning multiple 
addresses that all belong to the same subnet. In that case, the first 
such IP address is "primary" and all the additional addresses that 
belong to the same subnet are "secondary". My investigations tell me 
that "secondary" addresses are much lighter-weight. On the other hand, I 
don't seem to be able to assign more than about 280 "primary" addresses 
in the entire system before all the interfaces become unresponsive.

For additional information about "primary" vs. "secondary" addresses, 
take a look at:

http://www.linux-ip.net/html/linux-ip.html#tools-ip-address-add
http://www.linux-ip.net/gl/ip-cref/node33.html

Does anyone know why there appears to be a limit on the number of 
networks? Does anyone know where in the code this limit arises? Does 
anyone know a way to get around this limit?

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Question about limited primary addresses

[Patrick Turley]

Here is the evidence for my conclusions, quoted from our Bugzilla database.


------- Additional Comment #11 From Patrick Turley 2004-09-16 19:43

Our configuration uses the ip command to attach IP addresses to an
interface. Using these methods, it experience a severe limit on the
total number of addresses.

Tim has reported attaching thousands of addresses to his system test
boxes. There is also plenty of evidence on the Internet of other people
doing the same. All of the examples I looked at in detail were using the
ifconfig command.

In an attempt to discover if the ip command was the real problem, I
hunted down which ifconfig command would accomplish the same effect as
the ip command. Here are the ip commands required to attach a network to
a given device:


     ip addr add <networkaddress>/<prefixlength> brd + dev <device>
     ip link set <device> up


The equivalent ifconfig command is:


     ifconfig <device>:<aliasnumber> <address> netmask <netmask> up


I replaced the ip commands with the ifconfig command, ran the test and
got precisely the same results. The ifconfig command didn't seem to
improve anything.


------- Additional Comment #12 From Patrick Turley 2004-09-16 20:04

I used a script I downloaded from the Internet that uses ifconfig to
attach many IP addresses to a single interface. I used this script to
attach 1000 IP addresses to eth0 and saw no ill effects.

I used the low-level ip commands to examine the state of the network
more closely:


[root@rnsa root]# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc htb qlen 1000
     link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
     inet 192.168.0.1/24 brd 192.168.0.255 scope global eth0:0
     inet 13.0.0.1/24 brd 13.255.255.255 scope global eth0:1
     inet 13.0.1.1/24 brd 13.255.255.255 scope global eth0:251
     inet 13.0.2.1/24 brd 13.255.255.255 scope global eth0:501
     inet 13.0.3.1/24 brd 13.255.255.255 scope global eth0:751
     inet 13.0.0.2/24 brd 13.255.255.255 scope global secondary eth0:2
     inet 13.0.0.3/24 brd 13.255.255.255 scope global secondary eth0:3
     inet 13.0.0.4/24 brd 13.255.255.255 scope global secondary eth0:4
     inet 13.0.0.5/24 brd 13.255.255.255 scope global secondary eth0:5
     .
     .
     .
     inet 13.0.3.246/24 brd 13.255.255.255 scope global secondary eth0:996
     inet 13.0.3.247/24 brd 13.255.255.255 scope global secondary eth0:997
     inet 13.0.3.248/24 brd 13.255.255.255 scope global secondary eth0:998
     inet 13.0.3.249/24 brd 13.255.255.255 scope global secondary eth0:999
     inet 13.0.3.250/24 brd 13.255.255.255 scope global secondary eth0:1000


Notice that most of these addresses are tagged as "secondary". From

     http://www.linux-ip.net/html/linux-ip.html#tools-ip-address-add

we see that:


     When you add an address within the same CIDR network as another
     address on the same interface, the second address becomes a
     secondary address, meaning that if the first address is removed,
     the second address will also be purged from the interface.


This is consistent with the output of ip addr show. The script looped
through 1000 address, which is enough to span 4 class C networks. So,
the first address in each such network became a primary address, and all
the other addresses within those networks became secondaries. This
finally gives a concrete explanation for why we see many addresses on an
interface disappear when we only delete one.

From

     http://www.linux-ip.net/gl/ip-cref/node33.html

we learn that a little more about secondary addresses:


     1 secondary

         -- the address is not used when selecting the default source
         address of outgoing packets (Cf. Appendix A, p.[*].). An IP
         address becomes secondary if another address with the same
         prefix bits already exists. The first address is primary. It is
         the leader of the group of all secondary addresses. When the
         leader is deleted, all secondaries are purged too.


I also had a look at the route table:


     [root@rnsa root]# ip route show
     13.0.2.0/24    dev eth0  proto kernel  scope link  src 13.0.2.1
     13.0.3.0/24    dev eth0  proto kernel  scope link  src 13.0.3.1
     13.0.0.0/24    dev eth0  proto kernel  scope link  src 13.0.0.1
     13.0.1.0/24    dev eth0  proto kernel  scope link  src 13.0.1.1
     192.168.0.0/24 dev eth0  proto kernel  scope link  src 192.168.0.1
     10.1.0.0/16    dev eth1  proto kernel  scope link  src 10.1.1.18
     127.0.0.0/8    dev lo  scope link
     default via 10.1.254.254 dev eth1


This confirms what was said immediately above - none of the secondary
addresses are represented here - only the primary ones. This implies
that packets destined for one of the newly-created networks will only
take on a primary address as their source address.

------- Additional Comment #13 From Patrick Turley 2004-09-16 20:19

In our configuration, no two networks are ever allowed to overlap. That
means that *all* the addresses are primary. Take a look at the network
information for a system that has 100 VLANs and networks:


     [root@rnsa root]# ip addr show
     1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
         link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
         inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
     2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc htb qlen 1000
         link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
         inet 192.168.0.1/24 brd 192.168.0.255 scope global eth0
     3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc htb qlen 1000
         link/ether 00:09:6b:66:09:07 brd ff:ff:ff:ff:ff:ff
         inet 10.1.1.18/16 brd 10.1.255.255 scope global eth1
     1304: vlan1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
         link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
         inet 10.0.1.1/24 brd 10.0.1.255 scope global vlan1
     1305: vlan2: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
         link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
         inet 10.0.2.1/24 brd 10.0.2.255 scope global vlan2
     1306: vlan3: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
         link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
         inet 10.0.3.1/24 brd 10.0.3.255 scope global vlan3
         .
         .
         .
     1401: vlan98: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
         link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
         inet 10.0.98.1/24 brd 10.0.98.255 scope global vlan98
     1402: vlan99: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
         link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
         inet 10.0.99.1/24 brd 10.0.99.255 scope global vlan99
     1403: vlan100: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue
         link/ether 00:02:b3:cb:1e:ed brd ff:ff:ff:ff:ff:ff
         inet 10.0.100.1/24 brd 10.0.100.255 scope global vlan100


Since each address appears on a separate interface, they are all
primary. However, since they are all on disjoint networks, they would
still all be primary even if they all appeared on the same interface
(which is the original configuration that spawned this problem report).

Have a look at the routing table as well:


     [root@rnsa root]# ip route show
     10.0.20.0/24   dev vlan20  proto kernel  scope link  src 10.0.20.1
     10.0.21.0/24   dev vlan21  proto kernel  scope link  src 10.0.21.1
     10.0.22.0/24   dev vlan22  proto kernel  scope link  src 10.0.22.1
     .
     .
     .
     10.0.73.0/24   dev vlan73  proto kernel  scope link  src 10.0.73.1
     192.168.0.0/24 dev eth0    proto kernel  scope link  src 192.168.0.1
     10.0.100.0/24  dev vlan100 proto kernel  scope link  src 10.0.100.1
     10.0.99.0/24   dev vlan99  proto kernel  scope link  src 10.0.99.1
     10.0.98.0/24   dev vlan98  proto kernel  scope link  src 10.0.98.1
     10.0.97.0/24   dev vlan97  proto kernel  scope link  src 10.0.97.1
     10.0.96.0/24   dev vlan96  proto kernel  scope link  src 10.0.96.1
     10.1.0.0/16    dev eth1    proto kernel  scope link  src 10.1.1.18
     127.0.0.0/8    dev lo  scope link
     default via 10.1.254.254 dev eth1


There's an entry in the routing table for every VLAN (as there must be).
And, even if all these networks appeared on the same interface, there
would still need to be a routing rule for each.


------- Additional Comment #14 From Patrick Turley 2004-09-16 20:25

Everything I've learned so far leads me to conclude that the limit we're
running up against is the number of *primary* addresses. The problem
might be having too many primary addresses on a single interface, or it
might be the total number of primary addresses in the system.

Since each primary address requires a route, my first theory was that we
were putting too many entries in the routing table. I've since
discovered that the default size of the routing table is 32K, and
anecdotal evidence on the Internet suggests that a Linux router can deal
with this number comfortably.

Note, however, that N primary addresses implies at least N routing table
entries - but the reverse is not true. It's not likely that anyone else
is trying to configure more than a handful of addresses. The reason for
this apparent limit on the number of primary addresses is still a bit of
a mystery.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/