Question about limited primary addresses

Question about limited primary addresses

[Patrick Turley]

I am trying to configure a Linux box with all possible VLANs (4094 of 
them), and a subnet on each VLAN. Creating the VLANs isn't a problem. 
But, when I try to use "ip addr add ..." commands to assign an IP 
address to each VLAN interface, I get to do about 280 of them before all 
the interfaces on the box become unresponsive. I've also tried to assign 
the same IP addresses all to eth0, and I get the same result.

The problem seems to hinge on the distinction between primary and 
secondary addresses.

If you're reading the LARTC list, then you've probably had the need 
(e.g., load testing) to assign thousands of IP addresses to a single 
interface. However, it's usually the case that one is assigning multiple 
addresses that all belong to the same subnet. In that case, the first 
such IP address is "primary" and all the additional addresses that 
belong to the same subnet are "secondary". My investigations tell me 
that "secondary" addresses are much lighter-weight. On the other hand, I 
don't seem to be able to assign more than about 280 "primary" addresses 
in the entire system before all the interfaces become unresponsive.

For additional information about "primary" vs. "secondary" addresses, 
take a look at:

http://www.linux-ip.net/html/linux-ip.html#tools-ip-address-add
http://www.linux-ip.net/gl/ip-cref/node33.html

Does anyone know why there appears to be a limit on the number of 
networks? Does anyone know where in the code this limit arises? Does 
anyone know a way to get around this limit?

Re: Question about limited primary addresses

[Jason Opperisano]

On Sun, 2004-09-19 at 23:35, Patrick Turley wrote:
> I am trying to configure a Linux box with all possible VLANs (4094 of 
> them), and a subnet on each VLAN. Creating the VLANs isn't a problem. 
> But, when I try to use "ip addr add ..." commands to assign an IP 
> address to each VLAN interface, I get to do about 280 of them before all 
> the interfaces on the box become unresponsive. I've also tried to assign 
> the same IP addresses all to eth0, and I get the same result.
> 
> The problem seems to hinge on the distinction between primary and 
> secondary addresses.
> 
> If you're reading the LARTC list, then you've probably had the need 
> (e.g., load testing) to assign thousands of IP addresses to a single 
> interface. However, it's usually the case that one is assigning multiple 
> addresses that all belong to the same subnet. In that case, the first 
> such IP address is "primary" and all the additional addresses that 
> belong to the same subnet are "secondary". My investigations tell me 
> that "secondary" addresses are much lighter-weight. On the other hand, I 
> don't seem to be able to assign more than about 280 "primary" addresses 
> in the entire system before all the interfaces become unresponsive.
> 
> For additional information about "primary" vs. "secondary" addresses, 
> take a look at:
> 
> http://www.linux-ip.net/html/linux-ip.html#tools-ip-address-add
> http://www.linux-ip.net/gl/ip-cref/node33.html
> 
> Does anyone know why there appears to be a limit on the number of 
> networks? Does anyone know where in the code this limit arises? Does 
> anyone know a way to get around this limit?

i'm not aware of any limit.  i believe 2.0/2.2 had an alias limit of 256
per interface, but i could be way off base...  there's probably
something going on with you particular machine (i.e. memory problem).

my test to prove there's not a limit at 4096:

$ uname -a
Linux wolfpack 2.4.22-1.2199.nptl.ipsec.pom #2 Wed Aug 11 13:55:37 EDT
2004 i686 i686 i386 GNU/Linux

$ sudo ip addr add 220.1.1.1/24 dev eth0

$ ip -o -4 addr sh eth0
2: eth0    inet 220.1.1.1/24 scope global eth0

$ bash test	# see the contents of "test" below

$ ip -4 addr sh eth0 | wc -l
   4354

box is still responsive on random sampling of IP's (17.247.1.1,
16.212.1.1, 10.218.1.1, etc)...

contents of "test" are:

----
i=1
function addIP() {
  j=0
  while [ $j -le 255 ]; do
    sudo ip addr add $1.$j.1.1/24 dev eth0
    let "j = $j + 1"
  done
}

while [ $i -le 17 ]; do
   addIP $i
   let "i = $i + 1"
done
----

essentially--this script is adding IP's of 1.[0-255].1.1/24,
2.[0-255].1.1/24...up to 17.[0-255].1.1/24...for a total of 256*17 =
4352 IP's, all on different networks, all on eth0.

this is on my laptop i'm using to write this email (P4 2.2 GHz, 768 MB
RAM, FC1)

if this is not a valid reproduction of what you're trying to do, let me
know.

-j

-- 
Jason Opperisano <opie@817west.com>

Re: Question about limited primary addresses

[Patrick Turley]

Jason Opperisano wrote:
 > On Sun, 2004-09-19 at 23:35, Patrick Turley wrote:
 >
 >>I am trying to configure a Linux box with all possible VLANs (4094 of
 >>them), and a subnet on each VLAN. Creating the VLANs isn't a problem.
 >>But, when I try to use "ip addr add ..." commands to assign an IP
 >>address to each VLAN interface, I get to do about 280 of them before all
 >>the interfaces on the box become unresponsive. I've also tried to assign
 >>the same IP addresses all to eth0, and I get the same result.
.
.
.
 > i'm not aware of any limit.  i believe 2.0/2.2 had an alias limit of 256
 > per interface, but i could be way off base...  there's probably
 > something going on with you particular machine (i.e. memory problem).
 >
 > my test to prove there's not a limit at 4096:
.
.
.
 > contents of "test" are:
 >
 > ----
 > i=1
 > function addIP() {
 >   j=0
 >   while [ $j -le 255 ]; do
 >     sudo ip addr add $1.$j.1.1/24 dev eth0
 >     let "j = $j + 1"
 >   done
 > }
 >
 > while [ $i -le 17 ]; do
 >    addIP $i
 >    let "i = $i + 1"
 > done
 > ----

First of all, thank you very much for the time you spent investigating 
this problem.

Your script works fine on my test computer. I was able to assign all 
4000+ addresses with no problem.

I captured the commands emitted by your script and the commands emitted 
by our software and began the process of making them look more and more 
like each other until the essential difference became clear. I finally 
got to the point where the *only* difference was the IP addresses.

The addresses assigned by your script are (as you said) 
[1-17].[1-255].1.1/24.

The addresses I was assigning were 10.[1-2].[1-255].1/24.

So, the major difference seems to be that all my addresses were confined 
to the 10/8 subnet. It may be important to note that some of the 
addresses generated by your script *did* lie in that range, but the 
majority did not.

I adjusted your script very slightly to put all the addresses in the 
10/8 subnet:


     i=1
     function addIP() {
       j=0
       while [ $j -le 255 ]; do
         sudo ip addr add 10.$1.$j.1/24 dev eth0
         let "j = $j + 1"
       done
     }

     while [ $i -le 2 ]; do
        addIP $i
        let "i = $i + 1"
     done


When I ran this version, I saw the very same failure I've been seeing 
all along.

Would it be possible for you to run the modified version of this script 
on your machine and see if you have any problem?

Thanks again for your time.

Re: Question about limited primary addresses

[Jason Opperisano]

On Tue, 2004-09-21 at 14:51, Patrick Turley wrote:
> I adjusted your script very slightly to put all the addresses in the 
> 10/8 subnet:
> 
> 
>      i=1
>      function addIP() {
>        j=0
>        while [ $j -le 255 ]; do
>          sudo ip addr add 10.$1.$j.1/24 dev eth0
>          let "j = $j + 1"
>        done
>      }
> 
>      while [ $i -le 2 ]; do
>         addIP $i
>         let "i = $i + 1"
>      done
> 
> 
> When I ran this version, I saw the very same failure I've been seeing 
> all along.
> 
> Would it be possible for you to run the modified version of this script 
> on your machine and see if you have any problem?
> 
> Thanks again for your time.

not trying to complicate things, but i had to test this on a diff
machine (same HW, but running gentoo instead of FC1--can't blow up my
laptop right this moment).

details:

# uname -a
Linux vmg2 2.4.26-gentoo-r9 #2 Fri Sep 3 07:13:35 EDT 2004 i686 Intel(R)
Pentium(R) 4 CPU 2.20GHz GenuineIntel GNU/Linux

# ip -4 add sh eth0 | wc -l
513

# ip -4 add sh eth0 | head
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    inet 10.1.0.1/24 scope global eth0
    inet 10.1.1.1/24 scope global eth0
    inet 10.1.2.1/24 scope global eth0
    inet 10.1.3.1/24 scope global eth0
    inet 10.1.4.1/24 scope global eth0
    inet 10.1.5.1/24 scope global eth0
    inet 10.1.6.1/24 scope global eth0
    inet 10.1.7.1/24 scope global eth0
    inet 10.1.8.1/24 scope global eth0

# ip -4 add sh eth0 | tail
    inet 10.2.246.1/24 scope global eth0
    inet 10.2.247.1/24 scope global eth0
    inet 10.2.248.1/24 scope global eth0
    inet 10.2.249.1/24 scope global eth0
    inet 10.2.250.1/24 scope global eth0
    inet 10.2.251.1/24 scope global eth0
    inet 10.2.252.1/24 scope global eth0
    inet 10.2.253.1/24 scope global eth0
    inet 10.2.254.1/24 scope global eth0
    inet 10.2.255.1/24 scope global eth0

from a machine assigned 10.1.1.100/16 and 10.2.1.100/16, i can ping:

10.1.0.1, 10.1.1.1, 10.1.2.1, 10.1.3.1, 10.1.4.1, 10.2.254.1, 10.2.25.1,
10.1.25.1, 10.2.255.1

...on the test machine with all the 10.[1-2].[0-255].1/24 addresses.

we might be going about this all wrong...are you sure the problem is the
linux box with all the IP's, and not something else?  perhaps you're
overflowing the MAC table of a connected switch (just a thought)?

-j

-- 
Jason Opperisano <opie@817west.com>

Re: Question about limited primary addresses

[Patrick Turley]

Jason Opperisano wrote:
 > On Tue, 2004-09-21 at 14:51, Patrick Turley wrote:
 > # uname -a
 > Linux vmg2 2.4.26-gentoo-r9 #2 Fri Sep 3 07:13:35 EDT 2004 i686 Intel(R)
 > Pentium(R) 4 CPU 2.20GHz GenuineIntel GNU/Linux
 >
 > # ip -4 add sh eth0 | wc -l
 > 513
 >
 > # ip -4 add sh eth0 | head
 > 2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
 >     inet 10.1.0.1/24 scope global eth0
 >     inet 10.1.1.1/24 scope global eth0
 >     inet 10.1.2.1/24 scope global eth0
 >     inet 10.1.3.1/24 scope global eth0
 >     inet 10.1.4.1/24 scope global eth0
 >     inet 10.1.5.1/24 scope global eth0
 >     inet 10.1.6.1/24 scope global eth0
 >     inet 10.1.7.1/24 scope global eth0
 >     inet 10.1.8.1/24 scope global eth0
 >
 > # ip -4 add sh eth0 | tail
 >     inet 10.2.246.1/24 scope global eth0
 >     inet 10.2.247.1/24 scope global eth0
 >     inet 10.2.248.1/24 scope global eth0
 >     inet 10.2.249.1/24 scope global eth0
 >     inet 10.2.250.1/24 scope global eth0
 >     inet 10.2.251.1/24 scope global eth0
 >     inet 10.2.252.1/24 scope global eth0
 >     inet 10.2.253.1/24 scope global eth0
 >     inet 10.2.254.1/24 scope global eth0
 >     inet 10.2.255.1/24 scope global eth0
 >
 > from a machine assigned 10.1.1.100/16 and 10.2.1.100/16, i can ping:
 >
 > 10.1.0.1, 10.1.1.1, 10.1.2.1, 10.1.3.1, 10.1.4.1, 10.2.254.1, 10.2.25.1,
 > 10.1.25.1, 10.2.255.1
 >
 > ...on the test machine with all the 10.[1-2].[0-255].1/24 addresses.

I've found that ping is not a good test. Even with the networking 
broken, ping still works. Can you try to SSH/telnet/ftp/foo to the test 
machine?

Re: Question about limited primary addresses

[Patrick Turley]

I've discovered the source of this problem. Unfortunately, the cause is 
so embarassing that I'm not able to discuss it. I will now go commit 
sepuku. Thank you for your time and kind attention.

Patrick Turley wrote:
> Jason Opperisano wrote:
>  > On Tue, 2004-09-21 at 14:51, Patrick Turley wrote:
>  > # uname -a
>  > Linux vmg2 2.4.26-gentoo-r9 #2 Fri Sep 3 07:13:35 EDT 2004 i686 Intel(R)
>  > Pentium(R) 4 CPU 2.20GHz GenuineIntel GNU/Linux
>  >
>  > # ip -4 add sh eth0 | wc -l
>  > 513
>  >
>  > # ip -4 add sh eth0 | head
>  > 2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
>  >     inet 10.1.0.1/24 scope global eth0
>  >     inet 10.1.1.1/24 scope global eth0
>  >     inet 10.1.2.1/24 scope global eth0
>  >     inet 10.1.3.1/24 scope global eth0
>  >     inet 10.1.4.1/24 scope global eth0
>  >     inet 10.1.5.1/24 scope global eth0
>  >     inet 10.1.6.1/24 scope global eth0
>  >     inet 10.1.7.1/24 scope global eth0
>  >     inet 10.1.8.1/24 scope global eth0
>  >
>  > # ip -4 add sh eth0 | tail
>  >     inet 10.2.246.1/24 scope global eth0
>  >     inet 10.2.247.1/24 scope global eth0
>  >     inet 10.2.248.1/24 scope global eth0
>  >     inet 10.2.249.1/24 scope global eth0
>  >     inet 10.2.250.1/24 scope global eth0
>  >     inet 10.2.251.1/24 scope global eth0
>  >     inet 10.2.252.1/24 scope global eth0
>  >     inet 10.2.253.1/24 scope global eth0
>  >     inet 10.2.254.1/24 scope global eth0
>  >     inet 10.2.255.1/24 scope global eth0
>  >
>  > from a machine assigned 10.1.1.100/16 and 10.2.1.100/16, i can ping:
>  >
>  > 10.1.0.1, 10.1.1.1, 10.1.2.1, 10.1.3.1, 10.1.4.1, 10.2.254.1, 10.2.25.1,
>  > 10.1.25.1, 10.2.255.1
>  >
>  > ...on the test machine with all the 10.[1-2].[0-255].1/24 addresses.
> 
> I've found that ping is not a good test. Even with the networking 
> broken, ping still works. Can you try to SSH/telnet/ftp/foo to the test 
> machine?
> 
> 
>