Scaling Max IP address limitation

Scaling Max IP address limitation

[David Jones]

Hi,
I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0. 
But I am hitting a max limit of 4000 IP address . Seems like there is a 
limiting variable in linux kernel (which one? ) that prevents from 
adding more IP addresses than 4096. What do I need to change in Linux 
kernel  ( and then recompile ) to be able to add more IP addresses than 
4K addresses per system? ..
Thanks,
-d

Re: Scaling Max IP address limitation

[Andrew Morton]

On Sun, 24 Jun 2007 12:20:01 -0500 David Jones <djones@rossove.com> wrote:

> Hi,
> I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0. 
> But I am hitting a max limit of 4000 IP address . Seems like there is a 
> limiting variable in linux kernel (which one? ) that prevents from 
> adding more IP addresses than 4096. What do I need to change in Linux 
> kernel  ( and then recompile ) to be able to add more IP addresses than 
> 4K addresses per system? ..

(cc netdev)

Re: Scaling Max IP address limitation

[Robert Iakobashvili]

Hi,

> On Sun, 24 Jun 2007 12:20:01 -0500 David Jones <djones@rossove.com> wrote:

> > I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0.
> > But I am hitting a max limit of 4000 IP address . Seems like there is a
> > limiting variable in linux kernel (which one? ) that prevents from
> > adding more IP addresses than 4096. What do I need to change in Linux
> > kernel  ( and then recompile ) to be able to add more IP addresses than
> > 4K addresses per system? ..


We are adding tens of thousand IPv4 addresses using netlink interface
without any problems. The maximum we added was 60K of secondary
IPv4 addresses. It consumes some memory, however.

We have also added thousands of IPv6. I will try to test, if there is any
limit for doing it.

-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Scaling Max IP address limitation

[David Jones]

Robert Iakobashvili wrote:
> Hi,
>
>> On Sun, 24 Jun 2007 12:20:01 -0500 David Jones <djones@rossove.com> 
>> wrote:
>
>> > I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0.
>> > But I am hitting a max limit of 4000 IP address . Seems like there 
>> is a
>> > limiting variable in linux kernel (which one? ) that prevents from
>> > adding more IP addresses than 4096. What do I need to change in Linux
>> > kernel  ( and then recompile ) to be able to add more IP addresses 
>> than
>> > 4K addresses per system? ..
>
>
> We are adding tens of thousand IPv4 addresses using netlink interface
> without any problems. The maximum we added was 60K of secondary
> IPv4 addresses. It consumes some memory, however.
>
> We have also added thousands of IPv6. I will try to test, if there is any
> limit for doing it.
>

I am using the "ip add " command looping sequentially up until RTNETLINK 
starts refusing to add more IP addresses. I am using a simple shell 
script to do the trick. One quick fact : If I exhaust 4K addresses on 
one port , then I can not add more IP's  ( v4/v6 alike ) on any port on 
the system. So seems like its a system wide limitation . Tried digging 
through the kernel source code but no luck so far. So definitely need 
pointers in this regard.
How are you adding via Netlink interface ?
Thanks,
-d

Re: Scaling Max IP address limitation

[Robert Iakobashvili]

David,

On 6/25/07, David Jones <djones@rossove.com> wrote:

> >> > I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0.
> >> > But I am hitting a max limit of 4000 IP address . Seems like there
> >> is a
> >> > limiting variable in linux kernel (which one? ) that prevents from
> >> > adding more IP addresses than 4096. What do I need to change in Linux
> >> > kernel  ( and then recompile ) to be able to add more IP addresses
> >> than 4K addresses per system? ..
> >
> I am using the "ip add " command looping sequentially up until RTNETLINK
> starts refusing to add more IP addresses.
> How are you adding via Netlink interface ?

Yes.

OK. Now it looks that I am reproducing something.

Running curl-loader with 60K.conf (edit the name of interface) configuration:
#ulimit -n 80000
#curl-loader -f ./conf-examples/60K.conf -w
 it adds successfully 60 000 secondary IPv4 addresses as seen by
#ip addr | wc -l

When I tryed adding IPv6 addresses, using ipv6.conf with addresses
range edited:
IP_ADDR_MIN=  2001:db8:fff5:1::1
IP_ADDR_MAX= 2001:db8:fff5:ffff::1

I am getting after initial successes some errors:
"rtnl_talk(): RTNETLINK answers: Cannot allocate memory"
and
#ip addr | wc-l is 8194.

8K addresses added and no more? It might be a memory issue. Y can dig
into the code and look into the allocation process and limits on the
kernel memory for IPv6.

The physical memory on my computer is 480 MB.
kernel is vanilla 2.6.20.7.

Try to see, what happens, when you increase the memory on your comp,
if an option.

-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Scaling Max IP address limitation

[Jan Engelhardt]

On Jun 25 2007 11:47, Robert Iakobashvili wrote:
>
> I am getting after initial successes some errors:
> "rtnl_talk(): RTNETLINK answers: Cannot allocate memory"
> and
> #ip addr | wc-l is 8194.

I'd be surprised if it was 4096 on x86 and 8192 on x86_64...


	Jan
-- 

Re: Scaling Max IP address limitation

[Robert Iakobashvili]

Hi

On 6/25/07, Jan Engelhardt <jengelh@computergmbh.de> wrote:
>
> On Jun 25 2007 11:47, Robert Iakobashvili wrote:
> >
> > I am getting after initial successes some errors:
> > "rtnl_talk(): RTNETLINK answers: Cannot allocate memory"
> > and
> > #ip addr | wc-l is 8194.
>
> I'd be surprised if it was 4096 on x86 and 8192 on x86_64...

Missed to mention: the CPU is Pentium-4.


-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Scaling Max IP address limitation

[Jan Engelhardt]

On Jun 25 2007 12:41, Robert Iakobashvili wrote:
>> > I am getting after initial successes some errors:
>> > "rtnl_talk(): RTNETLINK answers: Cannot allocate memory"
>> > and
>> > #ip addr | wc-l is 8194.
>>
>> I'd be surprised if it was 4096 on x86 and 8192 on x86_64...
>
> Missed to mention: the CPU is Pentium-4.

That's like saying you've got a SPARC. Or a MIPS. Or a PPC.
(I can't infer from your answer whether that is running 32 or 64-bit
kernel, because there are P4s with and without 64-bit extensions.)


	Jan
-- 

Re: Scaling Max IP address limitation

[Robert Iakobashvili]

On 6/25/07, Jan Engelhardt <jengelh@computergmbh.de> wrote:
>
> On Jun 25 2007 12:41, Robert Iakobashvili wrote:
> >> > I am getting after initial successes some errors:
> >> > "rtnl_talk(): RTNETLINK answers: Cannot allocate memory"
> >> > and
> >> > #ip addr | wc-l is 8194.
> >>
> >> I'd be surprised if it was 4096 on x86 and 8192 on x86_64...
> >
> > Missed to mention: the CPU is Pentium-4.
>
> That's like saying you've got a SPARC. Or a MIPS. Or a PPC.
> (I can't infer from your answer whether that is running 32 or 64-bit
> kernel, because there are P4s with and without 64-bit extensions.)

32/32

>         Jan
> --
>


-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Scaling Max IP address limitation

[David Jones]

Ok I have tried it on a Pentium-M ( 32 Bit ,) with 512 MB RAM and  Core 
2 Duo with 1Gig RAM  ( running SMP kernel , 2 CPUS) with same results. 
Cant go more than ~4K addresses. I have tried them with vanilla and 
custom kernels all 2.6.19+ versions.  Results are same on both systems , 
so thats the reason I am thinking that there is some limit in kernel 
source tree which I cant seem to find . Really appreciate your help in 
this regard.
Thanks,
-d

Robert Iakobashvili wrote:
> On 6/25/07, Jan Engelhardt <jengelh@computergmbh.de> wrote:
>>
>> On Jun 25 2007 12:41, Robert Iakobashvili wrote:
>> >> > I am getting after initial successes some errors:
>> >> > "rtnl_talk(): RTNETLINK answers: Cannot allocate memory"
>> >> > and
>> >> > #ip addr | wc-l is 8194.
>> >>
>> >> I'd be surprised if it was 4096 on x86 and 8192 on x86_64...
>> >
>> > Missed to mention: the CPU is Pentium-4.
>>
>> That's like saying you've got a SPARC. Or a MIPS. Or a PPC.
>> (I can't infer from your answer whether that is running 32 or 64-bit
>> kernel, because there are P4s with and without 64-bit extensions.)
>
> 32/32
>
>>         Jan
>> -- 
>>
>
>

Re: Scaling Max IP address limitation

[Robert Iakobashvili]

Hi David,

On 6/25/07, David Jones <djones@rossove.com> wrote:
>  Ok I have tried it on a Pentium-M ( 32 Bit ,) with 512 MB RAM and  Core
> 2 Duo with 1Gig RAM  ( running SMP kernel , 2 CPUS) with same results.
> Cant go more than ~4K addresses. I have tried them with vanilla and
> custom kernels all  2.6.19+ versions.  Results are same on both systems ,
> so thats the reason I am thinking that there is some limit in kernel
>
> >> On Jun 25 2007 12:41, Robert Iakobashvili wrote:
> >> >> > I am getting after initial successes some errors:
> >> >> > "rtnl_talk(): RTNETLINK answers: Cannot allocate memory"
> >> >> > and
> >> >> > #ip addr | wc-l is 8194.
>

Looking into 2.6.20 sources:

IPv6 addresses are added via rtnetlink by:
inet6_rtm_newaddr ()
inet6_addr_add ()
ipv6_add_addr (),
where inet6_ifaddr structure is allocated by kzalloc ()

IPv4 addresses are added via rtnetlink by:
inet_rtm_newaddr () with further __inet_insert_ifa ()
inet_rtm_newaddr () allocates in_ifaddr structure by kzalloc ()

Structure inet6_ifaddr  is ~85-95 bytes (depending on config options,
arch, etc),
structure in_ifaddr is ~55-60 bytes (also depending). It looks like
the case, when
size does matter.
:)

When adding secondary IPv4 addresses from /proc/slabinfo is seen, that
slab "size-64" is used, whereas for IPv6 addresses "size-128" is in use.

There should be a way to make some fine tuning and, most probably, memory
allocation for the right pool (in your case "size-128").
I do recall some echo to /proc/slabinfo, but there were some evil details.

Probably, some great people of the netdev list can advise you, how to
increase memory resources in the right direction.
Hopefully it explains the facts and directs you.

Take care,


Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Scaling Max IP address limitation

[Robert Iakobashvili]

> On Sun, 24 Jun 2007 12:20:01 -0500 David Jones <djones@rossove.com> wrote:
> > I am trying to add multiple IP addresses ( v6 ) to my FC7 box on eth0.
> > But I am hitting a max limit of 4000 IP address . Seems like there is a
> > limiting variable in linux kernel (which one? ) that prevents from
> > adding more IP addresses than 4096. What do I need to change in Linux
> > kernel  ( and then recompile ) to be able to add more IP addresses than
> > 4K addresses per system? ..

How are you doing this?

Could it be some IPv6 issue like scope?


-- 
Sincerely,
Robert Iakobashvili,
coroberti %x40 gmail %x2e com
...........................................................
http://curl-loader.sourceforge.net
A web testing and traffic generation tool.

Re: Scaling Max IP address limitation

[Kyle Moffett]

On Jun 24, 2007, at 13:20:01, David Jones wrote:
> Hi, I am trying to add multiple IP addresses ( v6 ) to my FC7 box  
> on eth0. But I am hitting a max limit of 4000 IP address . Seems  
> like there is a limiting variable in linux kernel (which one? )  
> that prevents from adding more IP addresses than 4096. What do I  
> need to change in Linux kernel  ( and then recompile ) to be able  
> to add more IP addresses than 4K addresses per system? ..

Do you really need that many IP addresses?  When somebody finally  
gets around to implementing REDIRECT support for ip6tables then you  
could just redirect them all to the same port on the local system.   
Then with a happy little getsockopt() you can find out the original  
IP address for use in whatever application you are running.  That's  
likely to be a thousand times more efficient than binary searching  
through 5000+ mostly-sequential IP addresses per received packet.

<Unrelated wishful thinking>
I keep having hopeful dreams that one day netfilter will grow support  
for cross-protocol NAT (IE: NAT a TCPv4 connection over TCPv6 to the  
IPv6-only local web server, or vice versa).  It would seem that would  
require a merged "xtables" program.

Having routing table operations, IPsec transformations, etc just be  
another step in the firewall rules would also be useful.  It would be  
handy to be able to "-j ROUTE", then "-j IPSEC", then "-j ROUTE"  
again, to re-route the now-encapsulated IPsec packets to their proper  
destination.  That would also eliminate the sort-of-hacky problems  
with destination network interface in the bridging code: "-j BRIDGE"  
might be another step in the process, and conceivably you could have  
independent bridge MAC tables too.  You'd probably also want "-j  
BRIDGE_TEST" and "-j ROUTE_TEST" to compute the output network  
interface without actually modifying the addresses.

That would also appear to get rid of the need for all tables other  
than "filter" and all predefined chains other than "INPUT" and  
"OUTPUT".  Default rules would be these:
nettables -A INPUT -j CONNTRACK
nettables -A INPUT -j LOCALMATCH
nettables -A INPUT --for-this-host -j ACCEPT
nettables -A INPUT -j OUTPUT
nettables -A OUTPUT -j ROUTE
nettables -A OUTPUT -j TRANSMIT

Forwarded packets would be sent right into the OUTPUT chain from the  
INPUT chain by appropriate rules.  Instead of turning off  
ip_forwarding in /proc/sys, you could just change the "-j OUTPUT" in  
the INPUT chain to "-j ACCEPT", and it would be impossible to forward  
packets.  I can't see any functionality that we have today which a  
mechanism like this wouldn't support, aside from the fact that it  
hands the admin a loaded nuclear missile aimed at their foot  
(Flushing the INPUT chain would basically be analogous to committing  
network suicide, although there exist other ways to do that with  
netfilter today.
</Unrelated wishful thinking>

Cheers,
Kyle Moffett

Re: Scaling Max IP address limitation

[David Stevens]

> <Unrelated wishful thinking>
> I keep having hopeful dreams that one day netfilter will grow support 
> for cross-protocol NAT (IE: NAT a TCPv4 connection over TCPv6 to the 
> IPv6-only local web server, or vice versa).  It would seem that would 
> require a merged "xtables" program.

        You don't actually need it (at least for easy cases like that),
because IPv6 defines IPv4 mapped IPv6 addresses of the form
::ffff:a.b.c.d. These will generate IPv4 packets for a.b.c.d, from
a v6 socket.
        Unless you're using v6only binding (a sysctl option), you can
connect to v6-only servers using a v4 network and a v4 address of the
server. The peer address on those connections will show up as a v4
mapped address, and all the traffic will be v4, but the socket layer
is all v6.

                                                                +-DLS

Re: Scaling Max IP address limitation

[Jan Engelhardt]

On Jun 24 2007 15:08, Kyle Moffett wrote:
>
> Do you really need that many IP addresses?  When somebody finally gets
> around to implementing REDIRECT support for ip6tables then you could
> just redirect them all to the same port on the local system.

The way I see it, it's: "if someone gets around to implement *IPv6 NAT*"
(which, if its designers were asked, is contrary to the idea of ipv6).

> <Unrelated wishful thinking>
> I keep having hopeful dreams that one day netfilter will grow support for
> cross-protocol NAT (IE: NAT a TCPv4 connection over TCPv6 to the IPv6-only
> local web server, or vice versa).  It would seem that would require a merged
> "xtables" program.
>
> Having routing table operations, IPsec transformations, etc just be
> another step in the firewall rules would also be useful.  It would be
> handy to be able to "-j ROUTE", then "-j IPSEC", then "-j ROUTE" again,
> to re-route the now-encapsulated IPsec packets to their proper
> destination.

Absolutely...

> That would also eliminate the sort-of-hacky problems with
> destination network interface in the bridging code:

Where's the hack? iptables operates on what it sees, and it sees br0.
The physdev match is justified IMO.

> "-j BRIDGE" might be
> another step in the process, and conceivably you could have independent
> bridge MAC tables too.

Whether a packet goes out a bridge (was that the intention of -j BRIDGE?)
is determined by the routing table, which, in most cases, is just a matter
of destination address.

> You'd probably also want "-j BRIDGE_TEST" and
> "-j ROUTE_TEST" to compute the output network interface without actually
> modifying the addresses.
>
> That would also appear to get rid of the need for all tables other than
> "filter" and all predefined chains other than "INPUT" and "OUTPUT".  Default
> rules would be these:
> nettables -A INPUT -j CONNTRACK
> nettables -A INPUT -j LOCALMATCH
> nettables -A INPUT --for-this-host -j ACCEPT
> nettables -A INPUT -j OUTPUT

I'd prefer if Linux outputted its packets by default :)

> nettables -A OUTPUT -j ROUTE
> nettables -A OUTPUT -j TRANSMIT

pkttables it is!

But this idea may have its benefit: by not restricting rules to certain
positions like currently, throughput could be achieved. "Evil packets"
f.e. could be dropped really early. (Well, you could also drop them early
_today_, but a DROP in the mangle table will everyone make their eyes roll
:-)



	Jan
-- 

Re: Scaling Max IP address limitation

[david]

On Sun, 24 Jun 2007, Jan Engelhardt wrote:

>
> On Jun 24 2007 15:08, Kyle Moffett wrote:
>>
>> Do you really need that many IP addresses?  When somebody finally gets
>> around to implementing REDIRECT support for ip6tables then you could
>> just redirect them all to the same port on the local system.
>
> The way I see it, it's: "if someone gets around to implement *IPv6 NAT*"
> (which, if its designers were asked, is contrary to the idea of ipv6).

true, but back in the real world it's sometimes desriable to hid _chich_ 
specific machine somethign comes from. so I expect that implementation of 
NAT is going tohappen at some point before it's widely deployed.

David Lang

Re: Scaling Max IP address limitation

[Jan Engelhardt]

On Jun 24 2007 13:44, david@lang.hm wrote:
>> On Jun 24 2007 15:08, Kyle Moffett wrote:
>> >
>> > Do you really need that many IP addresses?  When somebody finally gets
>> > around to implementing REDIRECT support for ip6tables then you could
>> > just redirect them all to the same port on the local system.
>>
>> The way I see it, it's: "if someone gets around to implement *IPv6 NAT*"
>> (which, if its designers were asked, is contrary to the idea of ipv6).
>
> true, but back in the real world it's sometimes desriable to hid _chich_
> specific machine somethign comes from. so I expect that implementation of NAT
> is going tohappen at some point before it's widely deployed.

Client-transparent SOCKS5 proxy. It already exists today! ;-)
(Not as performant as an in-kernel NAT, though.)


	Jan
-- 

Re: Scaling Max IP address limitation

[Kyle Moffett]

On Jun 24, 2007, at 15:58:54, Jan Engelhardt wrote:
> On Jun 24 2007 15:08, Kyle Moffett wrote:
>> Do you really need that many IP addresses?  When somebody finally  
>> gets around to implementing REDIRECT support for ip6tables then  
>> you could just redirect them all to the same port on the local  
>> system.
>
> The way I see it, it's: "if someone gets around to implement *IPv6  
> NAT*" (which, if its designers were asked, is contrary to the idea  
> of ipv6).

I totally agree.  On the other hand, you need REDIRECT for things  
like transparent proxies which by definition aren't visible as  
anything other than a router or bridge.

>> Having routing table operations, IPsec transformations, etc just  
>> be another step in the firewall rules would also be useful.  It  
>> would be handy to be able to "-j ROUTE", then "-j IPSEC", then "-j  
>> ROUTE" again, to re-route the now-encapsulated IPsec packets to  
>> their proper destination.
> Absolutely...
>
>> That would also eliminate the sort-of-hacky problems with  
>> destination network interface in the bridging code:
>
> Where's the hack? iptables operates on what it sees, and it sees  
> br0.  The physdev match is justified IMO.

The problem is this:
I want to be able to filter bridged network traffic *both* based on  
the IP layer *and* the physical device it's going to be routed out.   
Due to fundamental problems with a statically-ordered architecture,  
it's impossible to get both, see commit  
68df071a201f06b08cdc07111c6d4af918e64edd (found here: http:// 
lists.netfilter.org/pipermail/netfilter-devel/2006-December/ 
026388.html).  Basically if you want such cross-layer hooks, right  
now your *ONLY* choice is to use marks with 2 drawbacks:  (1)  There  
are a very small number of marks which must be carefully allocated by  
your firewall-setup script  (2)  Marks are inherently extremely  
fragile for passing data between layers.

>> "-j BRIDGE" might be another step in the process, and conceivably  
>> you could have independent bridge MAC tables too.
>
> Whether a packet goes out a bridge (was that the intention of -j  
> BRIDGE?) is determined by the routing table, which, in most cases,  
> is just a matter of destination address.

No, the intent of "-j BRIDGE" would be _after_ "-j ROUTE" and some  
kind of "-j ARP", to actually compute which physical port a given  
packet should go.

>> That would also appear to get rid of the need for all tables other  
>> than
>> "filter" and all predefined chains other than "INPUT" and  
>> "OUTPUT".  Default
>> rules would be these:
>> nettables -A INPUT -j CONNTRACK
>> nettables -A INPUT -j LOCALMATCH
>> nettables -A INPUT --for-this-host -j ACCEPT
>> nettables -A INPUT -j OUTPUT
>
> I'd prefer if Linux outputted its packets by default :)

Well the problem is this:  Do you want the packet accepted locally or  
forwarded.  If forwarded, how do you want it routed, and which  
physical port do you want it to go out?  Without a statically-coded  
ordering for all those things there is no way to say what the  
"default" is.  You would need some way to switch between iptables/ 
ip6tables (for compatibility) and pkttables/nettables (for advanced  
functionality).

> But this idea may have its benefit: by not restricting rules to  
> certain positions like currently, throughput could be achieved.  
> "Evil packets" f.e. could be dropped really early. (Well, you could  
> also drop them early _today_, but a DROP in the mangle table will  
> everyone make their eyes roll

It does give you a million more ways to shoot yourself in the foot.   
Some things would have constraints like "output device must be  
set" (BRIDGE/ARP, for example).  If you accidentally stuck non- 
constrained things in the wrong order you could get totally-non-IP- 
compliant behavior.  On the other hand, it does give you many choices  
about IPsec before or after ROUTING (or after one routing step and  
before another), etc.

Cheers,
Kyle Moffett

Re: Scaling Max IP address limitation

[Patrick McHardy]

Jan Engelhardt wrote:
> On Jun 24 2007 15:08, Kyle Moffett wrote:
> 
>>Do you really need that many IP addresses?  When somebody finally gets
>>around to implementing REDIRECT support for ip6tables then you could
>>just redirect them all to the same port on the local system.
> 
> 
> The way I see it, it's: "if someone gets around to implement *IPv6 NAT*"
> (which, if its designers were asked, is contrary to the idea of ipv6).


You don't necessarily need NAT for REDIRECT, so we might actually have
an ip6tables REDIRECT some day. Check out the current TPROXY patches
for an example how to do it without NAT in case you're interested.