What's the right way to use a *large* number of source addresses?

What's the right way to use a *large* number of source addresses?

[Niels Möller]

Hi,

I have a client doing traffic generation for load testing. When they
experienced performance problems with assigning a large number (say,
100000) ip addresses to an interface, they wrote a custom and
proprietary source NAT kernel module which let's you set any desired
source address on a socket, and then sets up SNAT to that address. This
was a couple of years ago, and it appears to have worked fine.

However, the code is a bit complex, and duplicates functionality in the
iptables SNAT target and the connection tracking machinery in the
current kernel. If I could relicense the module under a free license, I
suspect it would be shot down for technical reasons.

So now I'm trying to figure out what's the Right Way to enable traffic
generations with a large number of source addresses, to possibly retire
the proprietary kernel module. I see a couple of different approaches:

1. Simply assign all addresses to be used to the interface, fixing any
   remaining performance problems.

   I've done a simple benchmark with a script assigning n addresses
   using "ip address add", and this seems to have O(n^2) complexity.
   E.g, assigning n=25500 addresses took 26 s, and doubling n, assigning
   51000 addresses, took 122 s, 4.6 times longer. Which isn't
   necessarily a problems once all the addresses are assigned, but it
   sounds a bit like there's a linear datastructure in there, not
   intended for a large number of addresses.

   A way to add an address range (or a prefix), using a *single* entry
   of whatever datastructures are used, would help.

2. Do source NAT. I think the current SNAT target does almost everything
   needed. It could be extended with some setsockopt to set the desired
   address on a per socket basis. Not sure where to store that info;
   either associate the desired address with the socket, and have the
   SNAT module look for that. Or maybe one could have setsockopt create
   a conntrack entry in advance, prior to connect.

   Main drawback of using NAT is the overhead for connection tracking;
   it would be preferable if the only per-connection state needed is the
   socket itself.

3. Just set the desired local address with the bind call. Currently,
   this gives an EADDRNOTAVAIL error, so the first step would be some
   option to allow arbitrary source addresses.

   For an arbitrary source address, the network stack can't guess the
   intended interface. So one would also need to support something like
   SO_BINDTODEVICE to tell it explicitly.

   And for replies to be passed up to the transport layer, one must set
   up some processing of incoming packets to deliver them to the local
   machine. It's very unclear to me if there's any good way to do that,
   maybe one needs a conntrack entry for each connection just like with
   SNAT. Even with conntrack I think this approach is a bit cleaner than
   SNAT, in that the transport-layer 5-tuple would be based on the
   address that really is used on the wire.

What do you think? From a user perspective, I think I'd prefer either
(1), or (3) with a single setsockopt call which means "I'm going to use
an arbitrary source address. Transmit my packets over interface X, and
arrange the processing of incoming packets so that replies arrive to
this socket.", and then specify the desired source address with bind()
as usual.

Best regards,
/Niels Möller

Re: What's the right way to use a *large* number of source addresses?

[Florian Westphal]

Niels Möller <nisse@southpole.se> wrote:
> So now I'm trying to figure out what's the Right Way to enable traffic
> generations with a large number of source addresses, to possibly retire
> the proprietary kernel module. I see a couple of different approaches:

[..]

> 3. Just set the desired local address with the bind call. Currently,
>    this gives an EADDRNOTAVAIL error, so the first step would be some
>    option to allow arbitrary source addresses.

See 'Making non-local sockets work' in Documentation/networking/tproxy.txt

RE: What's the right way to use a *large* number of source addresses?

[Lukas Tribus]

>> 3. Just set the desired local address with the bind call. Currently,
>> this gives an EADDRNOTAVAIL error, so the first step would be some
>> option to allow arbitrary source addresses.
>
> See 'Making non-local sockets work' in Documentation/networking/tproxy.txt


Yes, its a simple as setting IP_FREEBIND and then binding to the specific
source IP.

setsockopt(fd, SOL_IP, IP_FREEBIND, &one, sizeof(one))


You don't need to specify the interface. Just make sure your network
routes the return traffic back to your box.

Works for IPv6 as well (3.3+).



Lukas

 		 	   		  

Re: What's the right way to use a *large* number of source addresses?

[Niels Möller]

Florian Westphal <fw@strlen.de> writes:

> Niels Möller <nisse@southpole.se> wrote:

>> 3. Just set the desired local address with the bind call. Currently,
>>    this gives an EADDRNOTAVAIL error, so the first step would be some
>>    option to allow arbitrary source addresses.
>
> See 'Making non-local sockets work' in Documentation/networking/tproxy.txt

Great! Sounds like it should work fine with no additional kernel hacking.

Lukas Tribus <luky-37@hotmail.com> writes:

> Yes, its a simple as setting IP_FREEBIND and then binding to the specific
> source IP.
>
> setsockopt(fd, SOL_IP, IP_FREEBIND, &one, sizeof(one))

This sounds even easier, but I'm not sure it works. For my first test, I just tried

  sysctl net.ipv4.ip_nonlocal_bind=1

(which is the global equivalent of IP_FREEBIND). Then bind succeeds, but
connect fails with EINVAL,

  socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
  bind(3, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("10.200.0.1")}, 16) = 0
  connect(3, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("10.1.0.2")}, 16) = -1 EINVAL (Invalid argument)

I'll do some more experiments, using IP_TRANSPARENT, and setting up the
suggested iptables rules.

Thanks,
/Niels

Re: What's the right way to use a *large* number of source addresses?

[sowmini varadhan]

On Fri, May 23, 2014 at 5:38 AM, Niels Möller <nisse@southpole.se> wrote:


> 1. Simply assign all addresses to be used to the interface, fixing any
>    remaining performance problems.
>
>    I've done a simple benchmark with a script assigning n addresses
>    using "ip address add", and this seems to have O(n^2) complexity.
>    E.g, assigning n=25500 addresses took 26 s, and doubling n, assigning
>    51000 addresses, took 122 s, 4.6 times longer. Which isn't
>    necessarily a problems once all the addresses are assigned, but it
>    sounds a bit like there's a linear datastructure in there, not
>    intended for a large number of addresses.

I think the issue here is the netlink API: if you try to do the same thing
with ifconfig (instead of /sbin/ip), you'll find things are much faster,
which seems paradoxical because ip(7) and netlink sockets are the
recommended config paths. When I ran into this, it turned out that the
difference was due to the implementation- the ioctl path was able to
use the ifr_name efficiently to identify the target interface, unlike
the netlink config path.

But to solve  your specific problem, since you are implementing
a packet generator, wouldnt it be easier to craft the packet and feed
it over RAW or PF_PACKET sockets, allowing you more flexibility
(if you need it down the road) to set all fields, not just the
IP source address? Might mean a bit more work in your user-space,
to compute checksums etc. though.

--Sowmini

Re: What's the right way to use a *large* number of source addresses?

[Niels Möller]

Florian Westphal <fw@strlen.de> writes:

> Niels Möller <nisse@southpole.se> wrote:

>> 3. Just set the desired local address with the bind call. Currently,
>>    this gives an EADDRNOTAVAIL error, so the first step would be some
>>    option to allow arbitrary source addresses.
>
> See 'Making non-local sockets work' in Documentation/networking/tproxy.txt

I'd just like to say that I followed these docs, and I got a working
connection. Thanks again, I really couldn't locate this info on my own.

I do find the EINVAL error I got from connect, when only using sysctl
net.ipv4.ip_nonlocal_bind=1, and using bind with a nonlocal address, a
bit puzzling. I don't see any obviously better error to return, though.

Regards,
/Niels

Re: What's the right way to use a *large* number of source addresses?

[Eric Dumazet]

On Fri, 2014-05-23 at 08:53 -0400, sowmini varadhan wrote:

> I think the issue here is the netlink API: if you try to do the same thing
> with ifconfig (instead of /sbin/ip), you'll find things are much faster,
> which seems paradoxical because ip(7) and netlink sockets are the
> recommended config paths. When I ran into this, it turned out that the
> difference was due to the implementation- the ioctl path was able to
> use the ifr_name efficiently to identify the target interface, unlike
> the netlink config path.

Only if you launch on "ip" command for one IP address it might be slow.

ip populates a cache at startup.

Use the batch mode, and it will be much faster than ifconfig, as
ifconfig does not support this mode (you need one fork()/exec() per IP
address)

ip -batch filename

Re: What's the right way to use a *large* number of source addresses?

[Cong Wang]

On Fri, May 23, 2014 at 2:38 AM, Niels Möller <nisse@southpole.se> wrote:
> Hi,
....
>
> So now I'm trying to figure out what's the Right Way to enable traffic
> generations with a large number of source addresses, to possibly retire
> the proprietary kernel module. I see a couple of different approaches:
>
>

Isn't pktgen is what you are looking for? It provides src_min and src_max
to specify the range of source IPv4 addresses. But it does't understand
TCP, only UDP.

Re: What's the right way to use a *large* number of source addresses?

[Jamal Hadi Salim]

On 05/23/14 05:38, Niels Möller wrote:
> Hi,
>


>     Main drawback of using NAT is the overhead for connection tracking;
>     it would be preferable if the only per-connection state needed is the
>     socket itself.
>

If stateless nat is sufficient take a look at tc nat.

cheers,
jamal

Re: What's the right way to use a *large* number of source addresses?

[Jamal Hadi Salim]

On 05/23/14 10:14, Eric Dumazet wrote:

> Use the batch mode, and it will be much faster than ifconfig, as
> ifconfig does not support this mode (you need one fork()/exec() per IP
> address)
>
> ip -batch filename
>

The address dumping algorithm is a very likely contributor as well.
It tries to remember indices and then skips on the next iteration
all the way to where it left off.... has never been a big deal until
someone tries a substantial number of addresses.

cheers,
jamal

Re: What's the right way to use a *large* number of source addresses?

[Richard Weinberger]

On Sat, May 24, 2014 at 1:58 PM, Jamal Hadi Salim <jhs@mojatatu.com> wrote:
> On 05/23/14 05:38, Niels Möller wrote:
>>
>> Hi,
>>
>
>
>>     Main drawback of using NAT is the overhead for connection tracking;
>>     it would be preferable if the only per-connection state needed is the
>>     socket itself.
>>
>
> If stateless nat is sufficient take a look at tc nat.

How to use this?
My tc's manpage does not know anything about nat.

Maybe I'm mistaken but IIRC all non-netfilter nat code has been removed.

-- 
Thanks,
//richard

Re: What's the right way to use a *large* number of source addresses?

[Jamal Hadi Salim]

On 05/24/14 10:44, Richard Weinberger wrote:

> How to use this?
> My tc's manpage does not know anything about nat.
>

Start writing one and i will help.

> Maybe I'm mistaken but IIRC all non-netfilter nat code has been removed.
>

This is stateless nat nothing to do with netfilter (much more 
efficient). Here's some documentation for you ;->
----
hadi@jhs-1:~$ $TC filter add $DEV parent ffff: protocol ip u32 match ip 
dport 8080 0xffff flowid 1:1 action nat help
Illegal nat construct (help)
Usage: ... nat NAT
NAT := DIRECTION OLD NEW
DIRECTION := { ingress | egress }
OLD := PREFIX
NEW := ADDRESS
bad action parsing
parse_action: bad value (2:nat)!
Illegal "action"
-----

Google will give you more examples, but here's one:

Lets add a rule on the public facing bridge:

hadi@jhs-1:~$ $TC filter add dev virb0 parent ffff: protocol ip u32 
match ip dport 8080 0xffff flowid 1:1 action nat ingress 
192.168.10.15/32 10.0.0.15/32


now display it:
hadi@jhs-1:~$ $TC -s filter ls dev virbr0 parent ffff: protocol ip
filter pref 49152 u32
filter pref 49152 u32 fh 800: ht divisor 1
filter pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1
   match 00001f90/0000ffff at 20
	action order 1:  nat ingress 192.168.10.15/32 10.0.0.15 pass
         installed 96 sec used 96 sec	Action statistics:
	  Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
	  backlog 0b 0p requeues 0

You will need to add a reverse translation for internal->external

etc

cheers,
jamal

Re: What's the right way to use a *large* number of source addresses?

[Richard Weinberger]

Am 24.05.2014 17:13, schrieb Jamal Hadi Salim:
> On 05/24/14 10:44, Richard Weinberger wrote:
> 
>> How to use this?
>> My tc's manpage does not know anything about nat.
>>
> 
> Start writing one and i will help.

*grml* yet another awesome and undocumented feature buried in the network stack.

>> Maybe I'm mistaken but IIRC all non-netfilter nat code has been removed.
>>
> 
> This is stateless nat nothing to do with netfilter (much more efficient). Here's some documentation for you ;->

Digging through the code shows that this feature is rather new.
I had the old nat functionality of the routing subsystem in mind.

> ----
> hadi@jhs-1:~$ $TC filter add $DEV parent ffff: protocol ip u32 match ip dport 8080 0xffff flowid 1:1 action nat help
> Illegal nat construct (help)
> Usage: ... nat NAT
> NAT := DIRECTION OLD NEW
> DIRECTION := { ingress | egress }
> OLD := PREFIX
> NEW := ADDRESS
> bad action parsing
> parse_action: bad value (2:nat)!
> Illegal "action"
> -----
> 
> Google will give you more examples, but here's one:
> 
> Lets add a rule on the public facing bridge:
> 
> hadi@jhs-1:~$ $TC filter add dev virb0 parent ffff: protocol ip u32 match ip dport 8080 0xffff flowid 1:1 action nat ingress 192.168.10.15/32 10.0.0.15/32
> 
> 
> now display it:
> hadi@jhs-1:~$ $TC -s filter ls dev virbr0 parent ffff: protocol ip
> filter pref 49152 u32
> filter pref 49152 u32 fh 800: ht divisor 1
> filter pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1
>   match 00001f90/0000ffff at 20
>     action order 1:  nat ingress 192.168.10.15/32 10.0.0.15 pass
>         installed 96 sec used 96 sec    Action statistics:
>       Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>       backlog 0b 0p requeues 0
> 
> You will need to add a reverse translation for internal->external

Thanks for the howto.

Thanks,
//richard

Re: What's the right way to use a *large* number of source addresses?

[David Miller]

From: Richard Weinberger <richard@nod.at>
Date: Sat, 24 May 2014 18:02:23 +0200

> Digging through the code shows that this feature is rather new.

It's been there for several years, it's not new.

Re: What's the right way to use a *large* number of source addresses?

[Jamal Hadi Salim]

On 05/24/14 12:02, Richard Weinberger wrote:

> Digging through the code shows that this feature is rather new.
> I had the old nat functionality of the routing subsystem in mind.
>

This was the replacement for that. It is at least as old as
a grade one kid.

cheers,
jamal

Re: What's the right way to use a *large* number of source addresses?

[Niels Möller]

Cong Wang <cwang@twopensource.com> writes:

> On Fri, May 23, 2014 at 2:38 AM, Niels Möller <nisse@southpole.se> wrote:
>>
>> So now I'm trying to figure out what's the Right Way to enable traffic
>> generations with a large number of source addresses, to possibly retire
>> the proprietary kernel module. I see a couple of different approaches:
>
> Isn't pktgen is what you are looking for?

No, I'd like to have a user-space application using a plain tcp socket
as the endpoint.

In my experience, pktgen is useful if you want to test the limits of raw
packet performance of network cards and drivers, or generate load on
networking equipment. But not for generating load on a real application,
where the data needs to follow some higher level protocol.

Regards,
/Niels