ports beeing reused too fast

ports beeing reused too fast

[Octavian Purdila]

[-- Attachment #1: Type: text/plain, Size: 2073 bytes --]


Hi,

We've been running into an issue where a firewall would drop packets when an 
moderate (~360) connection rate was going through it. It looks like the 
firewall is dropping the SYNs that reuse ports "too fast". 

We have no issues with Linux 2.6.7, so I guess the behavior changed because of 
this this commit:

commit 6df716340da3a6fdd33d73d7ed4c6f7590ca1c42
Author: Stephen Hemminger <shemminger@osdl.org>
Date:   Thu Nov 3 16:33:23 2005 -0800

    [TCP/DCCP]: Randomize port selection


Now, I did some tests to confirm my suspicion. Basically, I am simulating a 
connection rate test (I've attached the .c to this email) by opening up 
connections and closing them - one at a time, and noting down the ports used, 
then looking for duplicate ports and printing the distance between the 
connection no.

Here is one of the runs, which make 1000 iterations:

listening (port 1242)
 port reused: 38203: distance 578 (624,46)
 port reused: 55693: distance 85 (147,62)
 port reused: 38269: distance 803 (872,69)
 port reused: 46239: distance 249 (344,95)
 port reused: 40981: distance 215 (319,104)
 port reused: 46246: distance 524 (641,117)
 port reused: 43990: distance 378 (498,120)
 port reused: 53766: distance 52 (232,180)
 port reused: 44199: distance 194 (383,189)
 port reused: 59464: distance 173 (384,211)
 port reused: 44417: distance 264 (492,228)
 port reused: 56989: distance 229 (553,324)
 port reused: 60117: distance 69 (394,325)
 port reused: 44549: distance 179 (566,387)
 port reused: 39213: distance 300 (801,501)
 port reused: 60166: distance 152 (671,519)
 port reused: 44178: distance 108 (712,604)
 port reused: 46516: distance 6 (792,786)
 port reused: 55754: distance 95 (969,874)
19 ports were being reused

Running the same test on 2.6.7 yields a "0 ports were being reused" on all 
tests that I've ran (10 or so).

Isn't it desirable to have the behavior from 2.6.7? 

I've looked over the  code and it looks right, so maybe net_random() is not 
random enough? Or maybe there are side effects because of the % port_range?

Thanks,
tavi


[-- Attachment #2: port-reuse.c --]
[-- Type: text/x-csrc, Size: 2365 bytes --]

#include <stdio.h>
#include <sys/types.h>
#include <netinet/ip.h>
#include <sys/socket.h>
#include <unistd.h>

#define N 1000
int ports[N], listen_port = 1234;

/* find duplicate ports and show distance between them */
int analyze(void)
{
	int i, j, n = 0;

	for(i = 0; i < N; i++) {
		for(j = i + 1; j < N; j++) {
			if (ports[i] != ports[j])
				continue;
			printf(" port reused: %d: distance %d (%d,%d)\n", ports[i], j-i, j, i);
			n++;
		}
	}

	printf("%d ports were being reused\n", n);

	return 0;
}

int client(void)
{
	int sock, i = 0;
	size_t tmp;
	struct sockaddr_in addr_bind = {
		.sin_family = AF_INET,
		.sin_addr.s_addr = INADDR_ANY,
		.sin_port = 0,
	}, addr_connect = {
		.sin_family = AF_INET,
		.sin_addr.s_addr = htonl(0x7f000001),
		.sin_port = htons(listen_port),
	}, addr_sockname;

	sleep(1);

 loop:
	sock = socket(AF_INET, SOCK_STREAM, 0);
	if (!sock) {
		perror("client: failed to create socket");
		return -1;
	}

	if (bind(sock, (struct sockaddr*)&addr_bind, sizeof(addr_bind)) < 0) {
		perror("client: failed to bind");
		return -1;
	}

	tmp = sizeof(addr_sockname);
	if (getsockname(sock, (struct sockaddr*)&addr_sockname, &tmp) < 0) {
		perror("client: getsockname failed");
		return -1;
	}

	ports[i] = ntohs(addr_sockname.sin_port);

	if (connect(sock, (struct sockaddr*)&addr_connect, sizeof(addr_connect)) < 0) {
		perror("client: failed to connect");
		return -1;
	}
	

	if (read(sock, &tmp, sizeof(tmp)) != 0) {
		perror("read failed");
		return -1;
	}

	close(sock);

	if (++i < N)
		goto loop;


	return analyze();
}


int server(void)
{
	int sock = socket(AF_INET, SOCK_STREAM, 0), sock2;
	struct sockaddr_in addr = {
		.sin_family = AF_INET,
		.sin_addr.s_addr = INADDR_ANY,
	};

 again:
	addr.sin_port = htons(listen_port);

	if (!sock) {
		perror("server: failed to create socket");
		return -1;
	}

	if (bind(sock, (struct sockaddr*)&addr, sizeof(addr)) < 0) {
		perror("server: failed to bind");
		listen_port++;
		goto again;
	}

	if (listen(sock, 128) < 0) {
		perror("server: failed to listen");
		return -1;
	}

	printf("listening (port %d)\n", listen_port);

	switch (fork()) {
	case -1:
		return -1;
	case 0:
		return client();
	}

 loop:
	sock2 = accept(sock, NULL, NULL);
	if (sock2 < 0) {
		perror("server: accept failed\n");
		return -1;
	}

	close(sock2);
	goto loop;
}


int main()
{

	return server();

}

Re: ports beeing reused too fast

[Eric Dumazet]

Octavian Purdila a écrit :
> Hi,
> 
> We've been running into an issue where a firewall would drop packets when an 
> moderate (~360) connection rate was going through it. It looks like the 
> firewall is dropping the SYNs that reuse ports "too fast". 
> 
> We have no issues with Linux 2.6.7, so I guess the behavior changed because of 
> this this commit:
> 
> commit 6df716340da3a6fdd33d73d7ed4c6f7590ca1c42
> Author: Stephen Hemminger <shemminger@osdl.org>
> Date:   Thu Nov 3 16:33:23 2005 -0800
> 
>     [TCP/DCCP]: Randomize port selection
> 
> 
> Now, I did some tests to confirm my suspicion. Basically, I am simulating a 
> connection rate test (I've attached the .c to this email) by opening up 
> connections and closing them - one at a time, and noting down the ports used, 
> then looking for duplicate ports and printing the distance between the 
> connection no.
> 
> Here is one of the runs, which make 1000 iterations:
> 
> listening (port 1242)
>  port reused: 38203: distance 578 (624,46)
>  port reused: 55693: distance 85 (147,62)
>  port reused: 38269: distance 803 (872,69)
>  port reused: 46239: distance 249 (344,95)
>  port reused: 40981: distance 215 (319,104)
>  port reused: 46246: distance 524 (641,117)
>  port reused: 43990: distance 378 (498,120)
>  port reused: 53766: distance 52 (232,180)
>  port reused: 44199: distance 194 (383,189)
>  port reused: 59464: distance 173 (384,211)
>  port reused: 44417: distance 264 (492,228)
>  port reused: 56989: distance 229 (553,324)
>  port reused: 60117: distance 69 (394,325)
>  port reused: 44549: distance 179 (566,387)
>  port reused: 39213: distance 300 (801,501)
>  port reused: 60166: distance 152 (671,519)
>  port reused: 44178: distance 108 (712,604)
>  port reused: 46516: distance 6 (792,786)
>  port reused: 55754: distance 95 (969,874)
> 19 ports were being reused
> 
> Running the same test on 2.6.7 yields a "0 ports were being reused" on all 
> tests that I've ran (10 or so).
> 
> Isn't it desirable to have the behavior from 2.6.7? 
> 
> I've looked over the  code and it looks right, so maybe net_random() is not 
> random enough? Or maybe there are side effects because of the % port_range?
> 

Random is random :)
Probability a port can be reused pretty fast is not nul.

So yes, behavior you discovered is expected, when we switched port selection
from a sequential one (not very secure btw) to a random one.

Any strong reason why a firewall would drop a SYN because ports were used in a 
previous session ?

Previous mode can be restored by application itself, using a bind() before
connect(), if this application knows it has a very high rate of connections
from a particular host to a particular host. (source ports range being
small anyway, so your firewall could complain again)

Re: ports beeing reused too fast

[Octavian Purdila]

On Saturday 09 May 2009 09:58:25 Eric Dumazet wrote:

> > I've looked over the  code and it looks right, so maybe net_random() is
> > not random enough? Or maybe there are side effects because of the %
> > port_range?
>
> Random is random :)
> Probability a port can be reused pretty fast is not nul.
>

Thinking again about it... you are right :)

> So yes, behavior you discovered is expected, when we switched port
> selection from a sequential one (not very secure btw) to a random one.
>
> Any strong reason why a firewall would drop a SYN because ports were used
> in a previous session ?

We don't know why the firewall (Cisco FWSM) is dropping the packets, may be a 
bug, limitation or miss-configuration. We are trying to track this down with 
the firewall vendor.

> Previous mode can be restored by application itself, using a bind() before
> connect(), if this application knows it has a very high rate of connections
> from a particular host to a particular host. (source ports range being
> small anyway, so your firewall could complain again)

Do you mean bind() with port != 0 ? Because I am already using bind() before 
connect().

Thanks,
tavi

Re: ports beeing reused too fast

[Eric Dumazet]

Octavian Purdila a écrit :
> On Saturday 09 May 2009 09:58:25 Eric Dumazet wrote:
> 
>>> I've looked over the  code and it looks right, so maybe net_random() is
>>> not random enough? Or maybe there are side effects because of the %
>>> port_range?
>> Random is random :)
>> Probability a port can be reused pretty fast is not nul.
>>
> 
> Thinking again about it... you are right :)
> 
>> So yes, behavior you discovered is expected, when we switched port
>> selection from a sequential one (not very secure btw) to a random one.
>>
>> Any strong reason why a firewall would drop a SYN because ports were used
>> in a previous session ?
> 
> We don't know why the firewall (Cisco FWSM) is dropping the packets, may be a 
> bug, limitation or miss-configuration. We are trying to track this down with 
> the firewall vendor.

Normally, the client machine should not reuse a port during the TIME_WAIT duration
(TCP_TIMEWAIT_LEN being 60 seconds on linux). Port selection being random or sequential,
it should avoid all ports recently used.

Maybe this firewall has a longer TIME_WAIT enforcement (something like 2 minutes)

> 
>> Previous mode can be restored by application itself, using a bind() before
>> connect(), if this application knows it has a very high rate of connections
>> from a particular host to a particular host. (source ports range being
>> small anyway, so your firewall could complain again)
> 
> Do you mean bind() with port != 0 ? Because I am already using bind() before 
> connect().
> 

Yes, but its obviously complex to handle at application side...

Re: ports beeing reused too fast

[Eric Dumazet]

Eric Dumazet a écrit :
> Octavian Purdila a écrit :
>> On Saturday 09 May 2009 09:58:25 Eric Dumazet wrote:
>>
>>>> I've looked over the  code and it looks right, so maybe net_random() is
>>>> not random enough? Or maybe there are side effects because of the %
>>>> port_range?
>>> Random is random :)
>>> Probability a port can be reused pretty fast is not nul.
>>>
>> Thinking again about it... you are right :)
>>
>>> So yes, behavior you discovered is expected, when we switched port
>>> selection from a sequential one (not very secure btw) to a random one.
>>>
>>> Any strong reason why a firewall would drop a SYN because ports were used
>>> in a previous session ?
>> We don't know why the firewall (Cisco FWSM) is dropping the packets, may be a 
>> bug, limitation or miss-configuration. We are trying to track this down with 
>> the firewall vendor.
> 
> Normally, the client machine should not reuse a port during the TIME_WAIT duration
> (TCP_TIMEWAIT_LEN being 60 seconds on linux). Port selection being random or sequential,
> it should avoid all ports recently used.
> 
> Maybe this firewall has a longer TIME_WAIT enforcement (something like 2 minutes)

Another thing to consider is your client/server use or not tcp timestamps (RFC 1323)

# should allow client to use fast reuse of ports (and trigger a firewall problem)
echo 1 >/proc/sys/net/ipv4/tcp_timestamps

If both machines allow tcp timestamps, then same ports can be reused pretty fast.

If firewall doesnt fully understand RFC 1323, it might explain some problem
with port randomization and shortened time between port reuse.

Re: ports beeing reused too fast

[Bill Fink]

On Sat, 09 May 2009, Eric Dumazet wrote:

> Octavian Purdila a écrit :
> > On Saturday 09 May 2009 09:58:25 Eric Dumazet wrote:
> > 
> >>> I've looked over the  code and it looks right, so maybe net_random() is
> >>> not random enough? Or maybe there are side effects because of the %
> >>> port_range?
> >> Random is random :)
> >> Probability a port can be reused pretty fast is not nul.
> >>
> > 
> > Thinking again about it... you are right :)
> > 
> >> So yes, behavior you discovered is expected, when we switched port
> >> selection from a sequential one (not very secure btw) to a random one.
> >>
> >> Any strong reason why a firewall would drop a SYN because ports were used
> >> in a previous session ?
> > 
> > We don't know why the firewall (Cisco FWSM) is dropping the packets, may be a 
> > bug, limitation or miss-configuration. We are trying to track this down with 
> > the firewall vendor.
> 
> Normally, the client machine should not reuse a port during the TIME_WAIT duration
> (TCP_TIMEWAIT_LEN being 60 seconds on linux). Port selection being random or sequential,
> it should avoid all ports recently used.
> 
> Maybe this firewall has a longer TIME_WAIT enforcement (something like 2 minutes)

But he had 19 ports being reused after only 1000 connect()s, which
with his stated ~360 (I'm assuming per second) connection rate,
would only take about 3 seconds.

						-Bill

Re: ports beeing reused too fast

[Octavian Purdila]

On Saturday 09 May 2009 22:31:35 Bill Fink wrote:

> > Normally, the client machine should not reuse a port during the TIME_WAIT
> > duration (TCP_TIMEWAIT_LEN being 60 seconds on linux). Port selection
> > being random or sequential, it should avoid all ports recently used.
> >
> > Maybe this firewall has a longer TIME_WAIT enforcement (something like 2
> > minutes)
>
> But he had 19 ports being reused after only 1000 connect()s, which
> with his stated ~360 (I'm assuming per second) connection rate,
> would only take about 3 seconds.
>

Time-wait doesn't apply in this case, because connections are closed by the 
server, thus they don't have to go through the time-wait state.

Thanks,
tavi

Re: ports beeing reused too fast

[Stephen Hemminger]

On Sat, 9 May 2009 15:31:35 -0400
Bill Fink <billfink@mindspring.com> wrote:

> On Sat, 09 May 2009, Eric Dumazet wrote:
> 
> > Octavian Purdila a écrit :
> > > On Saturday 09 May 2009 09:58:25 Eric Dumazet wrote:
> > > 
> > >>> I've looked over the  code and it looks right, so maybe net_random() is
> > >>> not random enough? Or maybe there are side effects because of the %
> > >>> port_range?
> > >> Random is random :)
> > >> Probability a port can be reused pretty fast is not nul.
> > >>
> > > 
> > > Thinking again about it... you are right :)
> > > 
> > >> So yes, behavior you discovered is expected, when we switched port
> > >> selection from a sequential one (not very secure btw) to a random one.
> > >>
> > >> Any strong reason why a firewall would drop a SYN because ports were used
> > >> in a previous session ?
> > > 
> > > We don't know why the firewall (Cisco FWSM) is dropping the packets, may be a 
> > > bug, limitation or miss-configuration. We are trying to track this down with 
> > > the firewall vendor.
> > 
> > Normally, the client machine should not reuse a port during the TIME_WAIT duration
> > (TCP_TIMEWAIT_LEN being 60 seconds on linux). Port selection being random or sequential,
> > it should avoid all ports recently used.
> > 
> > Maybe this firewall has a longer TIME_WAIT enforcement (something like 2 minutes)
> 
> But he had 19 ports being reused after only 1000 connect()s, which
> with his stated ~360 (I'm assuming per second) connection rate,
> would only take about 3 seconds.
> 
> 						-Bill

This the same thing as the Birthday paradox

-- 

Re: ports beeing reused too fast

[Octavian Purdila]

> > > >> Any strong reason why a firewall would drop a SYN because ports were
> > > >> used in a previous session ?
> > > >
> > > > We don't know why the firewall (Cisco FWSM) is dropping the packets,
> > > > may be a bug, limitation or miss-configuration. We are trying to
> > > > track this down with the firewall vendor.
> > >

Interestingly, we are seeing the same behavior with another high-end firewall 
(Juniper SRX). 

As mentioned previously, this can not be caused by the TW state, because the 
connections are getting closed on the server side.

Thanks,
tavi

Re: ports beeing reused too fast

[Stephen Hemminger]

On Tue, 12 May 2009 15:32:57 +0300
Octavian Purdila <opurdila@ixiacom.com> wrote:

> 
> > > > >> Any strong reason why a firewall would drop a SYN because ports were
> > > > >> used in a previous session ?
> > > > >
> > > > > We don't know why the firewall (Cisco FWSM) is dropping the packets,
> > > > > may be a bug, limitation or miss-configuration. We are trying to
> > > > > track this down with the firewall vendor.
> > > >
> 
> Interestingly, we are seeing the same behavior with another high-end firewall 
> (Juniper SRX). 
> 
> As mentioned previously, this can not be caused by the TW state, because the 
> connections are getting closed on the server side.
> 
> Thanks,
> tavi

I raised the issue to the original author of the proposed RFC and added
the issue to the ongoing review of the draft.

There is a suggested workaround using a bitmap but it seems like it would be
expensive to implement:
       http://ietfreport.isoc.org/all-ids/draft-ananth-tsvwg-timewait-00.txt

-- 

Re: ports beeing reused too fast

[Octavian Purdila]

> I raised the issue to the original author of the proposed RFC and added
> the issue to the ongoing review of the draft.
>
> There is a suggested workaround using a bitmap but it seems like it would
> be expensive to implement:
>       
> http://ietfreport.isoc.org/all-ids/draft-ananth-tsvwg-timewait-00.txt

OK, I now understand how TW could be an issue here - I didn't realize that we 
could have sockets in TW on the server side.

Since the workaround seems expensive, would it be acceptable to add a new 
sysctl option to disable port randomization?

Thanks,
tavi

Re: ports beeing reused too fast

[Stephen Hemminger]

On Tue, 12 May 2009 18:52:25 +0300
Octavian Purdila <opurdila@ixiacom.com> wrote:

> 
> > I raised the issue to the original author of the proposed RFC and added
> > the issue to the ongoing review of the draft.
> >
> > There is a suggested workaround using a bitmap but it seems like it would
> > be expensive to implement:
> >       
> > http://ietfreport.isoc.org/all-ids/draft-ananth-tsvwg-timewait-00.txt
> 
> OK, I now understand how TW could be an issue here - I didn't realize that we 
> could have sockets in TW on the server side.
> 
> Since the workaround seems expensive, would it be acceptable to add a new 
> sysctl option to disable port randomization?
> 
> Thanks,
> tavi
> 

Patches welcome.  Also, it matters whether application does bind() first
or port is assigned as part of the accept. In the later case the starting
point is a hash of the 5 tuple and changes only every 5 minutes.  Linux
also reuses ports in time wait, so it will reuse sockets more aggressively
than other hosts; this existed before port randomization.

--