bind() udp behavior 2.6.8.1

bind() udp behavior 2.6.8.1

[Adam Denenberg]

Hello,

 I am not subscribed to this list so please CC me personally in
response. 

 I am noticing some odd behavior with linux 2.6.8.1 on a redhat 8 box
when making udp requests.  It seems subsequent udp calls are all
allocating the same source ephemeral udp port.  I believe the kernel
should be randomizing these (or incrementing) these ports for subsequent
requests.  We ran a test C program that just put a gethostbyname_r call
in a for loop of 40 calls and all 40 requests used the same UDP source
port (32789).  This is causing our firewall to drop some packets since
it thinks it already closed that connection due to too many transactions
using same udp source/dest port passing thru in too short a time frame.

Is this a bug in how the linux kernel handles allocating udp source
ports? Freebsd seems to not do this, so i dont think this is a standard
methodology of how UDP/IP should function.

Any help would be appreciated..

thanks
adam

Re: bind() udp behavior 2.6.8.1

[Jan Engelhardt]

>Hello,
>
> I am not subscribed to this list so please CC me personally in
>response. 
>
> I am noticing some odd behavior with linux 2.6.8.1 on a redhat 8 box
>when making udp requests.  It seems subsequent udp calls are all
>allocating the same source ephemeral udp port.  I believe the kernel
>should be randomizing these (or incrementing) these ports for subsequent
>requests.

No, you can have a fixed port for any socket. (It's just a question whether 
you actually get the socket, because it might be in use.)

See http://linux01.org:2222/f/UHXT/examples/src/fastsock.c , which contains an 
example on how to choose a fixed port.

>  We ran a test C program that just put a gethostbyname_r call
>in a for loop of 40 calls and all 40 requests used the same UDP source
>port (32789).

Looks normal to me. It might select a random port upon "libc invocation" and 
use it for all further requests. This is in fact very valid, because UDP is 
connectionless; packets can go from anywhere to anywhere without any 
pre-work.

>  This is causing our firewall to drop some packets since
>it thinks it already closed that connection due to too many transactions
>using same udp source/dest port passing thru in too short a time frame.

Then, the firewall UDP implementation is broken. Note, an UDP connection *can 
not be closed*, because it never was "open". If it's trying to do something 
like 
	iptables -p udp -m state --state RELATED
it is doing it wrong, because that is an impossible situation.


Jan Engelhardt
-- 
ENOSPC

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

sorry "closed" was the wrong term here.  We are using a PIX Firewall
Module and it keeps a state table of all connections (tcp and udp). 
Thus when a new udp connection comes in with same high numbered source
port and the firewall has not removed that connection from its state
table, the firewall drops the packet.  The firewall needs about 60ms in
order to clear that connection from the state table, so if a second udp
request with the same high number port/ip comes thru before the firewall
clears the connection from the state table, it will drop the connection
(which is what we are seeing).

FreeBSD seems to increment future udp requests which prevents this
problem.  It just seems strange that the kernel would not randomize or
increment these source ports for udp requests.

has anyone encountered this issue?

thanks
adam

please CC me I am not on the list.

On Tue, 2004-12-14 at 11:04, Jan Engelhardt wrote:
> >Hello,
> >
> > I am not subscribed to this list so please CC me personally in
> >response. 
> >
> > I am noticing some odd behavior with linux 2.6.8.1 on a redhat 8 box
> >when making udp requests.  It seems subsequent udp calls are all
> >allocating the same source ephemeral udp port.  I believe the kernel
> >should be randomizing these (or incrementing) these ports for subsequent
> >requests.
> 
> No, you can have a fixed port for any socket. (It's just a question whether 
> you actually get the socket, because it might be in use.)
> 
> See http://linux01.org:2222/f/UHXT/examples/src/fastsock.c , which contains an 
> example on how to choose a fixed port.
> 
> >  We ran a test C program that just put a gethostbyname_r call
> >in a for loop of 40 calls and all 40 requests used the same UDP source
> >port (32789).
> 
> Looks normal to me. It might select a random port upon "libc invocation" and 
> use it for all further requests. This is in fact very valid, because UDP is 
> connectionless; packets can go from anywhere to anywhere without any 
> pre-work.
> 
> >  This is causing our firewall to drop some packets since
> >it thinks it already closed that connection due to too many transactions
> >using same udp source/dest port passing thru in too short a time frame.
> 
> Then, the firewall UDP implementation is broken. Note, an UDP connection *can 
> not be closed*, because it never was "open". If it's trying to do something 
> like 
> 	iptables -p udp -m state --state RELATED
> it is doing it wrong, because that is an impossible situation.
> 
> 
> Jan Engelhardt

Re: bind() udp behavior 2.6.8.1

[Jan Engelhardt]

>sorry "closed" was the wrong term here.  We are using a PIX Firewall
>Module and it keeps a state table of all connections (tcp and udp). 
>Thus when a new udp connection comes in with same high numbered source

UDP does not know connections. As such, _nobody_ can tell whether an UDP 
packet belongs to a logically existing "connection" or not.

>port and the firewall has not removed that connection from its state
>table, the firewall drops the packet.  The firewall needs about 60ms in
>order to clear that connection from the state table, so if a second udp
>request with the same high number port/ip comes thru before the firewall
>clears the connection from the state table, it will drop the connection
>(which is what we are seeing).
>
>FreeBSD seems to increment future udp requests which prevents this
>problem.  It just seems strange that the kernel would not randomize or
>increment these source ports for udp requests.

The kernel does not have problems with UDP, it's probably your firewall.

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

any firewall must keep some sort of state table even if it is udp.  How
else do you manage established connections?  It must know that some high
numbered port is making a udp dns request, and thus be able to allow
that request back thru to the high numbered port client b/c the
connection is already established.

what does any fireawall do if it sees one ip with the same high numbered
udp port make a request in a _very_ short amount of time (under 60ms for
this example)?  It must make a distintion between an attack and legit
traffic.  So if it sees one ip/port make multiple requests in too short
of a time frame, it will drop the traffic, as it probably should.  This
is causing erratic behavior when traffic traverses the firewall b/c the
linux kernel keeps allocating the same source high numbered ephemeral
port.  I would like to know if there is a way to alter this behavior b/c
it is breaking applications.

thanks
adam

please CC me as I am not on this list.

On Tue, 2004-12-14 at 11:44, Jan Engelhardt wrote:
> >sorry "closed" was the wrong term here.  We are using a PIX Firewall
> >Module and it keeps a state table of all connections (tcp and udp). 
> >Thus when a new udp connection comes in with same high numbered source
> 
> UDP does not know connections. As such, _nobody_ can tell whether an UDP 
> packet belongs to a logically existing "connection" or not.
> 
> >port and the firewall has not removed that connection from its state
> >table, the firewall drops the packet.  The firewall needs about 60ms in
> >order to clear that connection from the state table, so if a second udp
> >request with the same high number port/ip comes thru before the firewall
> >clears the connection from the state table, it will drop the connection
> >(which is what we are seeing).
> >
> >FreeBSD seems to increment future udp requests which prevents this
> >problem.  It just seems strange that the kernel would not randomize or
> >increment these source ports for udp requests.
> 
> The kernel does not have problems with UDP, it's probably your firewall.
> 

Re: bind() udp behavior 2.6.8.1

[Jan Engelhardt]

>any firewall must keep some sort of state table even if it is udp.  How

No.

>else do you manage established connections?  It must know that some high

You don't manage something that does not need managing. It's like firing a 
bullet at some. You can not tell whether there's still more bullets to come or 
not.

>numbered port is making a udp dns request, and thus be able to allow
>that request back thru to the high numbered port client b/c the
>connection is already established.

See linux-os's reply. UDP is connectionless.

>what does any fireawall do if it sees one ip with the same high numbered
>udp port make a request in a _very_ short amount of time (under 60ms for
>this example)?

I let it pass.

>It must make a distintion between an attack and legit
>traffic.

That's something VERY different. There is a difference between **knowing** 
that a set of packets _are related_ to each other and the time between two 
**arbitrary** packets.

>  So if it sees one ip/port make multiple requests in too short
>of a time frame, it will drop the traffic, as it probably should.

Depends on the definition of attack. Look it at that way:

localhost:32768 sends an UDP packet to dnsserver:53... but already the next 
packet CAN BE malicious. (Another process can take over the port if the former 
has dropped the socket.)

>This
>is causing erratic behavior when traffic traverses the firewall b/c the

Then fix the FW.

>linux kernel keeps allocating the same source high numbered ephemeral

In your case, the socket is allocated once for the whole program. This socket 
is _reused_.

>port.  I would like to know if there is a way to alter this behavior b/c
>it is breaking applications.

No, as said, your FW is broken.




Jan Engelhardt
-- 
ENOSPC

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

i am aware that UDP is connectionless.  However in terms of a firewall
this is different.  It _must_ keep a state table of some sorts otherwise
high port outbound connections destined for a DNS server will never be
let back in b/c the firewall will just say "Why is this dns server
making a udp connection to port 32768 on this client?".  Keeping a state
table allows this behavior thru the firewall as it should.

My issue is that linux is not randomizing or incrementing the ports it
uses for udp connections to prevent this sort of issue since udp is
connectionless.  We dont have sequence numbers or the sorts like TCP to
sort this out, we only have source ip and port.

Other OS's seem  to do this and thus apps are not getting broken when
connections are made thru the firewall, which is why i originally posted
the question.  I was hoping that maybe there is some workaround or that
hopefully someone else encountered this issue.  I am not saying the
firewall is not totally to blame, but i can see why it is behaving the
way it is when seeing tons of connections from the same udp ip/port come
in.

thanks again
adam



On Tue, 2004-12-14 at 12:10, Jan Engelhardt wrote:
> >any firewall must keep some sort of state table even if it is udp.  How
> 
> No.
> 
> >else do you manage established connections?  It must know that some high
> 
> You don't manage something that does not need managing. It's like firing a 
> bullet at some. You can not tell whether there's still more bullets to come or 
> not.
> 
> >numbered port is making a udp dns request, and thus be able to allow
> >that request back thru to the high numbered port client b/c the
> >connection is already established.
> 
> See linux-os's reply. UDP is connectionless.
> 
> >what does any fireawall do if it sees one ip with the same high numbered
> >udp port make a request in a _very_ short amount of time (under 60ms for
> >this example)?
> 
> I let it pass.
> 
> >It must make a distintion between an attack and legit
> >traffic.
> 
> That's something VERY different. There is a difference between **knowing** 
> that a set of packets _are related_ to each other and the time between two 
> **arbitrary** packets.
> 
> >  So if it sees one ip/port make multiple requests in too short
> >of a time frame, it will drop the traffic, as it probably should.
> 
> Depends on the definition of attack. Look it at that way:
> 
> localhost:32768 sends an UDP packet to dnsserver:53... but already the next 
> packet CAN BE malicious. (Another process can take over the port if the former 
> has dropped the socket.)
> 
> >This
> >is causing erratic behavior when traffic traverses the firewall b/c the
> 
> Then fix the FW.
> 
> >linux kernel keeps allocating the same source high numbered ephemeral
> 
> In your case, the socket is allocated once for the whole program. This socket 
> is _reused_.
> 
> >port.  I would like to know if there is a way to alter this behavior b/c
> >it is breaking applications.
> 
> No, as said, your FW is broken.
> 
> 
> 
> 
> Jan Engelhardt

Re: bind() udp behavior 2.6.8.1

[Jan Engelhardt]

>i am aware that UDP is connectionless.  However in terms of a firewall
>this is different.  It _must_ keep a state table of some sorts otherwise

must vs cannot.

>high port outbound connections destined for a DNS server will never be
>let back in b/c the firewall will just say "Why is this dns server
>making a udp connection to port 32768 on this client?".  Keeping a state
>table allows this behavior thru the firewall as it should.

no state for no connection.
if you want rate limiting, you can use iptables -m limit, or whatever your fw 
implements. Frankly, you can't distinguish DNS traffic from someone who 
hijacked your box by just looking at the packet's header. And your FW _has_ to 
account for the case that multiple, possibly "non-related" packets appear on 
the same source port.

>My issue is that linux is not randomizing or incrementing the ports it
>uses for udp connections to prevent this sort of issue since udp is
>connectionless.  We dont have sequence numbers or the sorts like TCP to

Works for me.

>sort this out, we only have source ip and port.

And the kernel keeps a rover. It's right here:

	grep udp_port_rover /usr/src/linux/net/ipv4/*.c

So whenever someone binds an udp socket to port 0, the kernel chooses 
something. And IF NOT (i.e port != 0 at the time you call bind), you gonna use 
THAT specific port.


Jan Engelhardt
-- 
ENOSPC

Re: bind() udp behavior 2.6.8.1

[Kyle Moffett]

On Dec 14, 2004, at 12:01, Adam Denenberg wrote:
> any firewall must keep some sort of state table even if it is udp.  How
> else do you manage established connections?  It must know that some 
> high
> numbered port is making a udp dns request, and thus be able to allow
> that request back thru to the high numbered port client b/c the
> connection is already established.
>
> what does any fireawall do if it sees one ip with the same high 
> numbered
> udp port make a request in a _very_ short amount of time (under 60ms 
> for
> this example)?  It must make a distintion between an attack and legit
> traffic.  So if it sees one ip/port make multiple requests in too short
> of a time frame, it will drop the traffic, as it probably should.  This
> is causing erratic behavior when traffic traverses the firewall b/c the
> linux kernel keeps allocating the same source high numbered ephemeral
> port.  I would like to know if there is a way to alter this behavior 
> b/c
> it is breaking applications.

When I wrote my user-space UDP over TCP tunneling software, I combined
the Internal IP and port and the External IP and port into a single 
hashed
value that I used as an index into my "psuedo-connection" hash, of which
each entry referenced an index in my 2000 item "pseudo-connection" 
table.
For each packet from an unrecognized host, I added a new hash entry and
table entry, then forwarded the packet.  When I get a new packet 
matching
an old-but-not-expired rule, I set the "last_seen" value to the current 
time.
When a connection is 5 minutes since "last_seen", then it can be removed
if there is pressure on the table, otherwise it will accept packets 
until it is
1 hour old, at which point it gets purged.  I've found that this system 
works
well at tunneling everything from Kerberos and OpenAFS to DNS without
problems.  Given that this relatively simple piece of sofware is able to
successfully manage a multitude of UDP connections, I would suggest
that an advanced connection-tracking firewall like yours has serious 
bugs
if it can't perform equally well.  Either that or it shouldn't be 
meddling in the
affairs of such UDP packets.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
!y?(-)
------END GEEK CODE BLOCK------

Re: bind() udp behavior 2.6.8.1

[Wichert Akkerman]

Previously Adam Denenberg wrote:
> i am aware that UDP is connectionless.  However in terms of a firewall
> this is different.  It _must_ keep a state table of some sorts otherwise
> high port outbound connections destined for a DNS server will never be
> let back in b/c the firewall will just say "Why is this dns server
> making a udp connection to port 32768 on this client?".  Keeping a state
> table allows this behavior thru the firewall as it should.

Just allow outgoing udp traffic from source port 53 from your DNS
server and you're done. 

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

i think you guys are all right.  However there is one concern.  Not 
clearing out a UDP connection in a firewall coming from a high port is 
indeed a security risk.  Allowing a high numbered udp port to remain 
open for a prolonged period of time would definitely impose a security 
risk which is why the PIX is doing what it does.  The linux server is 
"reusing" the same UDP high numbered socket however it is doing so 
exactly as the firewall is clearing its state table (60 ms) from the 
first connection which is what is causing the issue.

I think a firewall ought to be aware of such behavior, but at the same 
time be secure enough to not just leave high numbered udp ports wide 
open for attack.  I am trying to find out why the PIX chose 60 ms to 
clear out the UDP state table.  I think that is a random number and 
probably too short of a span for this to occur however i am still 
researching it.

Any other insight would be greatly appreciated.

adam

On Dec 14, 2004, at 5:07 PM, Kyle Moffett wrote:

> On Dec 14, 2004, at 12:01, Adam Denenberg wrote:
>> any firewall must keep some sort of state table even if it is udp.  
>> How
>> else do you manage established connections?  It must know that some 
>> high
>> numbered port is making a udp dns request, and thus be able to allow
>> that request back thru to the high numbered port client b/c the
>> connection is already established.
>>
>> what does any fireawall do if it sees one ip with the same high 
>> numbered
>> udp port make a request in a _very_ short amount of time (under 60ms 
>> for
>> this example)?  It must make a distintion between an attack and legit
>> traffic.  So if it sees one ip/port make multiple requests in too 
>> short
>> of a time frame, it will drop the traffic, as it probably should.  
>> This
>> is causing erratic behavior when traffic traverses the firewall b/c 
>> the
>> linux kernel keeps allocating the same source high numbered ephemeral
>> port.  I would like to know if there is a way to alter this behavior 
>> b/c
>> it is breaking applications.
>
> When I wrote my user-space UDP over TCP tunneling software, I combined
> the Internal IP and port and the External IP and port into a single 
> hashed
> value that I used as an index into my "psuedo-connection" hash, of 
> which
> each entry referenced an index in my 2000 item "pseudo-connection" 
> table.
> For each packet from an unrecognized host, I added a new hash entry and
> table entry, then forwarded the packet.  When I get a new packet 
> matching
> an old-but-not-expired rule, I set the "last_seen" value to the 
> current time.
> When a connection is 5 minutes since "last_seen", then it can be 
> removed
> if there is pressure on the table, otherwise it will accept packets 
> until it is
> 1 hour old, at which point it gets purged.  I've found that this 
> system works
> well at tunneling everything from Kerberos and OpenAFS to DNS without
> problems.  Given that this relatively simple piece of sofware is able 
> to
> successfully manage a multitude of UDP connections, I would suggest
> that an advanced connection-tracking firewall like yours has serious 
> bugs
> if it can't perform equally well.  Either that or it shouldn't be 
> meddling in the
> affairs of such UDP packets.
>
> Cheers,
> Kyle Moffett
>
> -----BEGIN GEEK CODE BLOCK-----
> Version: 3.12
> GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
> L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
> PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
> !y?(-)
> ------END GEEK CODE BLOCK------
>
>

Re: bind() udp behavior 2.6.8.1

[Kyle Moffett]

On Dec 14, 2004, at 21:23, Adam Denenberg wrote:
> i think you guys are all right.  However there is one concern.  Not 
> clearing out a UDP connection in a firewall coming from a high port is 
> indeed a security risk.  Allowing a high numbered udp port to remain 
> open for a prolonged period of time would definitely impose a security 
> risk which is why the PIX is doing what it does.  The linux server is 
> "reusing" the same UDP high numbered socket however it is doing so 
> exactly as the firewall is clearing its state table (60 ms) from the 
> first connection which is what is causing the issue.
>
> I think a firewall ought to be aware of such behavior, but at the same 
> time be secure enough to not just leave high numbered udp ports wide 
> open for attack.  I am trying to find out why the PIX chose 60 ms to 
> clear out the UDP state table.  I think that is a random number and 
> probably too short of a span for this to occur however i am still 
> researching it.
>
> Any other insight would be greatly appreciated.

60ms is certainly _way_ too small for most UDP traffic.  With something 
like
that, OpenAFS would die almost immediately.  I think the current OpenAFS
minimum is like 20 minutes, although somebody patched the OpenAFS
source to send a keepalive every 5 minutes, so it could be reduced.  
OTOH,
sending a keepalive every 60ms would take a _massive_ amount of
bandwidth even for one client, think about a couple hundred :-D.  Heck, 
I've
even seen pings on a regular basis that take longer than 60ms, which
means that even an infinitely fast kerberos server wouldn't respond 
quickly
enough :-D.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
!y?(-)
------END GEEK CODE BLOCK------

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

i have some more updated information for this.  It seems that linux is
generating a duplicate transaction ID in the udp header in the dns query
when making a dns request.  This piece seems to be what is preventing
the Firewall from distinguishing unique dns requests.  It sees a second
DNS request come from the linux server with the _same_ transaction ID in
the UDP header as it is marking that session closed since it already saw
the reply successfully.  So for example the linux server is making a dns
query with DNS Transaction ID 0x598d, then a response comes back with ID
0x598d, and then another query gets sent out with ID 0x598d.  When that
second response comes back, the firewall gets confused b/c it is in the
middle of closing the original 0x598d session and thus drops the packet,
causing a DNS timeout.

 Why is linux generating a redundant transaction ID in the dns header
here?  The first 2 requests have incremented values, but then for some
reason the third seems to have the same value which is causing issues. 
Has anyone encountered this behavior?

thanks again
adam

Please CC me i am not on the list.




On Tue, 2004-12-14 at 22:19, Kyle Moffett wrote:
> On Dec 14, 2004, at 21:23, Adam Denenberg wrote:
> > i think you guys are all right.  However there is one concern.  Not 
> > clearing out a UDP connection in a firewall coming from a high port is 
> > indeed a security risk.  Allowing a high numbered udp port to remain 
> > open for a prolonged period of time would definitely impose a security 
> > risk which is why the PIX is doing what it does.  The linux server is 
> > "reusing" the same UDP high numbered socket however it is doing so 
> > exactly as the firewall is clearing its state table (60 ms) from the 
> > first connection which is what is causing the issue.
> >
> > I think a firewall ought to be aware of such behavior, but at the same 
> > time be secure enough to not just leave high numbered udp ports wide 
> > open for attack.  I am trying to find out why the PIX chose 60 ms to 
> > clear out the UDP state table.  I think that is a random number and 
> > probably too short of a span for this to occur however i am still 
> > researching it.
> >
> > Any other insight would be greatly appreciated.
> 
> 60ms is certainly _way_ too small for most UDP traffic.  With something 
> like
> that, OpenAFS would die almost immediately.  I think the current OpenAFS
> minimum is like 20 minutes, although somebody patched the OpenAFS
> source to send a keepalive every 5 minutes, so it could be reduced.  
> OTOH,
> sending a keepalive every 60ms would take a _massive_ amount of
> bandwidth even for one client, think about a couple hundred :-D.  Heck, 
> I've
> even seen pings on a regular basis that take longer than 60ms, which
> means that even an infinitely fast kerberos server wouldn't respond 
> quickly
> enough :-D.
> 
> Cheers,
> Kyle Moffett
> 
> -----BEGIN GEEK CODE BLOCK-----
> Version: 3.12
> GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
> L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
> PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
> !y?(-)
> ------END GEEK CODE BLOCK------
> 
> 

Re: bind() udp behavior 2.6.8.1

[Jan Harkes]

On Wed, Dec 15, 2004 at 09:16:02AM -0500, Adam Denenberg wrote:
> the Firewall from distinguishing unique dns requests.  It sees a second
> DNS request come from the linux server with the _same_ transaction ID in
> the UDP header as it is marking that session closed since it already saw
> the reply successfully.  So for example the linux server is making a dns

Stupid guess here,

The reply got dropped after it passed your firewall and before it
reached the linux server. What you are seeing is simply a retransmit
which would also have happened if the original request got lost, or if
the reply was dropped before it reached your firewall, in which case the
firewall probably would have forwarded the retransmitted request without
a problem.

I would open the window before you throw the piece of garbage out.

Jan

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

almost yes.  The firewall never passes the retransmit onto the DNS
server since it has the same DNS ID, source port and source ip.  What is
happening is the following

request 1
--------------------
linux box.32789 (id 001) ->  FW -> DNS SERVER.53
DNS SERVER.53 (id 001) -> FW -> linux box.32789 

request 2
-------------------
linux box.32789 (id 002) -> FW -> DNS SERVER.53
DNS SERVER (id 002).53 -> FW -> linux box.32789

request 3
-----------------------
linux box (id 002).32789 -> FW -> NEVER GETS HERE, B/C ITS DROPPED

the time between request 2 and request 3 is under 60ms.  The firewall is
in the midst of clearing its table for the dns request with ID 002
already so it thinks its a duplicate and drops it.  So my question is,
why is the kernel not incrementing the DNS ID in this case? It does it
for almost all other tests that i can find, and the firewall does not
drop any traffic.  Only when the DNS ID does not increment does this
problem occur.  This does not seem to always be the default behavior.  I
wrote a small C program to just put a gethostbyname_r() in a for loop
and each DNS ID is incremented all 40 times.  But there are times when
this doesnt happen, and this seems to be what is causing the issue.  The
firewall needs some sort of identifier to know which dns request is
associated with which dns reply (source ip, source port, ID). 

this is the behavior I am trying to debug.

thanks
adam

Please CC me as i am not on the list.


On Wed, 2004-12-15 at 14:07, Jan Harkes wrote:
> On Wed, Dec 15, 2004 at 09:16:02AM -0500, Adam Denenberg wrote:
> > the Firewall from distinguishing unique dns requests.  It sees a second
> > DNS request come from the linux server with the _same_ transaction ID in
> > the UDP header as it is marking that session closed since it already saw
> > the reply successfully.  So for example the linux server is making a dns
> 
> Stupid guess here,
> 
> The reply got dropped after it passed your firewall and before it
> reached the linux server. What you are seeing is simply a retransmit
> which would also have happened if the original request got lost, or if
> the reply was dropped before it reached your firewall, in which case the
> firewall probably would have forwarded the retransmitted request without
> a problem.
> 
> I would open the window before you throw the piece of garbage out.
> 
> Jan
> 

Re: bind() udp behavior 2.6.8.1

[linux-os]

On Wed, 15 Dec 2004, Adam Denenberg wrote:

> almost yes.  The firewall never passes the retransmit onto the DNS
> server since it has the same DNS ID, source port and source ip.  What is
> happening is the following
>
> request 1
> --------------------
> linux box.32789 (id 001) ->  FW -> DNS SERVER.53
> DNS SERVER.53 (id 001) -> FW -> linux box.32789
>
> request 2
> -------------------
> linux box.32789 (id 002) -> FW -> DNS SERVER.53
> DNS SERVER (id 002).53 -> FW -> linux box.32789
>
> request 3
> -----------------------
> linux box (id 002).32789 -> FW -> NEVER GETS HERE, B/C ITS DROPPED
>
> the time between request 2 and request 3 is under 60ms.  The firewall is
> in the midst of clearing its table for the dns request with ID 002
> already so it thinks its a duplicate and drops it.  So my question is,
> why is the kernel not incrementing the DNS ID in this case? It does it
> for almost all other tests that i can find, and the firewall does not
> drop any traffic.  Only when the DNS ID does not increment does this
> problem occur.  This does not seem to always be the default behavior.  I
> wrote a small C program to just put a gethostbyname_r() in a for loop
> and each DNS ID is incremented all 40 times.  But there are times when
> this doesnt happen, and this seems to be what is causing the issue.  The
> firewall needs some sort of identifier to know which dns request is
> associated with which dns reply (source ip, source port, ID).
>
> this is the behavior I am trying to debug.
>
> thanks
> adam

The ID portion of the IP header, offset 32, is 16 bits of unique
identification that is supposed to be unique for the entire time
that any message should be in the system. That's what firewalls
and routers use to determine if it's a duplicate packet. You
never before stated that Linux was duplicating the ID portion,
and if it is, it's a bug. But, I'll bet that it isn't. Nothing
would work if it was. All TCP/IP messages are composed of
Datagrams. If these basic elements were mucked up, this would
have been discovered long before now, and if not discovered,
you wouldn't receive this message. Also UCP is connectionless
and stateless. If you have some box that handles it differently,
its broken.


Cheers,
Dick Johnson
Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by John Ashcroft.
                  98.36% of all statistics are fiction.

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

sorry for any confusion, but i am not referring to the Identification
field in the IP header but rather the "Transaction ID" field in the DNS
query portion of the packet.  I can reproduce this behavior on our linux
system where if i pump gethostbyname_r requests on the system at some
point it will reuse a transaction id in the DNS request.  This is my
lastest discovery in what is causing the requests to fail thru the
firewall.  So far my research has not turned up any reason as to why the
kernel would be re-using a transaction ID in the dns request.

The ip IDentification field is uniquely generated as it should be, just
not the Transaction ID field of the dns portion of the packet.

adam

Please CC me as I am not on this list.


On Wed, 2004-12-15 at 15:06, linux-os wrote:
> On Wed, 15 Dec 2004, Adam Denenberg wrote:
> 
> > almost yes.  The firewall never passes the retransmit onto the DNS
> > server since it has the same DNS ID, source port and source ip.  What is
> > happening is the following
> >
> > request 1
> > --------------------
> > linux box.32789 (id 001) ->  FW -> DNS SERVER.53
> > DNS SERVER.53 (id 001) -> FW -> linux box.32789
> >
> > request 2
> > -------------------
> > linux box.32789 (id 002) -> FW -> DNS SERVER.53
> > DNS SERVER (id 002).53 -> FW -> linux box.32789
> >
> > request 3
> > -----------------------
> > linux box (id 002).32789 -> FW -> NEVER GETS HERE, B/C ITS DROPPED
> >
> > the time between request 2 and request 3 is under 60ms.  The firewall is
> > in the midst of clearing its table for the dns request with ID 002
> > already so it thinks its a duplicate and drops it.  So my question is,
> > why is the kernel not incrementing the DNS ID in this case? It does it
> > for almost all other tests that i can find, and the firewall does not
> > drop any traffic.  Only when the DNS ID does not increment does this
> > problem occur.  This does not seem to always be the default behavior.  I
> > wrote a small C program to just put a gethostbyname_r() in a for loop
> > and each DNS ID is incremented all 40 times.  But there are times when
> > this doesnt happen, and this seems to be what is causing the issue.  The
> > firewall needs some sort of identifier to know which dns request is
> > associated with which dns reply (source ip, source port, ID).
> >
> > this is the behavior I am trying to debug.
> >
> > thanks
> > adam
> 
> The ID portion of the IP header, offset 32, is 16 bits of unique
> identification that is supposed to be unique for the entire time
> that any message should be in the system. That's what firewalls
> and routers use to determine if it's a duplicate packet. You
> never before stated that Linux was duplicating the ID portion,
> and if it is, it's a bug. But, I'll bet that it isn't. Nothing
> would work if it was. All TCP/IP messages are composed of
> Datagrams. If these basic elements were mucked up, this would
> have been discovered long before now, and if not discovered,
> you wouldn't receive this message. Also UCP is connectionless
> and stateless. If you have some box that handles it differently,
> its broken.
> 
> 
> Cheers,
> Dick Johnson
> Penguin : Linux version 2.6.9 on an i686 machine (5537.79 BogoMips).
>   Notice : All mail here is now cached for review by John Ashcroft.
>                   98.36% of all statistics are fiction.
> 

Re: bind() udp behavior 2.6.8.1

[Doug McNaught]

Adam Denenberg <adam@dberg.org> writes:

> sorry for any confusion, but i am not referring to the Identification
> field in the IP header but rather the "Transaction ID" field in the DNS
> query portion of the packet.  I can reproduce this behavior on our linux
> system where if i pump gethostbyname_r requests on the system at some
> point it will reuse a transaction id in the DNS request.  This is my
> lastest discovery in what is causing the requests to fail thru the
> firewall.  So far my research has not turned up any reason as to why the
> kernel would be re-using a transaction ID in the dns request.

That would be your DNS software reusing the transaction ID, since it's
part of the packet data, not the UDP header.  The kernel has nothing
to do with it.

-Doug

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

but this would be an issue with glibc then correct?


On Wed, 2004-12-15 at 15:45, Doug McNaught wrote:
> Adam Denenberg <adam@dberg.org> writes:
> 
> > sorry for any confusion, but i am not referring to the Identification
> > field in the IP header but rather the "Transaction ID" field in the DNS
> > query portion of the packet.  I can reproduce this behavior on our linux
> > system where if i pump gethostbyname_r requests on the system at some
> > point it will reuse a transaction id in the DNS request.  This is my
> > lastest discovery in what is causing the requests to fail thru the
> > firewall.  So far my research has not turned up any reason as to why the
> > kernel would be re-using a transaction ID in the dns request.
> 
> That would be your DNS software reusing the transaction ID, since it's
> part of the packet data, not the UDP header.  The kernel has nothing
> to do with it.
> 
> -Doug
> 

Re: bind() udp behavior 2.6.8.1

[Doug McNaught]

Adam Denenberg <adam@dberg.org> writes:

> but this would be an issue with glibc then correct?

Whatever's sending the DNS request--the resolver library, BIND, or
what have you...

-Doug

Re: bind() udp behavior 2.6.8.1

[Willy Tarreau]

Hi,

On Tue, Dec 14, 2004 at 12:38:02PM -0500, Adam Denenberg wrote:
 
> My issue is that linux is not randomizing or incrementing the ports it
> uses for udp connections to prevent this sort of issue since udp is
> connectionless.  We dont have sequence numbers or the sorts like TCP to
> sort this out, we only have source ip and port.

Most UDP services use the server socket to access remote servers. For
example, bind will only use its source port 53 to emit requests, ntpd
will only use source port 123, this *is* how they have been working for
ages. PIX is totally buggy and does not respect standards, it respects
what seems to be "common practices" (remember TCP ECN ?). If the
developpers have observed that freebsd wastes lots of source ports for
DNS requests, they will decide that using the same one is probably an
attack. I even remember that it does not support NTP correctly. When
an NTP server on your LAN "connects" to another one outside, it
translates the source port 123 to another source port < 1024, which
most servers and/or firewalls drop (they only let 123 or >=1024 in).

> Other OS's seem  to do this and thus apps are not getting broken when
> connections are made thru the firewall, which is why i originally posted
> the question.

Perhaps the other OS you have seen simply close the socket as soon as they
get their response, and have to create a new one for each new request...
How does any DNS server forward to outside through your PIX ?

> I was hoping that maybe there is some workaround or that
> hopefully someone else encountered this issue.  I am not saying the
> firewall is not totally to blame, but i can see why it is behaving the
> way it is when seeing tons of connections from the same udp ip/port come
> in.

I doubt you can reliably use any UDP-based application through this
firewall then... It seems more buggy than other pixes I have encountered,
and I think you can configure it to be less silly, but I don't know how.
You agree that a session is defined by these 5 numbers :
  - proto (udp, tcp, icmp, ...)
  - source ip, source port
  - destination ip, destination port

If these 5 parameters are the same within a certain time frame, the packet
belongs to a known session. If the combination is different, then it is a
new session, and there is no reason for the firewall to drop a new session
only based on the fact that 3 parameters out of 5 are the same as other ones.
It is as stupid as saying that you refuse to establish a new TCP connection
on some dest:80 because you just did it 30ms ago.

I you want to work around this buggy behaviour without reconfiguring the PIX,
you can also play with iptables on the linux machine to use a random source
port range :
  iptables -t nat -A OUTPUT -p udp --dport 53 -j SNAT --to-source <your ip>:1024-65534

But this is a dirty hack...

Regards,
Willy

Re: bind() udp behavior 2.6.8.1

[Willy Tarreau]

On Tue, Dec 14, 2004 at 09:23:43PM -0500, Adam Denenberg wrote:
> i think you guys are all right.  However there is one concern.  Not 
> clearing out a UDP connection in a firewall coming from a high port is 
> indeed a security risk.  Allowing a high numbered udp port to remain 
> open for a prolonged period of time would definitely impose a security 
> risk which is why the PIX is doing what it does.

Absolutely NO ! it is not "keeping the port open", it is "keeping a SESSION
open". The firewall should allow traffic from the same ip:port to the other
ip:port and from no other server on the net. You new session is totally
unrelated to the old one.

>  The linux server is 
> "reusing" the same UDP high numbered socket however it is doing so 
> exactly as the firewall is clearing its state table (60 ms) from the 
> first connection which is what is causing the issue.

it is not the same session if you connect to a different remote server,
and there is absolutely no reason to arbitrary prevent an internal server
from connecting to two external ones from the same IP:port. Of course, if
you reconnect to the same destIP:port, it should work because in this case
it is a continuation of the same session.

> I think a firewall ought to be aware of such behavior, but at the same 
> time be secure enough to not just leave high numbered udp ports wide 
> open for attack.  I am trying to find out why the PIX chose 60 ms to 
> clear out the UDP state table.  I think that is a random number and 
> probably too short of a span for this to occur however i am still 
> researching it.

it is not the timing which causes trouble, it is the way it confuses new
and already established sessions. Although 60 ms may seem short (you can
probably never resolve anything on ADSL with a loaded link), it may be
perfectly valid if the firewall agrees to open several sessions when you
connect to several servers. And if you connect several times to the same
server, of course it must re-open the session.

> Any other insight would be greatly appreciated.

unfortunately, googling for "pix udp problem" returns 25600 responses...

Regards,
willy
 

Re: bind() udp behavior 2.6.8.1

[Willy Tarreau]

On Wed, Dec 15, 2004 at 02:22:53PM -0500, Adam Denenberg wrote:
> So my question is, why is the kernel not incrementing the DNS ID in this case?

Uhh ?
The *kernel* should change IDs in the *DNS* request ? It is the libc which does
the DNS requests. The only ID the kernel could play with it the IP packet ID.

Willy

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

the issue with this currently is that the resolver is not using a new 
"Transaction ID" in the DNS porttion of the packet.  This is the one 
unique piece of informatino that helps distinguish what query belongs 
to what response (since same source port and ip is used).  So if the 
transaction id in the DNS header of the query is not updated, there is 
no real way to distinguish these dns requests.  Also keep in mind the 
linux box is requesting the same A record, so even that is not unique 
among these requests.

I am trying to figure out why the resolver (redhat 8 glibc) is behaing 
this way, b/c it is breaking applications.  If it simply updated its 
transaction ID in the packet i dont think this would be an issue.

thanks
adam


Please CC me I am not on this list.


On Dec 16, 2004, at 1:03 AM, Willy Tarreau wrote:

> On Tue, Dec 14, 2004 at 09:23:43PM -0500, Adam Denenberg wrote:
>> i think you guys are all right.  However there is one concern.  Not
>> clearing out a UDP connection in a firewall coming from a high port is
>> indeed a security risk.  Allowing a high numbered udp port to remain
>> open for a prolonged period of time would definitely impose a security
>> risk which is why the PIX is doing what it does.
>
> Absolutely NO ! it is not "keeping the port open", it is "keeping a 
> SESSION
> open". The firewall should allow traffic from the same ip:port to the 
> other
> ip:port and from no other server on the net. You new session is totally
> unrelated to the old one.
>
>>  The linux server is
>> "reusing" the same UDP high numbered socket however it is doing so
>> exactly as the firewall is clearing its state table (60 ms) from the
>> first connection which is what is causing the issue.
>
> it is not the same session if you connect to a different remote server,
> and there is absolutely no reason to arbitrary prevent an internal 
> server
> from connecting to two external ones from the same IP:port. Of course, 
> if
> you reconnect to the same destIP:port, it should work because in this 
> case
> it is a continuation of the same session.
>
>> I think a firewall ought to be aware of such behavior, but at the same
>> time be secure enough to not just leave high numbered udp ports wide
>> open for attack.  I am trying to find out why the PIX chose 60 ms to
>> clear out the UDP state table.  I think that is a random number and
>> probably too short of a span for this to occur however i am still
>> researching it.
>
> it is not the timing which causes trouble, it is the way it confuses 
> new
> and already established sessions. Although 60 ms may seem short (you 
> can
> probably never resolve anything on ADSL with a loaded link), it may be
> perfectly valid if the firewall agrees to open several sessions when 
> you
> connect to several servers. And if you connect several times to the 
> same
> server, of course it must re-open the session.
>
>> Any other insight would be greatly appreciated.
>
> unfortunately, googling for "pix udp problem" returns 25600 
> responses...
>
> Regards,
> willy
>
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

I disagree.  The linux server should be using unique Transaction ID's 
in the dns header for each unique dns request.  Otherwise there is no 
way to distinguish them (same A record request).  Of course the 
firewall is going to drop a reply that it thinks it already saw a reply 
for 30ms ago.

This appears to be a bug in the way glibc is handling things but i 
cannot be sure.  That is the goal of my investigation.

adam

Please CC me i am not on the list.

On Dec 15, 2004, at 2:07 PM, Jan Harkes wrote:

> On Wed, Dec 15, 2004 at 09:16:02AM -0500, Adam Denenberg wrote:
>> the Firewall from distinguishing unique dns requests.  It sees a 
>> second
>> DNS request come from the linux server with the _same_ transaction ID 
>> in
>> the UDP header as it is marking that session closed since it already 
>> saw
>> the reply successfully.  So for example the linux server is making a 
>> dns
>
> Stupid guess here,
>
> The reply got dropped after it passed your firewall and before it
> reached the linux server. What you are seeing is simply a retransmit
> which would also have happened if the original request got lost, or if
> the reply was dropped before it reached your firewall, in which case 
> the
> firewall probably would have forwarded the retransmitted request 
> without
> a problem.
>
> I would open the window before you throw the piece of garbage out.
>
> Jan
>
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: bind() udp behavior 2.6.8.1

[Jan Harkes]

On Thu, Dec 16, 2004 at 09:24:59AM -0500, Adam Denenberg wrote:
> I disagree.  The linux server should be using unique Transaction ID's 
> in the dns header for each unique dns request.  Otherwise there is no 
> way to distinguish them (same A record request).  Of course the 
> firewall is going to drop a reply that it thinks it already saw a reply 
> for 30ms ago.
> 
> This appears to be a bug in the way glibc is handling things but i 
> cannot be sure.  That is the goal of my investigation.

Ok, please stop it and _read_ the email, completely, a possible solution
will be at the end.

This is not a problem with the linux kernel, the application (resolver
library) is simply retransmitting the request and the kernel does
exactly as it is told. So first of all this is completely off-topic for
linux-kernel.

Second of all, it is -not- a glibc issue either. The reply packet was
probably lost after it has passed through the firewall but before it
reached the DNS client machine. So since it has not yet received the
response it is doing the right thing, retransmitting the request.

The problem is... your firewall, it seems to assume that once it has
seen a UDP reply packet that this packet will reach the destination and
the session is closed. This is an incorrect assumption since UDP is per
definition a lossy protocol (unreliable datagram protocol) and the
packet can and will be dropped by any routers, switches, the network
card in your machine, or even by the kernel if there is too much
traffic or a single bit error which makes the checksum fail.

One solution that might work is to run a local caching DNS daemon on a
machine behind the firewall (bind, dnsmasq or something) and set
/etc/resolv.conf to only send requests to the local machine. The local
caching daemon can then forward those requests from a known fixed local
port to your upstream DNS servers. This allows you to open a hole in the
firewall between local_dns_cache:53 and upstream_dns_server:53.

Jan

Re: bind() udp behavior 2.6.8.1

[Adam Denenberg]

Ok thanks for all the help.

One final note before i end this (since there still seems to be some 
misunderstanding) is that the reply comes back successful for the first 
of the 2 redundant transaction IDs.  Its not a retransmit.  ID 101 gets 
sent from the linux box and receives a response successfully.  It then 
sends another reqeust with ID 101 _after_ it received the response for 
101.  That is my issue and that is what is breaking the communication 
thru the firewall.  It should be using a new ID after a successful dns 
transaction but it is not.

I understand this has deviated from the topic of the linux kernel so we 
can consider this thread dead.

thanks everybody for all their input and help.

adam

On Dec 16, 2004, at 9:51 AM, Jan Harkes wrote:

> On Thu, Dec 16, 2004 at 09:24:59AM -0500, Adam Denenberg wrote:
>> I disagree.  The linux server should be using unique Transaction ID's
>> in the dns header for each unique dns request.  Otherwise there is no
>> way to distinguish them (same A record request).  Of course the
>> firewall is going to drop a reply that it thinks it already saw a 
>> reply
>> for 30ms ago.
>>
>> This appears to be a bug in the way glibc is handling things but i
>> cannot be sure.  That is the goal of my investigation.
>
> Ok, please stop it and _read_ the email, completely, a possible 
> solution
> will be at the end.
>
> This is not a problem with the linux kernel, the application (resolver
> library) is simply retransmitting the request and the kernel does
> exactly as it is told. So first of all this is completely off-topic for
> linux-kernel.
>
> Second of all, it is -not- a glibc issue either. The reply packet was
> probably lost after it has passed through the firewall but before it
> reached the DNS client machine. So since it has not yet received the
> response it is doing the right thing, retransmitting the request.
>
> The problem is... your firewall, it seems to assume that once it has
> seen a UDP reply packet that this packet will reach the destination and
> the session is closed. This is an incorrect assumption since UDP is per
> definition a lossy protocol (unreliable datagram protocol) and the
> packet can and will be dropped by any routers, switches, the network
> card in your machine, or even by the kernel if there is too much
> traffic or a single bit error which makes the checksum fail.
>
> One solution that might work is to run a local caching DNS daemon on a
> machine behind the firewall (bind, dnsmasq or something) and set
> /etc/resolv.conf to only send requests to the local machine. The local
> caching daemon can then forward those requests from a known fixed local
> port to your upstream DNS servers. This allows you to open a hole in 
> the
> firewall between local_dns_cache:53 and upstream_dns_server:53.
>
> Jan
>