Strange ip_conntrack values

Strange ip_conntrack values

[John]

Hi,

When I run the command

grep ^tcp /proc/net/ip_conntrack | awk '{print $4}' | sort | uniq -c

I get these lines ...

    26 CLOSE
    11 CLOSE_WAIT
   883 ESTABLISHED
    57 FIN_WAIT
    34 SYN_RECV
   116 SYN_SENT
 23720 TIME_WAIT

the TIME_WAIT number seems very strange ... network interrupts
increased a lot three months ago and I couldn't find an explanation
for this. The number of our visitors didn't increased like this ...

For example, before this "event", network interrups from MRTG, went
down to nearly 0 at 4 AM, but now even at this hour they are at 1000
....

We are under RH 7.2
2.4.26

I hope you could help me ...

Thanks a lot,

John

Re: Strange ip_conntrack values

[Antony Stone]

On Sunday 18 July 2004 11:31 am, John wrote:

> Hi,
>
> When I run the command
>
> grep ^tcp /proc/net/ip_conntrack | awk '{print $4}' | sort | uniq -c
>
> I get these lines ...
>
>     26 CLOSE
>     11 CLOSE_WAIT
>    883 ESTABLISHED
>     57 FIN_WAIT
>     34 SYN_RECV
>    116 SYN_SENT
>  23720 TIME_WAIT
>
> the TIME_WAIT number seems very strange ... network interrupts
> increased a lot three months ago and I couldn't find an explanation
> for this. The number of our visitors didn't increased like this ...

I agree this is strange, because the default TIME_WAIT timeout value is 2 
minutes (you haven't increased this, have you?), therefore this would suggest 
that nearly 24000 connections through your firewall were completed during the 
past two minutes...   This seems unlikely, especially in light of the number 
(883) you have in progress right now.

If you "grep TIME_WAIT /proc/net/ip_conntrack | more", do you see nearly all 
entries with the same source and/or destination address?   If so, investigate 
that machine.....

If not, I suggest a network sniffer (eg: ethereal) or some netfilter LOGging 
rules to see if you can identify what all this traffic is.

Regards,

Antony.

-- 
There are two possible outcomes:

 If the result confirms the hypothesis, then you've made a measurement.
 If the result is contrary to the hypothesis, then you've made a discovery.

 - Enrico Fermi

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Strange ip_conntrack values

[John]

> I agree this is strange, because the default TIME_WAIT timeout value is 2
> minutes (you haven't increased this, have you?), therefore this would suggest
> that nearly 24000 connections through your firewall were completed during the
> past two minutes...   This seems unlikely, especially in light of the number
> (883) you have in progress right now.

yes it's still à 2 min

> If you "grep TIME_WAIT /proc/net/ip_conntrack | more", do you see nearly all
> entries with the same source and/or destination address?   If so, investigate
> that machine.....

unfortunately not ...

> If not, I suggest a network sniffer (eg: ethereal) or some netfilter LOGging
> rules to see if you can identify what all this traffic is.

how can I do that ? could u help me achieving this ? I've installed
tcpdump and logged all connections between 4AM and 6AM but it's not
easy to find something ...

could it come from the firewall ?

thanks for your help

Re: Strange ip_conntrack values

[Antony Stone]

On Sunday 18 July 2004 12:28 pm, John wrote:

> > If not, I suggest a network sniffer (eg: ethereal) or some netfilter
> > LOGging rules to see if you can identify what all this traffic is.
>
> how can I do that ? could u help me achieving this ? I've installed
> tcpdump and logged all connections between 4AM and 6AM but it's not
> easy to find something ...

Tcpdump is a good packet sniffer but it does not show the data in a 
user-friendly format.

I suggest you install ethereal on a machine (does not have to be the firewall) 
and load the tcpdump output file into that.   It will help show you the 
connections in a meaningful format, and you can look for FIN-ACK packets 
which are not replied, multiple FIN-ACKs, etc.

Also, do you have a snapshot of /proc/net/ip_conntrack from any time during 
4am-6am?   If not, I suggest you take another tcpdump log (rather than 2 
hours, I suggest something much shorter, say 10 minutes, because the timer 
you are interested in expires after 2 minutes, so you should get enough 
examples of whatever's happening within a 10 minute window), and take a 
snapshot of /proc/net/ip_conntrack at the start and end of the tcpdump log 
(perhaps a couple of times in the middle as well).

That should give you a traffic stream (of a manageable size) to look at in 
ethereal and compare to the contents of the conntrack table to work out where 
the TIME_WAIT entries are coming from.

By the way, you're not blocking any packets which are important to closing 
connections, are you?   Such as FIN-ACK or RST?   Maybe checking the packet 
counters from "iptables -L -nvx; iptables -L -t nat -nvx" might show 
something interesting?


Regards,

Antony.

-- 
You can spend the whole of your life trying to be popular,
but at the end of the day the size of the crowd at your funeral
will be largely dictated by the weather.

 - Frank Skinner

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Strange ip_conntrack values

[John]

> Tcpdump is a good packet sniffer but it does not show the data in a
> user-friendly format.

I've tried to load my tcpdump file but ethereal doesn't recognize it
... is there a way to configure tcpdump fot that ?

my data are like this :

04:30:00.662037 IP deathpolka.nyogtha.org.62238 > mydomaine.net.http:
. ack 3067679957 win 64240
04:30:00.662331 IP sts-12e87.adsl.wanadoo.nl.4164 >
mydomaine.net.http: . ack 3331465322 win 17520
04:30:00.662617 IP deathpolka.nyogtha.org.62238 > mydomaine.net.http:
F 0:0(0) ack 1 win 64240

> I suggest you install ethereal on a machine (does not have to be the firewall)
> and load the tcpdump output file into that.   It will help show you the
> connections in a meaningful format, and you can look for FIN-ACK packets
> which are not replied, multiple FIN-ACKs, etc.
> 
> Also, do you have a snapshot of /proc/net/ip_conntrack from any time during
> 4am-6am?   If not, I suggest you take another tcpdump log (rather than 2
> hours, I suggest something much shorter, say 10 minutes, because the timer
> you are interested in expires after 2 minutes, so you should get enough
> examples of whatever's happening within a 10 minute window), and take a
> snapshot of /proc/net/ip_conntrack at the start and end of the tcpdump log
> (perhaps a couple of times in the middle as well).
> 
> That should give you a traffic stream (of a manageable size) to look at in
> ethereal and compare to the contents of the conntrack table to work out where
> the TIME_WAIT entries are coming from.

ok good idea I'll try this tonight

> By the way, you're not blocking any packets which are important to closing
> connections, are you?   Such as FIN-ACK or RST?   Maybe checking the packet
> counters from "iptables -L -nvx; iptables -L -t nat -nvx" might show
> something interesting?

I'm not enough experienced to try to interpret it . here is a copy if
u can have a look, I've not seen something too strange :

thanks for your help

Chain INPUT (policy DROP 15947 packets, 1548815 bytes)
    pkts      bytes target     prot opt in     out     source         
     destination
    3251   196544 MALFORMED  all  --  *      *       0.0.0.0/0        
   0.0.0.0/0          state INVALID
       0        0 MALFORMED  all  -f  *      *       0.0.0.0/0        
   0.0.0.0/0
       1       40 MALFORMED  tcp  --  *      *       0.0.0.0/0        
   0.0.0.0/0          state NEW tcp flags:0x3F/0x03
       0        0 MALFORMED  tcp  --  *      *       0.0.0.0/0        
   0.0.0.0/0          state NEW tcp flags:0x3F/0x29
  846218 117145998 ACCEPT     udp  --  *      *       0.0.0.0/0       
    0.0.0.0/0          state RELATED,ESTABLISHED
       0        0 ACCEPT     tcp  --  *      *       0.0.0.0/0        
   0.0.0.0/0          state RELATED tcp flags:!0x16/0x02
     162     7916 ACCEPT     tcp  --  *      *       0.0.0.0/0        
   0.0.0.0/0          state RELATED tcp flags:0x16/0x02
 2156908 103774704 ACCEPT     tcp  --  *      *       0.0.0.0/0       
    0.0.0.0/0          state ESTABLISHED tcp flags:0x16/0x02
423113683 25328463653 ACCEPT     tcp  --  *      *       0.0.0.0/0    
       0.0.0.0/0          state ESTABLISHED tcp flags:!0x16/0x02
   15377  1042496 ACCEPT     all  --  lo     *       127.0.0.1        
   0.0.0.0/0
   78330  4735198 ACCEPT     all  --  lo     *       MYIP        MYIP
    4288   339738 ACCEPT     udp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          udp spt:53
   37115  2929800 ACCEPT     udp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          udp dpt:53
       6      240 ACCEPT     tcp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          tcp spt:53
      36     1528 ACCEPT     tcp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          tcp dpt:53
130986256 6441434116 ACCEPT     tcp  --  eth0   *       0.0.0.0/0     
      0.0.0.0/0          state NEW tcp dpt:80 flags:0x16/0x02
  440496 18258310 ACCEPT     tcp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          state NEW tcp dpt:80 flags:!0x16/0x02
     123     6156 ACCEPT     tcp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          state NEW tcp dpt:22 flags:0x16/0x02
     317    15508 ACCEPT     tcp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          state NEW tcp dpt:21 flags:0x16/0x02
    6190   318804 ACCEPT     tcp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          state NEW tcp dpt:25 flags:0x16/0x02
    5747   278968 ACCEPT     tcp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          state NEW tcp dpt:110 flags:0x16/0x02
    1750    84000 ACCEPT     tcp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          state NEW tcp dpt:10000 flags:0x16/0x02
   39411  3711874 ACCEPT     icmp --  eth0   *       0.0.0.0/0        
   0.0.0.0/0
   95124  7419672 REJECT     udp  --  eth0   *       0.0.0.0/0        
   0.0.0.0/0          udp dpt:137 reject-with icmp-port-unreachable
   15552  1469086 LOG        all  --  *      *       0.0.0.0/0        
   0.0.0.0/0          limit: avg 1/sec burst 5 LOG flags 0 level 4
prefix `IPT [DROPED] : '

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source         
     destination

Chain OUTPUT (policy ACCEPT 325971709 packets, 53608243217 bytes)
    pkts      bytes target     prot opt in     out     source         
     destination

Chain MALFORMED (4 references)
    pkts      bytes target     prot opt in     out     source         
     destination
    3196   193448 LOG        all  --  *      *       0.0.0.0/0        
   0.0.0.0/0          limit: avg 1/sec burst 5 LOG flags 0 level 4
prefix `IPT [MALFORMED] : '
    3252   196584 REJECT     all  --  *      *       0.0.0.0/0        
   0.0.0.0/0          reject-with icmp-port-unreachable
Chain PREROUTING (policy ACCEPT 127860795 packets, 6290596491 bytes)
    pkts      bytes target     prot opt in     out     source         
     destination

Chain POSTROUTING (policy ACCEPT 140964 packets, 10420602 bytes)
    pkts      bytes target     prot opt in     out     source         
     destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source         
     destination

Re: Strange ip_conntrack values

[John]

> Tcpdump is a good packet sniffer but it does not show the data in a
> user-friendly format.

ok I've made another tcpdump for ethereal and it's ok;

I've checked and I get a lot of this scheme :

No.     Time        Source                Destination           Protocol Info
     10 0.004569    24.33.232.227         mydomain         TCP     
1488 > http [SYN] Seq=0 Ack=0 Win=64240 Len=0 MSS=1460
     11 0.004626    mydomain         24.33.232.227         TCP     
http > 1488 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460
    255 0.162181    24.33.232.227         mydomain         TCP     
1488 > http [ACK] Seq=1 Ack=1 Win=64240 Len=0
    258 0.165191    24.33.232.227         mydomain         TCP     
1488 > http [FIN, ACK] Seq=1 Ack=1 Win=64240 Len=0
    259 0.165313    mydomain         24.33.232.227         TCP     
http > 1488 [FIN, ACK] Seq=1 Ack=2 Win=5840 Len=0
    385 0.311935    24.33.232.227         mydomain         TCP     
1488 > http [ACK] Seq=2 Ack=2 Win=64240 Len=0

(this is the whole tcp stream)

for others I get the complete http exchange : get ...

is it normal ?

Ethereal is brand new for me so if you have some good tips to help me
find some interesting information ... thanks a lot

John

Re: Strange ip_conntrack values

[Antony Stone]

On Sunday 18 July 2004 2:56 pm, John wrote:

> > Tcpdump is a good packet sniffer but it does not show the data in a
> > user-friendly format.
>
> ok I've made another tcpdump for ethereal and it's ok;

Good.   Sounds like the last time you just redirected console output from 
tcpdump; this time you've actually made it save a file.

> I've checked and I get a lot of this scheme :
>
> No.     Time        Source                Destination           Protocol
> Info 10 0.004569    24.33.232.227         mydomain         TCP
> 1488 > http [SYN] Seq=0 Ack=0 Win=64240 Len=0 MSS=1460
>      11 0.004626    mydomain         24.33.232.227         TCP
> http > 1488 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460
>     255 0.162181    24.33.232.227         mydomain         TCP
> 1488 > http [ACK] Seq=1 Ack=1 Win=64240 Len=0
>     258 0.165191    24.33.232.227         mydomain         TCP
> 1488 > http [FIN, ACK] Seq=1 Ack=1 Win=64240 Len=0
>     259 0.165313    mydomain         24.33.232.227         TCP
> http > 1488 [FIN, ACK] Seq=1 Ack=2 Win=5840 Len=0
>     385 0.311935    24.33.232.227         mydomain         TCP
> 1488 > http [ACK] Seq=2 Ack=2 Win=64240 Len=0
>
> (this is the whole tcp stream)

Good.   That to me looks like a good TCP sequence, however there is one packet 
missing - the ACK response from "mydomain" to 24.33.232.227 in response to 
the FIN-ACK sent by 24.33.232.227 in packet number 1488.   Until this missing 
response is seen by netfilter, it will regard that connection as being in the 
TIME_WAIT state, however this will expire after 2 minutes.

> for others I get the complete http exchange : get ...

I can't explain why there isn't any more interesting data in the HTTP part of 
the above communication, however that's pretty irrelevant for TCP/IP.

> is it normal ?

Is it normal to see an HTTP connection with no data being transferred?   No, I 
wouldn't have thought so, but then I'm no expert on M$ IIS-5.1, which is what 
that web server is running.

> Ethereal is brand new for me so if you have some good tips to help me
> find some interesting information ... thanks a lot

The help system is pretty good; other than that there's documentation at 
http://www.ethereal.com   You probably won't find ethereal.org or 
ethereal.net quite so useful :)

Regards,

Antony.

-- 
GIT/E d- s+:--(-) a+ C++++$(---) UL++++$ P+(---)>++ L+++(++++)$ !E W(-) N(-) 
o? w--(---) O !M V+++(--) !PS !PE Y+ PGP+> t- !tv@ b+++ DI++ D--- e+++(*) h++ 
5? !X- !R K--? G-

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Strange ip_conntrack values

[John]

> Good.   That to me looks like a good TCP sequence, however there is one packet
> missing - the ACK response from "mydomain" to 24.33.232.227 in response to
> the FIN-ACK sent by 24.33.232.227 in packet number 1488.   Until this missing
> response is seen by netfilter, it will regard that connection as being in the
> TIME_WAIT state, however this will expire after 2 minutes.

yep strange ... for many sequences it's the same thing. the web server
is under Red Hat 7.2. Could it come from it? This phenomenon with many
network interrupts appeared one day without any important change from
us ...

is it possible to exclude the IP that run this sequence ?

what could I check ? do you think it could be a kind of attack? thanks
because I'm lost ... don't konw what to do with this problem

Re: Strange ip_conntrack values

[Antony Stone]

On Sunday 18 July 2004 5:19 pm, John wrote:

> > Good.   That to me looks like a good TCP sequence, however there is one
> > packet missing - the ACK response from "mydomain" to 24.33.232.227 in
> > response to the FIN-ACK sent by 24.33.232.227 in packet number 1488.  
> > Until this missing response is seen by netfilter, it will regard that
> > connection as being in the TIME_WAIT state, however this will expire
> > after 2 minutes.
>
> yep strange ... for many sequences it's the same thing. the web server
> is under Red Hat 7.2

Huh?

$ telnet 24.33.232.227 80
Trying 24.33.232.227...
Connected to 24.33.232.227.
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.0 200 OK
Accept-Ranges: bytes
Date: Sun, 18 Jul 2004 16:29:40 GMT
Content-Length: 988
Content-Type: text/html
Server: Microsoft-IIS/5.1
Content-Location: http://24.33.232.227/Default.htm
Last-Modified: Fri, 05 Sep 2003 15:24:32 GMT
ETag: "30633ecec173c31:9f5"

Connection closed by foreign host.
$

Note the line saying "Server: Microsoft-IIS/5.1"

I really don't think that can be running under Red Hat 7.2 :)

Anyway, getting back to the question at hand, I guess one valid approach is 
"is this causing you any problems?"   ie: Is your connection tracking table 
getting full and dropping new connections because of all these 2-minute 
timeouts you're seeing?

Antony.

-- 
Anything that improbable is effectively impossible.

 - Murray Gell-Mann, Nobel Prizewinner in Physics

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Strange ip_conntrack values

[John]

> 
> Note the line saying "Server: Microsoft-IIS/5.1"
> 
> I really don't think that can be running under Red Hat 7.2 :)
> 
> Anyway, getting back to the question at hand, I guess one valid approach is
> "is this causing you any problems?"   ie: Is your connection tracking table
> getting full and dropping new connections because of all these 2-minute
> timeouts you're seeing?

ok for http://24.33.232.227/
I was speaking of our server :))

but strange for : http://24.33.232.227/ why this yahoo page will make
a connection to our web pages ... maybe it's just spoofing ...

to answer your question, no problem at all for our server, but I don't
like not to be able to explain all these strange TCP connections ;) I
feel frustrated ...

Re: Strange ip_conntrack values

[Stephen Smoogen]

On Sun, 18 Jul 2004, John wrote:

>Hi,
>
>When I run the command
>
>grep ^tcp /proc/net/ip_conntrack | awk '{print $4}' | sort | uniq -c
>
>I get these lines ...
>
>    26 CLOSE
>    11 CLOSE_WAIT
>   883 ESTABLISHED
>    57 FIN_WAIT
>    34 SYN_RECV
>   116 SYN_SENT
> 23720 TIME_WAIT
>
>the TIME_WAIT number seems very strange ... network interrupts
>increased a lot three months ago and I couldn't find an explanation
>for this. The number of our visitors didn't increased like this ...

What modules are loaded into the kernel? There is a bug in the Red Hat 
7.x/8.0/9 kernels from a patch that Alan Cox had in his patchset long 
ago and should have been updated. The bug causes connections to sit 
in slabinfo forever and not get cleaned out from the conntrack module. I 
think I tracked it down to the conntrack_ftp but it could have been 
generic. 

The best bet on a 7.x/8 machine is to download and compile the latest
2.4.x kernel and possibly add POM items if you need them. On a 9 system
it would be better to upgrade to Fedora 1(or 2) as the 2.6 backported
code is a pain to get around. I found that for the 7.x series a stock 
kernel 2.4.24 kernel worked great because a lot of the patches Red Hat 
had incorporated into theirs was now in the mainline. 

Hope this helps.



-- 
Stephen John Smoogen		smoogen@lanl.gov
Los Alamos National Lab  CCN-5 Sched 5/40  PH: 4-0645
Ta-03 SM-1498 MailStop B255 DP 10S  Los Alamos, NM 87545
-- "We cannot have a free government without elections; and if the
-- rebellion could force us to forgo, or postpone, a national election,
-- it might fairly claim to have already conquered us." Abraham Lincoln