Packets missing state ?

Packets missing state ?

[Rajiv Shah]

Hi,
	I can see the following lines in my /var/log/messages

IN=eth1 OUT= MAC=00:03:47:6a:5c:6b:00:c0:49:25:d1:a3:08:00 SRC=69.6.16.110 
DST=xxx.xxx.xxx.xx LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=15968 PROTO=TCP SPT=80 
DPT=1821 WINDOW=65535 RES=0x00 ACK URGP=0

IN=eth1 OUT= MAC=00:03:47:6a:5c:6b:00:c0:49:25:d1:a3:08:00 SRC=69.6.16.110 
DST=xxx.xxx.xxx.xx LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=15968 PROTO=TCP 
SPT=110 DPT=4553 WINDOW=65535 RES=0x00 ACK URGP=0

where xxx.xxx.xxx.xx is my IP. This seems to be one of the return packets 
which is being logged. I can access the site without any problems. The mails 
come thru, can do a ssh login. But looking at the logs I feel there is some 
problem somewhere. Here is what I am checking for input packets.

**********************************************************************************************************
$IPT -A IN_FIREWALL -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A IN_FIREWALL -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A IN_FIREWALL -j LOG --log-prefix "IPT IN_FIREWALL: " $LOGOPT
$IPT -A IN_FIREWALL -j DROP
**********************************************************************************************************

-- 
 
======================================================
 
Warm Regards,
 
Rajiv Shah
Systems Engineer,
The Argon Company
www.theargoncompany.com
-----------------------------------------------------------------------------------------------------------
FREE sign-up for a virus-protected Secure Mailbox - www.tassm.com

Re: Packets missing state ?

[Antony Stone]

On Tuesday 23 December 2003 12:58 pm, Rajiv Shah wrote:

> Hi,
> 	I can see the following lines in my /var/log/messages
>
> IN=eth1 OUT= MAC=00:03:47:6a:5c:6b:00:c0:49:25:d1:a3:08:00 SRC=69.6.16.110
> DST=xxx.xxx.xxx.xx LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=15968 PROTO=TCP
> SPT=80 DPT=1821 WINDOW=65535 RES=0x00 ACK URGP=0
>
> IN=eth1 OUT= MAC=00:03:47:6a:5c:6b:00:c0:49:25:d1:a3:08:00 SRC=69.6.16.110
> DST=xxx.xxx.xxx.xx LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=15968 PROTO=TCP
> SPT=110 DPT=4553 WINDOW=65535 RES=0x00 ACK URGP=0
>
> where xxx.xxx.xxx.xx is my IP. This seems to be one of the return packets
> which is being logged. I can access the site without any problems. The
> mails come thru, can do a ssh login. But looking at the logs I feel there
> is some problem somewhere.

You say you can access the site without any problems.

I would expect the above log entries to happen when you close down the browser 
(or view a different site), but the remote server still continues to send a 
few packets.   They are no longer part of an established connection as far as 
netfilter is concerned, so they get logged.

Try checking the timestamps, or view the logfile in real time as you do your 
browsing - I'd expect these log entries to occur soon after you leave the 
site at the SRC address.

Nothing to worry about.

Antony.

-- 
What is this talk of "software release"?
Our software evolves and matures until it is capable of escape, leaving a 
bloody trail of designers and quality assurance people in its wake.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Packets missing state ?

[Vinayakam Murugan]

On Tuesday 23 December 2003 18:57, Vinayakam Murugan wrote:
> > You say you can access the site without any problems.
> >
> > I would expect the above log entries to happen when you close down the
> > browser (or view a different site), but the remote server still continues
> > to send a few packets.   They are no longer part of an established
> > connection as far as netfilter is concerned, so they get logged.
> >
> > Try checking the timestamps, or view the logfile in real time as you do
> > your browsing - I'd expect these log entries to occur soon after you
> > leave the site at the SRC address.
>
Thanks Antony. I'm Rajiv's colleague

 We were not able to reproduce it out here.Anyways in this scenario, would
 the packet would have the state as NEW?  All these logs are of packets with
 state NEW.

 Another peculiar log which we would like to get more info on are

 Dec 23 18:43:05 theargonserver kernel: IPT IN_FIREWALL_NEW: IN=eth1 OUT=
 MAC=00:03:47:6a:5c:6b:00:c0:49:25:d1:a3:08:00 SRC=80.15.238.66
 DST=xxx.xxx.xxx.xxx LEN=73 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF PROTO=UDP
 SPT=31819 DPT=53 LEN=53

 There are a lot of packets to port 53. Is this normal?

-- 
Warm Regards
~~~~~~~~~~~~~~~~~~~~~~~
Vinayakam Murugan

Viruses getting you down?
Get your virus protected mailbox at http://www.tassm.com

Linux: The choice of a GNU generation

Re: Packets missing state ?

[Antony Stone]

On Tuesday 23 December 2003 1:37 pm, Vinayakam Murugan wrote:

> On Tuesday 23 December 2003 18:57, Vinayakam Murugan wrote:
> > > You say you can access the site without any problems.
> > >
> > > I would expect the above log entries to happen when you close down the
> > > browser (or view a different site), but the remote server still
> > > continues to send a few packets.   They are no longer part of an
> > > established connection as far as netfilter is concerned, so they get
> > > logged.
> > >
> > > Try checking the timestamps, or view the logfile in real time as you do
> > > your browsing - I'd expect these log entries to occur soon after you
> > > leave the site at the SRC address.
>
> Thanks Antony. I'm Rajiv's colleague
>
>  We were not able to reproduce it out here.Anyways in this scenario, would
>  the packet would have the state as NEW?  All these logs are of packets
> with state NEW.

Yes, they are NEW because they are for a connection which netfilter has 
decided is over - therefore it is no longer ESTABLISHED, and any packets 
which continue to appear will be treated as a NEW connection.

>  Another peculiar log which we would like to get more info on are
>
>  Dec 23 18:43:05 theargonserver kernel: IPT IN_FIREWALL_NEW: IN=eth1 OUT=
>  MAC=00:03:47:6a:5c:6b:00:c0:49:25:d1:a3:08:00 SRC=80.15.238.66
>  DST=xxx.xxx.xxx.xxx LEN=73 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF PROTO=UDP
>  SPT=31819 DPT=53 LEN=53
>
>  There are a lot of packets to port 53. Is this normal?

Yes.   UDP port 53 means client-to-server DNS lookups.   This happens all the 
time on a normal network, and UDP is used so that if an answer doesn't come 
back quickly, the client will try another server instead.   Depending on the 
rules and logging you are doing, netfilter may again decide that some reply 
packets no longer match ESTABLISHED connections (netfilter keeps its own idea 
of what is an ESTABLISHED UDP connection, even though UDP is a connectionless 
protocol), and so such packets may be treated as spurious NEW packets which 
netfilter doesn't know what to do with.

Unless you experience DNS problems on your network (clients timeout or cannot 
find servers) I wouldn't bother about this; if you do have problems the 
easiest solution (which is really simple to do) is set up your own caching 
DNS server insternal to your network and tell the clients to do their lookups 
from there (the caching server will then do TCP requests to other servers on 
the Internet, and the whole thing will be much more efficient for you - I 
don't know if that will actually be noticeable to people in terms of better 
performance or lower bandwidth, though).

Antony.

-- 
Software development can be quick, high quality, or low cost.

The customer gets to pick any two out of three.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Packets missing state ?

[Chris Brenton]

On Tue, 2003-12-23 at 08:54, Antony Stone wrote:
>
> Yes, they are NEW because they are for a connection which netfilter has 
> decided is over - therefore it is no longer ESTABLISHED, and any packets 
> which continue to appear will be treated as a NEW connection.

ESTABLISHED state is maintained for 5 days, so I'm guessing the problem
is not there. When I've seen this before the traces usually show the
following:

*ESTABLISHED communications go fine
*One side issues a FIN/ACK to tear down the connection
*state timing is dropped from 5 days to 2 minutes
*Other end of the connection has more data to send &
Attempts transmission after the 2 min. timer expires
*Packet is dropped by the firewall and logged

Normally a FIN/ACK exchange takes place pretty quickly but per the RFCs
one end does not have to close as well if it does not think its
finished. These can cause the log entries you are experiencing.

I saw this *a lot* back in the beta days when Rusty had the FIN expire
set to one minute. Now that its at two minutes, it happens less
frequently.

Of course I have seen scanning in the wild like this that uses a fixed
source port of 80 so you'll just think* that its a state issue. The
scanning can identify static Vs. stateful firewalling as well as what
hosts are up or down. You need to log outbound HTTP to be able to tell
the difference.

> >  Another peculiar log which we would like to get more info on are
> >
> >  Dec 23 18:43:05 theargonserver kernel: IPT IN_FIREWALL_NEW: IN=eth1 OUT=
> >  MAC=00:03:47:6a:5c:6b:00:c0:49:25:d1:a3:08:00 SRC=80.15.238.66
> >  DST=xxx.xxx.xxx.xxx LEN=73 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF PROTO=UDP
> >  SPT=31819 DPT=53 LEN=53
> >
> >  There are a lot of packets to port 53. Is this normal?
> 
> Yes.   UDP port 53 means client-to-server DNS lookups. 

It could also be server to server queries as well as a load balancer.
See if you have an outbound query that immediately precedes this log
entry. If so, its someone pulling metrics to see what IP to serve up to
you. I'm leaning towards load balancer on this one as the source IP has
no PTR. It could however be one of your users on their home account and
their DNS entries are still pointed at the home office.

BTW, if the above was a successful recursive query you really need to
lock the box down. See the Bugtraq post I did back in Sept. on how
spammers can park their domain on your recursive DNS.

HTH,
C

Re: Packets missing state ?

[Antony Stone]

On Tuesday 23 December 2003 3:21 pm, Chris Brenton wrote:

> On Tue, 2003-12-23 at 08:54, Antony Stone wrote:

> > Yes.   UDP port 53 means client-to-server DNS lookups.
>
> It could also be server to server queries as well as a load balancer.

I thought server to server queries were always TCP?

Antony.

-- 
If builders made buildings the way programmers write programs, then the first 
woodpecker to come along would destroy civilisation.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Packets missing state ?

[Cedric Blancher]

Le mar 23/12/2003 à 16:33, Antony Stone a écrit :
> I thought server to server queries were always TCP?

DNS switches to TCP when payload exceed 512 bytes, such as AXFR.

-- 
http://www.netexit.com/~sid/
PGP KeyID: 157E98EE FingerPrint: FA62226DA9E72FA8AECAA240008B480E157E98EE
>> Hi! I'm your friendly neighbourhood signature virus.
>> Copy me to your signature file and help me spread! 

Re: Packets missing state ?

[Chris Brenton]

On Tue, 2003-12-23 at 10:33, Antony Stone wrote:
>
> > > Yes.   UDP port 53 means client-to-server DNS lookups.
> >
> > It could also be server to server queries as well as a load balancer.
> 
> I thought server to server queries were always TCP?

Nope. UDP is always used unless the "answer" will exceed a packet length
of 512 bytes. If this max is exceeded, as complete an answer as possible
is returned with the truncation bit turned on in the DNS header. After
that, you'll see the session switch over to TCP for a complete answer.

The other case is primary/master <--> secondary/slave communications for
the purpose of zone transfers. 

Dig and other tools will let you choose to use TCP, but the default is
UDP.

HTH,
C

Re: Packets missing state ?

[Cedric Blancher]

Le mar 23/12/2003 à 14:37, Vinayakam Murugan a écrit :
>  We were not able to reproduce it out here.Anyways in this scenario, would
>  the packet would have the state as NEW?  All these logs are of packets with
>  state NEW.

These packets are delayed ones, probably due to high latency (heavy load
on line ?). Most of the time, there's not much to worry about,
especially on DSL line when upload goes up.

>  Another peculiar log which we would like to get more info on are
>  Dec 23 18:43:05 theargonserver kernel: IPT IN_FIREWALL_NEW: IN=eth1 OUT=
>  MAC=00:03:47:6a:5c:6b:00:c0:49:25:d1:a3:08:00 SRC=80.15.238.66
>  DST=xxx.xxx.xxx.xxx LEN=73 TOS=0x00 PREC=0x00 TTL=55 ID=0 DF PROTO=UDP
>  SPT=31819 DPT=53 LEN=53
>  There are a lot of packets to port 53. Is this normal?

Seems that someone wants to ask some DNS stuff to your box :) Maybe this
IP is declared somewhere as authoritative for a domain or someone use it
as forwarder.

-- 
http://www.netexit.com/~sid/
PGP KeyID: 157E98EE FingerPrint: FA62226DA9E72FA8AECAA240008B480E157E98EE
>> Hi! I'm your friendly neighbourhood signature virus.
>> Copy me to your signature file and help me spread! 

Re: Packets missing state ?

[Vinayakam Murugan]

> These packets are delayed ones, probably due to high latency (heavy load
> on line ?). Most of the time, there's not much to worry about,
> especially on DSL line when upload goes up.

How can i uniquely identify such packets so that I don't log them and just 
drop them? Is this possible? 

> Seems that someone wants to ask some DNS stuff to your box :) Maybe this
> IP is declared somewhere as authoritative for a domain or someone use it
> as forwarder.

My primary objective is to log suspicious packets only. Any pointers on how I 
can do that?

-- 
Warm Regards
~~~~~~~~~~~~~~~~~~~~~~~
Vinayakam Murugan

Viruses getting you down?
Get your virus protected mailbox at http://www.tassm.com

Linux: The choice of a GNU generation

Re: Packets missing state ?

[Cedric Blancher]

Le mer 24/12/2003 à 05:12, Vinayakam Murugan a écrit :
> How can i uniquely identify such packets so that I don't log them and just 
> drop them? Is this possible? 

Yes you can. Just remove the logging rule.

> My primary objective is to log suspicious packets only. Any pointers on how I 
> can do that?

The thing with suspicious packets is they're only suspicious... That
means you cannot be sure that they're really malicious, so you can't
drop them. Practically, you'll get far more false positive than real
badly crafted packets.
You can spot what you feel suspicious, but afterwards, you'll have to
analyse your logs in order to see if they're malicious or not. In your
case, they don't seem to be (to me).

-- 
http://www.netexit.com/~sid/
PGP KeyID: 157E98EE FingerPrint: FA62226DA9E72FA8AECAA240008B480E157E98EE
>> Hi! I'm your friendly neighbourhood signature virus.
>> Copy me to your signature file and help me spread! 

Re: Packets missing state ?

[Vinayakam Murugan]

> Yes you can. Just remove the logging rule.

Thanks, Cedric :-) What I meant was is it safe to drop all new packets coming 
in? Is that the usual practice?

-- 
Warm Regards
~~~~~~~~~~~~~~~~~~~~~~~
Vinayakam Murugan

Viruses getting you down?
Get your virus protected mailbox at http://www.tassm.com

Linux: The choice of a GNU generation

Re: Packets missing state ?

[Cedric Blancher]

Le mer 24/12/2003 à 09:18, Vinayakam Murugan a écrit :
> What I meant was is it safe to drop all new packets coming 
> in? Is that the usual practice?

If you don't want incoming connections, you'll have to drop new packets
coming in and it is safe to do so. Just keep your ruleset going.

By the way, I have a comment on it.

	$IPT -A IN_FIREWALL -p tcp -m state --state \
		ESTABLISHED,RELATED -j ACCEPT
	$IPT -A IN_FIREWALL -p udp -m state --state \
		ESTABLISHED,RELATED -j ACCEPT
	$IPT -A IN_FIREWALL -j LOG --log-prefix "IPT IN_FIREWALL: " \
		$LOGOPT
	$IPT -A IN_FIREWALL -j DROP

It would be a good idea to let ICMP traffic go through, at least for
RELATED packets that are ICMP errors generated by your own connections.

	$IPT -A IN_FIREWALL -p icmp -m state --state RELATED -j ACCEPT

ESTABLISHED one are replies to ICMP requests (ping, timestamp, netmask
and info). It's up to you to decide weither letting your firewall ping
is a good idea or not.

-- 
http://www.netexit.com/~sid/
PGP KeyID: 157E98EE FingerPrint: FA62226DA9E72FA8AECAA240008B480E157E98EE
>> Hi! I'm your friendly neighbourhood signature virus.
>> Copy me to your signature file and help me spread!