RST instead of FIN?

RST instead of FIN?

[Jay Levitt]

[-- Attachment #1: Type: text/plain, Size: 1012 bytes --]

On my Mandrake 9.0 box (2.4.19-38 kernel), a few times an hour I see TCP packets after the end of a TCP session, which result in log/drops in iptables.  I ran ethereal to capture one such session, and found that in this instance I seem to be the party at fault - but I have no idea why or what to do...

66.35.250.206, a sourceforge.net box, connects to my mail server to deliver an e-mail, as follows:

sourceforge: [SYN]
me: [SYN, ACK]
sourceforge: [ACK]
[SMTP conversation ensues, switches to TLS, sends me an e-mail.  at the end..]
me: [RST]
sourceforge: [FIN, ACK]
me: [RST]
me: [RST]

Since I'd already RST the connection, the [FIN, ACK] was of course treated as un-ESTABLISHED, and so was logged and dropped.  

But what could cause me to be sending three RSTs at the end of a conversation instead of a FIN?  Could that be a sendmail problem (I'm running 8.12.10), a kernel problem, something else?  The whole conversation took 5 seconds, so there are no timeouts occurring...

Jay Levitt

[-- Attachment #2: Type: text/html, Size: 2126 bytes --]

Re: RST instead of FIN?

[Antony Stone]

On Saturday 10 April 2004 7:33 pm, Jay Levitt wrote:

> 66.35.250.206, a sourceforge.net box, connects to my mail server to deliver
> an e-mail, as follows:
>
> sourceforge: [SYN]
> me: [SYN, ACK]
> sourceforge: [ACK]
> [SMTP conversation ensues, switches to TLS, sends me an e-mail.  at the
> end..] me: [RST]
> sourceforge: [FIN, ACK]
> me: [RST]
> me: [RST]

Makes no sense to me.   Maybe you could check whether this happens when other 
mail servers send you an email (ie: is it a consistent thing with your mail 
server), or even whether it happens *every* time this particular mail server 
talks to you, or only sometimes?

I can't suggest why it might be happening, but I'd suggest turn on sendmail 
debugging/verbose logging in order to see what it thinks is really going on.   
I think it's an application-layer thing rather than a kernel/networking 
thing, but that's a guess.

Regards,

Antony.

-- 
Software development can be quick, high quality, or low cost.

The customer gets to pick any two out of three.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: RST instead of FIN?

[Jay Levitt]

It happens on many servers, not just sourceforge - in fact, I get it on the
netfilter server as well.  And it's only sometimes for any given server, not
all the time...

I'll try turning on sendmail logging.  I wasn't sure if there was anything
an application could even do to trigger an RST, but I haven't done tcp
programming in quite a while...
----- Original Message -----
From: "Antony Stone" <Antony@Soft-Solutions.co.uk>
To: <netfilter@lists.netfilter.org>
Sent: Saturday, April 10, 2004 6:54 PM
Subject: Re: RST instead of FIN?


> On Saturday 10 April 2004 7:33 pm, Jay Levitt wrote:
>
> > 66.35.250.206, a sourceforge.net box, connects to my mail server to
deliver
> > an e-mail, as follows:
> >
> > sourceforge: [SYN]
> > me: [SYN, ACK]
> > sourceforge: [ACK]
> > [SMTP conversation ensues, switches to TLS, sends me an e-mail.  at the
> > end..] me: [RST]
> > sourceforge: [FIN, ACK]
> > me: [RST]
> > me: [RST]
>
> Makes no sense to me.   Maybe you could check whether this happens when
other
> mail servers send you an email (ie: is it a consistent thing with your
mail
> server), or even whether it happens *every* time this particular mail
server
> talks to you, or only sometimes?
>
> I can't suggest why it might be happening, but I'd suggest turn on
sendmail
> debugging/verbose logging in order to see what it thinks is really going
on.
> I think it's an application-layer thing rather than a kernel/networking
> thing, but that's a guess.
>
> Regards,
>
> Antony.
>
> --
> Software development can be quick, high quality, or low cost.
>
> The customer gets to pick any two out of three.
>
>                                                      Please reply to the
list;
>                                                            please don't CC
me.
>
>
>

Re: RST instead of FIN?

[Chris Brenton]

On Sat, 2004-04-10 at 14:33, Jay Levitt wrote:
>
> sourceforge: [SYN]
> me: [SYN, ACK]
> sourceforge: [ACK]
> [SMTP conversation ensues, switches to TLS, sends me an e-mail.  at
> the end..]
> me: [RST]

Weird. Are you sure this is not a RST/ACK? 

> sourceforge: [FIN, ACK]

Looks like the RST was ignored (although hard to say since you did not
include time stamps). Does the source MAC on the RST match your system?
When I've seen this in the past its been an IDS or IPS attempting to
reset the session due to a suspicious payload, but they get the sequence
numbers wrong. Thus the RST/ACK gets ignored and the session continues.

> me: [RST]
> me: [RST]

If this is an RST rather than a RST/ACK, it could be your system is
losing session info and handling the ACKs like they are new packets
(maybe some kind of broken IP wrapper application?). The second RST is
*really* odd as its an error packet without any stimulus. That's not
suppose to happen. 

I'm guessing this is not the kernel or Sendmail, but I'm honestly not
sure what it is.

Happy hunting,
Chris

Re: RST instead of FIN?

[Jay Levitt]

Chris Brenton wrote:

> On Sat, 2004-04-10 at 14:33, Jay Levitt wrote:
> >
> > sourceforge: [SYN]
> > me: [SYN, ACK]
> > sourceforge: [ACK]
> > [SMTP conversation ensues, switches to TLS, sends me an e-mail.  at
> > the end..]
> > me: [RST]
>
> Weird. Are you sure this is not a RST/ACK?
Yep.  It's an RST only.

>
> > sourceforge: [FIN, ACK]
>
> Looks like the RST was ignored (although hard to say since you did not
> include time stamps). Does the source MAC on the RST match your system?

Sorry about that - I can't figure out how to get an abbreviated output from
Ethereal so I just retyped it.  I've included the full output of the last
few packets below, although I see now that timestamps are still missing!...
The RST was sent within microseconds of the last packet received.  The
source MAC is my own....

OOH!  Looks like I read this wrong the first time.  The first RST is me, for
reasons unknown, and the second two are sourceforge.  That's even weirder.
With timestamps:

#753 17:20:34.230099 sourceforge: last data packet of message body
#754 17:20:34.230181 me: RST
#755 17:20:34.230538 sourceforge: FIN, ACK
#756 17:20:34.318588 sourceforge: RST
#757 17:20:34.319745 sourceforge: RST

> When I've seen this in the past its been an IDS or IPS attempting to
> reset the session due to a suspicious payload, but they get the sequence
> numbers wrong. Thus the RST/ACK gets ignored and the session continues.

Interesting.  I'm not running an IDS/IPS.  Perhaps sourceforge is, but that
doesn't explain my sending the RST...

> > me: [RST]
> > me: [RST]
>
> If this is an RST rather than a RST/ACK, it could be your system is
> losing session info and handling the ACKs like they are new packets
> (maybe some kind of broken IP wrapper application?).

No wrappers installed here.. just iptables.

> The second RST is
> *really* odd as its an error packet without any stimulus. That's not
> suppose to happen.

Agreed..

> I'm guessing this is not the kernel or Sendmail, but I'm honestly not
> sure what it is.

Any ideas where I might seek out other experts?

Thanks for the help...

Jay

-------------------------

Frame 753 (95 bytes on wire, 95 bytes captured)
Ethernet II, Src: 00:20:78:d0:44:8f, Dst: 00:50:2c:01:62:8e
Internet Protocol, Src Addr: 66.35.250.206 (66.35.250.206), Dst Addr:
192.168.1.150 (192.168.1.150)
Transmission Control Protocol, Src Port: 42185 (42185), Dst Port: smtp (25),
Seq: 2495007464, Ack: 3573134794, Len: 29
    Source port: 42185 (42185)
    Destination port: smtp (25)
    Sequence number: 2495007464
    Next sequence number: 2495007493
    Acknowledgement number: 3573134794
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
    Window size: 14480
    Checksum: 0x4684 (correct)
    Options: (12 bytes)
Simple Mail Transfer Protocol

Frame 754 (54 bytes on wire, 54 bytes captured)
Ethernet II, Src: 00:50:2c:01:62:8e, Dst: 00:20:78:d0:44:8f
Internet Protocol, Src Addr: 192.168.1.150 (192.168.1.150), Dst Addr:
66.35.250.206 (66.35.250.206)
Transmission Control Protocol, Src Port: smtp (25), Dst Port: 42185 (42185),
Seq: 3573134794, Ack: 0, Len: 0
    Source port: smtp (25)
    Destination port: 42185 (42185)
    Sequence number: 3573134794
    Header length: 20 bytes
    Flags: 0x0004 (RST)
    Window size: 0
    Checksum: 0x8109 (correct)

Frame 755 (66 bytes on wire, 66 bytes captured)
Ethernet II, Src: 00:20:78:d0:44:8f, Dst: 00:50:2c:01:62:8e
Internet Protocol, Src Addr: 66.35.250.206 (66.35.250.206), Dst Addr:
192.168.1.150 (192.168.1.150)
Transmission Control Protocol, Src Port: 42185 (42185), Dst Port: smtp (25),
Seq: 2495007493, Ack: 3573134794, Len: 0
    Source port: 42185 (42185)
    Destination port: smtp (25)
    Sequence number: 2495007493
    Acknowledgement number: 3573134794
    Header length: 32 bytes
    Flags: 0x0011 (FIN, ACK)
    Window size: 14480
    Checksum: 0x877f (correct)
    Options: (12 bytes)

Frame 756 (60 bytes on wire, 60 bytes captured)
Ethernet II, Src: 00:20:78:d0:44:8f, Dst: 00:50:2c:01:62:8e
Internet Protocol, Src Addr: 66.35.250.206 (66.35.250.206), Dst Addr:
192.168.1.150 (192.168.1.150)
Transmission Control Protocol, Src Port: 42185 (42185), Dst Port: smtp (25),
Seq: 2495007464, Ack: 0, Len: 0
    Source port: 42185 (42185)
    Destination port: smtp (25)
    Sequence number: 2495007464
    Header length: 20 bytes
    Flags: 0x0004 (RST)
    Window size: 0
    Checksum: 0xac2e (correct)

Frame 757 (60 bytes on wire, 60 bytes captured)
Ethernet II, Src: 00:20:78:d0:44:8f, Dst: 00:50:2c:01:62:8e
Internet Protocol, Src Addr: 66.35.250.206 (66.35.250.206), Dst Addr:
192.168.1.150 (192.168.1.150)
Transmission Control Protocol, Src Port: 42185 (42185), Dst Port: smtp (25),
Seq: 2495007464, Ack: 0, Len: 0
    Source port: 42185 (42185)
    Destination port: smtp (25)
    Sequence number: 2495007464
    Header length: 20 bytes
    Flags: 0x0004 (RST)
    Window size: 0
    Checksum: 0xac2e (correct)

Re: RST instead of FIN?

[Ranjeet Shetye]

Jay,

you can use tethereal instead of ethereal and then its very easy to cut
and paste the textual output.

Ranjeet.

On Sun, 2004-04-11 at 11:01, Jay Levitt wrote:
> Chris Brenton wrote:
> 
> > On Sat, 2004-04-10 at 14:33, Jay Levitt wrote:
> > >
> > > sourceforge: [SYN]
> > > me: [SYN, ACK]
> > > sourceforge: [ACK]
> > > [SMTP conversation ensues, switches to TLS, sends me an e-mail.  at
> > > the end..]
> > > me: [RST]
> >
> > Weird. Are you sure this is not a RST/ACK?
> Yep.  It's an RST only.
> 
> >
> > > sourceforge: [FIN, ACK]
> >
> > Looks like the RST was ignored (although hard to say since you did not
> > include time stamps). Does the source MAC on the RST match your system?
> 
> Sorry about that - I can't figure out how to get an abbreviated output from
> Ethereal so I just retyped it.  I've included the full output of the last
> few packets below, although I see now that timestamps are still missing!...
> The RST was sent within microseconds of the last packet received.  The
> source MAC is my own....
> 
> OOH!  Looks like I read this wrong the first time.  The first RST is me, for
> reasons unknown, and the second two are sourceforge.  That's even weirder.
> With timestamps:
> 
> #753 17:20:34.230099 sourceforge: last data packet of message body
> #754 17:20:34.230181 me: RST
> #755 17:20:34.230538 sourceforge: FIN, ACK
> #756 17:20:34.318588 sourceforge: RST
> #757 17:20:34.319745 sourceforge: RST
> 
> > When I've seen this in the past its been an IDS or IPS attempting to
> > reset the session due to a suspicious payload, but they get the sequence
> > numbers wrong. Thus the RST/ACK gets ignored and the session continues.
> 
> Interesting.  I'm not running an IDS/IPS.  Perhaps sourceforge is, but that
> doesn't explain my sending the RST...
> 
> > > me: [RST]
> > > me: [RST]
> >
> > If this is an RST rather than a RST/ACK, it could be your system is
> > losing session info and handling the ACKs like they are new packets
> > (maybe some kind of broken IP wrapper application?).
> 
> No wrappers installed here.. just iptables.
> 
> > The second RST is
> > *really* odd as its an error packet without any stimulus. That's not
> > suppose to happen.
> 
> Agreed..
> 
> > I'm guessing this is not the kernel or Sendmail, but I'm honestly not
> > sure what it is.
> 
> Any ideas where I might seek out other experts?
> 
> Thanks for the help...
> 
> Jay
> 
> -------------------------
> 
> Frame 753 (95 bytes on wire, 95 bytes captured)
> Ethernet II, Src: 00:20:78:d0:44:8f, Dst: 00:50:2c:01:62:8e
> Internet Protocol, Src Addr: 66.35.250.206 (66.35.250.206), Dst Addr:
> 192.168.1.150 (192.168.1.150)
> Transmission Control Protocol, Src Port: 42185 (42185), Dst Port: smtp (25),
> Seq: 2495007464, Ack: 3573134794, Len: 29
>     Source port: 42185 (42185)
>     Destination port: smtp (25)
>     Sequence number: 2495007464
>     Next sequence number: 2495007493
>     Acknowledgement number: 3573134794
>     Header length: 32 bytes
>     Flags: 0x0018 (PSH, ACK)
>     Window size: 14480
>     Checksum: 0x4684 (correct)
>     Options: (12 bytes)
> Simple Mail Transfer Protocol
> 
> Frame 754 (54 bytes on wire, 54 bytes captured)
> Ethernet II, Src: 00:50:2c:01:62:8e, Dst: 00:20:78:d0:44:8f
> Internet Protocol, Src Addr: 192.168.1.150 (192.168.1.150), Dst Addr:
> 66.35.250.206 (66.35.250.206)
> Transmission Control Protocol, Src Port: smtp (25), Dst Port: 42185 (42185),
> Seq: 3573134794, Ack: 0, Len: 0
>     Source port: smtp (25)
>     Destination port: 42185 (42185)
>     Sequence number: 3573134794
>     Header length: 20 bytes
>     Flags: 0x0004 (RST)
>     Window size: 0
>     Checksum: 0x8109 (correct)
> 
> Frame 755 (66 bytes on wire, 66 bytes captured)
> Ethernet II, Src: 00:20:78:d0:44:8f, Dst: 00:50:2c:01:62:8e
> Internet Protocol, Src Addr: 66.35.250.206 (66.35.250.206), Dst Addr:
> 192.168.1.150 (192.168.1.150)
> Transmission Control Protocol, Src Port: 42185 (42185), Dst Port: smtp (25),
> Seq: 2495007493, Ack: 3573134794, Len: 0
>     Source port: 42185 (42185)
>     Destination port: smtp (25)
>     Sequence number: 2495007493
>     Acknowledgement number: 3573134794
>     Header length: 32 bytes
>     Flags: 0x0011 (FIN, ACK)
>     Window size: 14480
>     Checksum: 0x877f (correct)
>     Options: (12 bytes)
> 
> Frame 756 (60 bytes on wire, 60 bytes captured)
> Ethernet II, Src: 00:20:78:d0:44:8f, Dst: 00:50:2c:01:62:8e
> Internet Protocol, Src Addr: 66.35.250.206 (66.35.250.206), Dst Addr:
> 192.168.1.150 (192.168.1.150)
> Transmission Control Protocol, Src Port: 42185 (42185), Dst Port: smtp (25),
> Seq: 2495007464, Ack: 0, Len: 0
>     Source port: 42185 (42185)
>     Destination port: smtp (25)
>     Sequence number: 2495007464
>     Header length: 20 bytes
>     Flags: 0x0004 (RST)
>     Window size: 0
>     Checksum: 0xac2e (correct)
> 
> Frame 757 (60 bytes on wire, 60 bytes captured)
> Ethernet II, Src: 00:20:78:d0:44:8f, Dst: 00:50:2c:01:62:8e
> Internet Protocol, Src Addr: 66.35.250.206 (66.35.250.206), Dst Addr:
> 192.168.1.150 (192.168.1.150)
> Transmission Control Protocol, Src Port: 42185 (42185), Dst Port: smtp (25),
> Seq: 2495007464, Ack: 0, Len: 0
>     Source port: 42185 (42185)
>     Destination port: smtp (25)
>     Sequence number: 2495007464
>     Header length: 20 bytes
>     Flags: 0x0004 (RST)
>     Window size: 0
>     Checksum: 0xac2e (correct)
-- 

Ranjeet Shetye
Senior Software Engineer
Zultys Technologies
Ranjeet dot Shetye2 at Zultys dot com
http://www.zultys.com/
 
The views, opinions, and judgements expressed in this message are solely
those of the author. The message contents have not been reviewed or
approved by Zultys.