unexpected behaviour...?

unexpected behaviour...?

[Christof Nyffenegger]

Hello

We're in trouble with our brand new linux hot standby firewall cluster. it 
consists of two machines connected to four networks in total. we are about to 
migrate several connections to external networks to this environment and
tested the setup, i.e. the failover functionality which is provided by 
heartbeat 1.0.1-27 (rpm from SuSE 8.2). the firewall nodes are SuSE 8.2 with 
kernel 2.4.20. shown below is the test setup:


                ---------
               |         |
               | ssh/ftp |
               | server  |
                ---------
               192.168.2.50
                    |
    /---------------------------------------------------/ DMZ2
        |                     |                     |
        | - - - - - - - -192.168.2.3- - - - - - - - |
   192.168.2.1                                 192.168.2.2
    ---------                                   ---------
   |         |                                 |         |
   |  FW A   |                                 |  FW B   |
   |         |                                 |         |
    ---------                                   ---------
   192.168.1.1                                 192.168.1.2
        | - - - - - - - -192.168.1.3- - - - - - - - |
        |                    |                      |
    /---------------------------------------------------/ DMZ 1
                                          |
                                     192.168.1.50
                                      ---------
                                     |         |
                                     | ssh/ftp |
                                     | client  |
                                      ---------


- FW A has physical adresses 192.168.1.1 and 192.168.2.1
- FW B has physical adresses 192.168.1.2 and 192.168.2.2
- the cluster service adresses of DMZ 1 and DMZ 2 are
  192.168.1.3 and 192.168.2.3 resp.
- the default gateway on the client is 192.168.1.3
- the server has 192.168.2.3 as its default gateway
- the ruleset is built with fwbuilder 1.0.10
- the rules all have stateful inspection enabled
- established and related connections are accepted prior to first
  rule

test scenario:

- FW A is the active node (it has the service adresses 192.168.1.3
  and 192.168.2.3 rsp.)
- the client initiates an ssh connection to the ssh server in DMZ 2
- the client initiates an ftp connection to the ftp server in DMZ 2
- the client initiates a http donwload over the proxy server
  -> on FW A, the connections appear in the connection table
- failover: the service adresses are moved to FW B, heartbeat does a 
  gratuitous arp
- additionally, the interfaces on FW A are disabled manually
  (just to make things clear...)

now one would expect all the sessions to freeze immediately after failover.
this is the case indeed with the ftp session and the http download - they
freeze to ice because FW B has no established sessions in its connection table
when it acquires the service adresses and the connections are routed through it
(after the gratuitous arp from heartbeat). but surpise - the ssh session 
survives the failover. a few seconds after the failover it shows up in the 
connection table on FW B as an established session. Hmm.

this behaviour was reproduced at will, and we were running sniffers during
consecutive tests. the sniffer logs showed no new ssh session setup after
the failover took place, there was no new tcp handshake or so. the session
just continued as if nothing ever happened.

am i getting something wrong? or is this the expected behaviour?

Christof


__________________________________________________________________

Gesendet von Yahoo! Mail - http://mail.yahoo.de
Logos und Klingeltöne fürs Handy bei http://sms.yahoo.de

Re: unexpected behaviour...?

[Jim Carter]

On Sat, 23 Aug 2003, Christof Nyffenegger wrote:
> We're in trouble with our brand new linux hot standby firewall cluster. it
--- snip ---

> now one would expect all the sessions to freeze immediately after failover.
> this is the case indeed with the ftp session and the http download - they
> freeze to ice because FW B has no established sessions in its connection table
> when it acquires the service adresses and the connections are routed through it
> (after the gratuitous arp from heartbeat). but surpise - the ssh session
> survives the failover. a few seconds after the failover it shows up in the
> connection table on FW B as an established session. Hmm.
--- snip ---
> am i getting something wrong? or is this the expected behaviour?

It's the behavior I expect :-)  When the MAC address of the gateway changes
from firewall A to firewall B, the packets of the SSH connection go there
and are routed through, no sweat.

FTP is another matter.  The control channel is getting through, I'm sure,
but if you fail over, most likely firewall B would not be aware that that
it's a FTP control channel and would not do the conntrack magic necessary
to let the FTP server originate data connections to the client, or in
passive mode, to let the client originate data connections to the arbitrary
ports that the server tells it to use.  Similarly for any service that uses
multiple connections.

On the HTTP proxy action, if you failed over in the middle of a download,
of course it would freeze, because the proxy on the other server would not
have an established connection and the packets would be tossed, but
subsequent downloads should work, since each one is an independently opened
connection.  Unless for some reason the server on firewall A answered using
its own IP address and the client said, hey, I used the proxy on
192.168.2.3 (the shared address), but it's answering from 192.168.2.1, so
I'm supposed to use that from now on.  When the failover happened, the
client would find that it had been too smart for its own good.

You might try setting "ServerName 192.168.2.3:80" in
/etc/apache2/httpd.conf, and see if that makes a difference in the client's
behavior, because the default for ServerName is the box's own IP address
on the interface that the request came from.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: unexpected behaviour...?

[Christof Nyffenegger]

 --- Jim Carter <jimc@math.ucla.edu> schrieb: > On Sat, 23 Aug 2003, Christof
Nyffenegger wrote:
> > We're in trouble with our brand new linux hot standby firewall cluster. it
> --- snip ---
> 
> > now one would expect all the sessions to freeze immediately after failover.
> > this is the case indeed with the ftp session and the http download - they
> > freeze to ice because FW B has no established sessions in its connection
> table
> > when it acquires the service adresses and the connections are routed
> through it
> > (after the gratuitous arp from heartbeat). but surpise - the ssh session
> > survives the failover. a few seconds after the failover it shows up in the
> > connection table on FW B as an established session. Hmm.
> --- snip ---
> > am i getting something wrong? or is this the expected behaviour?
> 
> It's the behavior I expect :-)  When the MAC address of the gateway changes
> from firewall A to firewall B, the packets of the SSH connection go there
> and are routed through, no sweat.

but what about the connection table on firewall B? it contains no established
ssh session - in fact no session at all, since it didn't forward any traffic so
far. only established and related sessions are allowed in its rulebase, plus
explicitly defined NEW sessions of course (for example the forementioned ssh
connection). now i ask myself why the ssh session appears in the connection
table as an established session on firewall B if the session was not
established through firewall B? or maybe i missunderstand the concept of a
stateful firewall from the ground up...? 

> 
> FTP is another matter.  The control channel is getting through, I'm sure,
> but if you fail over, most likely firewall B would not be aware that that
> it's a FTP control channel and would not do the conntrack magic necessary
> to let the FTP server originate data connections to the client, or in
> passive mode, to let the client originate data connections to the arbitrary
> ports that the server tells it to use.  Similarly for any service that uses
> multiple connections.

the ftp session freezes because there is nor a control nor a related data
connection in the connection table on firewall B - no surprise.

> 
> On the HTTP proxy action, if you failed over in the middle of a download,
> of course it would freeze, because the proxy on the other server would not
> have an established connection and the packets would be tossed, but
> subsequent downloads should work, since each one is an independently opened
> connection.  Unless for some reason the server on firewall A answered using
> its own IP address and the client said, hey, I used the proxy on
> 192.168.2.3 (the shared address), but it's answering from 192.168.2.1, so
> I'm supposed to use that from now on.  When the failover happened, the
> client would find that it had been too smart for its own good.
> 

the proxy server is not running on the firewalls - it was a http donwload
*through* firewall A from a browser client to a squid proxy elsewhere in the
network. as i mentioned it's no surprise to see this session freeze after
failover. but the point is that the ssh session did NOT freeze after failover.
i think i understand why the ftp and http sessions froze, but i don't
understand why the ssh session did not.

Thanks for any pointers

__________________________________________________________________

Gesendet von Yahoo! Mail - http://mail.yahoo.de
Logos und Klingeltöne fürs Handy bei http://sms.yahoo.de

Re: unexpected behaviour...?

[Jim Carter]

On Tue, 26 Aug 2003, Christof Nyffenegger wrote:

>  --- Jim Carter <jimc@math.ucla.edu> schrieb:
> but what about the connection table on firewall B? it contains no established
> ssh session - in fact no session at all, since it didn't forward any traffic so
> far. only established and related sessions are allowed in its rulebase, plus
> explicitly defined NEW sessions of course (for example the forementioned ssh
> connection). now i ask myself why the ssh session appears in the connection
> table as an established session on firewall B if the session was not
> established through firewall B? or maybe i missunderstand the concept of a
> stateful firewall from the ground up...?

Hmmm, good point.  However, I think a packet is considered to be part of an
established connection because it has particular header bits set, not
because of IP address matching in the conntrack tables.  Someone who's more
familiar with the sources, could you please confirm or correct this
statement?

> the proxy server is not running on the firewalls - it was a http donwload
> *through* firewall A from a browser client to a squid proxy elsewhere in the
> network. as i mentioned it's no surprise to see this session freeze after
> failover. but the point is that the ssh session did NOT freeze after failover.
> i think i understand why the ftp and http sessions froze, but i don't
> understand why the ssh session did not.

Oh, I didn't understand the configuration.  Hmmm again.  I would really
expect the http and ssh connections to behave the same.  What are the
differences between the connections?  Are the services on different
operating systems (Linux / Windows), so the TCP stack behaves differently?
Maybe you could put the server-side DMZ's switch into promiscuous mode (so
you can snoop on all packets) and run tcpdump / ethereal / snort.  If any
machine is somehow using the IP address of the individual firewall box as
its gateway, its connections won't survive the failover, while if the
shared address is being used, as it should be, it should work.  The same
principle applies on the client side too.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: jimc@math.ucla.edu  http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: unexpected behaviour...?

[Christof Nyffenegger]

 --- Jim Carter <jimc@math.ucla.edu> schrieb:
> Hmmm, good point.  However, I think a packet is considered to be part of an
> established connection because it has particular header bits set, not
> because of IP address matching in the conntrack tables.  Someone who's more
> familiar with the sources, could you please confirm or correct this
> statement?

As far as i (thought to) know, a session has to go through the SYN_SENT and
SYN_RCVD states before reaching the ESTABLISHED state. the firewall box has to
see the TCP SYN, SYN/ACK, ACK packets of a new session before the session is
added to the connection table as an established session. if the firewall later
sees packets belonging to one of the established sessions in it's connection
table, it forwards them if the rulebase is configured so. packets of an
established session which can't be matched against a session in the connection
table - according to the source and destination sockets - are dropped. 

> Oh, I didn't understand the configuration.  Hmmm again.  I would really
> expect the http and ssh connections to behave the same.

Yep, thats the point. but they don't.

> What are the
> differences between the connections?  Are the services on different
> operating systems (Linux / Windows), so the TCP stack behaves differently?

The clients are win 2k and Linux, the servers are running Linux (SuSE 8.2,
Knoppix) and solaris 8.

> Maybe you could put the server-side DMZ's switch into promiscuous mode (so
> you can snoop on all packets) and run tcpdump / ethereal / snort.  If any
> machine is somehow using the IP address of the individual firewall box as
> its gateway, its connections won't survive the failover, while if the
> shared address is being used, as it should be, it should work.  The same
> principle applies on the client side too.

We were running sniffers on the win 2k client (winpcap/ethereal) and on the
servers (tcpdump) as well. on the client and the servers, the gateways were set
correctly to the failover addresses of the firewall boxes.
i'm aware that we didn't sniff the complete traffic on the switched network
segments, but since we didn't see any packets belonging to a new session setup
after failover (we would have seen them in the dumps), imho there is no need to
dig into the stack implementations at this stage. for me it looks really as if
netfilter is not working as it should in this case.


__________________________________________________________________

Gesendet von Yahoo! Mail - http://mail.yahoo.de
Logos und Klingeltöne fürs Handy bei http://sms.yahoo.de

Re: unexpected behaviour...?

[Jim Carter]

On Thu, 28 Aug 2003, Christof Nyffenegger wrote:
> The clients are win 2k and Linux, the servers are running Linux (SuSE 8.2,
> Knoppix) and solaris 8.

(jimc's summary: with a hot failover gateway that shouldn't be passing
random packets, a SSH and a HTTP connection are made, and then the gateway
is exchanged.  The replacement shows an established through-going TCP
connection for SSH, which works as if nothing bad happened, but the HTTP
connection freezes.)

Here at UCLA we had a nasty behavior: another site has a preferred MX and a
failover MX; they panicked over MSBlaster (I'm guessing) and blocked port
25 on the preferred MX.  Sendmail on our Solaris machines considered this a
temporary error, and did not try the failover MX, whereas on Linux,
sendmail did fail over.  It was almost the identical version, just
different OS's and different TCP stacks.

If you could try both ssh and http to the Linux boxes, and both to the
Solaris boxes, and both to the Windows boxes (not that you'd want to :-)
but you could test client-side issues), I'll bet the behavior goes with the
OS, not the port.  Obviously you've tested with both Windows and Linux
clients, so on this hypothesis the server's OS determines what happens.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA  90095-1555
Email: jimc@math.ucla.edu    http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: unexpected behaviour...?

[Chris Brenton]

Christof Nyffenegger wrote:
> 
>>Oh, I didn't understand the configuration.  Hmmm again.  I would really
>>expect the http and ssh connections to behave the same.
> 
> Yep, thats the point. but they don't.

I have not followed this whole thread, so if this idea has been 
suggested already please feel free to /dev/null this.

My experience has been that most SSH clients issue a keep alive while 
HTTP does not. If your accept rule is based on IP and port number only 
(in other words, you are not doing a --tcp-flags ALL SYN), its possible 
that the outbound ACK from the client is recreating the session in the 
state table.

Easy way to check this would be to write your SSH rule with the above 
switch. If SSH starts acting like HTTP (no longer re-creates the state 
table entry) then you know this is it.

HTH,
C

Re: unexpected behaviour...?

[Ralf Spenneberg]

Am Don, 2003-08-28 um 13.33 schrieb Christof Nyffenegger:
>  --- Jim Carter <jimc@math.ucla.edu> schrieb:
> > Hmmm, good point.  However, I think a packet is considered to be part of an
> > established connection because it has particular header bits set, not
> > because of IP address matching in the conntrack tables.  Someone who's more
> > familiar with the sources, could you please confirm or correct this
> > statement?
No bits in the header. It is simply matching the IP addresses, protocol
and ports. If you add the TCP-Window-Tracking patch it does more.
At the moment TCP is handled just like  UDP, ICMP, and generic
protocols.

If the client behind the firewall sends an ACK the connection is
automatically picked up by iptables.

Cheers,

Ralf
-- 
Ralf Spenneberg
RHCE, RHCX

Book: Intrusion Detection für Linux Server   http://www.spenneberg.com
IPsec-Howto				     http://www.ipsec-howto.org
Honeynet Project Mirror:                     http://honeynet.spenneberg.org

unexpected behaviour...?

[Christof Nyffenegger]

--- Ralf Spenneberg <lists@spenneberg.org> schrieb:

> No bits in the header. It is simply matching the IP addresses, protocol
> and ports. If you add the TCP-Window-Tracking patch it does more.
> At the moment TCP is handled just like  UDP, ICMP, and generic
> protocols.
>
> If the client behind the firewall sends an ACK the connection is
> automatically picked up by iptables.

OK, i see (after *reading* some netfilter documentation as
well....RTFM...sorry). I thought that netfilter is doing *real* tcp state
tracking which is not true. to quote Joszef Kadlecsik (from "connection
tracking" by James C. Stephens):

---- snip ----

"When a packet with the SYN+ACK flags set arrives in response to a packet with
SYN set the connection tracking thinks: "I have been just seeing a packet with
SYN+ACK which answers a SYN I had previously seen, so this is an ESTABLISHED
connection."

The important point here is that the conntrack states are not equivalent to tcp
states. We have already seen that a connection doesn't achieve the tcp
connection status of ESTABLISHED until the ACK after the SYN+ACK has been
received.

The representation of the tcp connection states in the state table is purely
for timeouts. You can prove this to yourself by sending an ACK packet through
your firewall to a non-existent machine (so that you don't get the RST back).
It will create a state table entry no problem because it it is the first packet
of a connection and so is treated as NEW (the entry will not be marked as
ASSURED though).

---- snip ----

Well, that explains it. thank you very much!


__________________________________________________________________

Gesendet von Yahoo! Mail - http://mail.yahoo.de
Logos und Klingeltöne fürs Handy bei http://sms.yahoo.de