Netfilter+IPsec patches

Netfilter+IPsec patches

[Alexander Samad]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]

Patrick

whilst debugging a ipsec bug I noticed these problems

when you do a tcpdump you the decrypted packet seems to show up
 twice it seems to be the exact same packet.

I am running this on a debian 2.6.4 kernel with the netfilter patchs
applied
(up to date cvs)


tcpdump output
=========
13:23:05.868512 0:a:8b:6a:30:8c 0:5:5d:64:c6:4e 0800 150:
202.154.115.130 > 138.130.55.80: ESP(spi=0x6e3852ef,seq=0x29)
13:23:05.868512 0:a:8b:6a:30:8c 0:5:5d:64:c6:4e 0800 98: 192.168.5.1 >
192.168.10.1: icmp: echo request (DF)
13:23:05.868512 0:a:8b:6a:30:8c 0:5:5d:64:c6:4e 0800 98: 192.168.5.1 >
192.168.10.1: icmp: echo request (DF)


my other problem is when I ping across the ipsec tunnel  from the remote
end to the server end I see the packets come in the interface, I see
them in the INPUT table and in the mangle table, but it never seems to
get back to the application



from ipsec auto --status
====
000 "roadwarrior.internet.nat"[4]:
192.168.8.0/22===138.130.55.80:4500[C=AU, ST=NSW, L=Sydney, O=A.Samad
Pty Ltd, OU=Alfred St, CN=sydlxfw01,
E=sydlxfw01@samad.com.au]---138.130.52.1...144.137.104.46:4500[C=AU,
ST=NSW, L=Sydney, O=A.Samad Pty Ltd, OU=Alfred St,
CN=asamadlx.samad.com.au, E=asamadlx@samad.com.au]===192.168.8.2/32;
erouted; eroute owner: #30

Thanks
Alex

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Netfilter+IPsec patches

[Patrick McHardy]

Alexander Samad wrote:
> Patrick
> 
> whilst debugging a ipsec bug I noticed these problems
> 
> when you do a tcpdump you the decrypted packet seems to show up
>  twice it seems to be the exact same packet.

Yes, this is a consequence of passing the packets though the stack again
once we know decapsulation is finished. It will also cause statistics
to account for the packet twice.

> 
> I am running this on a debian 2.6.4 kernel with the netfilter patchs
> applied
> (up to date cvs)
> 
> 
> tcpdump output
> =========
> 13:23:05.868512 0:a:8b:6a:30:8c 0:5:5d:64:c6:4e 0800 150:
> 202.154.115.130 > 138.130.55.80: ESP(spi=0x6e3852ef,seq=0x29)
> 13:23:05.868512 0:a:8b:6a:30:8c 0:5:5d:64:c6:4e 0800 98: 192.168.5.1 >
> 192.168.10.1: icmp: echo request (DF)
> 13:23:05.868512 0:a:8b:6a:30:8c 0:5:5d:64:c6:4e 0800 98: 192.168.5.1 >
> 192.168.10.1: icmp: echo request (DF)
> 
> 
> my other problem is when I ping across the ipsec tunnel  from the remote
> end to the server end I see the packets come in the interface, I see
> them in the INPUT table and in the mangle table, but it never seems to
> get back to the application

Please give some more details on the configuration, like:

Are you using NAT ?
Are you marking the packets in the mangle table ?
Are the packets forwarded when they get out of the tunnel ?

When you see the packets in the INPUT chain, does their source- and
destination address match your policy ?

Regards
Patrick

> 
> 
> 
> from ipsec auto --status
> ====
> 000 "roadwarrior.internet.nat"[4]:
> 192.168.8.0/22===138.130.55.80:4500[C=AU, ST=NSW, L=Sydney, O=A.Samad
> Pty Ltd, OU=Alfred St, CN=sydlxfw01,
> E=sydlxfw01@samad.com.au]---138.130.52.1...144.137.104.46:4500[C=AU,
> ST=NSW, L=Sydney, O=A.Samad Pty Ltd, OU=Alfred St,
> CN=asamadlx.samad.com.au, E=asamadlx@samad.com.au]===192.168.8.2/32;
> erouted; eroute owner: #30
> 
> Thanks
> Alex

Re: Netfilter+IPsec patches

[Alexander Samad]

[-- Attachment #1: Type: text/plain, Size: 2691 bytes --]

On Thu, May 27, 2004 at 02:56:46AM +0200, Patrick McHardy wrote:
> Alexander Samad wrote:
> >Patrick
> >
> >whilst debugging a ipsec bug I noticed these problems
> >
> >when you do a tcpdump you the decrypted packet seems to show up
> > twice it seems to be the exact same packet.
> 
> Yes, this is a consequence of passing the packets though the stack again
> once we know decapsulation is finished. It will also cause statistics
> to account for the packet twice.
Understand that, but I see the unencapsulated packet twice (tcpdump
see's it 3 times enc + 2 x unencap)
> 
> >
> >I am running this on a debian 2.6.4 kernel with the netfilter patchs
> >applied
> >(up to date cvs)
> >
> >
> >tcpdump output
> >=========
> >13:23:05.868512 0:a:8b:6a:30:8c 0:5:5d:64:c6:4e 0800 150:
> >202.154.115.130 > 138.130.55.80: ESP(spi=0x6e3852ef,seq=0x29)
> >13:23:05.868512 0:a:8b:6a:30:8c 0:5:5d:64:c6:4e 0800 98: 192.168.5.1 >
> >192.168.10.1: icmp: echo request (DF)
> >13:23:05.868512 0:a:8b:6a:30:8c 0:5:5d:64:c6:4e 0800 98: 192.168.5.1 >
> >192.168.10.1: icmp: echo request (DF)
> >
> >
> >my other problem is when I ping across the ipsec tunnel  from the remote
> >end to the server end I see the packets come in the interface, I see
> >them in the INPUT table and in the mangle table, but it never seems to
> >get back to the application
> 
> Please give some more details on the configuration, like:
> 
> Are you using NAT ?
> Are you marking the packets in the mangle table ?
> Are the packets forwarded when they get out of the tunnel ?
> 
> When you see the packets in the INPUT chain, does their source- and
> destination address match your policy ?
> 
> Regards
> Patrick
I did some futher testing, I was in NAT-T mode, when I removed the
nat'ing it started to work.

I had not being doing anything in mangle, but I did put in some lines to
accept and to count the packets.

I did the same on the INPUT chain as well and then placed a LOG before
(lines 1&2 on INPUT) and it seemed to have the right information.

The packets are destined for the box that terminates the IPSEC tunnel.
I am presume it met the policy as the packets where decrypted.



> 
> >
> >
> >
> >from ipsec auto --status
> >====
> >000 "roadwarrior.internet.nat"[4]:
> >192.168.8.0/22===138.130.55.80:4500[C=AU, ST=NSW, L=Sydney, O=A.Samad
> >Pty Ltd, OU=Alfred St, CN=sydlxfw01,
> >E=sydlxfw01@samad.com.au]---138.130.52.1...144.137.104.46:4500[C=AU,
> >ST=NSW, L=Sydney, O=A.Samad Pty Ltd, OU=Alfred St,
> >CN=asamadlx.samad.com.au, E=asamadlx@samad.com.au]===192.168.8.2/32;
> >erouted; eroute owner: #30
> >
> >Thanks
> >Alex
> 
> 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Netfilter+IPsec patches

[Stephen Frost]

[-- Attachment #1: Type: text/plain, Size: 2106 bytes --]

* Alexander Samad (alex@samad.com.au) wrote:
> On Thu, May 27, 2004 at 02:56:46AM +0200, Patrick McHardy wrote:
> > Please give some more details on the configuration, like:
> > 
> > Are you using NAT ?
> > Are you marking the packets in the mangle table ?
> > Are the packets forwarded when they get out of the tunnel ?
> > 
> > When you see the packets in the INPUT chain, does their source- and
> > destination address match your policy ?
> I did some futher testing, I was in NAT-T mode, when I removed the
> nat'ing it started to work.

I've run into a rather annoying problem.  I'm not sure if it's due to
the IPSEC patches, but I have some suspicion that it is.  Basically the
story goes like this:

I've got a bunch of network cards in my gateway, in this example we're
concerned w/ 3 of them- two connections to the internet, one internal.
For this to work I have to have source-based routing working (which it
used to, back when I was using 2.4).  It appears to still work fine for
connections which are *not* NAT'd.  For connections which are NAT'd it
goes like this:

eth0 - internet1 (has the 'default' route going out it)
eth1 - internet2 (has a seperate route table w/ a default route)
eth2 - internal

SYN comes in on eth0, NAT'd, goes out eth2, SYN+ACK comes back, that
gets NAT'd and goes out eth0.  All's happy there.

SYN comes in on eth1, NAT'd, goes out eth2, SYN+ACK comes back, that
gets NAT'd to the eth1 address but then gets sent out *eth0* instead.

pings (which aren't NAT'd) to the eth1 address work fine.  So do
traceroute's (again, not NAT'd).  If NAT'ing is turned off and the
machine accepts connections directly then TCP connections also work
fine.

Policy-based routing is in effect, and is working for things which are
not NAT'd.  Things which are NAT'd appear to be going out the main
table's default route though instead of being properly routed.

This is using 2.6.7 w/ recent (week-old) CVS iptables/pom-ng, the 
IPSEC patches and some others (though *not* the new conntrack code).

Thoughts?

	Thanks,

		Stephen

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Netfilter+IPsec patches

[Stephen Frost]

[-- Attachment #1: Type: text/plain, Size: 1141 bytes --]

* Stephen Frost (sfrost@snowman.net) wrote:
> I've got a bunch of network cards in my gateway, in this example we're
> concerned w/ 3 of them- two connections to the internet, one internal.
> For this to work I have to have source-based routing working (which it
> used to, back when I was using 2.4).  It appears to still work fine for
> connections which are *not* NAT'd.  For connections which are NAT'd it
> goes like this:

Alright, so, tried something funny-  If I add a source-route rule for 
the *internal* address of the machine then the source routing works (but,
unfortunately, this breaks things since that machine needs to be able to
accept connections from both internet connections).

I'm guessing this is done because of the packets are going through the 
stack twice, but only going through the routing code once, and that's
happening prior to the NAT'ing?

Please note, these packets aren't IPSEC'd and don't have anything to do
w/ IPSEC stuff.  I'm doing some other IPSEC stuff on one of the
connections at the moment, but that's all working fine (it's on
internet1, so that may help...).

	Stephen

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Netfilter+IPsec patches

[Philip Craig]

Stephen Frost wrote:
> I've run into a rather annoying problem.  I'm not sure if it's due to
> the IPSEC patches, but I have some suspicion that it is.  Basically the
> story goes like this:

This doesn't sound related to the IPSec patches.  Or did it previously
work without the patches?

> I've got a bunch of network cards in my gateway, in this example we're
> concerned w/ 3 of them- two connections to the internet, one internal.
> For this to work I have to have source-based routing working (which it
> used to, back when I was using 2.4).  It appears to still work fine for
> connections which are *not* NAT'd.  For connections which are NAT'd it
> goes like this:
> 
> eth0 - internet1 (has the 'default' route going out it)
> eth1 - internet2 (has a seperate route table w/ a default route)
> eth2 - internal
> 
> SYN comes in on eth0, NAT'd, goes out eth2, SYN+ACK comes back, that
> gets NAT'd and goes out eth0.  All's happy there.

So a packet with an internal source and a destination on the internet
is routed out eth0.  (Note that routing is prior to SNAT.)

> SYN comes in on eth1, NAT'd, goes out eth2, SYN+ACK comes back, that
> gets NAT'd to the eth1 address but then gets sent out *eth0* instead.

And again a packet with an internal source and a destination on the
internet is still routed out eth0.  Nothing unexpected here, although
it isn't what you want.

> pings (which aren't NAT'd) to the eth1 address work fine.  So do
> traceroute's (again, not NAT'd).  If NAT'ing is turned off and the
> machine accepts connections directly then TCP connections also work
> fine.

These are all connections to the machine's eth1 address, not internal
addresses?  So packets with an eth1 source and a destination on the
internet are routed out eth1.  But this is a different source from
above, so you can't really compare them.

> Policy-based routing is in effect, and is working for things which are
> not NAT'd.  Things which are NAT'd appear to be going out the main
> table's default route though instead of being properly routed.

I assume that currently you are only using the separate routing table
if the source address is eth1.  You need to modify this to also use
the separate routing table for replies to packets that came in eth1.
You can do this using CONNMARK.  Set the connmark for packets coming
in eth1, and restore it for packets coming in eth0, then add an ip rule
to use the separate routing table for that fwmark.

Alternatively, you could look at using the ROUTE target instead of
using policy based routing.  I've never tried it though.

-- 
Philip Craig - SnapGear, A CyberGuard Company - http://www.SnapGear.com

Re: Netfilter+IPsec patches

[Stephen Frost]

[-- Attachment #1: Type: text/plain, Size: 4739 bytes --]

* Philip Craig (philipc@snapgear.com) wrote:
> Stephen Frost wrote:
> >I've run into a rather annoying problem.  I'm not sure if it's due to
> >the IPSEC patches, but I have some suspicion that it is.  Basically the
> >story goes like this:
> 
> This doesn't sound related to the IPSec patches.  Or did it previously
> work without the patches?

I'm not 100% sure when the breakage occured, unfortunately.  I can tell
you that things were working fine under 2.4.20 or so.  Then I jumped to
2.6.7 w/ the ipsec patches and set up some ipsec stuffs.  I have a
suspicion of the ipsec patches because of what they do and, well, I'd
hope this isn't broken in the base kernel. :)  Check out the changes in
ipsec-03-policy-lookup to ip_nat_standalone.c; the 'nat_route_key' stuff
is what I'm wondering about, but honestly I don't really know.

> >I've got a bunch of network cards in my gateway, in this example we're
> >concerned w/ 3 of them- two connections to the internet, one internal.
> >For this to work I have to have source-based routing working (which it
> >used to, back when I was using 2.4).  It appears to still work fine for
> >connections which are *not* NAT'd.  For connections which are NAT'd it
> >goes like this:
> >
> >eth0 - internet1 (has the 'default' route going out it)
> >eth1 - internet2 (has a seperate route table w/ a default route)
> >eth2 - internal
> >
> >SYN comes in on eth0, NAT'd, goes out eth2, SYN+ACK comes back, that
> >gets NAT'd and goes out eth0.  All's happy there.
> 
> So a packet with an internal source and a destination on the internet
> is routed out eth0.  (Note that routing is prior to SNAT.)

This is DNAT'ing, btw, I wasn't specific about that before, sorry.
un-NAT'ing should happen pre-routing, I think...  Like this:
Packet shows up on eth1 from 1.1.1.1 -> 1.2.3.4
Gateway NAT's from 1.2.3.4 -> 10.1.2.3 (interal address)
Gateway does routing on NAT'd packet, finds eth2
Gateway sends packet out eth2 to the 10.1.2.3 machine
10.1.2.3 machine sends reply from 10.1.2.3 to 1.1.1.1
Gateway unNAT's from 10.1.2.3 -> 1.2.3.4
Gateway does routing on unNAT'd packet, should find eth1
Gateway sends packet out eth1 to the 1.1.1.1 machine

> >SYN comes in on eth1, NAT'd, goes out eth2, SYN+ACK comes back, that
> >gets NAT'd to the eth1 address but then gets sent out *eth0* instead.
> 
> And again a packet with an internal source and a destination on the
> internet is still routed out eth0.  Nothing unexpected here, although
> it isn't what you want.

It breaks policy routing...  At least, that's what it seems like to me.
I suppose I can give two internal addresses to the internal machines and
then do source-routing on the internal address but, really, that doesn't
seem quite right to me.

> >pings (which aren't NAT'd) to the eth1 address work fine.  So do
> >traceroute's (again, not NAT'd).  If NAT'ing is turned off and the
> >machine accepts connections directly then TCP connections also work
> >fine.
> 
> These are all connections to the machine's eth1 address, not internal
> addresses?  So packets with an eth1 source and a destination on the
> internet are routed out eth1.  But this is a different source from
> above, so you can't really compare them.

Hrmpf, I suppose not, but I didn't have an easy way to test the other
case.

> >Policy-based routing is in effect, and is working for things which are
> >not NAT'd.  Things which are NAT'd appear to be going out the main
> >table's default route though instead of being properly routed.
> 
> I assume that currently you are only using the separate routing table
> if the source address is eth1.  You need to modify this to also use
> the separate routing table for replies to packets that came in eth1.
> You can do this using CONNMARK.  Set the connmark for packets coming
> in eth1, and restore it for packets coming in eth0, then add an ip rule
> to use the separate routing table for that fwmark.

Well, eth1 has a number of different addresses, really, they're all
aliases off of it, but all but 1 of them are NAT'd to other machines.  I
was thinking I could do fwmark routing but, honestly, I thought this was
one of the nice things about policy routing, being able to route based
on the source address.  I'll have to go check, but I could have sworn
this worked for DNAT'ing and was in fact documented in LARTC as being
'how you do it'.

> Alternatively, you could look at using the ROUTE target instead of
> using policy based routing.  I've never tried it though.

Honestly, I don't particularly like the ROUTE target from a conceptual
standpoint.  I suppose it may help in some situations.

	Thanks,

		Stephen

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Netfilter+IPsec patches

[Alexander Samad]

[-- Attachment #1: Type: text/plain, Size: 5773 bytes --]

On Tue, Aug 17, 2004 at 11:45:49PM -0400, Stephen Frost wrote:
> * Philip Craig (philipc@snapgear.com) wrote:
> > Stephen Frost wrote:
> > >I've run into a rather annoying problem.  I'm not sure if it's due to
> > >the IPSEC patches, but I have some suspicion that it is.  Basically the
> > >story goes like this:
> > 
> > This doesn't sound related to the IPSec patches.  Or did it previously
> > work without the patches?
> 
> I'm not 100% sure when the breakage occured, unfortunately.  I can tell
> you that things were working fine under 2.4.20 or so.  Then I jumped to
> 2.6.7 w/ the ipsec patches and set up some ipsec stuffs.  I have a
> suspicion of the ipsec patches because of what they do and, well, I'd
> hope this isn't broken in the base kernel. :)  Check out the changes in
> ipsec-03-policy-lookup to ip_nat_standalone.c; the 'nat_route_key' stuff
> is what I'm wondering about, but honestly I don't really know.

As far as I remember these patches do change the way the packets flow
through netfitler, if there is a change to them (ie encap or decap) they
get rerouted again with the new information.  But this should only be on
ipsec enc packets, everything else should pass through.

Having said that I seem to remember having an issue with policy based
routing, it only working on the original information, I raised it here,
but got no repsonse back. But this was only for ipsec + Nat'ed packets

> 
> > >I've got a bunch of network cards in my gateway, in this example we're
> > >concerned w/ 3 of them- two connections to the internet, one internal.
> > >For this to work I have to have source-based routing working (which it
> > >used to, back when I was using 2.4).  It appears to still work fine for
> > >connections which are *not* NAT'd.  For connections which are NAT'd it
> > >goes like this:
> > >
> > >eth0 - internet1 (has the 'default' route going out it)
> > >eth1 - internet2 (has a seperate route table w/ a default route)
> > >eth2 - internal
> > >
> > >SYN comes in on eth0, NAT'd, goes out eth2, SYN+ACK comes back, that
> > >gets NAT'd and goes out eth0.  All's happy there.
> > 
> > So a packet with an internal source and a destination on the internet
> > is routed out eth0.  (Note that routing is prior to SNAT.)
I think this has changed, because packets can get enc after snat and
thus are allowed to be rerouted, but this should only happen to ipsec
packets 


> 
> This is DNAT'ing, btw, I wasn't specific about that before, sorry.
> un-NAT'ing should happen pre-routing, I think...  Like this:
> Packet shows up on eth1 from 1.1.1.1 -> 1.2.3.4
> Gateway NAT's from 1.2.3.4 -> 10.1.2.3 (interal address)
> Gateway does routing on NAT'd packet, finds eth2
> Gateway sends packet out eth2 to the 10.1.2.3 machine
> 10.1.2.3 machine sends reply from 10.1.2.3 to 1.1.1.1
> Gateway unNAT's from 10.1.2.3 -> 1.2.3.4
> Gateway does routing on unNAT'd packet, should find eth1
> Gateway sends packet out eth1 to the 1.1.1.1 machine
> 
> > >SYN comes in on eth1, NAT'd, goes out eth2, SYN+ACK comes back, that
> > >gets NAT'd to the eth1 address but then gets sent out *eth0* instead.
> > 
> > And again a packet with an internal source and a destination on the
> > internet is still routed out eth0.  Nothing unexpected here, although
> > it isn't what you want.
> 
> It breaks policy routing...  At least, that's what it seems like to me.
> I suppose I can give two internal addresses to the internal machines and
> then do source-routing on the internal address but, really, that doesn't
> seem quite right to me.
> 
> > >pings (which aren't NAT'd) to the eth1 address work fine.  So do
> > >traceroute's (again, not NAT'd).  If NAT'ing is turned off and the
> > >machine accepts connections directly then TCP connections also work
> > >fine.
> > 
> > These are all connections to the machine's eth1 address, not internal
> > addresses?  So packets with an eth1 source and a destination on the
> > internet are routed out eth1.  But this is a different source from
> > above, so you can't really compare them.
> 
> Hrmpf, I suppose not, but I didn't have an easy way to test the other
> case.
> 
> > >Policy-based routing is in effect, and is working for things which are
> > >not NAT'd.  Things which are NAT'd appear to be going out the main
> > >table's default route though instead of being properly routed.
> > 
> > I assume that currently you are only using the separate routing table
> > if the source address is eth1.  You need to modify this to also use
> > the separate routing table for replies to packets that came in eth1.
> > You can do this using CONNMARK.  Set the connmark for packets coming
> > in eth1, and restore it for packets coming in eth0, then add an ip rule
> > to use the separate routing table for that fwmark.
> 
> Well, eth1 has a number of different addresses, really, they're all
> aliases off of it, but all but 1 of them are NAT'd to other machines.  I
> was thinking I could do fwmark routing but, honestly, I thought this was
> one of the nice things about policy routing, being able to route based
> on the source address.  I'll have to go check, but I could have sworn
> this worked for DNAT'ing and was in fact documented in LARTC as being
> 'how you do it'.
> 
> > Alternatively, you could look at using the ROUTE target instead of
> > using policy based routing.  I've never tried it though.
> 
> Honestly, I don't particularly like the ROUTE target from a conceptual
> standpoint.  I suppose it may help in some situations.
> 
> 	Thanks,
> 
> 		Stephen

I think the only way to really test is build a kernel with out the ipsec
patchs and try or look through the code



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Netfilter+IPsec patches

[Philip Craig]

Stephen Frost wrote:
> This is DNAT'ing, btw, I wasn't specific about that before, sorry.
> un-NAT'ing should happen pre-routing, I think...  Like this:
> Packet shows up on eth1 from 1.1.1.1 -> 1.2.3.4
> Gateway NAT's from 1.2.3.4 -> 10.1.2.3 (interal address)
> Gateway does routing on NAT'd packet, finds eth2
> Gateway sends packet out eth2 to the 10.1.2.3 machine
> 10.1.2.3 machine sends reply from 10.1.2.3 to 1.1.1.1
> Gateway unNAT's from 10.1.2.3 -> 1.2.3.4
> Gateway does routing on unNAT'd packet, should find eth1
> Gateway sends packet out eth1 to the 1.1.1.1 machine

I don't know what the ipsec patches change, but for unpatched kernels,
this is how it works:  If you DNAT in one direction, there is an
automatic SNAT in the other direction (or unNAT as you call it)
which is performed in the POST_ROUTING hook.  So for your example,
the routing is based on a packet with source 10.1.2.3.

-- 
Philip Craig - SnapGear, A CyberGuard Company - http://www.SnapGear.com

Re: Netfilter+IPsec patches

[Patrick McHardy]

Stephen Frost wrote:

>* Stephen Frost (sfrost@snowman.net) wrote:
>  
>
>>I've got a bunch of network cards in my gateway, in this example we're
>>concerned w/ 3 of them- two connections to the internet, one internal.
>>For this to work I have to have source-based routing working (which it
>>used to, back when I was using 2.4).  It appears to still work fine for
>>connections which are *not* NAT'd.  For connections which are NAT'd it
>>goes like this:
>>    
>>
>
>Alright, so, tried something funny-  If I add a source-route rule for 
>the *internal* address of the machine then the source routing works (but,
>unfortunately, this breaks things since that machine needs to be able to
>accept connections from both internet connections).
>
>I'm guessing this is done because of the packets are going through the 
>stack twice, but only going through the routing code once, and that's
>happening prior to the NAT'ing?
>
>Please note, these packets aren't IPSEC'd and don't have anything to do
>w/ IPSEC stuff.  I'm doing some other IPSEC stuff on one of the
>connections at the moment, but that's all working fine (it's on
>internet1, so that may help...).
>  
>
It looks like it has something to do with the ipsec patches rerouting in
POSTROUTING after NAT. Please send your exact routes and rules, I can't
figure out the exact problem.

Regards
Patrick