Re: [IPSEC] Too many SADs!

Re: [IPSEC] Too many SADs!

[Wolfgang Walter]

We had the same problem. Seems to be a limitation of the pfkey-implementation 
of linux.

racoon and setkey both use the pfkey-interface.

We switched to iproute2 and openswan which both use the netfilter-interface. 
Therefor they can handle thousands of SAD and SPD rules.

Greetings,

Wolfgang Walter

Re: [IPSEC] Too many SADs!

[Stephen Frost]

[-- Attachment #1: Type: text/plain, Size: 1273 bytes --]

* Wolfgang Walter (wolfgang.walter@studentenwerk.mhn.de) wrote:
> We had the same problem. Seems to be a limitation of the pfkey-implementation 
> of linux.
> 
> racoon and setkey both use the pfkey-interface.
> 
> We switched to iproute2 and openswan which both use the netfilter-interface. 
> Therefor they can handle thousands of SAD and SPD rules.

Well, that's quite interesting.  I didn't realize there were multiple
interfaces to the IPSEC in Linux.  Additionally, the problem isn't that
I've got too many policies which end up requiring too many SADs- the
problem is that SADs are being created above and beyond what's actually
necessary for my policies, which is a problem.  I'm not entirely sure
why that's happening either.  At one point a SAD was being added every
second when there was *already* an apparently current SAD for the
required policy.  Not good, looks like a bug to me, and I would have
thought it was a kernel bug but I could be wrong there.

I'm certainly curious about the alternative interface to IPSEC in
Linux, and especially your claim that it's a 'netfilter' interface.
I'll certainly look into that...  What kernel are you using?  What
version of iproute2 and Openswan?  Do you have to patch the kernel?

	Stephen

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [IPSEC] Too many SADs!

[Michael Richardson]

-----BEGIN PGP SIGNED MESSAGE-----


>>>>> "Stephen" == Stephen Frost <sfrost@snowman.net> writes:
    Stephen> interfaces to the IPSEC in Linux.  Additionally, the
    Stephen> problem isn't that I've got too many policies which end up
    Stephen> requiring too many SADs- the  problem is that SADs are
    Stephen> being created above and beyond what's actually necessary
    Stephen> for my policies, which is a problem.  I'm not entirely sure

  There is certainly a bug in openswan 2.3.1drX, possibly in 2.3.0,
where more SPD entries get created than necessary.
  This would result in many SAD entries, since the incoming SAs are not
removed until they expire, or the remote end asks for them to be deleted.
 
  As the SAD interface in NETKEY provided by netfilter/pfkey does not
permit any kind of "insert here" option, it is possible that there is
some other bug whereby SAD entries multiply.

- -- 
] Michael Richardson          Xelerance Corporation, Ottawa, ON |  firewalls  [
] mcr @ xelerance.com           Now doing IPsec training, see   |net architect[
] http://www.sandelman.ca/mcr/    www.xelerance.com/training/   |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys

iQCVAwUBQkBoAIqHRg3pndX9AQEb3wQA4NNgcrdmwlloOJPJX+Z8xdfXNA42Gm1P
M7wDT2nFlOavn04FVNPdp45EzITyoICYHkRXSxhorb42lW5mWahRckSjbujMLw9W
bFdpeVqUj+gitmwAs5VYZ2C3KAxiws6puKnINWgxiZgOHiIkAUotAX6jRkPHF8E5
loREL0C1ykM=
=aC1v
-----END PGP SIGNATURE-----

Re: [IPSEC] Too many SADs!

[Stephen Frost]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

* Michael Richardson (mcr@sandelman.ottawa.on.ca) wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> 
> 
> >>>>> "Stephen" == Stephen Frost <sfrost@snowman.net> writes:
>     Stephen> interfaces to the IPSEC in Linux.  Additionally, the
>     Stephen> problem isn't that I've got too many policies which end up
>     Stephen> requiring too many SADs- the  problem is that SADs are
>     Stephen> being created above and beyond what's actually necessary
>     Stephen> for my policies, which is a problem.  I'm not entirely sure
> 
>   There is certainly a bug in openswan 2.3.1drX, possibly in 2.3.0,
> where more SPD entries get created than necessary.

Well, that's interesting, since my problem had been with racoon...

>   This would result in many SAD entries, since the incoming SAs are not
> removed until they expire, or the remote end asks for them to be deleted.
>  
>   As the SAD interface in NETKEY provided by netfilter/pfkey does not
> permit any kind of "insert here" option, it is possible that there is
> some other bug whereby SAD entries multiply.

Got me, but if you're seeing this with openswan too, well, that'd be
rather interesting and might point to a problem outside of the userspace
tools...

	Stephen

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [IPSEC] Too many SADs!

[Scott Mcdermott]

Wolfgang Walter on Tue 22/03 00:52 +0100:
> We had the same problem. Seems to be a limitation of the
> pfkey-implementation of linux.
> 
> racoon and setkey both use the pfkey-interface.
> 
> We switched to iproute2 and openswan which both use the
> netfilter-interface.  Therefor they can handle thousands
> of SAD and SPD rules.

What, openswan uses PF_KEY last I checked on kernel 2.6.  I
guess you can use KLIPS, but why would you? What's this
"netfilter-interface" to ipsec code?

I had the exact same problem the original poster had with
Racoon.  SPDs would multiply without bounds, seemingly
geometrically.

I switched to strongswan and the problems immediately
vanished.  There is some bug in racoon where it doesn't
replace SPDs.  I used the latest ipsec-utils and kernel and
this problem did not go away until I switched instead to
strongswan (still using PF_KEY) (it also worked with
openswan).

Re: [IPSEC] Too many SADs!

[Stephen Frost]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

* Scott Mcdermott (smcdermott@questra.com) wrote:
> What, openswan uses PF_KEY last I checked on kernel 2.6.  I
> guess you can use KLIPS, but why would you? What's this
> "netfilter-interface" to ipsec code?

This confused me too...

> I had the exact same problem the original poster had with
> Racoon.  SPDs would multiply without bounds, seemingly
> geometrically.

Yeah.  Not good. :(

> I switched to strongswan and the problems immediately
> vanished.  There is some bug in racoon where it doesn't
> replace SPDs.  I used the latest ipsec-utils and kernel and
> this problem did not go away until I switched instead to
> strongswan (still using PF_KEY) (it also worked with
> openswan).

Sounds like I may need to check out strongswan/openswan.  
I can tell you I wasn't exactly a fan of freeswan for a variety
of reasons.  I'm suprised there havn't been more people
talking about and looking into fixing this, kind of concerning..

	Thanks,

		Stephen

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [IPSEC] Too many SADs!

[Scott Mcdermott]

Stephen Frost on Tue 22/03 19:33 -0500:
> Sounds like I may need to check out strongswan/openswan.
> I can tell you I wasn't exactly a fan of freeswan for a
> variety of reasons.

What reasons? The userspace code with it is great (i.e. the
IKE daemon).  The kernel stuff may be a different matter.
You could use the native IPSEC code in the kernel instead.

I don't know what distribution you're using but I found it
simple to adapt the openswan .spec file to make a source RPM
for strongswan.

As I understand it, the Openswan project is motivated by
commercial interests, whereas Strongswan is in it for
security and correctness.  I had difficulty using Openswan
with AES (it wasn't accepting custom ciphers and DH groups
specified in the config file, and was sending bogus IKE
proposals with 65535 in all the fields of the first listed
transform) until I switched to Strongswan.  And if you are
doing anything with X.509, the author of that patch is the
one that forked Strongswan.  It has been very solid for me
since I switched off Racoon.

Re: [IPSEC] Too many SADs!

[Stephen Frost]

[-- Attachment #1: Type: text/plain, Size: 4841 bytes --]

* Scott Mcdermott (smcdermott@questra.com) wrote:
> Stephen Frost on Tue 22/03 19:33 -0500:
> > Sounds like I may need to check out strongswan/openswan.
> > I can tell you I wasn't exactly a fan of freeswan for a
> > variety of reasons.
> 
> What reasons? The userspace code with it is great (i.e. the
> IKE daemon).  The kernel stuff may be a different matter.
> You could use the native IPSEC code in the kernel instead.

With freeswan it was: kernel code sucking, difficulty getting
it going, the attitude of upstream esp wrt contributions from
people in the US (sorry, but that's where I am and I don't
plan on moving), and the evil configuration syntax which I
detest.

strongswan/openswan appears to have fixed at least some of
these issues (don't need to patch the kernel and so no KLIPS,
apparently better attitude and maybe not so hard to get it
going) but the evil configuration syntax is still there though
more hacked up to support the newer options which doesn't seem
to have done much for its looks.

I'm now currently concerned about a few things w/
strongswan/openswan: policy definitions given to the kernel by
*swan instead of more directly by me, transport/require ipsec
(esp. to a /24 or similar), AES crashing openswan in its latest
release, 2.6.11 'fwd' kernel policy support (is it there?),
and support for 'default route' tunnels (ie: tunnel from me
out my gw to the rest of the world- I want it encrypted from
me to my gw where it can possible be reencrypted to go out
another tunnel or decrypted and sent in the clear across the
net when ipsec isn't required).

I've also got concerns about the kernel-level stuff that's
independent of IKE: IPSEC and Netfilter interaction (or lack
thereof, though I'm using Patrick's patches now which do
help with that some) esp. wrt filtering, SNAT and DNAT,
IPSEC and routing (esp. wrt multiple ipsec tunnels on a
machine and passing packets w/ the whole decrypt - netfilter
- reencrypt for new tunnel bit), libpcap interaction with
IPSEC transport which doesn't seem to work so hot (it seems
that the IPSEC header has been 0'd out in the packet but
it's still there and libpcap thinks that's the body of the
packet and it can't know any different), and MTU issues
though so far these havn't given me *too* much trouble.

Sorry but I really just can't stand the *swan configuration
syntax. :)  I guess I'll have to give it a shot and possibly
live with it if it works so much better but the whole
left/right/middle/top/bottom/whatever bit just annoys the
hell out of me. :)

> I don't know what distribution you're using but I found it
> simple to adapt the openswan .spec file to make a source RPM
> for strongswan.

Debian/sid.  OpenSwan appears to be in sid w/ version 2.3.0
but I don't see StrongSwan.  Probably wouldn't be too hard
to adopt the OpenSwan deb stuff to StrongSwan and if it turns
out to work so well perhaps I'll upload it to the archive
(I'm a DD and stuff).

> As I understand it, the Openswan project is motivated by
> commercial interests, whereas Strongswan is in it for
> security and correctness.  I had difficulty using Openswan

Being in it for the money can have its advantages, like
people on staff paid full time to work on improving it. :)

> with AES (it wasn't accepting custom ciphers and DH groups
> specified in the config file, and was sending bogus IKE
> proposals with 65535 in all the fields of the first listed
> transform) until I switched to Strongswan.  And if you are
> doing anything with X.509, the author of that patch is the
> one that forked Strongswan.  It has been very solid for me
> since I switched off Racoon.

This is certainly one of my concerns- I'm currently intending
to use AES as much as possible (des3 for some Windows hosts)
and I noticed that the OpenSwan page says 2.3.1 has problems
with it.  Sounds like StrongSwan doesn't have this problem?
Given the similar codebase (at least, I thought they were
similar, perhaps they aren't?) how is it that it works for
one and not the other?

We're certainly using X.509 certs a fair bit and have some
rather serious security dependencies on those certs.  The
StrongSwan/OpenSwan configuration appears to better support
identification-by-certificate than Racoon did (as far as I
could tell anyway) though its requirements wrt what happens
if you want to use subjectAltName seemed a bit odd and may
pose a problem, though I'm hoping to be able to use the DN
for identification but I'm not the CA for these certs so
I'm not 100% sure what the best identifying attribute will
be for me.

Thanks for the comments and concern.  I'd be happy to hear
from others on any of my comments/concerns from above.  Just
looking for the best solution for my requirements.

	Stephen

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [IPSEC] Too many SADs!

[jean-mickael guerin]

> We switched to iproute2 and openswan which both use the netfilter-interface. 

Do you mean netlink interface ?

Jean-Mickael

Re: [IPSEC] Too many SADs!

[KOVACS Krisztian]

[-- Attachment #1: Type: text/plain, Size: 2888 bytes --]


  Hi,

2005-03-22, k keltezéssel 00.52-kor Wolfgang Walter ezt írta:
> We had the same problem. Seems to be a limitation of the pfkey-implementation 
> of linux.

  This is not really a limitation of the PFKEY implementation, at least
not a limitation related to some upper limit of the number of entries
(no such limit exists). The problem itself is caused by the too
simplistic SAD/SPD dumping code in the implementation, which does not
care about possibly lost messages. The problem is as follows: when
setkey requests a dump, the kernel responds with a stream of messages.
Each of these contains the dump of an SAD/SPD entry, and contains a
decreasing sequence number. The sequence number of the last message is
zero, this way the receiving user-space application can tell that the
dump is over.

  The problem occurs when there's not enough space in the socket buffer
of the PFKEY socket. In this case the kernel simply drops all messages
after the buffer becomes full, thus losing the precious end-of-dump
marker last message as well. Racoon's setkey obviously cannot cope with
this and is still waiting for the missing messages.

  The problem itself comes from the unreliable nature of PFKEY, so it's
not something which can be solved, but there are more ways to work it
around. 

  AFAIK early Solaris PFKEY implementations made sure the first and last
message in the dump is delivered. This way the user-space application
has the possibility to detect problems and do something about them.
Another solution is to make dumping an all-or-nothing operation, even at
the cost of the socket buffer growing past its upper limit (this is in
current NetBSD versions, for example). More information about the
problem and its NetBSD workaround can be found here:

  http://mail-index.netbsd.org/tech-net/2004/05/21/0006.html

  As a quick and dirty hack to see if you have the same problem, try
increasing the net.core.rmem_default sysctl value. If you set that to a
larger value than the memory needed for the complete dump you should
have no problems. Of course I wouldn't recommend this as a long-term
"solution".

  Oh, wait, this is where a limitation of ipsec-tools comes into the
picture. For some unknown reason, libipsec (PFKEY interface used by both
setkey and racoon) sets the socket buffer size to 128K for _all_ PFKEY
sockets. So even if you set a much higher default value through sysctl,
setkey dumbly sets the limit to 128K. IMHO this is a bug in ipsec-tools,
it should not impose such limitations. I've attached a quick-and-dirty
patch which removes those ugly setsockopts.

> We switched to iproute2 and openswan which both use the netfilter-interface. 
> Therefor they can handle thousands of SAD and SPD rules.

  Seems to justify the PFKEY problems I've summarized above.

-- 
 Regards,
  Krisztian Kovacs

[-- Attachment #2: ipsec-tools_pfkey_limitation_fix.diff --]
[-- Type: text/x-patch, Size: 519 bytes --]

--- ipsec-tools-0_5-branch/src/libipsec/pfkey.c	2004-09-13 16:09:18.000000000 +0200
+++ ipsec-tools-0_5-branch/src/libipsec/pfkey.c.new	2005-03-23 10:55:50.000000000 +0100
@@ -1713,13 +1713,6 @@
 		return -1;
 	}
 
-	/*
-	 * This is a temporary workaround for KAME PR 154.
-	 * Don't really care even if it fails.
-	 */
-	(void)setsockopt(so, SOL_SOCKET, SO_SNDBUF, &bufsiz, sizeof(bufsiz));
-	(void)setsockopt(so, SOL_SOCKET, SO_RCVBUF, &bufsiz, sizeof(bufsiz));
-
 	__ipsec_errcode = EIPSEC_NO_ERROR;
 	return so;
 }

Re: [IPSEC] Too many SADs!

[Wolfgang Walter]

Am Mittwoch, 23. März 2005 10:57 schrieben Sie:
>   Hi,
>
> 2005-03-22, k keltezéssel 00.52-kor Wolfgang Walter ezt írta:
> > We had the same problem. Seems to be a limitation of the
> > pfkey-implementation of linux.
>
>   This is not really a limitation of the PFKEY implementation, at least
> not a limitation related to some upper limit of the number of entries
> (no such limit exists). The problem itself is caused by the too
> simplistic SAD/SPD dumping code in the implementation, which does not
> care about possibly lost messages. The problem is as follows: when
> setkey requests a dump, the kernel responds with a stream of messages.
> Each of these contains the dump of an SAD/SPD entry, and contains a
> decreasing sequence number. The sequence number of the last message is
> zero, this way the receiving user-space application can tell that the
> dump is over.
>
>   The problem occurs when there's not enough space in the socket buffer
> of the PFKEY socket. In this case the kernel simply drops all messages
> after the buffer becomes full, thus losing the precious end-of-dump
> marker last message as well. Racoon's setkey obviously cannot cope with
> this and is still waiting for the missing messages.
>
>   The problem itself comes from the unreliable nature of PFKEY, so it's
> not something which can be solved, but there are more ways to work it
> around.
>
>   AFAIK early Solaris PFKEY implementations made sure the first and last
> message in the dump is delivered. This way the user-space application
> has the possibility to detect problems and do something about them.
> Another solution is to make dumping an all-or-nothing operation, even at
> the cost of the socket buffer growing past its upper limit (this is in
> current NetBSD versions, for example). More information about the
> problem and its NetBSD workaround can be found here:
>

Thanks for the explanation.

>   http://mail-index.netbsd.org/tech-net/2004/05/21/0006.html
>
>   As a quick and dirty hack to see if you have the same problem, try
> increasing the net.core.rmem_default sysctl value. If you set that to a
> larger value than the memory needed for the complete dump you should
> have no problems. Of course I wouldn't recommend this as a long-term
> "solution".

We tried that. Detected a hard limit you mention later.

I think one should not need to manipulate net.core.rmem_default. It would be 
better if racoon tried to set the buffer size high enough or had at least a 
option to set the size.

>
>   Oh, wait, this is where a limitation of ipsec-tools comes into the
> picture. For some unknown reason, libipsec (PFKEY interface used by both
> setkey and racoon) sets the socket buffer size to 128K for _all_ PFKEY
> sockets. So even if you set a much higher default value through sysctl,
> setkey dumbly sets the limit to 128K. IMHO this is a bug in ipsec-tools,
> it should not impose such limitations. I've attached a quick-and-dirty
> patch which removes those ugly setsockopts.
>

Though this may solve the problem with dump I still think racoon together with 
pfkey will not handle large spd-sets reliable.

We did the following test: start racoon with an empty set and then add 
spd-rules with setkey. Even with a relativ slow rate (say 1 rule/s) the 
kernel seems to drop pfkey-messages once and again and racoon will therfor 
not see these policies. I think that the kernel dropped these messages simply 
because it fails to allocate memory for the message.

Probably this will happen for other pfkey-messages from kernel to userspace, 
too.

> > We switched to iproute2 and openswan which both use the
> > netfilter-interface. Therefor they can handle thousands of SAD and SPD
> > rules.
>
>   Seems to justify the PFKEY problems I've summarized above.

Probably, for the dump-case especially.

But I think that the way racoon works makes it more difficult to ensure that 
kernel and racoon are synchronous.

The relationship between pluto and kernel is simple: pluto is the master of 
its rules: if you want to change policy-rules created by pluto you must not 
change them via the kernel but change them via pluto which then updates them 
in the kernel. On the other side pluto only manipulates its own rules.

With racoon you manipulate the ruleset in the kernel and racoon updates its 
own database from it. If a pfkey-message is lost racoon would have to know 
that so it can dump the spd (res. sad for messages concerning the sad).

I personally think that it would be better if racoon would not read the spd 
but instead create the spd-rules itself from its racoon.conf because your 
racoon.conf must fit your spd anyway (and sometimes racoon already needs to 
create spd-rules itself (for the road-warrior-case i.e.)).


Greetings,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts
EDV
Leopoldstraße 15
80802 München

Re: [IPSEC] Too many SADs!

[Wolfgang Walter]

> What, openswan uses PF_KEY last I checked on kernel 2.6.  I
> guess you can use KLIPS, but why would you? What's this
> "netfilter-interface" to ipsec code?
> 

Sorry, meant netlink-interface.

> I had the exact same problem the original poster had with
> Racoon.  SPDs would multiply without bounds, seemingly
> geometrically.

> I switched to strongswan and the problems immediately
> vanished.  There is some bug in racoon where it doesn't
> replace SPDs.  I used the latest ipsec-utils and kernel and
> this problem did not go away until I switched instead to
> strongswan (still using PF_KEY) (it also worked with
> openswan).

We don't use openswan with KLIPS but with native ipsec.

I'm rather sure that openswan 2.3.0 uses netlink with native ipsec - there is 
no pfkey-socket open when running pluto and pluto opens a netlink-socket.

Does not really matter. The problem of racoon is that it does a spd-dump when 
started. The kernel seems to run out of memory when generating such a huge 
pfkey-message.

The same is true for setkey. You can use it to add thousands of spd-rules but 
you may not dump (and so list) them (you can use iproute2 to check that 
setkey really added those entries).

So we use iproute2 to flush and list our spd and to set up static spd-rules 
(especially those for discard and none policies). We use pluto from openswan 
2.3.0 for IKE.

Greetings,

Wolfgang Walter

Re: [IPSEC] Too many SADs!

[Wolfgang Walter]

> > We had the same problem. Seems to be a limitation of the 
pfkey-implementation 
> > of linux.
> > 
> > racoon and setkey both use the pfkey-interface.
> > 
> > We switched to iproute2 and openswan which both use the 
netfilter-interface. 

Sorry, wanted to say netlink-interface.

> > Therefor they can handle thousands of SAD and SPD rules.
> 

> Well, that's quite interesting.  I didn't realize there were multiple
> interfaces to the IPSEC in Linux.  Additionally, the problem isn't that
> I've got too many policies which end up requiring too many SADs- the
> problem is that SADs are being created above and beyond what's actually
> necessary for my policies, which is a problem.  I'm not entirely sure
> why that's happening either.  At one point a SAD was being added every
> second when there was *already* an apparently current SAD for the
> required policy.  Not good, looks like a bug to me, and I would have
> thought it was a kernel bug but I could be wrong there.

Hmm, we saw this with racoon, too.

I think that linux sometimes can not send pfkey-messages because it can not 
allocate enough memory in that moment. The message get lost and then racoon 
and kernel have different views of the situation. I think the pfkey-protocoll 
is unreliable by design so racoon should handle that.

If you start racoon with no spd-rules and then start to add a lot of them with 
setkey you get a similar behaviour: some of the new rules don't make it to 
racoon and racoon will never learn them. (we tried this as a workaround 
because racoon does not start when more than about 400 spd-rules are already 
set).

That racoon does not start with more then about 400 spd-rules seems to be also 
a problem of the pfkey-implementation: the pfkey-answer to the dump-request 
which racoons does at startup gets to large: the kernel can not allocate 
enough memory.

(and this is the reason setkey can not list the rules, either, so you may use 
it to add more spd-rules it can later list).

> 
> I'm certainly curious about the alternative interface to IPSEC in
> Linux, and especially your claim that it's a 'netfilter' interface.

As mentioned, meant netlink-interface, not netfilter.

> I'll certainly look into that...  What kernel are you using?  What
> version of iproute2 and Openswan?  Do you have to patch the kernel?
> 

iproute2 from debian unstable (20041019-3).

openswan from debian unstable: openswan 2.3.0 (debian version: 2.3.0-2)

>         Stephen
>

Greetings,

Wolfgang Walter

[IPSEC] Too many SADs!

[Stephen Frost]

Greetings,

  This seems to be the right place for Linux 2.6 ipsec issues:

  Linux 2.6.10 + Virtual Server 1.9.4 + Patrick's IPSEC Netfilter patches
  i386 & amd64 (same source for both)
  Debian Racoon & ipsec-tools 0.5-4
  Setting policies using setkey (not using racoon-tool)
  Using both transport and tunnels

  Problem:

  ===# setkey -D | grep '^[0-9]' | wc -l
  recv: Resource temporarily unavailable
  443
  ===# setkey -D | grep mature | wc -l     
  recv: Resource temporarily unavailable
  443
  ===# setkey -D | grep tunnel | wc -l     
  recv: Resource temporarily unavailable
  18
  ===# setkey -D | grep transport | wc -l
  recv: Resource temporarily unavailable
  425
  ===# ps auwx | grep racoon
  root     17722  3.8  2.0 178268 168252 ?       Ss   Mar20  28:39 /usr/sbin/racoon
  ===# setkey -D -P | grep '^[0-9]' | wc -l
  34
  ===# setkey -D -P | grep transport | wc -l
  20
  ===# setkey -D -P | grep tunnel | wc -l   
  14

  I've seen the number of tunnel SADs go up a bunch too on another
  machine.  I see that there's been some changes in 2.6.11.3 (or so?)
  wrt IPSEC and __xfrm_state_find_acq_byseq(), would that likely fix
  this problem?  I don't tend to use /unique:x but rather /require; in
  my policies, would changing that fix this?  I had originally been
  using a /24 for my transport policy and thought changing that to be a
  bunch of /32 policies for the specific machines I'm talking to would
  help- it didn't.

  Occationally (generally when I first get ipsec going between a couple
  machines) I see pmtu problems which kill that ssh, but after that it
  works.  Not a big deal but I see alot of MTU discussion and patches,
  is that expected to be in 2.6.12?

  	Thanks for any help,

		Stephen