Problems with xfrm (IPSec) and multicast

Problems with xfrm (IPSec) and multicast

[Roar Bjørgum Rotvik]

Hi,

I have configured two Linux PC's to use IPSec to encrypt some mcast traffic, using "ip 
xfrm". Each PC has two network cards, one connected to a LAN (unencrypted side, also 
called red side) and one connected to the other node (encrypted side, also called black side).
Currently the setup uses static keys in the SA entries, so IKE is not a problem.

|<-- RED side -->|<- BLACK network ->|<-- RED side -->|
+-------+   +---------+           +---------+   +-------+
| LAN A +---+ IPSEC-A +-----------+ IPSEC-B +---+ LAN B |
+-------+   +---------+           +---------+   +-------+
10.0.10.0/24          192.168.0.0/24          10.0.20.0/24

Configuration:
Kernel tested: Linux-2.6.16.13 + 2.6.17-rc4
LAN A: 10.0.10.0/24
LAN B: 10.0.20.0/24
IPSEC-A: RED IP: 10.0.10.1, BLACK IP: 192.168.0.1
IPSEC-B: RED IP: 10.0.20.1, BLACK IP: 192.168.0.2
RED mcast group used: 239.192.20.1
BLACK mcast group used: 239.192.10.1

IPSEC-A SA and SP entries:
[root@ipsec-a ~]# ip xfrm state
src 192.168.0.1 dst 239.192.10.1
         proto esp spi 0x00000001 reqid 0 mode tunnel
         replay-window 4
         auth sha1 0x00000000010000000002000000000301
         enc aes 0x000000000000000000000000000000000000000000000001
         encap type espinudp sport 4500 dport 4500 addr 0.0.0.0

[root@ipsec-a ~]# ip xfrm policy
src 0.0.0.0/0 dst 239.192.20.1/32
         dir in priority 2147483648
         tmpl src 192.168.0.1 dst 239.192.10.1
                 proto esp reqid 0 mode tunnel
src 10.0.10.0/24 dst 239.192.20.1/32
         dir out priority 2147483648
         tmpl src 192.168.0.1 dst 239.192.10.1
                 proto esp reqid 0 mode tunnel
src 0.0.0.0/0 dst 239.192.20.1/32
         dir fwd priority 2147483648
         tmpl src 192.168.0.1 dst 239.192.10.1
                 proto esp reqid 0 mode tunnel

(The entries for IPSEC-B is similar, but the address 192.168.0.1 is changed to the IPSEC-B 
BLACK IP 192.168.0.2).
And I have a small userspace app that opens a socket and bind to port 4500 and issue 
setsockopt (fd, SOL_UDP, UDP_ENCAP) so that the kernel will accept UDP encap ESP packets.

When I send multicast traffic from IPSEC-A (bound to the RED interface 10.0.10.1) to mcast 
group 239.192.20.1, the traffic matches the out SP entry and is encrypted according to the 
SA entry and sent as UDP encap ESP to mcast group 239.192.10.1 on the BLACK network.

On IPSEC-B the UDP encap ESP packet is decrypted and is visible for userspace processes. 
So far so good.

But then I start sending similar mcast traffic the other way, but from IPSEC-B (bound to 
IPSEC-B RED IP 10.0.20.1). This traffic is also encrypted and sent to IPSEC-A.

But this packet is not decrypted at IPSEC-A, it seems to disappear. The IP and UDP SNMP 
counters increase for the received UDP encap ESP packet, but I cannot see what happens to 
the packet after the UDP layer. Seems like it is dropped somewhere in XFRM?

By sending some more packets from IPSEC-B (roughly 5-8 more packets), these packets 
suddenly starts to be decrypted at IPSEC-A and all is well. Until I start traffic the 
other way around again, when the same problem occurs at IPSEC-B.

So I cannot make encrypted multicast traffic to flow both ways at the same time, and has 
no clue as to why the first packets after changing direction is dropped somewhere.

Anyone have a clue to this observed problem with linux xfrm and multicast or a better 
solution for encrypted multicast on linux 2.6.x?

Any help with this is appreciated and more info (tcpdump, snmp stats and so on) can be 
obtained if needed.

-- 
Roar Bjørgum Rotvik

Re: Problems with xfrm (IPSec) and multicast

[Herbert Xu]

On Wed, Jun 14, 2006 at 01:09:59PM +0000, Roar Bj?rgum Rotvik wrote:
> 
> So I cannot make encrypted multicast traffic to flow both ways at the same 
> time, and has no clue as to why the first packets after changing direction 
> is dropped somewhere.

Sounds like conntrack.  Check /proc/net/ip_conntrack when this happens.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt