ProxyARP and IPSec

ProxyARP and IPSec

[H. Peter Anvin]

Hello all,

I am having a puzzlement combining ProxyARP and IPsec.  Specificially, I 
want to take a single address from a local LAN and extend it via IPsec 
to another site.

Unfortunately IPsec tunnels, unlike all other tunnels, don't have 
pseudo-devices associated with them.  I understand this to be from 
desire of uniformity with the other modes of IPsec, but this is one of 
many cases where it causes problems.

Specifically, Linux will not ProxyARP for an address unless it has a 
route for it, *and* that route either has a DNAT marking or points to a 
different interface than the input interface:

net/ipv4/arp.c:

    855                  } else if (IN_DEV_FORWARD(in_dev)) {
    856                          if ((rt->rt_flags&RTCF_DNAT) ||
    857                              (addr_type == RTN_UNICAST  && 
rt->u.dst.dev != dev &&
^^^^^^^^^^^^^^^^^^^^
    858                               (arp_fwd_proxy(in_dev, rt) || 
pneigh_lookup(&arp_tbl, &tip, dev, 0)))) {

However, since IPsec tunnels don't have interfaces associated with it, 
the route to the other side of the IPsec tunnel with point to the same 
interface (there is, elsewhere, a security policy associated with the 
address), and this selection will fail.

Does anyone know of a trick around this issue?

	-hpa

Re: ProxyARP and IPSec

[Thomas Graf]

* H. Peter Anvin <hpa@zytor.com> 2006-08-22 17:31
> Specifically, Linux will not ProxyARP for an address unless it has a 
> route for it, *and* that route either has a DNAT marking or points to a 
> different interface than the input interface:

I can think of a very ugly way: Use netfilter to match on the
arp packet prerouting, set nfmark to some value, create a routing
rule matching the fwmark again, have it look up a separate table
with a dummy route pointing to a dummy device. Make sure to have
a proxy neighbour entry as using the device proxy_arp sysctl would
fail again.

Re: ProxyARP and IPSec

[David Miller]

From: Thomas Graf <tgraf@suug.ch>
Date: Wed, 23 Aug 2006 21:14:25 +0200

> * H. Peter Anvin <hpa@zytor.com> 2006-08-22 17:31
> > Specifically, Linux will not ProxyARP for an address unless it has a 
> > route for it, *and* that route either has a DNAT marking or points to a 
> > different interface than the input interface:
> 
> I can think of a very ugly way: Use netfilter to match on the
> arp packet prerouting, set nfmark to some value, create a routing
> rule matching the fwmark again, have it look up a separate table
> with a dummy route pointing to a dummy device. Make sure to have
> a proxy neighbour entry as using the device proxy_arp sysctl would
> fail again.

This shows we have a usability problem if that's the only way
to do this :-)

What he's trying to accomplish doesn't sound all that weird,
does anyone have any other ideas?

Re: ProxyARP and IPSec

[Alexey Kuznetsov]

Hello!

> What he's trying to accomplish doesn't sound all that weird,

Absolutely sane.


> does anyone have any other ideas?

The question is where is this host really?

If it is far far away and connected only via IPsec tunnel with destionation
of tunnel different of host address

ip ro add THEHOST dev dummy0

should be enough. It asserts that THEHOST is not on eth0.
IPsec policy will figure out correct route, unless something is broken.

If tunnel endpoint is THEHOST, then it is necessary to make a prescription
how to reach it bypassing IPsec. This can be made with a rule telling
that THEHOST is reachable from router and only from router:

ip ru add from OUR_TUNNEL_ENDPOINT to THEHOST table XXX
ip ro add THEHOST via THAT_ROUTE_WHICH_IS_SUPPOSED_TO_KNOW table XXX

Alexey

Re: ProxyARP and IPSec

[H. Peter Anvin]

Alexey Kuznetsov wrote:
> 
> The question is where is this host really?
> 
> If it is far far away and connected only via IPsec tunnel with destionation
> of tunnel different of host address
> 
> ip ro add THEHOST dev dummy0
> 
> should be enough. It asserts that THEHOST is not on eth0.
> IPsec policy will figure out correct route, unless something is broken.
> 

Just tried it, and it works as advertised.

	-hpa

Re: ProxyARP and IPSec

[H. Peter Anvin]

H. Peter Anvin wrote:
> Alexey Kuznetsov wrote:
>>
>> The question is where is this host really?
>>
>> If it is far far away and connected only via IPsec tunnel with 
>> destionation
>> of tunnel different of host address
>>
>> ip ro add THEHOST dev dummy0
>>
>> should be enough. It asserts that THEHOST is not on eth0.
>> IPsec policy will figure out correct route, unless something is broken.
>>
> 
> Just tried it, and it works as advertised.
> 

... except that OpenSwan will rip out the route and install a route 
pointing to eth0, thus breaking the thing again.

	-hpa

Re: ProxyARP and IPSec

[Andy Gay]

On Wed, 2006-08-23 at 18:14 -0700, H. Peter Anvin wrote:
> H. Peter Anvin wrote:
> > Alexey Kuznetsov wrote:
> >>
> >> The question is where is this host really?
> >>
> >> If it is far far away and connected only via IPsec tunnel with 
> >> destionation
> >> of tunnel different of host address
> >>
> >> ip ro add THEHOST dev dummy0
> >>
> >> should be enough. It asserts that THEHOST is not on eth0.
> >> IPsec policy will figure out correct route, unless something is broken.
> >>
> > 
> > Just tried it, and it works as advertised.
> > 
> 
> ... except that OpenSwan will rip out the route and install a route 
> pointing to eth0, thus breaking the thing again.

Use a custom updown script with Openswan to fix that.

> 
> 	-hpa
> 
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: ProxyARP and IPSec

[H. Peter Anvin]

Andy Gay wrote:
>>>>
>>> Just tried it, and it works as advertised.
>>>
>> ... except that OpenSwan will rip out the route and install a route 
>> pointing to eth0, thus breaking the thing again.
> 
> Use a custom updown script with Openswan to fix that.
> 

*Nod.*

I'm thinking that David definitely has a point about having a usability 
problem, though.  All other kind of tunnels have endpoint devices 
associated with them, and that would make all these kinds of problems go 
away, plus would be nicer to deal with in iptables, dealing with routed 
IPsec connections, etc.

	-hpa

Re: ProxyARP and IPSec

[Thomas Graf]

* David Miller <davem@davemloft.net> 2006-08-23 15:14
> From: Thomas Graf <tgraf@suug.ch>
> Date: Wed, 23 Aug 2006 21:14:25 +0200
> 
> > * H. Peter Anvin <hpa@zytor.com> 2006-08-22 17:31
> > > Specifically, Linux will not ProxyARP for an address unless it has a 
> > > route for it, *and* that route either has a DNAT marking or points to a 
> > > different interface than the input interface:
> > 
> > I can think of a very ugly way: Use netfilter to match on the
> > arp packet prerouting, set nfmark to some value, create a routing
> > rule matching the fwmark again, have it look up a separate table
> > with a dummy route pointing to a dummy device. Make sure to have
> > a proxy neighbour entry as using the device proxy_arp sysctl would
> > fail again.
> 
> This shows we have a usability problem if that's the only way
> to do this :-)

What about adding blackhole device to be used for such routes.
I believe it would be good architecture to always use devices
to state directions packets are being received from and sent to.

Re: ProxyARP and IPSec

[Alexey Kuznetsov]

Hello!

> I'm thinking that David definitely has a point about having a usability 
> problem, though.  All other kind of tunnels have endpoint devices 
> associated with them, and that would make all these kinds of problems go 
> away, 

Yes, when you deal with sane practical setups, this approach is the only
reasonable one.

Unfortunately, IPsec is not something totally sane and practical :-),
"security gateway" case is small part of it and "routing" viewpoint
clashes fatally with another requirements. Pure result is that we use approach
where it is possible to do everything with some efforts, rather than approach
which is simple and intuitive, but does not allow to do many things.

It is possible to simulate simple life, creating "ipsecX" devices
with disabled xfrm and route all the tunnels there. That would be handy.

I would just advice to rename one of dummy devices to "ipsec0"
and route all the IPsec tunnels there. It is also simple.

What's about iptables, I am sorry, it is too flexible to control IPsec. :-)
One day, someone with enough of energy and stamina will make flow cache
to unify all the kinds of policy rules. Until that day, you have to tune
all three policy sets (routing, ipsec and iptables) separately and take
care of the cases, when one set has to cheat another. :-)

Alexey

Re: ProxyARP and IPSec

[H. Peter Anvin]

Alexey Kuznetsov wrote:
> Hello!
> 
>> I'm thinking that David definitely has a point about having a usability 
>> problem, though.  All other kind of tunnels have endpoint devices 
>> associated with them, and that would make all these kinds of problems go 
>> away, 
> 
> Yes, when you deal with sane practical setups, this approach is the only
> reasonable one.
> 
> Unfortunately, IPsec is not something totally sane and practical :-),
> "security gateway" case is small part of it and "routing" viewpoint
> clashes fatally with another requirements. Pure result is that we use approach
> where it is possible to do everything with some efforts, rather than approach
> which is simple and intuitive, but does not allow to do many things.
> 

Fair enough.  However, that does beg a question: is there any sane way 
to create the pseudo-device model on top of the current model, as a 
convenience layer?  That way you could get the best of both.

	-hpa

Re: ProxyARP and IPSec

[Stephen J. Bevan]

H. Peter Anvin writes:
 > Fair enough.  However, that does beg a question: is there any sane way 
 > to create the pseudo-device model on top of the current model, as a 
 > convenience layer?  That way you could get the best of both.

I assume you were using tunnel-mode IPsec and depending on exactly
what you want to do you may be able to replace it with transport mode
IPsec (or stay with tunnel if the extra 20 bytes of IP is not a
problem) to handle host<->host IPsec and use gre or ipip for overlay
network.  That way you get a virtual device (gre or ipip) you can
route to, run OSPF on, ... etc.

-- 
VGER BF report: U 0.498333

Re: ProxyARP and IPSec

[H. Peter Anvin]

Stephen J. Bevan wrote:
> H. Peter Anvin writes:
>  > Fair enough.  However, that does beg a question: is there any sane way 
>  > to create the pseudo-device model on top of the current model, as a 
>  > convenience layer?  That way you could get the best of both.
> 
> I assume you were using tunnel-mode IPsec and depending on exactly
> what you want to do you may be able to replace it with transport mode
> IPsec (or stay with tunnel if the extra 20 bytes of IP is not a
> problem) to handle host<->host IPsec and use gre or ipip for overlay
> network.  That way you get a virtual device (gre or ipip) you can
> route to, run OSPF on, ... etc.

<sarcasm>

What I great idea.  Now I just have to get every host I want to 
interoperate with to support a nonstandard configuration.  The scary 
part is that if I motivate it with "Linux is too stupid to handle 
standard tunnel-mode IPsec" I might actually get away with it.

</sarcasm>

Really... if saying our configuration is so screwed up that we have to 
run a different over-wire protocol isn't an admission of failure I don't 
know what is.  I suspect this contributes to the growth in OpenVPN as well.

	-hpa

-- 
VGER BF report: U 0.500003

Re: ProxyARP and IPSec

[Stephen J. Bevan]

 > <sarcasm>
 > 
 > What I great idea.  Now I just have to get every host I want to 
 > interoperate with to support a nonstandard configuration.  The scary 
 > part is that if I motivate it with "Linux is too stupid to handle 
 > standard tunnel-mode IPsec" I might actually get away with it.
 > 
 > </sarcasm>

Linux handles tunnel-mode IPsec in the same way that most IPsec
vendors did and many still do.  For example, Cisco IOS has pages and
pages of documentation about how to combine IPsec with GRE in order to
support securely running OSPF between sites, precisely because its
IPsec didn't offer a virtual interface.  However, Cisco (along with
Netscreen/Juniper and Fortinet) now additionally support IPsec that
uses a virtual interface and so you have a choice of using an
interface or not as you see fit.  I would be useful if Linux offered
the option but code talks and I'm not offering a patch so I'm not in a
position to complain about what Linux currently supports.


 > Really... if saying our configuration is so screwed up that we have to 
 > run a different over-wire protocol isn't an admission of failure I don't 

If you use ipip the over-wire protocol is identical, see RFC 3884
section 3.1 or you can test it right now using manual keying (remote
side uses tunnel mode, your side uses transport + ipip).  To use IKE
pluto would need to be hacked a bit, though most of the changes could
be handled via a updown script.


 > know what is.  I suspect this contributes to the growth in OpenVPN as well.

Haven't you heard, SSL based VPNs are the future :-)

-- 
VGER BF report: U 0.947229

Re: ProxyARP and IPSec

[Alexey Kuznetsov]

Hello!

> <sarcasm>
> 
> What I great idea.  Now I just have to get every host I want to 
> interoperate with to support a nonstandard configuration.  The scary 
> part is that if I motivate it with "Linux is too stupid to handle 
> standard tunnel-mode IPsec" I might actually get away with it.

sarcasm mode is not accepted. Linux does exactly "standard tunnel-mode IPsec".
It does not give you device to make you totally happy.

Probably, you are not aware that "standard IPsec tunnel device",
if it is created:

1. Probably, will not accept fragmented frames, because IPsec cannot
   handle them
2. Probably, will have undefined MTU (65536), because of 1
3. Probably, will screw up TCP because of 2
   etc.

Actually, this is the reason why it is not implemented.
It is dirty business. :-) And the person, who implements this,
has to be really... unscrupulous. :-)

Alexey

Re: ProxyARP and IPSec

[H. Peter Anvin]

Alexey Kuznetsov wrote:
> 
> sarcasm mode is not accepted. Linux does exactly "standard tunnel-mode IPsec".
> It does not give you device to make you totally happy.
> 

The sarcasm was a commentary to the "just switch protocols then" comment.

> Probably, you are not aware that "standard IPsec tunnel device",
> if it is created:
> 
> 1. Probably, will not accept fragmented frames, because IPsec cannot
>    handle them

Now that's interesting.  If this is true, and I have every reason to 
believe you know what you're talking about, then that would seem to be a 
problem which is hard to overcome for any security gateway, regardless 
of how it's implemented in the kernel -- the data originator might be on 
a totally different host!

Furthermore, it seems to directly contradict the following statement 
from RFC 4303 section 3.3.4:

"In tunnel mode, ESP is applied to an IP packet, which may be a fragment 
of an IP datagram."

That reading seems to imply that it should handle fragments just fine. 
Similarly, RFC 4301 section 13 ("differences from RFC 2401") states:

"For tunnel mode SAs, an SG, BITS, or BITW implementation is now allowed 
to fragment packets before applying IPsec.  This applies only to IPv4. 
For IPv6 packets, only the originator is allowed to fragment them."

(consistently with the overall IPv6 architecture.)

Transport mode is different, of course.

> 2. Probably, will have undefined MTU (65536), because of 1
> 3. Probably, will screw up TCP because of 2
>    etc.
> 
> Actually, this is the reason why it is not implemented.
> It is dirty business. :-) And the person, who implements this,
> has to be really... unscrupulous. :-)

I'm clearly failing to understand where, exactly, the problems lie.  I 
would appreciate any pointers and/or clue transfusion...

	-hpa

Re: ProxyARP and IPSec

[H. Peter Anvin]

Stephen J. Bevan wrote:

> 
>  > Really... if saying our configuration is so screwed up that we have to 
>  > run a different over-wire protocol isn't an admission of failure I don't 
> 
> If you use ipip the over-wire protocol is identical, see RFC 3884
> section 3.1 or you can test it right now using manual keying (remote
> side uses tunnel mode, your side uses transport + ipip).  To use IKE
> pluto would need to be hacked a bit, though most of the changes could
> be handled via a updown script.
> 

Interesting.  It might be interesting to implement userspace (e.g. 
OpenSwan) in such a way that all tunnel-mode IPsec is implemented this 
way automatically, using an ipip interface in the kernel.

	-hpa

Re: ProxyARP and IPSec

[Alexey Kuznetsov]

Hello!

> >1. Probably, will not accept fragmented frames, because IPsec cannot
> >   handle them
...
> I'm clearly failing to understand where, exactly, the problems lie.  I 
> would appreciate any pointers and/or clue transfusion...

I said "probably".

Look into old rfc2401, search for word "fragment".
Then search for the same word in new rfc4301. All those 100K of new text
deal with various design bugs in IPsec, mostly with pathologies encountered
in the case of security gateways. (Some section there are real fun: f.e.
look at section 7.2)

With this amount of thin places, there are no chances it will interoperate,
unless you use the most conservative approach.

My opinion? Scared. :-)

Alexey

Re: ProxyARP and IPSec

[Stephen J. Bevan]

Alexey Kuznetsov writes:
 > Probably, you are not aware that "standard IPsec tunnel device",
 > if it is created:
 > 
 > 1. Probably, will not accept fragmented frames, because IPsec cannot
 >    handle them

IPsec can handle them, though not particularly smoothly if the IPsec
tunnel is only supposed to carry a particular port&protocol
combination.


 > 2. Probably, will have undefined MTU (65536), because of 1

An MTU that is more likely to make most things work (at least over
Ethernet) is ETH_DATA_LEN - MAX_SA_LEN where MAX_SA_LEN is however
much is required for IPsec (something like IP + UDP if NAT-T + ESP
header + IV + padding + ESP trailer).  The simplest thing is to just
statically configure it.  However, some implementations dynamically
calculate the IPsec device MTU based on the maximum size required by
any of the IPsec SAs that will go over the interface, using either a
pessimistic (255) or optimistic (2) padding estimate.  This can cause
problems for OPSF adjacency if each side arrives at a different MTU
but that can be handled by either manually configuring the device MTU
or explicitly configuring the MTU that OSPF will advertise (I think
Quagga supports that).


 > Actually, this is the reason why it is not implemented.
 > It is dirty business. :-) And the person, who implements this,
 > has to be really... unscrupulous. :-)

Exactly the same issue occurs if one implements IPsec (or any other
encryption method) in user-level using a tun/tap device.  Consequently
while I agree that fragmentation causes an additional level of
problems if one wants to have port/protocol based selectors in IPsec,
I believe that most (but not all) VPN users are quite satisfied with
policies containing all traffic, all ports and so will not encounter
any IPsec specific problems.

Re: ProxyARP and IPSec

[H. Peter Anvin]

Thomas Graf wrote:
> What about adding blackhole device to be used for such routes.
> I believe it would be good architecture to always use devices
> to state directions packets are being received from and sent to.

The dummy device can be used for that.

	-hpa

Re: ProxyARP and IPSec

[Thomas Graf]

* H. Peter Anvin <hpa@zytor.com> 2006-09-07 15:28
> Thomas Graf wrote:
> >What about adding blackhole device to be used for such routes.
> >I believe it would be good architecture to always use devices
> >to state directions packets are being received from and sent to.
> 
> The dummy device can be used for that.

I was thinking a bit beyond that, a device similiar to ifb but
without a hard header to allow classification. Packets get
enqueued before skb->dst gets overwritten with dst->child and
dequeue handles the second half of xfrm4_output_finish2().

Re: ProxyARP and IPSec

[David Miller]

From: Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>
Date: Tue, 5 Sep 2006 13:05:30 +0400

> Look into old rfc2401, search for word "fragment".
> Then search for the same word in new rfc4301. All those 100K of new text
> deal with various design bugs in IPsec, mostly with pathologies encountered
> in the case of security gateways. (Some section there are real fun: f.e.
> look at section 7.2)

I even was not aware of this problem. :-)

Essentially, if you use ports as part of your selector,
then it is impossible to handle anything other than the
first fragment of a fragmented frame because the subsequent
fragments will not have the ports which you need in order
to match.

The suggestions in 7.2 involving a seperate SA for the non-first
fragments seem totally unrealistic, if you ask me.  They even say
the idea cannot work with ipv6, what is the point? :-)

Re: ProxyARP and IPSec

[Stephen J. Bevan]

David Miller writes:
 > Essentially, if you use ports as part of your selector,
 > then it is impossible to handle anything other than the
 > first fragment of a fragmented frame because the subsequent
 > fragments will not have the ports which you need in order
 > to match.

If you have port/protocol based selectors and you are firewalling then
re-assembly is already being done so IPsec will see the re-assembled
packet at little cost.  Alternately in a pure IPsec configuration it
possible to arrange things so that re-assembly is only done if
port/protocol based selectors are used.