NETMAP of destination *after* routing

NETMAP of destination *after* routing

[Charles Duffy]

Howdy, all. I'm looking at building NETMAP-like functionality into 
libvirt, such that groups of guest VMs (each group on its own bridge) 
can think they're sharing the same address space, but be separately 
addressable from outside (including the VM host itself). This has 
applications in automated QA -- being able to suspend a group of virtual 
machines in-flight, create an arbitrary number of copy-on-write images 
of these machines (each group of copies attached via a different bridge 
device) connected to different bridges, and being immediately able to 
separately address each copy via a distinct network address without 
reconfiguration.

Unfortunately, the current behavior of NETMAP -- translating the source 
address in POSTROUTING and the destination in PREROUTING -- doesn't 
appear to work for this purpose: I still need the original destination 
intact when routing to decide which bridge packets should go out.


How do 'yall suggest resolving this? I've played around with 
xtables-addons somewhat, and am pondering building a target to do 
translation in the mangle table on a packet-by-packet basis (as my 
present understanding -- correct or otherwise -- is that translating the 
destination post-routing with existing conntrack-based NAT functionality 
simply isn't feasible)... but at present I don't know what roadblocks 
are likely to be hit in the process.

Thoughts?

Thanks!

Re: NETMAP of destination *after* routing

[Jan Engelhardt]

On Wednesday 2008-08-06 18:57, Charles Duffy wrote:
>
> How do 'yall suggest resolving this? I've played around with
> xtables-addons somewhat, and am pondering building a target to do
> translation in the mangle table on a packet-by-packet basis (as my
> present understanding -- correct or otherwise -- is that

This is what RAWNAT (in the "RAWNAT" branch of xtables-addons) is
supposed to do, though I do not claim it being well-tested;
Manu ran into a few issues I still have not addressed :/

> translating the destination post-routing with existing
> conntrack-based NAT functionality simply isn't feasible)... but at
> present I don't know what roadblocks are likely to be hit in the
> process.

Re: NETMAP of destination *after* routing

[Sven-Haegar Koch]

On Wed, 6 Aug 2008, Charles Duffy wrote:

> Unfortunately, the current behavior of NETMAP -- translating the source
> address in POSTROUTING and the destination in PREROUTING -- doesn't appear to
> work for this purpose: I still need the original destination intact when
> routing to decide which bridge packets should go out.
> 
> 
> How do 'yall suggest resolving this?

Perhaps setting a mark based on the destination address at pre-routing,
use netmap, and then route based on the mark?
(Using one routing table a'la 'common-dest/mask -> bridgeX' per mark)

c'ya
sven

-- 

The Internet treats censorship as a routing problem, and routes around
it. (John Gilmore on http://www.cygnus.com/~gnu/)

Re: NETMAP of destination *after* routing

[Charles Duffy]

Sven-Haegar Koch wrote:
> On Wed, 6 Aug 2008, Charles Duffy wrote:
> 
>> Unfortunately, the current behavior of NETMAP -- translating the source
>> address in POSTROUTING and the destination in PREROUTING -- doesn't appear to
>> work for this purpose: I still need the original destination intact when
>> routing to decide which bridge packets should go out.
>>
>>
>> How do 'yall suggest resolving this?
> 
> Perhaps setting a mark based on the destination address at pre-routing,
> use netmap, and then route based on the mark?
> (Using one routing table a'la 'common-dest/mask -> bridgeX' per mark)

This works only for packets which came in from elsewhere in the network; 
packets emitted from the local host don't go through PREROUTING (and 
thus can't be marked, and also -- and worse -- don't get their 
destinations translated).

Perhaps I could also also add a rule setting an appropriate mark in 
OUTPUT. Not being able to communicate with the hosts from the local 
system is a showstopper, however, and I'd prefer to avoid munging the 
routing tables if possible to keep the patch to libvirt implementing 
this functionality minimal. (Robust infrastructure is already in place 
for modifying iptables rules for libvirt-managed networks; routing, not 
so much).

Re: NETMAP of destination *after* routing

[Grant Taylor]

On 08/07/08 09:55, Charles Duffy wrote:
> Perhaps I could also also add a rule setting an appropriate mark in 
> OUTPUT. Not being able to communicate with the hosts from the local 
> system is a showstopper, however, and I'd prefer to avoid munging the 
> routing tables if possible to keep the patch to libvirt implementing 
> this functionality minimal. (Robust infrastructure is already in place 
> for modifying iptables rules for libvirt-managed networks; routing, not 
> so much).

If you did get the rules in the PREROUTING and OUTPUT to work and they 
were the same, I'd suggest you put them in a common sub-chain and jump 
to it from both the PREROUTING and OUTPUT chains.  This will make 
maintenance much easier down the road.

Have you considered doing this on layer 2, doing so would allow your 
systems to have the same IP address.  You would in effect be altering 
the destination MAC address of the ethernet frames.  In fact, except for 
ARPing, you could easily have an unlimited number of hosts with the same 
address on the same physical network.  You will just need to translate 
IPs to the proper MAC at the time you want to communicate with them.

This will work from a traditional routing point of view.  However this 
does not take in to account that the target systems may see an IP 
conflict on the network.  To resolve this, you may want to put each 
system on its own micro network and bridge micro networks together as 
you see fit.



Grant. . . .

Re: NETMAP of destination *after* routing

[Jan Engelhardt]

On Thursday 2008-08-07 11:16, Grant Taylor wrote:
> On 08/07/08 09:55, Charles Duffy wrote:
>
>> Perhaps I could also also add a rule setting an appropriate mark
>> in OUTPUT. Not being able to communicate with the hosts from the
>> local system is a showstopper, however, and I'd prefer to avoid
>> munging the routing tables if possible to keep the patch to
>> libvirt implementing this functionality minimal. (Robust
>> infrastructure is already in place for modifying iptables rules
>> for libvirt-managed networks; routing, not so much).
>
> If you did get the rules in the PREROUTING and OUTPUT to work and
> they were the same, I'd suggest you put them in a common sub-chain
> and jump to it from both the PREROUTING and OUTPUT chains.  This
> will make maintenance much easier down the road.

That of course won't work because Xtables does chain inspection and
spots the illegal use of SNAT in a chain which is referenced from
PREROUTING.

Re: NETMAP of destination *after* routing

[Charles Duffy]

Jan Engelhardt wrote:
> On Wednesday 2008-08-06 18:57, Charles Duffy wrote:
>> How do 'yall suggest resolving this? I've played around with
>> xtables-addons somewhat, and am pondering building a target to do
>> translation in the mangle table on a packet-by-packet basis (as my
>> present understanding -- correct or otherwise -- is that
> 
> This is what RAWNAT (in the "RAWNAT" branch of xtables-addons) is
> supposed to do, though I do not claim it being well-tested;
> Manu ran into a few issues I still have not addressed :/

My prime concern with RAWNAT is its chances of being merged, given 
Patrick's concerns about the overhead of adding additional tables; 
otherwise, it looks to be just the tool for the job.

Is the workaround Manu provided (commenting out the skb_make_writable 
call) safe?

Re: NETMAP of destination *after* routing

[Grant Taylor]

On 08/07/08 11:03, Jan Engelhardt wrote:
> That of course won't work because Xtables does chain inspection and 
> spots the illegal use of SNAT in a chain which is referenced from 
> PREROUTING.

Sorry. I was thinking DNATing in both the PREROUTING and OUTPUT chains.



Grant. . . .