Re: NAT issue on a machine with both routing and bridging.

[Francois Goudal <fg@satcom1.com>]

Grant Taylor a écrit :
> On 06/23/08 09:22, Francois Goudal wrote:
>> So I decided to use virtual machines, like Xen (I tried UML as well, 
>> so my problem is not related to Xen specifically).
> 
> This is starting to sound like a project that I would work on.
> 

Thanks !

>> ................                                     ................
>> .    HOST A    .                                     .    HOST D    .
>> . 10.168.254.1 .                                     . 172.16.33.10 .
>> ................                                     ................
>>   |                                                               |
>>   |                                                               |
>>   |                                                               |
>>   | eth1                                                     eth0 |
>> .....................................................................
>> . |0.0.0.0                                                0.0.0.0 | .
>> . |__________________________________             ________________| .
>> .                                    |           |                  .
>> ............................         |_ br0      |                  .
>> .                     eth0 . vif1.0  |  0.0.0.0  |                  .
>> . XEN VM          _________._________|           |                  .
>> . HOST B         | 0.0.0.0 . 0.0.0.0             |                  .
>> .                |         .                     |_ br2             .
>> .           br0 _|         .                     |  172.16.33.200   .
>> . 10.168.254.51  |    eth1 . vif1.1              |        ^         .
>> .                |_________._________                     |         .
>> .                  0.0.0.0 . 0.0.0.0 |                    | Routing .
>> ............................         |_ br1               | + DNAT  .
>> .                                    |  10.168.254.250 <--'         .
>> .                                    |                              .
>> .        HOST C                                                     .
>> .....................................................................
> 
> (Nice ASCII art)
> 

:-)

>> Host C is a Xen Host machine that contains one Xen VM for the PEP 
>> stuff and which is responsible for the masquerading of packets.
> 
> So Host C is Dom 0 and Host B is a Dom U, correct.
> 

Yes, Host C is the Dom0 and Host B is a DomU here.

> <snip>
> 
>> But now, I want to get rid of the need of a special route on host D, 
>> so I want to setup DNAT/Masquerade on the Host C.
> 
> *nod*
> 
> <snip>
> 
>> So I suspect that on Host C, the packets that comes in the eth1 NIC 
>> are not just forwarded to the VM by the bridge, but detected somehow 
>> by the network stack and forwarded to eth0 (by some layer2 code ?) 
>> without being masqueraded, then.
> 
> Can we see the output of brctl on Host C (domain 0)?
> 

bridge name	bridge id		STP enabled	interfaces
br1		8000.c6eabf59b7a0	no		vif1.1
br0		8000.00304883f91f	no		eth1
							vif1.0
br2		8000.00304883f91e	no		eth0

This looks like the ASCII-art I did, I double checked all this, I don't 
think the problem comes from the bridge configuration, but you will 
probably tell me if you can see sth wrong here :-)

>> I have been working on trying to solve this during 2 days now but 
>> still I can't find a solution.
> 
> Is there a reason that you are not masquerading packets that leave br2 
> in Host C?
> 

I don't understand your question. I want them to be masqueraded, but the 
fact is that I can't get them masqueraded when they come from a machine 
connected to eth1 on the Dom0. But they are masqueraded when they come 
from the DomU. But I don't see any reason for that difference. On the 
Dom0, the eth1 interface is linked with a bridge to one interface of the 
DomU but no IP addresses are set (on eth1 itself, on the bridge 
interface it belongs to, and on the Xen backend interface which is in 
the bridge) so the traffic has to go through the DomU, so now, why is it 
working with the DomU itself but not with the hosts connected on eth1, I 
have no idea :-/

I had a look at the big Linux Network Packet Flow picture that describes 
how the packets are going through both ebtables and iptables rules, but 
I don't see anything that could be a problem.

>> Can anyone have a quick look and hopefully provide me an explaination 
>> and maybe some help to find a solution ?
> 
> I need to see how things are bridged in Host C to be sure.  I suspect 
> that either something is amiss in your bridging or where / how you were 
> doing your masquerading.
> 

for the masquerading, as I said, it's very simple :

iptables -t nat -A POSTROUTING -o xenbr0 -j MASQUERADE

And I tried with eth0 instead of xenbr0, and I tried with SNAT, 
specifying manually the IP address 172.16.33.200, but nothing worked.

Regarding the routing, The HostC has nothing special : One default route 
for each interface that has an IP address, so :
10.168.254.0 goes through br1
172.16.33.0 goes through xenbr0

On HostA, I have this :
10.168.254.0 goes through eth0
0.0.0.0 goes through gw 10.168.254.250

On HostB, I have :
10.168.254.0 goes through br0
0.0.0.0 goes through gw 10.168.254.250

And on HostD, I just have :
172.16.33.0 goes through eth0

So I need masquerading so that HostD can reply to HostA without having 
to setup a route on HostD to tell him how to do it.

> I will say that what you are wanting to do is sound and does work.  I 
> have deployed multiple systems running complex networks in vms, be it 
> UML (multiple incarnations) and VMWare (any incarnation needing a 
> Windows vm).  Presently I have multiple systems deployed that have one 
> host with up to 8 guest vms.  These types of systems sound overly 
> complex. but the networking is usually the least complex part of them.
> 

Yes, I'm aware this is quite complex, and I understand that it might be 
difficult to help, especially because I'm using a PEP software which 
might be quite difficult to setup if someone wants to reproduce the problem.
But still, as I said, the PEP stuff can be replaced by bridging the two 
interfaces in the DomU together, it does the same, and I am able to 
reproduce the problem with such a setup as well.

 > Don't give up.

I won't ;-)

Thank's for your time.
Best regards.
-- 
Francois Goudal
Satcom1