Troubles doing transparent proxy for virtual machines

Troubles doing transparent proxy for virtual machines

[Dan Higgins]

Hi iptables gurus. First here is the basic topology:

   Internet
      |
   Gateway
      |
Workstation---eth0---virbr0
                        |
                        +-----+-----+
                        |     |     |
                       vm1   vm2   vm3

I need to test a traffic analyzer running on my workstation, listening
on some port (say 8990) on eth0.

The rule [I think] I want is "any packets leaving virbr0 going anywhere
to port 80 must instead go to port 8990 on eth0". My software running on
port 8990 does its own check of the NAT packet mangling to push the
packets through after it inspects them.

I've been banging my head on this, with different variants of:
iptables -t nat -A PREROUTING -i virbr0 -p tcp --dport 80 -j DNAT \
     --to 10.0.0.10:8990
And I've tried the more generic method of using the mangle table with
--set-mark and ip rule add fwmark, but I'm not getting it.

I guess what's confusing me is that everything runs on the same box.

Thanks for any guidance,
Dan

Troubles doing transparent proxy for virtual machines

[Dan Higgins]

Hi iptables gurus. First here is the basic topology:

   Internet
      |
   Gateway
      |
Workstation---eth0---virbr0
                        |
                        +-----+-----+
                        |     |     |
                       vm1   vm2   vm3

I need to test a traffic analyzer running on my workstation, listening
on some port (say 8990) on eth0.

This is not my code, I'm just the poor guy who has to test it. 
Philosophically,
I don't even like what it does, but pragmatically, I got bills to pay.

The rule [I think] I want is:
"Any packets leaving virbr0 going anywhere to port 80 must instead go to 
port
8990 on eth0 (10.0.0.10)."
The software running on port 8990 does its own check of the NAT packet 
mangling
to push the packets through after it inspects them.

I've been banging my head on this, with different variants of:
iptables -t nat -A PREROUTING -i virbr0 -p tcp --dport 80 -j DNAT \
     --to 10.0.0.10:8990
And I've tried the more generic method of using the mangle table with
--set-mark and ip rule add fwmark, but I'm not getting it.

I guess what's confusing me is that everything runs on the same box.

Thanks for any guidance,
Dan

Re: Troubles doing transparent proxy for virtual machines

[Marek Kierdelewicz]

Hi,

>I guess what's confusing me is that everything runs on the same box.

Yup. Packet traverses nat table when it passes bridge and it cannot
traverse this table second time entering virbr0 interface.

Try this:
echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

For permament solution add appropriate entry in /etc/sysctl.conf.

Best regards,
Marek

Re: Troubles doing transparent proxy for virtual machines

[Dan Higgins]

On 06/03/2010 04:20 PM, Marek Kierdelewicz wrote:
> Hi,
>
>    
>> I guess what's confusing me is that everything runs on the same box.
>>      
> Yup. Packet traverses nat table when it passes bridge and it cannot
> traverse this table second time entering virbr0 interface.
>
> Try this:
> echo 0>  /proc/sys/net/bridge/bridge-nf-call-iptables
>    

Thanks for the response. I'm beginning to think that my strategy here is 
all wrong to begin with. Here's why:

1. The subnet for my VMs is 192.168.122.0/24. This is created by 
libvirtd when it starts up.

2. I start up Firefox in a VM, then go to google.com.

3. As I watch with tcpdump, I never even see 192.168.122.x in the output.

Maybe libvirtd is doing its own network voodoo before handing off to the 
kernel network stack.  [Shrug]  Should I expect tcpdump to show packets 
coming through a bridge (virbr0 or just br0)?

This is still pretty deep water to me.

Re: Troubles doing transparent proxy for virtual machines

[Marek Kierdelewicz]

Hi,

>Thanks for the response. I'm beginning to think that my strategy here
>is all wrong to begin with. Here's why:
>1. The subnet for my VMs is 192.168.122.0/24. This is created by 
>libvirtd when it starts up.
>2. I start up Firefox in a VM, then go to google.com.
>3. As I watch with tcpdump, I never even see 192.168.122.x in the
>output.
>Maybe libvirtd is doing its own network voodoo before handing off to
>the kernel network stack. 

Sorry, I've never used libvirtd, but if your virtualization backend is
kvm and it's configured for "userspace networking", iptables won't do
what you want.

> [Shrug]  Should I expect tcpdump to show
>packets coming through a bridge (virbr0 or just br0)?

In "bridged networking" you should see packets on VMs interface
(tapX) and bridge interface (brX).

Try setting it up. This page may be of some help:
http://wiki.libvirt.org/page/Networking

Best regards,
Marek