Multiple TAP Interfaces, with multiple bridges

Multiple TAP Interfaces, with multiple bridges

[J L]

Hi,

I am having an odd networking issue. It is one of those "it used to
work, and now it doesn't" kind of things. I can't work out what I am
doing differently.

I have a virtual machine, started with (among other things):
  -net nic,macaddr=fa:9e:0b:53:d2:7d,model=rtl8139 -net
tap,script=/images/1/ifup-eth0,downscript=/images/1/ifdown-eth0
  -net nic,macaddr=fa:02:4e:86:ed:ce,model=e1000 -net
tap,script=/images/1/ifup-eth1,downscript=/images/1/ifdown-eth1

The ifup-ethX script inserts the tap interface into the correct bridge
(of which there are multiple.)

The Virtual Machine is Centos 5.3, with a 2.6.27.21 kernel. The Host
is Ubuntu 9.10 with a 2.6.31 kernel.


My network then looks like:

The Virtual Machine has an eth0 interface, which is matched with tap0
on the host.
The Virtual Machine has an eth1 interface, which is matched with tap1
on the host.

The host has a bridge br0, which contains tap0 and eth0.
The host has a bridge br1, which contains tap1.

There is a server on the same network as the Host's eth0.

The Virtual Machines eth0 interface is down.
The Virtual Machines eth1 interface has an IP address of 192.168.1.10/24.
The Virtual Machine has a default gateway of 192.168.1.1.

The host's br0 has an IP address of 192.168.0.1/24.
The host's br1 has an IP address of 192.168.1.1/24.

The server has an IP address of 192.168.0.20/24, and a default gateway
of 192.168.0.1.

Firewalling is disabled everywhere. I have allowed time for the
bridges and STP to settle.



If I go to the Virtual Machine, and ping 192.168.0.20 (the server), I
would expect tcpdumps to show:
  * VM: eth1, dest MAC of Host's tap1/br0
  * Host: tap1, dest MAC of Host's tap1/br0
  * Host: br1, dest MAC of Host's tap1/br0
  * Host now routes from br1 to br0
  * Host: tap0, no packet
  * Host: br0, dest MAC of Server
  * Host: eth0, dest MAC of Server
  * Server: eth0, dest MAC of Server

What I actually get:
  * VM: eth1, dest MAC of Host's tap1/br0
  * Host: tap1, dest MAC of Host's tap1/br0
  * Host: br1, dest MAC of Host's tap1/br0
  * Host should, but does not route from br0 to br1
  * Host: tap0, dest MAC of ***Host's tap1/br0***
  * Host: br0, dest MAC of ***Host's tap1/br0**
  * Host: eth0, no packet
  * Server: eth0, no packet

As you can see, the packet has egressed *both* tap interfaces! Is this
expected behaviour? What can I do about this?




If I remove tap0 from the bridge, I then get:
  * VM: eth1, dest MAC of Host's tap1/br0
  * Host: tap1, dest MAC of Host's tap1/br0
  * Host: br1, dest MAC of Host's tap1/br0
  * Host should, but does not, route from br0 to br1
  * Host: tap0, no packet
  * Host: br0, no packet
  * Host: eth0, no packet
  * Server: eth0, no packet

This is the other half of my problem: in this case, with effectively
only one tap, the host is not routing between br1 and br0. The packet
just gets silently dropped. Does anyone know what I am doing wrong?

I hope I have managed to explain this well enough!

Thanks,
--
Jarrod Lowe

Re: Multiple TAP Interfaces, with multiple bridges

[Tom Lendacky]

On Wednesday 03 February 2010 10:56:43 am J L wrote:
> Hi,
> 
> I am having an odd networking issue. It is one of those "it used to
> work, and now it doesn't" kind of things. I can't work out what I am
> doing differently.
> 
> I have a virtual machine, started with (among other things):
>   -net nic,macaddr=fa:9e:0b:53:d2:7d,model=rtl8139 -net
> tap,script=/images/1/ifup-eth0,downscript=/images/1/ifdown-eth0
>   -net nic,macaddr=fa:02:4e:86:ed:ce,model=e1000 -net
> tap,script=/images/1/ifup-eth1,downscript=/images/1/ifdown-eth1
> 

I believe this has to do with the qemu vlan support. If you don't specify the 
vlan= option you end up with nics on the same vlan. You need to assign the two 
nics to separate vlans using vlan= on each net parameter, eg:


   -net nic,vlan=0,macaddr=fa:9e:0b:53:d2:7d,model=rtl8139 -net
 tap,vlan=0,script=/images/1/ifup-eth0,downscript=/images/1/ifdown-eth0
   -net nic,vlan=1,macaddr=fa:02:4e:86:ed:ce,model=e1000 -net
 tap,vlan=1,script=/images/1/ifup-eth1,downscript=/images/1/ifdown-eth1

Try that and see if you get the results you expect.

Tom

> The ifup-ethX script inserts the tap interface into the correct bridge
> (of which there are multiple.)
> 
> The Virtual Machine is Centos 5.3, with a 2.6.27.21 kernel. The Host
> is Ubuntu 9.10 with a 2.6.31 kernel.
> 
> 
> My network then looks like:
> 
> The Virtual Machine has an eth0 interface, which is matched with tap0
> on the host.
> The Virtual Machine has an eth1 interface, which is matched with tap1
> on the host.
> 
> The host has a bridge br0, which contains tap0 and eth0.
> The host has a bridge br1, which contains tap1.
> 
> There is a server on the same network as the Host's eth0.
> 
> The Virtual Machines eth0 interface is down.
> The Virtual Machines eth1 interface has an IP address of 192.168.1.10/24.
> The Virtual Machine has a default gateway of 192.168.1.1.
> 
> The host's br0 has an IP address of 192.168.0.1/24.
> The host's br1 has an IP address of 192.168.1.1/24.
> 
> The server has an IP address of 192.168.0.20/24, and a default gateway
> of 192.168.0.1.
> 
> Firewalling is disabled everywhere. I have allowed time for the
> bridges and STP to settle.
> 
> 
> 
> If I go to the Virtual Machine, and ping 192.168.0.20 (the server), I
> would expect tcpdumps to show:
>   * VM: eth1, dest MAC of Host's tap1/br0
>   * Host: tap1, dest MAC of Host's tap1/br0
>   * Host: br1, dest MAC of Host's tap1/br0
>   * Host now routes from br1 to br0
>   * Host: tap0, no packet
>   * Host: br0, dest MAC of Server
>   * Host: eth0, dest MAC of Server
>   * Server: eth0, dest MAC of Server
> 
> What I actually get:
>   * VM: eth1, dest MAC of Host's tap1/br0
>   * Host: tap1, dest MAC of Host's tap1/br0
>   * Host: br1, dest MAC of Host's tap1/br0
>   * Host should, but does not route from br0 to br1
>   * Host: tap0, dest MAC of ***Host's tap1/br0***
>   * Host: br0, dest MAC of ***Host's tap1/br0**
>   * Host: eth0, no packet
>   * Server: eth0, no packet
> 
> As you can see, the packet has egressed *both* tap interfaces! Is this
> expected behaviour? What can I do about this?
> 
> 
> 
> 
> If I remove tap0 from the bridge, I then get:
>   * VM: eth1, dest MAC of Host's tap1/br0
>   * Host: tap1, dest MAC of Host's tap1/br0
>   * Host: br1, dest MAC of Host's tap1/br0
>   * Host should, but does not, route from br0 to br1
>   * Host: tap0, no packet
>   * Host: br0, no packet
>   * Host: eth0, no packet
>   * Server: eth0, no packet
> 
> This is the other half of my problem: in this case, with effectively
> only one tap, the host is not routing between br1 and br0. The packet
> just gets silently dropped. Does anyone know what I am doing wrong?
> 
> I hope I have managed to explain this well enough!
> 
> Thanks,
> --
> Jarrod Lowe
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Multiple TAP Interfaces, with multiple bridges

[arnd]

On Wednesday 03 February 2010 17:56:43 J L wrote:
> I am having an odd networking issue. It is one of those "it used to
> work, and now it doesn't" kind of things. I can't work out what I am
> doing differently.
> 
> I have a virtual machine, started with (among other things):
>   -net nic,macaddr=fa:9e:0b:53:d2:7d,model=rtl8139 -net
> tap,script=/images/1/ifup-eth0,downscript=/images/1/ifdown-eth0
>   -net nic,macaddr=fa:02:4e:86:ed:ce,model=e1000 -net
> tap,script=/images/1/ifup-eth1,downscript=/images/1/ifdown-eth1

This seems to be missing a vlan= option at least for the second pair:

> What I actually get:
>   * VM: eth1, dest MAC of Host's tap1/br0
>   * Host: tap1, dest MAC of Host's tap1/br0
>   * Host: br1, dest MAC of Host's tap1/br0
>   * Host should, but does not route from br0 to br1
>   * Host: tap0, dest MAC of ***Host's tap1/br0***
>   * Host: br0, dest MAC of ***Host's tap1/br0**
>   * Host: eth0, no packet
>   * Server: eth0, no packet
> 
> As you can see, the packet has egressed both tap interfaces! Is this
> expected behaviour? What can I do about this?

Qemu forwards this packet to everything inside of the same vlan, which
is 0 by default. Does it work with this?

   -net nic,vlan=1,macaddr=fa:9e:0b:53:d2:7d,model=rtl8139 -net tap,vlan=1,script=/images/1/ifup-eth0,downscript=/images/1/ifdown-eth0
   -net nic,vlan=2,macaddr=fa:02:4e:86:ed:ce,model=e1000 -net tap,vlan=2,script=/images/1/ifup-eth1,downscript=/images/1/ifdown-eth1

> If I remove tap0 from the bridge, I then get:
>   * VM: eth1, dest MAC of Host's tap1/br0
>   * Host: tap1, dest MAC of Host's tap1/br0
>   * Host: br1, dest MAC of Host's tap1/br0
>   * Host should, but does not, route from br0 to br1
>   * Host: tap0, no packet
>   * Host: br0, no packet
>   * Host: eth0, no packet
>   * Server: eth0, no packet
> 
> This is the other half of my problem: in this case, with effectively
> only one tap, the host is not routing between br1 and br0. The packet
> just gets silently dropped. Does anyone know what I am doing wrong?

Maybe /proc/sys/net/ipv4/ip_forward is disabled?

	Arnd

Re: Multiple TAP Interfaces, with multiple bridges

[J L]

On 3 February 2010 17:16,  <arnd@arndb.de> wrote:
> On Wednesday 03 February 2010 17:56:43 J L wrote:
>> I am having an odd networking issue. It is one of those "it used to
>> work, and now it doesn't" kind of things. I can't work out what I am
>> doing differently.
>>
>> I have a virtual machine, started with (among other things):
>>   -net nic,macaddr=fa:9e:0b:53:d2:7d,model=rtl8139 -net
>> tap,script=/images/1/ifup-eth0,downscript=/images/1/ifdown-eth0
>>   -net nic,macaddr=fa:02:4e:86:ed:ce,model=e1000 -net
>> tap,script=/images/1/ifup-eth1,downscript=/images/1/ifdown-eth1
>
> This seems to be missing a vlan= option at least for the second pair:
>
>> What I actually get:
>>   * VM: eth1, dest MAC of Host's tap1/br0
>>   * Host: tap1, dest MAC of Host's tap1/br0
>>   * Host: br1, dest MAC of Host's tap1/br0
>>   * Host should, but does not route from br0 to br1
>>   * Host: tap0, dest MAC of ***Host's tap1/br0***
>>   * Host: br0, dest MAC of ***Host's tap1/br0**
>>   * Host: eth0, no packet
>>   * Server: eth0, no packet
>>
>> As you can see, the packet has egressed both tap interfaces! Is this
>> expected behaviour? What can I do about this?
>
> Qemu forwards this packet to everything inside of the same vlan, which
> is 0 by default. Does it work with this?
>
>   -net nic,vlan=1,macaddr=fa:9e:0b:53:d2:7d,model=rtl8139 -net tap,vlan=1,script=/images/1/ifup-eth0,downscript=/images/1/ifdown-eth0
>   -net nic,vlan=2,macaddr=fa:02:4e:86:ed:ce,model=e1000 -net tap,vlan=2,script=/images/1/ifup-eth1,downscript=/images/1/ifdown-eth1

Thanks, both to you and Tom, who both emailed this piece of clue at
the same time :)

My misunderstanding was in thinking that vlan=XX would mean the
packets would land on the bridge with that VLAN tag, not what it seems
to actually be doing, of being used to tie one-or-more '-net nic'
sections with one-or-more '-net tap' sections. That is, I though the
vlan=XX was host-wide, not guest-wide.

Don't know how it worked before - probably I just never noticed the
extra packets.


>> If I remove tap0 from the bridge, I then get:
>>   * VM: eth1, dest MAC of Host's tap1/br0
>>   * Host: tap1, dest MAC of Host's tap1/br0
>>   * Host: br1, dest MAC of Host's tap1/br0
>>   * Host should, but does not, route from br0 to br1
>>   * Host: tap0, no packet
>>   * Host: br0, no packet
>>   * Host: eth0, no packet
>>   * Server: eth0, no packet
>>
>> This is the other half of my problem: in this case, with effectively
>> only one tap, the host is not routing between br1 and br0. The packet
>> just gets silently dropped. Does anyone know what I am doing wrong?
>
> Maybe /proc/sys/net/ipv4/ip_forward is disabled?
Sorry, forgot to mention that bit. It is '1'.

I added a '-j LOG' rule to the FORWARD table (as the only rule, policy
ACCEPT), and can see that the packets from the VM never make it to the
FORWARD table.


>
>        Arnd
>


Thanks,
-- 
Jarrod Lowe