Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger <lists@wiesinger.com>]

On Mon, 21 Feb 2011, Arnd Bergmann wrote:

> On Monday 21 February 2011, Jan Kiszka wrote:
>>> Now I think I tried all useful possible combinations:
>>> 1.) macvtap0 and macvlan0 in bridged and non bridge mode
>>> 2.) macvlan0 based on eth0 or based on macvtap0
>>> 3.) Using and ip address on macvlan0 or not (tried both for 7b mac and
>>> 7c mac)
>>> 4.) Using 7b and 7c mac address in KVM (MAC in KVM is always the command
>>> line configured) and used tap interface from macvtap0 (as no second tap
>>> devices shows up)
>>>
>>> Summary:
>>> 1.) Using macvtap0 only with 7b mac on interface and also at qemu works
>>> well in bridged mode as well as non bridged mode but with limitation no
>>> guest/host communication possible, used interface in KVM is tap
>>> interface created by macvtap0. Quite logically that it doesn't work with
>>> guest/host because of missing hairpin mode on the switch. But according
>>> to http://virt.kernelnewbies.org/MacVTap bridge mode should work even
>>> without hairpinning available on the switch.
>>> 2.) macvlan0 doesn't create any tap interface therefore it can't be used
>>> as a device on KVM.
>>> 3.) Using 7c mac address in KVM doesn't work at all regardsless of
>>> setting ip addresses on macvlan0 or any other setup.
>>>
>>> @Arnd: Can you explain a setup in detail which should work also with
>>> host/guest communication. Thnx.
>>>
>>> Any further ideas?
>>> Which kernel is needed to work?
>>> (current: 2.6.34.7-56.fc13.x86_64)
>>
>> Actually, I tried this successfully over a 2.6.38-rcX, but I have no
>> idea what changes related to macvlan/tap went in since .34 and if this
>> is required for this.
>
> We had a few bugs in .34, but it should work in general.
>
> One thing to make sure is that the host has connectivity through the
> macvlan interface and the lower interface (eth0) has no IP address assigned
> but is 'up'.
>
> It could also be a bug in the NIC, you could try to set the NIC into
> promiscious mode using ethtool to work around that.

Hello,

After some further tests and looking at the iproute ip and kernel code I 
finally gave up because I thing such a setup it is not possible without 
breaking up/reconfiguring eth0. When I have to reconfigure eth0 I think a 
better approach is to configure a bridge which I finally did and works 
well.

I tried to explain/document the macvtap/macvlan concepts and limitations 
below. Please comment on it whether this is true or false.

macvtap/macvlan driver concepts and limitations:
1.) macvlan driver adds a MAC address to a lower interface device where 
the actual macvlanx device is based on
2.) macvtap driver is based on macvlan driver and macvtap driver adds 
additional functionality of interface <=> external program communication 
with stdin/stdout channel.
3.) Limitations: macvtap/macvlan based devices can only communicate with 
childs based on the same lower device (e.g. eth0 in this sample) but not 
to the lower device itself, only to the outside world of the interface
(I guess limitations are derived from function macvlan_queue_xmit: 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=drivers/net/macvlan.c;h=6ed577b065df136a18af5ee483810773ca4f43f8;hb=HEAD#l217)

Example:
# Lower device eth0 must not have an IP address assigned, but when also no communication is possible with macv* devices
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvtap0 type macvtap mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address 1a:46:0b:ca:bc:7b up
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvtap1 type macvtap mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap1 address 1a:46:0b:ca:bc:7c up
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvlan0 type macvtap mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address 1a:46:0b:ca:bc:7d up

Example usage:
eth0 should not be assigned any IP. macvlan0 should be used as the main 
host interface and the main IP address is configured.
macvtap0 and macvtap1 can be used in e.g. VMs.
Bridge communication is possible between the 3 interfaces and the outer 
world: macvtap0, macvtap1, macvlan0
E.g.:
1.) macvtap0 <=> macvtap1, macvlan0, eth0 external
2.) macvtap1 <=> macvtap0, macvlan0, eth0 external
3.) macvlan0 <=> macvtap0, macvtap1, eth0 external

But no communication is possible between lower device eth0 and the child 
interfaces macvtap0, macvtap1, macvlan0:
1.) eth0 <=> macvtap0
2.) eth0 <=> macvtap1
3.) eth0 <=> macvlan0

finally this makes the macvlan/macvtap approach useless because main eth0 
interface must still be broken in the chain and reconfigured which was 
against the requirements that eth0 should not be touched and reconfigured!

Bridge Setup for Fedora 13 is below (reconfigure with service network 
restart, be carefull on remote operation or have a serial console :-)!).

Nevertheless, thank you for your help and I'm ready for any other 
approach :-).

Ciao,
Gerhard

--
http://www.wiesinger.com/

############################################################
cat /etc/sysconfig/network-scripts/ifcfg-br0
# Bridge config
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
BROADCAST=192.168.0.255
IPADDR=192.168.0.9
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
DNS1=192.168.0.2
SEARCH="mydomain"
NM_CONTROLLED=
############################################################
cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0

# Removed for bridge config
#BOOTPROTO=static
#BROADCAST=192.168.0.255

#########################
# Onboard one
HWADDR=00:1f:c6:8c:59:1f
#########################

# Removed for bridge config
#IPADDR=192.168.0.9
#NETMASK=255.255.255.0
#NETWORK=192.168.0.0
ONBOOT=yes
#DNS1=192.168.0.2
#SEARCH="mydomain"
NM_CONTROLLED=

# New to add for bridge
# Bridge membership
BRIDGE=br0
BOOTPROTO=none
############################################################
cat /etc/sysconfig/network-scripts/_ifcfg-eth0.nonbridge
# Old config without bridge
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.0.255

#########################
# Onboard one
HWADDR=00:1f:c6:8c:59:1f
#########################

IPADDR=192.168.0.9
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
DNS1=192.168.0.2
SEARCH="mydomain"
NM_CONTROLLED=
############################################################