[Qemu-devel] Network bridging without adding bridge with brctl, possible?

[Qemu-devel] Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger]

Hello,

I like the way like VMWare Server 2.x does bridging: There is no need to 
change basic network interface settings and add a bridge between the 
existing network interface like e.g. discussed in:
http://www.linux-kvm.com/content/using-bridged-networking-virt-manager

VMWare Server 2.x does the following:
1.) Add vmnet network interfaces with vmnet.tar code (should be GPL v2).
2.) Starting a user mode process which I guess does the forwarding: /usr/bin/vmnet-bridge
/usr/bin/vmnet-bridge -d /var/run/vmnet-bridge-0.pid -n 0 -i eth0
3.) 2 other processes:
/usr/bin/vmnet-netifup -d /var/run/vmnet-netifup-vmnet1.pid /dev/vmnet1 vmnet1
/usr/bin/vmnet-dhcpd -cf /etc/vmware/vmnet1/dhcpd/dhcpd.conf -lf /etc/vmware/vmnet1/dhcpd/dhcpd.leases -pf /var/run/vmnet-dhcpd-vmnet1.pid vmnet1

/usr/bin/vmnet-bridge
Either the '-n' or '-p' option must be given exactly once.
Usage: vmnet-bridge (-n<number> | -p<uuid>) [-d<file name>] 
[(-e<interface> | -i<interface>)] [-s<number>] [-h]
        -i<interface> include interface
        -e<interface> exclude interface
        -g            prefer default gateway interface
        -n<number>    virtual network
        -p<uuid>      private virtual network
        -d<file name> process identifier file name
        -s<number>    optional pipe descriptor for service to send
                      a two byte startup status code
        -h            help

Might this also be easily possible with QEMU/KVM?
What do you thing about this? Worth to implement?
Any other suggestions to achieve this?

Thnx.

Ciao,
Gerhard

--
http://www.wiesinger.com/

[Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2409 bytes --]

On 2011-02-19 20:32, Gerhard Wiesinger wrote:
> Hello,
> 
> I like the way like VMWare Server 2.x does bridging: There is no need to
> change basic network interface settings and add a bridge between the
> existing network interface like e.g. discussed in:
> http://www.linux-kvm.com/content/using-bridged-networking-virt-manager
> 
> VMWare Server 2.x does the following:
> 1.) Add vmnet network interfaces with vmnet.tar code (should be GPL v2).
> 2.) Starting a user mode process which I guess does the forwarding:
> /usr/bin/vmnet-bridge
> /usr/bin/vmnet-bridge -d /var/run/vmnet-bridge-0.pid -n 0 -i eth0
> 3.) 2 other processes:
> /usr/bin/vmnet-netifup -d /var/run/vmnet-netifup-vmnet1.pid /dev/vmnet1
> vmnet1
> /usr/bin/vmnet-dhcpd -cf /etc/vmware/vmnet1/dhcpd/dhcpd.conf -lf
> /etc/vmware/vmnet1/dhcpd/dhcpd.leases -pf
> /var/run/vmnet-dhcpd-vmnet1.pid vmnet1
> 
> /usr/bin/vmnet-bridge
> Either the '-n' or '-p' option must be given exactly once.
> Usage: vmnet-bridge (-n<number> | -p<uuid>) [-d<file name>]
> [(-e<interface> | -i<interface>)] [-s<number>] [-h]
>        -i<interface> include interface
>        -e<interface> exclude interface
>        -g            prefer default gateway interface
>        -n<number>    virtual network
>        -p<uuid>      private virtual network
>        -d<file name> process identifier file name
>        -s<number>    optional pipe descriptor for service to send
>                      a two byte startup status code
>        -h            help
> 
> Might this also be easily possible with QEMU/KVM?
> What do you thing about this? Worth to implement?
> Any other suggestions to achieve this?

http://virt.kernelnewbies.org/MacVTap

Enabling host<->guest communication this way is still a bit unhandy
IMHO. You need a fairly recent iproute2 version, then set up macvtap
like this

    ip link add link eth1 name macvtap0 type macvtap mode bridge

And you additionally seem to need a separate macvlan device attached to
that bridge, configured to the IP of the host.

    ip link add link eth1 name macvlan0 type macvlan mode bridge
    ifconfig macvlan0 ...

Not sure if this is by design or due to internals of the networking
stack, but it looks unintuitive from user perspective. Maybe Arnd can
shed a light on this.

Of course, you could also simply offload all that setup to libvirt.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger]

On Sun, 20 Feb 2011, Jan Kiszka wrote:

> On 2011-02-19 20:32, Gerhard Wiesinger wrote:
>> Any other suggestions to achieve this?
>
> http://virt.kernelnewbies.org/MacVTap
>
> Enabling host<->guest communication this way is still a bit unhandy
> IMHO. You need a fairly recent iproute2 version, then set up macvtap
> like this
>
>    ip link add link eth1 name macvtap0 type macvtap mode bridge
>
> And you additionally seem to need a separate macvlan device attached to
> that bridge, configured to the IP of the host.
>
>    ip link add link eth1 name macvlan0 type macvlan mode bridge
>    ifconfig macvlan0 ...
>
> Not sure if this is by design or due to internals of the networking
> stack, but it looks unintuitive from user perspective. Maybe Arnd can
> shed a light on this.

Thnx Jan. That's exactly I'm looking for.

Is it possible to use then a legacy type network card for "old" guests?

e.g. ne2k_pci, i82551, i82557b, i82559er, rtl8139, e1000, pcnet
and also virtio, virtio-net-pci

> Of course, you could also simply offload all that setup to libvirt.

http://wiki.libvirt.org/page/Networking
But that still needs a bridge on ethernet level and break up the existing 
interface, right?

BTW: From: http://virt.kernelnewbies.org/MacVTap
As of QEMU 0.12:
qemu -net nic,model=virtio,addr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap11
Is there a newer Syntax with tap interfaces possible (QEMU GIT) without 
bash redirects?

Thnx.

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1903 bytes --]

On 2011-02-20 11:19, Gerhard Wiesinger wrote:
> On Sun, 20 Feb 2011, Jan Kiszka wrote:
> 
>> On 2011-02-19 20:32, Gerhard Wiesinger wrote:
>>> Any other suggestions to achieve this?
>>
>> http://virt.kernelnewbies.org/MacVTap
>>
>> Enabling host<->guest communication this way is still a bit unhandy
>> IMHO. You need a fairly recent iproute2 version, then set up macvtap
>> like this
>>
>>    ip link add link eth1 name macvtap0 type macvtap mode bridge
>>
>> And you additionally seem to need a separate macvlan device attached to
>> that bridge, configured to the IP of the host.
>>
>>    ip link add link eth1 name macvlan0 type macvlan mode bridge
>>    ifconfig macvlan0 ...
>>
>> Not sure if this is by design or due to internals of the networking
>> stack, but it looks unintuitive from user perspective. Maybe Arnd can
>> shed a light on this.
> 
> Thnx Jan. That's exactly I'm looking for.
> 
> Is it possible to use then a legacy type network card for "old" guests?
> 
> e.g. ne2k_pci, i82551, i82557b, i82559er, rtl8139, e1000, pcnet
> and also virtio, virtio-net-pci

For sure.

> 
>> Of course, you could also simply offload all that setup to libvirt.
> 
> http://wiki.libvirt.org/page/Networking
> But that still needs a bridge on ethernet level and break up the
> existing interface, right?

http://libvirt.org/formatdomain.html#elementsNICSDirect

But it looks like it doesn't handle the host-guest setup I described,
and bridging is supposed to be done the classic way.

> 
> BTW: From: http://virt.kernelnewbies.org/MacVTap
> As of QEMU 0.12:
> qemu -net nic,model=virtio,addr=1a:46:0b:ca:bc:7b -net tap,fd=3
> 3<>/dev/tap11
> Is there a newer Syntax with tap interfaces possible (QEMU GIT) without
> bash redirects?

Nope. Normally, the fd is passed in by the management tool. So the
existing interface was sufficient.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger]

On Sun, 20 Feb 2011, Jan Kiszka wrote:

> On 2011-02-19 20:32, Gerhard Wiesinger wrote:
>> Hello,
> http://virt.kernelnewbies.org/MacVTap
>
>    ip link add link eth1 name macvtap0 type macvtap mode bridge
>    ip link add link eth1 name macvlan0 type macvlan mode bridge
>    ifconfig macvlan0 ...


Hello Jan,

iproute2 2.6.37 is needed!
http://devresources.linuxfoundation.org/dev/iproute2/download/iproute2-2.6.37.tar.bz2
==================================================================================
1.) Only macvtap interface used
==================================================================================
For me it was just unlogically to use a VLAN type interface. Therefore I 
tried the following which worked nearly:
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvtap0 type macvtap mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address 1a:46:0b:ca:bc:7b up
ip link show macvtap0
ls -l /dev/tap*
# crw-rw---- 1 root root 249, 8 Feb 20 13:07 /dev/tap8
qemu-system-x86_64 ... some params ... -net nic,model=e1000,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap8

Booting Knoppix 6.2 with ping is ok, MAC address is also correct on second 
host.

Only problem is that from host to guest no networking is possible (only 
from guest to other host).

Any ideas? I guess some forwarding is missing? iptables forward missing?
==================================================================================
2.) Only macvtap interface used
==================================================================================
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvtap0 type macvtap mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address 1a:46:0b:ca:bc:7b up
#/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvlan0 type macvlan mode bridge
#/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address 1a:46:0b:ca:bc:7c up
ip link show macvtap0
ip link show macvlan0
ls -l /dev/tap*
# crw-rw---- 1 root root 249, 8 Feb 20 13:07 /dev/tap8
# No further TAP device created! (Reason why it doesn't work?)
# 7c MAC is used!?
qemu-system-x86_64 ... some params ... -net nic,model=e1000,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3 3<>/dev/tap8

Booting Knoppix 6.2 without any networking, eth0 in guest has 7c MAC 
address.

Any ideas?
==================================================================================

Thnx.

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3121 bytes --]

On 2011-02-20 13:49, Gerhard Wiesinger wrote:
> On Sun, 20 Feb 2011, Jan Kiszka wrote:
> 
>> On 2011-02-19 20:32, Gerhard Wiesinger wrote:
>>> Hello,
>> http://virt.kernelnewbies.org/MacVTap
>>
>>    ip link add link eth1 name macvtap0 type macvtap mode bridge
>>    ip link add link eth1 name macvlan0 type macvlan mode bridge
>>    ifconfig macvlan0 ...
> 
> 
> Hello Jan,
> 
> iproute2 2.6.37 is needed!
> http://devresources.linuxfoundation.org/dev/iproute2/download/iproute2-2.6.37.tar.bz2
> 
> ==================================================================================
> 
> 1.) Only macvtap interface used
> ==================================================================================
> 
> For me it was just unlogically to use a VLAN type interface. Therefore I
> tried the following which worked nearly:
> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
> macvtap0 type macvtap mode bridge
> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
> 1a:46:0b:ca:bc:7b up
> ip link show macvtap0
> ls -l /dev/tap*
> # crw-rw---- 1 root root 249, 8 Feb 20 13:07 /dev/tap8
> qemu-system-x86_64 ... some params ... -net
> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap8
> 
> Booting Knoppix 6.2 with ping is ok, MAC address is also correct on
> second host.
> 
> Only problem is that from host to guest no networking is possible (only
> from guest to other host).
> 
> Any ideas? I guess some forwarding is missing? iptables forward missing?

To my understanding macvlan(+macvtap) devices in bridge mode can only
talk to the outer world via the physical device they are attached to or
other macvlan nodes attached to that same device. Therefore...

> ==================================================================================
> 
> 2.) Only macvtap interface used
> ==================================================================================
> 
> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
> macvtap0 type macvtap mode bridge
> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
> 1a:46:0b:ca:bc:7b up
> #/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
> macvlan0 type macvlan mode bridge
> #/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address

...the need for this additional macvlan device. You just have to set an
IP on macvlan0.

Jan

> 1a:46:0b:ca:bc:7c up
> ip link show macvtap0
> ip link show macvlan0
> ls -l /dev/tap*
> # crw-rw---- 1 root root 249, 8 Feb 20 13:07 /dev/tap8
> # No further TAP device created! (Reason why it doesn't work?)
> # 7c MAC is used!?
> qemu-system-x86_64 ... some params ... -net
> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3 3<>/dev/tap8
> 
> Booting Knoppix 6.2 without any networking, eth0 in guest has 7c MAC
> address.
> 
> Any ideas?
> ==================================================================================
> 
> 
> Thnx.
> 
> Ciao,
> Gerhard
> 
> -- 
> http://www.wiesinger.com/
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger]

On Sun, 20 Feb 2011, Jan Kiszka wrote:

> On 2011-02-20 13:49, Gerhard Wiesinger wrote:
>> On Sun, 20 Feb 2011, Jan Kiszka wrote:
>>
>>> On 2011-02-19 20:32, Gerhard Wiesinger wrote:
>>>> Hello,
>>> http://virt.kernelnewbies.org/MacVTap
>>>
>>>    ip link add link eth1 name macvtap0 type macvtap mode bridge
>>>    ip link add link eth1 name macvlan0 type macvlan mode bridge
>>>    ifconfig macvlan0 ...
>>
>>
>> Hello Jan,
>>
>> iproute2 2.6.37 is needed!
>> http://devresources.linuxfoundation.org/dev/iproute2/download/iproute2-2.6.37.tar.bz2
>>
>> ==================================================================================
>>
>> 1.) Only macvtap interface used
>> ==================================================================================
>>
>> For me it was just unlogically to use a VLAN type interface. Therefore I
>> tried the following which worked nearly:
>> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>> macvtap0 type macvtap mode bridge
>> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
>> 1a:46:0b:ca:bc:7b up
>> ip link show macvtap0
>> ls -l /dev/tap*
>> # crw-rw---- 1 root root 249, 8 Feb 20 13:07 /dev/tap8
>> qemu-system-x86_64 ... some params ... -net
>> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap8
>>
>> Booting Knoppix 6.2 with ping is ok, MAC address is also correct on
>> second host.
>>
>> Only problem is that from host to guest no networking is possible (only
>> from guest to other host).
>>
>> Any ideas? I guess some forwarding is missing? iptables forward missing?
>
> To my understanding macvlan(+macvtap) devices in bridge mode can only
> talk to the outer world via the physical device they are attached to or
> other macvlan nodes attached to that same device. Therefore...
>
>> ==================================================================================
>>
>> 2.) Only macvtap interface used
>> ==================================================================================
>>
>> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>> macvtap0 type macvtap mode bridge
>> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
>> 1a:46:0b:ca:bc:7b up
>> #/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>> macvlan0 type macvlan mode bridge
>> #/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address
>
> ...the need for this additional macvlan device. You just have to set an
> IP on macvlan0.

Negative.

Tried:
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvtap0 type macvtap mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address 1a:46:0b:ca:bc:7b up
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvlan0 type macvlan mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address 1a:46:0b:ca:bc:7c up
ip link show macvtap0
ip link show macvlan0
ls -l /dev/tap*
# crw-rw---- 1 root root 249, 10 Feb 20 15:45 /dev/tap10
ifconfig macvlan0 192.168.0.23 up

DHCP is configure for 1a:46:0b:ca:bc:7b mac for 192.168.0.23 ip and 
1a:46:0b:ca:bc:7c mac for 192.168.0.24 ip.

# works without outer connection
qemu-system-x86_64 ... some params ... -net nic,model=e1000,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap10

# 2nd try:
ifconfig macvlan0 192.168.0.24 up
# No network at all
qemu-system-x86_64 ... some params ... -net nic,model=e1000,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3 3<>/dev/tap10

Seems to me quite logically because macvtap0 (and not macvlan0) is 
associated with /dev/tap10 but with another mac address set in KVM.

Any furher ideas?

Thnx.

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 4157 bytes --]

On 2011-02-20 16:19, Gerhard Wiesinger wrote:
> On Sun, 20 Feb 2011, Jan Kiszka wrote:
> 
>> On 2011-02-20 13:49, Gerhard Wiesinger wrote:
>>> On Sun, 20 Feb 2011, Jan Kiszka wrote:
>>>
>>>> On 2011-02-19 20:32, Gerhard Wiesinger wrote:
>>>>> Hello,
>>>> http://virt.kernelnewbies.org/MacVTap
>>>>
>>>>    ip link add link eth1 name macvtap0 type macvtap mode bridge
>>>>    ip link add link eth1 name macvlan0 type macvlan mode bridge
>>>>    ifconfig macvlan0 ...
>>>
>>>
>>> Hello Jan,
>>>
>>> iproute2 2.6.37 is needed!
>>> http://devresources.linuxfoundation.org/dev/iproute2/download/iproute2-2.6.37.tar.bz2
>>>
>>>
>>> ==================================================================================
>>>
>>>
>>> 1.) Only macvtap interface used
>>> ==================================================================================
>>>
>>>
>>> For me it was just unlogically to use a VLAN type interface. Therefore I
>>> tried the following which worked nearly:
>>> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>>> macvtap0 type macvtap mode bridge
>>> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
>>> 1a:46:0b:ca:bc:7b up
>>> ip link show macvtap0
>>> ls -l /dev/tap*
>>> # crw-rw---- 1 root root 249, 8 Feb 20 13:07 /dev/tap8
>>> qemu-system-x86_64 ... some params ... -net
>>> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap8
>>>
>>> Booting Knoppix 6.2 with ping is ok, MAC address is also correct on
>>> second host.
>>>
>>> Only problem is that from host to guest no networking is possible (only
>>> from guest to other host).
>>>
>>> Any ideas? I guess some forwarding is missing? iptables forward missing?
>>
>> To my understanding macvlan(+macvtap) devices in bridge mode can only
>> talk to the outer world via the physical device they are attached to or
>> other macvlan nodes attached to that same device. Therefore...
>>
>>> ==================================================================================
>>>
>>>
>>> 2.) Only macvtap interface used
>>> ==================================================================================
>>>
>>>
>>> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>>> macvtap0 type macvtap mode bridge
>>> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
>>> 1a:46:0b:ca:bc:7b up
>>> #/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>>> macvlan0 type macvlan mode bridge
>>> #/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address
>>
>> ...the need for this additional macvlan device. You just have to set an
>> IP on macvlan0.
> 
> Negative.
> 
> Tried:
> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
> macvtap0 type macvtap mode bridge
> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
> 1a:46:0b:ca:bc:7b up
> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
> macvlan0 type macvlan mode bridge
> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address
> 1a:46:0b:ca:bc:7c up
> ip link show macvtap0
> ip link show macvlan0
> ls -l /dev/tap*
> # crw-rw---- 1 root root 249, 10 Feb 20 15:45 /dev/tap10
> ifconfig macvlan0 192.168.0.23 up
> 
> DHCP is configure for 1a:46:0b:ca:bc:7b mac for 192.168.0.23 ip and
> 1a:46:0b:ca:bc:7c mac for 192.168.0.24 ip.
> 
> # works without outer connection
> qemu-system-x86_64 ... some params ... -net
> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap10
> 
> # 2nd try:
> ifconfig macvlan0 192.168.0.24 up
> # No network at all
> qemu-system-x86_64 ... some params ... -net
> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3 3<>/dev/tap10
> 
> Seems to me quite logically because macvtap0 (and not macvlan0) is
> associated with /dev/tap10 but with another mac address set in KVM.
> 
> Any furher ideas?

As you already noticed: you mixed up the MAC addresses. KVM's must be
the same as used for its frontend macvtap. The macvlan is only for the
host and has a separate one.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger]

On Sun, 20 Feb 2011, Jan Kiszka wrote:

> On 2011-02-20 16:19, Gerhard Wiesinger wrote:
>> On Sun, 20 Feb 2011, Jan Kiszka wrote:
>>
>>> On 2011-02-20 13:49, Gerhard Wiesinger wrote:
>>>> On Sun, 20 Feb 2011, Jan Kiszka wrote:
>>>>
>>>>> On 2011-02-19 20:32, Gerhard Wiesinger wrote:
>>>>>> Hello,
>>>>> http://virt.kernelnewbies.org/MacVTap
>>>>>
>>>>>    ip link add link eth1 name macvtap0 type macvtap mode bridge
>>>>>    ip link add link eth1 name macvlan0 type macvlan mode bridge
>>>>>    ifconfig macvlan0 ...
>>>>
>>>>
>>>> Hello Jan,
>>>>
>>>> iproute2 2.6.37 is needed!
>>>> http://devresources.linuxfoundation.org/dev/iproute2/download/iproute2-2.6.37.tar.bz2
>>>>
>>>>
>>>> ==================================================================================
>>>>
>>>>
>>>> 1.) Only macvtap interface used
>>>> ==================================================================================
>>>>
>>>>
>>>> For me it was just unlogically to use a VLAN type interface. Therefore I
>>>> tried the following which worked nearly:
>>>> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>>>> macvtap0 type macvtap mode bridge
>>>> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
>>>> 1a:46:0b:ca:bc:7b up
>>>> ip link show macvtap0
>>>> ls -l /dev/tap*
>>>> # crw-rw---- 1 root root 249, 8 Feb 20 13:07 /dev/tap8
>>>> qemu-system-x86_64 ... some params ... -net
>>>> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap8
>>>>
>>>> Booting Knoppix 6.2 with ping is ok, MAC address is also correct on
>>>> second host.
>>>>
>>>> Only problem is that from host to guest no networking is possible (only
>>>> from guest to other host).
>>>>
>>>> Any ideas? I guess some forwarding is missing? iptables forward missing?
>>>
>>> To my understanding macvlan(+macvtap) devices in bridge mode can only
>>> talk to the outer world via the physical device they are attached to or
>>> other macvlan nodes attached to that same device. Therefore...
>>>
>>>> ==================================================================================
>>>>
>>>>
>>>> 2.) Only macvtap interface used
>>>> ==================================================================================
>>>>
>>>>
>>>> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>>>> macvtap0 type macvtap mode bridge
>>>> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
>>>> 1a:46:0b:ca:bc:7b up
>>>> #/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>>>> macvlan0 type macvlan mode bridge
>>>> #/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address
>>>
>>> ...the need for this additional macvlan device. You just have to set an
>>> IP on macvlan0.
>>
>> Negative.
>>
>> Tried:
>> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>> macvtap0 type macvtap mode bridge
>> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
>> 1a:46:0b:ca:bc:7b up
>> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
>> macvlan0 type macvlan mode bridge
>> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address
>> 1a:46:0b:ca:bc:7c up
>> ip link show macvtap0
>> ip link show macvlan0
>> ls -l /dev/tap*
>> # crw-rw---- 1 root root 249, 10 Feb 20 15:45 /dev/tap10
>> ifconfig macvlan0 192.168.0.23 up
>>
>> DHCP is configure for 1a:46:0b:ca:bc:7b mac for 192.168.0.23 ip and
>> 1a:46:0b:ca:bc:7c mac for 192.168.0.24 ip.
>>
>> # works without outer connection
>> qemu-system-x86_64 ... some params ... -net
>> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap10
>>
>> # 2nd try:
>> ifconfig macvlan0 192.168.0.24 up
>> # No network at all
>> qemu-system-x86_64 ... some params ... -net
>> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3 3<>/dev/tap10
>>
>> Seems to me quite logically because macvtap0 (and not macvlan0) is
>> associated with /dev/tap10 but with another mac address set in KVM.
>>
>> Any furher ideas?
>
> As you already noticed: you mixed up the MAC addresses. KVM's must be
> the same as used for its frontend macvtap. The macvlan is only for the
> host and has a separate one.

I think I did everyting right in the last 1st try but it still didn't 
work:
1.) macvtap0: MAC: 1a:46:0b:ca:bc:7b
2.) macvlan0: MAC: 1a:46:0b:ca:bc:7c, 192.168.0.23
3.) KVM: MAC: 1a:46:0b:ca:bc:7b, assigned IP from DHCP: 1a:46:0b:ca:bc:7b
(looks like an IP address conflict inside guest and outside?)

That should be as you explained, right?

BTW: Posted a bugfix to hw/pcnet.c with AMD drivers.

Thnx.

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Arnd Bergmann]

On Sunday 20 February 2011, Gerhard Wiesinger wrote:
> >
> > Not sure if this is by design or due to internals of the networking
> > stack, but it looks unintuitive from user perspective. Maybe Arnd can
> > shed a light on this.

The lower device cannot be in bridge mode, because that would make the
logic in the kernel awfully complex. I agree that it's a bit unfortunate,
but it simplified the design a lot.

> 
> > Of course, you could also simply offload all that setup to libvirt.
> 
> http://wiki.libvirt.org/page/Networking
> But that still needs a bridge on ethernet level and break up the existing 
> interface, right?

No. macvtap is a tap that sits on an external interface, you don't
need a bridge if you use it.
 
> BTW: From: http://virt.kernelnewbies.org/MacVTap
> As of QEMU 0.12:
> qemu -net nic,model=virtio,addr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap11
> Is there a newer Syntax with tap interfaces possible (QEMU GIT) without 
> bash redirects?

I made some suggestions how to integrate it with qemu, but they never got in.
Libvirt makes it a lot easier, though. We discussed that it should be cleaned
up when the networking helper scripts make it into qemu. Not sure if that
ever happened.

	Arnd

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Arnd Bergmann]

On Sunday 20 February 2011, Gerhard Wiesinger wrote:
> >> qemu-system-x86_64 ... some params ... -net
> >> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3 3<>/dev/tap10
> >>
> >> Seems to me quite logically because macvtap0 (and not macvlan0) is
> >> associated with /dev/tap10 but with another mac address set in KVM.
> >>
> >> Any furher ideas?
> >
> > As you already noticed: you mixed up the MAC addresses. KVM's must be
> > the same as used for its frontend macvtap. The macvlan is only for the
> > host and has a separate one.
> 
> I think I did everyting right in the last 1st try but it still didn't 
> work:
> 1.) macvtap0: MAC: 1a:46:0b:ca:bc:7b
> 2.) macvlan0: MAC: 1a:46:0b:ca:bc:7c, 192.168.0.23
> 3.) KVM: MAC: 1a:46:0b:ca:bc:7b, assigned IP from DHCP: 1a:46:0b:ca:bc:7b
> (looks like an IP address conflict inside guest and outside?)
> 
> That should be as you explained, right?
> 

The qemu command above has the 7c mac address, which does not match.
Do you see the interface in the guest using "ip link show" ?

	Arnd

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger]

On Sun, 20 Feb 2011, Arnd Bergmann wrote:

> On Sunday 20 February 2011, Gerhard Wiesinger wrote:
>>>
>>> Not sure if this is by design or due to internals of the networking
>>> stack, but it looks unintuitive from user perspective. Maybe Arnd can
>>> shed a light on this.
>
> The lower device cannot be in bridge mode, because that would make the
> logic in the kernel awfully complex. I agree that it's a bit unfortunate,
> but it simplified the design a lot.
>
>>
>>> Of course, you could also simply offload all that setup to libvirt.
>>
>> http://wiki.libvirt.org/page/Networking
>> But that still needs a bridge on ethernet level and break up the existing
>> interface, right?
>
> No. macvtap is a tap that sits on an external interface, you don't
> need a bridge if you use it.
>
>> BTW: From: http://virt.kernelnewbies.org/MacVTap
>> As of QEMU 0.12:
>> qemu -net nic,model=virtio,addr=1a:46:0b:ca:bc:7b -net tap,fd=3 3<>/dev/tap11
>> Is there a newer Syntax with tap interfaces possible (QEMU GIT) without
>> bash redirects?
>
> I made some suggestions how to integrate it with qemu, but they never got in.
> Libvirt makes it a lot easier, though. We discussed that it should be cleaned
> up when the networking helper scripts make it into qemu. Not sure if that
> ever happened.

I think now it's a good time to integrate this :-)

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger]

On Sun, 20 Feb 2011, Arnd Bergmann wrote:

> On Sunday 20 February 2011, Gerhard Wiesinger wrote:
>>>> qemu-system-x86_64 ... some params ... -net
>>>> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3 3<>/dev/tap10
>>>>
>>>> Seems to me quite logically because macvtap0 (and not macvlan0) is
>>>> associated with /dev/tap10 but with another mac address set in KVM.
>>>>
>>>> Any furher ideas?
>>>
>>> As you already noticed: you mixed up the MAC addresses. KVM's must be
>>> the same as used for its frontend macvtap. The macvlan is only for the
>>> host and has a separate one.
>>
>> I think I did everyting right in the last 1st try but it still didn't
>> work:
>> 1.) macvtap0: MAC: 1a:46:0b:ca:bc:7b
>> 2.) macvlan0: MAC: 1a:46:0b:ca:bc:7c, 192.168.0.23
>> 3.) KVM: MAC: 1a:46:0b:ca:bc:7b, assigned IP from DHCP: 1a:46:0b:ca:bc:7b
>> (looks like an IP address conflict inside guest and outside?)
>>
>> That should be as you explained, right?
>
> The qemu command above has the 7c mac address, which does not match.
> Do you see the interface in the guest using "ip link show" ?

Only the second try had 7c mac address. 1st try had 7b mac address.

Now I tried the following still with same success that host/guest 
communication (because of missing hairpin mode of the switch) is NOT 
possible.
############################################################
# Used 7c mac in KVM
############################################################
# No need for bridge mode and iproute2 2.6.37
ip link add link eth0 name macvtap0 type macvtap
ip link set macvtap0 address 1a:46:0b:ca:bc:7b up
ip link add link macvtap0 name macvlan0 type macvlan
ip link set macvlan0 address 1a:46:0b:ca:bc:7c up
ip link show macvtap0
ip link show macvlan0
ls -l /dev/tap*
# crw-rw---- 1 root root 249, 18 Feb 21 07:21 /dev/tap18
ifconfig macvlan0 192.168.0.23 up
/root/download/qemu/git/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 ... -net nic,model=pcnet,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3
# Guest has 7c mac, external traffic NOT ok
ip link delete macvlan0 type macvlan
ls -l /dev/tap*
# Still present
# crw-rw---- 1 root root 249, 18 Feb 21 07:21 /dev/tap18
ip link delete macvtap0 type macvtap
ls -l /dev/tap*
# ls: cannot access /dev/tap*: No such file or directory
############################################################
# Used 7b mac in KVM
############################################################
# No need for bridge mode and iproute2 2.6.37
ip link add link eth0 name macvtap0 type macvtap
ip link set macvtap0 address 1a:46:0b:ca:bc:7b up
ip link add link macvtap0 name macvlan0 type macvlan
ip link set macvlan0 address 1a:46:0b:ca:bc:7c up
ip link show macvtap0
ip link show macvlan0
ls -l /dev/tap*
# crw-rw---- 1 root root 249, 18 Feb 21 07:21 /dev/tap18
ifconfig macvlan0 192.168.0.23 up
/root/download/qemu/git/qemu-kvm/x86_64-softmmu/qemu-system-x86_64... -net nic,model=pcnet,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3
# Guest has 7b mac, external traffic ok, internal not
ip link delete macvlan0 type macvlan
ls -l /dev/tap*
# Still present
# crw-rw---- 1 root root 249, 18 Feb 21 07:21 /dev/tap18
ip link delete macvtap0 type macvtap
ls -l /dev/tap*
# ls: cannot access /dev/tap*: No such file or directory
############################################################

Now I think I tried all useful possible combinations:
1.) macvtap0 and macvlan0 in bridged and non bridge mode
2.) macvlan0 based on eth0 or based on macvtap0
3.) Using and ip address on macvlan0 or not (tried both for 7b mac and 7c 
mac)
4.) Using 7b and 7c mac address in KVM (MAC in KVM is always the command 
line configured) and used tap interface from macvtap0 (as no second tap 
devices shows up)

Summary:
1.) Using macvtap0 only with 7b mac on interface and also at qemu works 
well in bridged mode as well as non bridged mode but with limitation no guest/host 
communication possible, used interface in KVM is tap interface created by 
macvtap0. Quite logically that it doesn't work with guest/host because of 
missing hairpin mode on the switch. But according to 
http://virt.kernelnewbies.org/MacVTap bridge mode should work even without 
hairpinning available on the switch.
2.) macvlan0 doesn't create any tap interface therefore it can't be used 
as a device on KVM.
3.) Using 7c mac address in KVM doesn't work at all regardsless of 
setting ip addresses on macvlan0 or any other setup.

@Arnd: Can you explain a setup in detail which should work also with 
host/guest communication. Thnx.

Any further ideas?
Which kernel is needed to work?
(current: 2.6.34.7-56.fc13.x86_64)

Thnx.

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 5092 bytes --]

On 2011-02-21 07:40, Gerhard Wiesinger wrote:
> On Sun, 20 Feb 2011, Arnd Bergmann wrote:
> 
>> On Sunday 20 February 2011, Gerhard Wiesinger wrote:
>>>>> qemu-system-x86_64 ... some params ... -net
>>>>> nic,model=e1000,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3 3<>/dev/tap10
>>>>>
>>>>> Seems to me quite logically because macvtap0 (and not macvlan0) is
>>>>> associated with /dev/tap10 but with another mac address set in KVM.
>>>>>
>>>>> Any furher ideas?
>>>>
>>>> As you already noticed: you mixed up the MAC addresses. KVM's must be
>>>> the same as used for its frontend macvtap. The macvlan is only for the
>>>> host and has a separate one.
>>>
>>> I think I did everyting right in the last 1st try but it still didn't
>>> work:
>>> 1.) macvtap0: MAC: 1a:46:0b:ca:bc:7b
>>> 2.) macvlan0: MAC: 1a:46:0b:ca:bc:7c, 192.168.0.23
>>> 3.) KVM: MAC: 1a:46:0b:ca:bc:7b, assigned IP from DHCP:
>>> 1a:46:0b:ca:bc:7b
>>> (looks like an IP address conflict inside guest and outside?)
>>>
>>> That should be as you explained, right?
>>
>> The qemu command above has the 7c mac address, which does not match.
>> Do you see the interface in the guest using "ip link show" ?
> 
> Only the second try had 7c mac address. 1st try had 7b mac address.
> 
> Now I tried the following still with same success that host/guest
> communication (because of missing hairpin mode of the switch) is NOT
> possible.
> ############################################################
> # Used 7c mac in KVM
> ############################################################
> # No need for bridge mode and iproute2 2.6.37
> ip link add link eth0 name macvtap0 type macvtap
> ip link set macvtap0 address 1a:46:0b:ca:bc:7b up
> ip link add link macvtap0 name macvlan0 type macvlan
> ip link set macvlan0 address 1a:46:0b:ca:bc:7c up
> ip link show macvtap0
> ip link show macvlan0
> ls -l /dev/tap*
> # crw-rw---- 1 root root 249, 18 Feb 21 07:21 /dev/tap18
> ifconfig macvlan0 192.168.0.23 up
> /root/download/qemu/git/qemu-kvm/x86_64-softmmu/qemu-system-x86_64 ...
> -net nic,model=pcnet,macaddr=1a:46:0b:ca:bc:7c -net tap,fd=3
> # Guest has 7c mac, external traffic NOT ok
> ip link delete macvlan0 type macvlan
> ls -l /dev/tap*
> # Still present
> # crw-rw---- 1 root root 249, 18 Feb 21 07:21 /dev/tap18
> ip link delete macvtap0 type macvtap
> ls -l /dev/tap*
> # ls: cannot access /dev/tap*: No such file or directory
> ############################################################
> # Used 7b mac in KVM
> ############################################################
> # No need for bridge mode and iproute2 2.6.37
> ip link add link eth0 name macvtap0 type macvtap
> ip link set macvtap0 address 1a:46:0b:ca:bc:7b up
> ip link add link macvtap0 name macvlan0 type macvlan
> ip link set macvlan0 address 1a:46:0b:ca:bc:7c up
> ip link show macvtap0
> ip link show macvlan0
> ls -l /dev/tap*
> # crw-rw---- 1 root root 249, 18 Feb 21 07:21 /dev/tap18
> ifconfig macvlan0 192.168.0.23 up
> /root/download/qemu/git/qemu-kvm/x86_64-softmmu/qemu-system-x86_64...
> -net nic,model=pcnet,macaddr=1a:46:0b:ca:bc:7b -net tap,fd=3
> # Guest has 7b mac, external traffic ok, internal not
> ip link delete macvlan0 type macvlan
> ls -l /dev/tap*
> # Still present
> # crw-rw---- 1 root root 249, 18 Feb 21 07:21 /dev/tap18
> ip link delete macvtap0 type macvtap
> ls -l /dev/tap*
> # ls: cannot access /dev/tap*: No such file or directory
> ############################################################
> 
> Now I think I tried all useful possible combinations:
> 1.) macvtap0 and macvlan0 in bridged and non bridge mode
> 2.) macvlan0 based on eth0 or based on macvtap0
> 3.) Using and ip address on macvlan0 or not (tried both for 7b mac and
> 7c mac)
> 4.) Using 7b and 7c mac address in KVM (MAC in KVM is always the command
> line configured) and used tap interface from macvtap0 (as no second tap
> devices shows up)
> 
> Summary:
> 1.) Using macvtap0 only with 7b mac on interface and also at qemu works
> well in bridged mode as well as non bridged mode but with limitation no
> guest/host communication possible, used interface in KVM is tap
> interface created by macvtap0. Quite logically that it doesn't work with
> guest/host because of missing hairpin mode on the switch. But according
> to http://virt.kernelnewbies.org/MacVTap bridge mode should work even
> without hairpinning available on the switch.
> 2.) macvlan0 doesn't create any tap interface therefore it can't be used
> as a device on KVM.
> 3.) Using 7c mac address in KVM doesn't work at all regardsless of
> setting ip addresses on macvlan0 or any other setup.
> 
> @Arnd: Can you explain a setup in detail which should work also with
> host/guest communication. Thnx.
> 
> Any further ideas?
> Which kernel is needed to work?
> (current: 2.6.34.7-56.fc13.x86_64)

Actually, I tried this successfully over a 2.6.38-rcX, but I have no
idea what changes related to macvlan/tap went in since .34 and if this
is required for this.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Arnd Bergmann]

On Monday 21 February 2011, Jan Kiszka wrote:
> > Now I think I tried all useful possible combinations:
> > 1.) macvtap0 and macvlan0 in bridged and non bridge mode
> > 2.) macvlan0 based on eth0 or based on macvtap0
> > 3.) Using and ip address on macvlan0 or not (tried both for 7b mac and
> > 7c mac)
> > 4.) Using 7b and 7c mac address in KVM (MAC in KVM is always the command
> > line configured) and used tap interface from macvtap0 (as no second tap
> > devices shows up)
> > 
> > Summary:
> > 1.) Using macvtap0 only with 7b mac on interface and also at qemu works
> > well in bridged mode as well as non bridged mode but with limitation no
> > guest/host communication possible, used interface in KVM is tap
> > interface created by macvtap0. Quite logically that it doesn't work with
> > guest/host because of missing hairpin mode on the switch. But according
> > to http://virt.kernelnewbies.org/MacVTap bridge mode should work even
> > without hairpinning available on the switch.
> > 2.) macvlan0 doesn't create any tap interface therefore it can't be used
> > as a device on KVM.
> > 3.) Using 7c mac address in KVM doesn't work at all regardsless of
> > setting ip addresses on macvlan0 or any other setup.
> > 
> > @Arnd: Can you explain a setup in detail which should work also with
> > host/guest communication. Thnx.
> > 
> > Any further ideas?
> > Which kernel is needed to work?
> > (current: 2.6.34.7-56.fc13.x86_64)
> 
> Actually, I tried this successfully over a 2.6.38-rcX, but I have no
> idea what changes related to macvlan/tap went in since .34 and if this
> is required for this.

We had a few bugs in .34, but it should work in general.

One thing to make sure is that the host has connectivity through the
macvlan interface and the lower interface (eth0) has no IP address assigned
but is 'up'.

It could also be a bug in the NIC, you could try to set the NIC into
promiscious mode using ethtool to work around that.

Also, the lower interface must not be connected to a bridge device.

	Arnd

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger]

On Mon, 21 Feb 2011, Arnd Bergmann wrote:

> On Monday 21 February 2011, Jan Kiszka wrote:
>>> Now I think I tried all useful possible combinations:
>>> 1.) macvtap0 and macvlan0 in bridged and non bridge mode
>>> 2.) macvlan0 based on eth0 or based on macvtap0
>>> 3.) Using and ip address on macvlan0 or not (tried both for 7b mac and
>>> 7c mac)
>>> 4.) Using 7b and 7c mac address in KVM (MAC in KVM is always the command
>>> line configured) and used tap interface from macvtap0 (as no second tap
>>> devices shows up)
>>>
>>> Summary:
>>> 1.) Using macvtap0 only with 7b mac on interface and also at qemu works
>>> well in bridged mode as well as non bridged mode but with limitation no
>>> guest/host communication possible, used interface in KVM is tap
>>> interface created by macvtap0. Quite logically that it doesn't work with
>>> guest/host because of missing hairpin mode on the switch. But according
>>> to http://virt.kernelnewbies.org/MacVTap bridge mode should work even
>>> without hairpinning available on the switch.
>>> 2.) macvlan0 doesn't create any tap interface therefore it can't be used
>>> as a device on KVM.
>>> 3.) Using 7c mac address in KVM doesn't work at all regardsless of
>>> setting ip addresses on macvlan0 or any other setup.
>>>
>>> @Arnd: Can you explain a setup in detail which should work also with
>>> host/guest communication. Thnx.
>>>
>>> Any further ideas?
>>> Which kernel is needed to work?
>>> (current: 2.6.34.7-56.fc13.x86_64)
>>
>> Actually, I tried this successfully over a 2.6.38-rcX, but I have no
>> idea what changes related to macvlan/tap went in since .34 and if this
>> is required for this.
>
> We had a few bugs in .34, but it should work in general.
>
> One thing to make sure is that the host has connectivity through the
> macvlan interface and the lower interface (eth0) has no IP address assigned
> but is 'up'.
>
> It could also be a bug in the NIC, you could try to set the NIC into
> promiscious mode using ethtool to work around that.

Hello,

After some further tests and looking at the iproute ip and kernel code I 
finally gave up because I thing such a setup it is not possible without 
breaking up/reconfiguring eth0. When I have to reconfigure eth0 I think a 
better approach is to configure a bridge which I finally did and works 
well.

I tried to explain/document the macvtap/macvlan concepts and limitations 
below. Please comment on it whether this is true or false.

macvtap/macvlan driver concepts and limitations:
1.) macvlan driver adds a MAC address to a lower interface device where 
the actual macvlanx device is based on
2.) macvtap driver is based on macvlan driver and macvtap driver adds 
additional functionality of interface <=> external program communication 
with stdin/stdout channel.
3.) Limitations: macvtap/macvlan based devices can only communicate with 
childs based on the same lower device (e.g. eth0 in this sample) but not 
to the lower device itself, only to the outside world of the interface
(I guess limitations are derived from function macvlan_queue_xmit: 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=drivers/net/macvlan.c;h=6ed577b065df136a18af5ee483810773ca4f43f8;hb=HEAD#l217)

Example:
# Lower device eth0 must not have an IP address assigned, but when also no communication is possible with macv* devices
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvtap0 type macvtap mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address 1a:46:0b:ca:bc:7b up
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvtap1 type macvtap mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap1 address 1a:46:0b:ca:bc:7c up
/root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name macvlan0 type macvtap mode bridge
/root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address 1a:46:0b:ca:bc:7d up

Example usage:
eth0 should not be assigned any IP. macvlan0 should be used as the main 
host interface and the main IP address is configured.
macvtap0 and macvtap1 can be used in e.g. VMs.
Bridge communication is possible between the 3 interfaces and the outer 
world: macvtap0, macvtap1, macvlan0
E.g.:
1.) macvtap0 <=> macvtap1, macvlan0, eth0 external
2.) macvtap1 <=> macvtap0, macvlan0, eth0 external
3.) macvlan0 <=> macvtap0, macvtap1, eth0 external

But no communication is possible between lower device eth0 and the child 
interfaces macvtap0, macvtap1, macvlan0:
1.) eth0 <=> macvtap0
2.) eth0 <=> macvtap1
3.) eth0 <=> macvlan0

finally this makes the macvlan/macvtap approach useless because main eth0 
interface must still be broken in the chain and reconfigured which was 
against the requirements that eth0 should not be touched and reconfigured!

Bridge Setup for Fedora 13 is below (reconfigure with service network 
restart, be carefull on remote operation or have a serial console :-)!).

Nevertheless, thank you for your help and I'm ready for any other 
approach :-).

Ciao,
Gerhard

--
http://www.wiesinger.com/

############################################################
cat /etc/sysconfig/network-scripts/ifcfg-br0
# Bridge config
DEVICE=br0
TYPE=Bridge
BOOTPROTO=static
BROADCAST=192.168.0.255
IPADDR=192.168.0.9
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
DNS1=192.168.0.2
SEARCH="mydomain"
NM_CONTROLLED=
############################################################
cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0

# Removed for bridge config
#BOOTPROTO=static
#BROADCAST=192.168.0.255

#########################
# Onboard one
HWADDR=00:1f:c6:8c:59:1f
#########################

# Removed for bridge config
#IPADDR=192.168.0.9
#NETMASK=255.255.255.0
#NETWORK=192.168.0.0
ONBOOT=yes
#DNS1=192.168.0.2
#SEARCH="mydomain"
NM_CONTROLLED=

# New to add for bridge
# Bridge membership
BRIDGE=br0
BOOTPROTO=none
############################################################
cat /etc/sysconfig/network-scripts/_ifcfg-eth0.nonbridge
# Old config without bridge
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.0.255

#########################
# Onboard one
HWADDR=00:1f:c6:8c:59:1f
#########################

IPADDR=192.168.0.9
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
DNS1=192.168.0.2
SEARCH="mydomain"
NM_CONTROLLED=
############################################################

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Jan Kiszka]

On 2011-02-23 07:38, Gerhard Wiesinger wrote:
> On Mon, 21 Feb 2011, Arnd Bergmann wrote:
> 
>> On Monday 21 February 2011, Jan Kiszka wrote:
>>>> Now I think I tried all useful possible combinations:
>>>> 1.) macvtap0 and macvlan0 in bridged and non bridge mode
>>>> 2.) macvlan0 based on eth0 or based on macvtap0
>>>> 3.) Using and ip address on macvlan0 or not (tried both for 7b mac and
>>>> 7c mac)
>>>> 4.) Using 7b and 7c mac address in KVM (MAC in KVM is always the
>>>> command
>>>> line configured) and used tap interface from macvtap0 (as no second tap
>>>> devices shows up)
>>>>
>>>> Summary:
>>>> 1.) Using macvtap0 only with 7b mac on interface and also at qemu works
>>>> well in bridged mode as well as non bridged mode but with limitation no
>>>> guest/host communication possible, used interface in KVM is tap
>>>> interface created by macvtap0. Quite logically that it doesn't work
>>>> with
>>>> guest/host because of missing hairpin mode on the switch. But according
>>>> to http://virt.kernelnewbies.org/MacVTap bridge mode should work even
>>>> without hairpinning available on the switch.
>>>> 2.) macvlan0 doesn't create any tap interface therefore it can't be
>>>> used
>>>> as a device on KVM.
>>>> 3.) Using 7c mac address in KVM doesn't work at all regardsless of
>>>> setting ip addresses on macvlan0 or any other setup.
>>>>
>>>> @Arnd: Can you explain a setup in detail which should work also with
>>>> host/guest communication. Thnx.
>>>>
>>>> Any further ideas?
>>>> Which kernel is needed to work?
>>>> (current: 2.6.34.7-56.fc13.x86_64)
>>>
>>> Actually, I tried this successfully over a 2.6.38-rcX, but I have no
>>> idea what changes related to macvlan/tap went in since .34 and if this
>>> is required for this.
>>
>> We had a few bugs in .34, but it should work in general.
>>
>> One thing to make sure is that the host has connectivity through the
>> macvlan interface and the lower interface (eth0) has no IP address
>> assigned
>> but is 'up'.
>>
>> It could also be a bug in the NIC, you could try to set the NIC into
>> promiscious mode using ethtool to work around that.
> 
> Hello,
> 
> After some further tests and looking at the iproute ip and kernel code I
> finally gave up because I thing such a setup it is not possible without
> breaking up/reconfiguring eth0. When I have to reconfigure eth0 I think
> a better approach is to configure a bridge which I finally did and works
> well.
> 
> I tried to explain/document the macvtap/macvlan concepts and limitations
> below. Please comment on it whether this is true or false.
> 
> macvtap/macvlan driver concepts and limitations:
> 1.) macvlan driver adds a MAC address to a lower interface device where
> the actual macvlanx device is based on
> 2.) macvtap driver is based on macvlan driver and macvtap driver adds
> additional functionality of interface <=> external program communication
> with stdin/stdout channel.
> 3.) Limitations: macvtap/macvlan based devices can only communicate with
> childs based on the same lower device (e.g. eth0 in this sample) but not
> to the lower device itself, only to the outside world of the interface
> (I guess limitations are derived from function macvlan_queue_xmit:
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=drivers/net/macvlan.c;h=6ed577b065df136a18af5ee483810773ca4f43f8;hb=HEAD#l217)
> 
> 
> Example:
> # Lower device eth0 must not have an IP address assigned, but when also
> no communication is possible with macv* devices
> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
> macvtap0 type macvtap mode bridge
> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap0 address
> 1a:46:0b:ca:bc:7b up
> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
> macvtap1 type macvtap mode bridge
> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvtap1 address
> 1a:46:0b:ca:bc:7c up
> /root/download/iproute2/iproute2-2.6.37/ip/ip link add link eth0 name
> macvlan0 type macvtap mode bridge
> /root/download/iproute2/iproute2-2.6.37/ip/ip link set macvlan0 address
> 1a:46:0b:ca:bc:7d up
> 
> Example usage:
> eth0 should not be assigned any IP. macvlan0 should be used as the main
> host interface and the main IP address is configured.
> macvtap0 and macvtap1 can be used in e.g. VMs.
> Bridge communication is possible between the 3 interfaces and the outer
> world: macvtap0, macvtap1, macvlan0
> E.g.:
> 1.) macvtap0 <=> macvtap1, macvlan0, eth0 external
> 2.) macvtap1 <=> macvtap0, macvlan0, eth0 external
> 3.) macvlan0 <=> macvtap0, macvtap1, eth0 external
> 
> But no communication is possible between lower device eth0 and the child
> interfaces macvtap0, macvtap1, macvlan0:
> 1.) eth0 <=> macvtap0
> 2.) eth0 <=> macvtap1
> 3.) eth0 <=> macvlan0

Right, but if I set IP(eth0) == IP(macvlan0), I'm able to communicate
between macvlan0 and mactapX, thus between guest and host. Just
re-checked here, still works (after resolving the usual MAC address mess
I caused by configuring manually).

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Arnd Bergmann]

On Wednesday 23 February 2011, Gerhard Wiesinger wrote:
> After some further tests and looking at the iproute ip and kernel code I 
> finally gave up because I thing such a setup it is not possible without 
> breaking up/reconfiguring eth0. When I have to reconfigure eth0 I think a 
> better approach is to configure a bridge which I finally did and works 
> well.
> 
> I tried to explain/document the macvtap/macvlan concepts and limitations 
> below. Please comment on it whether this is true or false.
> 
> macvtap/macvlan driver concepts and limitations:
> 1.) macvlan driver adds a MAC address to a lower interface device where 
> the actual macvlanx device is based on
> 2.) macvtap driver is based on macvlan driver and macvtap driver adds 
> additional functionality of interface <=> external program communication 
> with stdin/stdout channel.
> 3.) Limitations: macvtap/macvlan based devices can only communicate with 
> childs based on the same lower device (e.g. eth0 in this sample) but not 
> to the lower device itself, only to the outside world of the interface

Correct.

> finally this makes the macvlan/macvtap approach useless because main eth0 
> interface must still be broken in the chain and reconfigured which was 
> against the requirements that eth0 should not be touched and reconfigured!

Yes, that is unfortunate, but it's the same that you'd get with a bridge
device: 
When you have a bridge on top of eth0, you can no longer assign an IP
address to eth0 and let it communicate with the virtual ports on the
bridge. You need to instead set the IP address on the bridge itself.

Macvlan is slightly better because it allows you to have multiple
host devices that can each have their own MAC/IP address, unlike
the bridge, but of course it can not be connected to anything else
besides macvlan or macvtap ports.

	Arnd

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Gerhard Wiesinger]

On Wed, 23 Feb 2011, Jan Kiszka wrote:
> Right, but if I set IP(eth0) == IP(macvlan0), I'm able to communicate
> between macvlan0 and mactapX, thus between guest and host. Just
> re-checked here, still works (after resolving the usual MAC address mess
> I caused by configuring manually).

Thnx for the tipp.

Did you use MAC(eth0) == MAC(macvlan0) or MAC(eth0) <> MAC(macvlan0) to 
get it to work?

OK, that should be possible because of the layer2 "split brain" 
implementation of macvlan/macvtap between lower interface (e.g. eth0) and 
mavlan*/macvtap*. Therefore the macvlan*/macvtap* interfaces see only each 
other and the outer world/eth0 is another distinct layer2 domain.

My only concern is about layer3 (IP) and same IP address on the same host. 
Because of the "split brain" there shouldn't be any problem from the 
macvlan*/macvtap* point of view and also from the eth0 outside world view.

But from the view of "localhost"/iptables/routing the kernel sees 2 
identical IP addresses. Which one is used on a local ping or local 
connect?

I will try it out, too.

Ciao,
Gerhard

--
http://www.wiesinger.com/

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Jan Kiszka]

On 2011-02-24 07:49, Gerhard Wiesinger wrote:
> On Wed, 23 Feb 2011, Jan Kiszka wrote:
>> Right, but if I set IP(eth0) == IP(macvlan0), I'm able to communicate
>> between macvlan0 and mactapX, thus between guest and host. Just
>> re-checked here, still works (after resolving the usual MAC address mess
>> I caused by configuring manually).
> 
> Thnx for the tipp.
> 
> Did you use MAC(eth0) == MAC(macvlan0) or MAC(eth0) <> MAC(macvlan0) to 
> get it to work?

The latter (I just let macvlan/tap choose their MACs).

> 
> OK, that should be possible because of the layer2 "split brain" 
> implementation of macvlan/macvtap between lower interface (e.g. eth0) and 
> mavlan*/macvtap*. Therefore the macvlan*/macvtap* interfaces see only each 
> other and the outer world/eth0 is another distinct layer2 domain.
> 
> My only concern is about layer3 (IP) and same IP address on the same host. 
> Because of the "split brain" there shouldn't be any problem from the 
> macvlan*/macvtap* point of view and also from the eth0 outside world view.
> 
> But from the view of "localhost"/iptables/routing the kernel sees 2 
> identical IP addresses. Which one is used on a local ping or local 
> connect?

None of both, rather the lookback interface.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Re: Network bridging without adding bridge with brctl, possible?

[Arnd Bergmann]

On Thursday 24 February 2011, Jan Kiszka wrote:
> On 2011-02-24 07:49, Gerhard Wiesinger wrote:
> > On Wed, 23 Feb 2011, Jan Kiszka wrote:
> >> Right, but if I set IP(eth0) == IP(macvlan0), I'm able to communicate
> >> between macvlan0 and mactapX, thus between guest and host. Just
> >> re-checked here, still works (after resolving the usual MAC address mess
> >> I caused by configuring manually).
> > 
> > Thnx for the tipp.
> > 
> > Did you use MAC(eth0) == MAC(macvlan0) or MAC(eth0) <> MAC(macvlan0) to 
> > get it to work?
> 
> The latter (I just let macvlan/tap choose their MACs).

You cannot set the two to the same MAC address while they are up, but I think
you can do

ip link set eth0 down
ip link set macvlan0 ${MAC_ETH0}
ip link set macvlan0 up

Doing that is a bit tricky if eth0 is your only connection to the machine...

	Arnd