From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33030) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XPxLy-0002B0-Sq for qemu-devel@nongnu.org; Fri, 05 Sep 2014 13:31:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1XPxLt-00059X-KM for qemu-devel@nongnu.org; Fri, 05 Sep 2014 13:31:18 -0400 Received: from indium.canonical.com ([91.189.90.7]:42293) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XPxLt-00059N-8f for qemu-devel@nongnu.org; Fri, 05 Sep 2014 13:31:13 -0400 Received: from loganberry.canonical.com ([91.189.90.37]) by indium.canonical.com with esmtp (Exim 4.76 #1 (Debian)) id 1XPxLs-0000uG-DH for ; Fri, 05 Sep 2014 17:31:12 +0000 Received: from loganberry.canonical.com (localhost [127.0.0.1]) by loganberry.canonical.com (Postfix) with ESMTP id 5BCED2E80AF for ; Fri, 5 Sep 2014 17:31:12 +0000 (UTC) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Date: Fri, 05 Sep 2014 17:24:30 -0000 From: Serge Hallyn <1362755@bugs.launchpad.net> Sender: bounces@canonical.com References: <20140828184002.18278.30739.malonedeb@soybean.canonical.com> Message-Id: <20140905172430.GN16450@ubuntumail> Errors-To: bounces@canonical.com Subject: Re: [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself. Reply-To: Bug 1362755 <1362755@bugs.launchpad.net> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Does this also fail with a build of git://git.qemu.org/qemu.git? -- = You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1362755 Title: QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself. Status in QEMU: New Bug description: Guys, Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network. It does not route tagged VLAN packets. That's it, it is impossible to use Trusty acting as a QEmu 2.0 Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual tagged network within itself. QEmu 2.X guest does not route traffic when with tagged VLANs! So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as "firewalls / routers", and it have an easy to reproduce, connectivity problem. This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic) with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1). I have this very same setup up and running, on about ~100 physical servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them, the QEmu Hypervisors dedicated to host "guest acting as routers / firewalls", like a "borger gateway" for example, that it does not work as expected. One interesting thing to note is that, this BUG appear only, and only at, the QEmu Hypervisors dedicated to host guests that are used as `router / firewalls` (as I said above), others QEmu Hypervisors of my network does not suffer from this problem. Another interesting point is that it fails to route tagged VLAN packets only when these packets are originated from within the Hypervisor itself, I mean, packets from both host and other guests (not the router/firewall guest itself), suffer from this connectivity problem. As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a "border hypervisor". So, this proves that there is something wrong with QEmu. I already tested it with both `openvswitch-switch` and with `bridge- utils`, same bad results. So, don't waste your time trying `bridge- utils` (optional steps while reproducing it), you can keep OVS bridges from original design. I think that I'm using the best pratices to build this environment, as follows... * Topology * QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"): 1- Physical machine with 3 NICs; 2- Minimal Ubuntu 14.04.1 installed and upgraded; 3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd tcpd= ump". - eth0 connected to the Internet - VLAN tag 10; - eth1 connected to the LAN1 - VLAN tag 100; - eth2 connected to the LAN2 - VLAN tag 200; Guest (guest-fw-1.domain.com - the "border gateway" itself - regular guest acting as a router with iptables/ip6tables): 1- Virtual Machine with 3 NICs (VirtIO); 2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded; 3- Packages installed: "aiccu iptables vlan pv-grub-menu". OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your (virt-)lab network, including the `qemu-host-1` itself. Steps to reproduce * Preparing the `qemu-host-1` host: - Configure the /etc/network/interfaces with: --- # The loopback network interface auto lo iface lo inet loopback auto eth0 iface eth0 inet manual =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0up ip link set $IFACE up =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0down ip link set $IFACE d= own auto eth1 iface eth1 inet manual =C2=A0up ip link set dev $IFACE up =C2=A0down ip link set dev $IFACE down auto ovsbr1p1 iface ovsbr1p1 inet6 auto iface ovsbr1p1 inet static =C2=A0address 192.168.1.10 =C2=A0netmask 24 =C2=A0gateway 192.168.1.1 auto eth2 iface eth2 inet manual =C2=A0up ip link set $IFACE up =C2=A0down ip link set $IFACE down --- - Creating the Hypervisor OVS Bridges: ovs-vsctl add-br ovsbr0 ovs-vsctl add-br ovsbr1 ovs-vsctl add-br ovsbr2 - Attaching the bridges to the NICs: ovs-vsctl add-port ovsbr0 eth0 ovs-vsctl add-port ovsbr1 eth1 ovs-vsctl add-port ovsbr2 eth2 - Creating the OVS internal tagged interface (best practice?), so the QEmu Hypervisor itself can have its own IP (v4 and v6): ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=3D100 -- set interface ovsbr1p1 ty= pe=3Dinternal ovs-vsctl set interface ovsbr1p1 mac=3D\"32:ac:85:72:ab:fe\" =C2=A0NOTE: =C2=A0* I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6 with SLAAC, so, it remain fixed across host reboots. - Making Libvirt aware of OVS Bridges: Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml and ovsbr2.xml): --- ovsbr0.xml contents: =C2=A0ovsbr0 =C2=A0 =C2=A0 =C2=A0 --- --- ovsbr1.xml contents: =C2=A0ovsbr1 =C2=A0 =C2=A0 =C2=A0 --- --- ovsbr2.xml contents: =C2=A0ovsbr2 =C2=A0 =C2=A0 =C2=A0 --- Run: virsh net-define ovsbr0.xml virsh net-define ovsbr1.xml virsh net-define ovsbr2.xml virsh net-autostart ovsbr0 virsh net-autostart ovsbr1 virsh net-autostart ovsbr2 virsh net-start ovsbr0 virsh net-start ovsbr1 virsh net-start ovsbr2 - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtual Machine): 1- VM Configuration file (network-only / cutted): --- =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 --- 2- Configure "guest-fw-1.domain.com" (the router / firewall guest) /etc/network/interfaces file like this: --- auto vlan10 iface vlan10 inet static =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0vlan_raw_device eth0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 200.2.1.106 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 29 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0gateway 200.2.1.105 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0dns-nameserver 8.8.8.8 auto vlan100 iface vlan100 inet6 static =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0vlan_raw_device eth1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 2001:129X:2XX:810= X::2 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 64 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0dns-nameserver 2001:4860:= 4860::8844 2001:4860:4860::8888 iface vlan100 inet static =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0vlan_raw_device eth1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 192.168.4.1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 24 auto vlan200 iface vlan200 inet6 static =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0vlan_raw_device eth2 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 2001:1291:2de:10:= :1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 64 iface vlan200 inet static =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0vlan_raw_device eth2 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 172.16.0.1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 24 --- 3- Enable radvd for your LANs: --- # SERVERS interface vlan100 { =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0AdvSendAdvert on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0MinRtrAdvInterval 3; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0MaxRtrAdvInterval 10; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0AdvLinkMTU 1500; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0AdvDefaultPreference high; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0prefix 2001:1291:200:850a= ::/64 { =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0DeprecatePrefix on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0AdvOnLink on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0AdvAutonomous on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0AdvRouterAddr on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0}; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0route ::/0 { =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0RemoveRoute on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0}; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0RDNSS 2001:4860:4860::884= 4 2001:4860:4860::8888 { }; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0DNSSL domain.com.br { }; }; # DESKTOPS interface vlan200 { =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0AdvSendAdvert on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0MinRtrAdvInterval 3; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0MaxRtrAdvInterval 10; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0AdvLinkMTU 1500; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0AdvDefaultPreference high; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0prefix 2001:1291:2de:10::= /64 { =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0DeprecatePrefix on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0AdvOnLink on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0AdvAutonomous on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0AdvRouterAddr on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0}; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0route ::/0 { =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0RemoveRoute on; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0}; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0RDNSS 2001:4860:4860::884= 4 2001:4860:4860::8888 { }; =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0DNSSL igcorp.com.br { }; }; --- 4- HIT TUE BUG! =C2=A0Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will not work! Ping works... TCP connections doesn't. =C2=A0The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu 2.0 Virtual Machine hosted on itself, the guest `guest- fw-1.domain.com`. Details: --- root@qemu-host-1:~# ip r default via 192.168.4.1 dev ovsbr1p1 192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 root@qemu-host-1:~# ip -6 r | grep ovsbr1p1 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 8= 6397sec fe80::/64 dev ovsbr1p1 proto kernel metric 256 default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 = expires 27sec # ping6 okay... root@qemu-host-1:~# ping6 google.com -c1 PING google.com(2800:3f0:4001:815::1007) 56 data bytes 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=3D1 ttl=3D55 time=3D44.5 = ms --- google.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev =3D 44.579/44.579/44.579/0.000 ms # traceroute6 okay... root@qemu-host-1:~# traceroute6 google.com traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850= a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets =C2=A01 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 m= s 0.223 ms =C2=A02 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 2= 0.738 ms 20.902 ms =C2=A03 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 = ms =C2=A04 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms= 141.706 ms 21.058 ms =C2=A05 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069= ms 20.837 ms 20.903 ms =C2=A06 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms = 24.464 ms 24.649 ms =C2=A07 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.7= 34 ms 24.525 ms 25.273 ms =C2=A08 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.= 335 ms =C2=A09 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms = 36.375 ms 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms = 30.769 ms # NOTE: the first hope is the "guest-fw-1". # "apt-get update", not okay! *BUG* root@qemu-host-1:~# apt-get update 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting = to sec # it remains "Waiting for headers" forever... # While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1` shows: http://pastebin.com/2BUiNEfQ --- =C2=A0(OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!) =C2=A0Possible workarounds: is this an OpenvSwitch BUG? Lets try it with `bridge-utils` instead... =C2=A0* Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`, instead of openvswitch-switch. ------------------------ 1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch: - Reconfigure `qemu-host-1`s /etc/network/interfaces file with: --- auto br0 iface br0 inet manual =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_ports eth0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_maxwait 5 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_fd 1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_stp on auto br1 iface br1 inet manual =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_ports eth1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_maxwait 5 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_fd 1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_stp on auto vlan100 iface vlan100 inet6 auto =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0vlan_raw_device br1 iface vlan100 inet static =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0vlan_raw_device br1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 192.168.1.10 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 24 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0gateway 192.168.1.1 auto br2 iface br2 inet manual =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_ports eth2 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_maxwait 5 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_fd 1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bridge_stp on --- 2- New VM Configuration file (network-only section / cutted), adjusted to make use bridges from `bridge-utils` package: --- =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 --- * Start `guest-fw-1` as-is: =C2=A0virsh start guest-fw-1 New try: --- root@qemu-host-1:~# ip r default via 192.168.4.1 dev vlan100 192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 root@qemu-host-1:~# ip -6 r | grep vlan100 2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires 86= 397sec fe80::/64 dev vlan100 proto kernel metric 256 default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 ex= pires 27sec # ping6 okay... root@qemu-host-1:~# ping6 google.com -c1 PING google.com(2800:3f0:4001:815::1007) 56 data bytes 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=3D1 ttl=3D55 time=3D44.5 = ms --- google.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev =3D 44.579/44.579/44.579/0.000 ms # traceroute6 okay... root@qemu-host-1:~# traceroute6 google.com traceroute to google.com (2800:3f0:4001:815::1007) from 2001:1291:200:850= a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets =C2=A01 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 m= s 0.223 ms =C2=A02 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms 2= 0.738 ms 20.902 ms =C2=A03 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 = ms =C2=A04 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms= 141.706 ms 21.058 ms =C2=A05 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069= ms 20.837 ms 20.903 ms =C2=A06 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms = 24.464 ms 24.649 ms =C2=A07 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.7= 34 ms 24.525 ms 25.273 ms =C2=A08 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.= 335 ms =C2=A09 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms = 36.375 ms 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 ms 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms = 30.769 ms # BUG effect! "apt-get update", not okay! root@qemu-host-1:~# apt-get update 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting = to sec # it remains "Waiting for headers" forever... - So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges, falling back to OpenvSwitch as we started. ** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch. -- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf): --- name =3D "guest-fw-1" uuid =3D "17e031c7-1264-4979-8f06-c5e016469474" bootloader =3D "pygrub" memory =3D 2048 vcpus =3D 2 vif =3D [ 'bridge=3Dovsbr0', 'bridge=3Dovsbr1', 'bridge=3Dovsbr2', 'bridge=3Dovsbr3', 'bridge=3Dovsbr4', 'bridge=3Dovsbr5' ] disk =3D [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw'= ] --- Details - Working as expected when with Xen!! Look: --- root@qemu-host-1:~# ping6 -c 1 google.com PING google.com(2800:3f0:4001:815::1002) 56 data bytes 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=3D1 ttl=3D55 time=3D37.5 = ms root@qemu-host-1:~# ip -6 r | grep ovsbr1p1 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 8= 6394sec fe80::/64 dev ovsbr1p1 proto kernel metric 256 default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 = expires 24sec # *BUG dissapeared!* root@qemu-host-1:~# apt-get update Ign http://us.archive.ubuntu.com trusty InRelease Ign http://security.ubuntu.com trusty-security InRelease Ign http://us.archive.ubuntu.com trusty-proposed InRelease Ign http://us.archive.ubuntu.com trusty-updates InRelease Ign http://us.archive.ubuntu.com trusty-backports InRelease Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B] Hit http://us.archive.ubuntu.com trusty Release.gpg Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB] ........................ Ign http://us.archive.ubuntu.com trusty/main Translation-en_US Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US Fetched 1,011 kB in 19s (50.7 kB/s) Reading package lists... Done --- Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as expected! You guys can see that the `guest-fw-1` is working on top of Xen, as-is, I mean, the changes happened only at the Hypervisor itself, problem solved (not for QEmu)! But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the host `qemu-host-1` cannot browse the web again (`apt-get update` will not work if its gateway is a QEmu guest). =C2=A0** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake bridges". =C2=A0The presented workaround have one big downside, while it allows us = to keep using QEmu (and KSM), it requires a complete reconfiguration of the `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create a fake bridge, a new VirtIO NIC for your guest (this might add a bit of overhead for your hypervisor as a whole, I'm not sure), plus a lot of extra work... If you need to add a new VLAN to your `guest- fw-1`, you'll need to reboot it, to add a new VirtIO NIC (this isn't the best way to build hypervisors - not the best practice), this is just a real workaround that allows you to keep using QEmu (and benefits from KSM, Libvirt and etc)... =C2=A0While, when replacing QEmu by Xen, you don't need to change a single line within the guest itself... =C2=A0So, this network problem lies within the QEmu Virtual Machine! =C2=A0Doing this workaround: 1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it: ovs-vsctl add-br vlan10 ovsbr0 10 ovs-vsctl add-br vlan100 ovsbr1 100 ovs-vsctl add-br vlan200 ovsbr2 100 2- Making Libvirt aware of OVS Fake Bridges: Create 3 files, one for each fake bridge, like this (vlan10.xml, vlan100.xml and vlan200.xml): --- vlan10.xml contents: =C2=A0vlan10 =C2=A0 =C2=A0 =C2=A0 --- --- vlan100.xml contents: =C2=A0vlan100 =C2=A0 =C2=A0 =C2=A0 --- --- vlan200.xml contents: =C2=A0vlan200 =C2=A0 =C2=A0 =C2=A0 --- Run: virsh net-define vlan10.xml virsh net-define vlan100.xml virsh net-define vlan200.xml virsh net-autostart vlan10 virsh net-autostart vlan100 virsh net-autostart vlan200 virsh net-start vlan10 virsh net-start vlan100 virsh net-start vlan200 3- Reconfigure the `guest-fw-1` to make use of new "fake bridges": --- =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 --- 4- Reconfigure `guest-gw-1`s /etc/network/interfaces file: --- auto eth0 iface eth0 inet static # vlan_raw_device eth0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 200.2.1.106 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 29 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0gateway 200.2.1.105 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0dns-nameserver 8.8.8.8 auto eth1 iface eth1 inet6 static # vlan_raw_device eth1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 2001:129X:2XX:810= X::2 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 64 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0dns-nameserver 2001:4860:= 4860::8844 2001:4860:4860::8888 iface eth1 inet static # vlan_raw_device eth1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 192.168.4.1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 24 auto eth2 iface eth2 inet6 static # vlan_raw_device eth2 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 2001:1291:2de:10:= :1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 64 iface eth2 inet static # vlan_raw_device eth2 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0address 172.16.0.1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0netmask 24 --- 5- Details: Working as expected when with QEmu but, without tagging the VLAN within the `guest-fw-1` itself. --- root@qemu-host-1:~# ping6 -c 1 google.com PING google.com(2800:3f0:4001:815::1002) 56 data bytes 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=3D1 ttl=3D55 time=3D37.5 = ms root@qemu-host-1:~# ip -6 r | grep ovsbr1p1 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires 8= 6394sec fe80::/64 dev ovsbr1p1 proto kernel metric 256. default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 = expires 24sec # *BUG dissapeared!* root@qemu-host-1:~# apt-get update Ign http://us.archive.ubuntu.com trusty InRelease Ign http://security.ubuntu.com trusty-security InRelease Ign http://us.archive.ubuntu.com trusty-proposed InRelease Ign http://us.archive.ubuntu.com trusty-updates InRelease Ign http://us.archive.ubuntu.com trusty-backports InRelease Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B] Hit http://us.archive.ubuntu.com trusty Release.gpg Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB] ........................ Ign http://us.archive.ubuntu.com trusty/main Translation-en_US Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US Fetched 1,011 kB in 19s (50.7 kB/s) Reading package lists... Done --- Conclusion: A QEmu guest router does not route tagged VLAN packages that are originated at its host, neighter from others guests hosted at the same hypervisor. Making it impossible to create a virtual network within a hypervisor. Best Regards, Thiago Martins To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1362755/+subscriptions