From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60249) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XfgT0-00022H-Oe for qemu-devel@nongnu.org; Sat, 18 Oct 2014 22:43:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1XfgSt-0002gn-8V for qemu-devel@nongnu.org; Sat, 18 Oct 2014 22:43:34 -0400 Received: from mail-wi0-x22e.google.com ([2a00:1450:400c:c05::22e]:56192) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1XfgSs-0002gf-JG for qemu-devel@nongnu.org; Sat, 18 Oct 2014 22:43:27 -0400 Received: by mail-wi0-f174.google.com with SMTP id h11so5163686wiw.1 for ; Sat, 18 Oct 2014 19:43:25 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <54008C36.3070309@redhat.com> References: <20140828184002.18278.30739.malonedeb@soybean.canonical.com> <54008C36.3070309@redhat.com> From: =?UTF-8?B?TWFydGlueCAtIOOCuOOCp+ODvOODoOOCug==?= Date: Sat, 18 Oct 2014 23:42:54 -0300 Message-ID: Content-Type: multipart/alternative; boundary=047d7bfcf2f8e89b9d0505bd8f8d Subject: Re: [Qemu-devel] [Bug 1362755] [NEW] QEmu +2 does not route VLAN tagged packets, that are originated within the Hypervisor itself. List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: vyasevic@redhat.com Cc: qemu-devel@nongnu.org, 1362755@bugs.launchpad.net --047d7bfcf2f8e89b9d0505bd8f8d Content-Type: text/plain; charset=UTF-8 Hey Vlad, Sorry about this delay... I tried the following commands within the guest (that router/firewall " guest-fw-1.domain.com"): root@guest-fw-1:~# ethtool -K eth1 tso off root@guest-fw-1:~# ethtool -K eth1 gso off root@guest-fw-1:~# ethtool -K eth1 tx off But it did not fixed the problem... Then, I tried (it was still enabled for that "vlan device", just to test): root@guest-fw-1:~# ethtool -K vlan100 gso off Didn't worked either... --- root@guest-fw-1:~# grep vlan100 /proc/net/vlan/config vlan100 | 100 | eth1 --- Also, at the KVM host, its ovsbr1 (OpenvSwitch bridge), is attached to eth1, then, I tried to turn `tso/gso/tx off` there too but, no, same bad results. I tried to disable tso/gso/tx at another guest, of vlan100 net, didn't worked either. Regards, Thiago On 29 August 2014 11:20, Vlad Yasevich wrote: > [ realized that the bug and reporter were non cc'd, updated cc list] > > On 08/28/2014 02:40 PM, Thiago Martins wrote: > > Public bug reported: > > > > Guys, > > > > Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual network. > > It does not route tagged VLAN packets. > > > > The have a been a bunch of rather recent changes to the kernel to support > guest VLANs correctly. The issues have been around TSO/GSO implementation > in the kernel. > > Could try disabling TSO/GSO and tx checksums on the vlan devices in the > guest > and see if it solves your problem? > > If it does, could you try the kernel from > git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git > turn the offloads back on and see if the problem is solved? > > Thanks > -vlad > > > That's it, it is impossible to use Trusty acting as a QEmu 2.0 > > Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual > > tagged network within itself. QEmu 2.X guest does not route traffic when > > with tagged VLANs! > > > > So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as > > "firewalls / routers", and it have an easy to reproduce, connectivity > > problem. > > > > This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic) > > with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1). > > > > I have this very same setup up and running, on about ~100 physical > > servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of them, > > the QEmu Hypervisors dedicated to host "guest acting as routers / > > firewalls", like a "borger gateway" for example, that it does not work > > as expected. > > > > One interesting thing to note is that, this BUG appear only, and only > > at, the QEmu Hypervisors dedicated to host guests that are used as > > `router / firewalls` (as I said above), others QEmu Hypervisors of my > > network does not suffer from this problem. > > > > Another interesting point is that it fails to route tagged VLAN packets > > only when these packets are originated from within the Hypervisor > > itself, I mean, packets from both host and other guests (not the > > router/firewall guest itself), suffer from this connectivity problem. > > > > As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a > > "border hypervisor". So, this proves that there is something wrong with > > QEmu. > > > > I already tested it with both `openvswitch-switch` and with `bridge- > > utils`, same bad results. So, don't waste your time trying `bridge- > > utils` (optional steps while reproducing it), you can keep OVS bridges > > from original design. > > > > I think that I'm using the best pratices to build this environment, as > > follows... > > > > > > * Topology * > > > > > > QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border > hypervisor"): > > > > 1- Physical machine with 3 NICs; > > 2- Minimal Ubuntu 14.04.1 installed and upgraded; > > 3- Packages installed: "ubuntu-virt-server openvswitch-switch rdnssd > tcpdump". > > > > - eth0 connected to the Internet - VLAN tag 10; > > - eth1 connected to the LAN1 - VLAN tag 100; > > - eth2 connected to the LAN2 - VLAN tag 200; > > > > > > Guest (guest-fw-1.domain.com - the "border gateway" itself - regular > guest acting as a router with iptables/ip6tables): > > > > 1- Virtual Machine with 3 NICs (VirtIO); > > 2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded; > > 3- Packages installed: "aiccu iptables vlan pv-grub-menu". > > > > > > OBS: You'll need `virt-manager` to connect at `qemu-host-1` to install > > `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your > > (virt-)lab network, including the `qemu-host-1` itself. > > > > > > Steps to reproduce > > > > > > * Preparing the `qemu-host-1` host: > > > > - Configure the /etc/network/interfaces with: > > > > --- > > # The loopback network interface > > auto lo > > iface lo inet loopback > > > > auto eth0 > > iface eth0 inet manual > > up ip link set $IFACE up > > down ip link set $IFACE down > > > > auto eth1 > > iface eth1 inet manual > > up ip link set dev $IFACE up > > down ip link set dev $IFACE down > > > > auto ovsbr1p1 > > iface ovsbr1p1 inet6 auto > > > > iface ovsbr1p1 inet static > > address 192.168.1.10 > > netmask 24 > > gateway 192.168.1.1 > > > > auto eth2 > > iface eth2 inet manual > > up ip link set $IFACE up > > down ip link set $IFACE down > > --- > > > > > > - Creating the Hypervisor OVS Bridges: > > > > ovs-vsctl add-br ovsbr0 > > ovs-vsctl add-br ovsbr1 > > ovs-vsctl add-br ovsbr2 > > > > > > - Attaching the bridges to the NICs: > > > > ovs-vsctl add-port ovsbr0 eth0 > > ovs-vsctl add-port ovsbr1 eth1 > > ovs-vsctl add-port ovsbr2 eth2 > > > > > > - Creating the OVS internal tagged interface (best practice?), so the > QEmu Hypervisor itself can have its own IP (v4 and v6): > > > > ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=100 -- set interface ovsbr1p1 > type=internal > > ovs-vsctl set interface ovsbr1p1 mac=\"32:ac:85:72:ab:fe\" > > > > > > NOTE: > > > > * I'm fixing the MAC Address of ovsbr1p1 because I like to use IPv6 > > with SLAAC, so, it remain fixed across host reboots. > > > > > > - Making Libvirt aware of OVS Bridges: > > > > Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml > > and ovsbr2.xml): > > > > --- ovsbr0.xml contents: > > > > ovsbr0 > > > > > > > > > > --- > > > > --- ovsbr1.xml contents: > > > > ovsbr1 > > > > > > > > > > --- > > > > --- ovsbr2.xml contents: > > > > ovsbr2 > > > > > > > > > > --- > > > > > > Run: > > > > virsh net-define ovsbr0.xml > > virsh net-define ovsbr1.xml > > virsh net-define ovsbr2.xml > > > > virsh net-autostart ovsbr0 > > virsh net-autostart ovsbr1 > > virsh net-autostart ovsbr2 > > > > virsh net-start ovsbr0 > > virsh net-start ovsbr1 > > virsh net-start ovsbr2 > > > > > > - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum > Virtual Machine): > > > > 1- VM Configuration file (network-only / cutted): > > > > --- > > > > > > > > > >
function='0x0'/> > > > > > > > > > > > >
function='0x0'/> > > > > > > > > > > > >
function='0x0'/> > > > > --- > > > > 2- Configure "guest-fw-1.domain.com" (the router / firewall guest) > > /etc/network/interfaces file like this: > > > > --- > > auto vlan10 > > iface vlan10 inet static > > vlan_raw_device eth0 > > address 200.2.1.106 > > netmask 29 > > gateway 200.2.1.105 > > dns-nameserver 8.8.8.8 > > > > auto vlan100 > > iface vlan100 inet6 static > > vlan_raw_device eth1 > > address 2001:129X:2XX:810X::2 > > netmask 64 > > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888 > > > > iface vlan100 inet static > > vlan_raw_device eth1 > > address 192.168.4.1 > > netmask 24 > > > > auto vlan200 > > iface vlan200 inet6 static > > vlan_raw_device eth2 > > address 2001:1291:2de:10::1 > > netmask 64 > > > > iface vlan200 inet static > > vlan_raw_device eth2 > > address 172.16.0.1 > > netmask 24 > > --- > > > > 3- Enable radvd for your LANs: > > > > --- > > # SERVERS > > interface vlan100 { > > AdvSendAdvert on; > > MinRtrAdvInterval 3; > > MaxRtrAdvInterval 10; > > AdvLinkMTU 1500; > > AdvDefaultPreference high; > > prefix 2001:1291:200:850a::/64 { > > DeprecatePrefix on; > > AdvOnLink on; > > AdvAutonomous on; > > AdvRouterAddr on; > > }; > > route ::/0 { > > RemoveRoute on; > > }; > > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { }; > > DNSSL domain.com.br { }; > > }; > > # DESKTOPS > > interface vlan200 { > > AdvSendAdvert on; > > MinRtrAdvInterval 3; > > MaxRtrAdvInterval 10; > > AdvLinkMTU 1500; > > AdvDefaultPreference high; > > prefix 2001:1291:2de:10::/64 { > > DeprecatePrefix on; > > AdvOnLink on; > > AdvAutonomous on; > > AdvRouterAddr on; > > }; > > route ::/0 { > > RemoveRoute on; > > }; > > RDNSS 2001:4860:4860::8844 2001:4860:4860::8888 { }; > > DNSSL igcorp.com.br { }; > > }; > > --- > > > > 4- HIT TUE BUG! > > > > Go to `qemu-host-1.domain.com` and try to run "apt-get update", it will > > not work! Ping works... TCP connections doesn't. > > > > The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu > > 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`. > > > > Details: > > > > --- > > root@qemu-host-1:~# ip r > > default via 192.168.4.1 dev ovsbr1p1 > > 192.168.4.0/24 dev ovsbr1p1 proto kernel scope link src 192.168.4.2 > > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 > > > > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1 > > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires > 86397sec > > fe80::/64 dev ovsbr1p1 proto kernel metric 256 > > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 > expires 27sec > > > > > > # ping6 okay... > > root@qemu-host-1:~# ping6 google.com -c1 > > PING google.com(2800:3f0:4001:815::1007) 56 data bytes > > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms > > > > --- google.com ping statistics --- > > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms > > > > # traceroute6 okay... > > root@qemu-host-1:~# traceroute6 google.com > > traceroute to google.com (2800:3f0:4001:815::1007) from > 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets > > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms > 0.223 ms > > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms > 20.738 ms 20.902 ms > > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms > > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms > 141.706 ms 21.058 ms > > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 > ms 20.837 ms 20.903 ms > > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms > 24.464 ms 24.649 ms > > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 > ms 24.525 ms 25.273 ms > > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 > ms > > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms > 36.375 ms > > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 > ms > > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms > 30.769 ms > > > > # NOTE: the second hope is the "guest-fw-1". > > > > # "apt-get update", not okay! *BUG* > > > > root@qemu-host-1:~# apt-get update > > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] > [Connecting to sec > > > > # it remains "Waiting for headers" forever... > > > > # While waiting for "apt-get update" above, `tcpdump -ni ovsbr1p1` > > shows: > > > > http://pastebin.com/2BUiNEfQ > > > > --- > > > > > > (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not fix it!) > > > > Possible workarounds: is this an OpenvSwitch BUG? Lets try it with > > `bridge-utils` instead... > > > > * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`, > > instead of openvswitch-switch. > > > > > > ------------------------ > > > > 1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch: > > > > - Reconfigure `qemu-host-1`s /etc/network/interfaces file with: > > > > --- > > auto br0 > > iface br0 inet manual > > bridge_ports eth0 > > bridge_maxwait 5 > > bridge_fd 1 > > bridge_stp on > > > > auto br1 > > iface br1 inet manual > > bridge_ports eth1 > > bridge_maxwait 5 > > bridge_fd 1 > > bridge_stp on > > > > auto vlan100 > > iface vlan100 inet6 auto > > vlan_raw_device br1 > > > > iface vlan100 inet static > > vlan_raw_device br1 > > address 192.168.1.10 > > netmask 24 > > gateway 192.168.1.1 > > > > auto br2 > > iface br2 inet manual > > bridge_ports eth2 > > bridge_maxwait 5 > > bridge_fd 1 > > bridge_stp on > > --- > > > > > > 2- New VM Configuration file (network-only section / cutted), adjusted > > to make use bridges from `bridge-utils` package: > > > > --- > > > > > > > > > >
function='0x0'/> > > > > > > > > > > > >
function='0x0'/> > > > > > > > > > > > >
function='0x0'/> > > > > --- > > > > * Start `guest-fw-1` as-is: > > > > virsh start guest-fw-1 > > > > > > New try: > > > > --- > > root@qemu-host-1:~# ip r > > default via 192.168.4.1 dev vlan100 > > 192.168.4.0/24 dev vlan100 proto kernel scope link src 192.168.4.2 > > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 > > > > root@qemu-host-1:~# ip -6 r | grep vlan100 > > 2001:1291:200:850a::/64 dev vlan100 proto kernel metric 256 expires > 86397sec > > fe80::/64 dev vlan100 proto kernel metric 256 > > default via fe80::5054:ff:feb5:7744 dev vla100 proto ra metric 1024 > expires 27sec > > > > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1 > > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires > 86394sec > > fe80::/64 dev ovsbr1p1 proto kernel metric 256 > > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 > expires 24sec > > > > # ping6 okay... > > root@qemu-host-1:~# ping6 google.com -c1 > > PING google.com(2800:3f0:4001:815::1007) 56 data bytes > > 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=1 ttl=55 time=44.5 ms > > > > --- google.com ping statistics --- > > 1 packets transmitted, 1 received, 0% packet loss, time 0ms > > rtt min/avg/max/mdev = 44.579/44.579/44.579/0.000 ms > > > > # traceroute6 okay... > > root@qemu-host-1:~# traceroute6 google.com > > traceroute to google.com (2800:3f0:4001:815::1007) from > 2001:1291:200:850a:1054:3d86:369:d4b2, 30 hops max, 24 byte packets > > 1 2001:1291:200:850a::2 (2001:1291:200:850a::2) 0.394 ms 0.261 ms > 0.223 ms > > 2 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1) 21.536 ms > 20.738 ms 20.902 ms > > 3 brudi01.sixxs.net (2001:1291:2::b) 20.684 ms 20.74 ms 20.846 ms > > 4 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a) 197.392 ms > 141.706 ms 21.058 ms > > 5 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98::a) 21.069 > ms 20.837 ms 20.903 ms > > 6 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a) 24.564 ms > 24.464 ms 24.649 ms > > 7 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0:4b::b) 24.734 > ms 24.525 ms 25.273 ms > > 8 2001:1291:0:63::2 (2001:1291:0:63::2) 36.619 ms 36.245 ms 36.335 > ms > > 9 2001:4860::1:0:4f20 (2001:4860::1:0:4f20) 36.285 ms 41.017 ms > 36.375 ms > > 10 2001:4860:0:1::71 (2001:4860:0:1::71) 31.601 ms 31.623 ms 31.512 > ms > > 11 2800:3f0:4001:815::12 (2800:3f0:4001:815::12) 30.826 ms 30.683 ms > 30.769 ms > > > > > > # BUG effect! "apt-get update", not okay! > > > > root@qemu-host-1:~# apt-get update > > 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] > [Connecting to sec > > > > # it remains "Waiting for headers" forever... > > > > - So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges, > > falling back to OpenvSwitch as we started. > > > > > > ** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitch. > > > > > > -- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf): > > > > --- > > name = "guest-fw-1" > > > > uuid = "17e031c7-1264-4979-8f06-c5e016469474" > > > > bootloader = "pygrub" > > > > memory = 2048 > > > > vcpus = 2 > > > > vif = [ 'bridge=ovsbr0', 'bridge=ovsbr1', 'bridge=ovsbr2', > > 'bridge=ovsbr3', 'bridge=ovsbr4', 'bridge=ovsbr5' ] > > > > disk = [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,xvda,rw' ] > > --- > > > > Details - Working as expected when with Xen!! Look: > > > > --- > > root@qemu-host-1:~# ping6 -c 1 google.com > > PING google.com(2800:3f0:4001:815::1002) 56 data bytes > > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms > > > > > > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1 > > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires > 86394sec > > fe80::/64 dev ovsbr1p1 proto kernel metric 256 > > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 > expires 24sec > > > > # *BUG dissapeared!* > > > > root@qemu-host-1:~# apt-get update > > Ign http://us.archive.ubuntu.com trusty InRelease > > Ign http://security.ubuntu.com trusty-security InRelease > > Ign http://us.archive.ubuntu.com trusty-proposed InRelease > > Ign http://us.archive.ubuntu.com trusty-updates InRelease > > Ign http://us.archive.ubuntu.com trusty-backports InRelease > > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B] > > Hit http://us.archive.ubuntu.com trusty Release.gpg > > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB] > > ........................ > > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US > > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US > > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US > > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US > > Fetched 1,011 kB in 19s (50.7 kB/s) > > Reading package lists... Done > > --- > > > > > > Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as > expected! You guys can see that the `guest-fw-1` is working on top of Xen, > as-is, I mean, the changes happened only at the Hypervisor itself, problem > solved (not for QEmu)! > > > > But, QEmu still have a problem, if I remove Xen, back to QEmu, then, the > > host `qemu-host-1` cannot browse the web again (`apt-get update` will > > not work if its gateway is a QEmu guest). > > > > > > ** Workaround #3: Untagging the VLANs with OpenvSwitch and its "fake > bridges". > > > > The presented workaround have one big downside, while it allows us to > > keep using QEmu (and KSM), it requires a complete reconfiguration of the > > `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to create > > a fake bridge, a new VirtIO NIC for your guest (this might add a bit of > > overhead for your hypervisor as a whole, I'm not sure), plus a lot of > > extra work... If you need to add a new VLAN to your `guest-fw-1`, you'll > > need to reboot it, to add a new VirtIO NIC (this isn't the best way to > > build hypervisors - not the best practice), this is just a real > > workaround that allows you to keep using QEmu (and benefits from KSM, > > Libvirt and etc)... > > > > While, when replacing QEmu by Xen, you don't need to change a single > > line within the guest itself... > > > > So, this network problem lies within the QEmu Virtual Machine! > > > > Doing this workaround: > > > > 1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle it: > > > > ovs-vsctl add-br vlan10 ovsbr0 10 > > ovs-vsctl add-br vlan100 ovsbr1 100 > > ovs-vsctl add-br vlan200 ovsbr2 100 > > > > 2- Reconfigure the `guest-fw-1` to make use of new "fake bridges": > > > > --- > > > > > > > > > >
function='0x0'/> > > > > > > > > > > > >
function='0x0'/> > > > > > > > > > > > >
function='0x0'/> > > > > --- > > > > 3- Reconfigure `guest-gw-1`s /etc/network/interfaces file: > > > > --- > > auto eth0 > > iface eth0 inet static > > # vlan_raw_device eth0 > > address 200.2.1.106 > > netmask 29 > > gateway 200.2.1.105 > > dns-nameserver 8.8.8.8 > > > > auto eth1 > > iface eth1 inet6 static > > # vlan_raw_device eth1 > > address 2001:129X:2XX:810X::2 > > netmask 64 > > dns-nameserver 2001:4860:4860::8844 2001:4860:4860::8888 > > > > iface eth1 inet static > > # vlan_raw_device eth1 > > address 192.168.4.1 > > netmask 24 > > > > auto eth2 > > iface eth2 inet6 static > > # vlan_raw_device eth2 > > address 2001:1291:2de:10::1 > > netmask 64 > > > > iface eth2 inet static > > # vlan_raw_device eth2 > > address 172.16.0.1 > > netmask 24 > > --- > > > > 4- Details: Working as expected when with QEmu but, without tagging the > > VLAN within the `guest-fw-1` itself. > > > > --- > > root@qemu-host-1:~# ping6 -c 1 google.com > > PING google.com(2800:3f0:4001:815::1002) 56 data bytes > > 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=1 ttl=55 time=37.5 ms > > > > > > root@qemu-host-1:~# ip -6 r | grep ovsbr1p1 > > 2001:1291:200:850a::/64 dev ovsbr1p1 proto kernel metric 256 expires > 86394sec > > fe80::/64 dev ovsbr1p1 proto kernel metric 256. > > default via fe80::5054:ff:feb5:7744 dev ovsbr1p1 proto ra metric 1024 > expires 24sec > > > > # *BUG dissapeared!* > > > > root@qemu-host-1:~# apt-get update > > Ign http://us.archive.ubuntu.com trusty InRelease > > Ign http://security.ubuntu.com trusty-security InRelease > > Ign http://us.archive.ubuntu.com trusty-proposed InRelease > > Ign http://us.archive.ubuntu.com trusty-updates InRelease > > Ign http://us.archive.ubuntu.com trusty-backports InRelease > > Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B] > > Hit http://us.archive.ubuntu.com trusty Release.gpg > > Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB] > > ........................ > > Ign http://us.archive.ubuntu.com trusty/main Translation-en_US > > Ign http://us.archive.ubuntu.com trusty/multiverse Translation-en_US > > Ign http://us.archive.ubuntu.com trusty/restricted Translation-en_US > > Ign http://us.archive.ubuntu.com trusty/universe Translation-en_US > > Fetched 1,011 kB in 19s (50.7 kB/s) > > Reading package lists... Done > > --- > > > > Conclusion: > > > > A QEmu guest router does not route tagged VLAN packages that are > > originated at its host, neighter from others guests hosted at the same > > hypervisor. Making it impossible to create a virtual network within a > > hypervisor. > > > > Best Regards, > > Thiago Martins > > > > ** Affects: qemu > > Importance: Undecided > > Status: New > > > > > --047d7bfcf2f8e89b9d0505bd8f8d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hey Vlad,

Sorry about this delay...

I tried the following commands within the guest (that= router/firewall "guest-fw-1.= domain.com"):

root@guest-fw-1:~#=C2= =A0ethtool -K eth1 tso off
root@guest-fw-1:~#=C2=A0ethtool -K eth= 1 gso off
root@guest-fw-1:~#=C2=A0ethtool -K eth1 tx off

But it did not fixed the problem... Then, I trie= d (it was still enabled for that "vlan device", just to test):

root@guest-fw-1:~#=C2=A0ethtool -K vlan100 gso off

Didn't worked either...

---
root@guest-fw-1:~# grep vlan100 /proc/net/vlan/con= fig=C2=A0
vlan100 =C2=A0 =C2=A0 =C2=A0 =C2=A0| 100 =C2=A0| eth1
---

Also, at the KVM host, its ovsb= r1 (OpenvSwitch bridge), is attached to eth1, then, I tried to turn `tso/gs= o/tx off` there too but, no, same bad results.

I t= ried to disable tso/gso/tx at another guest, of vlan100 net, didn't wor= ked either.

Regards,
Thiago
<= div class=3D"gmail_extra">
On 29 August 2014 = 11:20, Vlad Yasevich <vyasevic@redhat.com> wrote:
[ realized that the bug and reporter were non cc&= #39;d, updated cc list]

On 08/28/2014 02:40 PM, Thiago Martins wrote:
> Public bug reported:
>
> Guys,
>
> Trusty QEmu 2.0 Hypervisor fails to create a consistent virtual networ= k.
> It does not route tagged VLAN packets.
>

The have a been a bunch of rather recent changes to the kernel to su= pport
guest VLANs correctly.=C2=A0 The issues have been around TSO/GSO implementa= tion
in the kernel.

Could try disabling TSO/GSO and tx checksums on the vlan devices in the gue= st
and see if it solves your problem?

If it does, could you try the kernel from
=C2=A0 git://git.kernel.org/pub/scm/linux/kernel/git/davem/= net.git
turn the offloads back on and see if the problem is solved?

Thanks
-vlad

> That's it, it is impossible to use Trusty acting as a QEmu 2.0
> Hypervisor (metapakage `ubuntu-virt-server`), to make a basic virtual<= br> > tagged network within itself. QEmu 2.X guest does not route traffic wh= en
> with tagged VLANs!
>
> So, Trusty QEmu 2.0 Hypervisor cannot be used to host guests acting as=
> "firewalls / routers", and it have an easy to reproduce, con= nectivity
> problem.
>
> This network problem affects Ubuntu 14.04.1 (Linux-3.13.0-35-generic)<= br> > with QEmu 2.0 (it also affects 14.10, Linux 3.16 - QEmu 2.1).
>
> I have this very same setup up and running, on about ~100 physical
> servers (others Trusty QEmu 2.0 Hypervisors), and in only a few of the= m,
> the QEmu Hypervisors dedicated to host "guest acting as routers /=
> firewalls", like a "borger gateway" for example, that i= t does not work
> as expected.
>
> One interesting thing to note is that, this BUG appear only, and only<= br> > at, the QEmu Hypervisors dedicated to host guests that are used as
> `router / firewalls` (as I said above), others QEmu Hypervisors of my<= br> > network does not suffer from this problem.
>
> Another interesting point is that it fails to route tagged VLAN packet= s
> only when these packets are originated from within the Hypervisor
> itself, I mean, packets from both host and other guests (not the
> router/firewall guest itself), suffer from this connectivity problem.<= br> >
> As a workaroung / fix, Xen-4.4 can be used, instead of QEmu 2.0, as a<= br> > "border hypervisor". So, this proves that there is something= wrong with
> QEmu.
>
> I already tested it with both `openvswitch-switch` and with `bridge- > utils`, same bad results. So, don't waste your time trying `bridge= -
> utils` (optional steps while reproducing it), you can keep OVS bridges=
> from original design.
>
> I think that I'm using the best pratices to build this environment= , as
> follows...
>
>
> * Topology *
>
>
> QEmu 2.0 Hypervisor - (qemu-host-1.domain.com - the "border hypervisor"= ):
>
> 1- Physical machine with 3 NICs;
> 2- Minimal Ubuntu 14.04.1 installed and upgraded;
> 3- Packages installed: "ubuntu-virt-server openvswitch-switch rdn= ssd tcpdump".
>
> - eth0 connected to the Internet - VLAN tag 10;
> - eth1 connected to the LAN1 - VLAN tag 100;
> - eth2 connected to the LAN2 - VLAN tag 200;
>
>
> Guest (gues= t-fw-1.domain.com - the "border gateway" itself - regular gue= st acting as a router with iptables/ip6tables):
>
> 1- Virtual Machine with 3 NICs (VirtIO);
> 2- Minimal Virtual Machine Ubuntu 14.04.1 installed and upgraded;
> 3- Packages installed: "aiccu iptables vlan pv-grub-menu". >
>
> OBS: You'll need `virt-manager` to connect at `qemu-host-1` to ins= tall
> `guest-fw-1`. Then, use `guest-fw-1` as a default gateway for your
> (virt-)lab network, including the `qemu-host-1` itself.
>
>
> Steps to reproduce
>
>
> * Preparing the `qemu-host-1` host:
>
> - Configure the /etc/network/interfaces with:
>
> ---
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> auto eth0
> iface eth0 inet manual
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0up ip link set $IFACE up
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0down ip link set $IFACE down
>
> auto eth1
> iface eth1 inet manual
>=C2=A0 =C2=A0 =C2=A0 =C2=A0up ip link set dev $IFACE up
>=C2=A0 =C2=A0 =C2=A0 =C2=A0down ip link set dev $IFACE down
>
> auto ovsbr1p1
> iface ovsbr1p1 inet6 auto
>
> iface ovsbr1p1 inet static
>=C2=A0 =C2=A0 =C2=A0 =C2=A0address 192.168.1.10
>=C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 24
>=C2=A0 =C2=A0 =C2=A0 =C2=A0gateway 192.168.1.1
>
> auto eth2
> iface eth2 inet manual
>=C2=A0 =C2=A0 =C2=A0 =C2=A0up ip link set $IFACE up
>=C2=A0 =C2=A0 =C2=A0 =C2=A0down ip link set $IFACE down
> ---
>
>
> - Creating the Hypervisor OVS Bridges:
>
> ovs-vsctl add-br ovsbr0
> ovs-vsctl add-br ovsbr1
> ovs-vsctl add-br ovsbr2
>
>
> - Attaching the bridges to the NICs:
>
> ovs-vsctl add-port ovsbr0 eth0
> ovs-vsctl add-port ovsbr1 eth1
> ovs-vsctl add-port ovsbr2 eth2
>
>
> - Creating the OVS internal tagged interface (best practice?), so the = QEmu Hypervisor itself can have its own IP (v4 and v6):
>
> ovs-vsctl add-port ovsbr1 ovsbr1p1 tag=3D100 -- set interface ovsbr1p1= type=3Dinternal
> ovs-vsctl set interface ovsbr1p1 mac=3D\"32:ac:85:72:ab:fe\"=
>
>
>=C2=A0 NOTE:
>
>=C2=A0 * I'm fixing the MAC Address of ovsbr1p1 because I like to u= se IPv6
> with SLAAC, so, it remain fixed across host reboots.
>
>
> - Making Libvirt aware of OVS Bridges:
>
> Create 3 files, one for each bridge, like this (ovsbr0.xml, ovsbr1.xml=
> and ovsbr2.xml):
>
> --- ovsbr0.xml contents:
> <network>
>=C2=A0 <name>ovsbr0</name>
>=C2=A0 <forward mode=3D'bridge'/>
>=C2=A0 <bridge name=3D'ovsbr0'/>
>=C2=A0 <virtualport type=3D'openvswitch'/>
> </network>
> ---
>
> --- ovsbr1.xml contents:
> <network>
>=C2=A0 <name>ovsbr1</name>
>=C2=A0 <forward mode=3D'bridge'/>
>=C2=A0 <bridge name=3D'ovsbr1'/>
>=C2=A0 <virtualport type=3D'openvswitch'/>
> </network>
> ---
>
> --- ovsbr2.xml contents:
> <network>
>=C2=A0 <name>ovsbr2</name>
>=C2=A0 <forward mode=3D'bridge'/>
>=C2=A0 <bridge name=3D'ovsbr2'/>
>=C2=A0 <virtualport type=3D'openvswitch'/>
> </network>
> ---
>
>
> Run:
>
> virsh net-define ovsbr0.xml
> virsh net-define ovsbr1.xml
> virsh net-define ovsbr2.xml
>
> virsh net-autostart ovsbr0
> virsh net-autostart ovsbr1
> virsh net-autostart ovsbr2
>
> virsh net-start ovsbr0
> virsh net-start ovsbr1
> virsh net-start ovsbr2
>
>
> - Creating the "guest-fw-1.domain.com" (Ubuntu 14.04.1 - Minimum Virtu= al Machine):
>
> 1- VM Configuration file (network-only / cutted):
>
> ---
>=C2=A0 =C2=A0 =C2=A0<interface type=3D'network'>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<mac address=3D'52:54:00:41:8c:3f'= ;/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<source network=3D'ovsbr0'/> >=C2=A0 =C2=A0 =C2=A0 =C2=A0<model type=3D'virtio'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<address type=3D'pci' domain=3D&#= 39;0x0000' bus=3D'0x00' slot=3D'0x03' function=3D'0= x0'/>
>=C2=A0 =C2=A0 =C2=A0</interface>
>=C2=A0 =C2=A0 =C2=A0<interface type=3D'network'>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<mac address=3D'52:54:00:27:b2:7d'= ;/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<source network=3D'ovsbr1'/> >=C2=A0 =C2=A0 =C2=A0 =C2=A0<model type=3D'virtio'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<address type=3D'pci' domain=3D&#= 39;0x0000' bus=3D'0x00' slot=3D'0x09' function=3D'0= x0'/>
>=C2=A0 =C2=A0 =C2=A0</interface>
>=C2=A0 =C2=A0 =C2=A0<interface type=3D'network'>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<mac address=3D'52:54:00:ff:35:5c'= ;/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<source network=3D'ovsbr2'/> >=C2=A0 =C2=A0 =C2=A0 =C2=A0<model type=3D'virtio'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<address type=3D'pci' domain=3D&#= 39;0x0000' bus=3D'0x00' slot=3D'0x0a' function=3D'0= x0'/>
>=C2=A0 =C2=A0 =C2=A0</interface>
> ---
>
> 2- Configure "guest-fw-1.domain.com" (the router / firewall guest)
> /etc/network/interfaces file like this:
>
> ---
> auto vlan10
> iface vlan10 inet static
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vlan_raw_device eth0
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 200.2.1.106
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 29
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0gateway 200.2.1.105
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0dns-nameserver 8.8.8.8
>
> auto vlan100
> iface vlan100 inet6 static
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vlan_raw_device eth1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 2001:129X:2XX:810X::2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 64
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0dns-nameserver 2001:4860:4860::8844 2= 001:4860:4860::8888
>
> iface vlan100 inet static
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vlan_raw_device eth1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 192.168.4.1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 24
>
> auto vlan200
> iface vlan200 inet6 static
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vlan_raw_device eth2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 2001:1291:2de:10::1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 64
>
> iface vlan200 inet static
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vlan_raw_device eth2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 172.16.0.1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 24
> ---
>
> 3- Enable radvd for your LANs:
>
> ---
> # SERVERS
> interface vlan100 {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvSendAdvert on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MinRtrAdvInterval 3;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MaxRtrAdvInterval 10;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvLinkMTU 1500;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvDefaultPreference high;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0prefix 2001:1291:200:850a::/64 {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Deprecate= Prefix on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvOnLink= on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvAutono= mous on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvRouter= Addr on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0};
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0route ::/0 {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RemoveRou= te on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0};
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RDNSS 2001:4860:4860::8844 2001:4860:= 4860::8888 { };
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0DNSSL domain.com.br { };
> };
> # DESKTOPS
> interface vlan200 {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvSendAdvert on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MinRtrAdvInterval 3;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0MaxRtrAdvInterval 10;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvLinkMTU 1500;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvDefaultPreference high;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0prefix 2001:1291:2de:10::/64 {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Deprecate= Prefix on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvOnLink= on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvAutono= mous on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0AdvRouter= Addr on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0};
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0route ::/0 {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RemoveRou= te on;
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0};
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RDNSS 2001:4860:4860::8844 2001:4860:= 4860::8888 { };
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0DNSSL igcorp.com.br { };
> };
> ---
>
> 4- HIT TUE BUG!
>
>=C2=A0 Go to `qemu-host-1.domain.com` and try to run "apt-get update", i= t will
> not work! Ping works... TCP connections doesn't.
>
>=C2=A0 The gateway of `qemu-host-1.domain.com` (through ovsbr1p1), is the QEmu
> 2.0 Virtual Machine hosted on itself, the guest `guest-fw-1.domain.com`.
>
> Details:
>
> ---
> root@qemu-host-1:~# ip r
> default via 192.168.4.1 dev ovsbr1p1
> 192.168.4.0/24= dev ovsbr1p1=C2=A0 proto kernel=C2=A0 scope link=C2=A0 src 192.168.4.2
> 192.168.122.0/24= dev virbr0=C2=A0 proto kernel=C2=A0 scope link=C2=A0 src 192.168.122.1
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1=C2=A0 proto kernel=C2=A0 metric 2= 56=C2=A0 expires 86397sec
> fe80::/64 dev ovsbr1p1=C2=A0 proto kernel=C2=A0 metric 256
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1=C2=A0 proto ra=C2=A0 = metric 1024=C2=A0 expires 27sec
>
>
> # ping6 okay...
> root@qemu-host-1:~# ping6 google.com -c1
> PING google.com(28= 00:3f0:4001:815::1007) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=3D1 ttl=3D55 time=3D44= .5 ms
>
> --- google.com pin= g statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev =3D 44.579/44.579/44.579/0.000 ms
>
> # traceroute6 okay...
> root@qemu-host-1:~# traceroute6 google.com
> traceroute to google.c= om (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2= , 30 hops max, 24 byte packets
>=C2=A0 1=C2=A0 2001:1291:200:850a::2 (2001:1291:200:850a::2)=C2=A0 0.39= 4 ms=C2=A0 0.261 ms=C2=A0 0.223 ms
>=C2=A0 2=C2=A0 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1)=C2=A0 21.53= 6 ms=C2=A0 20.738 ms=C2=A0 20.902 ms
>=C2=A0 3=C2=A0 b= rudi01.sixxs.net (2001:1291:2::b)=C2=A0 20.684 ms=C2=A0 20.74 ms=C2=A0 = 20.846 ms
>=C2=A0 4=C2=A0 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a)= =C2=A0 197.392 ms=C2=A0 141.706 ms=C2=A0 21.058 ms
>=C2=A0 5=C2=A0 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98:= :a)=C2=A0 21.069 ms=C2=A0 20.837 ms=C2=A0 20.903 ms
>=C2=A0 6=C2=A0 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a)=C2=A0 = 24.564 ms=C2=A0 24.464 ms=C2=A0 24.649 ms
>=C2=A0 7=C2=A0 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0= :4b::b)=C2=A0 24.734 ms=C2=A0 24.525 ms=C2=A0 25.273 ms
>=C2=A0 8=C2=A0 2001:1291:0:63::2 (2001:1291:0:63::2)=C2=A0 36.619 ms=C2= =A0 36.245 ms=C2=A0 36.335 ms
>=C2=A0 9=C2=A0 2001:4860::1:0:4f20 (2001:4860::1:0:4f20)=C2=A0 36.285 m= s=C2=A0 41.017 ms=C2=A0 36.375 ms
> 10=C2=A0 2001:4860:0:1::71 (2001:4860:0:1::71)=C2=A0 31.601 ms=C2=A0 3= 1.623 ms=C2=A0 31.512 ms
> 11=C2=A0 2800:3f0:4001:815::12 (2800:3f0:4001:815::12)=C2=A0 30.826 ms= =C2=A0 30.683 ms=C2=A0 30.769 ms
>
> # NOTE: the second hope is the "guest-fw-1".
>
> # "apt-get update", not okay! *BUG*
>
> root@qemu-host-1:~# apt-get update
> 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec >
> # it remains "Waiting for headers" forever...
>
> # While waiting for "apt-get update" above, `tcpdump -ni ovs= br1p1`
> shows:
>
> http://past= ebin.com/2BUiNEfQ
>
> ---
>
>
>=C2=A0 (OPTIONAL STEP - replace OpenvSwitch by bridge-utils - does not = fix it!)
>
>=C2=A0 Possible workarounds: is this an OpenvSwitch BUG? Lets try it wi= th
> `bridge-utils` instead...
>
>=C2=A0 * Reconfigure your "qemu-host-1.domain.com" to use `bridge-utils`= ,
> instead of openvswitch-switch.
>
>
> ------------------------
>
> 1- Preparing the host, now using `bridge-utils` instead of OpenvSwitch= :
>
> - Reconfigure `qemu-host-1`s /etc/network/interfaces file with:
>
> ---
> auto br0
> iface br0 inet manual
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_ports eth0
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_maxwait 5
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_fd 1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_stp on
>
> auto br1
> iface br1 inet manual
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_ports eth1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_maxwait 5
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_fd 1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_stp on
>
> auto vlan100
> iface vlan100 inet6 auto
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vlan_raw_device br1
>
> iface vlan100 inet static
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0vlan_raw_device br1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 192.168.1.10
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 24
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0gateway 192.168.1.1
>
> auto br2
> iface br2 inet manual
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_ports eth2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_maxwait 5
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_fd 1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bridge_stp on
> ---
>
>
> 2- New VM Configuration file (network-only section / cutted), adjusted=
> to make use bridges from `bridge-utils` package:
>
> ---
>=C2=A0 =C2=A0 =C2=A0<interface type=3D'bridge'>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<mac address=3D'52:54:00:41:8c:3f'= ;/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<source bridge=3D'br0'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<model type=3D'virtio'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<address type=3D'pci' domain=3D&#= 39;0x0000' bus=3D'0x00' slot=3D'0x03' function=3D'0= x0'/>
>=C2=A0 =C2=A0 =C2=A0</interface>
>=C2=A0 =C2=A0 =C2=A0<interface type=3D'bridge'>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<mac address=3D'52:54:00:27:b2:7d'= ;/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<source bridge=3D'br1'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<model type=3D'virtio'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<address type=3D'pci' domain=3D&#= 39;0x0000' bus=3D'0x00' slot=3D'0x09' function=3D'0= x0'/>
>=C2=A0 =C2=A0 =C2=A0</interface>
>=C2=A0 =C2=A0 =C2=A0<interface type=3D'bridge'>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<mac address=3D'52:54:00:ff:35:5c'= ;/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<source bridge=3D'br2'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<model type=3D'virtio'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<address type=3D'pci' domain=3D&#= 39;0x0000' bus=3D'0x00' slot=3D'0x0a' function=3D'0= x0'/>
>=C2=A0 =C2=A0 =C2=A0</interface>
> ---
>
> * Start `guest-fw-1` as-is:
>
>=C2=A0 virsh start guest-fw-1
>
>
> New try:
>
> ---
> root@qemu-host-1:~# ip r
> default via 192.168.4.1 dev vlan100
> 192.168.4.0/24= dev vlan100 proto kernel=C2=A0 scope link=C2=A0 src 192.168.4.2
> 192.168.122.0/24= dev virbr0=C2=A0 proto kernel=C2=A0 scope link=C2=A0 src 192.168.122.1
>
> root@qemu-host-1:~# ip -6 r | grep vlan100
> 2001:1291:200:850a::/64 dev vlan100=C2=A0 proto kernel=C2=A0 metric 25= 6=C2=A0 expires 86397sec
> fe80::/64 dev vlan100=C2=A0 proto kernel=C2=A0 metric 256
> default via fe80::5054:ff:feb5:7744 dev vla100=C2=A0 proto ra=C2=A0 me= tric 1024=C2=A0 expires 27sec
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1=C2=A0 proto kernel=C2=A0 metric 2= 56=C2=A0 expires 86394sec
> fe80::/64 dev ovsbr1p1=C2=A0 proto kernel=C2=A0 metric 256
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1=C2=A0 proto ra=C2=A0 = metric 1024=C2=A0 expires 24sec
>
> # ping6 okay...
> root@qemu-host-1:~# ping6 google.com -c1
> PING google.com(28= 00:3f0:4001:815::1007) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1007: icmp_seq=3D1 ttl=3D55 time=3D44= .5 ms
>
> --- google.com pin= g statistics ---
> 1 packets transmitted, 1 received, 0% packet loss, time 0ms
> rtt min/avg/max/mdev =3D 44.579/44.579/44.579/0.000 ms
>
> # traceroute6 okay...
> root@qemu-host-1:~# traceroute6 google.com
> traceroute to google.c= om (2800:3f0:4001:815::1007) from 2001:1291:200:850a:1054:3d86:369:d4b2= , 30 hops max, 24 byte packets
>=C2=A0 1=C2=A0 2001:1291:200:850a::2 (2001:1291:200:850a::2)=C2=A0 0.39= 4 ms=C2=A0 0.261 ms=C2=A0 0.223 ms
>=C2=A0 2=C2=A0 gw-1291.udi-01.br.sixxs.net (2001:1291:200:50a::1)=C2=A0 21.53= 6 ms=C2=A0 20.738 ms=C2=A0 20.902 ms
>=C2=A0 3=C2=A0 b= rudi01.sixxs.net (2001:1291:2::b)=C2=A0 20.684 ms=C2=A0 20.74 ms=C2=A0 = 20.846 ms
>=C2=A0 4=C2=A0 ge-0-2-0-71.seed.ula001.ctbc.com.br (2001:1291:2::a)= =C2=A0 197.392 ms=C2=A0 141.706 ms=C2=A0 21.058 ms
>=C2=A0 5=C2=A0 ge-5-2-0-0.core-d.ula001.ctbc.com.br (2001:1291:0:98:= :a)=C2=A0 21.069 ms=C2=A0 20.837 ms=C2=A0 20.903 ms
>=C2=A0 6=C2=A0 ae0-0.core-b.fac001.ctbc.com.br (2001:1291:0:d7::a)=C2=A0 = 24.564 ms=C2=A0 24.464 ms=C2=A0 24.649 ms
>=C2=A0 7=C2=A0 et-1-0-0-0.border-a.fac001.ctbc.com.br (2001:1291:0= :4b::b)=C2=A0 24.734 ms=C2=A0 24.525 ms=C2=A0 25.273 ms
>=C2=A0 8=C2=A0 2001:1291:0:63::2 (2001:1291:0:63::2)=C2=A0 36.619 ms=C2= =A0 36.245 ms=C2=A0 36.335 ms
>=C2=A0 9=C2=A0 2001:4860::1:0:4f20 (2001:4860::1:0:4f20)=C2=A0 36.285 m= s=C2=A0 41.017 ms=C2=A0 36.375 ms
> 10=C2=A0 2001:4860:0:1::71 (2001:4860:0:1::71)=C2=A0 31.601 ms=C2=A0 3= 1.623 ms=C2=A0 31.512 ms
> 11=C2=A0 2800:3f0:4001:815::12 (2800:3f0:4001:815::12)=C2=A0 30.826 ms= =C2=A0 30.683 ms=C2=A0 30.769 ms
>
>
> # BUG effect! "apt-get update", not okay!
>
> root@qemu-host-1:~# apt-get update
> 0% [Connecting to us.archive.ubuntu.com (2001:67c:1562::14)] [Connecting to sec >
> # it remains "Waiting for headers" forever...
>
> - So! It is not an OpenvSwitch BUG! Removing `bridge-utils` bridges, > falling back to OpenvSwitch as we started.
>
>
> ** Workaround #2: Use Xen-4.4 instead of QEmu 2.0 / Back to OpenvSwitc= h.
>
>
> -- VM conf (`guest-fw-1` needs to have /etc/init/hvc0.conf):
>
> ---
> name =3D "guest-fw-1"
>
> uuid =3D "17e031c7-1264-4979-8f06-c5e016469474"
>
> bootloader =3D "pygrub"
>
> memory =3D 2048
>
> vcpus =3D 2
>
> vif =3D [ 'bridge=3Dovsbr0', 'bridge=3Dovsbr1', 'b= ridge=3Dovsbr2',
> 'bridge=3Dovsbr3', 'bridge=3Dovsbr4', 'bridge=3Dov= sbr5' ]
>
> disk =3D [ 'tap:raw:/var/lib/libvirt/images/guest-fw-1-disk0.img,x= vda,rw' ]
> ---
>
> Details - Working as expected when with Xen!! Look:
>
> ---
> root@qemu-host-1:~# ping6 -c 1 google.com
> PING google.com(28= 00:3f0:4001:815::1002) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=3D1 ttl=3D55 time=3D37= .5 ms
>
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1=C2=A0 proto kernel=C2=A0 metric 2= 56=C2=A0 expires 86394sec
> fe80::/64 dev ovsbr1p1=C2=A0 proto kernel=C2=A0 metric 256
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1=C2=A0 proto ra=C2=A0 = metric 1024=C2=A0 expires 24sec
>
> # *BUG dissapeared!*
>
> root@qemu-host-1:~# apt-get update
> Ign http://= us.archive.ubuntu.com trusty InRelease
> Ign http://se= curity.ubuntu.com trusty-security InRelease
> Ign http://= us.archive.ubuntu.com trusty-proposed InRelease
> Ign http://= us.archive.ubuntu.com trusty-updates InRelease
> Ign http://= us.archive.ubuntu.com trusty-backports InRelease
> Get:1 http://= security.ubuntu.com trusty-security Release.gpg [933 B]
> Hit http://= us.archive.ubuntu.com trusty Release.gpg
> Get:2 http://= security.ubuntu.com trusty-security Release [59.7 kB]
> ........................
> Ign http://= us.archive.ubuntu.com trusty/main Translation-en_US
> Ign http://= us.archive.ubuntu.com trusty/multiverse Translation-en_US
> Ign http://= us.archive.ubuntu.com trusty/restricted Translation-en_US
> Ign http://= us.archive.ubuntu.com trusty/universe Translation-en_US
> Fetched 1,011 kB in 19s (50.7 kB/s)
> Reading package lists... Done
> ---
>
>
> Now, both Xen Dom0 (`qemu-host-1`) and DomU (`guest-fw-1`) works as ex= pected! You guys can see that the `guest-fw-1` is working on top of Xen, as= -is, I mean, the changes happened only at the Hypervisor itself, problem so= lved (not for QEmu)!
>
> But, QEmu still have a problem, if I remove Xen, back to QEmu, then, t= he
> host `qemu-host-1` cannot browse the web again (`apt-get update` will<= br> > not work if its gateway is a QEmu guest).
>
>
>=C2=A0 ** Workaround #3: Untagging the VLANs with OpenvSwitch and its &= quot;fake bridges".
>
>=C2=A0 The presented workaround have one big downside, while it allows = us to
> keep using QEmu (and KSM), it requires a complete reconfiguration of t= he
> `guest-fw-1` interfaces! Also, for each VLAN tag, you'll need to c= reate
> a fake bridge, a new VirtIO NIC for your guest (this might add a bit o= f
> overhead for your hypervisor as a whole, I'm not sure), plus a lot= of
> extra work... If you need to add a new VLAN to your `guest-fw-1`, you&= #39;ll
> need to reboot it, to add a new VirtIO NIC (this isn't the best wa= y to
> build hypervisors - not the best practice), this is just a real
> workaround that allows you to keep using QEmu (and benefits from KSM,<= br> > Libvirt and etc)...
>
>=C2=A0 While, when replacing QEmu by Xen, you don't need to change = a single
> line within the guest itself...
>
>=C2=A0 So, this network problem lies within the QEmu Virtual Machine! >
>=C2=A0 Doing this workaround:
>
> 1- Untagging the VLANs at OpenvSwitch, because QEmu can't handle i= t:
>
> ovs-vsctl add-br vlan10 ovsbr0 10
> ovs-vsctl add-br vlan100 ovsbr1 100
> ovs-vsctl add-br vlan200 ovsbr2 100
>
> 2- Reconfigure the `guest-fw-1` to make use of new "fake bridges&= quot;:
>
> ---
>=C2=A0 =C2=A0 =C2=A0<interface type=3D'network'>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<mac address=3D'52:54:00:41:8c:3f'= ;/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<source network=3D'vlan10'/> >=C2=A0 =C2=A0 =C2=A0 =C2=A0<model type=3D'virtio'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<address type=3D'pci' domain=3D&#= 39;0x0000' bus=3D'0x00' slot=3D'0x03' function=3D'0= x0'/>
>=C2=A0 =C2=A0 =C2=A0</interface>
>=C2=A0 =C2=A0 =C2=A0<interface type=3D'network'>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<mac address=3D'52:54:00:27:b2:7d'= ;/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<source network=3D'vlan100'/><= br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0<model type=3D'virtio'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<address type=3D'pci' domain=3D&#= 39;0x0000' bus=3D'0x00' slot=3D'0x09' function=3D'0= x0'/>
>=C2=A0 =C2=A0 =C2=A0</interface>
>=C2=A0 =C2=A0 =C2=A0<interface type=3D'network'>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<mac address=3D'52:54:00:ff:35:5c'= ;/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<source network=3D'vlan200'/><= br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0<model type=3D'virtio'/>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0<address type=3D'pci' domain=3D&#= 39;0x0000' bus=3D'0x00' slot=3D'0x0a' function=3D'0= x0'/>
>=C2=A0 =C2=A0 =C2=A0</interface>
> ---
>
> 3- Reconfigure `guest-gw-1`s /etc/network/interfaces file:
>
> ---
> auto eth0
> iface eth0 inet static
> #=C2=A0 =C2=A0 =C2=A0 =C2=A0 vlan_raw_device eth0
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 200.2.1.106
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 29
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0gateway 200.2.1.105
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0dns-nameserver 8.8.8.8
>
> auto eth1
> iface eth1 inet6 static
> #=C2=A0 =C2=A0 =C2=A0 =C2=A0 vlan_raw_device eth1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 2001:129X:2XX:810X::2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 64
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0dns-nameserver 2001:4860:4860::8844 2= 001:4860:4860::8888
>
> iface eth1 inet static
> #=C2=A0 =C2=A0 =C2=A0 =C2=A0 vlan_raw_device eth1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 192.168.4.1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 24
>
> auto eth2
> iface eth2 inet6 static
> #=C2=A0 =C2=A0 =C2=A0 =C2=A0 vlan_raw_device eth2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 2001:1291:2de:10::1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 64
>
> iface eth2 inet static
> #=C2=A0 =C2=A0 =C2=A0 =C2=A0 vlan_raw_device eth2
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0address 172.16.0.1
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0netmask 24
> ---
>
> 4- Details: Working as expected when with QEmu but, without tagging th= e
> VLAN within the `guest-fw-1` itself.
>
> ---
> root@qemu-host-1:~# ping6 -c 1 google.com
> PING google.com(28= 00:3f0:4001:815::1002) 56 data bytes
> 64 bytes from 2800:3f0:4001:815::1002: icmp_seq=3D1 ttl=3D55 time=3D37= .5 ms
>
>
> root@qemu-host-1:~# ip -6 r | grep ovsbr1p1
> 2001:1291:200:850a::/64 dev ovsbr1p1=C2=A0 proto kernel=C2=A0 metric 2= 56=C2=A0 expires 86394sec
> fe80::/64 dev ovsbr1p1=C2=A0 proto kernel=C2=A0 metric 256.
> default via fe80::5054:ff:feb5:7744 dev ovsbr1p1=C2=A0 proto ra=C2=A0 = metric 1024=C2=A0 expires 24sec
>
> # *BUG dissapeared!*
>
> root@qemu-host-1:~# apt-get update
> Ign http://= us.archive.ubuntu.com trusty InRelease
> Ign http://se= curity.ubuntu.com trusty-security InRelease
> Ign http://= us.archive.ubuntu.com trusty-proposed InRelease
> Ign http://= us.archive.ubuntu.com trusty-updates InRelease
> Ign http://= us.archive.ubuntu.com trusty-backports InRelease
> Get:1 http://= security.ubuntu.com trusty-security Release.gpg [933 B]
> Hit http://= us.archive.ubuntu.com trusty Release.gpg
> Get:2 http://= security.ubuntu.com trusty-security Release [59.7 kB]
> ........................
> Ign http://= us.archive.ubuntu.com trusty/main Translation-en_US
> Ign http://= us.archive.ubuntu.com trusty/multiverse Translation-en_US
> Ign http://= us.archive.ubuntu.com trusty/restricted Translation-en_US
> Ign http://= us.archive.ubuntu.com trusty/universe Translation-en_US
> Fetched 1,011 kB in 19s (50.7 kB/s)
> Reading package lists... Done
> ---
>
> Conclusion:
>
> A QEmu guest router does not route tagged VLAN packages that are
> originated at its host, neighter from others guests hosted at the same=
> hypervisor. Making it impossible to create a virtual network within a<= br> > hypervisor.
>
> Best Regards,
> Thiago Martins
>
> ** Affects: qemu
>=C2=A0 =C2=A0 =C2=A0 Importance: Undecided
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Status: New
>



--047d7bfcf2f8e89b9d0505bd8f8d--