From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41618) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bxgWa-0007iX-Dv for qemu-devel@nongnu.org; Fri, 21 Oct 2016 16:34:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bxgWX-0003MS-Aw for qemu-devel@nongnu.org; Fri, 21 Oct 2016 16:34:44 -0400 Received: from mail-it0-x235.google.com ([2607:f8b0:4001:c0b::235]:37230) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1bxgWX-0003M9-4s for qemu-devel@nongnu.org; Fri, 21 Oct 2016 16:34:41 -0400 Received: by mail-it0-x235.google.com with SMTP id m138so6757092itm.0 for ; Fri, 21 Oct 2016 13:34:40 -0700 (PDT) Received: from neil.localnet (206-248-184-9.dsl.teksavvy.com. [206.248.184.9]) by smtp.googlemail.com with ESMTPSA id e6sm71299ite.2.2016.10.21.13.34.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Oct 2016 13:34:39 -0700 (PDT) From: Neil Skrypuch Date: Fri, 21 Oct 2016 16:34:38 -0400 Message-ID: <1854617.40xUKIEeaj@neil> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Subject: [Qemu-devel] Mysterious RST connection with virtio-net NATting VM List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org I have a NATting VM (let's call this vm/nat) sitting in front of another VM (let's call this one vm/wget), with vm/wget residing in a private virtual network, with all network connectivity for vm/wget going through vm/nat. Additionally, I have a web server running on a physical machine from which vm/wget is trying to fetch a file. Graphically, it looks something like this: web server <-> vm/nat <-> vm/wget Now, when vm/wget tries to wget a file from the web server, it frequently gets a connection reset by peer network error, wget will retry the connection and resume the download, after which the cycle will repeat many times before the file eventually fully downloads. The problem is not limited to wget though, both scp and curl experience the same connection reset by peer issue quite consistently. After some collaborative debugging on IRC, tcpdump revealed that an RST is being generated which is breaking the network connection. Now, from the web server's point of view, the RST is coming from vm/wget and from vm/wget's point of view, the RST is coming from the web server. Further tcpdumping from both NICs on vm/nat reveals that vm/nat itself is generating the RSTs, although it is not clear why. Furthermore, the TCP sequence numbers on the RSTs are totally out of line with the ACKs in the stream, for example: 13:01:19.819882 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 1914061, win 65535, length 0 13:01:19.819886 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 1914061, win 65535, length 0 13:01:19.819889 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 1918441, win 65535, length 0 13:01:19.820127 IP 192.168.123.198.80 > 192.168.100.160.37675: Flags [R], seq 1390832295, win 0, length 0 We ran out of ideas on how to further debug the RST issue, but ideas are welcome. One other thing worth noting is that vm/nat is configured with a pair of virtio-net NICs. If I change the NICs to e1000 models, the RST issue vanishes. More specifically, only the NIC facing the web server on vm/nat needs to be changed to e1000 for the issue to go away, the other can remain virtio-net. Also, running a wget directly on vm/nat works in all cases. I tried a number of other things to try and narrow down the issue, none of which helped, including: - vhost=on - qemu versions 2.7.0, 2.3.0 and 2.0.0 - disabling kvm accel (ie, using TCG) The full command line for vm/nat is this: /usr/bin/qemu-system-x86_64 -machine accel=kvm -vga cirrus -m 512 -smp 4,cores=4,sockets=1,threads=1 -drive file=/var/lib/kvm/lb.polldev.com.img,format=raw,if=virtio -device virtio-net- pci,mac=52:54:00:42:45:15,netdev=net1 -netdev type=tap,script=/etc/qemu-ifup- br0,downscript=no,id=net1 -device virtio-net- pci,mac=52:54:00:42:45:16,netdev=net2 -netdev type=tap,script=/etc/qemu-ifup- br2,downscript=no,id=net2 -curses -monitor unix:/var/lib/kvm/monitor/lb,server,nowait -name "lb.polldev.com",process=lb.polldev.com I was able to reproduce this with different web server machines, as well as different vm/wget machines, as well as when moving vm/nat to a different host (albeit for the other host, the RST issue presents itself significantly less frequently). Since the problem goes away with e1000, my gut feeling is that virtio-net is somehow slightly mangling the packets which causes iptables to eventually RST the connection, although it's probably more subtle than that, since I'm sure I'm not the first person to try NAT in a virtio-net VM. Any thoughts? - Neil