[Qemu-devel] Mysterious RST connection with virtio-net NATting VM

[Qemu-devel] Mysterious RST connection with virtio-net NATting VM

[Neil Skrypuch]

I have a NATting VM (let's call this vm/nat) sitting in front of another VM 
(let's call this one vm/wget), with vm/wget residing in a private virtual 
network, with all network connectivity for vm/wget going through vm/nat. 
Additionally, I have a web server running on a physical machine from which 
vm/wget is trying to fetch a file. Graphically, it looks something like this:

web server <-> vm/nat <-> vm/wget

Now, when vm/wget tries to wget a file from the web server, it frequently gets 
a connection reset by peer network error, wget will retry the connection and 
resume the download, after which the cycle will repeat many times before the 
file eventually fully downloads. The problem is not limited to wget though, 
both scp and curl experience the same connection reset by peer issue quite 
consistently.

After some collaborative debugging on IRC, tcpdump revealed that an RST is 
being generated which is breaking the network connection. Now, from the web 
server's point of view, the RST is coming from vm/wget and from vm/wget's 
point of view, the RST is coming from the web server. Further tcpdumping from 
both NICs on vm/nat reveals that vm/nat itself is generating the RSTs, 
although it is not clear why. Furthermore, the TCP sequence numbers on the 
RSTs are totally out of line with the ACKs in the stream, for example:

13:01:19.819882 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 
1914061, win 65535, length 0
13:01:19.819886 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 
1914061, win 65535, length 0
13:01:19.819889 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.], ack 
1918441, win 65535, length 0
13:01:19.820127 IP 192.168.123.198.80 > 192.168.100.160.37675: Flags [R], seq 
1390832295, win 0, length 0

We ran out of ideas on how to further debug the RST issue, but ideas are 
welcome.

One other thing worth noting is that vm/nat is configured with a pair of 
virtio-net NICs. If I change the NICs to e1000 models, the RST issue vanishes. 
More specifically, only the NIC facing the web server on vm/nat needs to be 
changed to e1000 for the issue to go away, the other can remain virtio-net. 
Also, running a wget directly on vm/nat works in all cases. I tried a number 
of other things to try and narrow down the issue, none of which helped, 
including:

- vhost=on
- qemu versions 2.7.0, 2.3.0 and 2.0.0
- disabling kvm accel (ie, using TCG)

The full command line for vm/nat is this:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -vga cirrus -m 512 -smp 
4,cores=4,sockets=1,threads=1 -drive 
file=/var/lib/kvm/lb.polldev.com.img,format=raw,if=virtio -device virtio-net-
pci,mac=52:54:00:42:45:15,netdev=net1 -netdev type=tap,script=/etc/qemu-ifup-
br0,downscript=no,id=net1 -device virtio-net-
pci,mac=52:54:00:42:45:16,netdev=net2 -netdev type=tap,script=/etc/qemu-ifup-
br2,downscript=no,id=net2 -curses -monitor 
unix:/var/lib/kvm/monitor/lb,server,nowait -name 
"lb.polldev.com",process=lb.polldev.com

I was able to reproduce this with different web server machines, as well as 
different vm/wget machines, as well as when moving vm/nat to a different host 
(albeit for the other host, the RST issue presents itself significantly less 
frequently).

Since the problem goes away with e1000, my gut feeling is that virtio-net is 
somehow slightly mangling the packets which causes iptables to eventually RST 
the connection, although it's probably more subtle than that, since I'm sure 
I'm not the first person to try NAT in a virtio-net VM.

Any thoughts?

- Neil

Re: [Qemu-devel] Mysterious RST connection with virtio-net NATting VM

[Neil Skrypuch]

On October 21, 2016 04:34:38 PM Neil Skrypuch wrote:
> I have a NATting VM (let's call this vm/nat) sitting in front of another VM
> (let's call this one vm/wget), with vm/wget residing in a private virtual
> network, with all network connectivity for vm/wget going through vm/nat.
> Additionally, I have a web server running on a physical machine from which
> vm/wget is trying to fetch a file. Graphically, it looks something like
> this:
> 
> web server <-> vm/nat <-> vm/wget
> 
> Now, when vm/wget tries to wget a file from the web server, it frequently
> gets a connection reset by peer network error, wget will retry the
> connection and resume the download, after which the cycle will repeat many
> times before the file eventually fully downloads. The problem is not
> limited to wget though, both scp and curl experience the same connection
> reset by peer issue quite consistently.
> 
> After some collaborative debugging on IRC, tcpdump revealed that an RST is
> being generated which is breaking the network connection. Now, from the web
> server's point of view, the RST is coming from vm/wget and from vm/wget's
> point of view, the RST is coming from the web server. Further tcpdumping
> from both NICs on vm/nat reveals that vm/nat itself is generating the RSTs,
> although it is not clear why. Furthermore, the TCP sequence numbers on the
> RSTs are totally out of line with the ACKs in the stream, for example:
> 
> 13:01:19.819882 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.],
> ack 1914061, win 65535, length 0
> 13:01:19.819886 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.],
> ack 1914061, win 65535, length 0
> 13:01:19.819889 IP 192.168.100.160.37675 > 192.168.123.198.80: Flags [.],
> ack 1918441, win 65535, length 0
> 13:01:19.820127 IP 192.168.123.198.80 > 192.168.100.160.37675: Flags [R],
> seq 1390832295, win 0, length 0
> 
> We ran out of ideas on how to further debug the RST issue, but ideas are
> welcome.
> 
> One other thing worth noting is that vm/nat is configured with a pair of
> virtio-net NICs. If I change the NICs to e1000 models, the RST issue
> vanishes. More specifically, only the NIC facing the web server on vm/nat
> needs to be changed to e1000 for the issue to go away, the other can remain
> virtio-net. Also, running a wget directly on vm/nat works in all cases. I
> tried a number of other things to try and narrow down the issue, none of
> which helped, including:
> 
> - vhost=on
> - qemu versions 2.7.0, 2.3.0 and 2.0.0
> - disabling kvm accel (ie, using TCG)
> 
> The full command line for vm/nat is this:
> 
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -vga cirrus -m 512 -smp
> 4,cores=4,sockets=1,threads=1 -drive
> file=/var/lib/kvm/lb.polldev.com.img,format=raw,if=virtio -device
> virtio-net- pci,mac=52:54:00:42:45:15,netdev=net1 -netdev
> type=tap,script=/etc/qemu-ifup- br0,downscript=no,id=net1 -device
> virtio-net-
> pci,mac=52:54:00:42:45:16,netdev=net2 -netdev
> type=tap,script=/etc/qemu-ifup- br2,downscript=no,id=net2 -curses -monitor
> unix:/var/lib/kvm/monitor/lb,server,nowait -name
> "lb.polldev.com",process=lb.polldev.com
> 
> I was able to reproduce this with different web server machines, as well as
> different vm/wget machines, as well as when moving vm/nat to a different
> host (albeit for the other host, the RST issue presents itself
> significantly less frequently).
> 
> Since the problem goes away with e1000, my gut feeling is that virtio-net is
> somehow slightly mangling the packets which causes iptables to eventually
> RST the connection, although it's probably more subtle than that, since I'm
> sure I'm not the first person to try NAT in a virtio-net VM.
> 
> Any thoughts?
> 
> - Neil

I was able to narrow this down a bit. In particular, it only happens when 
vm/wget has net.ipv4.tcp_window_scaling = 0. Once I restore window scaling to 
the default value (1), I no longer get the mysterious RSTs and everything 
works as expected.

We actually don't have a good reason to disable window scaling in the first 
place (in fact it's pretty much required to get close to wire line speeds on 
most modern networks), so I'm considering this a solution. Though, it would be 
interesting to know how disabled window scaling causes the RST, there is 
almost certainly a bug there, somewhere.

- Neil