From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37191) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YG8uF-0006rf-3d for qemu-devel@nongnu.org; Tue, 27 Jan 2015 11:22:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YG8uA-0005DN-0O for qemu-devel@nongnu.org; Tue, 27 Jan 2015 11:22:23 -0500 Received: from mail.choopa.net ([216.155.136.52]:53092) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YG8u9-0005DJ-U1 for qemu-devel@nongnu.org; Tue, 27 Jan 2015 11:22:17 -0500 Received: from localhost (unknown [127.0.0.1]) by mail.choopa.net (iRedMail) with ESMTP id A91104AF4BC for ; Tue, 27 Jan 2015 11:22:17 -0500 (EST) Received: from mail.choopa.net ([127.0.0.1]) by localhost (mail.choopa.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y48EkYVzvlZh for ; Tue, 27 Jan 2015 11:22:17 -0500 (EST) Received: from [10.5.5.117] (office-nat.choopa.net [208.167.225.40]) by mail.choopa.net (iRedMail) with ESMTPSA id 10B2E4AF470 for ; Tue, 27 Jan 2015 11:22:16 -0500 (EST) Message-ID: <54C7BB36.80400@gameservers.com> Date: Tue, 27 Jan 2015 11:22:14 -0500 From: Brian Rak MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Subject: [Qemu-devel] Multicast network breaks under high load List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org I have two Ubuntu 14.04 KVM instances (kernel version 3.16.0-29-generic). (Running under qemu 2.2.0) They're connected using qemu's 'multicast' network type, and basic connectivity works fine. I can ping between the guests, they can transfer data, etc. I'll refer to them this way: guest1 - 10.99.0.10 guest2 - 10.99.0.11 The problem is when I send a large amount of traffic from guest2 to guest1, guest2's network stack just stops transmitting packets. This issue persists until I reboot guest2 (rebooting from the OS itself is enough, I do not need to terminate and relaunch the qemu process). Running tcpdump from within guest2, it sees incoming ARP packets: 16:09:29.197379 ARP, Request who-has 10.99.0.11 tell 10.99.0.10, length 28 16:09:30.196671 ARP, Request who-has 10.99.0.11 tell 10.99.0.10, length 28 but, it doesn't respond to them (even though the network config is fine): 3: eth1: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 10.99.0.11/8 brd 10.255.255.255 scope global eth1 valid_lft forever preferred_lft forever Once this breaks, if I try to ping guest1 from guest2 I see no bytes transmitted (via ifconfig & tcpdump). If I let ping run for a minute or two, I start seeing: ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available Things that I've tried, that haven't prevented the issue: * Kernel upgrade of guest2 to 3.17.1-031701-generic * Disabling GRO on guest2 Once this breaks, none of the following will fix it: * ifconfig eth1 down; ifconfig eth1 up It looks to me like the virtio driver is just giving up on transmitting packets for some reason. I'm not really sure how I would begin troubleshooting this. Any suggestions? For reference, guest2's network definition looks like this: -netdev socket,mcast=230.0.0.0:6000,id=hostnet1