From mboxrd@z Thu Jan 1 00:00:00 1970 From: Anthony Liguori Subject: Re: [Qemu-devel] Hitting 29 NIC limit Date: Thu, 14 Oct 2010 07:57:20 -0500 Message-ID: <4CB6FE30.1040904@codemonkey.ws> References: <4CB6388A.30006@codemonkey.ws> <4CB6F275.2060204@redhat.com> <20101014121047.GN11403@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Avi Kivity , Anjali Kulkarni , "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" To: "Daniel P. Berrange" Return-path: Received: from mail-iw0-f174.google.com ([209.85.214.174]:52124 "EHLO mail-iw0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754072Ab0JNM5W (ORCPT ); Thu, 14 Oct 2010 08:57:22 -0400 Received: by iwn41 with SMTP id 41so799202iwn.19 for ; Thu, 14 Oct 2010 05:57:22 -0700 (PDT) In-Reply-To: <20101014121047.GN11403@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On 10/14/2010 07:10 AM, Daniel P. Berrange wrote: > On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote: > >> On 10/14/2010 12:54 AM, Anthony Liguori wrote: >> >>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote: >>> >>>> Hi, >>>> >>>> Using the legacy way of starting up NICs, I am hitting a limitation >>>> after 29 >>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot >>>> limit on a single bus- 3 are already taken up) >>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to >>>> more, but ofcource that wont work. >>>> Is there any way to go beyond 29 NICs the legacy way? What is the >>>> maximum >>>> that can be supported by the qdev mothod? >>>> >>> I got up to 104 without trying very hard using the following script: >>> >>> args="" >>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do >>> for fn in 0 1 2 3 4 5 6 7; do >>> args="$args -netdev user,id=eth${slot}_${fn}" >>> args="$args -device >>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile=" >>> done >>> done >>> >>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} >>> -enable-kvm >>> >>> The key is to make the virtio-net devices multifunction and to fill >>> out all 8 functions for each slot. >>> >> This is unlikely to work right wrt pci hotplug. If we want to support a >> large number of interfaces, we need true multiport cards. >> > Or a PCI bridge to wire up more PCI buses, so we raise the max limit for > any type of device we emulate. > I've always been sceptical of this. When physical systems have a large number of NICs, it's via multiple functions, not a bunch of PCI bridges. With just a handful of 8-port NICs, you can exceed the current slot-based limit on physical hardware. It's not an extremely common configuration but it does exist. BTW, I don't think it's possible to hot-add physical functions. I believe I know of a card that supports dynamic add of physical functions (pre-dating SR-IOV). Regards, Anthony Liguori > Daniel >