From mboxrd@z Thu Jan 1 00:00:00 1970 From: Amos Kong Subject: Re: [PATCH v3] KVM: Resize kvm_io_range array dynamically Date: Thu, 01 Mar 2012 13:19:24 +0800 Message-ID: <4F4F06DC.6030400@redhat.com> References: <2b6f7414-7cae-4349-9f5d-eb6c3febf612@zmail13.collab.prod.int.phx2.redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org, jasowang@redhat.com, mtosatti@redhat.com, alex williamson , avi@redhat.com, levinsasha928@gmail.com To: Jan Kiszka Return-path: Received: from mx1.redhat.com ([209.132.183.28]:59914 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751160Ab2CAFTa (ORCPT ); Thu, 1 Mar 2012 00:19:30 -0500 In-Reply-To: <2b6f7414-7cae-4349-9f5d-eb6c3febf612@zmail13.collab.prod.int.phx2.redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On 01/03/12 00:34, Amos Kong wrote: > ----- Original Message ----- >> On 2012-02-29 16:22, Amos Kong wrote: >>> ----- Original Message ----- >>>> On 2012-02-29 14:30, Amos Kong wrote: >>>>> kvm_io_bus devices are used for ioevent, pit, pic, ioapic, >>>>> coalesced_mmio. >>>>> >>>>> Currently Qemu only emulates one PCI bus, it contains 32 slots, >>>>> one slot contains 8 functions, maximum of supported PCI devices: >>>>> 1 * 32 * 8 = 256. The maximum of coalesced mmio zone is 100, >>>>> each zone has an iobus devices. 300 io_bus devices is not enough. >>>>> >>>>> This patch makes the kvm_io_range array can be resized >>>>> dynamically. >>>> >>>> Is there any limit, or can userspace allocate arbitrary amounts of >>>> kernel memory this way? >>> >>> Hi Jan, >>> >>> There is a fixed array in linux-2.6/include/linux/kvm_host.h, >>> we can only register 300 devices. >>> >>> struct kvm_io_range { >>> gpa_t addr; >>> int len; >>> struct kvm_io_device *dev; >>> }; >>> >>> struct kvm_io_bus { >>> int dev_count; >>> #define NR_IOBUS_DEVS 300 >>> struct kvm_io_range range[NR_IOBUS_DEVS]; >>> }; >>> ^^^^^^^^^^^^^^ >> >> Right. But doesn't your patch remove precisely this limit? So what >> limits userspace now? To register 300 million devices...? > > Hi Jan, > > It seems we need to reserve the limitation in kvm_host.h > > #define NR_IOBUS_DEVS 600 > > /* Currently Qemu only emulates one PCI bus, it contains 32 slots, > one slot contains 8 functions. Only 29 slots can be used to > add multiple function devices. Maximum of supported PCI devices: > 29 * 8 = 232. Each virtio-blk device needs 1 iobus device, > each virtio-net(vhost) device requires 2 such devices to service > notifications (ioevent) from rx/tx queues. > The maximum of coalesced mmio zone is 100, each zone has an iobus > devices. ioevent, pit, ioapic take less iobus devices. > > So we can set max limitation to 600. */ One virtio-net(vhost=on) takes two iobus devices, and it needs three IRQs for MSI/MIS-X. I started a guest with 232 virtio-net(vhost=on), guest IRQ 24 to 191 were used for virtio-config/input/output, and only 56 virtio-nics' MSIX were enabled. 56 virtio-net(vhost=on) registered 56 * 2 = 112 iobus devices. It's safe to set the limit to 300, right ? > ----- check limit when register dev ---- > > virt/kvm/kvm_main.c: > > /* Caller must hold slots_lock. */ > int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr, > int len, struct kvm_io_device *dev) > { > struct kvm_io_bus *new_bus, *bus; > > bus = kvm->buses[bus_idx]; > if (bus->dev_count> NR_IOBUS_DEVS - 1) // can only register 600 devices > return -ENOSPC; > > Amos. -- Amos.