Re: [PATCH v3] KVM: Resize kvm_io_range array dynamically

[Amos Kong <akong@redhat.com>]

On 01/03/12 00:34, Amos Kong wrote:
> ----- Original Message -----
>> On 2012-02-29 16:22, Amos Kong wrote:
>>> ----- Original Message -----
>>>> On 2012-02-29 14:30, Amos Kong wrote:
>>>>> kvm_io_bus devices are used for ioevent, pit, pic, ioapic,
>>>>> coalesced_mmio.
>>>>>
>>>>> Currently Qemu only emulates one PCI bus, it contains 32 slots,
>>>>> one slot contains 8 functions, maximum of supported PCI devices:
>>>>>   1 * 32 * 8 = 256. The maximum of coalesced mmio zone is 100,
>>>>> each zone has an iobus devices. 300 io_bus devices is not enough.
>>>>>
>>>>> This patch makes the kvm_io_range array can be resized
>>>>> dynamically.
>>>>
>>>> Is there any limit, or can userspace allocate arbitrary amounts of
>>>> kernel memory this way?
>>>
>>> Hi Jan,
>>>
>>> There is a fixed array in linux-2.6/include/linux/kvm_host.h,
>>> we can only register 300 devices.
>>>
>>> struct kvm_io_range {
>>>          gpa_t addr;
>>>          int len;
>>>          struct kvm_io_device *dev;
>>> };
>>>
>>> struct kvm_io_bus {
>>>          int                   dev_count;
>>> #define NR_IOBUS_DEVS 300
>>>          struct kvm_io_range range[NR_IOBUS_DEVS];
>>> };
>>>                                    ^^^^^^^^^^^^^^
>>
>> Right. But doesn't your patch remove precisely this limit? So what
>> limits userspace now? To register 300 million devices...?
>
> Hi Jan,
>
> It seems we need to reserve the limitation in kvm_host.h
>
> #define NR_IOBUS_DEVS 600
>
> /* Currently Qemu only emulates one PCI bus, it contains 32 slots,
> one slot contains 8 functions. Only 29 slots can be used to
> add multiple function devices. Maximum of supported PCI devices:
> 29 * 8 = 232. Each virtio-blk device needs 1 iobus device,
> each virtio-net(vhost) device requires 2 such devices to service
> notifications (ioevent) from rx/tx queues.
> The maximum of coalesced mmio zone is 100, each zone has an iobus
> devices. ioevent, pit, ioapic take less iobus devices.
>
> So we can set max limitation to 600. */

One virtio-net(vhost=on) takes two iobus devices,
and it needs three IRQs for MSI/MIS-X.
I started a guest with 232 virtio-net(vhost=on),
guest IRQ 24 to 191 were used for virtio-config/input/output,
and only 56 virtio-nics' MSIX were enabled.
56 virtio-net(vhost=on) registered 56 * 2 = 112 iobus devices.

It's safe to set the limit to 300, right ?


> ----- check limit when register dev ----
>
> virt/kvm/kvm_main.c:
>
> /* Caller must hold slots_lock. */
> int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr,
>                              int len, struct kvm_io_device *dev)
> {
>          struct kvm_io_bus *new_bus, *bus;
>
>          bus = kvm->buses[bus_idx];
>          if (bus->dev_count>  NR_IOBUS_DEVS - 1)   // can only register 600 devices
>                  return -ENOSPC;
>
> Amos.

-- 
			Amos.