KVM cpu limitations

KVM cpu limitations

[John Paul Walters]

Hi,

We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores.  I see that x86 guests are currently limited to 64 VCPUs.  Is there any reason for this hard limitation?  It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256.  Qemu-kvm and possibly SeaBIOS seem to require changes as well.  Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests?

thanks,
JP 

Re: KVM cpu limitations

[Avi Kivity]

On 07/21/2011 02:20 AM, John Paul Walters wrote:
> Hi,
>
> We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores.  I see that x86 guests are currently limited to 64 VCPUs.  Is there any reason for this hard limitation?  It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256.  Qemu-kvm and possibly SeaBIOS seem to require changes as well.  Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests?
>

And here I am on record saying no one wants this...

kvm.git has patches increasing the limit to 254 (256 is not possible due 
to the APIC ID being 8  bits and two IDs being reserved).

Latest seabios appears to have no cpu limits; qemu is limited to 255.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: KVM cpu limitations

[John Paul Walters]

On Jul 21, 2011, at 2:10 AM, Avi Kivity wrote:

> On 07/21/2011 02:20 AM, John Paul Walters wrote:
>> Hi,
>> 
>> We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores.  I see that x86 guests are currently limited to 64 VCPUs.  Is there any reason for this hard limitation?  It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256.  Qemu-kvm and possibly SeaBIOS seem to require changes as well.  Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests?
>> 
> 
> And here I am on record saying no one wants this...
> 
> kvm.git has patches increasing the limit to 254 (256 is not possible due to the APIC ID being 8  bits and two IDs being reserved).
> 
> Latest seabios appears to have no cpu limits; qemu is limited to 255.
> 


Hi again,

I've applied the 254 core patches (below) from kvm.git on a RHEL 6.1 kernel.  The new modules build and insert fine.

https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=052fa7f4c5e79262cffcdc90bdd94172e00d45e3
https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=29a07f8e31980599c586ea7d1f84957bc7fe98ed

However, whenever I try to boot a system with more than 83 CPUs, the system fails to boot with:

Booting from Hard Disk...
Boot failed: could not read the boot disk

I'm using qemu-kvm.git with the following command line:
/opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -hda big_image_2.qcow2 -m 8388 -redir tcp:52109::22

Does anyone have any suggestions?

thanks,
JP


> -- 
> I have a truly marvellous patch that fixes the bug which this
> signature is too narrow to contain.
> 

Re: KVM cpu limitations

[Avi Kivity]

On 08/08/2011 10:18 PM, John Paul Walters wrote:
> On Jul 21, 2011, at 2:10 AM, Avi Kivity wrote:
>
> >  On 07/21/2011 02:20 AM, John Paul Walters wrote:
> >>  Hi,
> >>
> >>  We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores.  I see that x86 guests are currently limited to 64 VCPUs.  Is there any reason for this hard limitation?  It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256.  Qemu-kvm and possibly SeaBIOS seem to require changes as well.  Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests?
> >>
> >
> >  And here I am on record saying no one wants this...
> >
> >  kvm.git has patches increasing the limit to 254 (256 is not possible due to the APIC ID being 8  bits and two IDs being reserved).
> >
> >  Latest seabios appears to have no cpu limits; qemu is limited to 255.
> >
>
>
> Hi again,
>
> I've applied the 254 core patches (below) from kvm.git on a RHEL 6.1 kernel.  The new modules build and insert fine.
>
> https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=052fa7f4c5e79262cffcdc90bdd94172e00d45e3
> https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=29a07f8e31980599c586ea7d1f84957bc7fe98ed
>
> However, whenever I try to boot a system with more than 83 CPUs, the system fails to boot with:
>
> Booting from Hard Disk...
> Boot failed: could not read the boot disk
>
> I'm using qemu-kvm.git with the following command line:
> /opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -hda big_image_2.qcow2 -m 8388 -redir tcp:52109::22
>
> Does anyone have any suggestions?
>
>

Most likely a seabios failure.  Suggest you enable debugging in seabios 
and see what's going on; also copy the seabios mailing list.

-- 
error compiling committee.c: too many arguments to function

Re: KVM cpu limitations

[John Paul Walters]

On Aug 9, 2011, at 5:13 AM, Avi Kivity wrote:

> On 08/08/2011 10:18 PM, John Paul Walters wrote:
>> On Jul 21, 2011, at 2:10 AM, Avi Kivity wrote:
>> 
>> >  On 07/21/2011 02:20 AM, John Paul Walters wrote:
>> >>  Hi,
>> >>
>> >>  We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores.  I see that x86 guests are currently limited to 64 VCPUs.  Is there any reason for this hard limitation?  It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256.  Qemu-kvm and possibly SeaBIOS seem to require changes as well.  Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests?
>> >>
>> >
>> >  And here I am on record saying no one wants this...
>> >
>> >  kvm.git has patches increasing the limit to 254 (256 is not possible due to the APIC ID being 8  bits and two IDs being reserved).
>> >
>> >  Latest seabios appears to have no cpu limits; qemu is limited to 255.
>> >
>> 
>> 
>> Hi again,
>> 
>> I've applied the 254 core patches (below) from kvm.git on a RHEL 6.1 kernel.  The new modules build and insert fine.
>> 
>> https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=052fa7f4c5e79262cffcdc90bdd94172e00d45e3
>> https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=29a07f8e31980599c586ea7d1f84957bc7fe98ed
>> 
>> However, whenever I try to boot a system with more than 83 CPUs, the system fails to boot with:
>> 
>> Booting from Hard Disk...
>> Boot failed: could not read the boot disk
>> 
>> I'm using qemu-kvm.git with the following command line:
>> /opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -hda big_image_2.qcow2 -m 8388 -redir tcp:52109::22
>> 
>> Does anyone have any suggestions?
>> 
>> 
> 
> Most likely a seabios failure.  Suggest you enable debugging in seabios and see what's going on; also copy the seabios mailing list.
> 
Hi Avi,

I've enabled debugging in seabios (#define DEBUG_BIOS) and get the output below.  Note that with the help of folks in the KVM irc channel I'm able to start a 254 core instance using the KVM tool, so the problem seems to be limited to qemu/seabios.

best,
JP

jwalters@uv /tmp/qemu_test_jp $ /opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -drive file=big_image_2.qcow2,if=virtio -m 8388
warning: subregion collision fffe0000/20000 vs 0/12c400000
VNC server running on `::1:5901'
Start bios (version pre-0.6.3-20110315_112143-titi)
Ram Size=0xe0000000 (0x000000012c400000 high)
Relocating init from 0x000e49d0 to 0xdffe1880 (size 58968)
CPU Mhz=2002
PCI: pci_bios_init_bus_rec bus = 0x0
PIIX3/PIIX4 init: elcr=00 0c
PCI: bus=0 devfn=0x00: vendor_id=0x8086 device_id=0x1237
PCI: bus=0 devfn=0x08: vendor_id=0x8086 device_id=0x7000
PCI: bus=0 devfn=0x09: vendor_id=0x8086 device_id=0x7010
region 4: 0x0000c000
PCI: bus=0 devfn=0x0b: vendor_id=0x8086 device_id=0x7113
PCI: bus=0 devfn=0x10: vendor_id=0x1013 device_id=0x00b8
region 0: 0xf0000000
region 1: 0xf2000000
region 6: 0xf2010000
PCI: bus=0 devfn=0x18: vendor_id=0x10ec device_id=0x8139
region 0: 0x0000c100
region 1: 0xf2020000
region 6: 0xf2030000
PCI: bus=0 devfn=0x20: vendor_id=0x1af4 device_id=0x1001
region 0: 0x0000c200
region 1: 0xf2040000
Found 84 cpu(s) max supported 84 cpu(s)
MP table addr=0x000fd4b0 MPC table addr=0x000fd4c0 size=1892
SMBIOS ptr=0x000fd490 table=0xdffff030
ACPI tables: RSDP=0x000fd460 RSDT=0xdfffa5c0
Scan for VGA option rom
Running option rom at c000:0003
VGABios $Id$
Turning on vga text mode console
SeaBIOS (version pre-0.6.3-20110315_112143-titi)

Found 1 lpt ports
Found 1 serial ports
ATA controller 0 at 1f0/3f4/0 (irq 14 dev 9)
ATA controller 1 at 170/374/0 (irq 15 dev 9)
found virtio-blk at 0:4
ebda moved from 9fc00 to 9dc00
WARNING - Unable to allocate resource at init_virtio_blk:107!
WARNING - Unable to allocate resource at init_atadrive:740!
PS2 keyboard initialized
All threads complete.
Scan for option roms
Running option rom at c900:0003
pmm call arg1=1
pmm call arg1=0
pmm call arg1=1
pmm call arg1=0
Searching bootorder for: /pci@i0cf8/*@3
Searching bootorder for: /rom@genroms/vapic.bin
Running option rom at ca00:0003
Returned 40960 bytes of ZoneHigh
e820 map has 8 items:
  0: 0000000000000000 - 000000000009dc00 = 1
  1: 000000000009dc00 - 00000000000a0000 = 2
  2: 00000000000f0000 - 0000000000100000 = 2
  3: 0000000000100000 - 00000000dfffa000 = 1
  4: 00000000dfffa000 - 00000000e0000000 = 2
  5: 00000000feffc000 - 00000000ff000000 = 2
  6: 00000000fffc0000 - 0000000100000000 = 2
  7: 0000000100000000 - 000000022c400000 = 1
enter handle_19:
  NULL
Booting from ROM...
Booting from c900:0372
enter handle_18:
  NULL
Booting from Hard Disk...
Boot failed: could not read the boot disk

enter handle_18:
  NULL
Booting from Floppy...
Boot failed: could not read the boot disk

enter handle_18:
  NULL
No bootable device.




> -- 
> error compiling committee.c: too many arguments to function
> 

Re: KVM cpu limitations

[Kevin O'Connor]

On Tue, Aug 09, 2011 at 02:32:02PM -0400, John Paul Walters wrote:
> I've enabled debugging in seabios (#define DEBUG_BIOS) and get the
> output below.  Note that with the help of folks in the KVM irc
> channel I'm able to start a 254 core instance using the KVM tool, so
> the problem seems to be limited to qemu/seabios.
[...]
> jwalters@uv /tmp/qemu_test_jp $ /opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -drive file=big_image_2.qcow2,if=virtio -m 8388
> Found 84 cpu(s) max supported 84 cpu(s)
> MP table addr=0x000fd4b0 MPC table addr=0x000fd4c0 size=1892
> SMBIOS ptr=0x000fd490 table=0xdffff030
> ACPI tables: RSDP=0x000fd460 RSDT=0xdfffa5c0
[...]
> WARNING - Unable to allocate resource at init_virtio_blk:107!
> WARNING - Unable to allocate resource at init_atadrive:740!

You're hitting a weird interaction with the mptable - its allocation
just barely fits, and since it takes up all the ram in the f-segment
it causes other allocations to fail (which are needed to hold info on
the boot drive).

This should no longer be an issue in the latest SeaBIOS from git.
Alternatively, if you increase the number of CPUs, you should see the
machine boot becuase only the mptable allocation will fail.

-Kevin

Re: KVM cpu limitations

[John Paul Walters]

[-- Attachment #1.1: Type: text/plain, Size: 640 bytes --]


On Aug 9, 2011, at 11:04 PM, Kevin O'Connor wrote:

> On Tue, Aug 09, 2011 at 02:32:02PM -0400, John Paul Walters wrote:
> 
> You're hitting a weird interaction with the mptable - its allocation
> just barely fits, and since it takes up all the ram in the f-segment
> it causes other allocations to fail (which are needed to hold info on
> the boot drive).
> 
> This should no longer be an issue in the latest SeaBIOS from git.
> Alternatively, if you increase the number of CPUs, you should see the
> machine boot becuase only the mptable allocation will fail.
> 

Thanks Kevin.  I updated SeaBIOS and I'm able to boot the VM now.

JP




[-- Attachment #1.2: Type: text/html, Size: 1016 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
SeaBIOS mailing list
SeaBIOS@seabios.org
http://www.seabios.org/mailman/listinfo/seabios