* KVM cpu limitations @ 2011-07-20 23:20 John Paul Walters 2011-07-21 6:10 ` Avi Kivity 0 siblings, 1 reply; 7+ messages in thread From: John Paul Walters @ 2011-07-20 23:20 UTC (permalink / raw) To: kvm Hi, We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores. I see that x86 guests are currently limited to 64 VCPUs. Is there any reason for this hard limitation? It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256. Qemu-kvm and possibly SeaBIOS seem to require changes as well. Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests? thanks, JP ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KVM cpu limitations 2011-07-20 23:20 KVM cpu limitations John Paul Walters @ 2011-07-21 6:10 ` Avi Kivity 2011-08-08 19:18 ` John Paul Walters 0 siblings, 1 reply; 7+ messages in thread From: Avi Kivity @ 2011-07-21 6:10 UTC (permalink / raw) To: John Paul Walters; +Cc: kvm On 07/21/2011 02:20 AM, John Paul Walters wrote: > Hi, > > We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores. I see that x86 guests are currently limited to 64 VCPUs. Is there any reason for this hard limitation? It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256. Qemu-kvm and possibly SeaBIOS seem to require changes as well. Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests? > And here I am on record saying no one wants this... kvm.git has patches increasing the limit to 254 (256 is not possible due to the APIC ID being 8 bits and two IDs being reserved). Latest seabios appears to have no cpu limits; qemu is limited to 255. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KVM cpu limitations 2011-07-21 6:10 ` Avi Kivity @ 2011-08-08 19:18 ` John Paul Walters 2011-08-09 9:13 ` Avi Kivity 0 siblings, 1 reply; 7+ messages in thread From: John Paul Walters @ 2011-08-08 19:18 UTC (permalink / raw) To: Avi Kivity; +Cc: kvm On Jul 21, 2011, at 2:10 AM, Avi Kivity wrote: > On 07/21/2011 02:20 AM, John Paul Walters wrote: >> Hi, >> >> We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores. I see that x86 guests are currently limited to 64 VCPUs. Is there any reason for this hard limitation? It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256. Qemu-kvm and possibly SeaBIOS seem to require changes as well. Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests? >> > > And here I am on record saying no one wants this... > > kvm.git has patches increasing the limit to 254 (256 is not possible due to the APIC ID being 8 bits and two IDs being reserved). > > Latest seabios appears to have no cpu limits; qemu is limited to 255. > Hi again, I've applied the 254 core patches (below) from kvm.git on a RHEL 6.1 kernel. The new modules build and insert fine. https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=052fa7f4c5e79262cffcdc90bdd94172e00d45e3 https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=29a07f8e31980599c586ea7d1f84957bc7fe98ed However, whenever I try to boot a system with more than 83 CPUs, the system fails to boot with: Booting from Hard Disk... Boot failed: could not read the boot disk I'm using qemu-kvm.git with the following command line: /opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -hda big_image_2.qcow2 -m 8388 -redir tcp:52109::22 Does anyone have any suggestions? thanks, JP > -- > I have a truly marvellous patch that fixes the bug which this > signature is too narrow to contain. > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KVM cpu limitations 2011-08-08 19:18 ` John Paul Walters @ 2011-08-09 9:13 ` Avi Kivity 2011-08-09 18:32 ` John Paul Walters 0 siblings, 1 reply; 7+ messages in thread From: Avi Kivity @ 2011-08-09 9:13 UTC (permalink / raw) To: John Paul Walters; +Cc: kvm On 08/08/2011 10:18 PM, John Paul Walters wrote: > On Jul 21, 2011, at 2:10 AM, Avi Kivity wrote: > > > On 07/21/2011 02:20 AM, John Paul Walters wrote: > >> Hi, > >> > >> We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores. I see that x86 guests are currently limited to 64 VCPUs. Is there any reason for this hard limitation? It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256. Qemu-kvm and possibly SeaBIOS seem to require changes as well. Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests? > >> > > > > And here I am on record saying no one wants this... > > > > kvm.git has patches increasing the limit to 254 (256 is not possible due to the APIC ID being 8 bits and two IDs being reserved). > > > > Latest seabios appears to have no cpu limits; qemu is limited to 255. > > > > > Hi again, > > I've applied the 254 core patches (below) from kvm.git on a RHEL 6.1 kernel. The new modules build and insert fine. > > https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=052fa7f4c5e79262cffcdc90bdd94172e00d45e3 > https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=29a07f8e31980599c586ea7d1f84957bc7fe98ed > > However, whenever I try to boot a system with more than 83 CPUs, the system fails to boot with: > > Booting from Hard Disk... > Boot failed: could not read the boot disk > > I'm using qemu-kvm.git with the following command line: > /opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -hda big_image_2.qcow2 -m 8388 -redir tcp:52109::22 > > Does anyone have any suggestions? > > Most likely a seabios failure. Suggest you enable debugging in seabios and see what's going on; also copy the seabios mailing list. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KVM cpu limitations 2011-08-09 9:13 ` Avi Kivity @ 2011-08-09 18:32 ` John Paul Walters 2011-08-10 3:04 ` Kevin O'Connor 0 siblings, 1 reply; 7+ messages in thread From: John Paul Walters @ 2011-08-09 18:32 UTC (permalink / raw) To: Avi Kivity; +Cc: kvm, seabios On Aug 9, 2011, at 5:13 AM, Avi Kivity wrote: > On 08/08/2011 10:18 PM, John Paul Walters wrote: >> On Jul 21, 2011, at 2:10 AM, Avi Kivity wrote: >> >> > On 07/21/2011 02:20 AM, John Paul Walters wrote: >> >> Hi, >> >> >> >> We have a 256 core SGI Ultraviolet machine running RHEL 6.1 with qemu-kvm 0.13, and we'd like to be able to start large guest VMs of up to 256 cores. I see that x86 guests are currently limited to 64 VCPUs. Is there any reason for this hard limitation? It appears that we can't get around this limitation by simply redefining the kernel's KVM_MAX_VCPUS to 256. Qemu-kvm and possibly SeaBIOS seem to require changes as well. Can anyone offer any suggestions as to how straightforward it would be to increase the number of CPUs that we can allocate to KVM guests? >> >> >> > >> > And here I am on record saying no one wants this... >> > >> > kvm.git has patches increasing the limit to 254 (256 is not possible due to the APIC ID being 8 bits and two IDs being reserved). >> > >> > Latest seabios appears to have no cpu limits; qemu is limited to 255. >> > >> >> >> Hi again, >> >> I've applied the 254 core patches (below) from kvm.git on a RHEL 6.1 kernel. The new modules build and insert fine. >> >> https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=052fa7f4c5e79262cffcdc90bdd94172e00d45e3 >> https://git.kernel.org/?p=virt/kvm/kvm.git;a=commit;h=29a07f8e31980599c586ea7d1f84957bc7fe98ed >> >> However, whenever I try to boot a system with more than 83 CPUs, the system fails to boot with: >> >> Booting from Hard Disk... >> Boot failed: could not read the boot disk >> >> I'm using qemu-kvm.git with the following command line: >> /opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -hda big_image_2.qcow2 -m 8388 -redir tcp:52109::22 >> >> Does anyone have any suggestions? >> >> > > Most likely a seabios failure. Suggest you enable debugging in seabios and see what's going on; also copy the seabios mailing list. > Hi Avi, I've enabled debugging in seabios (#define DEBUG_BIOS) and get the output below. Note that with the help of folks in the KVM irc channel I'm able to start a 254 core instance using the KVM tool, so the problem seems to be limited to qemu/seabios. best, JP jwalters@uv /tmp/qemu_test_jp $ /opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -drive file=big_image_2.qcow2,if=virtio -m 8388 warning: subregion collision fffe0000/20000 vs 0/12c400000 VNC server running on `::1:5901' Start bios (version pre-0.6.3-20110315_112143-titi) Ram Size=0xe0000000 (0x000000012c400000 high) Relocating init from 0x000e49d0 to 0xdffe1880 (size 58968) CPU Mhz=2002 PCI: pci_bios_init_bus_rec bus = 0x0 PIIX3/PIIX4 init: elcr=00 0c PCI: bus=0 devfn=0x00: vendor_id=0x8086 device_id=0x1237 PCI: bus=0 devfn=0x08: vendor_id=0x8086 device_id=0x7000 PCI: bus=0 devfn=0x09: vendor_id=0x8086 device_id=0x7010 region 4: 0x0000c000 PCI: bus=0 devfn=0x0b: vendor_id=0x8086 device_id=0x7113 PCI: bus=0 devfn=0x10: vendor_id=0x1013 device_id=0x00b8 region 0: 0xf0000000 region 1: 0xf2000000 region 6: 0xf2010000 PCI: bus=0 devfn=0x18: vendor_id=0x10ec device_id=0x8139 region 0: 0x0000c100 region 1: 0xf2020000 region 6: 0xf2030000 PCI: bus=0 devfn=0x20: vendor_id=0x1af4 device_id=0x1001 region 0: 0x0000c200 region 1: 0xf2040000 Found 84 cpu(s) max supported 84 cpu(s) MP table addr=0x000fd4b0 MPC table addr=0x000fd4c0 size=1892 SMBIOS ptr=0x000fd490 table=0xdffff030 ACPI tables: RSDP=0x000fd460 RSDT=0xdfffa5c0 Scan for VGA option rom Running option rom at c000:0003 VGABios $Id$ Turning on vga text mode console SeaBIOS (version pre-0.6.3-20110315_112143-titi) Found 1 lpt ports Found 1 serial ports ATA controller 0 at 1f0/3f4/0 (irq 14 dev 9) ATA controller 1 at 170/374/0 (irq 15 dev 9) found virtio-blk at 0:4 ebda moved from 9fc00 to 9dc00 WARNING - Unable to allocate resource at init_virtio_blk:107! WARNING - Unable to allocate resource at init_atadrive:740! PS2 keyboard initialized All threads complete. Scan for option roms Running option rom at c900:0003 pmm call arg1=1 pmm call arg1=0 pmm call arg1=1 pmm call arg1=0 Searching bootorder for: /pci@i0cf8/*@3 Searching bootorder for: /rom@genroms/vapic.bin Running option rom at ca00:0003 Returned 40960 bytes of ZoneHigh e820 map has 8 items: 0: 0000000000000000 - 000000000009dc00 = 1 1: 000000000009dc00 - 00000000000a0000 = 2 2: 00000000000f0000 - 0000000000100000 = 2 3: 0000000000100000 - 00000000dfffa000 = 1 4: 00000000dfffa000 - 00000000e0000000 = 2 5: 00000000feffc000 - 00000000ff000000 = 2 6: 00000000fffc0000 - 0000000100000000 = 2 7: 0000000100000000 - 000000022c400000 = 1 enter handle_19: NULL Booting from ROM... Booting from c900:0372 enter handle_18: NULL Booting from Hard Disk... Boot failed: could not read the boot disk enter handle_18: NULL Booting from Floppy... Boot failed: could not read the boot disk enter handle_18: NULL No bootable device. > -- > error compiling committee.c: too many arguments to function > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KVM cpu limitations 2011-08-09 18:32 ` John Paul Walters @ 2011-08-10 3:04 ` Kevin O'Connor 2011-08-10 20:43 ` John Paul Walters 0 siblings, 1 reply; 7+ messages in thread From: Kevin O'Connor @ 2011-08-10 3:04 UTC (permalink / raw) To: John Paul Walters; +Cc: Avi Kivity, kvm, seabios On Tue, Aug 09, 2011 at 02:32:02PM -0400, John Paul Walters wrote: > I've enabled debugging in seabios (#define DEBUG_BIOS) and get the > output below. Note that with the help of folks in the KVM irc > channel I'm able to start a 254 core instance using the KVM tool, so > the problem seems to be limited to qemu/seabios. [...] > jwalters@uv /tmp/qemu_test_jp $ /opt/qemu.git/bin/qemu-system-x86_64 -smp 84 -drive file=big_image_2.qcow2,if=virtio -m 8388 > Found 84 cpu(s) max supported 84 cpu(s) > MP table addr=0x000fd4b0 MPC table addr=0x000fd4c0 size=1892 > SMBIOS ptr=0x000fd490 table=0xdffff030 > ACPI tables: RSDP=0x000fd460 RSDT=0xdfffa5c0 [...] > WARNING - Unable to allocate resource at init_virtio_blk:107! > WARNING - Unable to allocate resource at init_atadrive:740! You're hitting a weird interaction with the mptable - its allocation just barely fits, and since it takes up all the ram in the f-segment it causes other allocations to fail (which are needed to hold info on the boot drive). This should no longer be an issue in the latest SeaBIOS from git. Alternatively, if you increase the number of CPUs, you should see the machine boot becuase only the mptable allocation will fail. -Kevin ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: KVM cpu limitations 2011-08-10 3:04 ` Kevin O'Connor @ 2011-08-10 20:43 ` John Paul Walters 0 siblings, 0 replies; 7+ messages in thread From: John Paul Walters @ 2011-08-10 20:43 UTC (permalink / raw) To: Kevin O'Connor; +Cc: seabios, Avi Kivity, kvm [-- Attachment #1.1: Type: text/plain, Size: 640 bytes --] On Aug 9, 2011, at 11:04 PM, Kevin O'Connor wrote: > On Tue, Aug 09, 2011 at 02:32:02PM -0400, John Paul Walters wrote: > > You're hitting a weird interaction with the mptable - its allocation > just barely fits, and since it takes up all the ram in the f-segment > it causes other allocations to fail (which are needed to hold info on > the boot drive). > > This should no longer be an issue in the latest SeaBIOS from git. > Alternatively, if you increase the number of CPUs, you should see the > machine boot becuase only the mptable allocation will fail. > Thanks Kevin. I updated SeaBIOS and I'm able to boot the VM now. JP [-- Attachment #1.2: Type: text/html, Size: 1016 bytes --] [-- Attachment #2: Type: text/plain, Size: 137 bytes --] _______________________________________________ SeaBIOS mailing list SeaBIOS@seabios.org http://www.seabios.org/mailman/listinfo/seabios ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2011-08-10 20:43 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-07-20 23:20 KVM cpu limitations John Paul Walters 2011-07-21 6:10 ` Avi Kivity 2011-08-08 19:18 ` John Paul Walters 2011-08-09 9:13 ` Avi Kivity 2011-08-09 18:32 ` John Paul Walters 2011-08-10 3:04 ` Kevin O'Connor 2011-08-10 20:43 ` John Paul Walters
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox