Problem with SMP boot

Problem with SMP boot

[Tim Chen]

Hi,

I am only able to bring up a single cpu on my guest even when I
specify a 4 cpu smp config.  My test machine 
is a two socket dual core woodcrest with 4GB RAM. I am
using a stock RHEL5 kernel on my guest and 2.6.20-rc5 
on my host. 

The command I used is:
qemu-kvm -hda vdisk2.img -smp 4 -boot c -m 1024 -no-acpi

I've tried also without -no-acpi with the same result.

Now when I switch to qemu-system-x86_64 without kvm, the guest boots
with all 4 cpus being specified.  

>From the boot log, it looks like it is APIC related.
Is this a known problem?

Thanks.

Tim Chen

------------Boot log on guest below----------------

boot log for command
qemu-kvm -hda vdisk2.img -smp 4 -boot c -m 1024 -no-acpi

Linux version 2.6.18-1.3002.el5 (brewbuilder-sQhldQRnEDERsRnJhv8rMPXAX3CI6PSWQQ4Iyu8u01E@public.gmane.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-51)) #1 SMP Tue Jan 9 21:19:58 EST 2007
Command line: ro root=LABEL=/ rhgb quiet
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 0000000040000000 (usable)
 BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
DMI not present or invalid.
ACPI: Unable to locate RSDP
No NUMA configuration found
Faking a node at 0000000000000000-0000000040000000
Bootmem setup node 0 0000000000000000-0000000040000000
Intel MultiProcessor Specification v1.4
    Virtual Wire compatibility mode.
OEM ID: QEMUCPU  Product ID: 0.1          APIC at: 0xFEE00000
Processor #0 6:0 APIC version 17
Processor #1 6:0 APIC version 17
Processor #2 6:0 APIC version 17
Processor #3 6:0 APIC version 17
I/O APIC #4 Version 17 at 0xFEC00000.
Setting APIC routing to physical flat
Processors: 4
Nosave address range: 000000000009f000 - 00000000000a0000
Nosave address range: 00000000000a0000 - 00000000000e8000
Nosave address range: 00000000000e8000 - 0000000000100000
Allocating PCI resources starting at 50000000 (gap: 40000000:bffc0000)
SMP: Allowing 4 CPUs, 0 hotplug CPUs
Built 1 zonelists.  Total pages: 257255
Kernel command line: ro root=LABEL=/ rhgb quiet
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
Memory: 1025624k/1048576k available (2397k kernel code, 22564k reserved, 1222k data, 196k init)
Calibrating delay using timer specific routine.. 5372.61 BogoMIPS (lpj=2686305)
Security Framework v1.0.0 initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU 0/0 -> Node 0
SMP alternatives: switching to UP code
Using IO-APIC 4
activating NMI Watchdog ... done.
Using local APIC timer interrupts.
result 4636414
Detected 4.636 MHz APIC timer.
SMP alternatives: switching to SMP code
Booting processor 1/4 APIC 0x1
Not responding.
Inquiring remote APIC #1...
... APIC #1 ID: failed
... APIC #1 VERSION: failed
... APIC #1 SPIV: failed
SMP alternatives: switching to SMP code
Booting processor 2/3 APIC 0x2
Not responding.
Inquiring remote APIC #2...
... APIC #2 ID: failed
... APIC #2 VERSION: failed
... APIC #2 SPIV: failed
SMP alternatives: switching to SMP code
Booting processor 3/2 APIC 0x3
Not responding.
Inquiring remote APIC #3...
... APIC #3 ID: failed
... APIC #3 VERSION: failed
... APIC #3 SPIV: failed
Brought up 1 CPUs
testing NMI watchdog ... CPU#0: NMI appears to be stuck (0->0)!

-----------------------------------------------------
Boot log for sucessfule brought up of 4 cpus with command:
qemu-system-x86_64 -hda vdisk2.img -smp 4 -boot c -m 1024 -no-acpi


Linux version 2.6.18-1.3002.el5 (brewbuilder-sQhldQRnEDERsRnJhv8rMPXAX3CI6PSWQQ4Iyu8u01E@public.gmane.org) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-51)) #1 SMP Tue Jan 9 21:19:58 EST 2007
Command line: ro root=LABEL=/ rhgb acpi=off
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 0000000040000000 (usable)
 BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
DMI not present or invalid.
No NUMA configuration found
Faking a node at 0000000000000000-0000000040000000
Bootmem setup node 0 0000000000000000-0000000040000000
Intel MultiProcessor Specification v1.4
    Virtual Wire compatibility mode.
OEM ID: QEMUCPU  Product ID: 0.1          APIC at: 0xFEE00000
Processor #0 6:0 APIC version 17
Processor #1 6:0 APIC version 17
Processor #2 6:0 APIC version 17
Processor #3 6:0 APIC version 17
I/O APIC #4 Version 17 at 0xFEC00000.
Setting APIC routing to physical flat
Processors: 4
Nosave address range: 000000000009f000 - 00000000000a0000
Nosave address range: 00000000000a0000 - 00000000000e8000
Nosave address range: 00000000000e8000 - 0000000000100000
Allocating PCI resources starting at 50000000 (gap: 40000000:bffc0000)
SMP: Allowing 4 CPUs, 0 hotplug CPUs
Built 1 zonelists.  Total pages: 257255
Kernel command line: ro root=LABEL=/ rhgb acpi=off
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
Memory: 1025624k/1048576k available (2397k kernel code, 22564k reserved, 1222k data, 196k init)
Calibrating delay using timer specific routine.. 5481.21 BogoMIPS (lpj=2740609)
Security Framework v1.0.0 initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 256
 starting audispd
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU 0/0 -> Node 0
SMP alternatives: switching to UP code
Using IO-APIC 4
activating NMI Watchdog ... done.
Using local APIC timer interrupts.
result 62501278
Detected 62.501 MHz APIC timer.
SMP alternatives: switching to SMP code
Booting processor 1/4 APIC 0x1
Initializing CPU#1
calibrate_delay_direct() failed to get a good estimate for loops_per_jiffy.
Probably due to long platform interrupts. Consider using "lpj=" boot option.
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU 1/0 -> Node 0
QEMU Virtual CPU version 0.8.2 stepping 03
CPU 1: Syncing TSC to CPU 0.
CPU 1: synchronized TSC with CPU 0 (last diff -108 cycles, maxerr 5205480 cycles)
SMP alternatives: switching to SMP code
Booting processor 2/4 APIC 0x2
Initializing CPU#2
calibrate_delay_direct() failed to get a good estimate for loops_per_jiffy.
Probably due to long platform interrupts. Consider using "lpj=" boot option.
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU 2/0 -> Node 0
QEMU Virtual CPU version 0.8.2 stepping 03
CPU 2: Syncing TSC to CPU 0.
CPU 2: synchronized TSC with CPU 0 (last diff 1312276 cycles, maxerr 7810648 cycles)
SMP alternatives: switching to SMP code
Booting processor 3/4 APIC 0x3
Initializing CPU#3
calibrate_delay_direct() failed to get a good estimate for loops_per_jiffy.
Probably due to long platform interrupts. Consider using "lpj=" boot option.
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU 3/0 -> Node 0
QEMU Virtual CPU version 0.8.2 stepping 03
CPU 3: Syncing TSC to CPU 0.
CPU 3: synchronized TSC with CPU 0 (last diff 2613452 cycles, maxerr 10414152 cycles)
Brought up 4 CPUs



-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Problem with SMP boot

[Matthew Hall]

Tim Chen wrote:
> Now when I switch to qemu-system-x86_64 without kvm, the guest boots
> with all 4 cpus being specified.  
> 
>>From the boot log, it looks like it is APIC related.
> Is this a known problem?

Boot the /host/ kernel with 'noapic nolapic' and see if that makes a 
difference for you. There is a performance impact under load, but it 
made a world of difference for me when I had issues which were similarly 
kvm specific.

Matt

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Problem with SMP boot

[Tim Chen]

On Fri, 2007-01-26 at 06:06 +0000, Matthew Hall wrote:

> 
> Boot the /host/ kernel with 'noapic nolapic' and see if that makes a 
> difference for you. There is a performance impact under load, but it 
> made a world of difference for me when I had issues which were similarly 
> kvm specific.

Unfortunately, that did not solve the problem and I see the same error
in my boot log as previously reported.  

Tim

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Problem with SMP boot

[Tim Chen]

On Fri, 2007-01-26 at 19:58 +0200, Avi Kivity wrote:
>   
> 
> kvm only supports uniprocessor guests at this time.
> 

Thanks for the clarification.  Is SMP support for guests
expected soon?

The FAQ is a bit misleading: 
http://kvm.qumranet.com/kvmwiki/FAQ

It says "Does kvm support SMP hosts?  Yes". 
I misinterpreted it meaning that SMP 
support for guests is available.

Tim

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Problem with SMP boot

[Avi Kivity]

Tim Chen wrote:
> Hi,
>
> I am only able to bring up a single cpu on my guest even when I
> specify a 4 cpu smp config.  My test machine 
> is a two socket dual core woodcrest with 4GB RAM. I am
> using a stock RHEL5 kernel on my guest and 2.6.20-rc5 
> on my host. 
>
> The command I used is:
> qemu-kvm -hda vdisk2.img -smp 4 -boot c -m 1024 -no-acpi
>
> I've tried also without -no-acpi with the same result.
>
>   

kvm only supports uniprocessor guests at this time.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Problem with SMP boot

[Matthew Hall]

Avi Kivity wrote:
> kvm only supports uniprocessor guests at this time.

Out of interest, if/when kvm supports smp guests aswell as hosts would 
the guest be presented with a multiple virtual or physical cpu's?
Ie. If the guest has multiple virtual cpu's the process would only run 
as a single task which would then affine to a single cpu on a multiple 
cpu host.

So could kvm let an smp guest share the physical cpu's the host has?

Matt


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Problem with SMP boot

[Avi Kivity]

Matthew Hall wrote:
> Avi Kivity wrote:
>   
>> kvm only supports uniprocessor guests at this time.
>>     
>
> Out of interest, if/when

when.


>  kvm supports smp guests aswell as hosts would 
> the guest be presented with a multiple virtual or physical cpu's?
> Ie. If the guest has multiple virtual cpu's the process would only run 
> as a single task which would then affine to a single cpu on a multiple 
> cpu host.
>
> So could kvm let an smp guest share the physical cpu's the host has?
>   

kvm maps a guest vcpu to a host thread.  In turn, the host scheduler 
maps host threads to host cpus, so the mapping of vcpus to cpus depends 
on resource availability: if you have enough cpus, the scheduler will 
assign each vcpu to a separate cpu.

The user will also be able to affect the mapping by binding vcpu threads 
to specific processors.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: Problem with SMP boot

[Avi Kivity]

Tim Chen wrote:
> On Fri, 2007-01-26 at 19:58 +0200, Avi Kivity wrote:
>   
>>   
>>
>> kvm only supports uniprocessor guests at this time.
>>
>>     
>
> Thanks for the clarification.  Is SMP support for guests
> expected soon?
>   

For the next few weeks I plan to focus on stability, correctness, and a 
stable userspace interface.  I will accept patches outside these areas, 
though.


> The FAQ is a bit misleading: 
> http://kvm.qumranet.com/kvmwiki/FAQ
>
> It says "Does kvm support SMP hosts?  Yes". 
> I misinterpreted it meaning that SMP 
> support for guests is available.
>   

I added an entry for smp guests.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV