cpu hotplug

cpu hotplug

[Conrad Wood]

Hi everyone,

I'm currently looking into hotplugging CPUs. 
This exclusively with linux-guests and linux-host.
I have found some (conflicting) information about this on the net.

As far as I read:
* seabios doesn't support SMP (but why does the linux-guest show
multiple cpus?)
* there is no uniform way of informing the linux kernel that a cpu has
added (Found references to a unisys way of doing it though, which seems
to be still implemented)
* the kernel needs the acpi tables to determine how many cpus it has,
thus the support in the bios would involve an aml-compiler to
hot-compile new acpi tables.

1) *If* I understood this right, wouldn't it be feasible to patch the
linux-kernel to forego the check on the acpi tables and read them
straight from a kvm provided area (even in a kvm-proprietary format?).
Then at least the aml-compiler wouldn't be necessary, or am I wrong
there?

2) Assuming I understood this right, I'd further assume, I could create
some sort of kernel event (udev) to initiate a cpu_starting event,
couldn't I?

Some feedback would be much appreciated ;)


Conrad

Re: cpu hotplug

[Kevin O'Connor]

On Fri, Sep 17, 2010 at 11:39:36PM +0200, Conrad Wood wrote:
> Hi everyone,
> 
> I'm currently looking into hotplugging CPUs. 
> This exclusively with linux-guests and linux-host.
> I have found some (conflicting) information about this on the net.
> 
> As far as I read:
> * seabios doesn't support SMP (but why does the linux-guest show
> multiple cpus?)
> * there is no uniform way of informing the linux kernel that a cpu has
> added (Found references to a unisys way of doing it though, which seems
> to be still implemented)
> * the kernel needs the acpi tables to determine how many cpus it has,
> thus the support in the bios would involve an aml-compiler to
> hot-compile new acpi tables.

SeaBIOS version 0.6.1 has ACPI support for cpu-hotplug.  It's been
tested on Linux and Windows.  Search the mail archive for "cpu
hotplug" for the details.

-Kevin

Re: cpu hotplug

[Conrad Wood]

On Sat, 2010-09-18 at 10:32 -0400, Kevin O'Connor wrote:
> On Fri, Sep 17, 2010 at 11:39:36PM +0200, Conrad Wood wrote:
> > Hi everyone,
> > 
> > I'm currently looking into hotplugging CPUs. 
> > This exclusively with linux-guests and linux-host.
> > I have found some (conflicting) information about this on the net.
> > 
> > As far as I read:
> > * seabios doesn't support SMP (but why does the linux-guest show
> > multiple cpus?)
> > * there is no uniform way of informing the linux kernel that a cpu has
> > added (Found references to a unisys way of doing it though, which seems
> > to be still implemented)
> > * the kernel needs the acpi tables to determine how many cpus it has,
> > thus the support in the bios would involve an aml-compiler to
> > hot-compile new acpi tables.
> 
> SeaBIOS version 0.6.1 has ACPI support for cpu-hotplug.  It's been
> tested on Linux and Windows.  Search the mail archive for "cpu
> hotplug" for the details.
> 
> -Kevin
> 

Now... that would save some time ;)
Thank you! 

Conrad

Re: cpu hotplug

[Conrad Wood]

On Sat, 2010-09-18 at 10:32 -0400, Kevin O'Connor wrote:
> On Fri, Sep 17, 2010 at 11:39:36PM +0200, Conrad Wood wrote:
> > Hi everyone,
> > 
> > I'm currently looking into hotplugging CPUs. 
> > This exclusively with linux-guests and linux-host.
> > I have found some (conflicting) information about this on the net.
> > 
> > As far as I read:
> > * seabios doesn't support SMP (but why does the linux-guest show
> > multiple cpus?)
> > * there is no uniform way of informing the linux kernel that a cpu has
> > added (Found references to a unisys way of doing it though, which seems
> > to be still implemented)
> > * the kernel needs the acpi tables to determine how many cpus it has,
> > thus the support in the bios would involve an aml-compiler to
> > hot-compile new acpi tables.
> 
> SeaBIOS version 0.6.1 has ACPI support for cpu-hotplug.  It's been
> tested on Linux and Windows.  Search the mail archive for "cpu
> hotplug" for the details.
> 
> -Kevin
> 

hm... after upgrading to seabios 0.6.1 and qemu-kvm 0.13.50 (git today)
I get:

qemu-kvm/hw/qdev.c:94: qdev_create_from_info: Assertion
`bus->allow_hotplug' failed.

when I issue cpu_set 4 online on the cmd line. ;(

starting with commandline:

/usr/local/kvm/bin/qemu-system-x86_64 -M pc \
-bios /usr/local/kvm/seabios/latest \
-cpu phenom -enable-kvm -m 1024 -smp 4,cores=2,maxcpus=30,threads=1
-name testapp -uuid 5ee3d553-f6a4-0f33-6823-ee16a16a7fb4 -nodefaults \
-numa 2 \
-monitor telnet:localhost:9401,server,nowait -rtc base=utc -boot
${BOOTSTRAP} -device
rtl8139,vlan=0,id=net0,mac=52:54:01:65:e6:70,bus=pci.0,addr=0x4 \
-net tap,ifname=cnw0 -chardev pty,id=serial0 -device
isa-serial,chardev=serial0 -k en-gb -vga cirrus \
-drive
file=/srv/kvm/test-app.qcow2,if=virtio,cache=writeback,bus=0,id=a0,index=0,media=disk,format=qcow2,serial=1,boot=on,snapshot=${SNAPSHOT} \
-nodefaults \
-${WD} \
--enable-nesting \


any ideas ?

Conrad

Re: cpu hotplug

[Gleb Natapov]

On Sat, Sep 18, 2010 at 08:27:54PM +0200, Conrad Wood wrote:
> On Sat, 2010-09-18 at 10:32 -0400, Kevin O'Connor wrote:
> > On Fri, Sep 17, 2010 at 11:39:36PM +0200, Conrad Wood wrote:
> > > Hi everyone,
> > > 
> > > I'm currently looking into hotplugging CPUs. 
> > > This exclusively with linux-guests and linux-host.
> > > I have found some (conflicting) information about this on the net.
> > > 
> > > As far as I read:
> > > * seabios doesn't support SMP (but why does the linux-guest show
> > > multiple cpus?)
> > > * there is no uniform way of informing the linux kernel that a cpu has
> > > added (Found references to a unisys way of doing it though, which seems
> > > to be still implemented)
> > > * the kernel needs the acpi tables to determine how many cpus it has,
> > > thus the support in the bios would involve an aml-compiler to
> > > hot-compile new acpi tables.
> > 
> > SeaBIOS version 0.6.1 has ACPI support for cpu-hotplug.  It's been
> > tested on Linux and Windows.  Search the mail archive for "cpu
> > hotplug" for the details.
> > 
> > -Kevin
> > 
> 
> hm... after upgrading to seabios 0.6.1 and qemu-kvm 0.13.50 (git today)
> I get:
> 
> qemu-kvm/hw/qdev.c:94: qdev_create_from_info: Assertion
> `bus->allow_hotplug' failed.
> 
> when I issue cpu_set 4 online on the cmd line. ;(
> 
> starting with commandline:
> 
> /usr/local/kvm/bin/qemu-system-x86_64 -M pc \
> -bios /usr/local/kvm/seabios/latest \
> -cpu phenom -enable-kvm -m 1024 -smp 4,cores=2,maxcpus=30,threads=1
> -name testapp -uuid 5ee3d553-f6a4-0f33-6823-ee16a16a7fb4 -nodefaults \
> -numa 2 \
> -monitor telnet:localhost:9401,server,nowait -rtc base=utc -boot
> ${BOOTSTRAP} -device
> rtl8139,vlan=0,id=net0,mac=52:54:01:65:e6:70,bus=pci.0,addr=0x4 \
> -net tap,ifname=cnw0 -chardev pty,id=serial0 -device
> isa-serial,chardev=serial0 -k en-gb -vga cirrus \
> -drive
> file=/srv/kvm/test-app.qcow2,if=virtio,cache=writeback,bus=0,id=a0,index=0,media=disk,format=qcow2,serial=1,boot=on,snapshot=${SNAPSHOT} \
> -nodefaults \
> -${WD} \
> --enable-nesting \
> 
> 
> any ideas ?
> 
Known problem in qemu. There was a patch for this, but qemu maintainers
think it is not good enough.

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

On Sun, 2010-09-19 at 08:38 +0200, Gleb Natapov wrote:
> On Sat, Sep 18, 2010 at 08:27:54PM +0200, Conrad Wood wrote:
> > On Sat, 2010-09-18 at 10:32 -0400, Kevin O'Connor wrote:
> > > On Fri, Sep 17, 2010 at 11:39:36PM +0200, Conrad Wood wrote:
> > > > Hi everyone,
> > > > 
> > > > I'm currently looking into hotplugging CPUs. 
> > > > This exclusively with linux-guests and linux-host.
> > > > I have found some (conflicting) information about this on the net.
> > > > 
[...]

Well, I got a bit further - thanks so far for your help.
Given
1. kvm-72
2. linux-guest-kernel 2.6.35.4
3. kernel cmdline "maxcpus=32 possible_cpus=32"

I can set "cpu_set 5 online" in the monitor and get indeed a new cpu
appearing in /sys/devices..., which I can then set online in the guest
by doing echo 1 >/sys/devices/.../online.

/proc/cpuinfo in the guest then reports the new cpu.

But... why does the command "info cpus" in the monitor report all cpus
apart from CPU0 as (halted) ?

echo 1 >/sys/devices.../online makes no difference to what "info cpus"
reports, even though the guests /proc/cpuinfo shows expected behaviour.
Sorry to bother everyone, just trying to understand this.

How, if at all, are the guests' cpus mapped to physical cpus or cores on
the host?

Conrad

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 12:41:37PM +0200, Conrad Wood wrote:
> On Sun, 2010-09-19 at 08:38 +0200, Gleb Natapov wrote:
> > On Sat, Sep 18, 2010 at 08:27:54PM +0200, Conrad Wood wrote:
> > > On Sat, 2010-09-18 at 10:32 -0400, Kevin O'Connor wrote:
> > > > On Fri, Sep 17, 2010 at 11:39:36PM +0200, Conrad Wood wrote:
> > > > > Hi everyone,
> > > > > 
> > > > > I'm currently looking into hotplugging CPUs. 
> > > > > This exclusively with linux-guests and linux-host.
> > > > > I have found some (conflicting) information about this on the net.
> > > > > 
> [...]
> 
> Well, I got a bit further - thanks so far for your help.
> Given
> 1. kvm-72
> 2. linux-guest-kernel 2.6.35.4
> 3. kernel cmdline "maxcpus=32 possible_cpus=32"
> 
> I can set "cpu_set 5 online" in the monitor and get indeed a new cpu
> appearing in /sys/devices..., which I can then set online in the guest
> by doing echo 1 >/sys/devices/.../online.
> 
> /proc/cpuinfo in the guest then reports the new cpu.
> 
> But... why does the command "info cpus" in the monitor report all cpus
> apart from CPU0 as (halted) ?
> 
Because they are halted? Run multiple cpu hogs in the guest and see if
cpus will still be halted as reported by monitor. Also kvm-72 is so
ancient that I am not sure it reports cpu state correctly in monitor at
all.

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

On Sun, 2010-09-19 at 12:46 +0200, Gleb Natapov wrote:
[...]
> > But... why does the command "info cpus" in the monitor report all cpus
> > apart from CPU0 as (halted) ?
> > 
> Because they are halted? Run multiple cpu hogs in the guest and see if
> cpus will still be halted as reported by monitor. Also kvm-72 is so
> ancient that I am not sure it reports cpu state correctly in monitor at
> all.
> 
> --
fair enough, upgraded to 0.12.5, same results.

tried a cpu-hog (small loop forking 5 times and each fork doing some
integer arithmetics for a while) - no change in info cpus

thank you, but I still don't get it ;-)

Conrad

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 01:09:39PM +0200, Conrad Wood wrote:
> On Sun, 2010-09-19 at 12:46 +0200, Gleb Natapov wrote:
> [...]
> > > But... why does the command "info cpus" in the monitor report all cpus
> > > apart from CPU0 as (halted) ?
> > > 
> > Because they are halted? Run multiple cpu hogs in the guest and see if
> > cpus will still be halted as reported by monitor. Also kvm-72 is so
> > ancient that I am not sure it reports cpu state correctly in monitor at
> > all.
> > 
> > --
> fair enough, upgraded to 0.12.5, same results.
> 
> tried a cpu-hog (small loop forking 5 times and each fork doing some
> integer arithmetics for a while) - no change in info cpus
> 
> thank you, but I still don't get it ;-)
> 
At what rip are they halted? Try to run without kvm and see if the
result is the same. Can you provide output of "info cpus" here?
Are you using qemu-kvm or qemu BTW?

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

On Sun, 2010-09-19 at 13:13 +0200, Gleb Natapov wrote:
> On Sun, Sep 19, 2010 at 01:09:39PM +0200, Conrad Wood wrote:
> > On Sun, 2010-09-19 at 12:46 +0200, Gleb Natapov wrote:
> > [...]
> > > > But... why does the command "info cpus" in the monitor report all cpus
> > > > apart from CPU0 as (halted) ?
> > > > 
> > > Because they are halted? Run multiple cpu hogs in the guest and see if
> > > cpus will still be halted as reported by monitor. Also kvm-72 is so
> > > ancient that I am not sure it reports cpu state correctly in monitor at
> > > all.
> > > 
> > > --
> > fair enough, upgraded to 0.12.5, same results.
> > 
> > tried a cpu-hog (small loop forking 5 times and each fork doing some
> > integer arithmetics for a while) - no change in info cpus
> > 
> > thank you, but I still don't get it ;-)
> > 
> At what rip are they halted? Try to run without kvm and see if the
> result is the same. Can you provide output of "info cpus" here?
> Are you using qemu-kvm or qemu BTW?

I am using qemu-kvm (debian package Version: 0.12.5+dfsg-3).
Here's the "info cpus"  with and without kvm acceleration.
basically qemu shows the behavior I'd expect, kvm does not. Or do I
misunderstand something?

I wonder. if this is relevant: The host only has a single cpu, with 4
cores, I had expected those to be seen as cpus in the guest, similar how
linux lists each core as a cpu in the host.



************************************************************
kvm -no-kvm:

after guest has booted:
(qemu) info cpus
* CPU #0: pc=0xffffffff81047730 (halted) thread_id=17121
  CPU #1: pc=0xffffffff81047730 (halted) thread_id=17121
  CPU #2: pc=0xffffffff81047730 (halted) thread_id=17121
  CPU #3: pc=0xffffffff81047730 (halted) thread_id=17121

running cpuhog with 5 forks:
(qemu) info cpus
* CPU #0: pc=0x0000000000400698 thread_id=17121
  CPU #1: pc=0x0000000000400698 thread_id=17121
  CPU #2: pc=0x0000000000400698 thread_id=17121
  CPU #3: pc=0x0000000000400698 thread_id=17121

************************************************************
kvm -enable-kvm

after guest has booted:
(qemu) info cpus
* CPU #0: pc=0xffffffff81047730 thread_id=27082
  CPU #1: pc=0xffffffff81047730 (halted) thread_id=27083
  CPU #2: pc=0xffffffff81047730 (halted) thread_id=27084
  CPU #3: pc=0xffffffff81047730 (halted) thread_id=27085

running cpuhog with 5 forks:
(qemu) info cpus
* CPU #0: pc=0x00000000004006b6 thread_id=27082
  CPU #1: pc=0x00000000004006b6 (halted) thread_id=27083
  CPU #2: pc=0x00000000004006b6 (halted) thread_id=27084
  CPU #3: pc=0x00000000004006b6 (halted) thread_id=27085

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 01:32:28PM +0200, Conrad Wood wrote:
> On Sun, 2010-09-19 at 13:13 +0200, Gleb Natapov wrote:
> > On Sun, Sep 19, 2010 at 01:09:39PM +0200, Conrad Wood wrote:
> > > On Sun, 2010-09-19 at 12:46 +0200, Gleb Natapov wrote:
> > > [...]
> > > > > But... why does the command "info cpus" in the monitor report all cpus
> > > > > apart from CPU0 as (halted) ?
> > > > > 
> > > > Because they are halted? Run multiple cpu hogs in the guest and see if
> > > > cpus will still be halted as reported by monitor. Also kvm-72 is so
> > > > ancient that I am not sure it reports cpu state correctly in monitor at
> > > > all.
> > > > 
> > > > --
> > > fair enough, upgraded to 0.12.5, same results.
> > > 
> > > tried a cpu-hog (small loop forking 5 times and each fork doing some
> > > integer arithmetics for a while) - no change in info cpus
> > > 
> > > thank you, but I still don't get it ;-)
> > > 
> > At what rip are they halted? Try to run without kvm and see if the
> > result is the same. Can you provide output of "info cpus" here?
> > Are you using qemu-kvm or qemu BTW?
> 
> I am using qemu-kvm (debian package Version: 0.12.5+dfsg-3).
> Here's the "info cpus"  with and without kvm acceleration.
> basically qemu shows the behavior I'd expect, kvm does not. Or do I
> misunderstand something?
> 
No you are not. So your qemu-kvm version has cpu state reporting bug too. It is
not serious bug, just annoying. pc shows that otherwise everything is
working as expected.

> I wonder. if this is relevant: The host only has a single cpu, with 4
> cores, I had expected those to be seen as cpus in the guest, similar how
> linux lists each core as a cpu in the host.
Guest and host has different cpu topology.

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

On Sun, 2010-09-19 at 13:39 +0200, Gleb Natapov wrote:
> On Sun, Sep 19, 2010 at 01:32:28PM +0200, Conrad Wood wrote:
> > On Sun, 2010-09-19 at 13:13 +0200, Gleb Natapov wrote:
> > > On Sun, Sep 19, 2010 at 01:09:39PM +0200, Conrad Wood wrote:
> > > > On Sun, 2010-09-19 at 12:46 +0200, Gleb Natapov wrote:
> > > > [...]
> > > > > > But... why does the command "info cpus" in the monitor report all cpus
> > > > > > apart from CPU0 as (halted) ?
> > > > > > 
> > > > > Because they are halted? Run multiple cpu hogs in the guest and see if
> > > > > cpus will still be halted as reported by monitor. Also kvm-72 is so
> > > > > ancient that I am not sure it reports cpu state correctly in monitor at
> > > > > all.
> > > > > 
> > > > > --
> > > > fair enough, upgraded to 0.12.5, same results.
> > > > 
> > > > tried a cpu-hog (small loop forking 5 times and each fork doing some
> > > > integer arithmetics for a while) - no change in info cpus
> > > > 
> > > > thank you, but I still don't get it ;-)
> > > > 
> > > At what rip are they halted? Try to run without kvm and see if the
> > > result is the same. Can you provide output of "info cpus" here?
> > > Are you using qemu-kvm or qemu BTW?
> > 
> > I am using qemu-kvm (debian package Version: 0.12.5+dfsg-3).
> > Here's the "info cpus"  with and without kvm acceleration.
> > basically qemu shows the behavior I'd expect, kvm does not. Or do I
> > misunderstand something?
> > 
> No you are not. So your qemu-kvm version has cpu state reporting bug too. It is
> not serious bug, just annoying. pc shows that otherwise everything is
> working as expected.

I see. Is that a known bug, is someone working on that already?
If not... could you point me to the relevant code sections/docs where I
can fix that?
It actually is quite relevant for me, as I have to manage quite a lot of
VMs and need to be able to query how many cpus/cores they're using ;(

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 01:51:29PM +0200, Conrad Wood wrote:
> On Sun, 2010-09-19 at 13:39 +0200, Gleb Natapov wrote:
> > On Sun, Sep 19, 2010 at 01:32:28PM +0200, Conrad Wood wrote:
> > > On Sun, 2010-09-19 at 13:13 +0200, Gleb Natapov wrote:
> > > > On Sun, Sep 19, 2010 at 01:09:39PM +0200, Conrad Wood wrote:
> > > > > On Sun, 2010-09-19 at 12:46 +0200, Gleb Natapov wrote:
> > > > > [...]
> > > > > > > But... why does the command "info cpus" in the monitor report all cpus
> > > > > > > apart from CPU0 as (halted) ?
> > > > > > > 
> > > > > > Because they are halted? Run multiple cpu hogs in the guest and see if
> > > > > > cpus will still be halted as reported by monitor. Also kvm-72 is so
> > > > > > ancient that I am not sure it reports cpu state correctly in monitor at
> > > > > > all.
> > > > > > 
> > > > > > --
> > > > > fair enough, upgraded to 0.12.5, same results.
> > > > > 
> > > > > tried a cpu-hog (small loop forking 5 times and each fork doing some
> > > > > integer arithmetics for a while) - no change in info cpus
> > > > > 
> > > > > thank you, but I still don't get it ;-)
> > > > > 
> > > > At what rip are they halted? Try to run without kvm and see if the
> > > > result is the same. Can you provide output of "info cpus" here?
> > > > Are you using qemu-kvm or qemu BTW?
> > > 
> > > I am using qemu-kvm (debian package Version: 0.12.5+dfsg-3).
> > > Here's the "info cpus"  with and without kvm acceleration.
> > > basically qemu shows the behavior I'd expect, kvm does not. Or do I
> > > misunderstand something?
> > > 
> > No you are not. So your qemu-kvm version has cpu state reporting bug too. It is
> > not serious bug, just annoying. pc shows that otherwise everything is
> > working as expected.
> 
> I see. Is that a known bug, is someone working on that already?
> If not... could you point me to the relevant code sections/docs where I
> can fix that?
> It actually is quite relevant for me, as I have to manage quite a lot of
> VMs and need to be able to query how many cpus/cores they're using ;(
> 
I fixed it several times in the past[0][1][3], but it keeps re-appearing :) I
hope current qemu-kvm head is OK.

[0] 865aa3fec2b3d83c9b6edd452050a30ef6eaf3b8 
[1] 8c7d4cb7163d1c97df4439e2e0edb5918abf106b
[2] d6d233543d543fcf34df6190bbce5daf38d870f5

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

On Sun, 2010-09-19 at 13:59 +0200, Gleb Natapov wrote:
> On Sun, Sep 19, 2010 at 01:51:29PM +0200, Conrad Wood wrote:
> > On Sun, 2010-09-19 at 13:39 +0200, Gleb Natapov wrote:
> > > On Sun, Sep 19, 2010 at 01:32:28PM +0200, Conrad Wood wrote:
> > > > On Sun, 2010-09-19 at 13:13 +0200, Gleb Natapov wrote:
> > > > > On Sun, Sep 19, 2010 at 01:09:39PM +0200, Conrad Wood wrote:
> > > > > > On Sun, 2010-09-19 at 12:46 +0200, Gleb Natapov wrote:
> > > > > > [...]
> > > > > > > > But... why does the command "info cpus" in the monitor report all cpus
> > > > > > > > apart from CPU0 as (halted) ?
> > > > > > > > 
> > > > > > > Because they are halted? Run multiple cpu hogs in the guest and see if
> > > > > > > cpus will still be halted as reported by monitor. Also kvm-72 is so
> > > > > > > ancient that I am not sure it reports cpu state correctly in monitor at
> > > > > > > all.
> > > > > > > 
> > > > > > > --
> > > > > > fair enough, upgraded to 0.12.5, same results.
> > > > > > 
> > > > > > tried a cpu-hog (small loop forking 5 times and each fork doing some
> > > > > > integer arithmetics for a while) - no change in info cpus
> > > > > > 
> > > > > > thank you, but I still don't get it ;-)
> > > > > > 
> > > > > At what rip are they halted? Try to run without kvm and see if the
> > > > > result is the same. Can you provide output of "info cpus" here?
> > > > > Are you using qemu-kvm or qemu BTW?
> > > > 
> > > > I am using qemu-kvm (debian package Version: 0.12.5+dfsg-3).
> > > > Here's the "info cpus"  with and without kvm acceleration.
> > > > basically qemu shows the behavior I'd expect, kvm does not. Or do I
> > > > misunderstand something?
> > > > 
> > > No you are not. So your qemu-kvm version has cpu state reporting bug too. It is
> > > not serious bug, just annoying. pc shows that otherwise everything is
> > > working as expected.
> > 
> > I see. Is that a known bug, is someone working on that already?
> > If not... could you point me to the relevant code sections/docs where I
> > can fix that?
> > It actually is quite relevant for me, as I have to manage quite a lot of
> > VMs and need to be able to query how many cpus/cores they're using ;(
> > 
> I fixed it several times in the past[0][1][3], but it keeps re-appearing :) I
> hope current qemu-kvm head is OK.
> 
> [0] 865aa3fec2b3d83c9b6edd452050a30ef6eaf3b8 
> [1] 8c7d4cb7163d1c97df4439e2e0edb5918abf106b
> [2] d6d233543d543fcf34df6190bbce5daf38d870f5

got it.
I'll keep an eye on head and once cpu-hotplug works with head I'll test
the state reporting ;)
Thank you!!!

Conrad

Re: cpu hotplug

[Avi Kivity]

  On 09/19/2010 01:59 PM, Gleb Natapov wrote:
> >
> I fixed it several times in the past[0][1][3], but it keeps re-appearing :) I
> hope current qemu-kvm head is OK.
>
> [0] 865aa3fec2b3d83c9b6edd452050a30ef6eaf3b8
> [1] 8c7d4cb7163d1c97df4439e2e0edb5918abf106b
> [2] d6d233543d543fcf34df6190bbce5daf38d870f5
>

kvm-autotest and you won't have to fix it again...

-- 
error compiling committee.c: too many arguments to function

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 05:20:38PM +0200, Avi Kivity wrote:
>  On 09/19/2010 01:59 PM, Gleb Natapov wrote:
> >>
> >I fixed it several times in the past[0][1][3], but it keeps re-appearing :) I
> >hope current qemu-kvm head is OK.
> >
> >[0] 865aa3fec2b3d83c9b6edd452050a30ef6eaf3b8
> >[1] 8c7d4cb7163d1c97df4439e2e0edb5918abf106b
> >[2] d6d233543d543fcf34df6190bbce5daf38d870f5
> >
> 
> kvm-autotest and you won't have to fix it again...
> 
Heh. Can't argue with this.

--
			Gleb.

Re: cpu hotplug

[Kevin O'Connor]

On Sun, Sep 19, 2010 at 08:38:12AM +0200, Gleb Natapov wrote:
> On Sat, Sep 18, 2010 at 08:27:54PM +0200, Conrad Wood wrote:
> > hm... after upgrading to seabios 0.6.1 and qemu-kvm 0.13.50 (git today)
> > I get:
[...]
> > any ideas ?
> > 
> Known problem in qemu. There was a patch for this, but qemu maintainers
> think it is not good enough.

Old versions of kvm do work - for example, qemu-kvm-0.12.5 works for
me.

-Kevin

Re: cpu hotplug

[Conrad Wood]

On Sun, 2010-09-19 at 09:26 -0400, Kevin O'Connor wrote:
> On Sun, Sep 19, 2010 at 08:38:12AM +0200, Gleb Natapov wrote:
> > On Sat, Sep 18, 2010 at 08:27:54PM +0200, Conrad Wood wrote:
> > > hm... after upgrading to seabios 0.6.1 and qemu-kvm 0.13.50 (git today)
> > > I get:
> [...]
> > > any ideas ?
> > > 
> > Known problem in qemu. There was a patch for this, but qemu maintainers
> > think it is not good enough.
> 
> Old versions of kvm do work - for example, qemu-kvm-0.12.5 works for
> me.
> 
> -Kevin
> 

hm... that's the version I am using und it does not work here ;(
the hotplug itself seems to work but they're reported as halted.
Could you confirm that the cpus in the monitor as reported by "info
cpus" are not reported as "(halted)" ?

Then it seems to be a local misconfiguration.

(with -enable-kvm on amd)

Thanks!

Conrad

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 03:29:35PM +0200, Conrad Wood wrote:
> On Sun, 2010-09-19 at 09:26 -0400, Kevin O'Connor wrote:
> > On Sun, Sep 19, 2010 at 08:38:12AM +0200, Gleb Natapov wrote:
> > > On Sat, Sep 18, 2010 at 08:27:54PM +0200, Conrad Wood wrote:
> > > > hm... after upgrading to seabios 0.6.1 and qemu-kvm 0.13.50 (git today)
> > > > I get:
> > [...]
> > > > any ideas ?
> > > > 
> > > Known problem in qemu. There was a patch for this, but qemu maintainers
> > > think it is not good enough.
> > 
> > Old versions of kvm do work - for example, qemu-kvm-0.12.5 works for
> > me.
> > 
> > -Kevin
> > 
> 
> hm... that's the version I am using und it does not work here ;(
> the hotplug itself seems to work but they're reported as halted.
I told you this is cosmetic. They are working.

> Could you confirm that the cpus in the monitor as reported by "info
> cpus" are not reported as "(halted)" ?
> 
> Then it seems to be a local misconfiguration.
> 
> (with -enable-kvm on amd)
> 
> Thanks!
> 
> Conrad

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

> > hm... that's the version I am using und it does not work here ;(
> > the hotplug itself seems to work but they're reported as halted.
> I told you this is cosmetic. They are working.

I agree, the hotplug itself seems to be working!

> > 
> > Then it seems to be a local misconfiguration.

and if so, I would like to know what I have misconfigured. ;-)
and what I can do to persuade this version to provide me with
information about which cpus are online and offline.

Conrad

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 03:35:47PM +0200, Conrad Wood wrote:
> 
> > > hm... that's the version I am using und it does not work here ;(
> > > the hotplug itself seems to work but they're reported as halted.
> > I told you this is cosmetic. They are working.
> 
> I agree, the hotplug itself seems to be working!
So why are you saying "I am using und it does not work here" above?
> 
> > > 
> > > Then it seems to be a local misconfiguration.
> 
> and if so, I would like to know what I have misconfigured. ;-)
> and what I can do to persuade this version to provide me with
> information about which cpus are online and offline.
> 
halted state is not the way to check for whether cpu is online or
offline. cpu may be online but executing hlt instruction so 
its state will be halted, but cpu itself is online. Actually with kvm
today you are not able to check that cpu is offline if it was ever
online. Query a guest for cpu status.

--
			Gleb.

Re: cpu hotplug

[Kevin O'Connor]

On Sun, Sep 19, 2010 at 03:40:45PM +0200, Gleb Natapov wrote:
> halted state is not the way to check for whether cpu is online or
> offline. cpu may be online but executing hlt instruction so 
> its state will be halted, but cpu itself is online. Actually with kvm
> today you are not able to check that cpu is offline if it was ever
> online. Query a guest for cpu status.

In that regard, "info cpus" does seem to work - as it only shows me
the cpus that have ever been online.  Ejecting a cpu in the guest does
not remove it from "info cpus", but then, I don't think kvm does
remove it from being available after an eject.

-Kevin

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 09:46:21AM -0400, Kevin O'Connor wrote:
> On Sun, Sep 19, 2010 at 03:40:45PM +0200, Gleb Natapov wrote:
> > halted state is not the way to check for whether cpu is online or
> > offline. cpu may be online but executing hlt instruction so 
> > its state will be halted, but cpu itself is online. Actually with kvm
> > today you are not able to check that cpu is offline if it was ever
> > online. Query a guest for cpu status.
> 
> In that regard, "info cpus" does seem to work - as it only shows me
> the cpus that have ever been online.  Ejecting a cpu in the guest does
> not remove it from "info cpus", but then, I don't think kvm does
> remove it from being available after an eject.
> 
Exactly. That what I meant by "Actually with kvm today you are not able
to check that cpu is offline if it was ever online". So if cpu was never
online it is easy to check that it is offline (it will not be reported
by "info cpus"). But if it was inserted and then ejected it will still
be present in "info cpus", but not used by a guest OS.

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

On Sun, 2010-09-19 at 15:50 +0200, Gleb Natapov wrote:
> On Sun, Sep 19, 2010 at 09:46:21AM -0400, Kevin O'Connor wrote:
> > On Sun, Sep 19, 2010 at 03:40:45PM +0200, Gleb Natapov wrote:
> > > halted state is not the way to check for whether cpu is online or
> > > offline. cpu may be online but executing hlt instruction so 
> > > its state will be halted, but cpu itself is online. Actually with kvm
> > > today you are not able to check that cpu is offline if it was ever
> > > online. Query a guest for cpu status.
> > 
> > In that regard, "info cpus" does seem to work - as it only shows me
> > the cpus that have ever been online.  Ejecting a cpu in the guest does
> > not remove it from "info cpus", but then, I don't think kvm does
> > remove it from being available after an eject.
> > 
> Exactly. That what I meant by "Actually with kvm today you are not able
> to check that cpu is offline if it was ever online". So if cpu was never
> online it is easy to check that it is offline (it will not be reported
> by "info cpus"). But if it was inserted and then ejected it will still
> be present in "info cpus", but not used by a guest OS.
> 
> --
> 			Gleb.
> 

1) Thanks for clarifying "online" vs "halted" - that makes sense and is
probably part of what confused me. I need to get the "online/offline"
status of cpus, not if they are halted or not. I understand this is
currently not possible with kvm. As I have no direct control over the
guest OS, I do however need a solution on the host. Any clues how this
could be implemented would be much appreciated.
This is important to me.

2) In my case, the cpus are ALWAYS reported as halted, regardless of the
current state in the guest. 
After the guest booted, The "cpu infos" displays precisely the amount of
cpus as initially set by the -smp parameter of kvm. (as halted).
any additional cpus added later will be displayed, also as halted -
again regardless whether or not the guest switches or had switched them
to online. It is reported as "halted" even whilst the guest is running a
cpu-hogger which would stop the cpu from halting.
I don't mind so much about this cosmetic display problem ;) 


Conrad

Re: cpu hotplug

[Kevin O'Connor]

On Sun, Sep 19, 2010 at 04:07:50PM +0200, Conrad Wood wrote:
> 1) Thanks for clarifying "online" vs "halted" - that makes sense and is
> probably part of what confused me. I need to get the "online/offline"
> status of cpus, not if they are halted or not. I understand this is
> currently not possible with kvm. As I have no direct control over the
> guest OS, I do however need a solution on the host. Any clues how this
> could be implemented would be much appreciated.
> This is important to me.

I thought I read somewhere that kvm does not currently support taking
a cpu offline - the guest can only voluntarily stop using a cpu.
(Gleb would know better, but I think he was trying to say the same
thing.)

> 2) In my case, the cpus are ALWAYS reported as halted, regardless of the
> current state in the guest. 
> After the guest booted, The "cpu infos" displays precisely the amount of
> cpus as initially set by the -smp parameter of kvm. (as halted).
> any additional cpus added later will be displayed, also as halted -
> again regardless whether or not the guest switches or had switched them
> to online. It is reported as "halted" even whilst the guest is running a
> cpu-hogger which would stop the cpu from halting.
> I don't mind so much about this cosmetic display problem ;) 

I suppose one could check the host cpu usage of the thread reported in
"info cpus".

-Kevin

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 04:07:50PM +0200, Conrad Wood wrote:
> On Sun, 2010-09-19 at 15:50 +0200, Gleb Natapov wrote:
> > On Sun, Sep 19, 2010 at 09:46:21AM -0400, Kevin O'Connor wrote:
> > > On Sun, Sep 19, 2010 at 03:40:45PM +0200, Gleb Natapov wrote:
> > > > halted state is not the way to check for whether cpu is online or
> > > > offline. cpu may be online but executing hlt instruction so 
> > > > its state will be halted, but cpu itself is online. Actually with kvm
> > > > today you are not able to check that cpu is offline if it was ever
> > > > online. Query a guest for cpu status.
> > > 
> > > In that regard, "info cpus" does seem to work - as it only shows me
> > > the cpus that have ever been online.  Ejecting a cpu in the guest does
> > > not remove it from "info cpus", but then, I don't think kvm does
> > > remove it from being available after an eject.
> > > 
> > Exactly. That what I meant by "Actually with kvm today you are not able
> > to check that cpu is offline if it was ever online". So if cpu was never
> > online it is easy to check that it is offline (it will not be reported
> > by "info cpus"). But if it was inserted and then ejected it will still
> > be present in "info cpus", but not used by a guest OS.
> > 
> > --
> > 			Gleb.
> > 
> 
> 1) Thanks for clarifying "online" vs "halted" - that makes sense and is
> probably part of what confused me. I need to get the "online/offline"
> status of cpus, not if they are halted or not. I understand this is
> currently not possible with kvm. As I have no direct control over the
> guest OS, I do however need a solution on the host. Any clues how this
> could be implemented would be much appreciated.
> This is important to me.
Lets establish some terminology to understand each other better:
Lets use CPU inserted/ejected to refer to cpu being available/not
available to a guest.
Lest use CPU online/offline to refer to cpu been used by a guest OS.

CPU hot-plug works like this:
  cpu is inserted (plugged into cpu socket on MB)
  guest OS is notifies through ACPI
  guest OS starts to use newly added CPU (makes it online)

CPU hot-unplug works like this: 
  guest OS is notifies through ACPI that cpu is going to be removed
  guest OS stops to use cpu (makes it offline)
  cpu is ejected (removed from a socket on MB) (kvm never does that part)

So which part do you want to know about from a host?

You can't really know from a host if cpu is used by guest OS (if it is
online/offline in s guest in our terminology)

inserted/ejected is management action (it can't be initiated by a
guest), so you can keep track of it by yourself. KVM doesn't do it
for you right now. More precisely it does, but doesn't have a way to
quire it.

> 
> 2) In my case, the cpus are ALWAYS reported as halted, regardless of the
> current state in the guest. 
This is cosmetic but. It does not affect any operation.

> After the guest booted, The "cpu infos" displays precisely the amount of
> cpus as initially set by the -smp parameter of kvm. (as halted).
> any additional cpus added later will be displayed, also as halted -
> again regardless whether or not the guest switches or had switched them
> to online. It is reported as "halted" even whilst the guest is running a
> cpu-hogger which would stop the cpu from halting.
> I don't mind so much about this cosmetic display problem ;) 
> 
Rightly so since even when it works information it displays is already
outdated at the moment you see it anyway. It is useful for debugging,
thus I keep fixing it when it breaks.

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

[..]

> Lets establish some terminology to understand each other better:
> Lets use CPU inserted/ejected to refer to cpu being available/not
> available to a guest.
> Lest use CPU online/offline to refer to cpu been used by a guest OS.
> 
> CPU hot-plug works like this:
>   cpu is inserted (plugged into cpu socket on MB)
>   guest OS is notifies through ACPI
>   guest OS starts to use newly added CPU (makes it online)
> 
> CPU hot-unplug works like this: 
>   guest OS is notifies through ACPI that cpu is going to be removed
>   guest OS stops to use cpu (makes it offline)
>   cpu is ejected (removed from a socket on MB) (kvm never does that part)
> 
> So which part do you want to know about from a host?
> 
> You can't really know from a host if cpu is used by guest OS (if it is
> online/offline in s guest in our terminology)
> 
> inserted/ejected is management action (it can't be initiated by a
> guest), so you can keep track of it by yourself. KVM doesn't do it
> for you right now. More precisely it does, but doesn't have a way to
> quire it.

I need to know if a cpu is currently inserted or ejected.

I gather a cpu which is currently "inserted", may potentially be used by
a guest os. (Wether he actually uses a cpu, puts it online or offline,
isn't my concern). 

The amount of time and number of cpus available to a guest is what I
like to keep track of.

Conrad

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 05:16:24PM +0200, Conrad Wood wrote:
> [..]
> 
> > Lets establish some terminology to understand each other better:
> > Lets use CPU inserted/ejected to refer to cpu being available/not
> > available to a guest.
> > Lest use CPU online/offline to refer to cpu been used by a guest OS.
> > 
> > CPU hot-plug works like this:
> >   cpu is inserted (plugged into cpu socket on MB)
> >   guest OS is notifies through ACPI
> >   guest OS starts to use newly added CPU (makes it online)
> > 
> > CPU hot-unplug works like this: 
> >   guest OS is notifies through ACPI that cpu is going to be removed
> >   guest OS stops to use cpu (makes it offline)
> >   cpu is ejected (removed from a socket on MB) (kvm never does that part)
> > 
> > So which part do you want to know about from a host?
> > 
> > You can't really know from a host if cpu is used by guest OS (if it is
> > online/offline in s guest in our terminology)
> > 
> > inserted/ejected is management action (it can't be initiated by a
> > guest), so you can keep track of it by yourself. KVM doesn't do it
> > for you right now. More precisely it does, but doesn't have a way to
> > quire it.
> 
> I need to know if a cpu is currently inserted or ejected.
> 
The only way to do that currently is to track it in your management app.

> I gather a cpu which is currently "inserted", may potentially be used by
> a guest os. (Wether he actually uses a cpu, puts it online or offline,
> isn't my concern). 
> 
> The amount of time and number of cpus available to a guest is what I
> like to keep track of.
> 
> Conrad
> 

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

> > I need to know if a cpu is currently inserted or ejected.
> > 
> The only way to do that currently is to track it in your management app.

The Management App does know which one should be inserted or ejected.
I see now way of enforcing it though.

This is what I had in mind: (simplified)

1. Management App increase cpus
2. kvm (as instructed by app): insert cpu x
3. guest os: online cpu x
4. do some stuff in the guest...
5. guest os: offline cpu x
6. Management App decrease cpus
7. kvm (as instructed by app): eject cpu x 

However after step 7 the guest can turn the cpu online again by issuing 
echo 1 >/sys/devices/.../cpuX/online
If the guest does so, the Management App has no way of knowing and thus
cannot provide accurate logs - that is why it is so important.

Conrad

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 05:37:18PM +0200, Conrad Wood wrote:
> 
> > > I need to know if a cpu is currently inserted or ejected.
> > > 
> > The only way to do that currently is to track it in your management app.
> 
> The Management App does know which one should be inserted or ejected.
> I see now way of enforcing it though.
> 
> This is what I had in mind: (simplified)
> 
> 1. Management App increase cpus
> 2. kvm (as instructed by app): insert cpu x
> 3. guest os: online cpu x
> 4. do some stuff in the guest...
> 5. guest os: offline cpu x
> 6. Management App decrease cpus
> 7. kvm (as instructed by app): eject cpu x 
> 
> However after step 7 the guest can turn the cpu online again by issuing 
> echo 1 >/sys/devices/.../cpuX/online
There will be no /sys/devices/.../cpuX/online in guest after step 7.

> If the guest does so, the Management App has no way of knowing and thus
> cannot provide accurate logs - that is why it is so important.
> 
> Conrad
> 
> 

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

> > However after step 7 the guest can turn the cpu online again by issuing 
> > echo 1 >/sys/devices/.../cpuX/online
> There will be no /sys/devices/.../cpuX/online in guest after step 7.

Well then at least in my version there's a bug, because it still is
there and never goes away. (kvm 0.12.5)
And changing it to online works as well.
Which version are you using?


Conrad

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 05:44:00PM +0200, Conrad Wood wrote:
> 
> > > However after step 7 the guest can turn the cpu online again by issuing 
> > > echo 1 >/sys/devices/.../cpuX/online
> > There will be no /sys/devices/.../cpuX/online in guest after step 7.
> 
> Well then at least in my version there's a bug, because it still is
> there and never goes away. (kvm 0.12.5)
> And changing it to online works as well.
Can you provide exact steps you are using? After doing cpu_set x offline
cpu x should disappear from a guest. Otherwise cpu eject didn't work.

> Which version are you using?
> 
Nothing :) I am saying how it should work. Last time I played with cpu
hot-plug was with qemu-kvm 0.11.x.

--
			Gleb.

Re: cpu hotplug

[Kevin O'Connor]

On Sun, Sep 19, 2010 at 05:53:47PM +0200, Gleb Natapov wrote:
> On Sun, Sep 19, 2010 at 05:44:00PM +0200, Conrad Wood wrote:
> > 
> > > > However after step 7 the guest can turn the cpu online again by issuing 
> > > > echo 1 >/sys/devices/.../cpuX/online
> > > There will be no /sys/devices/.../cpuX/online in guest after step 7.
> > 
> > Well then at least in my version there's a bug, because it still is
> > there and never goes away. (kvm 0.12.5)
> > And changing it to online works as well.
> Can you provide exact steps you are using? After doing cpu_set x offline
> cpu x should disappear from a guest. Otherwise cpu eject didn't work.

I'm confused.  The "cpu_set x offline" doesn't appear to do anything
for me.  It does not disable the cpu, nor does it inform the guest to
disable the cpu.

The only way I'm able to remove a cpu is to run in the guest:

echo 1 > /sys/devices/LNXSYSTM:00/LNXSYBUS:00/LNXCPU:xx/eject

which (as far as I know) is purely a voluntary guest disable of the
cpu.  It doesn't (as far as I know) even tell kvm that it is no longer
using the cpu.  (The acpi cpu eject method in seabios just issues a
sleep(200) call.)

-Kevin

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 12:04:51PM -0400, Kevin O'Connor wrote:
> On Sun, Sep 19, 2010 at 05:53:47PM +0200, Gleb Natapov wrote:
> > On Sun, Sep 19, 2010 at 05:44:00PM +0200, Conrad Wood wrote:
> > > 
> > > > > However after step 7 the guest can turn the cpu online again by issuing 
> > > > > echo 1 >/sys/devices/.../cpuX/online
> > > > There will be no /sys/devices/.../cpuX/online in guest after step 7.
> > > 
> > > Well then at least in my version there's a bug, because it still is
> > > there and never goes away. (kvm 0.12.5)
> > > And changing it to online works as well.
> > Can you provide exact steps you are using? After doing cpu_set x offline
> > cpu x should disappear from a guest. Otherwise cpu eject didn't work.
> 
> I'm confused.  The "cpu_set x offline" doesn't appear to do anything
> for me.  It does not disable the cpu, nor does it inform the guest to
> disable the cpu.
> 
> The only way I'm able to remove a cpu is to run in the guest:
> 
> echo 1 > /sys/devices/LNXSYSTM:00/LNXSYBUS:00/LNXCPU:xx/eject
> 
> which (as far as I know) is purely a voluntary guest disable of the
> cpu.  It doesn't (as far as I know) even tell kvm that it is no longer
> using the cpu.  (The acpi cpu eject method in seabios just issues a
> sleep(200) call.)
> 
Then cpu eject method in seabios has a bug. When "cpu_set x offline" is
called qemu sets status bit in gpe and injects ACPI interrupt. ACPI
should do Notify() on cpu object when it happens. 

--
			Gleb.

Re: cpu hotplug

[Kevin O'Connor]

On Sun, Sep 19, 2010 at 06:12:14PM +0200, Gleb Natapov wrote:
> On Sun, Sep 19, 2010 at 12:04:51PM -0400, Kevin O'Connor wrote:
> > I'm confused.  The "cpu_set x offline" doesn't appear to do anything
> > for me.  It does not disable the cpu, nor does it inform the guest to
> > disable the cpu.
[...]
> Then cpu eject method in seabios has a bug. When "cpu_set x offline" is
> called qemu sets status bit in gpe and injects ACPI interrupt. ACPI
> should do Notify() on cpu object when it happens. 

I was wrong.  The "cpu_set x offline" does send an event to the guest
OS.  SeaBIOS even forwards the event along - as far as I can tell a
Notify(CPxx, 3) event is generated by SeaBIOS.

My Windows 7 ultimate beta seems to receive the event okay (it pops up
a dialog box which says you can't unplug cpus).

Unfortunately, my test linux guest OS (FC13) doesn't seem to do
anything with the unplug Notify event.  I've tried with the original
FC13 and with a fully updated version - no luck.

So, I'm guessing this has something to do with the guest OS.

-Kevin

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 06:03:31PM -0400, Kevin O'Connor wrote:
> On Sun, Sep 19, 2010 at 06:12:14PM +0200, Gleb Natapov wrote:
> > On Sun, Sep 19, 2010 at 12:04:51PM -0400, Kevin O'Connor wrote:
> > > I'm confused.  The "cpu_set x offline" doesn't appear to do anything
> > > for me.  It does not disable the cpu, nor does it inform the guest to
> > > disable the cpu.
> [...]
> > Then cpu eject method in seabios has a bug. When "cpu_set x offline" is
> > called qemu sets status bit in gpe and injects ACPI interrupt. ACPI
> > should do Notify() on cpu object when it happens. 
> 
> I was wrong.  The "cpu_set x offline" does send an event to the guest
> OS.  SeaBIOS even forwards the event along - as far as I can tell a
> Notify(CPxx, 3) event is generated by SeaBIOS.
> 
> My Windows 7 ultimate beta seems to receive the event okay (it pops up
> a dialog box which says you can't unplug cpus).
> 
It may react to Eject() method.

> Unfortunately, my test linux guest OS (FC13) doesn't seem to do
> anything with the unplug Notify event.  I've tried with the original
> FC13 and with a fully updated version - no luck.
> 
> So, I'm guessing this has something to do with the guest OS.
> 
Can you verify that _STA() return zero after cpu unplug?

--
			Gleb.

Re: cpu hotplug

[Kevin O'Connor]

On Mon, Sep 20, 2010 at 08:50:17AM +0200, Gleb Natapov wrote:
> On Sun, Sep 19, 2010 at 06:03:31PM -0400, Kevin O'Connor wrote:
> > I was wrong.  The "cpu_set x offline" does send an event to the guest
> > OS.  SeaBIOS even forwards the event along - as far as I can tell a
> > Notify(CPxx, 3) event is generated by SeaBIOS.
> > 
> > My Windows 7 ultimate beta seems to receive the event okay (it pops up
> > a dialog box which says you can't unplug cpus).
> > 
> It may react to Eject() method.

The eject method is called by the OS to notify the host.  Right now
SeaBIOS's eject method doesn't do anything.

> > Unfortunately, my test linux guest OS (FC13) doesn't seem to do
> > anything with the unplug Notify event.  I've tried with the original
> > FC13 and with a fully updated version - no luck.
> > 
> > So, I'm guessing this has something to do with the guest OS.
> > 
> Can you verify that _STA() return zero after cpu unplug?

I've verified that.  I've also verified that Linux doesn't call the
_STA method after Notify(CPxx, 3).  It does call _STA on startup and
after a Notify(CPxx, 1) event.  So, the Linux kernel in my FC13 guest
just seems to be ignoring Notify(3) events.  (According to ACPI spec,
the guest should shutdown the cpu and then call the eject method.)

-Kevin

Re: cpu hotplug

[Gleb Natapov]

On Mon, Sep 20, 2010 at 09:07:16PM -0400, Kevin O'Connor wrote:
> On Mon, Sep 20, 2010 at 08:50:17AM +0200, Gleb Natapov wrote:
> > On Sun, Sep 19, 2010 at 06:03:31PM -0400, Kevin O'Connor wrote:
> > > I was wrong.  The "cpu_set x offline" does send an event to the guest
> > > OS.  SeaBIOS even forwards the event along - as far as I can tell a
> > > Notify(CPxx, 3) event is generated by SeaBIOS.
> > > 
> > > My Windows 7 ultimate beta seems to receive the event okay (it pops up
> > > a dialog box which says you can't unplug cpus).
> > > 
> > It may react to Eject() method.
> 
> The eject method is called by the OS to notify the host.  Right now
> SeaBIOS's eject method doesn't do anything.
> 
Yes. What I meant is it may react on presence of Eject() method. In my
experience Windows consider all devices with Eject() method as
hot-pluggable. But actually IIRC Windows 7 gave me this dialog box with
BOCHS BIOS too and there we didn't have Eject() method.

> > > Unfortunately, my test linux guest OS (FC13) doesn't seem to do
> > > anything with the unplug Notify event.  I've tried with the original
> > > FC13 and with a fully updated version - no luck.
> > > 
> > > So, I'm guessing this has something to do with the guest OS.
> > > 
> > Can you verify that _STA() return zero after cpu unplug?
> 
> I've verified that.  I've also verified that Linux doesn't call the
> _STA method after Notify(CPxx, 3).  It does call _STA on startup and
> after a Notify(CPxx, 1) event.  So, the Linux kernel in my FC13 guest
> just seems to be ignoring Notify(3) events.  (According to ACPI spec,
> the guest should shutdown the cpu and then call the eject method.)
> 
In older kernels _STA was called on Notify(3), but recently cpu hot-plug
in Linux was changed. Can you check what happens if you'll call
Notify(1) on unplug? Spec says that the value is:

 Device Check. Used to notify OSPM that the device either
 appeared or disappeared.

so may be it should be called on hot-plug and hot-unplug.

--
			Gleb.

Re: cpu hotplug

[Gleb Natapov]

On Mon, Sep 20, 2010 at 09:07:16PM -0400, Kevin O'Connor wrote:
> On Mon, Sep 20, 2010 at 08:50:17AM +0200, Gleb Natapov wrote:
> > On Sun, Sep 19, 2010 at 06:03:31PM -0400, Kevin O'Connor wrote:
> > > I was wrong.  The "cpu_set x offline" does send an event to the guest
> > > OS.  SeaBIOS even forwards the event along - as far as I can tell a
> > > Notify(CPxx, 3) event is generated by SeaBIOS.
> > > 
> > > My Windows 7 ultimate beta seems to receive the event okay (it pops up
> > > a dialog box which says you can't unplug cpus).
> > > 
> > It may react to Eject() method.
> 
> The eject method is called by the OS to notify the host.  Right now
> SeaBIOS's eject method doesn't do anything.
> 
> > > Unfortunately, my test linux guest OS (FC13) doesn't seem to do
> > > anything with the unplug Notify event.  I've tried with the original
> > > FC13 and with a fully updated version - no luck.
> > > 
> > > So, I'm guessing this has something to do with the guest OS.
> > > 
> > Can you verify that _STA() return zero after cpu unplug?
> 
> I've verified that.  I've also verified that Linux doesn't call the
> _STA method after Notify(CPxx, 3).  It does call _STA on startup and
> after a Notify(CPxx, 1) event.  So, the Linux kernel in my FC13 guest
> just seems to be ignoring Notify(3) events.  (According to ACPI spec,
> the guest should shutdown the cpu and then call the eject method.)
> 
To remove cpu completely you need to eject it from the guest:
echo 1 > /sys/bus/acpi/drivers/processor/LNXCPU\:03/eject

I didn't found a way to trigger this from a host. May be with some udev
magic it can be done.

--
			Gleb.

Re: cpu hotplug

[Conrad Wood]

On Sun, 2010-09-19 at 12:04 -0400, Kevin O'Connor wrote:
> On Sun, Sep 19, 2010 at 05:53:47PM +0200, Gleb Natapov wrote:
> > On Sun, Sep 19, 2010 at 05:44:00PM +0200, Conrad Wood wrote:
> > > 
> > > > > However after step 7 the guest can turn the cpu online again by issuing 
> > > > > echo 1 >/sys/devices/.../cpuX/online
> > > > There will be no /sys/devices/.../cpuX/online in guest after step 7.
> > > 
> > > Well then at least in my version there's a bug, because it still is
> > > there and never goes away. (kvm 0.12.5)
> > > And changing it to online works as well.
> > Can you provide exact steps you are using? After doing cpu_set x offline
> > cpu x should disappear from a guest. Otherwise cpu eject didn't work.
> 
> I'm confused.  The "cpu_set x offline" doesn't appear to do anything
> for me.  It does not disable the cpu, nor does it inform the guest to
> disable the cpu.


I second that - same here.


a script with my exact steps is below. Result is reproducible.


When run on my system it will output:


"Known CPUs" are defined as cpus with an entry
in /sys/devices/system/cpu on the guest
"Inserted CPUs" are defined as cpus listed by info cpus in qemu monitor
"online CPUs" are defined as cpus listed in /proc/cpuinfo on the guest

Inserted CPUs: 4, CPUs known to the guest: 4, ONLINE in guest: 4
Inserting CPU #4
Inserted CPUs: 5, CPUs known to the guest: 5, ONLINE in guest: 4
Ejecting CPU #4
Inserted CPUs: 5, CPUs known to the guest: 5, ONLINE in guest: 4
Setting all available cpus to online...
bash: line 0: echo: write error: Invalid argument
bash: line 0: echo: write error: Invalid argument
bash: line 0: echo: write error: Invalid argument
Inserted CPUs: 5, CPUs known to the guest: 5, ONLINE in guest: 5
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
after cpu_set 4 offline this should not come online again!!




---------------------------------------------------
#!/bin/bash
VM=192.168.2.35
MONITORHOST=localhost
MONITORPORT=9401

# count how many cpus are in /proc/cpuinfo on the guest
countOnline() {
 ONLINECPUS=`ssh -lroot ${VM} "grep processor /proc/cpuinfo | wc -l"`
}

# count how many cpus are in /sys/devices/...
countCPUsKnown() {
 KNOWNCPUS=`ssh -lroot ${VM} "find /sys/devices/system/cpu/ -name
'cpu?'| wc -l"`
}

# count how many cpus does kvm list
countInserted() {
 INSERTEDCPUS=`echo "info cpus" | nc  ${MONITORHOST} ${MONITORPORT} |
grep -av '(qemu)'|grep -va 'for more inform'|wc -l`
}

printInfo() {
countOnline
countCPUsKnown
countInserted
echo "Inserted CPUs: ${INSERTEDCPUS}, CPUs known to the guest:
${KNOWNCPUS}, ONLINE in guest: ${ONLINECPUS}"
}


echo "\"Known CPUs\" are defined as cpus with an entry
in /sys/devices/system/cpu on the guest"
echo "\"Inserted CPUs\" are defined as cpus listed by info cpus in qemu
monitor"
echo "\"online CPUs\" are defined as cpus listed in /proc/cpuinfo on the
guest"
echo

printInfo

echo "Inserting CPU #4"
echo "cpu_set 4 online" | nc ${MONITORHOST} ${MONITORPORT} >/dev/null

printInfo

echo "Ejecting CPU #4"
echo "cpu_set 4 offline" | nc ${MONITORHOST} ${MONITORPORT} >/dev/null

printInfo

echo "Setting all available cpus to online..."
ssh -lroot ${VM} "find /sys/devices/system/cpu/ -type f -path
'*/cpu?/online' -exec bash -c \"echo 1 >{}\" \;"


printInfo

Re: cpu hotplug

[Kevin O'Connor]

On Sun, Sep 19, 2010 at 06:24:33PM +0200, Conrad Wood wrote:
> 
> a script with my exact steps is below. Result is reproducible.
> 
[...]
> echo "Ejecting CPU #4"
> echo "cpu_set 4 offline" | nc ${MONITORHOST} ${MONITORPORT} >/dev/null
> 
> printInfo
> 
> echo "Setting all available cpus to online..."
> ssh -lroot ${VM} "find /sys/devices/system/cpu/ -type f -path
> '*/cpu?/online' -exec bash -c \"echo 1 >{}\" \;"

It's not clear if you can trust your guests or not.  If you can trust
the guest OS, it should be safe to run:

echo 1 > /sys/devices/LNXSYSTM:00/LNXSYBUS:00/LNXCPU:xx/eject

in the guest.  A normal linux guest wont reactivate the cpu after
that.

-Kevin

Re: cpu hotplug

[Gleb Natapov]

On Sun, Sep 19, 2010 at 05:57:25PM -0400, Kevin O'Connor wrote:
> On Sun, Sep 19, 2010 at 06:24:33PM +0200, Conrad Wood wrote:
> > 
> > a script with my exact steps is below. Result is reproducible.
> > 
> [...]
> > echo "Ejecting CPU #4"
> > echo "cpu_set 4 offline" | nc ${MONITORHOST} ${MONITORPORT} >/dev/null
> > 
> > printInfo
> > 
> > echo "Setting all available cpus to online..."
> > ssh -lroot ${VM} "find /sys/devices/system/cpu/ -type f -path
> > '*/cpu?/online' -exec bash -c \"echo 1 >{}\" \;"
> 
> It's not clear if you can trust your guests or not.  If you can trust
> the guest OS, it should be safe to run:
> 
> echo 1 > /sys/devices/LNXSYSTM:00/LNXSYBUS:00/LNXCPU:xx/eject
> 
> in the guest.  A normal linux guest wont reactivate the cpu after
> that.
> 
You can trust you guest more then you can trust a users of your guest.
It is easy for a user to do echo, it is much harder to modify guest to not
remove /sys entry for un-plugged cpu. 

But I agree that kvm shouldn't allow to reactivate un-plugged cpu even
if it is not destroy it on un-plug.

--
			Gleb.

Re: cpu hotplug

[Kevin O'Connor]

On Sun, Sep 19, 2010 at 03:29:35PM +0200, Conrad Wood wrote:
> On Sun, 2010-09-19 at 09:26 -0400, Kevin O'Connor wrote:
> > On Sun, Sep 19, 2010 at 08:38:12AM +0200, Gleb Natapov wrote:
> > > Known problem in qemu. There was a patch for this, but qemu maintainers
> > > think it is not good enough.
> > Old versions of kvm do work - for example, qemu-kvm-0.12.5 works for
> > me.
> hm... that's the version I am using und it does not work here ;(
> the hotplug itself seems to work but they're reported as halted.
> Could you confirm that the cpus in the monitor as reported by "info
> cpus" are not reported as "(halted)" ?

I sent my email before I saw that you and Gleb had continued the
thread.

It works in the sense that if I run two cpu hogs in a guest with one
cpu I use only one host cpu used.  When I turn on an additional guest
cpu, I see two host cpus being used.

But otherwise, "info cpus" does seem inaccurate.

-Kevin

Re: cpu hotplug

[Conrad Wood]

On Sun, 2010-09-19 at 09:38 -0400, Kevin O'Connor wrote:
> On Sun, Sep 19, 2010 at 03:29:35PM +0200, Conrad Wood wrote:
> > On Sun, 2010-09-19 at 09:26 -0400, Kevin O'Connor wrote:
> > > On Sun, Sep 19, 2010 at 08:38:12AM +0200, Gleb Natapov wrote:
> > > > Known problem in qemu. There was a patch for this, but qemu maintainers
> > > > think it is not good enough.
> > > Old versions of kvm do work - for example, qemu-kvm-0.12.5 works for
> > > me.
> > hm... that's the version I am using und it does not work here ;(
> > the hotplug itself seems to work but they're reported as halted.
> > Could you confirm that the cpus in the monitor as reported by "info
> > cpus" are not reported as "(halted)" ?
> 
> I sent my email before I saw that you and Gleb had continued the
> thread.
> 
> It works in the sense that if I run two cpu hogs in a guest with one
> cpu I use only one host cpu used.  When I turn on an additional guest
> cpu, I see two host cpus being used.
> 
> But otherwise, "info cpus" does seem inaccurate.
> 
> -Kevin
> 

thanks for confirming that. got a little puzzled there.

Cheers,

Conrad

CPU HOTPLUG

[Ozgur O Kilic]

Hello,

I am trying to hot plug and hot(un)plug vCPU to a KVM guest. At the
fallowing link, I found out we can hotplug vCPUs on the run time with "cpu
add" but the hot(un)plug is still in progress. Is there any update on it.
Can we use device add/remove for CPUs?
http://wiki.qemu.org/Features/CPUHotplug

Also, I also find out that we can use virsh (libvirt) to add and remove
vCPUs on the fly but I think it is just putting vCPU to sleep or wake up.
Is it a good method to change CPU numbers or will it be  performance
overhead such as does sleeping vCPUs get any time  on the host, are those
get scheduled?

Thank you, Happy new year,
Ozgur