VCPU hotplug problem

VCPU hotplug problem

[Gabriel Southern]

Hello,

I am working on a project using the VCPU hotplug feature in Xen, and I
think that there is a problem with how Linux is processing CPU hotplug
events.  The problem I have seen is that the /proc/stat file does not
change the number of CPU entries in response to the: xm vcpu-set
<domain> <num> command.  However, the /proc/cpuinfo file is updated
correctly.

The reason this is important is that some applications use the
sysconf(_SC_NPROCESSORS_ONLN) to determine how many threads to run.
In Linux this function parses the /proc/stat file to determine how
many processors are available.  However, the /proc/stat file is not
being updated correctly in response to the xm vcpu-set command.

It would be a big help for me if anyone else could test to see if they
experience the same problem.  I'm not sure if this would qualify as a
problem with Linux or with Xen; however, any ideas for fixing it are
appreciated as well.

Thanks,

-Gabriel

Re: VCPU hotplug problem

[Chris Lalancette]

Gabriel Southern wrote:
> Hello,
> 
> I am working on a project using the VCPU hotplug feature in Xen, and I
> think that there is a problem with how Linux is processing CPU hotplug
> events.  The problem I have seen is that the /proc/stat file does not
> change the number of CPU entries in response to the: xm vcpu-set
> <domain> <num> command.  However, the /proc/cpuinfo file is updated
> correctly.
> 
> The reason this is important is that some applications use the
> sysconf(_SC_NPROCESSORS_ONLN) to determine how many threads to run.
> In Linux this function parses the /proc/stat file to determine how
> many processors are available.  However, the /proc/stat file is not
> being updated correctly in response to the xm vcpu-set command.
> 
> It would be a big help for me if anyone else could test to see if they
> experience the same problem.  I'm not sure if this would qualify as a
> problem with Linux or with Xen; however, any ideas for fixing it are
> appreciated as well.

Hm, it would be helpful if you told us what version of Xen you are
running and which kernel you are running.  For what it is worth, running
a RHEL-5.1 PV guest on a RHEL-5.1 dom0 (both x86_64) seems to do the
right thing for me; xm vcpu-set <dom> 1 updates both /proc/cpuinfo and
/proc/stat, and xm vcpu-set <dom> 2 updates both again.

Chris Lalancette

Re: VCPU hotplug problem

[Gabriel Southern]

Thanks for the reply.  I had Ubuntu 7.10 x86_64 installed, and I
compiled the Xen 3.2 source that I downloaded from xen.org.  So my
kernel is 2.6.18.8-xen x86_64 and I am running Xen 3.2.  I had the
same problem with Xen 3.1.

Which version of the Linux kernel does RHEL-5.1 use?  I'm curious to
know if the problem I am seeing is a result of something I did wrong
in compiling Xen, or if it is a kernel bug that has been fixed in a
later release of Linux.

-Gabriel

On Feb 11, 2008 10:42 PM, Chris Lalancette <clalance@redhat.com> wrote:

>
> Hm, it would be helpful if you told us what version of Xen you are
> running and which kernel you are running.  For what it is worth, running
> a RHEL-5.1 PV guest on a RHEL-5.1 dom0 (both x86_64) seems to do the
> right thing for me; xm vcpu-set <dom> 1 updates both /proc/cpuinfo and
> /proc/stat, and xm vcpu-set <dom> 2 updates both again.
>
> Chris Lalancette
>

Re: VCPU hotplug problem

[Keir Fraser]

>> It would be a big help for me if anyone else could test to see if they
>> experience the same problem.  I'm not sure if this would qualify as a
>> problem with Linux or with Xen; however, any ideas for fixing it are
>> appreciated as well.
> 
> Hm, it would be helpful if you told us what version of Xen you are
> running and which kernel you are running.  For what it is worth, running
> a RHEL-5.1 PV guest on a RHEL-5.1 dom0 (both x86_64) seems to do the
> right thing for me; xm vcpu-set <dom> 1 updates both /proc/cpuinfo and
> /proc/stat, and xm vcpu-set <dom> 2 updates both again.

The issue is the result of a deliberate change in some kernels (including
our 2.6.18-xen tree) in which fs/proc/proc_misc.c:show_stat() iterates over
each_possible_cpu() rather than each_online_cpu(). This of course means it
prints info for all offline cpus as well as online cpus.

The original reason for this is that some user-space tools, such as older
versions of top, would get quite upset when CPUs are hot-unplugged while
they run. This would cause top to crash! I don't know whether this was fixed
by now in the user-space packages. The issue was originally seen years ago
so I suppose it's fairly likely it has been, and the hack could be removed.

 -- Keir

Re: VCPU hotplug problem

[Gabriel Southern]

Thanks for the explanation.  When I started looking at the problem, I
had not previously looked at the Xen source code, and I made the
mistake of looking at the Linux 2.6.18 kernel from kernel.org, without
the xen patches applied.  So when I was tracing through the code
everything looked fine because it had the for_each_online_cpu() macro
in place.  After reading your message, I checked the code that I was
actually compiling and it had the change you described.  Changing from
for_each_possible_cpu() back to for_each_online_cpu() fixed the
problem for me.

Incidentially in a Ubuntu 7.10 VM, after this change is made, top will
still crash if a CPU is hot-unplugged while top is running.  However,
in centos-5.1 the hot-unplug events are handled differently and top
does not crash.  Thanks again for the help.

-Gabriel

>
> The issue is the result of a deliberate change in some kernels (including
> our 2.6.18-xen tree) in which fs/proc/proc_misc.c:show_stat() iterates over
> each_possible_cpu() rather than each_online_cpu(). This of course means it
> prints info for all offline cpus as well as online cpus.
>
> The original reason for this is that some user-space tools, such as older
> versions of top, would get quite upset when CPUs are hot-unplugged while
> they run. This would cause top to crash! I don't know whether this was fixed
> by now in the user-space packages. The issue was originally seen years ago
> so I suppose it's fairly likely it has been, and the hack could be removed.
>
>  -- Keir
>