[PATCH 0/8] RFC: vcpu pinning at qemu start

[PATCH 0/8] RFC: vcpu pinning at qemu start

[Glauber Costa]

Hi guys,

Here's a first series of patch aiming at vcpu pinning support in qemu.
Ideally, as vcpu as just normal threads, the usual userspace tools can be used
to set cpu affinities mask.

However, It makes it very difficult to _start_ a vm with vcpus pinned, since
we don't know the thread ids from qemu in advance, nor do we know when are the
vcpus created.

The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus
with the specified affinities.

Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH 1/8] add thread id to vcpu structure

[Glauber Costa]

This allow us to track which thread is currently running the cpu
Signed-off-by: Glauber Costa <gcosta@redhat.com>
---
 qemu/qemu-kvm.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/qemu/qemu-kvm.c b/qemu/qemu-kvm.c
index 051946e..43e0f2e 100644
--- a/qemu/qemu-kvm.c
+++ b/qemu/qemu-kvm.c
@@ -47,6 +47,7 @@ struct vcpu_info {
     int signalled;
     int stop;
     int stopped;
+    int thread_id;
 } vcpu_info[256];
 
 CPUState *qemu_kvm_cpu_env(int index)
-- 
1.5.0.6


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH 2/8] provide a gettid function

[Glauber Costa]

Since glibc does not provide a gettid call directly, only through
syscall, we wrap one for kvm

Signed-off-by: Glauber Costa <gcosta@redhat.com>
---
 qemu/qemu-kvm.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/qemu/qemu-kvm.c b/qemu/qemu-kvm.c
index 43e0f2e..8ee3bf8 100644
--- a/qemu/qemu-kvm.c
+++ b/qemu/qemu-kvm.c
@@ -19,6 +19,7 @@ int kvm_irqchip = 1;
 #include <libkvm.h>
 #include <pthread.h>
 #include <sys/utsname.h>
+#include <sys/syscall.h>
 
 extern void perror(const char *s);
 
@@ -50,6 +51,11 @@ struct vcpu_info {
     int thread_id;
 } vcpu_info[256];
 
+static inline unsigned long kvm_get_thread_id(void)
+{
+    return syscall(SYS_gettid);
+}
+
 CPUState *qemu_kvm_cpu_env(int index)
 {
     return vcpu_info[index].env;
-- 
1.5.0.6


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH 3/8] get thread id at thread's creation

[Glauber Costa]

store the thread id through our new call as soon as
the process is created.

Signed-off-by: Glauber Costa <gcosta@redhat.com>
---
 qemu/qemu-kvm.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/qemu/qemu-kvm.c b/qemu/qemu-kvm.c
index 8ee3bf8..1aeb97b 100644
--- a/qemu/qemu-kvm.c
+++ b/qemu/qemu-kvm.c
@@ -335,6 +335,7 @@ static void *ap_main_loop(void *_env)
 
     vcpu = &vcpu_info[env->cpu_index];
     vcpu->env = env;
+    vcpu->thread_id = kvm_get_thread_id();
     sigfillset(&signals);
     //sigdelset(&signals, SIG_IPI);
     sigprocmask(SIG_BLOCK, &signals, NULL);
@@ -381,6 +382,7 @@ int kvm_init_ap(void)
 
     vcpu = &vcpu_info[0];
     vcpu->env = first_cpu;
+    vcpu->thread_id = kvm_get_thread_id();
     signal(SIG_IPI, sig_ipi_handler);
     for (i = 1; i < smp_cpus; ++i) {
         kvm_init_new_ap(i, env);
-- 
1.5.0.6


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH 4/8] store and set cpu affinities

[Glauber Costa]

This patch provides a cpu_set variable to vcpu_info structure.
It stores the current cpu mask for a thread.

We also provide a wrapper for storing a provided affinity, and a
function to set default affinities: The default affinities are current
process'

Signed-off-by: Glauber Costa <gcosta@redhat.com>
---
 qemu/qemu-kvm.c |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/qemu/qemu-kvm.c b/qemu/qemu-kvm.c
index 1aeb97b..a36fbf6 100644
--- a/qemu/qemu-kvm.c
+++ b/qemu/qemu-kvm.c
@@ -49,8 +49,25 @@ struct vcpu_info {
     int stop;
     int stopped;
     int thread_id;
+    cpu_set_t cpu_set;
 } vcpu_info[256];
 
+void kvm_store_cpu_affinity(int cpu, cpu_set_t *map)
+{
+	memcpy(&vcpu_info[cpu].cpu_set, map, sizeof(*map));
+}
+
+void kvm_register_default_affinities(void)
+{
+    int i;
+    cpu_set_t dfl;
+    sched_getaffinity(0, sizeof(dfl), &dfl);
+
+    for  (i = 0; i < 256; i++) {
+        memcpy(&vcpu_info[i].cpu_set, &dfl, sizeof(dfl));
+    }
+}
+
 static inline unsigned long kvm_get_thread_id(void)
 {
     return syscall(SYS_gettid);
-- 
1.5.0.6


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH 5/8] initialize affinities

[Glauber Costa]

store default process affinities before we get the chance to process
any options

Signed-off-by: Glauber Costa <gcosta@redhat.com>
---
 qemu/vl.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/qemu/vl.c b/qemu/vl.c
index f10fbd8..6a94724 100644
--- a/qemu/vl.c
+++ b/qemu/vl.c
@@ -8717,6 +8717,9 @@ int main(int argc, char **argv)
     }
 #endif
 
+#ifdef USE_KVM
+    kvm_register_default_affinities();
+#endif
     register_machines();
     machine = first_machine;
     cpu_model = NULL;
-- 
1.5.0.6


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH 6/8] stabilish default affinity for newly created cpus

[Glauber Costa]

Signed-off-by: Glauber Costa <gcosta@redhat.com>
---
 qemu/qemu-kvm.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/qemu/qemu-kvm.c b/qemu/qemu-kvm.c
index a36fbf6..80fe8e5 100644
--- a/qemu/qemu-kvm.c
+++ b/qemu/qemu-kvm.c
@@ -68,6 +68,12 @@ void kvm_register_default_affinities(void)
     }
 }
 
+static inline void kvm_default_affinity(int cpu)
+{
+    sched_setaffinity(vcpu_info[cpu].thread_id, sizeof(cpu_set_t),
+                      &vcpu_info[cpu].cpu_set);
+}
+
 static inline unsigned long kvm_get_thread_id(void)
 {
     return syscall(SYS_gettid);
@@ -353,6 +359,7 @@ static void *ap_main_loop(void *_env)
     vcpu = &vcpu_info[env->cpu_index];
     vcpu->env = env;
     vcpu->thread_id = kvm_get_thread_id();
+    kvm_default_affinity(env->cpu_index);
     sigfillset(&signals);
     //sigdelset(&signals, SIG_IPI);
     sigprocmask(SIG_BLOCK, &signals, NULL);
@@ -400,6 +407,7 @@ int kvm_init_ap(void)
     vcpu = &vcpu_info[0];
     vcpu->env = first_cpu;
     vcpu->thread_id = kvm_get_thread_id();
+    kvm_default_affinity(0);
     signal(SIG_IPI, sig_ipi_handler);
     for (i = 1; i < smp_cpus; ++i) {
         kvm_init_new_ap(i, env);
-- 
1.5.0.6


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH 7/8] process a cpu affinity mask

[Glauber Costa]

This patch provides a function that process a cpu affinity list
in the form x,y,z... into a cpu_set_t variable.

Signed-off-by: Glauber Costa <gcosta@redhat.com>
---
 qemu/vl.c |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/qemu/vl.c b/qemu/vl.c
index 6a94724..4715594 100644
--- a/qemu/vl.c
+++ b/qemu/vl.c
@@ -8646,6 +8646,23 @@ void *qemu_alloc_physram(unsigned long memory)
 
     return area;
 }
+#ifdef USE_KVM
+#include <sched.h>
+void process_cpu_set(const char *map, cpu_set_t *set)
+{
+    char *ptr = map;
+    int c;
+    CPU_ZERO(set);
+    do {
+        c = atoi(ptr);
+        CPU_SET(c, set);
+        ptr = strchr(ptr, ',');
+        if (!ptr)
+            break;
+        ptr++;
+    } while (*ptr);
+}
+#endif
 
 int main(int argc, char **argv)
 {
-- 
1.5.0.6


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

[PATCH 8/8] provide -cpu-map option

[Glauber Costa]

this patch introduces a -cpu-map option.
It has the form C:x,y,z..., and have the effect
of setting the affinity mask of vcpu C to processors
x,y,z...

Signed-off-by: Glauber Costa <gcosta@redhat.com>
---
 qemu/vl.c |   13 +++++++++++++
 1 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/qemu/vl.c b/qemu/vl.c
index 4715594..fa830dd 100644
--- a/qemu/vl.c
+++ b/qemu/vl.c
@@ -8059,6 +8059,7 @@ static void help(int exitcode)
 	   "-no-kvm         disable KVM hardware virtualization\n"
 #endif
 	   "-no-kvm-irqchip disable KVM kernel mode PIC/IOAPIC/LAPIC\n"
+           "-cpu-map C:x,y,z... set cpu 'C' affinity to processors x,y,z...\n"
 #endif
 #ifdef TARGET_I386
            "-std-vga        simulate a standard VGA card with VESA Bochs Extensions\n"
@@ -8177,6 +8178,7 @@ enum {
     QEMU_OPTION_no_acpi,
     QEMU_OPTION_no_kvm,
     QEMU_OPTION_no_kvm_irqchip,
+    QEMU_OPTION_cpu_map,
     QEMU_OPTION_no_reboot,
     QEMU_OPTION_show_cursor,
     QEMU_OPTION_daemonize,
@@ -8263,6 +8265,7 @@ const QEMUOption qemu_options[] = {
     { "no-kvm", 0, QEMU_OPTION_no_kvm },
 #endif
     { "no-kvm-irqchip", 0, QEMU_OPTION_no_kvm_irqchip },
+    { "cpu-map", HAS_ARG, QEMU_OPTION_cpu_map },
 #endif
 #if defined(TARGET_PPC) || defined(TARGET_SPARC)
     { "g", 1, QEMU_OPTION_g },
@@ -9211,6 +9214,16 @@ int main(int argc, char **argv)
 		kvm_irqchip = 0;
 		break;
 	    }
+            case QEMU_OPTION_cpu_map: {
+                int c = atoi(optarg);
+                char *ptr = strchr(optarg, ':');
+                cpu_set_t set;
+                if (!ptr)
+                    fprintf(stderr, "invalid cpu mapping %s\n", ptr);
+                process_cpu_set(++ptr, &set);
+                kvm_store_cpu_affinity(c, &set);
+                break;
+            }
 #endif
             case QEMU_OPTION_usb:
                 usb_enabled = 1;
-- 
1.5.0.6


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Joerg Roedel]

On Tue, Mar 04, 2008 at 01:21:53PM -0300, Glauber Costa wrote:
> Hi guys,
> 
> Here's a first series of patch aiming at vcpu pinning support in qemu.
> Ideally, as vcpu as just normal threads, the usual userspace tools can be used
> to set cpu affinities mask.
> 
> However, It makes it very difficult to _start_ a vm with vcpus pinned, since
> we don't know the thread ids from qemu in advance, nor do we know when are the
> vcpus created.
> 
> The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus
> with the specified affinities.
> 
> Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!

Cool, this goes into the same direction as I planned for KVM-NUMA
support. Do you plan to extend vcpu pinning into that direction?

Joerg

-- 
           |           AMD Saxony Limited Liability Company & Co. KG
 Operating |         Wilschdorfer Landstr. 101, 01109 Dresden, Germany
 System    |                  Register Court Dresden: HRA 4896
 Research  |              General Partner authorized to represent:
 Center    |             AMD Saxony LLC (Wilmington, Delaware, US)
           | General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Glauber Costa]

Joerg Roedel wrote:
> On Tue, Mar 04, 2008 at 01:21:53PM -0300, Glauber Costa wrote:
>> Hi guys,
>>
>> Here's a first series of patch aiming at vcpu pinning support in qemu.
>> Ideally, as vcpu as just normal threads, the usual userspace tools can be used
>> to set cpu affinities mask.
>>
>> However, It makes it very difficult to _start_ a vm with vcpus pinned, since
>> we don't know the thread ids from qemu in advance, nor do we know when are the
>> vcpus created.
>>
>> The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus
>> with the specified affinities.
>>
>> Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
> 
> Cool, this goes into the same direction as I planned for KVM-NUMA
> support. Do you plan to extend vcpu pinning into that direction?
I don't have any immediate plans, but it is surely interesting. If the 
patches (or something inspired in them) make it, there's something we 
can draw support for.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Joerg Roedel]

On Tue, Mar 04, 2008 at 03:11:27PM -0300, Glauber Costa wrote:
> Joerg Roedel wrote:
> >On Tue, Mar 04, 2008 at 01:21:53PM -0300, Glauber Costa wrote:
> >>Hi guys,
> >>
> >>Here's a first series of patch aiming at vcpu pinning support in qemu.
> >>Ideally, as vcpu as just normal threads, the usual userspace tools can be used
> >>to set cpu affinities mask.
> >>
> >>However, It makes it very difficult to _start_ a vm with vcpus pinned, since
> >>we don't know the thread ids from qemu in advance, nor do we know when are the
> >>vcpus created.
> >>
> >>The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus
> >>with the specified affinities.
> >>
> >>Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
> >Cool, this goes into the same direction as I planned for KVM-NUMA
> >support. Do you plan to extend vcpu pinning into that direction?
> I don't have any immediate plans, but it is surely interesting. If the patches (or something inspired in them) make it, there's something we can draw support for.

There are patches for HVM-NUMA support on Xen developed by André
Przywara. I think they are easy to port to KVM. Maybe it is the better
aproach than implementing simple vcpu pinning.

Joerg

-- 
           |           AMD Saxony Limited Liability Company & Co. KG
 Operating |         Wilschdorfer Landstr. 101, 01109 Dresden, Germany
 System    |                  Register Court Dresden: HRA 4896
 Research  |              General Partner authorized to represent:
 Center    |             AMD Saxony LLC (Wilmington, Delaware, US)
           | General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-devel mailing list
kvm-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Anthony Liguori]

Glauber Costa wrote:
> Hi guys,
>
> Here's a first series of patch aiming at vcpu pinning support in qemu.
> Ideally, as vcpu as just normal threads, the usual userspace tools can be used
> to set cpu affinities mask.
>
> However, It makes it very difficult to _start_ a vm with vcpus pinned, since
> we don't know the thread ids from qemu in advance, nor do we know when are the
> vcpus created.
>
> The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus
> with the specified affinities.
>
> Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
>
>   

So why exactly is this useful?  I have a hard time constructing a 
reasonable use-case in my mind for something like this.

Regards,

Anthony Liguori


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Glauber Costa]

Anthony Liguori wrote:
> Glauber Costa wrote:
>> Hi guys,
>>
>> Here's a first series of patch aiming at vcpu pinning support in qemu.
>> Ideally, as vcpu as just normal threads, the usual userspace tools can 
>> be used
>> to set cpu affinities mask.
>>
>> However, It makes it very difficult to _start_ a vm with vcpus pinned, 
>> since
>> we don't know the thread ids from qemu in advance, nor do we know when 
>> are the
>> vcpus created.
>>
>> The patches introduce a -cpu-map option, that, if specified, starts 
>> the virtual cpus
>> with the specified affinities.
>>
>> Comments? Welcome. Random rants? Not welcome, but... how can I stop 
>> you? So go ahead!
>>
>>   
> 
> So why exactly is this useful?  I have a hard time constructing a 
> reasonable use-case in my mind for something like this.

My main interest is in management tools being able to specify pinning
set ups at VM creation time.

As I said, it can be done through tools like taskset, but then you'd 
have to know:
  * when are the threads created
  * which thread ids corresponds to each cpu

And of course, for an amount of time, the threads will be running in a 
"wrong" cpu, which may affect workloads running there. (which is a case 
cpu pinning usually tries to address)




-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Anthony Liguori]

Glauber Costa wrote:
> My main interest is in management tools being able to specify pinning
> set ups at VM creation time.
>
> As I said, it can be done through tools like taskset, but then you'd 
> have to know:
>  * when are the threads created
>  * which thread ids corresponds to each cpu
>
> And of course, for an amount of time, the threads will be running in a 
> "wrong" cpu, which may affect workloads running there. (which is a 
> case cpu pinning usually tries to address)

A management tool can start QEMU with -S to prevent any CPUs from 
running, query the VCPU=>thread id relationship (modifying info cpus 
would be a good thing to do for this), taskset, and then run 'cont' in 
the monitor if they desperately need this functionality.  However, I 
don't think the vast majority of people need this particular functionality.

My feeling is that adding an interface to do this in QEMU encourages 
people to not use the existing Linux tools for this or worse yet, to 
think they can do a better job than Linux.  The whole reason this exists 
in Xen is that Xen's schedulers were incapable of doing CPU migration 
historically (which is no longer true since the credit scheduler).  It 
was necessary to specify pinning upon creation or you were stuck with 
round-robin placement.  So libvirt has APIs for this because they were 
part of the Xen API because it was needed to get reasonable performance 
at some point in time on Xen.  I don't think this behavior is useful for 
KVM though.  Just because Xen does it doesn't imply that we should do it.

Regards,

Anthony Liguori

>
>


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Daniel P. Berrange]

On Tue, Mar 04, 2008 at 01:28:24PM -0600, Anthony Liguori wrote:
> Glauber Costa wrote:
> > My main interest is in management tools being able to specify pinning
> > set ups at VM creation time.
> >
> > As I said, it can be done through tools like taskset, but then you'd 
> > have to know:
> >  * when are the threads created
> >  * which thread ids corresponds to each cpu
> >
> > And of course, for an amount of time, the threads will be running in a 
> > "wrong" cpu, which may affect workloads running there. (which is a 
> > case cpu pinning usually tries to address)
> 
> A management tool can start QEMU with -S to prevent any CPUs from 
> running, query the VCPU=>thread id relationship (modifying info cpus 
> would be a good thing to do for this), taskset, and then run 'cont' in 
> the monitor if they desperately need this functionality.  However, I 
> don't think the vast majority of people need this particular functionality.

I fully expected to have to run QEMU with -S and then use cont if I were
todo CPU pinning from libvirt.

The only info I'd need to get is the  PID <-> vCPU mapping data. Then
I can use regular Linux taskset capabilities from libvirt to assign the
initial pCPU <-> vCPU mapping and finally run 'cont'.

> My feeling is that adding an interface to do this in QEMU encourages 
> people to not use the existing Linux tools for this or worse yet, to 
> think they can do a better job than Linux.  The whole reason this exists 
> in Xen is that Xen's schedulers were incapable of doing CPU migration 
> historically (which is no longer true since the credit scheduler).  It 
> was necessary to specify pinning upon creation or you were stuck with 
> round-robin placement.  So libvirt has APIs for this because they were 
> part of the Xen API because it was needed to get reasonable performance 
> at some point in time on Xen.  I don't think this behavior is useful for 
> KVM though.  Just because Xen does it doesn't imply that we should do it.

I agree that adding QEMU commands for stuff which Linux already has APIs
and tools is a bad idea. QEMU/KVM is much nicer to manage than Xen, 
precisely because I can already use Linux APIs & process management tools.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Glauber Costa]

Daniel P. Berrange wrote:
> On Tue, Mar 04, 2008 at 01:28:24PM -0600, Anthony Liguori wrote:
>> Glauber Costa wrote:
>>> My main interest is in management tools being able to specify pinning
>>> set ups at VM creation time.
>>>
>>> As I said, it can be done through tools like taskset, but then you'd 
>>> have to know:
>>>  * when are the threads created
>>>  * which thread ids corresponds to each cpu
>>>
>>> And of course, for an amount of time, the threads will be running in a 
>>> "wrong" cpu, which may affect workloads running there. (which is a 
>>> case cpu pinning usually tries to address)
>> A management tool can start QEMU with -S to prevent any CPUs from 
>> running, query the VCPU=>thread id relationship (modifying info cpus 
>> would be a good thing to do for this), taskset, and then run 'cont' in 
>> the monitor if they desperately need this functionality.  However, I 
>> don't think the vast majority of people need this particular functionality.
> 
> I fully expected to have to run QEMU with -S and then use cont if I were
> todo CPU pinning from libvirt.
> 
> The only info I'd need to get is the  PID <-> vCPU mapping data. Then
> I can use regular Linux taskset capabilities from libvirt to assign the
> initial pCPU <-> vCPU mapping and finally run 'cont'.
> 
>> My feeling is that adding an interface to do this in QEMU encourages 
>> people to not use the existing Linux tools for this or worse yet, to 
>> think they can do a better job than Linux.  The whole reason this exists 
>> in Xen is that Xen's schedulers were incapable of doing CPU migration 
>> historically (which is no longer true since the credit scheduler).  It 
>> was necessary to specify pinning upon creation or you were stuck with 
>> round-robin placement.  So libvirt has APIs for this because they were 
>> part of the Xen API because it was needed to get reasonable performance 
>> at some point in time on Xen.  I don't think this behavior is useful for 
>> KVM though.  Just because Xen does it doesn't imply that we should do it.
> 
> I agree that adding QEMU commands for stuff which Linux already has APIs
> and tools is a bad idea. QEMU/KVM is much nicer to manage than Xen, 
> precisely because I can already use Linux APIs & process management tools.

I totally agree this is ideal, and I did not start this after thinking a 
little bit about
this situation. The main point is that we don't know when the cpus are 
created, and it does not seem to me that we will without a considerable 
amount of work.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Glauber Costa]

Anthony Liguori wrote:
> Glauber Costa wrote:
>> My main interest is in management tools being able to specify pinning
>> set ups at VM creation time.
>>
>> As I said, it can be done through tools like taskset, but then you'd 
>> have to know:
>>  * when are the threads created
>>  * which thread ids corresponds to each cpu
>>
>> And of course, for an amount of time, the threads will be running in a 
>> "wrong" cpu, which may affect workloads running there. (which is a 
>> case cpu pinning usually tries to address)
> 
> A management tool can start QEMU with -S to prevent any CPUs from 
> running, query the VCPU=>thread id relationship (modifying info cpus 
> would be a good thing to do for this), taskset, and then run 'cont' in 
> the monitor if they desperately need this functionality.  However, I 
> don't think the vast majority of people need this particular functionality.

No, it can't. Because at the time qemu starts, no vcpu -> thread id 
relationship exists at all. And we don't know when it will.

It would be a different story if there were some kind of api that could
warn qemu
 > My feeling is that adding an interface to do this in QEMU encourages
 > people to not use the existing Linux tools for this or worse yet, to
 > think they can do a better job than Linux.

I agree with you that we should stick with linux tools, and that's why I 
didn't provide any kind of runtime setting via qemu monitor to do this 
(with the infrastructure, it would be trivial). taskset will do.

> The whole reason this exists 
> in Xen is that Xen's schedulers were incapable of doing CPU migration 
> historically (which is no longer true since the credit scheduler).  It 
> was necessary to specify pinning upon creation or you were stuck with 
> round-robin placement.  So libvirt has APIs for this because they were 
> part of the Xen API because it was needed to get reasonable performance 
> at some point in time on Xen.  I don't think this behavior is useful for 
> KVM though.  Just because Xen does it doesn't imply that we should do it.
No, not just because xen does.

I do however feel it useful, since starting a vm and then let it run 
unchanged is definitely an useful use case. And as I tried to show you,
I can't see a good way to do that for pinning.

> Regards,
> 
> Anthony Liguori
> 
>>
>>
> 


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Anthony Liguori]

Glauber Costa wrote:
> Anthony Liguori wrote:
>
> No, it can't. Because at the time qemu starts, no vcpu -> thread id 
> relationship exists at all. And we don't know when it will.

Sure we do.  The vcpu -> thread id relationship is valid after 
kvm_init_ap() is called which is after machine init but before the 
select loop is entered for the first time.  Therefore, if you start qemu 
with -S, then connect on the monitor, and do an info cpus, you could be 
guaranteed to be told the mapping.

The threads are *idle* at this point so there's no harm if they were 
started on the "wrong" CPU.  You can now taskset to your hearts content 
and then when you're happy with placement, you can issue a 'cont' so 
that the VM actually starts running.  I saw "wrong" because you can 
still taskset the initial creation guaranteeing that the threads are 
created on the right group of physical CPUs, you just can't specify the 
exact mapping until you start interacting with the monitor.

Regards,

Anthony Liguori

>> Regards,
>>
>> Anthony Liguori
>>
>>>
>>>
>>
>


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Avi Kivity]

Anthony Liguori wrote:
> Glauber Costa wrote:
>   
>> Anthony Liguori wrote:
>>
>> No, it can't. Because at the time qemu starts, no vcpu -> thread id 
>> relationship exists at all. And we don't know when it will.
>>     
>
> Sure we do.  The vcpu -> thread id relationship is valid after 
> kvm_init_ap() is called which is after machine init but before the 
> select loop is entered for the first time.  Therefore, if you start qemu 
> with -S, then connect on the monitor, and do an info cpus, you could be 
> guaranteed to be told the mapping.
>
> The threads are *idle* at this point so there's no harm if they were 
> started on the "wrong" CPU.  You can now taskset to your hearts content 
> and then when you're happy with placement, you can issue a 'cont' so 
> that the VM actually starts running.  I saw "wrong" because you can 
> still taskset the initial creation guaranteeing that the threads are 
> created on the right group of physical CPUs, you just can't specify the 
> exact mapping until you start interacting with the monitor.
>
>   

Good points.  Initially I thought we ought to abstract the 
implementation and not expose the vcpu thread id, but I'm beginning to 
thing that due the wide variety of options (affinity, page migration, 
priority, cpu control groups) and the relative obscurity of the feature 
(which as you point out, isn't needed in the common case), we can export 
the thread id and let the management tools deal with it directly.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Glauber Costa]

Anthony Liguori wrote:
> Glauber Costa wrote:
>> Anthony Liguori wrote:
>>
>> No, it can't. Because at the time qemu starts, no vcpu -> thread id 
>> relationship exists at all. And we don't know when it will.
> 
> Sure we do.  The vcpu -> thread id relationship is valid after 
> kvm_init_ap() is called which is after machine init but before the 
> select loop is entered for the first time.  Therefore, if you start qemu 
> with -S, then connect on the monitor, and do an info cpus, you could be 
> guaranteed to be told the mapping.

I missed that. This changes everything. I now completely agree with you.

I'll post patches that expose the relationship, if it's better.

> The threads are *idle* at this point so there's no harm if they were 
> started on the "wrong" CPU.  You can now taskset to your hearts content 
> and then when you're happy with placement, you can issue a 'cont' so 
> that the VM actually starts running.  I saw "wrong" because you can 
> still taskset the initial creation guaranteeing that the threads are 
> created on the right group of physical CPUs, you just can't specify the 
> exact mapping until you start interacting with the monitor.
> 
> Regards,
> 
> Anthony Liguori
> 
>>> Regards,
>>>
>>> Anthony Liguori
>>>
>>>>
>>>>
>>>
>>
> 


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Avi Kivity]

Anthony Liguori wrote:
> Glauber Costa wrote:
>> My main interest is in management tools being able to specify pinning
>> set ups at VM creation time.
>>
>> As I said, it can be done through tools like taskset, but then you'd 
>> have to know:
>>  * when are the threads created
>>  * which thread ids corresponds to each cpu
>>
>> And of course, for an amount of time, the threads will be running in 
>> a "wrong" cpu, which may affect workloads running there. (which is a 
>> case cpu pinning usually tries to address)
>
> A management tool can start QEMU with -S to prevent any CPUs from 
> running, query the VCPU=>thread id relationship (modifying info cpus 
> would be a good thing to do for this), taskset, and then run 'cont' in 
> the monitor if they desperately need this functionality.  However, I 
> don't think the vast majority of people need this particular 
> functionality.
>


Affinity control is probably useful mostly for numa configurations, 
where you want to restrict virtual cpus to run on the cores closest to 
memory.  However it may well be that the scheduler is already good 
enough to do this on its own.


> My feeling is that adding an interface to do this in QEMU encourages 
> people to not use the existing Linux tools for this or worse yet, to 
> think they can do a better job than Linux.  The whole reason this 
> exists in Xen is that Xen's schedulers were incapable of doing CPU 
> migration historically (which is no longer true since the credit 
> scheduler).  It was necessary to specify pinning upon creation or you 
> were stuck with round-robin placement.  So libvirt has APIs for this 
> because they were part of the Xen API because it was needed to get 
> reasonable performance at some point in time on Xen.  I don't think 
> this behavior is useful for KVM though.  Just because Xen does it 
> doesn't imply that we should do it.
>

In the brutal world of hypervisors, if your competitor has a feature, 
you must have it too.  I often get asked about cpu pinning in kvm.

[I'd like to see how Xen implements swapping, though]

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Anthony Liguori]

Avi Kivity wrote:
> Affinity control is probably useful mostly for numa configurations, 
> where you want to restrict virtual cpus to run on the cores closest to 
> memory.  However it may well be that the scheduler is already good 
> enough to do this on its own.

In that case, you need to use numactl to set a NUMA policy so it's 
pretty natural that you would also be using taskset.  Assuming you're 
trying to keep a VM local to a particular node (we don't expose a 
virtual SRAT/SLIT table so that's all we can sanely do right now), it 
doesn't matter what CPU each VCPU thread lands on as long as they stay 
within the node.  So taskset is perfectly capable to address this need 
today.

> In the brutal world of hypervisors, if your competitor has a feature, 
> you must have it too.  I often get asked about cpu pinning in kvm.

And the answer to give is, of course, we support it through taskset :-)

Regards,

Anthony Liguori

> [I'd like to see how Xen implements swapping, though]
>


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Avi Kivity]

Glauber Costa wrote:
> Hi guys,
>
> Here's a first series of patch aiming at vcpu pinning support in qemu.
> Ideally, as vcpu as just normal threads, the usual userspace tools can be used
> to set cpu affinities mask.
>
> However, It makes it very difficult to _start_ a vm with vcpus pinned, since
> we don't know the thread ids from qemu in advance, nor do we know when are the
> vcpus created.
>
> The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus
> with the specified affinities.
>
> Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
>
>   

A monitor interface would be more useful than a command line option, as 
it allows you to migrate the vcpus at runtime, and also control 
hotplugged cpus.  For unmanaged use, taskset is probably sufficient to 
control affinity from the command line.

Normally I encourage splitting patches, but this is a bit extreme.  1 
and 3 are pointless without each other, 4 and 5, 7 and 8.  Hope that 
doesn't interfere with any pay-per-patch contract.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

Re: [PATCH 0/8] RFC: vcpu pinning at qemu start

[Glauber Costa]

Avi Kivity wrote:
> Glauber Costa wrote:
>> Hi guys,
>>
>> Here's a first series of patch aiming at vcpu pinning support in qemu.
>> Ideally, as vcpu as just normal threads, the usual userspace tools can 
>> be used
>> to set cpu affinities mask.
>>
>> However, It makes it very difficult to _start_ a vm with vcpus pinned, 
>> since
>> we don't know the thread ids from qemu in advance, nor do we know when 
>> are the
>> vcpus created.
>>
>> The patches introduce a -cpu-map option, that, if specified, starts 
>> the virtual cpus
>> with the specified affinities.
>>
>> Comments? Welcome. Random rants? Not welcome, but... how can I stop 
>> you? So go ahead!
>>
>>   
> 
> A monitor interface would be more useful than a command line option, as 
> it allows you to migrate the vcpus at runtime, and also control 
> hotplugged cpus.  For unmanaged use, taskset is probably sufficient to 
> control affinity from the command line.
> 
> Normally I encourage splitting patches, but this is a bit extreme.  1 
> and 3 are pointless without each other, 4 and 5, 7 and 8.  Hope that 
> doesn't interfere with any pay-per-patch contract.
> 
I'll post a new series that just expose the thread ids, so the 
management tools can use plain taskset. As for the split, there's 
nothing in my contract, but after all the x86 integration, it became a 
mania for me. Ingo made me this way.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/