[PATCH 0/4] Virtual Machine Time Accounting

[PATCH 0/4] Virtual Machine Time Accounting

[Laurent Vivier]

The aim of these four patches is to introduce Virtual Machine time accounting.

_Ingo_, as these patches modify files of the scheduler, could you have a look to
them, please ?

[PATCH 1/4] as recent CPUs introduce a third running state, after "user" and
"system", we need a new field, "guest", in cpustat to store the time used by
the CPU to run virtual CPU. Modify /proc/stat to display this new field.

[PATCH 2/4] like for cpustat, introduce the "gtime" (guest time of the task) and
"cgtime" (guest time of the task children) fields for the
tasks. Modify signal_struct and task_struct. Modify /proc/<pid>/stat to display
these new fields.

[PATCH 3/4] modify account_system_time() to add cputime to cpustat->guest if we
are running a VCPU. We add this cputime to  cpustat->user instead of
cpustat->system because this part of KVM code is in fact user code although it
is executed in the kernel. We duplicate VCPU time between guest and user to
allow an unmodified "top(1)" to display correct value. A modified "top(1)" is
able to display good cpu user time and cpu guest time by subtracting cpu guest
time from cpu user time. Update "gtime" and "cgtime" in signal_struct and
task_struct accordingly.

[PATCH 4/4] Modify KVM to update guest time accounting.

Signed-off-by: Laurent Vivier <Laurent.Vivier@bull.net>
-- 
------------- Laurent.Vivier@bull.net  --------------
          "Software is hard" - Donald Knuth

Re: [kvm-devel] [PATCH 0/4] Virtual Machine Time Accounting

[Avi Kivity]

Laurent Vivier wrote:
> The aim of these four patches is to introduce Virtual Machine time accounting.
>   

1-3 look good to me.  4 needs minor edits...

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH 0/4] Virtual Machine Time Accounting

[John Stoffel]

Laurent> The aim of these four patches is to introduce Virtual Machine
Laurent> time accounting.

So what does this buy us?  What increased functionality?  

Laurent> [PATCH 1/4] as recent CPUs introduce a third running state,
Laurent> after "user" and "system", we need a new field, "guest", in
Laurent> cpustat to store the time used by the CPU to run virtual
Laurent> CPU. Modify /proc/stat to display this new field.

What happens when you have multiple guests, is this state cummulative?  

I've glanced over these patches and the #ifdefs are ugly..., esp since
the overhead seems to exist even if you don't run any Guest CPUs.  Or
whatever you're doing here.

Please justify this patch more in terms of what it provides and why
it's a good thing to have.  

Thanks,
John

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Laurent Vivier]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]

Hi John,

John Stoffel wrote:
> 
> Laurent> The aim of these four patches is to introduce Virtual Machine
> Laurent> time accounting.
> 
> So what does this buy us?  What increased functionality?  
> 

functionnalities:

- allow to measure time spent by a CPU in a virtual CPU.
- allow to display in /proc/state this value by CPU
- allow to display in /proc/<pid>/state this value by process
- allow KVM to use these 3 previous functionnalities

> Laurent> [PATCH 1/4] as recent CPUs introduce a third running state,
> Laurent> after "user" and "system", we need a new field, "guest", in
> Laurent> cpustat to store the time used by the CPU to run virtual
> Laurent> CPU. Modify /proc/stat to display this new field.
> 
> What happens when you have multiple guests, is this state cummulative?  

Yes, like system time and user time are.

> I've glanced over these patches and the #ifdefs are ugly..., esp since

#ifdefs are ugly, I agree... I can remove them.
But "#ifdef" allows to save space and time (and it is not a beauty show).

> the overhead seems to exist even if you don't run any Guest CPUs.  Or
> whatever you're doing here.

Can you explain ?

> Please justify this patch more in terms of what it provides and why
> it's a good thing to have.  

Well... it provides the time spent by CPUs in virtual CPUs.
It is usefull only if you run Virtual Machine on your host (like KVM, lguest,
kqemu...) to know how many time your CPUs dedicate to virtual machine execution.
It's a good thing to have like user time and system time are good things to
have... do you use 'top' or 'time' sometime ?


Laurent
-- 
------------- Laurent.Vivier@bull.net  --------------
          "Software is hard" - Donald Knuth


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Jeremy Fitzhardinge]

Laurent Vivier wrote:
> functionnalities:
>
> - allow to measure time spent by a CPU in a virtual CPU.
> - allow to display in /proc/state this value by CPU
> - allow to display in /proc/<pid>/state this value by process
> - allow KVM to use these 3 previous functionnalities
>   

So, currently time spent in a kvm guest is accumulated as qemu-kvm
usertime, right?  Given that qemu knows when its running in qemu vs
guest context, couldn't it provide the breakdown between user and guest
time (ditto lguest)?

    J

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Rusty Russell]

On Mon, 2007-08-20 at 09:13 -0700, Jeremy Fitzhardinge wrote:
> Laurent Vivier wrote:
> > functionnalities:
> >
> > - allow to measure time spent by a CPU in a virtual CPU.
> > - allow to display in /proc/state this value by CPU
> > - allow to display in /proc/<pid>/state this value by process
> > - allow KVM to use these 3 previous functionnalities
> >   
> 
> So, currently time spent in a kvm guest is accumulated as qemu-kvm
> usertime, right?  Given that qemu knows when its running in qemu vs
> guest context, couldn't it provide the breakdown between user and guest
> time (ditto lguest)?

No, unfortunately it's accumulated as system time.

Rusty.

Re: [kvm-devel] [PATCH 0/4] Virtual Machine Time Accounting

[Avi Kivity]

Jeremy Fitzhardinge wrote:
> Laurent Vivier wrote:
>   
>> functionnalities:
>>
>> - allow to measure time spent by a CPU in a virtual CPU.
>> - allow to display in /proc/state this value by CPU
>> - allow to display in /proc/<pid>/state this value by process
>> - allow KVM to use these 3 previous functionnalities
>>   
>>     
>
> So, currently time spent in a kvm guest is accumulated as qemu-kvm
> usertime, right?  Given that qemu knows when its running in qemu vs
> guest context, couldn't it provide the breakdown between user and guest
> time (ditto lguest)?
>   

qemu doesn't (and shouldn't) do accounting; that's best done by 
interrupt driven code.

The patches do account for guest time in a separate counter; guest time 
is added to both user time and the new counter.  This allows an old 
'top' to see guest time (accounted as user time), and a new 'top' to 
separate guest time and user time by performing the appropriate 
mathematical operation.


-- 
error compiling committee.c: too many arguments to function

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Peter Zijlstra]

These patches never seem to have made it onto LKML?!

On Mon, 2007-08-20 at 15:13 +0200, Laurent Vivier wrote:
> The aim of these four patches is to introduce Virtual Machine time accounting.
> 
> _Ingo_, as these patches modify files of the scheduler, could you have a look to
> them, please ?
> 
> [PATCH 1/4] as recent CPUs introduce a third running state, after "user" and
> "system", we need a new field, "guest", in cpustat to store the time used by
> the CPU to run virtual CPU. Modify /proc/stat to display this new field.
> 
> [PATCH 2/4] like for cpustat, introduce the "gtime" (guest time of the task) and
> "cgtime" (guest time of the task children) fields for the
> tasks. Modify signal_struct and task_struct. Modify /proc/<pid>/stat to display
> these new fields.
> 
> [PATCH 3/4] modify account_system_time() to add cputime to cpustat->guest if we
> are running a VCPU. We add this cputime to  cpustat->user instead of
> cpustat->system because this part of KVM code is in fact user code although it
> is executed in the kernel. We duplicate VCPU time between guest and user to
> allow an unmodified "top(1)" to display correct value. A modified "top(1)" is
> able to display good cpu user time and cpu guest time by subtracting cpu guest
> time from cpu user time. Update "gtime" and "cgtime" in signal_struct and
> task_struct accordingly.
> 
> [PATCH 4/4] Modify KVM to update guest time accounting.

Isn't this exactly what CONFIG_VIRT_CPU_ACCOUNTING is about?

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Martin Schwidefsky]

On Tue, 04 Aug 2009 16:16:38 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> These patches never seem to have made it onto LKML?!
> 
> On Mon, 2007-08-20 at 15:13 +0200, Laurent Vivier wrote:
> > The aim of these four patches is to introduce Virtual Machine time accounting.
> > 
> > _Ingo_, as these patches modify files of the scheduler, could you have a look to
> > them, please ?
> > 
> > [PATCH 1/4] as recent CPUs introduce a third running state, after "user" and
> > "system", we need a new field, "guest", in cpustat to store the time used by
> > the CPU to run virtual CPU. Modify /proc/stat to display this new field.
> > 
> > [PATCH 2/4] like for cpustat, introduce the "gtime" (guest time of the task) and
> > "cgtime" (guest time of the task children) fields for the
> > tasks. Modify signal_struct and task_struct. Modify /proc/<pid>/stat to display
> > these new fields.
> > 
> > [PATCH 3/4] modify account_system_time() to add cputime to cpustat->guest if we
> > are running a VCPU. We add this cputime to  cpustat->user instead of
> > cpustat->system because this part of KVM code is in fact user code although it
> > is executed in the kernel. We duplicate VCPU time between guest and user to
> > allow an unmodified "top(1)" to display correct value. A modified "top(1)" is
> > able to display good cpu user time and cpu guest time by subtracting cpu guest
> > time from cpu user time. Update "gtime" and "cgtime" in signal_struct and
> > task_struct accordingly.
> > 
> > [PATCH 4/4] Modify KVM to update guest time accounting.
> 
> Isn't this exactly what CONFIG_VIRT_CPU_ACCOUNTING is about?

Not really, CONFIG_VIRT_CPU_ACCOUNT is a mechanism to sort out the
steal time in the >guest< and to increase the precision of the cpu
accounting values in general. The patches from Laurent improve the
code in the >host< that sorts out guest time vs. system time. The
patches do make sense to me.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Peter Zijlstra]

On Tue, 2009-08-04 at 17:07 +0200, Martin Schwidefsky wrote:
> On Tue, 04 Aug 2009 16:16:38 +0200
> Peter Zijlstra <peterz@infradead.org> wrote:
> 
> > These patches never seem to have made it onto LKML?!
> > 
> > On Mon, 2007-08-20 at 15:13 +0200, Laurent Vivier wrote:
> > > The aim of these four patches is to introduce Virtual Machine time accounting.
> > > 
> > > _Ingo_, as these patches modify files of the scheduler, could you have a look to
> > > them, please ?
> > > 
> > > [PATCH 1/4] as recent CPUs introduce a third running state, after "user" and
> > > "system", we need a new field, "guest", in cpustat to store the time used by
> > > the CPU to run virtual CPU. Modify /proc/stat to display this new field.
> > > 
> > > [PATCH 2/4] like for cpustat, introduce the "gtime" (guest time of the task) and
> > > "cgtime" (guest time of the task children) fields for the
> > > tasks. Modify signal_struct and task_struct. Modify /proc/<pid>/stat to display
> > > these new fields.
> > > 
> > > [PATCH 3/4] modify account_system_time() to add cputime to cpustat->guest if we
> > > are running a VCPU. We add this cputime to  cpustat->user instead of
> > > cpustat->system because this part of KVM code is in fact user code although it
> > > is executed in the kernel. We duplicate VCPU time between guest and user to
> > > allow an unmodified "top(1)" to display correct value. A modified "top(1)" is
> > > able to display good cpu user time and cpu guest time by subtracting cpu guest
> > > time from cpu user time. Update "gtime" and "cgtime" in signal_struct and
> > > task_struct accordingly.
> > > 
> > > [PATCH 4/4] Modify KVM to update guest time accounting.
> > 
> > Isn't this exactly what CONFIG_VIRT_CPU_ACCOUNTING is about?
> 
> Not really, CONFIG_VIRT_CPU_ACCOUNT is a mechanism to sort out the
> steal time in the >guest< and to increase the precision of the cpu
> accounting values in general. The patches from Laurent improve the
> code in the >host< that sorts out guest time vs. system time.

Ah, that wasn't clear, it keeps mentioning guest all over the place ;-)

So its going to split user time into user and guest. Does that really
make sense? For the host kernel it really is just another user process,
no?

>  The patches do make sense to me.

No idea there as they never made it to my lkml folder.

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Martin Schwidefsky]

On Tue, 04 Aug 2009 18:26:41 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> On Tue, 2009-08-04 at 17:07 +0200, Martin Schwidefsky wrote:
> > On Tue, 04 Aug 2009 16:16:38 +0200
> > Peter Zijlstra <peterz@infradead.org> wrote:
> > 
> > > These patches never seem to have made it onto LKML?!
> > > 
> > > On Mon, 2007-08-20 at 15:13 +0200, Laurent Vivier wrote:
> > > > The aim of these four patches is to introduce Virtual Machine time accounting.
> > > > 
> > > > _Ingo_, as these patches modify files of the scheduler, could you have a look to
> > > > them, please ?
> > > > 
> > > > [PATCH 1/4] as recent CPUs introduce a third running state, after "user" and
> > > > "system", we need a new field, "guest", in cpustat to store the time used by
> > > > the CPU to run virtual CPU. Modify /proc/stat to display this new field.
> > > > 
> > > > [PATCH 2/4] like for cpustat, introduce the "gtime" (guest time of the task) and
> > > > "cgtime" (guest time of the task children) fields for the
> > > > tasks. Modify signal_struct and task_struct. Modify /proc/<pid>/stat to display
> > > > these new fields.
> > > > 
> > > > [PATCH 3/4] modify account_system_time() to add cputime to cpustat->guest if we
> > > > are running a VCPU. We add this cputime to  cpustat->user instead of
> > > > cpustat->system because this part of KVM code is in fact user code although it
> > > > is executed in the kernel. We duplicate VCPU time between guest and user to
> > > > allow an unmodified "top(1)" to display correct value. A modified "top(1)" is
> > > > able to display good cpu user time and cpu guest time by subtracting cpu guest
> > > > time from cpu user time. Update "gtime" and "cgtime" in signal_struct and
> > > > task_struct accordingly.
> > > > 
> > > > [PATCH 4/4] Modify KVM to update guest time accounting.
> > > 
> > > Isn't this exactly what CONFIG_VIRT_CPU_ACCOUNTING is about?
> > 
> > Not really, CONFIG_VIRT_CPU_ACCOUNT is a mechanism to sort out the
> > steal time in the >guest< and to increase the precision of the cpu
> > accounting values in general. The patches from Laurent improve the
> > code in the >host< that sorts out guest time vs. system time.
> 
> Ah, that wasn't clear, it keeps mentioning guest all over the place ;-)
> 
> So its going to split user time into user and guest. Does that really
> make sense? For the host kernel it really is just another user process,
> no?

The code (at least in parts) is already upstream. Look at the
account_guest_time function:

static void account_guest_time(struct task_struct *p, cputime_t cputime,
                               cputime_t cputime_scaled)
{
        cputime64_t tmp;
        struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;

        tmp = cputime_to_cputime64(cputime);

        /* Add guest time to process. */
        p->utime = cputime_add(p->utime, cputime);
        p->utimescaled = cputime_add(p->utimescaled, cputime_scaled);
        account_group_user_time(p, cputime);
        p->gtime = cputime_add(p->gtime, cputime);

        /* Add guest time to cpustat. */
        cpustat->user = cputime64_add(cpustat->user, tmp);
        cpustat->guest = cputime64_add(cpustat->guest, tmp);
}

The cpu time for a guest is added to p->utime AND p->gtime. That is
done not to break existing tools that know nothing about guest time.
A guest time aware tool can subtract the p->gtime from p->utime to
get the time spent by the process outside of the guest context.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Peter Zijlstra]

On Tue, 2009-08-04 at 19:29 +0200, Martin Schwidefsky wrote:

> > So its going to split user time into user and guest. Does that really
> > make sense? For the host kernel it really is just another user process,
> > no?
> 
> The code (at least in parts) is already upstream. Look at the
> account_guest_time function:
> 
> static void account_guest_time(struct task_struct *p, cputime_t cputime,
>                                cputime_t cputime_scaled)
> {
>         cputime64_t tmp;
>         struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
> 
>         tmp = cputime_to_cputime64(cputime);
> 
>         /* Add guest time to process. */
>         p->utime = cputime_add(p->utime, cputime);
>         p->utimescaled = cputime_add(p->utimescaled, cputime_scaled);
>         account_group_user_time(p, cputime);
>         p->gtime = cputime_add(p->gtime, cputime);
> 
>         /* Add guest time to cpustat. */
>         cpustat->user = cputime64_add(cpustat->user, tmp);
>         cpustat->guest = cputime64_add(cpustat->guest, tmp);
> }
> 
> The cpu time for a guest is added to p->utime AND p->gtime. That is
> done not to break existing tools that know nothing about guest time.
> A guest time aware tool can subtract the p->gtime from p->utime to
> get the time spent by the process outside of the guest context.

But why? How a vcpu anything other than yet another userspace process?

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Martin Schwidefsky]

On Tue, 04 Aug 2009 19:33:14 +0200
Peter Zijlstra <peterz@infradead.org> wrote:

> On Tue, 2009-08-04 at 19:29 +0200, Martin Schwidefsky wrote:
> 
> > > So its going to split user time into user and guest. Does that really
> > > make sense? For the host kernel it really is just another user process,
> > > no?
> > 
> > The code (at least in parts) is already upstream. Look at the
> > account_guest_time function:
> > 
> > static void account_guest_time(struct task_struct *p, cputime_t cputime,
> >                                cputime_t cputime_scaled)
> > {
> >         cputime64_t tmp;
> >         struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
> > 
> >         tmp = cputime_to_cputime64(cputime);
> > 
> >         /* Add guest time to process. */
> >         p->utime = cputime_add(p->utime, cputime);
> >         p->utimescaled = cputime_add(p->utimescaled, cputime_scaled);
> >         account_group_user_time(p, cputime);
> >         p->gtime = cputime_add(p->gtime, cputime);
> > 
> >         /* Add guest time to cpustat. */
> >         cpustat->user = cputime64_add(cpustat->user, tmp);
> >         cpustat->guest = cputime64_add(cpustat->guest, tmp);
> > }
> > 
> > The cpu time for a guest is added to p->utime AND p->gtime. That is
> > done not to break existing tools that know nothing about guest time.
> > A guest time aware tool can subtract the p->gtime from p->utime to
> > get the time spent by the process outside of the guest context.
> 
> But why? How a vcpu anything other than yet another userspace process?

Because you have two different contexts where you spent time, the
"normal" process context and the "guest" process context. To know how
much of your cpu time is spent on virtualization overhead is an
interesting number.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [PATCH 0/4] Virtual Machine Time Accounting

[Jeremy Fitzhardinge]

On 08/04/09 10:33, Peter Zijlstra wrote:
> On Tue, 2009-08-04 at 19:29 +0200, Martin Schwidefsky wrote:
>
>   
>>> So its going to split user time into user and guest. Does that really
>>> make sense? For the host kernel it really is just another user process,
>>> no?
>>>       
>> The code (at least in parts) is already upstream. Look at the
>> account_guest_time function:
>>
>> static void account_guest_time(struct task_struct *p, cputime_t cputime,
>>                                cputime_t cputime_scaled)
>> {
>>         cputime64_t tmp;
>>         struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
>>
>>         tmp = cputime_to_cputime64(cputime);
>>
>>         /* Add guest time to process. */
>>         p->utime = cputime_add(p->utime, cputime);
>>         p->utimescaled = cputime_add(p->utimescaled, cputime_scaled);
>>         account_group_user_time(p, cputime);
>>         p->gtime = cputime_add(p->gtime, cputime);
>>
>>         /* Add guest time to cpustat. */
>>         cpustat->user = cputime64_add(cpustat->user, tmp);
>>         cpustat->guest = cputime64_add(cpustat->guest, tmp);
>> }
>>
>> The cpu time for a guest is added to p->utime AND p->gtime. That is
>> done not to break existing tools that know nothing about guest time.
>> A guest time aware tool can subtract the p->gtime from p->utime to
>> get the time spent by the process outside of the guest context.
>>     
>
> But why? How a vcpu anything other than yet another userspace process?
>   

Normally a running process can either be accumulating user or system
time.  From the host's perspective, the time spent running a vcpu is
more or less a form of system time[*], but the CPU happens to be in an
odd state (for kvm it would be in an actual completely different CPU
mode; for lguest it would just be configured unusually).

[* I think?  I assume qemu goes into a syscall for a while to run the
vcpu, then returns to qemu userspace when it needs to handle some
exception ]

I don't see that vcpu time is so particularly distinct from any other
kind of system time that it really needs a whole special form of time
accounting, unless the scheduler is being modified to schedule guests
differently from normal processes.  Otherwise it just seems like
something that could be relegated to one of the existing
tracing/benchmarking/profiling mechanisms.

(Full disclosure: these patches are completely irrelevant to Xen.)

    J