[PATCH] cpu idle ticks show twice in xen pvm guest

[PATCH] cpu idle ticks show twice in xen pvm guest

[Zhenzhong Duan]

Run below test on xen pvm.
# x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60  \
&& y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') \
&& echo -e  "X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)

@ X:58562301
@ Y:58574282
@ IDLE: 199.600

Normal idle percent should be around 100%.
xen_timer_interrupt called account_idle_ticks to account hypervisor stolen idle ticks 
but these ticks will be accounted again when idle ticks restarted.

Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
Signed-off-by: Joe Jin <joe.jin@oracle.com>

diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
index 163b467..5dcbc91 100644
--- a/arch/x86/xen/time.c
+++ b/arch/x86/xen/time.c
@@ -151,7 +151,6 @@ static void do_stolen_accounting(void)
 
 	ticks = iter_div_u64_rem(blocked, NS_PER_TICK, &blocked);
 	__this_cpu_write(xen_residual_blocked, blocked);
-	account_idle_ticks(ticks);
 }
 
 /* Get the TSC speed from Xen */

Re: [PATCH] cpu idle ticks show twice in xen pvm guest

[Konrad Rzeszutek Wilk]

On Wed, Oct 05, 2011 at 10:11:58PM -0700, Zhenzhong Duan wrote:
> Run below test on xen pvm.
> # x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60  \
> && y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') \
> && echo -e  "X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)
> 
> @ X:58562301
> @ Y:58574282
> @ IDLE: 199.600
> 
> Normal idle percent should be around 100%.
> xen_timer_interrupt called account_idle_ticks to account hypervisor stolen idle ticks 
> but these ticks will be accounted again when idle ticks restarted.
> 
> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
> Signed-off-by: Joe Jin <joe.jin@oracle.com>

Please in the future also CC the maintainers (you can get that using
the scripts/get_maintainer.pl).

Jeremy, any thoughts?
> 
> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
> index 163b467..5dcbc91 100644
> --- a/arch/x86/xen/time.c
> +++ b/arch/x86/xen/time.c
> @@ -151,7 +151,6 @@ static void do_stolen_accounting(void)
>  
>  	ticks = iter_div_u64_rem(blocked, NS_PER_TICK, &blocked);
>  	__this_cpu_write(xen_residual_blocked, blocked);
> -	account_idle_ticks(ticks);
>  }
>  
>  /* Get the TSC speed from Xen */
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] cpu idle ticks show twice in xen pvm guest

[Jeremy Fitzhardinge]

On 10/10/2011 08:53 AM, Konrad Rzeszutek Wilk wrote:
> On Wed, Oct 05, 2011 at 10:11:58PM -0700, Zhenzhong Duan wrote:
>> Run below test on xen pvm.
>> # x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60  \
>> && y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') \
>> && echo -e  "X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)
>>
>> @ X:58562301
>> @ Y:58574282
>> @ IDLE: 199.600
>>
>> Normal idle percent should be around 100%.
>> xen_timer_interrupt called account_idle_ticks to account hypervisor stolen idle ticks 
>> but these ticks will be accounted again when idle ticks restarted.
>>
>> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
>> Signed-off-by: Joe Jin <joe.jin@oracle.com>
> Please in the future also CC the maintainers (you can get that using
> the scripts/get_maintainer.pl).
>
> Jeremy, any thoughts?

Does this affect the accounting of stolen ticks?  If it does, that's not
necessarily a showstopper for this patch, but we'll need to do some more
thinking about it.  Certainly, accurate accounting for idleness is
important.

    J

>> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
>> index 163b467..5dcbc91 100644
>> --- a/arch/x86/xen/time.c
>> +++ b/arch/x86/xen/time.c
>> @@ -151,7 +151,6 @@ static void do_stolen_accounting(void)
>>  
>>  	ticks = iter_div_u64_rem(blocked, NS_PER_TICK, &blocked);
>>  	__this_cpu_write(xen_residual_blocked, blocked);
>> -	account_idle_ticks(ticks);
>>  }
>>  
>>  /* Get the TSC speed from Xen */
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] cpu idle ticks show twice in xen pvm guest

[DuanZhenzhong]

Jeremy Fitzhardinge wrote:
> On 10/10/2011 08:53 AM, Konrad Rzeszutek Wilk wrote:
>   
>> On Wed, Oct 05, 2011 at 10:11:58PM -0700, Zhenzhong Duan wrote:
>>     
>>> Run below test on xen pvm.
>>> # x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60  \
>>> && y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') \
>>> && echo -e  "X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)
>>>
>>> @ X:58562301
>>> @ Y:58574282
>>> @ IDLE: 199.600
>>>
>>> Normal idle percent should be around 100%.
>>> xen_timer_interrupt called account_idle_ticks to account hypervisor stolen idle ticks 
>>> but these ticks will be accounted again when idle ticks restarted.
>>>
>>> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
>>> Signed-off-by: Joe Jin <joe.jin@oracle.com>
>>>       
>> Please in the future also CC the maintainers (you can get that using
>> the scripts/get_maintainer.pl).
>>
>> Jeremy, any thoughts?
>>     
>
> Does this affect the accounting of stolen ticks?  If it does, that's not
> necessarily a showstopper for this patch, but we'll need to do some more
> thinking about it.  Certainly, accurate accounting for idleness is
> important.
>
>     J
>
>   
Thanks Konrad for point out that.
In Original kernel, tick_nohz_restart_sched_tick and 
do_stolen_accounting both called account_idle_ticks.
For a idle pvm, idle/iowait maybe doubled apparently.
Normally Y-X should get close to 6000 in idle system, but in my test 
it's 11981.
But I am still not sure if patch has any impact to other part. Thanks
>>> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
>>> index 163b467..5dcbc91 100644
>>> --- a/arch/x86/xen/time.c
>>> +++ b/arch/x86/xen/time.c
>>> @@ -151,7 +151,6 @@ static void do_stolen_accounting(void)
>>>  
>>>  	ticks = iter_div_u64_rem(blocked, NS_PER_TICK, &blocked);
>>>  	__this_cpu_write(xen_residual_blocked, blocked);
>>> -	account_idle_ticks(ticks);
>>>  }
>>>  
>>>  /* Get the TSC speed from Xen */
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> Please read the FAQ at  http://www.tux.org/lkml/
>>>       
>
>   


-- 
Regards
zhenzhong
--
Oracle Building, No.24 Building, Zhongguancun Software Park
Haidian District, Beijing 100193, China

Re: [PATCH] cpu idle ticks show twice in xen pvm guest

[Ferenc Wagner]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

Jeremy Fitzhardinge <jeremy@goop.org> writes:

>> On Wed, Oct 05, 2011 at 10:11:58PM -0700, Zhenzhong Duan wrote:
>>
>>> Run below test on xen pvm.
>>> # x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60  \
>>> && y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') \
>>> && echo -e  "X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)
>>>
>>> @ X:58562301
>>> @ Y:58574282
>>> @ IDLE: 199.600
>>>
>>> Normal idle percent should be around 100%.
>>> xen_timer_interrupt called account_idle_ticks to account hypervisor stolen idle ticks 
>>> but these ticks will be accounted again when idle ticks restarted.
>>>
>>> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
>>> Signed-off-by: Joe Jin <joe.jin@oracle.com>
>
> Does this affect the accounting of stolen ticks?  If it does, that's not
> necessarily a showstopper for this patch, but we'll need to do some more
> thinking about it.  Certainly, accurate accounting for idleness is
> important.

Please see also http://thread.gmane.org/gmane.linux.kernel/734441, where
I found that the counter doubling isn't always present under 2.6.26.
However, after going to 2.6.32 (Debian lenny-backports kernel, 4th of
April on the graph below) that instability seems to disappear.  Please
note that the following graph shows halved idle and iowait percentages.


[-- Attachment #2: Modified Munin graph --]
[-- Type: image/png, Size: 17739 bytes --]

[-- Attachment #3: Type: text/plain, Size: 708 bytes --]


(I haven't collected steal values, so the numbers don't sum up to 100%.)
I'd be grateful if this discrepancy could be cleared up eventually!
It's heartening to see some progress after more than three years. :)

Actually, as Munin doesn't half the idle and iowait values, but
truncates the (then overflowing) graph at 100%, I was rather surprised
to see iowait completely disappear after the kernel upgrade, and
concluded that it was somehow converted into buggy-looping in blkfront.
Now I see this isn't the case, but the steadily increasing system CPU
usage between reboots is still a mystery.  I'll start a separate thread
for that, just wanted to provide some motivation for this topic.
--
Thanks,
Feri.

Re: [PATCH] cpu idle ticks show twice in xen pvm guest

[Konrad Rzeszutek Wilk]

On Fri, Dec 09, 2011 at 04:47:39PM +0100, Ferenc Wagner wrote:
> Jeremy Fitzhardinge <jeremy@goop.org> writes:
> 
> >> On Wed, Oct 05, 2011 at 10:11:58PM -0700, Zhenzhong Duan wrote:
> >>
> >>> Run below test on xen pvm.
> >>> # x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60  \
> >>> && y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') \
> >>> && echo -e  "X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)
> >>>
> >>> @ X:58562301
> >>> @ Y:58574282
> >>> @ IDLE: 199.600
> >>>
> >>> Normal idle percent should be around 100%.
> >>> xen_timer_interrupt called account_idle_ticks to account hypervisor stolen idle ticks 
> >>> but these ticks will be accounted again when idle ticks restarted.
> >>>
> >>> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
> >>> Signed-off-by: Joe Jin <joe.jin@oracle.com>
> >
> > Does this affect the accounting of stolen ticks?  If it does, that's not
> > necessarily a showstopper for this patch, but we'll need to do some more
> > thinking about it.  Certainly, accurate accounting for idleness is
> > important.
> 
> Please see also http://thread.gmane.org/gmane.linux.kernel/734441, where
> I found that the counter doubling isn't always present under 2.6.26.
> However, after going to 2.6.32 (Debian lenny-backports kernel, 4th of
> April on the graph below) that instability seems to disappear.  Please
> note that the following graph shows halved idle and iowait percentages.
> 

What happenend in Feb?

> 
> (I haven't collected steal values, so the numbers don't sum up to 100%.)
> I'd be grateful if this discrepancy could be cleared up eventually!
> It's heartening to see some progress after more than three years. :)
> 
> Actually, as Munin doesn't half the idle and iowait values, but
> truncates the (then overflowing) graph at 100%, I was rather surprised
> to see iowait completely disappear after the kernel upgrade, and
> concluded that it was somehow converted into buggy-looping in blkfront.
> Now I see this isn't the case, but the steadily increasing system CPU
> usage between reboots is still a mystery.  I'll start a separate thread
> for that, just wanted to provide some motivation for this topic.

Did you add more memory in the system?
> --
> Thanks,
> Feri.

Re: [PATCH] cpu idle ticks show twice in xen pvm guest

[Ferenc Wagner]

Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> writes:

> On Fri, Dec 09, 2011 at 04:47:39PM +0100, Ferenc Wagner wrote:
>
>> Jeremy Fitzhardinge <jeremy@goop.org> writes:
>> 
>>>> On Wed, Oct 05, 2011 at 10:11:58PM -0700, Zhenzhong Duan wrote:
>>>>
>>>>> Run below test on xen pvm.
>>>>> # x=$(cat /proc/stat | grep cpu0 | awk '{print $5}') && sleep 60  \
>>>>> && y=$(cat /proc/stat | grep cpu0 | awk '{print $5}') \
>>>>> && echo -e  "X:$x\nY:$y\nIDLE:" $(echo "scale=3; ($y-$x)/6000*100" | bc)
>>>>>
>>>>> @ X:58562301
>>>>> @ Y:58574282
>>>>> @ IDLE: 199.600
>>>>>
>>>>> Normal idle percent should be around 100%.
>>>>> xen_timer_interrupt called account_idle_ticks to account hypervisor stolen idle ticks 
>>>>> but these ticks will be accounted again when idle ticks restarted.
>>>>>
>>>>> Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
>>>>> Signed-off-by: Joe Jin <joe.jin@oracle.com>
>>>
>>> Does this affect the accounting of stolen ticks?  If it does, that's not
>>> necessarily a showstopper for this patch, but we'll need to do some more
>>> thinking about it.  Certainly, accurate accounting for idleness is
>>> important.
>> 
>> Please see also http://thread.gmane.org/gmane.linux.kernel/734441, where
>> I found that the counter doubling isn't always present under 2.6.26.
>> However, after going to 2.6.32 (Debian lenny-backports kernel, 4th of
>> April on the graph below) that instability seems to disappear.  Please
>> note that the following graph shows halved idle and iowait percentages.
>
> What happenend in Feb?

Probably a live migration or a reboot, I can't remember.  See
http://article.gmane.org/gmane.comp.emulators.xen.devel/57771, where I
described a very similar occurence in detail.

>> (I haven't collected steal values, so the numbers don't sum up to 100%.)
>> I'd be grateful if this discrepancy could be cleared up eventually!
>> It's heartening to see some progress after more than three years. :)
>> 
>> Actually, as Munin doesn't half the idle and iowait values, but
>> truncates the (then overflowing) graph at 100%, I was rather surprised
>> to see iowait completely disappear after the kernel upgrade, and
>> concluded that it was somehow converted into buggy-looping in blkfront.
>> Now I see this isn't the case, but the steadily increasing system CPU
>> usage between reboots is still a mystery.  I'll start a separate thread
>> for that, just wanted to provide some motivation for this topic.
>
> Did you add more memory in the system?

I didn't.  Neither in February (when idle and iowait accounting
temporarily doubled), nor in April (since when the accounting is stable
and the system CPU usage is growing between reboots under the new
kernel).
-- 
Thanks,
Feri.