IDLE domain is scheduled more than dom0

IDLE domain is scheduled more than dom0

[Tian, Kevin]

Hi, Stephan,
	Latest hg tree has set your sEDF scheduler as default, and then
I saw some strange behavior as mentioned in subject on XEN/IA64. The
source tree is based on xen-ia64-unstable.hg set up by Dan, which was
just newly created yesterday and had default scheduler change as above
too.

	An example is: (No DomU created)
                      Total cpu time
IDLE                   93def9294f
Dom0                  5420288a1b

	Shouldn't IDLE domain not be scheduled for most time? Because
idle task will call into PAL for power save on XEN/IA64, the performance
is really, really bad to boot Dom0. The net effect is about ten times
slower. After adding "sched=bvt", everything back to normal.

	I'm not sure whether same problem also exists on x86, maybe not
that obvious. Is there any special sEDF parameter setting to be done
arch specific...?

	For previous BVT scheduler, the evt (effective virtual time) for
IDLE task is set as -1, which has ideal effect to always schedule Dom0
as long as it's runable. IDLE domain should never be scheduled after
Dom0 is up, unless explicit sleep request issued from Dom0.

	However after looking into sEDF code, it seems that current
running ED is very likely to be moved into wait queue after a period,
while IDLE is always on run queue. Then regarding case with only Dom0
and IDLE, IDLE will be scheduled more frequently than expected, even
when dom0 doesn't request to sleep explicitly. When multiple domains
exist, that phenomenon is instead not that obvious, since domains will
gear to run queue alternatively and prevent IDLE to be scheduled.

	So maybe some special consideration should be added to IDLE
domain...  I'm actually a stranger to this new scheduler, and please
correct me if I throw water on incorrect direction. ;-)

Thanks,
Kevin

Re: IDLE domain is scheduled more than dom0

[Dan Magenheimer]

> 	Shouldn't IDLE domain not be scheduled for most time? Because
> idle task will call into PAL for power save on XEN/IA64, the performance
> is really, really bad to boot Dom0. The net effect is about ten times
> slower. After adding "sched=bvt", everything back to normal.

If the sedf scheduler is scheduling the idle domain when
domain0 is runnable, surely this is affecting performance
on x86 also and is a bug that should be fixed?

Has anyone done any performance testing (on x86) since
sedf was checked in as the default?

Dan

Re: Re: IDLE domain is scheduled more than dom0

[Andrew Theurer]

On Friday 08 July 2005 09:53, Dan Magenheimer wrote:
> > 	Shouldn't IDLE domain not be scheduled for most time? Because
> > idle task will call into PAL for power save on XEN/IA64, the
> > performance is really, really bad to boot Dom0. The net effect is
> > about ten times slower. After adding "sched=bvt", everything back
> > to normal.
>
> If the sedf scheduler is scheduling the idle domain when
> domain0 is runnable, surely this is affecting performance
> on x86 also and is a bug that should be fixed?
>
> Has anyone done any performance testing (on x86) since
> sedf was checked in as the default?

Just tried launching some cpu bound tasks in dom0, and I get only 75% 
cpu util for dom0.  I'll try the other domain scheduler and see if it 
clears it up.

-Andrew


 cpu0:  [075.3] d0-0[075.3]
 cpu1:  [075.3] d0-1[075.3]
 cpu2:  [075.3] d0-2[075.3]
 cpu3:  [074.8] d0-3[074.8]

Re: Re: IDLE domain is scheduled more than dom0

[Andrew Theurer]

On Friday 08 July 2005 11:33, Andrew Theurer wrote:
> On Friday 08 July 2005 09:53, Dan Magenheimer wrote:
> > > 	Shouldn't IDLE domain not be scheduled for most time? Because
> > > idle task will call into PAL for power save on XEN/IA64, the
> > > performance is really, really bad to boot Dom0. The net effect is
> > > about ten times slower. After adding "sched=bvt", everything back
> > > to normal.
> >
> > If the sedf scheduler is scheduling the idle domain when
> > domain0 is runnable, surely this is affecting performance
> > on x86 also and is a bug that should be fixed?
> >
> > Has anyone done any performance testing (on x86) since
> > sedf was checked in as the default?
>
> Just tried launching some cpu bound tasks in dom0, and I get only 75%
> cpu util for dom0.  I'll try the other domain scheduler and see if it
> clears it up.

OK, just confirmed bvt works as expected in ia32.

-Andrew

Re: Re: IDLE domain is scheduled more than dom0

[Bin Ren]

I think it's quite likely that by default SEDF doesn't allow  domains
to use slack CPU time, i.e. non  work-conserving. Each domain is given
an absolute percentage of the total CPU time. Try use command "xm sedf
dom-id 0 0 0 1 0" to give a domain access to slack CPU time. Hopefully
this could restore the performance.

BTW, on my uniprocessor test machine with latest xen-unstable,
xenlinux 2.6.11.12, domU sees significant drop in network throughputs
(~40% less!) I'm interested in whether other people encounter similar
situations, especially on SMP machines.

Cheers,
Bin

On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote:
> On Friday 08 July 2005 11:33, Andrew Theurer wrote:
> > On Friday 08 July 2005 09:53, Dan Magenheimer wrote:
> > > >   Shouldn't IDLE domain not be scheduled for most time? Because
> > > > idle task will call into PAL for power save on XEN/IA64, the
> > > > performance is really, really bad to boot Dom0. The net effect is
> > > > about ten times slower. After adding "sched=bvt", everything back
> > > > to normal.
> > >
> > > If the sedf scheduler is scheduling the idle domain when
> > > domain0 is runnable, surely this is affecting performance
> > > on x86 also and is a bug that should be fixed?
> > >
> > > Has anyone done any performance testing (on x86) since
> > > sedf was checked in as the default?
> >
> > Just tried launching some cpu bound tasks in dom0, and I get only 75%
> > cpu util for dom0.  I'll try the other domain scheduler and see if it
> > clears it up.
> 
> OK, just confirmed bvt works as expected in ia32.
> 
> -Andrew
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Re: IDLE domain is scheduled more than dom0

[Stephan Diestelhorst]

>BTW, on my uniprocessor test machine with latest xen-unstable,

>xenlinux 2.6.11.12, domU sees significant drop in network throughputs
>(~40% less!) I'm interested in whether other people encounter similar
>situations, especially on SMP machines.
>
>
>  
>
Mhmhm, this is interesting, what are the timing parameters for your domains?

Thanks,
   Stephan

>On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote:
>  
>
>>On Friday 08 July 2005 11:33, Andrew Theurer wrote:
>>    
>>
>>>On Friday 08 July 2005 09:53, Dan Magenheimer wrote:
>>>      
>>>
>>>>>  Shouldn't IDLE domain not be scheduled for most time? Because
>>>>>idle task will call into PAL for power save on XEN/IA64, the
>>>>>performance is really, really bad to boot Dom0. The net effect is
>>>>>about ten times slower. After adding "sched=bvt", everything back
>>>>>to normal.
>>>>>          
>>>>>
>>>>If the sedf scheduler is scheduling the idle domain when
>>>>domain0 is runnable, surely this is affecting performance
>>>>on x86 also and is a bug that should be fixed?
>>>>
>>>>Has anyone done any performance testing (on x86) since
>>>>sedf was checked in as the default?
>>>>        
>>>>
>>>Just tried launching some cpu bound tasks in dom0, and I get only 75%
>>>cpu util for dom0.  I'll try the other domain scheduler and see if it
>>>clears it up.
>>>      
>>>
>>OK, just confirmed bvt works as expected in ia32.
>>
>>-Andrew
>>
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xensource.com
>>http://lists.xensource.com/xen-devel
>>
>>    
>>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>  
>

Re: Re: IDLE domain is scheduled more than dom0

[Bin Ren]

Default SEDF settings. I have never changed any SEDF parameters. It's
uniprocessor. With some netback driver instrumentation, I find out
that SEDF preempts a lot more aggresive even than BVT. For example,
when dom1 is sending data to an external host, during each context
switch the number of pages dom0 takes off the Tx ring and delivers to
the NIC always (yes, 100% of the time) fall between 1-8. With BVT, 70%
of batch-sizes fall between 1-8 with the rest scattered around 20-40.
In short, there are too frequent preemption/context switches with SEDF
on uniprocessors.

- Bin

On 7/9/05, Stephan Diestelhorst <sd386@cam.ac.uk> wrote:
> >BTW, on my uniprocessor test machine with latest xen-unstable,
> 
> >xenlinux 2.6.11.12, domU sees significant drop in network throughputs
> >(~40% less!) I'm interested in whether other people encounter similar
> >situations, especially on SMP machines.
> >
> >
> >
> >
> Mhmhm, this is interesting, what are the timing parameters for your domains?
> 
> Thanks,
>    Stephan
> 
> >On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote:
> >
> >
> >>On Friday 08 July 2005 11:33, Andrew Theurer wrote:
> >>
> >>
> >>>On Friday 08 July 2005 09:53, Dan Magenheimer wrote:
> >>>
> >>>
> >>>>>  Shouldn't IDLE domain not be scheduled for most time? Because
> >>>>>idle task will call into PAL for power save on XEN/IA64, the
> >>>>>performance is really, really bad to boot Dom0. The net effect is
> >>>>>about ten times slower. After adding "sched=bvt", everything back
> >>>>>to normal.
> >>>>>
> >>>>>
> >>>>If the sedf scheduler is scheduling the idle domain when
> >>>>domain0 is runnable, surely this is affecting performance
> >>>>on x86 also and is a bug that should be fixed?
> >>>>
> >>>>Has anyone done any performance testing (on x86) since
> >>>>sedf was checked in as the default?
> >>>>
> >>>>
> >>>Just tried launching some cpu bound tasks in dom0, and I get only 75%
> >>>cpu util for dom0.  I'll try the other domain scheduler and see if it
> >>>clears it up.
> >>>
> >>>
> >>OK, just confirmed bvt works as expected in ia32.
> >>
> >>-Andrew
> >>
> >>_______________________________________________
> >>Xen-devel mailing list
> >>Xen-devel@lists.xensource.com
> >>http://lists.xensource.com/xen-devel
> >>
> >>
> >>
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
> >http://lists.xensource.com/xen-devel
> >
> >
> 
>

Re: Re: IDLE domain is scheduled more than dom0

[Stephan Diestelhorst]

Andrew Theurer schrieb:

>On Friday 08 July 2005 09:53, Dan Magenheimer wrote:
>  
>
>>>	Shouldn't IDLE domain not be scheduled for most time? Because
>>>idle task will call into PAL for power save on XEN/IA64, the
>>>performance is really, really bad to boot Dom0. The net effect is
>>>about ten times slower. After adding "sched=bvt", everything back
>>>to normal.
>>>      
>>>
>>If the sedf scheduler is scheduling the idle domain when
>>domain0 is runnable, surely this is affecting performance
>>on x86 also and is a bug that should be fixed?
>>
>>Has anyone done any performance testing (on x86) since
>>sedf was checked in as the default?
>>    
>>
>
>Just tried launching some cpu bound tasks in dom0, and I get only 75% 
>cpu util for dom0.  I'll try the other domain scheduler and see if it 
>clears it up.
>
>-Andrew
>
>
> cpu0:  [075.3] d0-0[075.3]
> cpu1:  [075.3] d0-1[075.3]
> cpu2:  [075.3] d0-2[075.3]
> cpu3:  [074.8] d0-3[074.8]
>  
>
This is the current default behaviour of domain 0, it gets 75% of the
CPU time and no slack time. This is easily changeable, and I guess it is
usefull to do so!
Stephan

Re: Re: IDLE domain is scheduled more than dom0

[Keir Fraser]

On 9 Jul 2005, at 19:14, Stephan Diestelhorst wrote:

> This is the current default behaviour of domain 0, it gets 75% of the
> CPU time and no slack time. This is easily changeable, and I guess it 
> is
> usefull to do so!

The default for all domains should be that they are eligible for slack 
time. Alternatively (or perhaps additionally), CPU time should be given 
to ineligible domains in preference to idling the CPU. There's no good 
reason for the scheduler not to be work conserving, even if no domains 
are marked as wanting slack time.

  -- Keir

RE: Re: IDLE domain is scheduled more than dom0

[Magenheimer, Dan (HP Labs Fort Collins)]

Academically, this may not be a bug, but I think its
fair to argue that domain0 should be a special case
for using "slack" CPU time, at least by default.

Indeed, an argument could/should be made that the
whole concept of an idle domain should go away
entirely.

Dan 

> -----Original Message-----
> From: Bin Ren [mailto:bin.ren@gmail.com] 
> Sent: Friday, July 08, 2005 1:07 PM
> To: Andrew Theurer
> Cc: Magenheimer, Dan (HP Labs Fort Collins); 
> xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0
> 
> I think it's quite likely that by default SEDF doesn't allow  domains
> to use slack CPU time, i.e. non  work-conserving. Each domain is given
> an absolute percentage of the total CPU time. Try use command "xm sedf
> dom-id 0 0 0 1 0" to give a domain access to slack CPU time. Hopefully
> this could restore the performance.
> 
> BTW, on my uniprocessor test machine with latest xen-unstable,
> xenlinux 2.6.11.12, domU sees significant drop in network throughputs
> (~40% less!) I'm interested in whether other people encounter similar
> situations, especially on SMP machines.
> 
> Cheers,
> Bin
> 
> On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote:
> > On Friday 08 July 2005 11:33, Andrew Theurer wrote:
> > > On Friday 08 July 2005 09:53, Dan Magenheimer wrote:
> > > > >   Shouldn't IDLE domain not be scheduled for most 
> time? Because
> > > > > idle task will call into PAL for power save on XEN/IA64, the
> > > > > performance is really, really bad to boot Dom0. The 
> net effect is
> > > > > about ten times slower. After adding "sched=bvt", 
> everything back
> > > > > to normal.
> > > >
> > > > If the sedf scheduler is scheduling the idle domain when
> > > > domain0 is runnable, surely this is affecting performance
> > > > on x86 also and is a bug that should be fixed?
> > > >
> > > > Has anyone done any performance testing (on x86) since
> > > > sedf was checked in as the default?
> > >
> > > Just tried launching some cpu bound tasks in dom0, and I 
> get only 75%
> > > cpu util for dom0.  I'll try the other domain scheduler 
> and see if it
> > > clears it up.
> > 
> > OK, just confirmed bvt works as expected in ia32.
> > 
> > -Andrew
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> 

RE: Re: IDLE domain is scheduled more than dom0

[Magenheimer, Dan (HP Labs Fort Collins)]

Um, why was the default scheduler changed?  Is all this
easily fixed or should the default be changed back to
bvt until sedf demonstrates equal (or preferably better)
performance on some range of benchmarks? 

> -----Original Message-----
> From: Bin Ren [mailto:bin.ren@gmail.com] 
> Sent: Saturday, July 09, 2005 12:29 PM
> To: Stephan Diestelhorst
> Cc: Andrew Theurer; Magenheimer, Dan (HP Labs Fort Collins); 
> xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0
> 
> Default SEDF settings. I have never changed any SEDF parameters. It's
> uniprocessor. With some netback driver instrumentation, I find out
> that SEDF preempts a lot more aggresive even than BVT. For example,
> when dom1 is sending data to an external host, during each context
> switch the number of pages dom0 takes off the Tx ring and delivers to
> the NIC always (yes, 100% of the time) fall between 1-8. With BVT, 70%
> of batch-sizes fall between 1-8 with the rest scattered around 20-40.
> In short, there are too frequent preemption/context switches with SEDF
> on uniprocessors.
> 
> - Bin
> 
> On 7/9/05, Stephan Diestelhorst <sd386@cam.ac.uk> wrote:
> > >BTW, on my uniprocessor test machine with latest xen-unstable,
> > 
> > >xenlinux 2.6.11.12, domU sees significant drop in network 
> throughputs
> > >(~40% less!) I'm interested in whether other people 
> encounter similar
> > >situations, especially on SMP machines.
> > >
> > >
> > >
> > >
> > Mhmhm, this is interesting, what are the timing parameters 
> for your domains?
> > 
> > Thanks,
> >    Stephan
> > 
> > >On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote:
> > >
> > >
> > >>On Friday 08 July 2005 11:33, Andrew Theurer wrote:
> > >>
> > >>
> > >>>On Friday 08 July 2005 09:53, Dan Magenheimer wrote:
> > >>>
> > >>>
> > >>>>>  Shouldn't IDLE domain not be scheduled for most time? Because
> > >>>>>idle task will call into PAL for power save on XEN/IA64, the
> > >>>>>performance is really, really bad to boot Dom0. The 
> net effect is
> > >>>>>about ten times slower. After adding "sched=bvt", 
> everything back
> > >>>>>to normal.
> > >>>>>
> > >>>>>
> > >>>>If the sedf scheduler is scheduling the idle domain when
> > >>>>domain0 is runnable, surely this is affecting performance
> > >>>>on x86 also and is a bug that should be fixed?
> > >>>>
> > >>>>Has anyone done any performance testing (on x86) since
> > >>>>sedf was checked in as the default?
> > >>>>
> > >>>>
> > >>>Just tried launching some cpu bound tasks in dom0, and I 
> get only 75%
> > >>>cpu util for dom0.  I'll try the other domain scheduler 
> and see if it
> > >>>clears it up.
> > >>>
> > >>>
> > >>OK, just confirmed bvt works as expected in ia32.
> > >>
> > >>-Andrew
> > >>
> > >>_______________________________________________
> > >>Xen-devel mailing list
> > >>Xen-devel@lists.xensource.com
> > >>http://lists.xensource.com/xen-devel
> > >>
> > >>
> > >>
> > >
> > >_______________________________________________
> > >Xen-devel mailing list
> > >Xen-devel@lists.xensource.com
> > >http://lists.xensource.com/xen-devel
> > >
> > >
> > 
> >
> 

RE: Re: IDLE domain is scheduled more than dom0

[Ian Pratt]

> > cpu0:  [075.3] d0-0[075.3]
> > cpu1:  [075.3] d0-1[075.3]
> > cpu2:  [075.3] d0-2[075.3]
> > cpu3:  [074.8] d0-3[074.8]
> >  
> >
> This is the current default behaviour of domain 0, it gets 
> 75% of the CPU time and no slack time. This is easily 
> changeable, and I guess it is usefull to do so!

All domains should certainly be eligible for slack time by default
otherwise we'll just confuse people. 

If there are parameter tweaks that help dom0 work better as an IO domain
that's fine, but we should at least it make work as expected for
computer bound stuff too (not that people should really be doing such
stuff in dom0..)

[I suspect most people are only interested in using the weight parameter
rather than setting actual gurantees anyhow] 

I suspect the logic behind giving dom0 a large guarantee is that it that
when it blocks (which it will frequently) its large guarantee will
likely cause it to be scheduled immediately it unblocks, which is
"often" good for IO (not always, as promoting batching may be better in
other circumstances).

We need to get more benchmark numbers that compare sedf and bvt under
more realistic workloads. I'll be particularly interested in reaim and
kernel build numbers.

Stephan: please can you post a fix to the default parameters.

Thanks,
Ian

RE: Re: IDLE domain is scheduled more than dom0

[Tian, Kevin]

>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Stephan
>Diestelhorst
>Sent: Monday, July 11, 2005 11:34 PM
>
>Kevin Tian's measurements (on IA64 hardware):
>SEDF: dom0 (15/20, xoff)
>Dom0                  0x5420288a1b = 361316780571 ~ 361.3 sec
>IDLE                  0x93def9294f = 635101063503 ~ 635.1 sec
>-> total boot time: 996.4 sec ?!
>   ratio: 0.57

This data was collected until login prompt. Currently still some issue with timer, so that value may not be exact accurate. However I did observe that total boot time is about 5 times slower than BVT or the 3rd case, with IDE probe especially slow. I haven't figure out exact reason yet, and just wonder whether some different behavior on IA64 may exaggerate that issue...

>
>SEDF: dom0(15/20, xon)
>Dom0                  0x1040A91728 =  69804300072 ~  69.8 sec
>IDLE                  0x19CE88F3E7 = 110839264231 ~ 110.8 sec
>-> total boot time: 180.6 sec
>   ratio: 0.63

Sorry for this inaccurate data, upon which I stopped test at IDE probe due to slow progress. Rough sense is similar to first case, and anyhow the ratio is still not acceptable.

>
>SEDF: dom0(20/20, xoff)
>Dom0                  0x2D61AF8D5D = 194912423261 ~ 194.9 sec
>IDLE                    0x48FD92BF =   1224577727 ~   1.2 sec
>-> total boot time: 196.1 sec
>   ratio: 162.42

This one was also collected for whole boot process until login prompt.

>
>As you see, for me the fiddling with the parameters of sedf doesn't make much
>difference (even to BVT) and the idle-task always has 4-5 times as much CPU
>time as dom0. In my setup this is due to mounting of NFS devices, which takes
>quite a while, where dom0 is blocked most of the times. So our times might
>not be comparable.

So yes, they're not comparable. In your environment, too many I/O of Dom0 gives up time slice actively, which may shade effect when IDLE is scheduled more unexpectedly. However in my test environment, Dom0 never blocks actively even when doing I/O operation (Current status), which can be considered as a special case to make that corner case more obvious...

Thanks a lot,
Kevin

RE: Re: IDLE domain is scheduled more than dom0

[Tian, Kevin]

Hi, Stephan,
	I finally track down the problem to be caused by some bug in XEN/IA64 itself. Due to some code not suitable running with IDLE domain, sometimes the delta for next timer tick was changed to a very large value when current context is IDLE. Then IDLE ran a relative long period without timer interrupt happening. Finally evaluated cpu_time for IDLE became large due to above bug, though only be scheduled several times. Similar bugs were hidden previously on bvt scheduler, due to IDLE never be scheduled after DomN was up. 

	With a simple workaround, now IDLE only occupied about 1% time, with Dom0 to grasp 99%. Then I'll push out a patch to fix that issue soon. Sorry for blaming your scheduler and it actually looks very nice based on data collected by you. ;-)

Thanks a lot for help,
Kevin

>-----Original Message-----
>From: xen-devel-bounces@lists.xensource.com
>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tian, Kevin
>Sent: Tuesday, July 12, 2005 12:10 PM
>To: Stephan Diestelhorst
>Cc: xen-devel@lists.xensource.com
>Subject: RE: [Xen-devel] Re: IDLE domain is scheduled more than dom0
>
>>From: xen-devel-bounces@lists.xensource.com
>>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Stephan
>>Diestelhorst
>>Sent: Monday, July 11, 2005 11:34 PM
>>
>>Kevin Tian's measurements (on IA64 hardware):
>>SEDF: dom0 (15/20, xoff)
>>Dom0                  0x5420288a1b = 361316780571 ~ 361.3 sec
>>IDLE                  0x93def9294f = 635101063503 ~ 635.1 sec
>>-> total boot time: 996.4 sec ?!
>>   ratio: 0.57
>
>This data was collected until login prompt. Currently still some issue with timer,
>so that value may not be exact accurate. However I did observe that total boot
>time is about 5 times slower than BVT or the 3rd case, with IDE probe
>especially slow. I haven't figure out exact reason yet, and just wonder whether
>some different behavior on IA64 may exaggerate that issue...
>
>>
>>SEDF: dom0(15/20, xon)
>>Dom0                  0x1040A91728 =  69804300072 ~  69.8 sec
>>IDLE                  0x19CE88F3E7 = 110839264231 ~ 110.8 sec
>>-> total boot time: 180.6 sec
>>   ratio: 0.63
>
>Sorry for this inaccurate data, upon which I stopped test at IDE probe due to
>slow progress. Rough sense is similar to first case, and anyhow the ratio is still
>not acceptable.
>
>>
>>SEDF: dom0(20/20, xoff)
>>Dom0                  0x2D61AF8D5D = 194912423261 ~ 194.9 sec
>>IDLE                    0x48FD92BF =   1224577727 ~   1.2 sec
>>-> total boot time: 196.1 sec
>>   ratio: 162.42
>
>This one was also collected for whole boot process until login prompt.
>
>>
>>As you see, for me the fiddling with the parameters of sedf doesn't make
>much
>>difference (even to BVT) and the idle-task always has 4-5 times as much
>CPU
>>time as dom0. In my setup this is due to mounting of NFS devices, which
>takes
>>quite a while, where dom0 is blocked most of the times. So our times might
>>not be comparable.
>
>So yes, they're not comparable. In your environment, too many I/O of Dom0
>gives up time slice actively, which may shade effect when IDLE is scheduled
>more unexpectedly. However in my test environment, Dom0 never blocks
>actively even when doing I/O operation (Current status), which can be
>considered as a special case to make that corner case more obvious...
>
>Thanks a lot,
>Kevin
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel

Re: Re: IDLE domain is scheduled more than dom0

[Stephan Diestelhorst]

Hi Kevin,
  that's good news! 
> Hi, Stephan,
>  I finally track down the problem to be caused by some bug in XEN/IA64
> itself. Due to some code not suitable running with IDLE domain, sometimes
> the delta for next timer tick was changed to a very large value when
> current context is IDLE. Then IDLE ran a relative long period without timer
> interrupt happening. Finally evaluated cpu_time for IDLE became large due
> to above bug, though only be scheduled several times. Similar bugs were
> hidden previously on bvt scheduler, due to IDLE never be scheduled after
> DomN was up.
>
>  With a simple workaround, now IDLE only occupied about 1% time, with Dom0
> to grasp 99%. Then I'll push out a patch to fix that issue soon.
> Sorry for  blaming your scheduler and it actually looks very nice based on
> data collected by you. ;-)

Don't worry, similar things happened to me... With boot-times up to 10 
minutes, because I didn't realize that time-outs were relative, not absolute. 
Resulted in an nice exponential curve :-)

Thanks for your testing!

Stephan
>
> >-----Original Message-----
> >From: xen-devel-bounces@lists.xensource.com
> >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tian, Kevin
> >Sent: Tuesday, July 12, 2005 12:10 PM
> >To: Stephan Diestelhorst
> >Cc: xen-devel@lists.xensource.com
> >Subject: RE: [Xen-devel] Re: IDLE domain is scheduled more than dom0
> >
> >>From: xen-devel-bounces@lists.xensource.com
> >>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Stephan
> >>Diestelhorst
> >>Sent: Monday, July 11, 2005 11:34 PM
> >>
> >>Kevin Tian's measurements (on IA64 hardware):
> >>SEDF: dom0 (15/20, xoff)
> >>Dom0                  0x5420288a1b = 361316780571 ~ 361.3 sec
> >>IDLE                  0x93def9294f = 635101063503 ~ 635.1 sec
> >>-> total boot time: 996.4 sec ?!
> >>   ratio: 0.57
> >
> >This data was collected until login prompt. Currently still some issue
> > with timer, so that value may not be exact accurate. However I did
> > observe that total boot time is about 5 times slower than BVT or the 3rd
> > case, with IDE probe especially slow. I haven't figure out exact reason
> > yet, and just wonder whether some different behavior on IA64 may
> > exaggerate that issue...
> >
> >>SEDF: dom0(15/20, xon)
> >>Dom0                  0x1040A91728 =  69804300072 ~  69.8 sec
> >>IDLE                  0x19CE88F3E7 = 110839264231 ~ 110.8 sec
> >>-> total boot time: 180.6 sec
> >>   ratio: 0.63
> >
> >Sorry for this inaccurate data, upon which I stopped test at IDE probe due
> > to slow progress. Rough sense is similar to first case, and anyhow the
> > ratio is still not acceptable.
> >
> >>SEDF: dom0(20/20, xoff)
> >>Dom0                  0x2D61AF8D5D = 194912423261 ~ 194.9 sec
> >>IDLE                    0x48FD92BF =   1224577727 ~   1.2 sec
> >>-> total boot time: 196.1 sec
> >>   ratio: 162.42
> >
> >This one was also collected for whole boot process until login prompt.
> >
> >>As you see, for me the fiddling with the parameters of sedf doesn't make
> >
> >much
> >
> >>difference (even to BVT) and the idle-task always has 4-5 times as much
> >
> >CPU
> >
> >>time as dom0. In my setup this is due to mounting of NFS devices, which
> >
> >takes
> >
> >>quite a while, where dom0 is blocked most of the times. So our times
> >> might not be comparable.
> >
> >So yes, they're not comparable. In your environment, too many I/O of Dom0
> >gives up time slice actively, which may shade effect when IDLE is
> > scheduled more unexpectedly. However in my test environment, Dom0 never
> > blocks actively even when doing I/O operation (Current status), which can
> > be considered as a special case to make that corner case more obvious...
> >
> >Thanks a lot,
> >Kevin
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@lists.xensource.com
> >http://lists.xensource.com/xen-devel