pv_ops (2.6.31) and CONFIG_NO_HZ

pv_ops (2.6.31) and CONFIG_NO_HZ

[Dan Magenheimer]

Has anyone looked at using CONFIG_NO_HZ with an upstream
pv_ops kernel?  This seems like an intelligent combination...
since pvclock is a good timesource, it seems like there
should be no reason to have a regular tick, and one
could pack more idle domains into one physical machine.

I tried enabling CONFIG_NO_HZ from an otherwise working .config
file for 2.6.31 and the new kernel (running as a PV domU) panics early
in boot ("Kernel panic -- not syncing: Attempted to kill init!"
with a call trace of syscall_call->sys_exit_group->do_group_exit->
do_exit->panic).  So I'm wondering if I did something stupid
or it is known to not work.

Thanks,
Dan

Re: pv_ops (2.6.31) and CONFIG_NO_HZ

[Keir Fraser]

Periodic ticks are disabled for non-running PV guests, so I don't think
no-hz has as much potential win as you might think.

 -- Keir

On 02/11/2009 21:45, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> Has anyone looked at using CONFIG_NO_HZ with an upstream
> pv_ops kernel?  This seems like an intelligent combination...
> since pvclock is a good timesource, it seems like there
> should be no reason to have a regular tick, and one
> could pack more idle domains into one physical machine.
> 
> I tried enabling CONFIG_NO_HZ from an otherwise working .config
> file for 2.6.31 and the new kernel (running as a PV domU) panics early
> in boot ("Kernel panic -- not syncing: Attempted to kill init!"
> with a call trace of syscall_call->sys_exit_group->do_group_exit->
> do_exit->panic).  So I'm wondering if I did something stupid
> or it is known to not work.
> 
> Thanks,
> Dan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: pv_ops (2.6.31) and CONFIG_NO_HZ

[Dan Magenheimer]

> Periodic ticks are disabled for non-running PV guests, so I 
> don't think
> no-hz has as much potential win as you might think.

In this case, I was mostly interested in HV guests, but
happened to have a 2.6.31 build in a handy PV guest so
thought I would give it a try.

But as long as we are on the PV topic: Does this mean that
a PV guest that is idle will never get scheduled due to
needing to process a tick?  I do see that an idle
PV domain is doing about 250 kernel rdtsc/second, so
is this just because the Xen scheduler has nothing
better to do with a pcpu so it schedules an "idle"
vcpu on it?

> On 02/11/2009 21:45, "Dan Magenheimer" 
> <dan.magenheimer@oracle.com> wrote:
> 
> > Has anyone looked at using CONFIG_NO_HZ with an upstream
> > pv_ops kernel?  This seems like an intelligent combination...
> > since pvclock is a good timesource, it seems like there
> > should be no reason to have a regular tick, and one
> > could pack more idle domains into one physical machine.
> > 
> > I tried enabling CONFIG_NO_HZ from an otherwise working .config
> > file for 2.6.31 and the new kernel (running as a PV domU) 
> panics early
> > in boot ("Kernel panic -- not syncing: Attempted to kill init!"
> > with a call trace of syscall_call->sys_exit_group->do_group_exit->
> > do_exit->panic).  So I'm wondering if I did something stupid
> > or it is known to not work.
> > 
> > Thanks,
> > Dan
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 
> 
>

Re: pv_ops (2.6.31) and CONFIG_NO_HZ

[Keir Fraser]

On 02/11/2009 22:29, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> But as long as we are on the PV topic: Does this mean that
> a PV guest that is idle will never get scheduled due to
> needing to process a tick?  I do see that an idle
> PV domain is doing about 250 kernel rdtsc/second, so
> is this just because the Xen scheduler has nothing
> better to do with a pcpu so it schedules an "idle"
> vcpu on it?

A PV guest is supposed to set a one-shot timer on the next event in its
timer queues. So it depends what sorts of events get put on its timer queues
-- if there are periodic events, it will still request seemingly periodic
one-shot wakeups.

 -- Keir

Re: pv_ops (2.6.31) and CONFIG_NO_HZ

[Jeremy Fitzhardinge]

On 11/02/09 13:45, Dan Magenheimer wrote:
> Has anyone looked at using CONFIG_NO_HZ with an upstream
> pv_ops kernel?  This seems like an intelligent combination...
> since pvclock is a good timesource, it seems like there
> should be no reason to have a regular tick, and one
> could pack more idle domains into one physical machine.
>   

Yes, I've always recommeded NO_HZ.  The pvops kernel disables the 100Hz
periodic timer and always uses the high-precision timers for its
clockevents.  The core kernel still asks for periodic interrupts on
occasion, so it isn't completely tickless, but that's not really
something we should solve at the Xen level.

> I tried enabling CONFIG_NO_HZ from an otherwise working .config
> file for 2.6.31 and the new kernel (running as a PV domU) panics early
> in boot ("Kernel panic -- not syncing: Attempted to kill init!"
> with a call trace of syscall_call->sys_exit_group->do_group_exit->
> do_exit->panic).  So I'm wondering if I did something stupid
> or it is known to not work.
>   

No, that's strange.  However, someone else reported something similar
which turns out to be triggered by the pvclock vsyscall stuff.   I
haven't worked out what's failing yet.  (Or why NO_HZ would depend on
it, though it might be related to the highres timers options.)

Can you post the full crash message?

    J

RE: pv_ops (2.6.31) and CONFIG_NO_HZ

[Dan Magenheimer]

> On 11/02/09 13:45, Dan Magenheimer wrote:
> > Has anyone looked at using CONFIG_NO_HZ with an upstream
> > pv_ops kernel?  This seems like an intelligent combination...
> > since pvclock is a good timesource, it seems like there
> > should be no reason to have a regular tick, and one
> > could pack more idle domains into one physical machine.  
> 
> Yes, I've always recommeded NO_HZ.  The pvops kernel disables 
> the 100Hz
> periodic timer and always uses the high-precision timers for its
> clockevents.  The core kernel still asks for periodic interrupts on
> occasion, so it isn't completely tickless, but that's not really
> something we should solve at the Xen level.

OK thanks.  I'll look into this a bit further for HVMs.

> > I tried enabling CONFIG_NO_HZ from an otherwise working .config
> > file for 2.6.31 and the new kernel (running as a PV domU) 
> panics early
> > in boot ("Kernel panic -- not syncing: Attempted to kill init!"
> > with a call trace of syscall_call->sys_exit_group->do_group_exit->
> > do_exit->panic).  So I'm wondering if I did something stupid
> > or it is known to not work.

> No, that's strange.  However, someone else reported something similar
> which turns out to be triggered by the pvclock vsyscall stuff.   I
> haven't worked out what's failing yet.  (Or why NO_HZ would depend on
> it, though it might be related to the highres timers options.)

This was a kernel.org 2.6.31 kernel, not from your git tree
so no pvclock vsyscall stuff would be in it.
(Sorry I guess the subject was confusing.)

> Can you post the full crash message?

No, most of it scrolls away in the VNC window.  But
since I now know it is supposed to work, I'll look into it
further.

Thanks,
Dan