Tickless/dyntick kernel, highres timer and general time crapectomy

Tickless/dyntick kernel, highres timer and general time crapectomy

[Ralf Baechle]

Time to send bring this to a larger audience.

I'm working on getting dyntick and highres timer support working for MIPS.
This unavoidably implies dumping most of the MIPS-private time
infrastructure we've piled up over the past decade.  Which really is a
major crapectomy.  A first cut of the patches which are tested to run
well on uniprocessor and VSMP Malta kernels is at:

  ftp://ftp.linux-mips.org/pub/linux/mips/people/ralf/dyntick-quilt

It will also likely work on various other simple systems.  A more recent
version of these patches which I haven't yet gotten around to test on
silicon is available at:

  ftp://ftp.linux-mips.org/pub/linux/mips/people/ralf/linux-time.patches

The patchset is both large and intrusive.  One of the things that are still
missing is support for additional clocksources and clockevent_devices.
Particularly Alchemy (and any other system supporting clock scaling) will
need some work there since the cp0 count/compare timer are not useful on
such systems.  Help with those would be greatly appreciated.

  Ralf

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Sergei Shtylyov]

Hello.

Ralf Baechle wrote:
> Particularly Alchemy (and any other system supporting clock scaling) will
> need some work there since the cp0 count/compare timer are not useful on
> such systems.

    Hm, why it's being used then? :-)

>   Ralf

WBR, Sergei

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Ralf Baechle]

On Wed, Jun 06, 2007 at 11:04:57PM +0400, Sergei Shtylyov wrote:

> >Particularly Alchemy (and any other system supporting clock scaling) will
> >need some work there since the cp0 count/compare timer are not useful on
> >such systems.
> 
>    Hm, why it's being used then? :-)

From a looks the whole Alchemy power managment code fishy.

  Ralf

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Ralf Baechle]

On Wed, Jun 06, 2007 at 11:04:57PM +0400, Sergei Shtylyov wrote:

> >Particularly Alchemy (and any other system supporting clock scaling) will
> >need some work there since the cp0 count/compare timer are not useful on
> >such systems.
> 
>    Hm, why it's being used then? :-)

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Franck Bui-Huu]

Hi Ralf,

Ralf Baechle wrote:
> Time to send bring this to a larger audience.
>
> I'm working on getting dyntick and highres timer support working for MIPS.
> This unavoidably implies dumping most of the MIPS-private time
> infrastructure we've piled up over the past decade.  Which really is a
> major crapectomy.  A first cut of the patches which are tested to
> run

That's definitely true. I wrote my own version of clockevent support
yesterday based on your patchset "dyntick-quilt" and I end up rewrite
the whole time.c. The biggest part of the job would be to split this
into several patches to ease the review but I doubt it worth it since
we rewrite it almost from scratch.

Another issue I have is to implement clockevent set_mode() method. You
left it empty but I think we need at least to shut down the timer
interrupt when setting the clock event device. Strangely I haven't
found a "generic" way to stop them through cp0. Have I missed
something ? If not, that would mean that either we need a new hook to
achieve that or we find a way/hack to do that in a mips generic
way. Advice on this point would be appreciated.

> well on uniprocessor and VSMP Malta kernels is at:

>
>   ftp://ftp.linux-mips.org/pub/linux/mips/people/ralf/dyntick-quilt
>
> It will also likely work on various other simple systems.  A more recent
> version of these patches which I haven't yet gotten around to test on
> silicon is available at:
>
>   ftp://ftp.linux-mips.org/pub/linux/mips/people/ralf/linux-time.patches
>

BTW any idea when "time-ntp-make-cmos-update-generic.patch" is going
to be merged into mainline ? Note: I think there's a bug in
notify_cmos_timer(). The following test should be negated, shouldn't
it ?

+	if (no_sync_cmos_clock)
+		mod_timer(&sync_cmos_timer, jiffies + 1);

The other patch named "time-move-to_tm-to-lib.patch" create a new file
in arch/mips/lib directory. This new file is called
"to_tm.c". Shouldn't we call it something less specific like "time.c" ?

-- 
               Franck

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Atsushi Nemoto]

On Thu, 7 Jun 2007 09:59:53 +0200, "Franck Bui-Huu" <vagabon.xyz@gmail.com> wrote:
> The other patch named "time-move-to_tm-to-lib.patch" create a new file
> in arch/mips/lib directory. This new file is called
> "to_tm.c". Shouldn't we call it something less specific like "time.c" ?

In long term, I think it should be removed and replaced by
rtc_time_to_tm() in drivers/rtc/rtc-lib.c (but it will require some
adjustments on caller-side for tm_year value).

---
Atsushi Nemoto

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Ralf Baechle]

On Thu, Jun 07, 2007 at 09:59:53AM +0200, Franck Bui-Huu wrote:

> >I'm working on getting dyntick and highres timer support working for MIPS.
> >This unavoidably implies dumping most of the MIPS-private time
> >infrastructure we've piled up over the past decade.  Which really is a
> >major crapectomy.  A first cut of the patches which are tested to
> >run
> 
> That's definitely true. I wrote my own version of clockevent support
> yesterday based on your patchset "dyntick-quilt" and I end up rewrite
> the whole time.c. The biggest part of the job would be to split this
> into several patches to ease the review but I doubt it worth it since
> we rewrite it almost from scratch.

I'm actually getting closer and closer to the point where keep things in
a nicely split patchset stops being workable.

> Another issue I have is to implement clockevent set_mode() method. You
> left it empty but I think we need at least to shut down the timer
> interrupt when setting the clock event device. Strangely I haven't
> found a "generic" way to stop them through cp0. Have I missed
> something ?

You can mask the count/compare interrupt in the status register like any
other interrupt.  Keep in mind that on many CPUs this interrupt is
shared with the performance counter interrupt so cannot always be disabled.
There is no other disable bit for this interrupt than the IE bit in the
status register.  So it may just have to be ignored.

Other issues to solve:

  o The R4000/R4400 (others?) allow the use of hwint5 for either the
    count/compare interrupt or as a normal hardware interrupt.  The switch
    happens based on a bootstream setting.  There is no config register
    bit for this, so we either have to hardcode the knowledge about the
    affected machines or construct a probe.  Where the count/compare is
    not usable we cannot use this as a clockevent device.
  o Old revisions of the R4000 have a bug where if the count register is
    read in exactly the moment where it matches the compare register the
    timer interrupt is lost.  This means the system will be interrupt-less
    for the next typicall like 86s (at the typical 100MHz clock for the
    affected processors).  The workaround needs to be implemented.

Both are currently the least of my concerns, there is much bigger fish to
catch ...

> BTW any idea when "time-ntp-make-cmos-update-generic.patch" is going
> to be merged into mainline ? Note: I think there's a bug in
> notify_cmos_timer(). The following test should be negated, shouldn't
> it ?
> 
> +	if (no_sync_cmos_clock)
> +		mod_timer(&sync_cmos_timer, jiffies + 1);

In theory the patch should be in -mm to be merged early just after .22.

> The other patch named "time-move-to_tm-to-lib.patch" create a new file
> in arch/mips/lib directory. This new file is called
> "to_tm.c". Shouldn't we call it something less specific like "time.c" ?

I actually may drop that one for now.  Two problems with it:

 o to_tm() is being used in modules but having the function in a lib means
   it may not be linked into the kernel so the modules might fail.
 o arch/mips/lib is not the right place anyway.  There are several nearly
   identical copies of the function around under arch/ so they should be
   unified.

  Ralf

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Franck Bui-Huu]

Ralf Baechle wrote:
> On Thu, Jun 07, 2007 at 09:59:53AM +0200, Franck Bui-Huu wrote:
>
>>> I'm working on getting dyntick and highres timer support working for MIPS.
>>> This unavoidably implies dumping most of the MIPS-private time
>>> infrastructure we've piled up over the past decade.  Which really is a
>>> major crapectomy.  A first cut of the patches which are tested to
>>> run
>> That's definitely true. I wrote my own version of clockevent support
>> yesterday based on your patchset "dyntick-quilt" and I end up rewrite
>> the whole time.c. The biggest part of the job would be to split this
>> into several patches to ease the review but I doubt it worth it since
>> we rewrite it almost from scratch.
>
> I'm actually getting closer and closer to the point where keep things in
> a nicely split patchset stops being workable.
>

Actually I'm wondering if we shouldn't create a new file
"arch/mips/kernel/time2.c" which will be a complete rewrite of the
old one (interrupt handler, function pointers, clocksource,
clockevent). This file would be the future replacement of the old
time.c. This new file would be used only if the board have been
updated accordingly. That may help to migrate...

>> Another issue I have is to implement clockevent set_mode() method. You
>> left it empty but I think we need at least to shut down the timer
>> interrupt when setting the clock event device. Strangely I haven't
>> found a "generic" way to stop them through cp0. Have I missed
>> something ?
>
> You can mask the count/compare interrupt in the status register like any
> other interrupt.  Keep in mind that on many CPUs this interrupt is
> shared with the performance counter interrupt so cannot always be
> disabled.

Well this interrupt could be shared with other devices too, couldn't it ?
If so only the board code can disable it.

> There is no other disable bit for this interrupt than the IE bit in the
> status register.  So it may just have to be ignored.
>

That would mean we can't have a tickless system in these cases, wouldn't
it ?

> Other issues to solve:
>
>   o The R4000/R4400 (others?) allow the use of hwint5 for either the
>     count/compare interrupt or as a normal hardware interrupt.  The switch
>     happens based on a bootstream setting.  There is no config register
>     bit for this, so we either have to hardcode the knowledge about the
>     affected machines or construct a probe.  Where the count/compare is
>     not usable we cannot use this as a clockevent device.
>   o Old revisions of the R4000 have a bug where if the count register is
>     read in exactly the moment where it matches the compare register the
>     timer interrupt is lost.  This means the system will be interrupt-less
>     for the next typicall like 86s (at the typical 100MHz clock for the
>     affected processors).  The workaround needs to be implemented.
>
> Both are currently the least of my concerns, there is much bigger fish to
> catch ...
>

I agree a first implementation without these concerns addressed would
be easier. But it's still good to keep them in mind.

>> BTW any idea when "time-ntp-make-cmos-update-generic.patch" is going
>> to be merged into mainline ? Note: I think there's a bug in
>> notify_cmos_timer(). The following test should be negated, shouldn't
>> it ?
>>
>> +	if (no_sync_cmos_clock)
>> +		mod_timer(&sync_cmos_timer, jiffies + 1);
>
> In theory the patch should be in -mm to be merged early just after .22.
>

Haven't found any trace of it.

-- 
               Franck

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Sergei Shtylyov]

Hello.

Franck Bui-Huu wrote:

> Actually I'm wondering if we shouldn't create a new file
> "arch/mips/kernel/time2.c" which will be a complete rewrite of the
> old one (interrupt handler, function pointers, clocksource,
> clockevent). This file would be the future replacement of the old
> time.c. This new file would be used only if the board have been
> updated accordingly. That may help to migrate...

    We've been there and done that -- for George Anzinger's HRT. :-)

>>> Another issue I have is to implement clockevent set_mode() method. You
>>> left it empty but I think we need at least to shut down the timer
>>> interrupt when setting the clock event device. Strangely I haven't
>>> found a "generic" way to stop them through cp0. Have I missed
>>> something ?

>> You can mask the count/compare interrupt in the status register like any
>> other interrupt.  Keep in mind that on many CPUs this interrupt is
>> shared with the performance counter interrupt so cannot always be
>> disabled.

> Well this interrupt could be shared with other devices too, couldn't it ?
> If so only the board code can disable it.

>> There is no other disable bit for this interrupt than the IE bit in the
>> status register.  So it may just have to be ignored.

> That would mean we can't have a tickless system in these cases, wouldn't
> it ?

    No, it doesn't. Even on dyntick kernels, interrupts do happen several 
times a second. Dynticks have nothing to do with disabling timer interrupts...

WBR, Sergei

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Franck Bui-Huu]

Hi,

[ weird, Gmail thought you were a spamer... ]

Sergei Shtylyov wrote:
>
>    No, it doesn't. Even on dyntick kernels, interrupts do happen several
> times a second. Dynticks have nothing to do with disabling timer
> interrupts...
>

That's true however if your system has 2 clock devices. One is the r4k-hpt
and the other one soemthing else with a higher rating. If you don't stop
r4k-hpt interrupts, how does it work ?

-- 
               Franck

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Sergei Shtylyov]

Hello.

Franck Bui-Huu wrote:

> [ weird, Gmail thought you were a spamer... ]

    Well, I guess I should thank my ISP which indeed hosts many spammers. :-)

> Sergei Shtylyov wrote:

>>    No, it doesn't. Even on dyntick kernels, interrupts do happen several
>> times a second. Dynticks have nothing to do with disabling timer
>> interrupts...
>>
> 
> That's true however if your system has 2 clock devices. One is the r4k-hpt
> and the other one soemthing else with a higher rating. If you don't stop
> r4k-hpt interrupts, how does it work ?

    The unwanted events just gets ignored by higher-level code, IIRC... 
Classic PowerPC CPUs have the same problem (even worse, actually) -- one can't 
disable the decrementer interrupt at all.

WBR, Sergei

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Ralf Baechle]

On Thu, Jun 07, 2007 at 04:44:11PM +0200, Franck Bui-Huu wrote:

> >   No, it doesn't. Even on dyntick kernels, interrupts do happen several
> >times a second. Dynticks have nothing to do with disabling timer
> >interrupts...
> >
> 
> That's true however if your system has 2 clock devices. One is the r4k-hpt
> and the other one soemthing else with a higher rating. If you don't stop
> r4k-hpt interrupts, how does it work ?

To some degree this question is hypothetic because generally the cp0
count/compare timer will be the highest rated counter.

But even if so, the basic solution is the same - just ignore the interrupt
whenever it happens to be triggered.  Or if it isn't shared with an
active performance counter interrupt, you could even disable_irq() it.

  Ralf

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Franck Bui-Huu]

Hi Ralf,

Ralf Baechle wrote:
> On Thu, Jun 07, 2007 at 04:44:11PM +0200, Franck Bui-Huu wrote:
>
>>>   No, it doesn't. Even on dyntick kernels, interrupts do happen several
>>> times a second. Dynticks have nothing to do with disabling timer
>>> interrupts...
>>>
>> That's true however if your system has 2 clock devices. One is the r4k-hpt
>> and the other one soemthing else with a higher rating. If you don't stop
>> r4k-hpt interrupts, how does it work ?
>
> To some degree this question is hypothetic because generally the cp0
> count/compare timer will be the highest rated counter.
>

Well it increments every other clock. So it's not impossible to have a
an other higher rated counter.

> But even if so, the basic solution is the same - just ignore the interrupt
> whenever it happens to be triggered.  Or if it isn't shared with an
> active performance counter interrupt, you could even disable_irq() it.
>

OK, but the current code doesn't seem to support very well multiple
clock event devices. For example the global_cd array is not updated if
a new clock event device is registered. Even ll_timer_interrupt()
handler should be renamed something like ll_hpt_interrupt() for
example.

Thanks,


-- 
               Franck

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Ralf Baechle]

On Fri, Jun 08, 2007 at 10:29:42AM +0200, Franck Bui-Huu wrote:

> Well it increments every other clock. So it's not impossible to have a
> an other higher rated counter.

In practice that's very rare.  Otoh there are reasons why the cp0 counter
might be unusable - clockscaling, no interrupt, CPU powered off.

> >But even if so, the basic solution is the same - just ignore the interrupt
> >whenever it happens to be triggered.  Or if it isn't shared with an
> >active performance counter interrupt, you could even disable_irq() it.
> 
> OK, but the current code doesn't seem to support very well multiple
> clock event devices. For example the global_cd array is not updated if
> a new clock event device is registered. Even ll_timer_interrupt()
> handler should be renamed something like ll_hpt_interrupt() for
> example.

global_cd is meant to only hold the pointers to all processors' count/compare
clockevent devices, nothing else.  So if another clockevent device should
have a higher rating on a particular CPU the content of global_cd[] just
doesn't matter.

  Ralf

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Atsushi Nemoto]

On Thu, 7 Jun 2007 16:48:01 +0100, Ralf Baechle <ralf@linux-mips.org> wrote:
> But even if so, the basic solution is the same - just ignore the interrupt
> whenever it happens to be triggered.  Or if it isn't shared with an
> active performance counter interrupt, you could even disable_irq() it.

Using disable_irq() on the counter/compare interrupt might make the
WAIT instruction really not wait on some chips.  It is implementation
dependent wheather an assertion of masked interrupt break the WAIT
instruction or not.  Busy looping in cpu_idle() would not preferred,
but I'm not sure this is really a problem yet.  Maybe I should have
closer look at dyntick.

---
Atsushi Nemoto

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Ralf Baechle]

On Thu, Jun 07, 2007 at 03:11:48PM +0200, Franck Bui-Huu wrote:

> >In theory the patch should be in -mm to be merged early just after .22.

My bad, I meant -rt - which I'm not following.  Realtime is one step
further than I want to look ahead right now.

  Ralf

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Ralf Baechle]

On Thu, Jun 07, 2007 at 03:11:48PM +0200, Franck Bui-Huu wrote:

> Actually I'm wondering if we shouldn't create a new file
> "arch/mips/kernel/time2.c" which will be a complete rewrite of the
> old one (interrupt handler, function pointers, clocksource,
> clockevent). This file would be the future replacement of the old
> time.c. This new file would be used only if the board have been
> updated accordingly. That may help to migrate...

And we'll still be stuck with the compat crap for years - no thank you.
One of the strength of Linux has always been that if necessary we've been
able to turn the code base upside down as needed - even if sometimes it
takes a snow plough to move old stuff out of the way ;-)

> >You can mask the count/compare interrupt in the status register like any
> >other interrupt.  Keep in mind that on many CPUs this interrupt is
> >shared with the performance counter interrupt so cannot always be
> >disabled.
> 
> Well this interrupt could be shared with other devices too, couldn't it ?
> If so only the board code can disable it.

No, the boot mode bit controls a multiplexer so there is either count/compare
or the external interrupt.

> >There is no other disable bit for this interrupt than the IE bit in the
> >status register.  So it may just have to be ignored.
> >
> 
> That would mean we can't have a tickless system in these cases, wouldn't
> it ?

It would simply mean we will receive a useless every 2^32 cycles - nothing
terrible.  The tickless kernel isn't really tickless.  It's more accurately
called dyntick.  One thing that keeps us from completly getting rid of
regular timer interrupts under all circumstances is the current scheduler.
Only if a CPU is idle Linux can stop the regular timer interrupts for it.
With all the software interrupts that happen to be running on a usual
system we expect just a hand full of interrupts per second.  And the good
news is that mingo's current scheduler work is going to remove the concept
of time slices from the scheduler and once that is done we _finally_ will
be able to go even on non-idle CPUs.

  Ralf

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Franck Bui-Huu]

On 6/7/07, Ralf Baechle <ralf@linux-mips.org> wrote:
> On Thu, Jun 07, 2007 at 03:11:48PM +0200, Franck Bui-Huu wrote:
>
> > Actually I'm wondering if we shouldn't create a new file
> > "arch/mips/kernel/time2.c" which will be a complete rewrite of the
> > old one (interrupt handler, function pointers, clocksource,
> > clockevent). This file would be the future replacement of the old
> > time.c. This new file would be used only if the board have been
> > updated accordingly. That may help to migrate...
>
> And we'll still be stuck with the compat crap for years - no thank you.
> One of the strength of Linux has always been that if necessary we've been
> able to turn the code base upside down as needed - even if sometimes it
> takes a snow plough to move old stuff out of the way ;-)
>

OK.

> > >You can mask the count/compare interrupt in the status register like any
> > >other interrupt.  Keep in mind that on many CPUs this interrupt is
> > >shared with the performance counter interrupt so cannot always be
> > >disabled.
> >
> > Well this interrupt could be shared with other devices too, couldn't it ?
> > If so only the board code can disable it.
>
> No, the boot mode bit controls a multiplexer so there is either count/compare
> or the external interrupt.
>

The board I'm working on shares count/compare interrupt with others
devices. Actually an external interrupt controller, which is a simple
MUX, allows several devices to share the same irq line. To know which
device generates an irq you need to probe this irq controller.

So it seems to me that if the hardware designers decide to not reserve
the highest irq line for the timer irq, you're in trouble...

> > >There is no other disable bit for this interrupt than the IE bit in the
> > >status register.  So it may just have to be ignored.
> > >
> >
> > That would mean we can't have a tickless system in these cases, wouldn't
> > it ?
>
> It would simply mean we will receive a useless every 2^32 cycles - nothing
> terrible.  The tickless kernel isn't really tickless.  It's more accurately
> called dyntick.  One thing that keeps us from completly getting rid of
> regular timer interrupts under all circumstances is the current scheduler.
> Only if a CPU is idle Linux can stop the regular timer interrupts for it.
> With all the software interrupts that happen to be running on a usual
> system we expect just a hand full of interrupts per second.  And the good
> news is that mingo's current scheduler work is going to remove the concept
> of time slices from the scheduler and once that is done we _finally_ will
> be able to go even on non-idle CPUs.
>

OK.

thanks,
-- 
               Franck

Re: Tickless/dyntick kernel, highres timer and general time crapectomy

[Ralf Baechle]

On Fri, Jun 08, 2007 at 11:07:23AM +0200, Franck Bui-Huu wrote:

> devices. Actually an external interrupt controller, which is a simple
> MUX, allows several devices to share the same irq line. To know which
> device generates an irq you need to probe this irq controller.
> 
> So it seems to me that if the hardware designers decide to not reserve
> the highest irq line for the timer irq, you're in trouble...

Yuck.  That's a new configuration ...

Thinking about a really bad one.  If your processor is a MIPS R2 processor,
fine.  If it's older you can't reliably test for a timer interrupt
pending other than by masking all performance counter interrupts, then
testing if IE7 is still pending in the cause register.  If of course that
probe isn't reliable because there is another device you will have to do
more guessing by trying to acknowledge the cp0 compare interrupt.  If IE7
transitioned to 0, there was a timer interrupt pending if not there was
an external interrupt pending and there there may or may not have been
a compare interrupt have been pending.  And if the external goes 1->0 at
just that time you may believe you got a timer interrupt anyway.
Things get slightly better if the external interrupt controller has a
register where interrupt pending can be queried but it's still ...
interesting.

I guess the easy solution in such a case is to not use the compare
interrupt in such a configuration and just ack it.  Or go for an R2
processor.

  Ralf