hires timer patchset [was Re: 2.6.19 -mm merge plans]

hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Bill Rugolsky Jr.]

On Fri, Sep 22, 2006 at 09:01:06PM +0200, Ingo Molnar wrote:
> 
> * Pavel Machek <pavel@suse.cz> wrote:
> 
> > > would be nice to merge the -hrt queue that goes right ontop this 
> > > queue. Even if HIGH_RES_TIMERS is "default n" in the beginning. That 
> > > gives us high-res timers and dynticks which are both very important 
> > > features to certain classes of users/devices.
> > 
> > dynticks give benefit of 0.3W, or 20minutes (IIRC) from 8hours on 
> > thinkpad x60... and they were around for way too long. (When baseline 
> > is hz=250, it is 0.5W from hz=1000 baseline). It would be cool to 
> > finally merge them.
> 
> note that this is a new implementation of dynticks though, not Con's 
> older stuff which you probably used, right? But it's fairly low-impact 
> (just a few lines ontop of hrtimers, here and there), ontop of the 
> long-existing -hrt queue.

Just a data point, FWIW; I've made no systematic effort to find regressions.

We've been running Thomas's (pre-dynticks) 2.6.16-hrt patchset (and
HZ=1000) without issue as part of my standard kernel build on x86 and
x86_64 UP/SMP production hosts -- workstations, PostgreSQL (and other)
servers, and routers --- since I experienced latency/ntp problems with
an unpatched kernel using sata_nv on Tyan 2895 Nvidia CK804-chipset mobo
back in March 2006.  I've also been running the (originally x86-only)
2.6.17-dynticks patch on a Tyan Athlon SMP workstation mobo, and occasionally
on my laptop, so far without issue.

I originally chose Thomas's -hrt variant of John Stultz's timekeeping
patchset because it had known-working x86_64 support ...

We're not doing anything exotic (such as audio, packet capture,
or serving thousands of connections) that might stress the timer or
clock system much, but the x86 routers have GigE traffic and the
x86_64 PostgreSQL servers are often heavily loaded.

Regards,

	Bill Rugolsky

Re: hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Voluspa]

Sorry about dropping CC-s. I'm reading archived lkml and can't see them.
Adding the obvious candidates instead (Andrew, you're not obvious, but if
you'll pull in Thomas's stuff, you'll be...)

On 2006-09-22 20:29:40 Bill Rugolsky Jr. wrote:
> On Fri, Sep 22, 2006 at 09:01:06PM +0200, Ingo Molnar wrote:
> > 
> > * Pavel Machek wrote:
> > 
> > > > would be nice to merge the -hrt queue that goes right ontop
[...]
> > note that this is a new implementation of dynticks though, not Con's
> > older stuff which you probably used, right? But it's fairly
> > low-impact (just a few lines ontop of hrtimers, here and there),
> > ontop of the long-existing -hrt queue.
>
> Just a data point, FWIW; I've made no systematic effort to find
> regressions.
>
> We've been running Thomas's (pre-dynticks) 2.6.16-hrt patchset (and
> HZ=1000) without issue as part of my standard kernel build on x86 and
> x86_64 UP/SMP production hosts -- workstations, PostgreSQL (and other)
> servers, and routers --- since I experienced latency/ntp problems with
> an unpatched kernel using sata_nv on Tyan 2895 Nvidia CK804-chipset
> mobo back in March 2006.  I've also been running the (originally
> x86-only) 2.6.17-dynticks patch on a Tyan Athlon SMP workstation mobo,
> and occasionally on my laptop, so far without issue.

Here's another data point: I tried 2.6.18-rt3 today on a x86_64
notebook. I'm on an eternal quest for extended battery time, so
NO_HZ would be perfect. Long story shortened, HIGH_RES_TIMERS
(prerequisite for NO_HZ) caused the CPU to never step down from max
speed (ondemand, powernow_k8) at 2200 MHz.

In addition, something invisible used very frequent bursts of ~30% SYS
CPU. Turning from PREEMPT_RT to PREEMPT_DESKTOP introduced occasional
bursts of ~50% USER CPU (mixed with the SYS). Toggling RCU model
made no difference.

Going from the default HIGH_RES_RESOLUTION=1000 to 10000 and even
100000 had no impact on the level or frequency of the bursts. 

This resulted in the CPU temp never going lower than 53C, and thusly
the fan kept spinning at 'level 2' - it needs 49C to fall back to
normal speed. That sound tend to drive me insane...

Turning off HIGH_RES_TIMER allowed the CPU to enter the lowest 800 MHz
immediately, and fan to slow down after the normal time, post-boot.

I'd be glad to help investigate this issue, but since -rt is completely
new territory I'd need exact instructions about which knobs to turn.
The CPU usage cause _was_ invisible, at least to "top". And I had all
USB stuff unplugged, and no ehci_hcd loaded (that's another story,
written in the bugzilla). Completely plain console, no X, no net, no
nothing.

Here's a snippet from dmesg pertaining to the timers:

ACPI: PM-Timer IO Port: 0x4008
[...]
Event source pit configured with caps set: 07
time.c: Detected 2201.306 MHz processor.
[...]
Calibrating delay using timer specific routine.. 4404.80 BogoMIPS
(lpj=2202402)
[...]
Using local APIC timer interrupts.
result 12507441
Detected 12.507 MHz APIC timer.
lapic max_delta_ns: 670689262
Event source pit new caps set: 03
Event source lapic configured with caps set: 04
[...]
Real Time Clock Driver v1.12ac
[...]
Time: tsc clocksource has been installed.
Event source pit disabled
Event source lapic configured with caps set: 08
hrtimers: Switched to high resolution mode CPU 0

Oh, and here's a compiler warning fix (when PREEMPT_RT is off):

diff -Nurp linux-2.6.18-rt3/kernel/hrtimer.c linux-2.6.18-rt3-db/kernel/hrtimer.c
--- linux-2.6.18-rt3/kernel/hrtimer.c	2006-09-23 01:12:41.000000000 +0200
+++ linux-2.6.18-rt3-db/kernel/hrtimer.c	2006-09-23 02:16:29.000000000 +0200
@@ -786,7 +786,7 @@ static inline void hrtimer_raise_softirq
 	raise_softirq_irqoff(HRTIMER_SOFTIRQ);
 }
 
-static inline int hrtimer_adjust_softirq_prio(struct hrtimer_cpu_base *base)
+static inline void hrtimer_adjust_softirq_prio(struct hrtimer_cpu_base *base)
 {
 }
 
Mvh
Mats Johannesson

Re: hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Voluspa]

Andi,

See http://marc.theaimsgroup.com/?l=linux-kernel&m=115897798118500&w=2
for my original report.

I've now built without NO_HZ (but still HIGH_RES_TIMERS) and that fixes
the issue. You've got most of my machine specs since previous, even an
output from dmidecode, but feel free to ask for more if this bug
interests you.

I'm not particularly eager to get it solved... the nvidia driver still
needs one more hack for the glue to build against -rt3.

Mvh
Mats Johannesson

Re: hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Andi Kleen]

On Sat, Sep 23, 2006 at 05:25:17PM +0200, Voluspa wrote:
> Andi,
> 
> See http://marc.theaimsgroup.com/?l=linux-kernel&m=115897798118500&w=2
> for my original report.
> 
> I've now built without NO_HZ (but still HIGH_RES_TIMERS) and that fixes
> the issue. You've got most of my machine specs since previous, even an
> output from dmidecode, but feel free to ask for more if this bug
> interests you.
> 
> I'm not particularly eager to get it solved... the nvidia driver still
> needs one more hack for the glue to build against -rt3.

-rt* bugs are handled by Ingo and Thomas.

-Andi

Re: hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Daniel Walker]

On Sat, 2006-09-23 at 04:17 +0200, Voluspa wrote:

> Here's another data point: I tried 2.6.18-rt3 today on a x86_64
> notebook. I'm on an eternal quest for extended battery time, so
> NO_HZ would be perfect. Long story shortened, HIGH_RES_TIMERS
> (prerequisite for NO_HZ) caused the CPU to never step down from max
> speed (ondemand, powernow_k8) at 2200 MHz.
> 
> In addition, something invisible used very frequent bursts of ~30% SYS
> CPU. Turning from PREEMPT_RT to PREEMPT_DESKTOP introduced occasional
> bursts of ~50% USER CPU (mixed with the SYS). Toggling RCU model
> made no difference.
> 

It seems like you don't need all of 2.6.18-rt3 , you just want dynamic
tick .. You can obtain just the HRT/dynamic tick patch from here,

http://www.tglx.de/projects/hrtimers/2.6.18/

Daniel

Re: hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Voluspa]

On Sat, 23 Sep 2006 11:09:26 -0700 Daniel Walker wrote:
> On Sat, 2006-09-23 at 04:17 +0200, Voluspa wrote:
> 
> > Here's another data point: I tried 2.6.18-rt3 today on a x86_64
> > notebook. I'm on an eternal quest for extended battery time, so
> > NO_HZ would be perfect. Long story shortened, HIGH_RES_TIMERS
> > (prerequisite for NO_HZ) caused the CPU to never step down from max
> > speed (ondemand, powernow_k8) at 2200 MHz.
> > 
> > In addition, something invisible used very frequent bursts of ~30%
> > SYS CPU. Turning from PREEMPT_RT to PREEMPT_DESKTOP introduced
> > occasional bursts of ~50% USER CPU (mixed with the SYS). Toggling
> > RCU model made no difference.
> > 
> 
> It seems like you don't need all of 2.6.18-rt3 , you just want dynamic
> tick .. You can obtain just the HRT/dynamic tick patch from here,
> 
> http://www.tglx.de/projects/hrtimers/2.6.18/

Thank you pointing that out. I didn't realise that the features were so
sharply divided. Perhaps I'll even be able to debug it locally, then -
wearing ear-mufflers. I'm not kidding about the fan noise.

At the very least the experiment will tell if it's a -rt combo problem
or NO_HZ on its own.

Mvh
Mats Johannesson

Re: hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Voluspa]

On Sat, 23 Sep 2006 11:09:26 -0700 Daniel Walker wrote:
> On Sat, 2006-09-23 at 04:17 +0200, Voluspa wrote:
[...]
> It seems like you don't need all of 2.6.18-rt3 , you just want dynamic
> tick .. You can obtain just the HRT/dynamic tick patch from here,
> 
> http://www.tglx.de/projects/hrtimers/2.6.18/

I'm not good enough at C yet to immediately see what needs fixing. And
the hour is too late anyways:

  BUILD   arch/x86_64/boot/bzImage
Root device is (3, 2)
Boot sector 512 bytes.
Setup is 4709 bytes.
System is 1461 kB
Kernel: arch/x86_64/boot/bzImage is ready  (#1)
  Building modules, stage 2.
  MODPOST
[usual ACPI section mismatch warning deleted]
WARNING: "monotonic_clock" [drivers/char/hangcheck-timer.ko] undefined!
WARNING: "hrtimer_stop_sched_tick" [drivers/acpi/processor.ko] undefined!
WARNING: "hrtimer_restart_sched_tick" [drivers/acpi/processor.ko] undefined!
CC     arch/x86_64/kernel/cpufreq/powernow-k8.mod.o
LD [M] arch/x86_64/kernel/cpufreq/powernow-k8.ko

Mvh
Mats Johannesson

Re: hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Ingo Molnar]

* Voluspa <lista1@comhem.se> wrote:

> WARNING: "monotonic_clock" [drivers/char/hangcheck-timer.ko] undefined!

turn off the CONFIG_HANGCHECK_TIMER option.

> WARNING: "hrtimer_stop_sched_tick" [drivers/acpi/processor.ko] undefined!
> WARNING: "hrtimer_restart_sched_tick" [drivers/acpi/processor.ko] undefined!

add these two lins to the end of kernel/hrtimer.c:

EXPORT_SYMBOL_GPL(hrtimer_stop_sched_tick);
EXPORT_SYMBOL_GPL(hrtimer_restart_sched_tick);

	Ingo

Re: hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Voluspa]

On Sat, 23 Sep 2006 22:20:27 +0200 Ingo Molnar wrote:
> 
> * Voluspa wrote:
> 
> > WARNING: "monotonic_clock" [drivers/char/hangcheck-timer.ko]
> > undefined!
> 
> turn off the CONFIG_HANGCHECK_TIMER option.
> 
> > WARNING: "hrtimer_stop_sched_tick" [drivers/acpi/processor.ko]
> > undefined! WARNING:
> > "hrtimer_restart_sched_tick" [drivers/acpi/processor.ko] undefined!
> 
> add these two lins to the end of kernel/hrtimer.c:
> 
> EXPORT_SYMBOL_GPL(hrtimer_stop_sched_tick);
> EXPORT_SYMBOL_GPL(hrtimer_restart_sched_tick);

My mind was clouded close to bedtime. I now remember the fix that
was published already at 2.6.17 time, from Steven Rostedt:

http://marc.theaimsgroup.com/?l=linux-kernel&m=115124086410874&w=2

Well, result is that NO_HZ indeed is the culprit for this CPU
issue. /proc/interrupts showed the timer to be stuck on an initial 3044
triggers after boot, while NMI: counted up almost as fast as LOC: (if
that tells any tale). Observing "top -d 1" for awhile revealed SYS
bursting (almost regularly alternating) between 50% and 100% CPU each 2
to 3 seconds. In between it was 0. USER also had the same pattern, but
with much longer duration. Perhaps 10 seconds from one show to the next.

I've gotten the broken out hrt-dyntick1 patches so will be able to
experiment on my own - slowly, on spare time.

I am of course available for any thoughts or trials you can come up
with in the meantime.

Mvh
Mats Johannesson

Re: hires timer patchset [was Re: 2.6.19 -mm merge plans]

[Arjan van de Ven]

> I'm not particularly eager to get it solved... the nvidia driver still
> needs one more hack for the glue to build against -rt3.

that's one for nvidia not this list