2.6.27-rc6: nohz + s2ram = need to press keys to get progress

2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Pavel Machek]

Hi!

The old "you have to press keys to get machine to progress" seems to
be back :-(. Thinkpad x60.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Thomas Gleixner]

On Fri, 12 Sep 2008, Pavel Machek wrote:
> Hi!
> 
> The old "you have to press keys to get machine to progress" seems to
> be back :-(. Thinkpad x60.

/me cries

did this happen between rc5 and rc6 ?

Thanks,

	tglx

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Rafael J. Wysocki]

On Friday, 12 of September 2008, Pavel Machek wrote:
> Hi!
> 
> The old "you have to press keys to get machine to progress" seems to
> be back :-(. Thinkpad x60.

I guess the last bunch of HPET/clockevents patches caused that to happen.

We seem to be doing a "one step forward, one step back" thing here ...

Thanks,
Rafael

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Thomas Gleixner]

On Fri, 12 Sep 2008, Rafael J. Wysocki wrote:

> On Friday, 12 of September 2008, Pavel Machek wrote:
> > Hi!
> > 
> > The old "you have to press keys to get machine to progress" seems to
> > be back :-(. Thinkpad x60.
> 
> I guess the last bunch of HPET/clockevents patches caused that to happen.

Guessing is hardly a good method to get down to the root cause of
those problems.
 
> We seem to be doing a "one step forward, one step back" thing here ...

Well, the fixes for HPET/clockevents fix real bugs and there are no
evident side effects vs. s2ram inside. quite the contrary.

Thanks,

	tglx

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Rafael J. Wysocki]

On Friday, 12 of September 2008, Thomas Gleixner wrote:
> On Fri, 12 Sep 2008, Rafael J. Wysocki wrote:
> 
> > On Friday, 12 of September 2008, Pavel Machek wrote:
> > > Hi!
> > > 
> > > The old "you have to press keys to get machine to progress" seems to
> > > be back :-(. Thinkpad x60.
> > 
> > I guess the last bunch of HPET/clockevents patches caused that to happen.
> 
> Guessing is hardly a good method to get down to the root cause of
> those problems.

Well, sure.

> > We seem to be doing a "one step forward, one step back" thing here ...
> 
> Well, the fixes for HPET/clockevents fix real bugs and there are no
> evident side effects vs. s2ram inside. quite the contrary.

So that must be something different.

Thanks,
Rafael

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Thomas Gleixner]

On Fri, 12 Sep 2008, Rafael J. Wysocki wrote:
> > > We seem to be doing a "one step forward, one step back" thing here ...
> > 
> > Well, the fixes for HPET/clockevents fix real bugs and there are no
> > evident side effects vs. s2ram inside. quite the contrary.
> 
> So that must be something different.

well, there is no way to exclude those patches for sure, but Pavel
should be able to identify the one which causes problems.

Thanks,

	tglx
 

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Thomas Gleixner]

Pavel,

On Fri, 12 Sep 2008, Pavel Machek wrote:
> The old "you have to press keys to get machine to progress" seems to
> be back :-(. Thinkpad x60.

Is there a chance that we get some more information than that ?

Thanks,

	tglx

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Pavel Machek]

On Sun 2008-09-14 04:56:48, Thomas Gleixner wrote:
> Pavel,
> 
> On Fri, 12 Sep 2008, Pavel Machek wrote:
> > The old "you have to press keys to get machine to progress" seems to
> > be back :-(. Thinkpad x60.
> 
> Is there a chance that we get some more information than that ?

Yep.

It does not happen after _every_ s2ram, but when it happens the system
limps around in half-dead state with non-blinking cursor etc. Next
s2ram will not fix it.

nohz=off helps.

Will try 2.6.27-rc6 w/o any custom patches next (not that I have
anything interesting in that area), and then probably 2.6.26.

(I had nohz turned off before 2.6.27-rc6...)

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Pavel Machek]

On Sun 2008-09-14 12:09:13, Pavel Machek wrote:
> On Sun 2008-09-14 04:56:48, Thomas Gleixner wrote:
> > Pavel,
> > 
> > On Fri, 12 Sep 2008, Pavel Machek wrote:
> > > The old "you have to press keys to get machine to progress" seems to
> > > be back :-(. Thinkpad x60.
> > 
> > Is there a chance that we get some more information than that ?
> 
> Yep.
> 
> It does not happen after _every_ s2ram, but when it happens the system
> limps around in half-dead state with non-blinking cursor etc. Next
> s2ram will not fix it.
> 
> nohz=off helps.
> 
> Will try 2.6.27-rc6 w/o any custom patches next (not that I have
> anything interesting in that area), and then probably 2.6.26.

It _does_ happen with mainline 2.6.27-rc6.
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Pavel Machek]

On Sun 2008-09-14 12:14:06, Pavel Machek wrote:
> On Sun 2008-09-14 12:09:13, Pavel Machek wrote:
> > On Sun 2008-09-14 04:56:48, Thomas Gleixner wrote:
> > > Pavel,
> > > 
> > > On Fri, 12 Sep 2008, Pavel Machek wrote:
> > > > The old "you have to press keys to get machine to progress" seems to
> > > > be back :-(. Thinkpad x60.
> > > 
> > > Is there a chance that we get some more information than that ?
> > 
> > Yep.
> > 
> > It does not happen after _every_ s2ram, but when it happens the system
> > limps around in half-dead state with non-blinking cursor etc. Next
> > s2ram will not fix it.
> > 
> > nohz=off helps.
> > 
> > Will try 2.6.27-rc6 w/o any custom patches next (not that I have
> > anything interesting in that area), and then probably 2.6.26.
> 
> It _does_ happen with mainline 2.6.27-rc6.

s2ram hs problem in mainline 2.6.26, too. (Different problem: no
ammount pressing shift helps there.) nohz=off cures it, too.

How to proceed?

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Thomas Gleixner]

On Sun, 14 Sep 2008, Pavel Machek wrote:
> > It _does_ happen with mainline 2.6.27-rc6.
> 
> s2ram hs problem in mainline 2.6.26, too. (Different problem: no
> ammount pressing shift helps there.) nohz=off cures it, too.

Ok, so it's not a regression.

Is the problem in the suspend path or in the resume ?

Thanks,

	tglx

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Pavel Machek]

On Sun 2008-09-14 10:51:04, Thomas Gleixner wrote:
> On Sun, 14 Sep 2008, Pavel Machek wrote:
> > > It _does_ happen with mainline 2.6.27-rc6.
> > 
> > s2ram hs problem in mainline 2.6.26, too. (Different problem: no
> > ammount pressing shift helps there.) nohz=off cures it, too.
> 
> Ok, so it's not a regression.
> 
> Is the problem in the suspend path or in the resume ?

During resume, on both 2.6.26 and 2.6.27-rc6. (27-rc6 actually resumes
if I keep hitting shift, and I get "sleepy" system after that --
cursor does not blink, but machine can be used -- as long as I keep
hitting keys).

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Thomas Gleixner]

On Sun, 14 Sep 2008, Pavel Machek wrote:
> On Sun 2008-09-14 10:51:04, Thomas Gleixner wrote:
> > On Sun, 14 Sep 2008, Pavel Machek wrote:
> > > > It _does_ happen with mainline 2.6.27-rc6.
> > > 
> > > s2ram hs problem in mainline 2.6.26, too. (Different problem: no
> > > ammount pressing shift helps there.) nohz=off cures it, too.
> > 
> > Ok, so it's not a regression.
> > 
> > Is the problem in the suspend path or in the resume ?
> 
> During resume, on both 2.6.26 and 2.6.27-rc6. (27-rc6 actually resumes
> if I keep hitting shift, and I get "sleepy" system after that --
> cursor does not blink, but machine can be used -- as long as I keep
> hitting keys).

Hmm. Can you please provide the output of /proc/timer_list when the
system is in that "sleepy" state.

Thanks,

	tglx

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Pavel Machek]

On Sun 2008-09-14 12:15:04, Thomas Gleixner wrote:
> On Sun, 14 Sep 2008, Pavel Machek wrote:
> > On Sun 2008-09-14 10:51:04, Thomas Gleixner wrote:
> > > On Sun, 14 Sep 2008, Pavel Machek wrote:
> > > > > It _does_ happen with mainline 2.6.27-rc6.
> > > > 
> > > > s2ram hs problem in mainline 2.6.26, too. (Different problem: no
> > > > ammount pressing shift helps there.) nohz=off cures it, too.
> > > 
> > > Ok, so it's not a regression.
> > > 
> > > Is the problem in the suspend path or in the resume ?
> > 
> > During resume, on both 2.6.26 and 2.6.27-rc6. (27-rc6 actually resumes
> > if I keep hitting shift, and I get "sleepy" system after that --
> > cursor does not blink, but machine can be used -- as long as I keep
> > hitting keys).
> 
> Hmm. Can you please provide the output of /proc/timer_list when the
> system is in that "sleepy" state.

-rc5 seems to work ok... is there some patch I should try to revert
first?

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Thomas Gleixner]

On Mon, 15 Sep 2008, Pavel Machek wrote:
> > 
> > Hmm. Can you please provide the output of /proc/timer_list when the
> > system is in that "sleepy" state.

Any chance to get that output ?
 
> -rc5 seems to work ok... is there some patch I should try to revert
> first?

The relevant 9 changes are from:

7c1e76897492d92b6a1c2d6892494d39ded9680c

to

72d43d9bc9210d24d09202eaf219eac09e17b339

Thanks,

	tglx

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 828 bytes --]

On Mon 2008-09-15 07:08:01, Thomas Gleixner wrote:
> On Mon, 15 Sep 2008, Pavel Machek wrote:
> > > 
> > > Hmm. Can you please provide the output of /proc/timer_list when the
> > > system is in that "sleepy" state.
> 
> Any chance to get that output ?

Yep ;-).

> > -rc5 seems to work ok... is there some patch I should try to revert
> > first?
> 
> The relevant 9 changes are from:
> 
> 7c1e76897492d92b6a1c2d6892494d39ded9680c

I did 

cg-seek 7c1e76897492d92b6a1c2d6892494d39ded9680c

...and resulting kernel seems to work okay.

> to
> 
> 72d43d9bc9210d24d09202eaf219eac09e17b339

cg-seek here, and problem is back. Good. I can get the
/proc/timer_list ;-)... attached.
									Pavel


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: delme --]
[-- Type: text/plain, Size: 3425 bytes --]

Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 78679262236 nsecs

cpu: 0
 clock 0:
  .index:      0
  .resolution: 4000250 nsecs
  .get_time:   ktime_get_real
active timers:
 clock 1:
  .index:      1
  .resolution: 4000250 nsecs
  .get_time:   ktime_get
active timers:
 #0: <f5d67eb8>, it_real_fn, S:01, do_setitimer, rpc.nfsd/1488
 # expires at 79800076403 nsecs [in 1120814167 nsecs]
 #1: <f5d67eb8>, it_real_fn, S:01, do_setitimer, rpc.mountd/1486
 # expires at 79800101756 nsecs [in 1120839520 nsecs]
 #2: <f5d67eb8>, it_real_fn, S:01, do_setitimer, syslogd/1446
 # expires at 106836398898 nsecs [in 28157136662 nsecs]
 #3: <f5d67eb8>, it_real_fn, S:01, do_setitimer, qmgr/1603
 # expires at 389997707387 nsecs [in 311318445151 nsecs]
 #4: <f5d67eb8>, it_real_fn, S:01, do_setitimer, pickup/1602
 # expires at 411025536414 nsecs [in 332346274178 nsecs]
 #5: <f5d67eb8>, it_real_fn, S:01, do_setitimer, master/1601
 # expires at 411025663664 nsecs [in 332346401428 nsecs]
 #6: <f5d67eb8>, it_real_fn, S:01, do_setitimer, mutt/1627
 # expires at 949848265489 nsecs [in 871169003253 nsecs]
 #7: <f5d67eb8>, it_real_fn, S:01, do_setitimer, smtp/1606
 # expires at 6056997804606 nsecs [in 5978318542370 nsecs]
 #8: <f5d67eb8>, it_real_fn, S:01, do_setitimer, bounce/1638
 # expires at 6056997871724 nsecs [in 5978318609488 nsecs]
  .nohz_mode      : 1
  .idle_tick      : 78676896750 nsecs
  .tick_stopped   : 1
  .idle_jiffies   : 4294911965
  .idle_calls     : 10530
  .idle_sleeps    : 4823
  .idle_entrytime : 78676079360 nsecs
  .idle_waketime  : 78676019925 nsecs
  .idle_exittime  : 78676052541 nsecs
  .idle_sleeptime : 72788008686 nsecs
  .last_jiffies   : 4294911965
  .next_jiffies   : 4294911988
  .idle_expires   : 78768000000 nsecs
jiffies: 4294911965

cpu: 1
 clock 0:
  .index:      0
  .resolution: 4000250 nsecs
  .get_time:   ktime_get_real
active timers:
 clock 1:
  .index:      1
  .resolution: 4000250 nsecs
  .get_time:   ktime_get
active timers:
  .nohz_mode      : 1
  .idle_tick      : 78543351750 nsecs
  .tick_stopped   : 0
  .idle_jiffies   : 4294911931
  .idle_calls     : 7365
  .idle_sleeps    : 1350
  .idle_entrytime : 78540354359 nsecs
  .idle_waketime  : 76836307895 nsecs
  .idle_exittime  : 78676087671 nsecs
  .idle_sleeptime : 77463218593 nsecs
  .last_jiffies   : 4294911931
  .next_jiffies   : 2431226
  .idle_expires   : 10024904000000 nsecs
jiffies: 4294911965


Tick Device: mode:     1
Clock Event Device: hpet
 max_delta_ns:   2147483647
 min_delta_ns:   5000
 mult:           61496114
 shift:          32
 mode:           3
 next_event:     26228000000 nsecs
 set_next_event: hpet_legacy_next_event
 set_mode:       hpet_legacy_set_mode
 event_handler:  tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000000


Tick Device: mode:     1
Clock Event Device: lapic
 max_delta_ns:   807384766
 min_delta_ns:   1443
 mult:           44624068
 shift:          32
 mode:           3
 next_event:     78680897000 nsecs
 set_next_event: lapic_next_event
 set_mode:       lapic_timer_setup
 event_handler:  tick_nohz_handler

Tick Device: mode:     1
Clock Event Device: lapic
 max_delta_ns:   807384766
 min_delta_ns:   1443
 mult:           44624068
 shift:          32
 mode:           3
 next_event:     78683360500 nsecs
 set_next_event: lapic_next_event
 set_mode:       lapic_timer_setup
 event_handler:  tick_nohz_handler

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Pavel Machek]

> > to
> > 
> > 72d43d9bc9210d24d09202eaf219eac09e17b339
> 
> cg-seek here, and problem is back. Good. I can get the
> /proc/timer_list ;-)... attached.

(Actually, timer_list was from more recent kernel, sorry about that, I
was confused. But I verified, and 72d... is still broken).

Trying commit 7cfb0435330364f90f274a26ecdc5f47f738498c now... bad,
too.

Trying 1fb9b7d29d8e85ba3196eaa7ab871bf76fc98d36... bad, too.

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Thomas Gleixner]

On Mon, 15 Sep 2008, Pavel Machek wrote:
> > cg-seek here, and problem is back. Good. I can get the
> > /proc/timer_list ;-)... attached.
> 
> (Actually, timer_list was from more recent kernel, sorry about that, I
> was confused. But I verified, and 72d... is still broken).

Is the timer_list output is from that "sleepy" state ? If not please
provide one.

Thanks,
	tglx

Re: 2.6.27-rc6: nohz + s2ram = need to press keys to get progress

[Thomas Gleixner]

On Tue, 16 Sep 2008, Thomas Gleixner wrote:
> On Mon, 15 Sep 2008, Pavel Machek wrote:
> > > cg-seek here, and problem is back. Good. I can get the

Does the patch below fix it ?

Thanks,

	tglx
----
diff --git a/kernel/time/clockevents.c b/kernel/time/clockevents.c
index 1876b52..eb8736f 100644
--- a/kernel/time/clockevents.c
+++ b/kernel/time/clockevents.c
@@ -69,6 +69,9 @@ void clockevents_set_mode(struct clock_event_device *dev,
 		dev->set_mode(mode, dev);
 		dev->mode = mode;
 	}
+
+	if (mode == CLOCK_EVT_MODE_SHUTDOWN)
+		dev->next_event.tv64 = KTIME_MAX;
 }
 
 /**
diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c
index 2f5a382..6b0b230 100644
--- a/kernel/time/tick-broadcast.c
+++ b/kernel/time/tick-broadcast.c
@@ -434,6 +434,19 @@ again:
 }
 
 /*
+ * We shutdown the device which will stop anyway, but we keep the
+ * next_event untouched as it carries the information when we
+ * broadcast.
+ */
+static void tick_broadcast_shutdown_device(struct clock_event_device *dev)
+{
+	if (dev->mode != CLOCK_EVT_MODE_SHUTDOWN) {
+		dev->set_mode(CLOCK_EVT_MODE_SHUTDOWN, dev);
+		dev->mode = CLOCK_EVT_MODE_SHUTDOWN;
+	}
+}
+
+/*
  * Powerstate information: The system enters/leaves a state, where
  * affected devices might stop
  */
@@ -464,7 +477,7 @@ void tick_broadcast_oneshot_control(unsigned long reason)
 	if (reason == CLOCK_EVT_NOTIFY_BROADCAST_ENTER) {
 		if (!cpu_isset(cpu, tick_broadcast_oneshot_mask)) {
 			cpu_set(cpu, tick_broadcast_oneshot_mask);
-			clockevents_set_mode(dev, CLOCK_EVT_MODE_SHUTDOWN);
+			tick_broadcast_shutdown_device(dev);
 			if (dev->next_event.tv64 < bc->next_event.tv64)
 				tick_broadcast_set_event(dev->next_event, 1);
 		}