ondemand vs suspend.

ondemand vs suspend.

[Dave Jones]

Here's a puzzling case that Peter stumbled across when experimenting
with suspend on one of the new dual-core Intel-based Mac Mini's.

The hibernate process would just lock up, and never shut down.

 4190 ?        S<     0:00  \_ [kstopmachine]
 4191 ?        Z<     0:00      \_ [kstopmachine] <defunct>
  
dmesg showed that we had offlined the 2nd core..

PM: suspend-to-disk mode set to 'platform'
Freezing cpus ...
Breaking affinity for irq 0
Breaking affinity for irq 14
Breaking affinity for irq 217
Breaking affinity for irq 225
CPU 1 is now offline
SMP alternatives: switching to UP code

and sysrq-t showed that we were erm.. spinning.

ondemand      D EDDD1300  3452  1462     11          2160  1345 (L-TLB)
       f7f8eec0 f7f8eef4 f7f8eef4 eddd1300 003d10ba c05a4c2c 0000000a f7eb4fcc
       f7eb4eb0 c06e0c00 c19c4860 eddd1300 003d10ba 00000000 00000000 00000000
       f7f8eedc c041fc12 00000296 ffffffff f7f8eef4 00000246 00000246 c06e65d8
Call Trace:
 <c0607f89> __down+0x131/0x14a  <c0605432> __down_failed+0xa/0x10
 <c043aa55> .text.lock.cpu+0x2b/0x32  <c043a7ad> lock_cpu_hotplug+0xa/0xc
 <c05a3dae> __cpufreq_driver_target+0x15/0x6d  <f8bce3da> do_dbs_timer+0x229/0x287 [cpufreq_ondemand]
 <c04321fb> run_workqueue+0x81/0xc0  <c0432c35> worker_thread+0xce/0x101
 <c0435353> kthread+0xa3/0xd0  <c0402005> kernel_thread_helper+0x5/0xb
pm-hibernate  D F395EB00  1816  3934   2495                     (NOTLB)
       e632ade8 c0465c6c 00000004 f395eb00 003d10ba f2c31800 00000007 ddc7e44c
       ddc7e330 c1b1b190 c19c4860 f395eb00 003d10ba 00000000 00000000 c0481c9c
       f8bcfa40 f8bcfa40 e632a000 e632adf8 00000000 c0438ad8 f8bce646 f8bcfa54
Call Trace:
 <c06075ba> __mutex_lock_slowpath+0x19b/0x3e3  <c0607826> mutex_lock+0x24/0x27
 <f8bce646> cpufreq_governor_dbs+0x20e/0x2b0 [cpufreq_ondemand]  <c05a385f> __cpufreq_governor+0x57/0xd8
 <c05a4106> cpufreq_remove_dev+0x197/0x208  <c05a4716> cpufreq_cpu_callback+0x49/0x52
 <c0609795> notifier_call_chain+0x18/0x29  <c042f9da> blocking_notifier_call_chain+0x37/0x4f
 <c043a98d> cpu_down+0x12b/0x1c8  <c04420b0> disable_nonboot_cpus+0x35/0xa1
 <c043fc93> prepare_processes+0x13/0x3b  <c043fedd> pm_suspend_disk+0x9/0xf3
 <c043eead> enter_state+0x53/0x1c1  <c043f0a1> state_store+0x86/0x9c
 <c04a4d14> subsys_attr_store+0x20/0x25  <c04a4e18> sysfs_write_file+0xab/0xd1
 <c046b25c> vfs_write+0xab/0x154  <c046b890> sys_write+0x3b/0x60
 <c060837f> syscall_call+0x7/0xb
kstopmachine  S ED25F800  3428  4190     11  4191          2160 (L-TLB)
       dc184fac c19c4860 c04f473a ed25f800 003d10ba c041115b 00000003 f7e38e4c
       f7e38d30 c1acf050 c19cc860 ed25f800 003d10ba 00000000 00000001 00000246
       e632ae94 dc184f90 00000246 e632ae90 00000000 c041ee49 00000000 00000000
Call Trace:
 <c044579f> do_stop+0xf9/0x129  <c0435353> kthread+0xa3/0xd0
 <c0402005> kernel_thread_helper+0x5/0xb
kstopmachine  X ED25F800  3736  4191   4190                     (L-TLB)
       f7ecefb4 c0605fda f7ecef78 ed25f800 003d10ba 003d10ba 00000002 f2e622cc
       f2e621b0 ddc7e330 c19c4860 ed25f800 003d10ba 00000000 00000000 00000000
       f7ecef94 00000202 00000202 f2ec1ad8 00000000 c0608093 f7ecefb4 00000202
Call Trace:
 <c0427428> do_exit+0x7a1/0x7ab  <c040200b> kernel_thread_helper_end+0x0/0xe

D    pm-hibernate: 3934 [ddc7e330, 118] blocked on mutex: [f8bcfa40] {dbs_mutex}
.. held by:          ondemand: 1462 [f7eb4eb0, 110]
... acquired at:               do_dbs_timer+0x13/0x287 [cpufreq_ondemand]


This doesn't happen if the govenor is 'userspace', so I'm wondering if
we're doing something silly in ondemand when we've offlined a CPU.

Any ideas ?

		Dave

-- 
http://www.codemonkey.org.uk

RE: ondemand vs suspend.

[Pallipadi, Venkatesh]

Is this with 2.6.16 based kernel?

Andi pushed in a patch to fix this recently 
Here -
http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=c
ommit;h=6810b548b25114607e0814612d84125abccc0a4f


Thanks,
Venki 

>-----Original Message-----
>From: Dave Jones [mailto:davej@redhat.com] 
>Sent: Tuesday, June 13, 2006 1:53 PM
>To: cpufreq@lists.linux.org.uk
>Cc: Pallipadi, Venkatesh; pjones@redhat.com
>Subject: ondemand vs suspend.
>
>Here's a puzzling case that Peter stumbled across when experimenting
>with suspend on one of the new dual-core Intel-based Mac Mini's.
>
>The hibernate process would just lock up, and never shut down.
>
> 4190 ?        S<     0:00  \_ [kstopmachine]
> 4191 ?        Z<     0:00      \_ [kstopmachine] <defunct>
>  
>dmesg showed that we had offlined the 2nd core..
>
>PM: suspend-to-disk mode set to 'platform'
>Freezing cpus ...
>Breaking affinity for irq 0
>Breaking affinity for irq 14
>Breaking affinity for irq 217
>Breaking affinity for irq 225
>CPU 1 is now offline
>SMP alternatives: switching to UP code
>
>and sysrq-t showed that we were erm.. spinning.
>
>ondemand      D EDDD1300  3452  1462     11          2160  1345 (L-TLB)
>       f7f8eec0 f7f8eef4 f7f8eef4 eddd1300 003d10ba c05a4c2c 
>0000000a f7eb4fcc
>       f7eb4eb0 c06e0c00 c19c4860 eddd1300 003d10ba 00000000 
>00000000 00000000
>       f7f8eedc c041fc12 00000296 ffffffff f7f8eef4 00000246 
>00000246 c06e65d8
>Call Trace:
> <c0607f89> __down+0x131/0x14a  <c0605432> __down_failed+0xa/0x10
> <c043aa55> .text.lock.cpu+0x2b/0x32  <c043a7ad> 
>lock_cpu_hotplug+0xa/0xc
> <c05a3dae> __cpufreq_driver_target+0x15/0x6d  <f8bce3da> 
>do_dbs_timer+0x229/0x287 [cpufreq_ondemand]
> <c04321fb> run_workqueue+0x81/0xc0  <c0432c35> 
>worker_thread+0xce/0x101
> <c0435353> kthread+0xa3/0xd0  <c0402005> kernel_thread_helper+0x5/0xb
>pm-hibernate  D F395EB00  1816  3934   2495                     (NOTLB)
>       e632ade8 c0465c6c 00000004 f395eb00 003d10ba f2c31800 
>00000007 ddc7e44c
>       ddc7e330 c1b1b190 c19c4860 f395eb00 003d10ba 00000000 
>00000000 c0481c9c
>       f8bcfa40 f8bcfa40 e632a000 e632adf8 00000000 c0438ad8 
>f8bce646 f8bcfa54
>Call Trace:
> <c06075ba> __mutex_lock_slowpath+0x19b/0x3e3  <c0607826> 
>mutex_lock+0x24/0x27
> <f8bce646> cpufreq_governor_dbs+0x20e/0x2b0 
>[cpufreq_ondemand]  <c05a385f> __cpufreq_governor+0x57/0xd8
> <c05a4106> cpufreq_remove_dev+0x197/0x208  <c05a4716> 
>cpufreq_cpu_callback+0x49/0x52
> <c0609795> notifier_call_chain+0x18/0x29  <c042f9da> 
>blocking_notifier_call_chain+0x37/0x4f
> <c043a98d> cpu_down+0x12b/0x1c8  <c04420b0> 
>disable_nonboot_cpus+0x35/0xa1
> <c043fc93> prepare_processes+0x13/0x3b  <c043fedd> 
>pm_suspend_disk+0x9/0xf3
> <c043eead> enter_state+0x53/0x1c1  <c043f0a1> state_store+0x86/0x9c
> <c04a4d14> subsys_attr_store+0x20/0x25  <c04a4e18> 
>sysfs_write_file+0xab/0xd1
> <c046b25c> vfs_write+0xab/0x154  <c046b890> sys_write+0x3b/0x60
> <c060837f> syscall_call+0x7/0xb
>kstopmachine  S ED25F800  3428  4190     11  4191          2160 (L-TLB)
>       dc184fac c19c4860 c04f473a ed25f800 003d10ba c041115b 
>00000003 f7e38e4c
>       f7e38d30 c1acf050 c19cc860 ed25f800 003d10ba 00000000 
>00000001 00000246
>       e632ae94 dc184f90 00000246 e632ae90 00000000 c041ee49 
>00000000 00000000
>Call Trace:
> <c044579f> do_stop+0xf9/0x129  <c0435353> kthread+0xa3/0xd0
> <c0402005> kernel_thread_helper+0x5/0xb
>kstopmachine  X ED25F800  3736  4191   4190                     (L-TLB)
>       f7ecefb4 c0605fda f7ecef78 ed25f800 003d10ba 003d10ba 
>00000002 f2e622cc
>       f2e621b0 ddc7e330 c19c4860 ed25f800 003d10ba 00000000 
>00000000 00000000
>       f7ecef94 00000202 00000202 f2ec1ad8 00000000 c0608093 
>f7ecefb4 00000202
>Call Trace:
> <c0427428> do_exit+0x7a1/0x7ab  <c040200b> 
>kernel_thread_helper_end+0x0/0xe
>
>D    pm-hibernate: 3934 [ddc7e330, 118] blocked on mutex: 
>[f8bcfa40] {dbs_mutex}
>.. held by:          ondemand: 1462 [f7eb4eb0, 110]
>... acquired at:               do_dbs_timer+0x13/0x287 
>[cpufreq_ondemand]
>
>
>This doesn't happen if the govenor is 'userspace', so I'm wondering if
>we're doing something silly in ondemand when we've offlined a CPU.
>
>Any ideas ?
>
>		Dave
>
>-- 
>http://www.codemonkey.org.uk
>

Re: ondemand vs suspend.

[Dave Jones]

On Tue, Jun 13, 2006 at 04:03:12PM -0700, Pallipadi, Venkatesh wrote:
 > 
 > Is this with 2.6.16 based kernel?

This was based on 2.6.17rc6

 > Andi pushed in a patch to fix this recently 
 > Here -
 > http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=c
 > ommit;h=6810b548b25114607e0814612d84125abccc0a4f

so it should have this.

		Dave

-- 
http://www.codemonkey.org.uk

RE: ondemand vs suspend.

[Pallipadi, Venkatesh]

Yes. I am able to reproduce this 2.6.17-rc6 and trying to narrow it
down.

Also, there is a known issue with ondemand and resuming anyway as the
second CPU will not be able to resume with ondemand, but instead will go
back to 'default' governor on initialization. 
So, even with this bug fixed, we cannot suspend-resume with ondemand
governor. Workaround for now can be just to switch to default governor
before suspend and go back to ondemand after resume, using initscripts. 

Thanks,
Venki

>-----Original Message-----
>From: Dave Jones [mailto:davej@redhat.com] 
>Sent: Tuesday, June 13, 2006 1:53 PM
>To: cpufreq@lists.linux.org.uk
>Cc: Pallipadi, Venkatesh; pjones@redhat.com
>Subject: ondemand vs suspend.
>
>Here's a puzzling case that Peter stumbled across when experimenting
>with suspend on one of the new dual-core Intel-based Mac Mini's.
>
>The hibernate process would just lock up, and never shut down.
>
> 4190 ?        S<     0:00  \_ [kstopmachine]
> 4191 ?        Z<     0:00      \_ [kstopmachine] <defunct>
>  
>dmesg showed that we had offlined the 2nd core..
>
>PM: suspend-to-disk mode set to 'platform'
>Freezing cpus ...
>Breaking affinity for irq 0
>Breaking affinity for irq 14
>Breaking affinity for irq 217
>Breaking affinity for irq 225
>CPU 1 is now offline
>SMP alternatives: switching to UP code
>
>and sysrq-t showed that we were erm.. spinning.
>
>ondemand      D EDDD1300  3452  1462     11          2160  1345 (L-TLB)
>       f7f8eec0 f7f8eef4 f7f8eef4 eddd1300 003d10ba c05a4c2c 
>0000000a f7eb4fcc
>       f7eb4eb0 c06e0c00 c19c4860 eddd1300 003d10ba 00000000 
>00000000 00000000
>       f7f8eedc c041fc12 00000296 ffffffff f7f8eef4 00000246 
>00000246 c06e65d8
>Call Trace:
> <c0607f89> __down+0x131/0x14a  <c0605432> __down_failed+0xa/0x10
> <c043aa55> .text.lock.cpu+0x2b/0x32  <c043a7ad> 
>lock_cpu_hotplug+0xa/0xc
> <c05a3dae> __cpufreq_driver_target+0x15/0x6d  <f8bce3da> 
>do_dbs_timer+0x229/0x287 [cpufreq_ondemand]
> <c04321fb> run_workqueue+0x81/0xc0  <c0432c35> 
>worker_thread+0xce/0x101
> <c0435353> kthread+0xa3/0xd0  <c0402005> kernel_thread_helper+0x5/0xb
>pm-hibernate  D F395EB00  1816  3934   2495                     (NOTLB)
>       e632ade8 c0465c6c 00000004 f395eb00 003d10ba f2c31800 
>00000007 ddc7e44c
>       ddc7e330 c1b1b190 c19c4860 f395eb00 003d10ba 00000000 
>00000000 c0481c9c
>       f8bcfa40 f8bcfa40 e632a000 e632adf8 00000000 c0438ad8 
>f8bce646 f8bcfa54
>Call Trace:
> <c06075ba> __mutex_lock_slowpath+0x19b/0x3e3  <c0607826> 
>mutex_lock+0x24/0x27
> <f8bce646> cpufreq_governor_dbs+0x20e/0x2b0 
>[cpufreq_ondemand]  <c05a385f> __cpufreq_governor+0x57/0xd8
> <c05a4106> cpufreq_remove_dev+0x197/0x208  <c05a4716> 
>cpufreq_cpu_callback+0x49/0x52
> <c0609795> notifier_call_chain+0x18/0x29  <c042f9da> 
>blocking_notifier_call_chain+0x37/0x4f
> <c043a98d> cpu_down+0x12b/0x1c8  <c04420b0> 
>disable_nonboot_cpus+0x35/0xa1
> <c043fc93> prepare_processes+0x13/0x3b  <c043fedd> 
>pm_suspend_disk+0x9/0xf3
> <c043eead> enter_state+0x53/0x1c1  <c043f0a1> state_store+0x86/0x9c
> <c04a4d14> subsys_attr_store+0x20/0x25  <c04a4e18> 
>sysfs_write_file+0xab/0xd1
> <c046b25c> vfs_write+0xab/0x154  <c046b890> sys_write+0x3b/0x60
> <c060837f> syscall_call+0x7/0xb
>kstopmachine  S ED25F800  3428  4190     11  4191          2160 (L-TLB)
>       dc184fac c19c4860 c04f473a ed25f800 003d10ba c041115b 
>00000003 f7e38e4c
>       f7e38d30 c1acf050 c19cc860 ed25f800 003d10ba 00000000 
>00000001 00000246
>       e632ae94 dc184f90 00000246 e632ae90 00000000 c041ee49 
>00000000 00000000
>Call Trace:
> <c044579f> do_stop+0xf9/0x129  <c0435353> kthread+0xa3/0xd0
> <c0402005> kernel_thread_helper+0x5/0xb
>kstopmachine  X ED25F800  3736  4191   4190                     (L-TLB)
>       f7ecefb4 c0605fda f7ecef78 ed25f800 003d10ba 003d10ba 
>00000002 f2e622cc
>       f2e621b0 ddc7e330 c19c4860 ed25f800 003d10ba 00000000 
>00000000 00000000
>       f7ecef94 00000202 00000202 f2ec1ad8 00000000 c0608093 
>f7ecefb4 00000202
>Call Trace:
> <c0427428> do_exit+0x7a1/0x7ab  <c040200b> 
>kernel_thread_helper_end+0x0/0xe
>
>D    pm-hibernate: 3934 [ddc7e330, 118] blocked on mutex: 
>[f8bcfa40] {dbs_mutex}
>.. held by:          ondemand: 1462 [f7eb4eb0, 110]
>... acquired at:               do_dbs_timer+0x13/0x287 
>[cpufreq_ondemand]
>
>
>This doesn't happen if the govenor is 'userspace', so I'm wondering if
>we're doing something silly in ondemand when we've offlined a CPU.
>
>Any ideas ?
>
>		Dave
>
>-- 
>http://www.codemonkey.org.uk
>

RE: ondemand vs suspend.

[Peter Jones]

On Tue, 2006-06-13 at 19:50 -0700, Pallipadi, Venkatesh wrote:
> Yes. I am able to reproduce this 2.6.17-rc6 and trying to narrow it
> down.
> 
> Also, there is a known issue with ondemand and resuming anyway as the
> second CPU will not be able to resume with ondemand, but instead will go
> back to 'default' governor on initialization. 
> So, even with this bug fixed, we cannot suspend-resume with ondemand
> governor. Workaround for now can be just to switch to default governor
> before suspend and go back to ondemand after resume, using initscripts. 

Yeah, that's what I implemented for our pm utils earlier today.

-- 
  Peter

Re: ondemand vs suspend.

[Jeremy Fitzhardinge]

Pallipadi, Venkatesh wrote:
> Yes. I am able to reproduce this 2.6.17-rc6 and trying to narrow it
> down.
>
> Also, there is a known issue with ondemand and resuming anyway as the
> second CPU will not be able to resume with ondemand, but instead will go
> back to 'default' governor on initialization. 
> So, even with this bug fixed, we cannot suspend-resume with ondemand
> governor. Workaround for now can be just to switch to default governor
> before suspend and go back to ondemand after resume, using initscripts. 
>   
I've noticed another problem, which may be related.  Sometimes after 
resume, cpufreq seems to lose the ability to control the CPU speeds: 
they stay at the lowest performance level, and don't change regardless 
of what governor I set.  I'm seeing this on a T2400 Core Duo.  I'm not 
sure if this is a speedfreq-centrino problem or cpufreq itself.

    J

Re: ondemand vs suspend.

[Venkatesh Pallipadi]

On Tue, Jun 13, 2006 at 07:50:56PM -0700, Pallipadi, Venkatesh wrote:
> 
> Yes. I am able to reproduce this 2.6.17-rc6 and trying to narrow it
> down.
> 

Root caused this to a deadlock in cpufreq and ondemand. The deadlock is
due to non-existant ordering between cpu_hotplug lock and dbs_mutex.
Basically a race condition between cpu_down() and do_dbs_timer().

cpu_down() flow:
1 cpu_down() for CPU 1
2 Takes the cpu_hotplug lock
3 Calls pre-down notifiers
4   notifier handler in cpufreq calls cpufreq_driver_target
5     cpufreq_driver_target calls cpu_hotplu lock/unlock
      It is OK as cpu_hotplug lock is recusive for same process
6 CPU 1 goes down
7 CPU 0 calls post down notifiers for CPU 1
8   notifier handler in cpufreq calls governor event for stop
9     this ondemand governor routine takes dbs_mutex

Basically cpu_hotplug lock being taken before dbs_mutex in this path.

There is another event that gets triggered periodically in do_dbs_timer().
This runs in the context of ondemand workqueue and it takes dbs_mutex first
and takes cpu_hotplug later, inside __cpufreq_driver_target() call. This
ordering conflicts with mutex ordering in cpu_down and causes a deadlock.

Attached patch fixes the issue for both ondemand and conservative governors.

Signed-off-by: Venkatesh Pallipadi <venkatesh.pallipadi@intel.com>

--- linux-2.6.17/drivers/cpufreq/cpufreq_ondemand.c	2006-06-18 00:10:55.000000000 -0700
+++ linux-2.6.17/drivers/cpufreq/cpufreq_ondemand.c.new	2006-06-18 10:41:45.000000000 -0700
@@ -71,6 +71,14 @@ static DEFINE_PER_CPU(struct cpu_dbs_inf
 
 static unsigned int dbs_enable;	/* number of CPUs using this policy */
 
+/*
+ * DEADLOCK ALERT! There is a ordering requirement between cpu_hotplug
+ * lock and dbs_mutex. cpu_hotplug lock should always be held before
+ * dbs_mutex. If any function that can potentially take cpu_hotplug lock
+ * (like __cpufreq_driver_target()) is being called with dbs_mutex taken, then
+ * cpu_hotplug lock should be taken before that. Note that cpu_hotplug lock
+ * is recursive for the same process. -Venki
+ */
 static DEFINE_MUTEX (dbs_mutex);
 static DECLARE_WORK	(dbs_work, do_dbs_timer, NULL);
 
@@ -363,6 +371,7 @@ static void dbs_check_cpu(int cpu)
 static void do_dbs_timer(void *data)
 {
 	int i;
+	lock_cpu_hotplug();
 	mutex_lock(&dbs_mutex);
 	printk("CPU %d, dbs mutex taken\n", smp_processor_id());
 	for_each_online_cpu(i)
@@ -370,6 +379,7 @@ static void do_dbs_timer(void *data)
 	queue_delayed_work(dbs_workq, &dbs_work,
 			   usecs_to_jiffies(dbs_tuners_ins.sampling_rate));
 	mutex_unlock(&dbs_mutex);
+	unlock_cpu_hotplug();
 }
 
 static inline void dbs_timer_init(void)
@@ -470,6 +480,7 @@ static int cpufreq_governor_dbs(struct c
 		break;
 
 	case CPUFREQ_GOV_LIMITS:
+		lock_cpu_hotplug();
 		mutex_lock(&dbs_mutex);
 		if (policy->max < this_dbs_info->cur_policy->cur)
 			__cpufreq_driver_target(
@@ -480,6 +491,7 @@ static int cpufreq_governor_dbs(struct c
 					this_dbs_info->cur_policy,
 					policy->min, CPUFREQ_RELATION_L);
 		mutex_unlock(&dbs_mutex);
+		unlock_cpu_hotplug();
 		break;
 	}
 	return 0;
--- linux-2.6.17/drivers/cpufreq/cpufreq_conservative.c	2006-06-17 13:41:32.000000000 -0700
+++ linux-2.6.17/drivers/cpufreq/cpufreq_conservative.c.new	2006-06-18 10:42:30.000000000 -0700
@@ -72,6 +72,14 @@ static DEFINE_PER_CPU(struct cpu_dbs_inf
 
 static unsigned int dbs_enable;	/* number of CPUs using this policy */
 
+/*
+ * DEADLOCK ALERT! There is a ordering requirement between cpu_hotplug
+ * lock and dbs_mutex. cpu_hotplug lock should always be held before
+ * dbs_mutex. If any function that can potentially take cpu_hotplug lock
+ * (like __cpufreq_driver_target()) is being called with dbs_mutex taken, then
+ * cpu_hotplug lock should be taken before that. Note that cpu_hotplug lock
+ * is recursive for the same process. -Venki
+ */
 static DEFINE_MUTEX 	(dbs_mutex);
 static DECLARE_WORK	(dbs_work, do_dbs_timer, NULL);
 
@@ -414,12 +422,14 @@ static void dbs_check_cpu(int cpu)
 static void do_dbs_timer(void *data)
 { 
 	int i;
+	lock_cpu_hotplug();
 	mutex_lock(&dbs_mutex);
 	for_each_online_cpu(i)
 		dbs_check_cpu(i);
 	schedule_delayed_work(&dbs_work, 
 			usecs_to_jiffies(dbs_tuners_ins.sampling_rate));
 	mutex_unlock(&dbs_mutex);
+	unlock_cpu_hotplug();
 } 
 
 static inline void dbs_timer_init(void)
@@ -514,6 +524,7 @@ static int cpufreq_governor_dbs(struct c
 		break;
 
 	case CPUFREQ_GOV_LIMITS:
+		lock_cpu_hotplug();
 		mutex_lock(&dbs_mutex);
 		if (policy->max < this_dbs_info->cur_policy->cur)
 			__cpufreq_driver_target(
@@ -524,6 +535,7 @@ static int cpufreq_governor_dbs(struct c
 					this_dbs_info->cur_policy,
 				       	policy->min, CPUFREQ_RELATION_L);
 		mutex_unlock(&dbs_mutex);
+		unlock_cpu_hotplug();
 		break;
 	}
 	return 0;

Re: ondemand vs suspend.

[Venkatesh Pallipadi]

On Mon, Jun 19, 2006 at 03:52:59PM -0700, Jeremy Fitzhardinge wrote:
> Pallipadi, Venkatesh wrote:
> > Yes. I am able to reproduce this 2.6.17-rc6 and trying to narrow it
> > down.
> >
> > Also, there is a known issue with ondemand and resuming anyway as the
> > second CPU will not be able to resume with ondemand, but instead will go
> > back to 'default' governor on initialization. 
> > So, even with this bug fixed, we cannot suspend-resume with ondemand
> > governor. Workaround for now can be just to switch to default governor
> > before suspend and go back to ondemand after resume, using initscripts. 
> >   
> I've noticed another problem, which may be related.  Sometimes after 
> resume, cpufreq seems to lose the ability to control the CPU speeds: 
> they stay at the lowest performance level, and don't change regardless 
> of what governor I set.  I'm seeing this on a T2400 Core Duo.  I'm not 
> sure if this is a speedfreq-centrino problem or cpufreq itself.
> 
>     J

Does it only happen on CPU 1 or both the CPUs? What does /sys/.../cpufreq
look like once you get to this stuck at lowest freq state? Does 
scaling_available_frequencies still show all the freqs.

Thanks,
Venki

Re: ondemand vs suspend.

[Jeremy Fitzhardinge]

[-- Attachment #1: Type: text/plain, Size: 769 bytes --]

Venkatesh Pallipadi wrote:
> Does it only happen on CPU 1 or both the CPUs? What does /sys/.../cpufreq
> look like once you get to this stuck at lowest freq state? Does 
> scaling_available_frequencies still show all the freqs.
>   

Perhaps it isn't related to suspend.

My startup scripts set the governor for both CPUs to conservative.  Then 
later, not long after boot, with no suspend, I set them to 
"performance", but they don't switch speed.  If I switch to "userspace", 
writing to scaling_setspeed has no effect.

Confused.  Could this be related to the dual-coreness of the CPU?  Maybe 
things start misbehaving on policy switches?

This is 2.6.17-mm1 with the patch you just posted applied.

I've attached all the cpufreq sysfs files for each core.

    J


[-- Attachment #2: cpu0.txt --]
[-- Type: text/plain, Size: 793 bytes --]

cpufreq/scaling_setspeed:
1000000
cpufreq/stats/trans_table:
   From  :    To
         :   1833000   1333000   1000000 
  1833000:         0         0         1 
  1333000:         0         0         0 
  1000000:         0         0         0 
cpufreq/stats/time_in_state:
1833000 0
1333000 0
1000000 188851
cpufreq/stats/total_trans:
1
cpufreq/scaling_cur_freq:
1000000
cpufreq/cpuinfo_cur_freq:
1000000
cpufreq/scaling_available_frequencies:
1833000 1333000 1000000 
cpufreq/scaling_available_governors:
conservative ondemand powersave userspace performance 
cpufreq/scaling_driver:
centrino
cpufreq/scaling_governor:
userspace
cpufreq/affected_cpus:
0
cpufreq/scaling_max_freq:
1000000
cpufreq/scaling_min_freq:
1000000
cpufreq/cpuinfo_max_freq:
1833000
cpufreq/cpuinfo_min_freq:
1000000

[-- Attachment #3: cpu1.txt --]
[-- Type: text/plain, Size: 793 bytes --]

cpufreq/scaling_setspeed:
1000000
cpufreq/stats/trans_table:
   From  :    To
         :   1833000   1333000   1000000 
  1833000:         0         0         0 
  1333000:         0         0         0 
  1000000:         0         0         0 
cpufreq/stats/time_in_state:
1833000 0
1333000 0
1000000 187899
cpufreq/stats/total_trans:
0
cpufreq/scaling_cur_freq:
1000000
cpufreq/cpuinfo_cur_freq:
1000000
cpufreq/scaling_available_frequencies:
1833000 1333000 1000000 
cpufreq/scaling_available_governors:
conservative ondemand powersave userspace performance 
cpufreq/scaling_driver:
centrino
cpufreq/scaling_governor:
userspace
cpufreq/affected_cpus:
1
cpufreq/scaling_max_freq:
1000000
cpufreq/scaling_min_freq:
1000000
cpufreq/cpuinfo_max_freq:
1833000
cpufreq/cpuinfo_min_freq:
1000000

[-- Attachment #4: Type: text/plain, Size: 147 bytes --]

_______________________________________________
Cpufreq mailing list
Cpufreq@lists.linux.org.uk
http://lists.linux.org.uk/mailman/listinfo/cpufreq

Re: ondemand vs suspend.

[Jeremy Fitzhardinge]

Venkatesh Pallipadi wrote:
> Does it only happen on CPU 1 or both the CPUs? What does /sys/.../cpufreq
> look like once you get to this stuck at lowest freq state? Does 
> scaling_available_frequencies still show all the freqs.
>   
Did you have a chance to look at this, or should I do more investigation?

    J

RE: ondemand vs suspend.

[Pallipadi, Venkatesh]

>-----Original Message-----
>From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] 
>Sent: Friday, June 23, 2006 3:45 PM
>To: Pallipadi, Venkatesh
>Cc: Dave Jones; cpufreq@lists.linux.org.uk; pjones@redhat.com
>Subject: Re: ondemand vs suspend.
>
>Venkatesh Pallipadi wrote:
>> Does it only happen on CPU 1 or both the CPUs? What does 
>/sys/.../cpufreq
>> look like once you get to this stuck at lowest freq state? Does 
>> scaling_available_frequencies still show all the freqs.
>>   
>Did you have a chance to look at this, or should I do more 
>investigation?
>


I am not able to reprocude this on my Core Duo system.  At this point I
will need some more information on what exactly happens at this point of
failure to start looking at the code.. Like questions above. Does
scaling_available_frequencies still show all freqs. What is *max *min
values in /sys/.../cpufreq/ ..

Thanks,
Venki

Re: ondemand vs suspend.

[Jeremy Fitzhardinge]

Pallipadi, Venkatesh wrote:
> I am not able to reprocude this on my Core Duo system.  At this point I
> will need some more information on what exactly happens at this point of
> failure to start looking at the code.. Like questions above. Does
> scaling_available_frequencies still show all freqs. What is *max *min
> values in /sys/.../cpufreq/ ..
>   
Yes, all the expected frequencies are there.  I sent you mail with all 
the contents of /sys/.../cpu?/cpufreq/... the other day; should I resend it?

Thanks,
    J

RE: ondemand vs suspend.

[Pallipadi, Venkatesh]

 

>-----Original Message-----
>From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] 
>Sent: Monday, June 26, 2006 12:23 PM
>To: Pallipadi, Venkatesh
>Cc: Dave Jones; cpufreq@lists.linux.org.uk; pjones@redhat.com
>Subject: Re: ondemand vs suspend.
>
>Pallipadi, Venkatesh wrote:
>> I am not able to reprocude this on my Core Duo system.  At 
>this point I
>> will need some more information on what exactly happens at 
>this point of
>> failure to start looking at the code.. Like questions above. Does
>> scaling_available_frequencies still show all freqs. What is *max *min
>> values in /sys/.../cpufreq/ ..
>>   
>Yes, all the expected frequencies are there.  I sent you mail with all 
>the contents of /sys/.../cpu?/cpufreq/... the other day; 
>should I resend it?
>

Sorry. I had missed your earlier mail.

Looking at the info you had sent..
> cpufreq/scaling_max_freq:
> 1000000
> cpufreq/scaling_min_freq:
> 1000000
> cpufreq/cpuinfo_max_freq:
> 1833000
> cpufreq/cpuinfo_min_freq:
> 1000000

Looks like scaling_max_freq on both CPUs got set to 1000000 somehow. Due
to that all the frequency changes are failing from then on. If you
manually write 1833000 to scaling_max_freq, things should come back to
normal.

Now the question is: Why is scaling_max_freq getting set to 1000000.
There was one patch here from thomas, that fixed a race between _PPC
call to reduce the freq and some other action in setting cpufreq
governor was resulting in something like this. But, that patch is
already in mm1. 
http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=c
ommitdiff;h=7970e08bf066900efcd7794a1a338c11eb8f5141;hp=f1f76afd71e0f17a
f9a35fcb649f4bab53304a4d

I guess, there is some other corner case where scaling_max_freq can get
set.

> My startup scripts set the governor for both CPUs to conservative.
Then 
> later, not long after boot, with no suspend, I set them to 
> "performance", but they don't switch speed. 

Somewhere inbetween this max got changed. Can you check it soon after
changing the governor to conservative?

Thanks,
Venki

Re: ondemand vs suspend.

[Jeremy Fitzhardinge]

Pallipadi, Venkatesh wrote:
> Looks like scaling_max_freq on both CPUs got set to 1000000 somehow. Due
> to that all the frequency changes are failing from then on. If you
> manually write 1833000 to scaling_max_freq, things should come back to
> normal.
>   
Yup.  I'd overlooked that one.

Something very strange is going on.  If I do:

   1. set both CPUs to conservative
   2. set CPU1 to performance
   3. looks OK for a while, then...
   4. CPU1's max speed drops to 1GHz
   5. (then goes back; both CPUs switch to full speed; all kinds of
      strangeness)

I have the gnome cpufreq applet running, but its just reading 
scaling_cur_freq for each CPU.
Hm, and now both CPUs are apparently running at full speed.

If the two cores are in fact locked together, then this config doesn't 
make much sense.  But then, the resulting behaviour doesn't either.

> Now the question is: Why is scaling_max_freq getting set to 1000000.
> There was one patch here from thomas, that fixed a race between _PPC
> call to reduce the freq and some other action in setting cpufreq
> governor was resulting in something like this. But, that patch is
> already in mm1. 
>   
Which mm1?  I'm still running 2.6.17-rc6-mm2, because I'm depending on 
some AHCI suspend/resume patches which haven't been updated yet.  Hm, 
but that change does seem pretty old.

> I guess, there is some other corner case where scaling_max_freq can get
> set.
>   
Yes.  This doesn't appear to be a race with anything, since the 
spontaneous change seems to happen unrelated to anything else, unless 
reading the sysfs entries can cause a problem.

    J

RE: ondemand vs suspend.

[Pallipadi, Venkatesh]

 

>-----Original Message-----
>From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] 
>Sent: Monday, June 26, 2006 1:59 PM
>To: Pallipadi, Venkatesh
>Cc: Dave Jones; cpufreq@lists.linux.org.uk; pjones@redhat.com
>Subject: Re: ondemand vs suspend.
>
>Pallipadi, Venkatesh wrote:
>> Looks like scaling_max_freq on both CPUs got set to 1000000 
>somehow. Due
>> to that all the frequency changes are failing from then on. If you
>> manually write 1833000 to scaling_max_freq, things should 
>come back to
>> normal.
>>   
>Yup.  I'd overlooked that one.
>
>Something very strange is going on.  If I do:
>
>   1. set both CPUs to conservative
>   2. set CPU1 to performance
>   3. looks OK for a while, then...
>   4. CPU1's max speed drops to 1GHz
>   5. (then goes back; both CPUs switch to full speed; all kinds of
>      strangeness)
>
>I have the gnome cpufreq applet running, but its just reading 
>scaling_cur_freq for each CPU.
>Hm, and now both CPUs are apparently running at full speed.


Strange. I guess only scaling_max_freq is changing and not
cpuinfo_max_freq. Right?
Can you try stopping the applet and see whether anything changes?
Is the system idle while these changes happen or the load is varying
causing some frequency changes by conservative on CPU 0?

I am not able to reproduce this with periodic reading of
scaling_cur_freq. May be soemthing else is happening from userlevel that
is causing this..

>> governor was resulting in something like this. But, that patch is
>> already in mm1. 
>>   
>Which mm1?  I'm still running 2.6.17-rc6-mm2, because I'm depending on 
>some AHCI suspend/resume patches which haven't been updated yet.  Hm, 
>but that change does seem pretty old.

I was referring to linux-2.6.17-mm1. But, that patch should be there in
2.6.17-rc6 as well.

>> I guess, there is some other corner case where 
>scaling_max_freq can get
>> set.
>>   
>Yes.  This doesn't appear to be a race with anything, since the 
>spontaneous change seems to happen unrelated to anything else, unless 
>reading the sysfs entries can cause a problem.
>

Can you enable cpufreq.debug and capture the log. 

Thanks,
Venki

Re: ondemand vs suspend.

[Jeremy Fitzhardinge]

Pallipadi, Venkatesh wrote:
> Strange. I guess only scaling_max_freq is changing and not
> cpuinfo_max_freq. Right?
> Can you try stopping the applet and see whether anything changes?
> Is the system idle while these changes happen or the load is varying
> causing some frequency changes by conservative on CPU 0?
>
> I am not able to reproduce this with periodic reading of
> scaling_cur_freq. May be soemthing else is happening from userlevel that
> is causing this..
>   
Sorry, I haven't got around to looking at this closely again.  However, 
the behaviour with 2.6.17-mm3 has changed a bit.  Now on resume, the 
cpufreq applets exit for some reason, and both CPUs end up with a max 
speed of 1GHz.

> Can you enable cpufreq.debug and capture the log. 
>   

Will do.

    J