* Re: [BUG] hotplug cpus on ia64 [not found] ` <1212154614.12349.244.camel@twins> @ 2008-06-03 22:17 ` Cliff Wickman 2008-06-04 13:50 ` Dimitri Sivanich 2008-06-05 12:49 ` Peter Zijlstra 0 siblings, 2 replies; 7+ messages in thread From: Cliff Wickman @ 2008-06-03 22:17 UTC (permalink / raw) To: Peter Zijlstra; +Cc: sivanich, linux-kernel On Fri, May 30, 2008 at 03:36:54PM +0200, Peter Zijlstra wrote: > On Thu, 2008-05-29 at 11:32 -0500, Cliff Wickman wrote: > > >> I built an ia64 kernel from Andrew's tree (2.6.26-rc2-mm1) > > >> and get a very predictable hotplug cpu problem. > > >> billberry1:/tmp/cpw # ./dis > > >> disabled cpu 17 > > >> enabled cpu 17 > > >> billberry1:/tmp/cpw # ./dis > > >> disabled cpu 17 > > >> enabled cpu 17 > > >> billberry1:/tmp/cpw # ./dis > > >> > > >> The script that disables the cpu always hangs (unkillable) > > >> on the 3rd attempt. > > > > > And a bit further: > > > The kstopmachine thread always sits on the run queue (real time) for about > > > 30 minutes before running. > > > > And a bit further: > > > > The kstopmachine thread is queued as real-time on the downed cpu: > > >> rq -f 17 > > CPU# runq address size Lock current task time name > > ========================================================================== > > 17 0xe000046003059540 3 U 0xe0000360f06f8000 0 swapper > > Total of 3 queued: > > 3 real time tasks: px *(rt_rq *)0xe000046003059608 > > exclusive queue: > > slot 0 > > 0xe0000760f4628000 0 migration/17 > > 0xe0000760f4708000 0 kstopmachine > > 0xe0000760f6678000 0 watchdog/17 > > > > I put in counters and see that schedule() is never again entered by cpu 17 > > after it is downed the 3rd time. > > (it is entered after being up'd the first two times) > > > > The kstopmachine thread is bound to cpu 17 by __stop_machine_run()'s call > > to kthread_bind(). > > > > A cpu does not schedule after being downed, of course. But it does again > > after being up'd. > > Why would the second up be different? Following it, if the cpu is > > downed it never schedules again. > > > > If I always bind kstopmachine to cpu 0 the problem disappears. > > does: > > echo -1 > /proc/sys/kernel/sched_rt_runtime_us > > fix the problem? Yes! It does. Dimitri Sivanich has run into what looks like a similar problem. Hope the above workaround is a good clue to its solution. -- Cliff Wickman Silicon Graphics, Inc. cpw@sgi.com (651) 683-3824 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-03 22:17 ` [BUG] hotplug cpus on ia64 Cliff Wickman @ 2008-06-04 13:50 ` Dimitri Sivanich 2008-06-05 12:49 ` Peter Zijlstra 1 sibling, 0 replies; 7+ messages in thread From: Dimitri Sivanich @ 2008-06-04 13:50 UTC (permalink / raw) To: Cliff Wickman; +Cc: Peter Zijlstra, linux-kernel On Tue, Jun 03, 2008 at 05:17:59PM -0500, Cliff Wickman wrote: > > On Fri, May 30, 2008 at 03:36:54PM +0200, Peter Zijlstra wrote: > > On Thu, 2008-05-29 at 11:32 -0500, Cliff Wickman wrote: > > > I put in counters and see that schedule() is never again entered by cpu 17 > > > after it is downed the 3rd time. > > > (it is entered after being up'd the first two times) > > > > > > The kstopmachine thread is bound to cpu 17 by __stop_machine_run()'s call > > > to kthread_bind(). > > > > > > A cpu does not schedule after being downed, of course. But it does again > > > after being up'd. > > > Why would the second up be different? Following it, if the cpu is > > > downed it never schedules again. > > > > > > If I always bind kstopmachine to cpu 0 the problem disappears. > > > > does: > > > > echo -1 > /proc/sys/kernel/sched_rt_runtime_us > > > > fix the problem? > > Yes! It does. > > Dimitri Sivanich has run into what looks like a similar problem. > Hope the above workaround is a good clue to its solution. This fixes the problem I was seeing as well. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-03 22:17 ` [BUG] hotplug cpus on ia64 Cliff Wickman 2008-06-04 13:50 ` Dimitri Sivanich @ 2008-06-05 12:49 ` Peter Zijlstra 2008-06-05 13:51 ` Dimitri Sivanich 2008-06-10 10:19 ` Ingo Molnar 1 sibling, 2 replies; 7+ messages in thread From: Peter Zijlstra @ 2008-06-05 12:49 UTC (permalink / raw) To: Cliff Wickman; +Cc: sivanich, linux-kernel On Tue, 2008-06-03 at 17:17 -0500, Cliff Wickman wrote: > On Fri, May 30, 2008 at 03:36:54PM +0200, Peter Zijlstra wrote: > > On Thu, 2008-05-29 at 11:32 -0500, Cliff Wickman wrote: > > > >> I built an ia64 kernel from Andrew's tree (2.6.26-rc2-mm1) > > > >> and get a very predictable hotplug cpu problem. > > > >> billberry1:/tmp/cpw # ./dis > > > >> disabled cpu 17 > > > >> enabled cpu 17 > > > >> billberry1:/tmp/cpw # ./dis > > > >> disabled cpu 17 > > > >> enabled cpu 17 > > > >> billberry1:/tmp/cpw # ./dis > > > >> > > > >> The script that disables the cpu always hangs (unkillable) > > > >> on the 3rd attempt. > > > > > > > And a bit further: > > > > The kstopmachine thread always sits on the run queue (real time) for about > > > > 30 minutes before running. > > > > > > And a bit further: > > > > > > The kstopmachine thread is queued as real-time on the downed cpu: > > > >> rq -f 17 > > > CPU# runq address size Lock current task time name > > > ========================================================================== > > > 17 0xe000046003059540 3 U 0xe0000360f06f8000 0 swapper > > > Total of 3 queued: > > > 3 real time tasks: px *(rt_rq *)0xe000046003059608 > > > exclusive queue: > > > slot 0 > > > 0xe0000760f4628000 0 migration/17 > > > 0xe0000760f4708000 0 kstopmachine > > > 0xe0000760f6678000 0 watchdog/17 > > > > > > I put in counters and see that schedule() is never again entered by cpu 17 > > > after it is downed the 3rd time. > > > (it is entered after being up'd the first two times) > > > > > > The kstopmachine thread is bound to cpu 17 by __stop_machine_run()'s call > > > to kthread_bind(). > > > > > > A cpu does not schedule after being downed, of course. But it does again > > > after being up'd. > > > Why would the second up be different? Following it, if the cpu is > > > downed it never schedules again. > > > > > > If I always bind kstopmachine to cpu 0 the problem disappears. > > > > does: > > > > echo -1 > /proc/sys/kernel/sched_rt_runtime_us > > > > fix the problem? > > Yes! It does. > > Dimitri Sivanich has run into what looks like a similar problem. > Hope the above workaround is a good clue to its solution. Does the below fix it? Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> --- kernel/sched.c | 15 +++++-- kernel/sched_rt.c | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 115 insertions(+), 9 deletions(-) Index: linux-2.6/kernel/sched_rt.c =================================================================== --- linux-2.6.orig/kernel/sched_rt.c +++ linux-2.6/kernel/sched_rt.c @@ -280,6 +280,9 @@ static int balance_runtime(struct rt_rq continue; spin_lock(&iter->rt_runtime_lock); + if (iter->rt_runtime == RUNTIME_INF) + goto next; + diff = iter->rt_runtime - iter->rt_time; if (diff > 0) { do_div(diff, weight); @@ -293,12 +296,105 @@ static int balance_runtime(struct rt_rq break; } } +next: spin_unlock(&iter->rt_runtime_lock); } spin_unlock(&rt_b->rt_runtime_lock); return more; } + +static void __disable_runtime(struct rq *rq) +{ + struct root_domain *rd = rq->rd; + struct rt_rq *rt_rq; + + if (unlikely(!scheduler_running)) + return; + + for_each_leaf_rt_rq(rt_rq, rq) { + struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq); + s64 want; + int i; + + spin_lock(&rt_b->rt_runtime_lock); + spin_lock(&rt_rq->rt_runtime_lock); + if (rt_rq->rt_runtime == RUNTIME_INF || + rt_rq->rt_runtime == rt_b->rt_runtime) + goto balanced; + spin_unlock(&rt_rq->rt_runtime_lock); + + want = rt_b->rt_runtime - rt_rq->rt_runtime; + + for_each_cpu_mask(i, rd->span) { + struct rt_rq *iter = sched_rt_period_rt_rq(rt_b, i); + s64 diff; + + if (iter == rt_rq) + continue; + + spin_lock(&iter->rt_runtime_lock); + if (want > 0) { + diff = min_t(s64, iter->rt_runtime, want); + iter->rt_runtime -= diff; + want -= diff; + } else { + iter->rt_runtime -= want; + want -= want; + } + spin_unlock(&iter->rt_runtime_lock); + + if (!want) + break; + } + + spin_lock(&rt_rq->rt_runtime_lock); + BUG_ON(want); +balanced: + rt_rq->rt_runtime = RUNTIME_INF; + spin_unlock(&rt_rq->rt_runtime_lock); + spin_unlock(&rt_b->rt_runtime_lock); + } +} + +static void disable_runtime(struct rq *rq) +{ + unsigned long flags; + + spin_lock_irqsave(&rq->lock, flags); + __disable_runtime(rq); + spin_unlock_irqrestore(&rq->lock, flags); +} + +static void __enable_runtime(struct rq *rq) +{ + struct root_domain *rd = rq->rd; + struct rt_rq *rt_rq; + + if (unlikely(!scheduler_running)) + return; + + for_each_leaf_rt_rq(rt_rq, rq) { + struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq); + + spin_lock(&rt_b->rt_runtime_lock); + spin_lock(&rt_rq->rt_runtime_lock); + rt_rq->rt_runtime = rt_b->rt_runtime; + rt_rq->rt_time = 0; + spin_unlock(&rt_rq->rt_runtime_lock); + spin_unlock(&rt_b->rt_runtime_lock); + } +} + +static void enable_runtime(struct rq *rq) +{ + unsigned long flags; + + spin_lock_irqsave(&rq->lock, flags); + __enable_runtime(rq); + spin_unlock_irqrestore(&rq->lock, flags); +} + #endif static inline int rt_se_prio(struct sched_rt_entity *rt_se) @@ -328,14 +424,13 @@ static int sched_rt_runtime_exceeded(str #ifdef CONFIG_SMP if (rt_rq->rt_time > runtime) { - int more; - spin_unlock(&rt_rq->rt_runtime_lock); - more = balance_runtime(rt_rq); + balance_runtime(rt_rq); spin_lock(&rt_rq->rt_runtime_lock); - if (more) - runtime = sched_rt_runtime(rt_rq); + runtime = sched_rt_runtime(rt_rq); + if (runtime == RUNTIME_INF) + return 0; } #endif @@ -1157,6 +1252,8 @@ static void join_domain_rt(struct rq *rq { if (rq->rt.overloaded) rt_set_overload(rq); + + __enable_runtime(rq); } /* Assumes rq->lock is held */ @@ -1164,6 +1261,8 @@ static void leave_domain_rt(struct rq *r { if (rq->rt.overloaded) rt_clear_overload(rq); + + __disable_runtime(rq); } /* Index: linux-2.6/kernel/sched.c =================================================================== --- linux-2.6.orig/kernel/sched.c +++ linux-2.6/kernel/sched.c @@ -7455,20 +7455,27 @@ int sched_create_sysfs_power_savings_ent static int update_sched_domains(struct notifier_block *nfb, unsigned long action, void *hcpu) { + int cpu = (int)(long)hcpu; + switch (action) { - case CPU_UP_PREPARE: - case CPU_UP_PREPARE_FROZEN: case CPU_DOWN_PREPARE: case CPU_DOWN_PREPARE_FROZEN: + disable_runtime(cpu_rq(cpu)); + /* fall-through */ + case CPU_UP_PREPARE: + case CPU_UP_PREPARE_FROZEN: detach_destroy_domains(&cpu_online_map); return NOTIFY_OK; - case CPU_UP_CANCELED: - case CPU_UP_CANCELED_FROZEN: + case CPU_DOWN_FAILED: case CPU_DOWN_FAILED_FROZEN: case CPU_ONLINE: case CPU_ONLINE_FROZEN: + enable_runtime(cpu_rq(cpu)); + /* fall-through */ + case CPU_UP_CANCELED: + case CPU_UP_CANCELED_FROZEN: case CPU_DEAD: case CPU_DEAD_FROZEN: /* ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-05 12:49 ` Peter Zijlstra @ 2008-06-05 13:51 ` Dimitri Sivanich 2008-06-05 14:18 ` Peter Zijlstra 2008-06-10 10:19 ` Ingo Molnar 1 sibling, 1 reply; 7+ messages in thread From: Dimitri Sivanich @ 2008-06-05 13:51 UTC (permalink / raw) To: Peter Zijlstra; +Cc: Cliff Wickman, linux-kernel On Thu, Jun 05, 2008 at 02:49:58PM +0200, Peter Zijlstra wrote: > > Does the below fix it? > I don't believe so: :~ # taskset -p $$ pid 4502's current affinity mask: 8 :~ # cd /sys/devices/system :/sys/devices/system # cd cpu :/sys/devices/system/cpu # cd cpu2 :/sys/devices/system/cpu/cpu2 # cat online 1 :/sys/devices/system/cpu/cpu2 # echo 0 >online :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: 8 :/sys/devices/system/cpu/cpu2 # taskset -cp 0-3 $$ pid 4502's current affinity list: 3 pid 4502's new affinity list: 0,1,3 :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: b :/sys/devices/system/cpu/cpu2 # echo 1 >online :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: b :/sys/devices/system/cpu/cpu2 # taskset -cp 0-3 $$ pid 4502's current affinity list: 0,1,3 pid 4502's new affinity list: 0-3 :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: f :/sys/devices/system/cpu/cpu2 # echo 0 >online :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: b :/sys/devices/system/cpu/cpu2 # echo 1 >online :/sys/devices/system/cpu/cpu2 # taskset -p $$ (above command now hangs) (ps output) 0xe0000060b5650000 4502 4349 0 2 S 0xe0000060b5650390 bash 0xe0000060b8da0000 4843 4502 0 2 D 0xe0000060b8da0390 bash Stack traceback for pid 4843 0xe0000060b8da0000 4843 4502 0 2 D 0xe0000060b8da0390 bash 0xa0000001007d44b0 schedule+0x1210 args (0xe0000060ba470ce4, 0xa000000100dae190, 0xe000006003129200, 0xa000000100084b70, 0x48c, 0xe0000060b8dafda8, 0xe000006003129200, 0x200, 0xe0000060f780fe80) 0xa0000001007d4ac0 schedule_timeout+0x40 args (0x7fffffffffffffff, 0x0, 0x0, 0xa0000001007d2f00, 0x309, 0xe000006003129200) 0xa0000001007d2f00 wait_for_common+0x240 args (0xe0000060b8dafe08, 0x7fffffffffffffff, 0x2, 0xa0000001007d3280, 0x207, 0xe0000060ba470070) 0xa0000001007d3280 wait_for_completion+0x40 args (0xe0000060b8dafe08, 0xa00000010008d990, 0x38a, 0xffffffffffff9200) 0xa00000010008d990 sched_exec+0x1b0 args (0x2, 0xe0000060ba470000, 0xe0000060ba470010, 0xe000006003129200, 0xa00000010017e980, 0x58e, 0xa00000010017dce0) 0xa00000010017e980 do_execve+0xa0 args (0xe0000060f39e5000, 0x60000000000394b0, 0x6000000000056150, 0xe0000060b8dafe40, 0xe0000060f799f100, 0xe0000060f799bb00, 0xe0000060f799bbd8, 0x60000000000620b1, 0xa000000100013940) 0xa000000100013940 sys_execve+0x60 args (0xe0000060f39e5000, 0xe0000060f39e5000, 0x6000000000056150, 0xe0000060b8dafe40, 0xa00000010000a270, 0x50e, 0x2000000000028490) 0xa00000010000a270 ia64_execve+0x30 args (0x60000000000620a0, 0x60000000000394b0, 0x6000000000056150, 0x0, 0xc00000000000058e, 0x400000000003d020, 0x60000000000394b0, 0x0, 0xa00000010000aba0) 0xa00000010000aba0 ia64_ret_from_syscall args (0x60000000000620a0, 0x60000000000394b0, 0x6000000000056150, 0x0, 0xc00000000000058e, 0x400000000003d020, 0x60000000000394b0, 0x0) 0xa000000000010720 __kernel_syscall_via_break args (0x60000000000620a0, 0x60000000000394b0, 0x6000000000056150, 0x0, 0xc00000000000058e, 0x400000000003d020, 0x60000000000394b0, 0x0) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-05 13:51 ` Dimitri Sivanich @ 2008-06-05 14:18 ` Peter Zijlstra 0 siblings, 0 replies; 7+ messages in thread From: Peter Zijlstra @ 2008-06-05 14:18 UTC (permalink / raw) To: Dimitri Sivanich; +Cc: Cliff Wickman, linux-kernel On Thu, 2008-06-05 at 08:51 -0500, Dimitri Sivanich wrote: > On Thu, Jun 05, 2008 at 02:49:58PM +0200, Peter Zijlstra wrote: > > > > Does the below fix it? > > > > I don't believe so: Humpfh :-( I'll continue looking then... Thanks for testing. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-05 12:49 ` Peter Zijlstra 2008-06-05 13:51 ` Dimitri Sivanich @ 2008-06-10 10:19 ` Ingo Molnar 1 sibling, 0 replies; 7+ messages in thread From: Ingo Molnar @ 2008-06-10 10:19 UTC (permalink / raw) To: Peter Zijlstra; +Cc: Cliff Wickman, sivanich, linux-kernel * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote: > > > does: > > > > > > echo -1 > /proc/sys/kernel/sched_rt_runtime_us > > > > > > fix the problem? > > > > Yes! It does. > > > > Dimitri Sivanich has run into what looks like a similar problem. > > Hope the above workaround is a good clue to its solution. > > Does the below fix it? while it's not the full fix i've applied it to tip/sched-devel for more testing. Thanks, Ingo ^ permalink raw reply [flat|nested] 7+ messages in thread
* [BUG] cpu hotplug vs scheduler @ 2008-05-13 14:33 Avi Kivity 2008-05-14 8:13 ` Dmitry Adamushko 0 siblings, 1 reply; 7+ messages in thread From: Avi Kivity @ 2008-05-13 14:33 UTC (permalink / raw) To: linux-kernel; +Cc: Ingo Molnar I'm testing host cpu hotplug with kvm. Basically running 7 guests on a 4 core machine, offlining and onlining host cpus at random. Eventually I hit this: [4298303.496645] Booting processor 3/7 ip 6000 [4298303.506116] Initializing CPU#3 [4298303.506116] Calibrating delay using timer specific routine.. 5319.66 BogoMIPS (lpj=2659833) [4298303.506116] CPU: L1 I cache: 32K, L1 D cache: 32K [4298303.506116] CPU: L2 cache: 4096K [4298303.506116] CPU: Physical Processor ID: 3 [4298303.506116] CPU: Processor Core ID: 1 [4298303.506116] x86 PAT enabled: cpu 3, old 0x7040600070406, new 0x7010600070106 [4298303.582937] CPU3: Intel(R) Xeon(R) CPU 5150 @ 2.66GHz stepping 06 [4298303.585087] checking TSC synchronization [CPU#0 -> CPU#3]: passed. [4298303.707287] Switched to high resolution mode on CPU 3 [4298303.712943] kvm: enabling virtualization on CPU3 [4298303.713955] CPU0 attaching sched-domain: [4298303.713901] BUG: unable to handle kernel NULL pointer dereference at 0000000000000158 [4298303.713901] IP: [<ffffffff8022e722>] pick_next_task_fair+0x55/0x7c [4298303.713901] PGD 0 [4298303.713901] Oops: 0000 [1] PREEMPT SMP [4298303.713901] CPU 3 [4298303.713901] Modules linked in: kvm_intel kvm netconsole autofs4 nfs lockd nfs_acl sunrpc bridge llc acpi_cpufreq backlight sg e1000 button serio_raw rtc_cmos rtc_core rtc_lib ata_piix dm_snapshot dm_mod ahci libata dock sd_mod scsi_mod [last unloaded: kvm] [4298303.713901] Pid: 15115, comm: migration/3 Not tainted 2.6.26-rc2 #723 [4298303.713901] RIP: 0010:[<ffffffff8022e722>] [<ffffffff8022e722>] pick_next_task_fair+0x55/0x7c [4298303.713901] RSP: 0018:ffff81004fdfbe20 EFLAGS: 00010046 [4298303.713901] RAX: 0000000000000000 RBX: ffff81000103df80 RCX: 0000000000000000 [4298303.713901] RDX: ffff81000103e038 RSI: 000000003b9aca00 RDI: ffff81000103df00 [4298303.713901] RBP: ffff81004fdfbe40 R08: ffff81004fdfbdd0 R09: ffff81000103a0a0 [4298303.713901] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000000 [4298303.713901] R13: ffff81000103df00 R14: ffffffff8060a140 R15: 0000000000000003 [4298303.713901] FS: 0000000000000000(0000) GS:ffff81007f806a80(0000) knlGS:0000000000000000 [4298303.713901] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [4298303.713901] CR2: 0000000000000158 CR3: 0000000000201000 CR4: 00000000000026a0 [4298303.713901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [4298303.713901] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [4298303.713901] Process migration/3 (pid: 15115, threadinfo ffff81004fdfa000, task ffff81003f5f0700) [4298303.713901] Stack: ffff81004fdfbe40 ffff81000103df00 ffffffff804543c0 ffff810020985bf8 [4298303.713901] ffff81004fdfbee0 ffffffff804373fe ffff81004fdfbe80 ffffffff8023060a [4298303.713901] ffff81000103df00 ffff81003f5f0700 0000000000000000 ffff81003f5f0a60 [4298303.713901] Call Trace: [4298303.713901] [<ffffffff804373fe>] schedule+0x414/0x6ab [4298303.713901] [<ffffffff8023060a>] ? hrtick_set+0x9d/0xe8 [4298303.713901] [<ffffffff8043772f>] ? thread_return+0x9a/0xbf [4298303.713901] [<ffffffff80231652>] migration_thread+0x185/0x22d [4298303.713901] [<ffffffff802314cd>] ? migration_thread+0x0/0x22d [4298303.713901] [<ffffffff8024afe6>] kthread+0x49/0x77 [4298303.713901] [<ffffffff8020d228>] child_rip+0xa/0x12 [4298303.713901] [<ffffffff8024af9d>] ? kthread+0x0/0x77 [4298303.713901] [<ffffffff8020d21e>] ? child_rip+0x0/0x12 [4298303.713901] [4298303.713901] [4298303.713901] Code: c0 74 28 48 8b 7b 58 4c 8d 60 f0 48 85 ff 74 10 4c 89 e6 e8 df cc ff ff 85 c0 75 04 4c 8b 63 58 4c 89 e6 48 89 df e8 4a e5 ff ff <49> 8b 9c 24 58 01 00 00 48 85 db 75 bf 49 83 ec 38 4c 89 ef 4c [4298303.713901] RIP [<ffffffff8022e722>] pick_next_task_fair+0x55/0x7c This seems to be the assignment to cfs_rq after pick_next_entity(). I'm running kvm.git, which is currently 2.6.26-rc2 plus a few kvm patches. It could be kvm's fault, but it doesn't appear so from the traces. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] cpu hotplug vs scheduler 2008-05-13 14:33 [BUG] cpu hotplug vs scheduler Avi Kivity @ 2008-05-14 8:13 ` Dmitry Adamushko 2008-05-21 14:48 ` [BUG] hotplug cpus on ia64 Cliff Wickman 0 siblings, 1 reply; 7+ messages in thread From: Dmitry Adamushko @ 2008-05-14 8:13 UTC (permalink / raw) To: Avi Kivity Cc: linux-kernel, Ingo Molnar, Heiko Carstens, Peter Zijlstra, Srivatsa Vaddagiri [-- Attachment #1: Type: text/plain, Size: 2720 bytes --] Hi, > [ ... ] > > [4298303.713901] Call Trace: > [4298303.713901] [<ffffffff804373fe>] schedule+0x414/0x6ab > [4298303.713901] [<ffffffff8023060a>] ? hrtick_set+0x9d/0xe8 > [4298303.713901] [<ffffffff8043772f>] ? thread_return+0x9a/0xbf > [4298303.713901] [<ffffffff80231652>] migration_thread+0x185/0x22d > [4298303.713901] [<ffffffff802314cd>] ? migration_thread+0x0/0x22d > [4298303.713901] [<ffffffff8024afe6>] kthread+0x49/0x77 > [4298303.713901] [<ffffffff8020d228>] child_rip+0xa/0x12 > [4298303.713901] [<ffffffff8024af9d>] ? kthread+0x0/0x77 > [4298303.713901] [<ffffffff8020d21e>] ? child_rip+0x0/0x12 > [4298303.713901] > [4298303.713901] > [4298303.713901] Code: c0 74 28 48 8b 7b 58 4c 8d 60 f0 48 85 ff 74 10 4c > 89 e6 e8 df cc ff ff 85 c0 75 04 4c 8b 63 58 4c 89 e6 48 89 df e8 4a e5 ff > ff <49> 8b 9c 24 58 01 00 00 48 85 db 75 bf 49 83 ec 38 4c 89 ef 4c > [4298303.713901] RIP [<ffffffff8022e722>] pick_next_task_fair+0x55/0x7c > > This seems to be the assignment to cfs_rq after pick_next_entity(). [ cc'ed a few folks. ] So the cfs-tree likely gets out-of-sync. I pressume, it won't be reproducible with CONFIG_SCHED_GROUP options being disabled. Anyway, would you try one of these debug-patches (not sure about the workability of the second one though :-/) Let's check what are the values for 'cfs_rq->weight.load/nr_running'. thanks in advance, (non-whitespace-damaged versions are enclosed) --- --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -1291,6 +1291,12 @@ static struct task_struct *pick_next_task_fair(struct rq *rq) do { se = pick_next_entity(cfs_rq); + + if (unlikely(!se)) + printk(KERN_ERR "BUG: se == NULL but nr_running (%ld), load (%ld)," + " rq-nr_running (%ld), rq-load (%ld)\n", + cfs_rq->nr_running, cfs_rq->load.weight, rq->nr_running, rq->load.weight); + cfs_rq = group_cfs_rq(se); } while (cfs_rq); --- --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -1280,6 +1280,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_str uct *p) resched_task(curr); } +static void sysrq_sched_debug_show(void); + static struct task_struct *pick_next_task_fair(struct rq *rq) { struct task_struct *p; @@ -1291,6 +1293,10 @@ static struct task_struct *pick_next_task_fair(struct rq *rq) do { se = pick_next_entity(cfs_rq); + + if (unlikely(!se)) + sysrq_sched_debug_show(); + cfs_rq = group_cfs_rq(se); } while (cfs_rq); --- -- Best regards, Dmitry Adamushko [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: cfs_rq-debug-2.patch --] [-- Type: text/x-patch; name=cfs_rq-debug-2.patch, Size: 630 bytes --] diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c index e24ecd3..1dcc470 100644 --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -1280,6 +1280,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p) resched_task(curr); } +static void sysrq_sched_debug_show(void); + static struct task_struct *pick_next_task_fair(struct rq *rq) { struct task_struct *p; @@ -1291,6 +1293,10 @@ static struct task_struct *pick_next_task_fair(struct rq *rq) do { se = pick_next_entity(cfs_rq); + + if (unlikely(!se)) + sysrq_sched_debug_show(); + cfs_rq = group_cfs_rq(se); } while (cfs_rq); [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: cfs_rq-debug-3.patch --] [-- Type: text/x-patch; name=cfs_rq-debug-3.patch, Size: 542 bytes --] diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c index e24ecd3..e21e020 100644 --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -1291,6 +1291,12 @@ static struct task_struct *pick_next_task_fair(struct rq *rq) do { se = pick_next_entity(cfs_rq); + + if (unlikely(!se)) + printk(KERN_ERR "BUG: se == NULL but nr_running (%ld), load (%ld)," + " rq-nr_running (%ld), rq-load (%ld)\n", + cfs_rq->nr_running, cfs_rq->load.weight, rq->nr_running, rq->load.weight); + cfs_rq = group_cfs_rq(se); } while (cfs_rq); ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [BUG] hotplug cpus on ia64 2008-05-14 8:13 ` Dmitry Adamushko @ 2008-05-21 14:48 ` Cliff Wickman 0 siblings, 0 replies; 7+ messages in thread From: Cliff Wickman @ 2008-05-21 14:48 UTC (permalink / raw) To: dmitry.adamushko, avi, mingo, heiko.carstens, a.p.zijlstra, vatsa Cc: Avi Kivity, linux-kernel, Ingo Molnar, Heiko Carstens, PeterZijlst Gentlemen, I built an ia64 kernel from Andrew's tree (2.6.26-rc2-mm1) and get a very predictable hotplug cpu problem. billberry1:/tmp/cpw # ./dis disabled cpu 17 enabled cpu 17 billberry1:/tmp/cpw # ./dis disabled cpu 17 enabled cpu 17 billberry1:/tmp/cpw # ./dis The script that disables the cpu always hangs (unkillable) on the 3rd attempt. I haven't spent any debugging time on it yet. Just wondering if you've seen it? It doesn't seem to happen x86_64. -Cliff Wickman cpw@sgi.com ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2008-06-10 10:20 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <E1K1l2y-0007bu-44@eag09.americas.sgi.com>
[not found] ` <1212154614.12349.244.camel@twins>
2008-06-03 22:17 ` [BUG] hotplug cpus on ia64 Cliff Wickman
2008-06-04 13:50 ` Dimitri Sivanich
2008-06-05 12:49 ` Peter Zijlstra
2008-06-05 13:51 ` Dimitri Sivanich
2008-06-05 14:18 ` Peter Zijlstra
2008-06-10 10:19 ` Ingo Molnar
2008-05-13 14:33 [BUG] cpu hotplug vs scheduler Avi Kivity
2008-05-14 8:13 ` Dmitry Adamushko
2008-05-21 14:48 ` [BUG] hotplug cpus on ia64 Cliff Wickman
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.