* Re: [BUG] hotplug cpus on ia64 [not found] ` <1212154614.12349.244.camel@twins> @ 2008-06-03 22:17 ` Cliff Wickman 2008-06-04 13:50 ` Dimitri Sivanich 2008-06-05 12:49 ` Peter Zijlstra 0 siblings, 2 replies; 7+ messages in thread From: Cliff Wickman @ 2008-06-03 22:17 UTC (permalink / raw) To: Peter Zijlstra; +Cc: sivanich, linux-kernel On Fri, May 30, 2008 at 03:36:54PM +0200, Peter Zijlstra wrote: > On Thu, 2008-05-29 at 11:32 -0500, Cliff Wickman wrote: > > >> I built an ia64 kernel from Andrew's tree (2.6.26-rc2-mm1) > > >> and get a very predictable hotplug cpu problem. > > >> billberry1:/tmp/cpw # ./dis > > >> disabled cpu 17 > > >> enabled cpu 17 > > >> billberry1:/tmp/cpw # ./dis > > >> disabled cpu 17 > > >> enabled cpu 17 > > >> billberry1:/tmp/cpw # ./dis > > >> > > >> The script that disables the cpu always hangs (unkillable) > > >> on the 3rd attempt. > > > > > And a bit further: > > > The kstopmachine thread always sits on the run queue (real time) for about > > > 30 minutes before running. > > > > And a bit further: > > > > The kstopmachine thread is queued as real-time on the downed cpu: > > >> rq -f 17 > > CPU# runq address size Lock current task time name > > ========================================================================== > > 17 0xe000046003059540 3 U 0xe0000360f06f8000 0 swapper > > Total of 3 queued: > > 3 real time tasks: px *(rt_rq *)0xe000046003059608 > > exclusive queue: > > slot 0 > > 0xe0000760f4628000 0 migration/17 > > 0xe0000760f4708000 0 kstopmachine > > 0xe0000760f6678000 0 watchdog/17 > > > > I put in counters and see that schedule() is never again entered by cpu 17 > > after it is downed the 3rd time. > > (it is entered after being up'd the first two times) > > > > The kstopmachine thread is bound to cpu 17 by __stop_machine_run()'s call > > to kthread_bind(). > > > > A cpu does not schedule after being downed, of course. But it does again > > after being up'd. > > Why would the second up be different? Following it, if the cpu is > > downed it never schedules again. > > > > If I always bind kstopmachine to cpu 0 the problem disappears. > > does: > > echo -1 > /proc/sys/kernel/sched_rt_runtime_us > > fix the problem? Yes! It does. Dimitri Sivanich has run into what looks like a similar problem. Hope the above workaround is a good clue to its solution. -- Cliff Wickman Silicon Graphics, Inc. cpw@sgi.com (651) 683-3824 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-03 22:17 ` [BUG] hotplug cpus on ia64 Cliff Wickman @ 2008-06-04 13:50 ` Dimitri Sivanich 2008-06-05 12:49 ` Peter Zijlstra 1 sibling, 0 replies; 7+ messages in thread From: Dimitri Sivanich @ 2008-06-04 13:50 UTC (permalink / raw) To: Cliff Wickman; +Cc: Peter Zijlstra, linux-kernel On Tue, Jun 03, 2008 at 05:17:59PM -0500, Cliff Wickman wrote: > > On Fri, May 30, 2008 at 03:36:54PM +0200, Peter Zijlstra wrote: > > On Thu, 2008-05-29 at 11:32 -0500, Cliff Wickman wrote: > > > I put in counters and see that schedule() is never again entered by cpu 17 > > > after it is downed the 3rd time. > > > (it is entered after being up'd the first two times) > > > > > > The kstopmachine thread is bound to cpu 17 by __stop_machine_run()'s call > > > to kthread_bind(). > > > > > > A cpu does not schedule after being downed, of course. But it does again > > > after being up'd. > > > Why would the second up be different? Following it, if the cpu is > > > downed it never schedules again. > > > > > > If I always bind kstopmachine to cpu 0 the problem disappears. > > > > does: > > > > echo -1 > /proc/sys/kernel/sched_rt_runtime_us > > > > fix the problem? > > Yes! It does. > > Dimitri Sivanich has run into what looks like a similar problem. > Hope the above workaround is a good clue to its solution. This fixes the problem I was seeing as well. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-03 22:17 ` [BUG] hotplug cpus on ia64 Cliff Wickman 2008-06-04 13:50 ` Dimitri Sivanich @ 2008-06-05 12:49 ` Peter Zijlstra 2008-06-05 13:51 ` Dimitri Sivanich 2008-06-10 10:19 ` Ingo Molnar 1 sibling, 2 replies; 7+ messages in thread From: Peter Zijlstra @ 2008-06-05 12:49 UTC (permalink / raw) To: Cliff Wickman; +Cc: sivanich, linux-kernel On Tue, 2008-06-03 at 17:17 -0500, Cliff Wickman wrote: > On Fri, May 30, 2008 at 03:36:54PM +0200, Peter Zijlstra wrote: > > On Thu, 2008-05-29 at 11:32 -0500, Cliff Wickman wrote: > > > >> I built an ia64 kernel from Andrew's tree (2.6.26-rc2-mm1) > > > >> and get a very predictable hotplug cpu problem. > > > >> billberry1:/tmp/cpw # ./dis > > > >> disabled cpu 17 > > > >> enabled cpu 17 > > > >> billberry1:/tmp/cpw # ./dis > > > >> disabled cpu 17 > > > >> enabled cpu 17 > > > >> billberry1:/tmp/cpw # ./dis > > > >> > > > >> The script that disables the cpu always hangs (unkillable) > > > >> on the 3rd attempt. > > > > > > > And a bit further: > > > > The kstopmachine thread always sits on the run queue (real time) for about > > > > 30 minutes before running. > > > > > > And a bit further: > > > > > > The kstopmachine thread is queued as real-time on the downed cpu: > > > >> rq -f 17 > > > CPU# runq address size Lock current task time name > > > ========================================================================== > > > 17 0xe000046003059540 3 U 0xe0000360f06f8000 0 swapper > > > Total of 3 queued: > > > 3 real time tasks: px *(rt_rq *)0xe000046003059608 > > > exclusive queue: > > > slot 0 > > > 0xe0000760f4628000 0 migration/17 > > > 0xe0000760f4708000 0 kstopmachine > > > 0xe0000760f6678000 0 watchdog/17 > > > > > > I put in counters and see that schedule() is never again entered by cpu 17 > > > after it is downed the 3rd time. > > > (it is entered after being up'd the first two times) > > > > > > The kstopmachine thread is bound to cpu 17 by __stop_machine_run()'s call > > > to kthread_bind(). > > > > > > A cpu does not schedule after being downed, of course. But it does again > > > after being up'd. > > > Why would the second up be different? Following it, if the cpu is > > > downed it never schedules again. > > > > > > If I always bind kstopmachine to cpu 0 the problem disappears. > > > > does: > > > > echo -1 > /proc/sys/kernel/sched_rt_runtime_us > > > > fix the problem? > > Yes! It does. > > Dimitri Sivanich has run into what looks like a similar problem. > Hope the above workaround is a good clue to its solution. Does the below fix it? Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> --- kernel/sched.c | 15 +++++-- kernel/sched_rt.c | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 115 insertions(+), 9 deletions(-) Index: linux-2.6/kernel/sched_rt.c =================================================================== --- linux-2.6.orig/kernel/sched_rt.c +++ linux-2.6/kernel/sched_rt.c @@ -280,6 +280,9 @@ static int balance_runtime(struct rt_rq continue; spin_lock(&iter->rt_runtime_lock); + if (iter->rt_runtime == RUNTIME_INF) + goto next; + diff = iter->rt_runtime - iter->rt_time; if (diff > 0) { do_div(diff, weight); @@ -293,12 +296,105 @@ static int balance_runtime(struct rt_rq break; } } +next: spin_unlock(&iter->rt_runtime_lock); } spin_unlock(&rt_b->rt_runtime_lock); return more; } + +static void __disable_runtime(struct rq *rq) +{ + struct root_domain *rd = rq->rd; + struct rt_rq *rt_rq; + + if (unlikely(!scheduler_running)) + return; + + for_each_leaf_rt_rq(rt_rq, rq) { + struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq); + s64 want; + int i; + + spin_lock(&rt_b->rt_runtime_lock); + spin_lock(&rt_rq->rt_runtime_lock); + if (rt_rq->rt_runtime == RUNTIME_INF || + rt_rq->rt_runtime == rt_b->rt_runtime) + goto balanced; + spin_unlock(&rt_rq->rt_runtime_lock); + + want = rt_b->rt_runtime - rt_rq->rt_runtime; + + for_each_cpu_mask(i, rd->span) { + struct rt_rq *iter = sched_rt_period_rt_rq(rt_b, i); + s64 diff; + + if (iter == rt_rq) + continue; + + spin_lock(&iter->rt_runtime_lock); + if (want > 0) { + diff = min_t(s64, iter->rt_runtime, want); + iter->rt_runtime -= diff; + want -= diff; + } else { + iter->rt_runtime -= want; + want -= want; + } + spin_unlock(&iter->rt_runtime_lock); + + if (!want) + break; + } + + spin_lock(&rt_rq->rt_runtime_lock); + BUG_ON(want); +balanced: + rt_rq->rt_runtime = RUNTIME_INF; + spin_unlock(&rt_rq->rt_runtime_lock); + spin_unlock(&rt_b->rt_runtime_lock); + } +} + +static void disable_runtime(struct rq *rq) +{ + unsigned long flags; + + spin_lock_irqsave(&rq->lock, flags); + __disable_runtime(rq); + spin_unlock_irqrestore(&rq->lock, flags); +} + +static void __enable_runtime(struct rq *rq) +{ + struct root_domain *rd = rq->rd; + struct rt_rq *rt_rq; + + if (unlikely(!scheduler_running)) + return; + + for_each_leaf_rt_rq(rt_rq, rq) { + struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq); + + spin_lock(&rt_b->rt_runtime_lock); + spin_lock(&rt_rq->rt_runtime_lock); + rt_rq->rt_runtime = rt_b->rt_runtime; + rt_rq->rt_time = 0; + spin_unlock(&rt_rq->rt_runtime_lock); + spin_unlock(&rt_b->rt_runtime_lock); + } +} + +static void enable_runtime(struct rq *rq) +{ + unsigned long flags; + + spin_lock_irqsave(&rq->lock, flags); + __enable_runtime(rq); + spin_unlock_irqrestore(&rq->lock, flags); +} + #endif static inline int rt_se_prio(struct sched_rt_entity *rt_se) @@ -328,14 +424,13 @@ static int sched_rt_runtime_exceeded(str #ifdef CONFIG_SMP if (rt_rq->rt_time > runtime) { - int more; - spin_unlock(&rt_rq->rt_runtime_lock); - more = balance_runtime(rt_rq); + balance_runtime(rt_rq); spin_lock(&rt_rq->rt_runtime_lock); - if (more) - runtime = sched_rt_runtime(rt_rq); + runtime = sched_rt_runtime(rt_rq); + if (runtime == RUNTIME_INF) + return 0; } #endif @@ -1157,6 +1252,8 @@ static void join_domain_rt(struct rq *rq { if (rq->rt.overloaded) rt_set_overload(rq); + + __enable_runtime(rq); } /* Assumes rq->lock is held */ @@ -1164,6 +1261,8 @@ static void leave_domain_rt(struct rq *r { if (rq->rt.overloaded) rt_clear_overload(rq); + + __disable_runtime(rq); } /* Index: linux-2.6/kernel/sched.c =================================================================== --- linux-2.6.orig/kernel/sched.c +++ linux-2.6/kernel/sched.c @@ -7455,20 +7455,27 @@ int sched_create_sysfs_power_savings_ent static int update_sched_domains(struct notifier_block *nfb, unsigned long action, void *hcpu) { + int cpu = (int)(long)hcpu; + switch (action) { - case CPU_UP_PREPARE: - case CPU_UP_PREPARE_FROZEN: case CPU_DOWN_PREPARE: case CPU_DOWN_PREPARE_FROZEN: + disable_runtime(cpu_rq(cpu)); + /* fall-through */ + case CPU_UP_PREPARE: + case CPU_UP_PREPARE_FROZEN: detach_destroy_domains(&cpu_online_map); return NOTIFY_OK; - case CPU_UP_CANCELED: - case CPU_UP_CANCELED_FROZEN: + case CPU_DOWN_FAILED: case CPU_DOWN_FAILED_FROZEN: case CPU_ONLINE: case CPU_ONLINE_FROZEN: + enable_runtime(cpu_rq(cpu)); + /* fall-through */ + case CPU_UP_CANCELED: + case CPU_UP_CANCELED_FROZEN: case CPU_DEAD: case CPU_DEAD_FROZEN: /* ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-05 12:49 ` Peter Zijlstra @ 2008-06-05 13:51 ` Dimitri Sivanich 2008-06-05 14:18 ` Peter Zijlstra 2008-06-10 10:19 ` Ingo Molnar 1 sibling, 1 reply; 7+ messages in thread From: Dimitri Sivanich @ 2008-06-05 13:51 UTC (permalink / raw) To: Peter Zijlstra; +Cc: Cliff Wickman, linux-kernel On Thu, Jun 05, 2008 at 02:49:58PM +0200, Peter Zijlstra wrote: > > Does the below fix it? > I don't believe so: :~ # taskset -p $$ pid 4502's current affinity mask: 8 :~ # cd /sys/devices/system :/sys/devices/system # cd cpu :/sys/devices/system/cpu # cd cpu2 :/sys/devices/system/cpu/cpu2 # cat online 1 :/sys/devices/system/cpu/cpu2 # echo 0 >online :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: 8 :/sys/devices/system/cpu/cpu2 # taskset -cp 0-3 $$ pid 4502's current affinity list: 3 pid 4502's new affinity list: 0,1,3 :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: b :/sys/devices/system/cpu/cpu2 # echo 1 >online :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: b :/sys/devices/system/cpu/cpu2 # taskset -cp 0-3 $$ pid 4502's current affinity list: 0,1,3 pid 4502's new affinity list: 0-3 :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: f :/sys/devices/system/cpu/cpu2 # echo 0 >online :/sys/devices/system/cpu/cpu2 # taskset -p $$ pid 4502's current affinity mask: b :/sys/devices/system/cpu/cpu2 # echo 1 >online :/sys/devices/system/cpu/cpu2 # taskset -p $$ (above command now hangs) (ps output) 0xe0000060b5650000 4502 4349 0 2 S 0xe0000060b5650390 bash 0xe0000060b8da0000 4843 4502 0 2 D 0xe0000060b8da0390 bash Stack traceback for pid 4843 0xe0000060b8da0000 4843 4502 0 2 D 0xe0000060b8da0390 bash 0xa0000001007d44b0 schedule+0x1210 args (0xe0000060ba470ce4, 0xa000000100dae190, 0xe000006003129200, 0xa000000100084b70, 0x48c, 0xe0000060b8dafda8, 0xe000006003129200, 0x200, 0xe0000060f780fe80) 0xa0000001007d4ac0 schedule_timeout+0x40 args (0x7fffffffffffffff, 0x0, 0x0, 0xa0000001007d2f00, 0x309, 0xe000006003129200) 0xa0000001007d2f00 wait_for_common+0x240 args (0xe0000060b8dafe08, 0x7fffffffffffffff, 0x2, 0xa0000001007d3280, 0x207, 0xe0000060ba470070) 0xa0000001007d3280 wait_for_completion+0x40 args (0xe0000060b8dafe08, 0xa00000010008d990, 0x38a, 0xffffffffffff9200) 0xa00000010008d990 sched_exec+0x1b0 args (0x2, 0xe0000060ba470000, 0xe0000060ba470010, 0xe000006003129200, 0xa00000010017e980, 0x58e, 0xa00000010017dce0) 0xa00000010017e980 do_execve+0xa0 args (0xe0000060f39e5000, 0x60000000000394b0, 0x6000000000056150, 0xe0000060b8dafe40, 0xe0000060f799f100, 0xe0000060f799bb00, 0xe0000060f799bbd8, 0x60000000000620b1, 0xa000000100013940) 0xa000000100013940 sys_execve+0x60 args (0xe0000060f39e5000, 0xe0000060f39e5000, 0x6000000000056150, 0xe0000060b8dafe40, 0xa00000010000a270, 0x50e, 0x2000000000028490) 0xa00000010000a270 ia64_execve+0x30 args (0x60000000000620a0, 0x60000000000394b0, 0x6000000000056150, 0x0, 0xc00000000000058e, 0x400000000003d020, 0x60000000000394b0, 0x0, 0xa00000010000aba0) 0xa00000010000aba0 ia64_ret_from_syscall args (0x60000000000620a0, 0x60000000000394b0, 0x6000000000056150, 0x0, 0xc00000000000058e, 0x400000000003d020, 0x60000000000394b0, 0x0) 0xa000000000010720 __kernel_syscall_via_break args (0x60000000000620a0, 0x60000000000394b0, 0x6000000000056150, 0x0, 0xc00000000000058e, 0x400000000003d020, 0x60000000000394b0, 0x0) ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-05 13:51 ` Dimitri Sivanich @ 2008-06-05 14:18 ` Peter Zijlstra 0 siblings, 0 replies; 7+ messages in thread From: Peter Zijlstra @ 2008-06-05 14:18 UTC (permalink / raw) To: Dimitri Sivanich; +Cc: Cliff Wickman, linux-kernel On Thu, 2008-06-05 at 08:51 -0500, Dimitri Sivanich wrote: > On Thu, Jun 05, 2008 at 02:49:58PM +0200, Peter Zijlstra wrote: > > > > Does the below fix it? > > > > I don't believe so: Humpfh :-( I'll continue looking then... Thanks for testing. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] hotplug cpus on ia64 2008-06-05 12:49 ` Peter Zijlstra 2008-06-05 13:51 ` Dimitri Sivanich @ 2008-06-10 10:19 ` Ingo Molnar 1 sibling, 0 replies; 7+ messages in thread From: Ingo Molnar @ 2008-06-10 10:19 UTC (permalink / raw) To: Peter Zijlstra; +Cc: Cliff Wickman, sivanich, linux-kernel * Peter Zijlstra <a.p.zijlstra@chello.nl> wrote: > > > does: > > > > > > echo -1 > /proc/sys/kernel/sched_rt_runtime_us > > > > > > fix the problem? > > > > Yes! It does. > > > > Dimitri Sivanich has run into what looks like a similar problem. > > Hope the above workaround is a good clue to its solution. > > Does the below fix it? while it's not the full fix i've applied it to tip/sched-devel for more testing. Thanks, Ingo ^ permalink raw reply [flat|nested] 7+ messages in thread
* [BUG] cpu hotplug vs scheduler @ 2008-05-13 14:33 Avi Kivity 2008-05-14 8:13 ` Dmitry Adamushko 0 siblings, 1 reply; 7+ messages in thread From: Avi Kivity @ 2008-05-13 14:33 UTC (permalink / raw) To: linux-kernel; +Cc: Ingo Molnar I'm testing host cpu hotplug with kvm. Basically running 7 guests on a 4 core machine, offlining and onlining host cpus at random. Eventually I hit this: [4298303.496645] Booting processor 3/7 ip 6000 [4298303.506116] Initializing CPU#3 [4298303.506116] Calibrating delay using timer specific routine.. 5319.66 BogoMIPS (lpj=2659833) [4298303.506116] CPU: L1 I cache: 32K, L1 D cache: 32K [4298303.506116] CPU: L2 cache: 4096K [4298303.506116] CPU: Physical Processor ID: 3 [4298303.506116] CPU: Processor Core ID: 1 [4298303.506116] x86 PAT enabled: cpu 3, old 0x7040600070406, new 0x7010600070106 [4298303.582937] CPU3: Intel(R) Xeon(R) CPU 5150 @ 2.66GHz stepping 06 [4298303.585087] checking TSC synchronization [CPU#0 -> CPU#3]: passed. [4298303.707287] Switched to high resolution mode on CPU 3 [4298303.712943] kvm: enabling virtualization on CPU3 [4298303.713955] CPU0 attaching sched-domain: [4298303.713901] BUG: unable to handle kernel NULL pointer dereference at 0000000000000158 [4298303.713901] IP: [<ffffffff8022e722>] pick_next_task_fair+0x55/0x7c [4298303.713901] PGD 0 [4298303.713901] Oops: 0000 [1] PREEMPT SMP [4298303.713901] CPU 3 [4298303.713901] Modules linked in: kvm_intel kvm netconsole autofs4 nfs lockd nfs_acl sunrpc bridge llc acpi_cpufreq backlight sg e1000 button serio_raw rtc_cmos rtc_core rtc_lib ata_piix dm_snapshot dm_mod ahci libata dock sd_mod scsi_mod [last unloaded: kvm] [4298303.713901] Pid: 15115, comm: migration/3 Not tainted 2.6.26-rc2 #723 [4298303.713901] RIP: 0010:[<ffffffff8022e722>] [<ffffffff8022e722>] pick_next_task_fair+0x55/0x7c [4298303.713901] RSP: 0018:ffff81004fdfbe20 EFLAGS: 00010046 [4298303.713901] RAX: 0000000000000000 RBX: ffff81000103df80 RCX: 0000000000000000 [4298303.713901] RDX: ffff81000103e038 RSI: 000000003b9aca00 RDI: ffff81000103df00 [4298303.713901] RBP: ffff81004fdfbe40 R08: ffff81004fdfbdd0 R09: ffff81000103a0a0 [4298303.713901] R10: 0000000000000000 R11: 0000000000000003 R12: 0000000000000000 [4298303.713901] R13: ffff81000103df00 R14: ffffffff8060a140 R15: 0000000000000003 [4298303.713901] FS: 0000000000000000(0000) GS:ffff81007f806a80(0000) knlGS:0000000000000000 [4298303.713901] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [4298303.713901] CR2: 0000000000000158 CR3: 0000000000201000 CR4: 00000000000026a0 [4298303.713901] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [4298303.713901] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [4298303.713901] Process migration/3 (pid: 15115, threadinfo ffff81004fdfa000, task ffff81003f5f0700) [4298303.713901] Stack: ffff81004fdfbe40 ffff81000103df00 ffffffff804543c0 ffff810020985bf8 [4298303.713901] ffff81004fdfbee0 ffffffff804373fe ffff81004fdfbe80 ffffffff8023060a [4298303.713901] ffff81000103df00 ffff81003f5f0700 0000000000000000 ffff81003f5f0a60 [4298303.713901] Call Trace: [4298303.713901] [<ffffffff804373fe>] schedule+0x414/0x6ab [4298303.713901] [<ffffffff8023060a>] ? hrtick_set+0x9d/0xe8 [4298303.713901] [<ffffffff8043772f>] ? thread_return+0x9a/0xbf [4298303.713901] [<ffffffff80231652>] migration_thread+0x185/0x22d [4298303.713901] [<ffffffff802314cd>] ? migration_thread+0x0/0x22d [4298303.713901] [<ffffffff8024afe6>] kthread+0x49/0x77 [4298303.713901] [<ffffffff8020d228>] child_rip+0xa/0x12 [4298303.713901] [<ffffffff8024af9d>] ? kthread+0x0/0x77 [4298303.713901] [<ffffffff8020d21e>] ? child_rip+0x0/0x12 [4298303.713901] [4298303.713901] [4298303.713901] Code: c0 74 28 48 8b 7b 58 4c 8d 60 f0 48 85 ff 74 10 4c 89 e6 e8 df cc ff ff 85 c0 75 04 4c 8b 63 58 4c 89 e6 48 89 df e8 4a e5 ff ff <49> 8b 9c 24 58 01 00 00 48 85 db 75 bf 49 83 ec 38 4c 89 ef 4c [4298303.713901] RIP [<ffffffff8022e722>] pick_next_task_fair+0x55/0x7c This seems to be the assignment to cfs_rq after pick_next_entity(). I'm running kvm.git, which is currently 2.6.26-rc2 plus a few kvm patches. It could be kvm's fault, but it doesn't appear so from the traces. -- error compiling committee.c: too many arguments to function ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [BUG] cpu hotplug vs scheduler 2008-05-13 14:33 [BUG] cpu hotplug vs scheduler Avi Kivity @ 2008-05-14 8:13 ` Dmitry Adamushko 2008-05-21 14:48 ` [BUG] hotplug cpus on ia64 Cliff Wickman 0 siblings, 1 reply; 7+ messages in thread From: Dmitry Adamushko @ 2008-05-14 8:13 UTC (permalink / raw) To: Avi Kivity Cc: linux-kernel, Ingo Molnar, Heiko Carstens, Peter Zijlstra, Srivatsa Vaddagiri [-- Attachment #1: Type: text/plain, Size: 2720 bytes --] Hi, > [ ... ] > > [4298303.713901] Call Trace: > [4298303.713901] [<ffffffff804373fe>] schedule+0x414/0x6ab > [4298303.713901] [<ffffffff8023060a>] ? hrtick_set+0x9d/0xe8 > [4298303.713901] [<ffffffff8043772f>] ? thread_return+0x9a/0xbf > [4298303.713901] [<ffffffff80231652>] migration_thread+0x185/0x22d > [4298303.713901] [<ffffffff802314cd>] ? migration_thread+0x0/0x22d > [4298303.713901] [<ffffffff8024afe6>] kthread+0x49/0x77 > [4298303.713901] [<ffffffff8020d228>] child_rip+0xa/0x12 > [4298303.713901] [<ffffffff8024af9d>] ? kthread+0x0/0x77 > [4298303.713901] [<ffffffff8020d21e>] ? child_rip+0x0/0x12 > [4298303.713901] > [4298303.713901] > [4298303.713901] Code: c0 74 28 48 8b 7b 58 4c 8d 60 f0 48 85 ff 74 10 4c > 89 e6 e8 df cc ff ff 85 c0 75 04 4c 8b 63 58 4c 89 e6 48 89 df e8 4a e5 ff > ff <49> 8b 9c 24 58 01 00 00 48 85 db 75 bf 49 83 ec 38 4c 89 ef 4c > [4298303.713901] RIP [<ffffffff8022e722>] pick_next_task_fair+0x55/0x7c > > This seems to be the assignment to cfs_rq after pick_next_entity(). [ cc'ed a few folks. ] So the cfs-tree likely gets out-of-sync. I pressume, it won't be reproducible with CONFIG_SCHED_GROUP options being disabled. Anyway, would you try one of these debug-patches (not sure about the workability of the second one though :-/) Let's check what are the values for 'cfs_rq->weight.load/nr_running'. thanks in advance, (non-whitespace-damaged versions are enclosed) --- --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -1291,6 +1291,12 @@ static struct task_struct *pick_next_task_fair(struct rq *rq) do { se = pick_next_entity(cfs_rq); + + if (unlikely(!se)) + printk(KERN_ERR "BUG: se == NULL but nr_running (%ld), load (%ld)," + " rq-nr_running (%ld), rq-load (%ld)\n", + cfs_rq->nr_running, cfs_rq->load.weight, rq->nr_running, rq->load.weight); + cfs_rq = group_cfs_rq(se); } while (cfs_rq); --- --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -1280,6 +1280,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_str uct *p) resched_task(curr); } +static void sysrq_sched_debug_show(void); + static struct task_struct *pick_next_task_fair(struct rq *rq) { struct task_struct *p; @@ -1291,6 +1293,10 @@ static struct task_struct *pick_next_task_fair(struct rq *rq) do { se = pick_next_entity(cfs_rq); + + if (unlikely(!se)) + sysrq_sched_debug_show(); + cfs_rq = group_cfs_rq(se); } while (cfs_rq); --- -- Best regards, Dmitry Adamushko [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: cfs_rq-debug-2.patch --] [-- Type: text/x-patch; name=cfs_rq-debug-2.patch, Size: 630 bytes --] diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c index e24ecd3..1dcc470 100644 --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -1280,6 +1280,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p) resched_task(curr); } +static void sysrq_sched_debug_show(void); + static struct task_struct *pick_next_task_fair(struct rq *rq) { struct task_struct *p; @@ -1291,6 +1293,10 @@ static struct task_struct *pick_next_task_fair(struct rq *rq) do { se = pick_next_entity(cfs_rq); + + if (unlikely(!se)) + sysrq_sched_debug_show(); + cfs_rq = group_cfs_rq(se); } while (cfs_rq); [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: cfs_rq-debug-3.patch --] [-- Type: text/x-patch; name=cfs_rq-debug-3.patch, Size: 542 bytes --] diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c index e24ecd3..e21e020 100644 --- a/kernel/sched_fair.c +++ b/kernel/sched_fair.c @@ -1291,6 +1291,12 @@ static struct task_struct *pick_next_task_fair(struct rq *rq) do { se = pick_next_entity(cfs_rq); + + if (unlikely(!se)) + printk(KERN_ERR "BUG: se == NULL but nr_running (%ld), load (%ld)," + " rq-nr_running (%ld), rq-load (%ld)\n", + cfs_rq->nr_running, cfs_rq->load.weight, rq->nr_running, rq->load.weight); + cfs_rq = group_cfs_rq(se); } while (cfs_rq); ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [BUG] hotplug cpus on ia64 2008-05-14 8:13 ` Dmitry Adamushko @ 2008-05-21 14:48 ` Cliff Wickman 0 siblings, 0 replies; 7+ messages in thread From: Cliff Wickman @ 2008-05-21 14:48 UTC (permalink / raw) To: dmitry.adamushko, avi, mingo, heiko.carstens, a.p.zijlstra, vatsa Cc: Avi Kivity, linux-kernel, Ingo Molnar, Heiko Carstens, PeterZijlst Gentlemen, I built an ia64 kernel from Andrew's tree (2.6.26-rc2-mm1) and get a very predictable hotplug cpu problem. billberry1:/tmp/cpw # ./dis disabled cpu 17 enabled cpu 17 billberry1:/tmp/cpw # ./dis disabled cpu 17 enabled cpu 17 billberry1:/tmp/cpw # ./dis The script that disables the cpu always hangs (unkillable) on the 3rd attempt. I haven't spent any debugging time on it yet. Just wondering if you've seen it? It doesn't seem to happen x86_64. -Cliff Wickman cpw@sgi.com ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2008-06-10 10:20 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <E1K1l2y-0007bu-44@eag09.americas.sgi.com>
[not found] ` <1212154614.12349.244.camel@twins>
2008-06-03 22:17 ` [BUG] hotplug cpus on ia64 Cliff Wickman
2008-06-04 13:50 ` Dimitri Sivanich
2008-06-05 12:49 ` Peter Zijlstra
2008-06-05 13:51 ` Dimitri Sivanich
2008-06-05 14:18 ` Peter Zijlstra
2008-06-10 10:19 ` Ingo Molnar
2008-05-13 14:33 [BUG] cpu hotplug vs scheduler Avi Kivity
2008-05-14 8:13 ` Dmitry Adamushko
2008-05-21 14:48 ` [BUG] hotplug cpus on ia64 Cliff Wickman
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox