From: Mike Galbraith <efault@gmx.de>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>,
LKML <linux-kernel@vger.kernel.org>,
linux-rt-users <linux-rt-users@vger.kernel.org>
Subject: Re: [ANNOUNCE] 3.0.4-rt13
Date: Tue, 13 Sep 2011 17:17:37 +0200 [thread overview]
Message-ID: <1315927057.6445.2.camel@marge.simson.net> (raw)
In-Reply-To: <1315921009.5977.6.camel@twins>
On Tue, 2011-09-13 at 15:36 +0200, Peter Zijlstra wrote:
> On Mon, 2011-09-12 at 16:53 +0200, Mike Galbraith wrote:
> > On Mon, 2011-09-12 at 15:52 +0200, Mike Galbraith wrote:
> >
> > > 36717 <...>-6266 [002] 242.543129: sys_semop <-system_call_fastpath
> > > 36718 <...>-6266 [002] 242.543129: sys_semtimedop <-sys_semop
> > > 36719 <...>-6266 [002] 242.543131: ipc_lock_check <-sys_semtimedop
> > > 36720 <...>-6266 [002] 242.543131: ipc_lock <-ipc_lock_check
> > > 36721 <...>-6266 [002] 242.543132: __rcu_read_lock <-ipc_lock
> > > 36722 <...>-6266 [002] 242.543133: migrate_disable <-ipc_lock
> > > 36723 <...>-6266 [002] 242.543134: migrate_disable: migrate_disable: in_atomic:0 p->migrate_disable_atomic:0 p->migrate_disable:0
> > > 36724 <...>-6266 [002] 242.543134: pin_current_cpu <-migrate_disable
> > > 36725 <...>-6266 [002] 242.543134: _raw_spin_lock_irqsave <-migrate_disable
> > > 36726 <...>-6266 [002] 242.543135: _raw_spin_unlock_irqrestore <-migrate_disable
> > > 36727 <...>-6266 [002] 242.543135: rt_spin_lock <-ipc_lock
> > > 36728 <...>-6266 [002] 242.543136: ipcperms <-sys_semtimedop
> > > 36729 <...>-6266 [002] 242.543137: ns_capable <-ipcperms
> > > 36730 <...>-6266 [002] 242.543138: cap_capable <-ns_capable
> > > 36731 <...>-6266 [002] 242.543138: pid_vnr <-sys_semtimedop
> > > 36732 <...>-6266 [002] 242.543139: try_atomic_semop <-sys_semtimedop
> > > 36733 <...>-6266 [002] 242.543140: do_smart_update <-sys_semtimedop
> > > 36734 <...>-6266 [002] 242.543140: update_queue <-do_smart_update
> > > 36735 <...>-6266 [002] 242.543141: try_atomic_semop <-update_queue
> > > 36736 <...>-6266 [002] 242.543142: update_queue <-do_smart_update
> > > 36737 <...>-6266 [002] 242.543142: try_atomic_semop <-update_queue
> > > 36738 <...>-6266 [002] 242.543143: update_queue <-do_smart_update
> > > 36739 <...>-6266 [002] 242.543143: try_atomic_semop <-update_queue
> > > 36740 <...>-6266 [002] 242.543144: get_seconds <-do_smart_update
> > > 36741 <...>-6266 [002] 242.543144: rt_spin_unlock <-sys_semtimedop
> > > 36742 <...>-6266 [002] 242.543144: migrate_enable <-sys_semtimedop
> > > 36743 <...>-6266 [002] 242.543145: migrate_enable: migrate_enable: in_atomic:1 p->migrate_disable_atomic:0 p->migrate_disable:1
> >
> > Hm. Seems this is home grown a non-preemptive wakeup in the making.
>
> Does the below cure things? It breaks !rt builds, but we can cure that if it works..
Warning is gone, ship it ;-)
[ 216.115993] BUG: soft lockup - CPU#1 stuck for 23s! [lmsched:6247]
[ 216.115996] Modules linked in: snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device edd nfsd lockd nfs_acl auth_rpcgss sunrpc parport_pc parport bridge ipv6 stp cpufreq_conservative cpufreq_ondemand cpufreq_userspace cpufreq_powersave microcode acpi_cpufreq mperf nls_iso8859_1 nls_cp437 vfat fat fuse ext3 jbd dm_mod snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep kvm_intel usb_storage snd_pcm snd_timer kvm sr_mod cdrom usb_libusual uas sg e1000e snd firewire_ohci firewire_core i2c_i801 soundcore snd_page_alloc crc_itu_t button ext4 mbcache jbd2 crc16 usbhid hid uhci_hcd sd_mod ehci_hcd usbcore rtc_cmos ahci libahci libata scsi_mod fan processor thermal
[ 216.116000] CPU 1
[ 216.116000] Modules linked in: snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device edd nfsd lockd nfs_acl auth_rpcgss sunrpc parport_pc parport bridge ipv6 stp cpufreq_conservative cpufreq_ondemand cpufreq_userspace cpufreq_powersave microcode acpi_cpufreq mperf nls_iso8859_1 nls_cp437 vfat fat fuse ext3 jbd dm_mod snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep kvm_intel usb_storage snd_pcm snd_timer kvm sr_mod cdrom usb_libusual uas sg e1000e snd firewire_ohci firewire_core i2c_i801 soundcore snd_page_alloc crc_itu_t button ext4 mbcache jbd2 crc16 usbhid hid uhci_hcd sd_mod ehci_hcd usbcore rtc_cmos ahci libahci libata scsi_mod fan processor thermal
[ 216.116000]
[ 216.116000] Pid: 6247, comm: lmsched Not tainted 3.0.4-rt13 #2049 MEDIONPC MS-7502/MS-7502
[ 216.116000] RIP: 0010:[<ffffffff81359b62>] [<ffffffff81359b62>] _raw_spin_lock+0x22/0x30
[ 216.116000] RSP: 0018:ffff8801f1a97b98 EFLAGS: 00000293
[ 216.116000] RAX: 0000000000006900 RBX: ffffffff81036ec0 RCX: 0000000000000001
[ 216.116000] RDX: ffff8801ed3d2300 RSI: 0000000000000282 RDI: ffff8801f1a97f70
[ 216.116000] RBP: ffff8801f1a97b98 R08: ffff88022fcdf128 R09: 0000000000000040
[ 216.116000] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff8135b02e
[ 216.116000] R13: 0000000000000001 R14: 0000000000000001 R15: ffff8801ed3d2300
[ 216.116000] FS: 00007f5989dde720(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000
[ 216.116000] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 216.116000] CR2: 00007f0f32fc2000 CR3: 0000000222212000 CR4: 00000000000006e0
[ 216.116000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 216.116000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 216.116000] Process lmsched (pid: 6247, threadinfo ffff8801f1a96000, task ffff8801ed3d2300)
[ 216.116000] Stack:
[ 216.116000] ffff8801f1a97c48 ffffffff81359353 ffff88022693a540 ffff88022fcdf040
[ 216.116000] ffff88022693a540 0000000000000005 ffff88022fcdf040 ffff88022693a540
[ 216.116000] ffff8801f1a97bf8 ffffffff81036024 0000000000000001 ffff88022693a540
[ 216.116000] Call Trace:
[ 216.116000] [<ffffffff81359353>] rt_spin_lock_slowlock+0x33/0x1b0
[ 216.116000] [<ffffffff81036024>] ? check_preempt_curr+0x84/0xa0
[ 216.116000] [<ffffffff8103d84f>] ? ttwu_do_wakeup+0x5f/0x130
[ 216.116000] [<ffffffff81359836>] rt_spin_lock+0x26/0x30
[ 216.116000] [<ffffffff8103eea9>] __wake_up+0x39/0x70
[ 216.116000] [<ffffffff81174d70>] wake_up_sem_queue_do+0x60/0x70
[ 216.116000] [<ffffffff8117611d>] sys_semtimedop+0x36d/0xb40
[ 216.116000] [<ffffffff8106dfdc>] ? __hrtimer_start_range_ns+0x15c/0x2c0
[ 216.116000] [<ffffffff810964fb>] ? rcu_read_unlock_special+0x1bb/0x1f0
[ 216.116000] [<ffffffff8103db42>] ? migrate_enable+0x192/0x280
[ 216.116000] [<ffffffff8103dd8d>] ? migrate_disable+0x15d/0x1f0
[ 216.116000] [<ffffffff8103db42>] ? migrate_enable+0x192/0x280
[ 216.116000] [<ffffffff8105c0eb>] ? do_sigtimedwait+0xab/0x1d0
[ 216.116000] [<ffffffff8105c299>] ? sys_rt_sigtimedwait+0x89/0xe0
[ 216.116000] [<ffffffff81068005>] ? sys_timer_settime+0x185/0x220
[ 216.116000] [<ffffffff81176a00>] copy_semundo+0xf0/0x100
[ 216.116000] [<ffffffff8135a72b>] system_call_fastpath+0x16/0x1b
[ 216.116000] Code: 00 00 00 00 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 65 48 8b 04 25 c8 95 00 00 ff 80 44 e0 ff ff b8 00 01 00 00 f0 66 0f c1 07
[ 216.116000] e0 74 06 f3 90 8a 07 eb f6 c9 c3 66 90 55 48 89 e5 66 66 66
[ 216.116000] Call Trace:
[ 216.116000] [<ffffffff81359353>] rt_spin_lock_slowlock+0x33/0x1b0
[ 216.116000] [<ffffffff81036024>] ? check_preempt_curr+0x84/0xa0
[ 216.116000] [<ffffffff8103d84f>] ? ttwu_do_wakeup+0x5f/0x130
[ 216.116000] [<ffffffff81359836>] rt_spin_lock+0x26/0x30
[ 216.116000] [<ffffffff8103eea9>] __wake_up+0x39/0x70
[ 216.116000] [<ffffffff81174d70>] wake_up_sem_queue_do+0x60/0x70
[ 216.116000] [<ffffffff8117611d>] sys_semtimedop+0x36d/0xb40
[ 216.116000] [<ffffffff8106dfdc>] ? __hrtimer_start_range_ns+0x15c/0x2c0
[ 216.116000] [<ffffffff810964fb>] ? rcu_read_unlock_special+0x1bb/0x1f0
[ 216.116000] [<ffffffff8103db42>] ? migrate_enable+0x192/0x280
[ 216.116000] [<ffffffff8103dd8d>] ? migrate_disable+0x15d/0x1f0
[ 216.116000] [<ffffffff8103db42>] ? migrate_enable+0x192/0x280
[ 216.116000] [<ffffffff8105c0eb>] ? do_sigtimedwait+0xab/0x1d0
[ 216.116000] [<ffffffff8105c299>] ? sys_rt_sigtimedwait+0x89/0xe0
[ 216.116000] [<ffffffff81068005>] ? sys_timer_settime+0x185/0x220
[ 216.116000] [<ffffffff81176a00>] copy_semundo+0xf0/0x100
[ 216.116000] [<ffffffff8135a72b>] system_call_fastpath+0x16/0x1b
[ 216.116000] Kernel panic - not syncing: softlockup: hung tasks
[ 216.116000] Pid: 6247, comm: lmsched Not tainted 3.0.4-rt13 #2049
[ 216.116000] Call Trace:
[ 216.116000] <IRQ> [<ffffffff81357080>] panic+0xa0/0x1a8
[ 216.116000] [<ffffffff8108fd63>] watchdog_timer_fn+0x183/0x190
[ 216.116000] [<ffffffff8106d683>] __run_hrtimer+0x73/0x240
[ 216.116000] [<ffffffff8108fbe0>] ? __touch_watchdog+0x30/0x30
[ 216.116000] [<ffffffff8106e2f4>] hrtimer_interrupt+0x174/0x340
[ 216.116000] [<ffffffff8135bb29>] smp_apic_timer_interrupt+0x69/0x99
[ 216.116000] [<ffffffff8135b033>] apic_timer_interrupt+0x13/0x20
[ 216.116000] <EOI> [<ffffffff81036ec0>] ? update_curr_rt+0x180/0x230
[ 216.116000] [<ffffffff81359b62>] ? _raw_spin_lock+0x22/0x30
[ 216.116000] [<ffffffff81038bf0>] ? enqueue_task_rt+0x120/0x2e0
[ 216.116000] [<ffffffff81359353>] rt_spin_lock_slowlock+0x33/0x1b0
[ 216.116000] [<ffffffff81036024>] ? check_preempt_curr+0x84/0xa0
[ 216.116000] [<ffffffff8103d84f>] ? ttwu_do_wakeup+0x5f/0x130
[ 216.116000] [<ffffffff81359836>] rt_spin_lock+0x26/0x30
[ 216.116000] [<ffffffff8103eea9>] __wake_up+0x39/0x70
[ 216.116000] [<ffffffff81174d70>] wake_up_sem_queue_do+0x60/0x70
[ 216.116000] [<ffffffff8117611d>] sys_semtimedop+0x36d/0xb40
[ 216.116000] [<ffffffff8106dfdc>] ? __hrtimer_start_range_ns+0x15c/0x2c0
[ 216.116000] [<ffffffff810964fb>] ? rcu_read_unlock_special+0x1bb/0x1f0
[ 216.116000] [<ffffffff8103db42>] ? migrate_enable+0x192/0x280
[ 216.116000] [<ffffffff8103dd8d>] ? migrate_disable+0x15d/0x1f0
[ 216.116000] [<ffffffff8103db42>] ? migrate_enable+0x192/0x280
[ 216.116000] [<ffffffff8105c0eb>] ? do_sigtimedwait+0xab/0x1d0
[ 216.116000] [<ffffffff8105c299>] ? sys_rt_sigtimedwait+0x89/0xe0
[ 216.116000] [<ffffffff81068005>] ? sys_timer_settime+0x185/0x220
[ 216.116000] [<ffffffff81176a00>] copy_semundo+0xf0/0x100
[ 216.116000] [<ffffffff8135a72b>] system_call_fastpath+0x16/0x1b
next prev parent reply other threads:[~2011-09-13 15:17 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-09-10 9:12 [ANNOUNCE] 3.0.4-rt13 Thomas Gleixner
2011-09-10 14:53 ` Madovsky
2011-09-10 17:27 ` Rolando Martins
2011-09-11 10:35 ` Mike Galbraith
2011-09-11 10:35 ` Mike Galbraith
2011-09-11 17:01 ` Mike Galbraith
2011-09-12 7:24 ` Thomas Gleixner
2011-09-12 8:59 ` Peter Zijlstra
2011-09-12 9:05 ` Mike Galbraith
2011-09-12 13:52 ` Mike Galbraith
2011-09-12 14:53 ` Mike Galbraith
2011-09-13 13:36 ` Peter Zijlstra
2011-09-13 15:17 ` Mike Galbraith [this message]
2011-09-13 15:08 ` Peter Zijlstra
2011-09-13 15:28 ` Mike Galbraith
2011-09-13 16:13 ` Peter Zijlstra
2011-09-21 10:17 ` rt14: strace -> migrate_disable_atomic imbalance Mike Galbraith
2011-09-21 17:01 ` Peter Zijlstra
2011-09-21 18:50 ` Peter Zijlstra
2011-09-21 18:50 ` Peter Zijlstra
2011-09-22 4:46 ` Mike Galbraith
2011-09-22 6:31 ` Peter Zijlstra
2011-09-22 8:38 ` Peter Zijlstra
2011-09-22 10:00 ` Peter Zijlstra
2011-09-22 10:00 ` Peter Zijlstra
2011-09-22 11:55 ` Mike Galbraith
2011-09-22 12:09 ` Peter Zijlstra
2011-09-22 13:42 ` Mike Galbraith
2011-09-22 14:05 ` Mike Galbraith
2011-09-22 15:20 ` Peter Zijlstra
2011-09-22 14:34 ` Peter Zijlstra
2011-09-22 14:38 ` Mike Galbraith
2011-09-22 14:41 ` Mike Galbraith
2011-09-22 14:41 ` Peter Zijlstra
2011-09-22 14:46 ` Mike Galbraith
2011-09-22 14:46 ` Mike Galbraith
2011-09-22 11:31 ` Peter Zijlstra
2011-09-22 11:46 ` Peter Zijlstra
2011-09-22 11:46 ` Peter Zijlstra
2011-09-22 14:52 ` Oleg Nesterov
2011-09-22 15:13 ` Peter Zijlstra
2011-09-14 9:57 ` [PATCH -rt] ipc/sem: Rework semaphore wakeups Peter Zijlstra
2011-09-14 13:02 ` Mike Galbraith
2011-09-14 18:48 ` Manfred Spraul
2011-09-14 19:23 ` Peter Zijlstra
2011-09-15 17:04 ` Manfred Spraul
2011-09-12 10:04 ` [ANNOUNCE] 3.0.4-rt13 Peter Zijlstra
2011-09-12 11:33 ` Mike Galbraith
2011-09-11 18:14 ` Mike Galbraith
2011-09-12 7:33 ` Thomas Gleixner
2011-09-12 8:05 ` Mike Galbraith
2011-09-12 8:43 ` Mike Galbraith
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1315927057.6445.2.camel@marge.simson.net \
--to=efault@gmx.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.