* Re: [Bugme-new] [Bug 35662] New: softlockup with kernel 2.6.39 [not found] <bug-35662-10286@https.bugzilla.kernel.org/> @ 2011-05-23 23:22 ` Andrew Morton 2011-05-23 23:41 ` Hussam Al-Tayeb 0 siblings, 1 reply; 6+ messages in thread From: Andrew Morton @ 2011-05-23 23:22 UTC (permalink / raw) To: linux-mm; +Cc: bugzilla-daemon, bugme-daemon, Hussam Al-Tayeb (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Mon, 23 May 2011 08:13:30 GMT bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=35662 > > Summary: softlockup with kernel 2.6.39 > Product: IO/Storage > Version: 2.5 > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: Other > AssignedTo: io_other@kernel-bugs.osdl.org > ReportedBy: ht990332@gmail.com > Regression: No I'll mark this as a regression. > > After upgrading to kernel 2.6.39, I started having soft lockups due to disk > activity. anything more that low disk activity would cause a problem in an > application A. > dmesg would spit out something like [ 1920.307498] INFO: task java:25665 > blocked for more than 120 seconds. > [ 1920.307499] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this > message. > [ 1920.307500] java D f036df98 0 25665 25393 0x00000000 > [ 1920.307503] f036dee0 00000086 c15899c8 f036df98 e1c09590 00000001 00cbde28 > 00000170 > [ 1920.307507] f036de98 00000064 f036de60 f036de60 f036de68 f036de68 e1c09590 > c14e1440 > [ 1920.307511] 081c6000 c14e1440 f5506440 e1c09590 e1c08450 00000000 ffffffff > c15899c8 > [ 1920.307515] Call Trace: > [ 1920.307518] [<c1073c8d>] ? get_futex_key+0x6d/0x1d0 > [ 1920.307520] [<c10742c5>] ? futex_wake+0xe5/0x100 > [ 1920.307522] [<c132fd65>] rwsem_down_failed_common+0x95/0xe0 > [ 1920.307525] [<c1027640>] ? vmalloc_sync_all+0x120/0x120 > [ 1920.307527] [<c132fde2>] rwsem_down_read_failed+0x12/0x14 > [ 1920.307529] [<c132fe1f>] call_rwsem_down_read_failed+0x7/0xc > [ 1920.307531] [<c132f69d>] ? down_read+0xd/0x10 > [ 1920.307534] [<c1027787>] do_page_fault+0x147/0x420 > [ 1920.307536] [<c10760e4>] ? sys_futex+0xc4/0x130 > [ 1920.307538] [<c1027640>] ? vmalloc_sync_all+0x120/0x120 > [ 1920.307540] [<c1330c4b>] error_code+0x67/0x6c > > Application A would then stop being able to read/write from disk. Other running > applications would still be able to read/write fine to the disk. > I could even copy the data application A to another folder or delete it. > This isn't a hard lockup and I could still continue to use the computer but > then it'll hang at shutdown. > At first I thought the disk (which I bought 12 days ago) is bad so I ran > badblocks -vs and didn't find a single bad block. I ran smartctl long test and > the disk is fine. It started to feel like some ext4 regression. > > I downgraded to kernel 2.6.38.6 and performed a disk intensive action which was > recompiling libreoffice. This worked without a problem. > I also tried the application A which was giving problems earlier but I couldn't > see a problem again. So I compiled libreoffice again to check and didn't have > lockups. > It appears that we forgot to release mmap_sem. Also, > all paritions apart from swap and /boot and encrypted with LuKs. This involves dm-crypt, correct? I see quite a lot of dm-crypt related bug reports float past. But there have been very few changes in dm-crypt recently, and something broke in 2.6.39... -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Bugme-new] [Bug 35662] New: softlockup with kernel 2.6.39 2011-05-23 23:22 ` [Bugme-new] [Bug 35662] New: softlockup with kernel 2.6.39 Andrew Morton @ 2011-05-23 23:41 ` Hussam Al-Tayeb 2011-05-23 23:48 ` Andrew Morton 0 siblings, 1 reply; 6+ messages in thread From: Hussam Al-Tayeb @ 2011-05-23 23:41 UTC (permalink / raw) To: Andrew Morton; +Cc: linux-mm, bugzilla-daemon, bugme-daemon YEs, lsmod | grep dm_crypt dm_crypt 12887 2 dm_mod 55464 5 dm_crypt Still no lockups since downgrading to 2.6.38.6 which was a few hours before I filed the bug report. I could reinstall 2.6.39 if there is anything you would like me to test. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Bugme-new] [Bug 35662] New: softlockup with kernel 2.6.39 2011-05-23 23:41 ` Hussam Al-Tayeb @ 2011-05-23 23:48 ` Andrew Morton 2011-05-24 7:01 ` Hussam Al-Tayeb 0 siblings, 1 reply; 6+ messages in thread From: Andrew Morton @ 2011-05-23 23:48 UTC (permalink / raw) To: Hussam Al-Tayeb; +Cc: linux-mm, bugzilla-daemon, bugme-daemon On Tue, 24 May 2011 02:41:51 +0300 Hussam Al-Tayeb <ht990332@gmail.com> wrote: > YEs, > lsmod | grep dm_crypt > dm_crypt 12887 2 > dm_mod 55464 5 dm_crypt > > Still no lockups since downgrading to 2.6.38.6 which was a few hours before I > filed the bug report. > I could reinstall 2.6.39 if there is anything you would like me to test. Thanks. CONFIG_PROVE_LOCKING=y would be useful, if you didn't already have it set. Also after the kernel has failed, kill everything off and do a `ps aux' and look for any other tasks which are stuck in "D" state. Backtraces for any such processes can be obtained with "echo w > /proc/sysrq-trigger". -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Bugme-new] [Bug 35662] New: softlockup with kernel 2.6.39 2011-05-23 23:48 ` Andrew Morton @ 2011-05-24 7:01 ` Hussam Al-Tayeb 2011-05-26 5:03 ` Hussam Al-Tayeb 2011-05-26 16:33 ` Hussam Al-Tayeb 0 siblings, 2 replies; 6+ messages in thread From: Hussam Al-Tayeb @ 2011-05-24 7:01 UTC (permalink / raw) To: Andrew Morton; +Cc: Hussam Al-Tayeb, linux-mm, bugzilla-daemon, bugme-daemon Actually it was definitely more than a few hours before filing the bug report. I did recompile libreoffice twice as a 'stress test' under kernel 2.6.38.6 I'm using kernel 2.6.39 now and waiting for the lockup to happen again. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Bugme-new] [Bug 35662] New: softlockup with kernel 2.6.39 2011-05-24 7:01 ` Hussam Al-Tayeb @ 2011-05-26 5:03 ` Hussam Al-Tayeb 2011-05-26 16:33 ` Hussam Al-Tayeb 1 sibling, 0 replies; 6+ messages in thread From: Hussam Al-Tayeb @ 2011-05-26 5:03 UTC (permalink / raw) To: Andrew Morton; +Cc: linux-mm, bugzilla-daemon, bugme-daemon It happened again. also while doing a test build of libreoffice (very disk intensive stuff). The build process seems frozen. I did cat /var/log/kernel.log and run /bin/dmesg but didn't find any error. so I typed init s then echo w > /proc/sysrq-trigger and then I typed ps aux | grep D root 23 0.0 0.0 0 0 ? DN May25 0:06 [khugepaged] root 8673 0.0 0.0 4412 832 pts/2 S+ 08:02 0:00 grep D then I typed exit and I found the following in my kernel.log 11/05/26 07:54:00 udevd[7360] starting version 170 11/05/26 07:54:01 leds_ss4200 no LED devices found 11/05/26 07:54:01 intel_rng Firmware space is locked read-only. If you can't or 11/05/26 07:54:01 intel_rng don't want to disable this in firmware setup, and if 11/05/26 07:54:01 intel_rng you are certain that your system has a functional 11/05/26 07:54:01 intel_rng RNG, try using the 'no_fwh_detect' option. 11/05/26 07:54:32 SysRq Show Blocked State 11/05/26 07:54:32 task PC stack pid father 11/05/26 07:54:32 khugepaged D f4393ca4 0 23 2 0x00000000 11/05/26 07:54:32 f4393cb4 00000046 00000002 f4393ca4 f4044dcc 00000000 c0a8204e 000033ba 11/05/26 07:54:32 00000000 e20eba00 f4393c5c c10026ce 00000000 0098c500 00000001 c14e1440 11/05/26 07:54:32 f4393c64 c14e1440 f5506440 f4044da0 f4042f70 e20eba00 f4393d0c 00000286 11/05/26 07:54:32 Call Trace 11/05/26 07:54:32 [<c10026ce>] ? __switch_to+0xce/0x180 11/05/26 07:54:32 [<c1052076>] ? lock_timer_base.isra.31+0x26/0x50 11/05/26 07:54:32 [<c1052106>] ? try_to_del_timer_sync+0x66/0x100 11/05/26 07:54:32 [<c1052076>] ? lock_timer_base.isra.31+0x26/0x50 11/05/26 07:54:32 [<c132e3ca>] schedule_timeout+0x12a/0x2e0 11/05/26 07:54:32 [<c1051630>] ? init_timer_deferrable_key+0x20/0x20 11/05/26 07:54:32 [<c132e231>] io_schedule_timeout+0x81/0xd0 11/05/26 07:54:32 [<c10d7ea2>] congestion_wait+0x52/0xe0 11/05/26 07:54:32 [<c1061750>] ? abort_exclusive_wait+0x80/0x80 11/05/26 07:54:32 [<c10f2501>] compact_zone+0x721/0x750 11/05/26 07:54:32 [<c10f25a9>] compact_zone_order+0x79/0xa0 11/05/26 07:54:32 [<c10f266d>] try_to_compact_pages+0x9d/0xd0 11/05/26 07:54:32 [<c10c6eee>] __alloc_pages_direct_compact+0x7e/0x160 11/05/26 07:54:32 [<c10c73ad>] __alloc_pages_nodemask+0x3dd/0x7b0 11/05/26 07:54:32 [<c10e5f7c>] ? page_add_new_anon_rmap+0x8c/0xa0 11/05/26 07:54:32 [<c10faea8>] khugepaged+0x418/0xe10 11/05/26 07:54:32 [<c1061750>] ? abort_exclusive_wait+0x80/0x80 11/05/26 07:54:32 [<c10faa90>] ? khugepaged_defrag_store+0x50/0x50 11/05/26 07:54:32 [<c1061098>] kthread+0x68/0x70 11/05/26 07:54:32 [<c1061030>] ? kthread_worker_fn+0x150/0x150 11/05/26 07:54:32 [<c133147e>] kernel_thread_helper+0x6/0xd 11/05/26 07:54:32 Sched Debug Version v0.10, 2.6.39-ARCH #1 11/05/26 07:54:32 ktime : 56907897.973146 11/05/26 07:54:32 sched_clk : 56892856.276187 11/05/26 07:54:32 cpu_clk : 56907897.973092 11/05/26 07:54:32 jiffies : 16982371 11/05/26 07:54:32 sched_clock_stable : 0 11/05/26 07:54:32 11/05/26 07:54:32 sysctl_sched 11/05/26 07:54:32 .sysctl_sched_latency : 12.000000 11/05/26 07:54:32 .sysctl_sched_min_granularity : 1.500000 11/05/26 07:54:32 .sysctl_sched_wakeup_granularity : 2.000000 11/05/26 07:54:32 .sysctl_sched_child_runs_first : 0 11/05/26 07:54:32 .sysctl_sched_features : 7279 11/05/26 07:54:32 .sysctl_sched_tunable_scaling : 1 (logaritmic) 11/05/26 07:54:32 11/05/26 07:54:32 cpu#0, 2933.889 MHz 11/05/26 07:54:32 .nr_running : 1 11/05/26 07:54:32 .load : 1024 11/05/26 07:54:32 .nr_switches : 66131673 11/05/26 07:54:32 .nr_load_updates : 4725405 11/05/26 07:54:32 .nr_uninterruptible : 0 11/05/26 07:54:32 .next_balance : 16.982350 11/05/26 07:54:32 .curr->pid : 7949 11/05/26 07:54:32 .clock : 56907897.673964 11/05/26 07:54:32 .cpu_load[0] : 0 11/05/26 07:54:32 .cpu_load[1] : 0 11/05/26 07:54:32 .cpu_load[2] : 0 11/05/26 07:54:32 .cpu_load[3] : 0 11/05/26 07:54:32 .cpu_load[4] : 0 11/05/26 07:54:32 11/05/26 07:54:32 cfs_rq[0] autogroup-105 11/05/26 07:54:32 .exec_clock : 0.000000 11/05/26 07:54:32 .MIN_vruntime : 0.000001 11/05/26 07:54:32 .min_vruntime : 19.599979 11/05/26 07:54:32 .max_vruntime : 0.000001 11/05/26 07:54:32 .spread : 0.000000 11/05/26 07:54:32 .spread0 : -13633115.574964 11/05/26 07:54:32 .nr_spread_over : 0 11/05/26 07:54:32 .nr_running : 1 11/05/26 07:54:32 .load : 1024 11/05/26 07:54:32 .load_avg : 0.000000 11/05/26 07:54:32 .load_period : 9.999999 11/05/26 07:54:32 .load_contrib : 0 11/05/26 07:54:32 .load_tg : 0 11/05/26 07:54:32 .se->exec_start : 56907897.673964 11/05/26 07:54:32 .se->vruntime : 13633135.174943 11/05/26 07:54:32 .se->sum_exec_runtime : 19.279330 11/05/26 07:54:32 .se->load.weight : 1024 11/05/26 07:54:32 11/05/26 07:54:32 cfs_rq[0] 11/05/26 07:54:32 .exec_clock : 0.000000 11/05/26 07:54:32 .MIN_vruntime : 0.000001 11/05/26 07:54:32 .min_vruntime : 13633135.174943 11/05/26 07:54:32 .max_vruntime : 0.000001 11/05/26 07:54:32 .spread : 0.000000 11/05/26 07:54:32 .spread0 : 0.000000 11/05/26 07:54:32 .nr_spread_over : 0 11/05/26 07:54:32 .nr_running : 1 11/05/26 07:54:32 .load : 1024 11/05/26 07:54:32 .load_avg : 0.000000 11/05/26 07:54:32 .load_period : 0.000000 11/05/26 07:54:32 .load_contrib : 0 11/05/26 07:54:32 .load_tg : 0 11/05/26 07:54:32 11/05/26 07:54:32 runnable tasks 11/05/26 07:54:32 task PID tree-key switches prio exec-runtime sum-exec sum-sleep 11/05/26 07:54:32 ---------------------------------------------------------------------------------------------------------- 11/05/26 07:54:32 R bash 7949 19.599979 63 120 0 0 0.000000 0.000000 0.000000 /autogroup-105 11/05/26 07:54:32 11/05/26 07:54:32 cpu#1, 2933.889 MHz 11/05/26 07:54:32 .nr_running : 0 11/05/26 07:54:32 .load : 0 11/05/26 07:54:32 .nr_switches : 66462726 11/05/26 07:54:32 .nr_load_updates : 4571341 11/05/26 07:54:32 .nr_uninterruptible : 1 11/05/26 07:54:32 .next_balance : 16.982370 11/05/26 07:54:32 .curr->pid : 0 11/05/26 07:54:32 .clock : 56907897.675594 11/05/26 07:54:32 .cpu_load[0] : 0 11/05/26 07:54:32 .cpu_load[1] : 0 11/05/26 07:54:32 .cpu_load[2] : 0 11/05/26 07:54:32 .cpu_load[3] : 0 11/05/26 07:54:32 .cpu_load[4] : 5 11/05/26 07:54:32 11/05/26 07:54:32 cfs_rq[1] 11/05/26 07:54:32 .exec_clock : 0.000000 11/05/26 07:54:32 .MIN_vruntime : 0.000001 11/05/26 07:54:32 .min_vruntime : 13650531.404239 11/05/26 07:54:32 .max_vruntime : 0.000001 11/05/26 07:54:32 .spread : 0.000000 11/05/26 07:54:32 .spread0 : 17396.229296 11/05/26 07:54:32 .nr_spread_over : 0 11/05/26 07:54:32 .nr_running : 0 11/05/26 07:54:32 .load : 0 11/05/26 07:54:32 .load_avg : 0.000000 11/05/26 07:54:32 .load_period : 0.000000 11/05/26 07:54:32 .load_contrib : 0 11/05/26 07:54:32 .load_tg : 0 11/05/26 07:54:32 11/05/26 07:54:32 runnable tasks 11/05/26 07:54:32 task PID tree-key switches prio exec-runtime sum-exec sum-sleep 11/05/26 07:54:32 ---------------------------------------------------------------------------------------------------------- 11/05/26 07:54:32 11/05/26 07:55:01 usb 3-2 USB disconnect, device number 2 11/05/26 07:55:02 ppp0 Features changed: 0x00006800 -> 0x00006000 11/05/26 07:55:02 usb 3-2 new low speed USB device number 3 using uhci_hcd 11/05/26 07:55:03 input PIXART USB OPTICAL MOUSE as /devices/pci0000:00/0000:00:1d.1/usb3/3-2/3-2:1.0/input/input9 11/05/26 07:55:03 generic-usb 0003 93A:2510.0002: input,hidraw0: USB HID v1.11 Mouse [PIXART USB OPTICAL MOUSE] on usb-0000:00:1d.1-2/input0 11/05/26 07:55:05 0000 2:05.0: tulip_stop_rxtx() failed (CSR5 0xfc664010 CSR6 0xff972113) 11/05/26 07:55:05 net eth0 Setting full-duplex based on MII#1 link partner capability of 45e1 11/05/26 07:55:05 w83627ehf Found W83627DHG chip at 0x290 11/05/26 07:55:12 eth0 no IPv6 routers present 11/05/26 07:55:19 EXT4-fs (dm-0) re-mounted. Opts: user_xattr,commit=0 11/05/26 07:55:19 EXT4-fs (sda1) re-mounted. Opts: user_xattr,commit=0 11/05/26 07:55:20 EXT4-fs (dm-1) re-mounted. Opts: user_xattr,commit=0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Bugme-new] [Bug 35662] New: softlockup with kernel 2.6.39 2011-05-24 7:01 ` Hussam Al-Tayeb 2011-05-26 5:03 ` Hussam Al-Tayeb @ 2011-05-26 16:33 ` Hussam Al-Tayeb 1 sibling, 0 replies; 6+ messages in thread From: Hussam Al-Tayeb @ 2011-05-26 16:33 UTC (permalink / raw) To: Hussam Al-Tayeb; +Cc: Andrew Morton, linux-mm, bugzilla-daemon, bugme-daemon Another time. [ 8520.275830] INFO: task khugepaged:26 blocked for more than 120 seconds. [ 8520.275834] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 8520.275837] khugepaged D c1455200 0 26 2 0x00000000 [ 8520.275843] f3f9fedc 00000046 f59af000 c1455200 00000400 c1455200 c1455580 f3c451f0 [ 8520.275850] 0000000a f3f9fe88 c10c5078 c1455c80 00000003 f5506380 c14c4380 f3c451f0 [ 8520.275857] f3c453b4 33c86581 0000069a c14c4380 f5506380 f3c451f0 c4525640 f3f9fee8 [ 8520.275864] Call Trace: [ 8520.275874] [<c10c5078>] ? __alloc_pages_direct_compact+0xe8/0x160 [ 8520.275880] [<c10ff3f8>] ? __mem_cgroup_try_charge+0x2d8/0x4e0 [ 8520.275885] [<c10fd138>] ? memcg_check_events+0x28/0x160 [ 8520.275891] [<c131b885>] rwsem_down_failed_common+0x95/0xe0 [ 8520.275895] [<c131b8e2>] rwsem_down_write_failed+0x12/0x20 [ 8520.275900] [<c131b94a>] call_rwsem_down_write_failed+0x6/0x8 [ 8520.275904] [<c131b0f5>] ? down_write+0x15/0x17 [ 8520.275908] [<c10f8c72>] khugepaged+0x552/0xdf0 [ 8520.275913] [<c10613a0>] ? autoremove_wake_function+0x0/0x40 [ 8520.275922] [<c10f8720>] ? khugepaged+0x0/0xdf0 [ 8520.275924] [<c1060d08>] kthread+0x68/0x70 [ 8520.275926] [<c1060ca0>] ? kthread+0x0/0x70 [ 8520.275929] [<c1003d7e>] kernel_thread_helper+0x6/0x18 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-05-26 16:34 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <bug-35662-10286@https.bugzilla.kernel.org/>
2011-05-23 23:22 ` [Bugme-new] [Bug 35662] New: softlockup with kernel 2.6.39 Andrew Morton
2011-05-23 23:41 ` Hussam Al-Tayeb
2011-05-23 23:48 ` Andrew Morton
2011-05-24 7:01 ` Hussam Al-Tayeb
2011-05-26 5:03 ` Hussam Al-Tayeb
2011-05-26 16:33 ` Hussam Al-Tayeb
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).