* Random freeze (Re: mmotm 2008-11-19-02-19 uploaded)
@ 2008-11-20 6:09 Tetsuo Handa
2008-11-20 23:05 ` Valdis.Kletnieks
0 siblings, 1 reply; 7+ messages in thread
From: Tetsuo Handa @ 2008-11-20 6:09 UTC (permalink / raw)
To: linux-kernel; +Cc: akpm
Hello.
> The mm-of-the-moment snapshot 2008-11-19-02-19 has been uploaded to
Recent mmotm randomly freezes on /sbin/modprobe and read(). 2.6.28-rc2-mm1 was OK.
Config is at http://I-love.SAKURA.ne.jp/tmp/config-2.6.28-rc5-mm1
----- Quoting from http://I-love.SAKURA.ne.jp/tmp/messages4.txt (2008-11-19-02-19) -----
kjournald starting. Commit interval 5 seconds
EXT3-fs warning: checktime reached, running e2fsck is recommended
EXT3 FS on sdb1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting. Commit interval 5 seconds
EXT3 FS on sdc1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
/dev/sdb1 on /usr/src/vanilla type ext3 (rw,noatime,nodiratime)
/dev/sdc1 on /usr/src/distro type ext3 (rw,noatime,nodiratime)
Cleaning /tmp /var/run /var/lock.
Detecting hardware: open_sock(): No such device
agpgart pcnet32 piix BusLogic
Skipping unavailable/built-in agpgart module.
pcnet32 disabled in configuration.
Skipping unavailable/built-in piix module.
Skipping already loaded module BusLogic.
Running 0dns-down to make sure resolv.conf is ok...done.
Setting up networking...done.
Starting hotplug subsystem:
pci
ignoring pci display device 00:0f.0
BusLogic: already loaded
SysRq : Changing Loglevel
Loglevel set to 8
CPA self-test:
4k 227326 large 0 gb 0 x 227326[c0000000-f77fd000] miss 0
4k 227326 large 0 gb 0 x 227326[c0000000-f77fd000] miss 0
4k 227326 large 0 gb 0 x 227326[c0000000-f77fd000] miss 0
ok.
SysRq : Show Blocked State
task PC stack pid father
kjournald D f6a43a40 5944 807 2
f6359e3c 00000046 c0207ff6 f6a43a40 c21a4740 f6a229c0 00201488 00000000
f6688a00 00000000 f6358000 c21a4740 f65e4108 f65e3ea0 f6a43a40 000047bc
c1f0525c f6359e34 c023e48b c21a4740 c21a4740 00000000 c1f0525c f6359e48
Call Trace:
[<c0207ff6>] ? read_tsc+0x9/0x26
[<c023e48b>] ? ktime_get_ts+0x43/0x47
[<c05b5a3e>] io_schedule+0x47/0x79
[<c02ba64e>] sync_buffer+0x37/0x3b
[<c05b5c0d>] __wait_on_bit+0x30/0x56
[<c02ba617>] ? sync_buffer+0x0/0x3b
[<c05b5c9a>] out_of_line_wait_on_bit+0x67/0x6f
[<c02ba617>] ? sync_buffer+0x0/0x3b
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c02ba6a9>] __wait_on_buffer+0x19/0x1c
[<c02f6a14>] journal_commit_transaction+0x6e4/0xbf4
[<c0247f75>] ? lock_release_holdtime+0x91/0x97
[<c024ab0b>] ? trace_hardirqs_on_caller+0x120/0x148
[<c024ab3e>] ? trace_hardirqs_on+0xb/0xd
[<c0233247>] ? try_to_del_timer_sync+0x54/0x5c
[<c02f8972>] kjournald+0xb8/0x229
[<c023c3c0>] ? autoremove_wake_function+0x0/0x33
[<c023c3c0>] ? autoremove_wake_function+0x0/0x33
[<c02252d2>] ? complete+0x36/0x3e
[<c02f88ba>] ? kjournald+0x0/0x229
[<c023bf0a>] kthread+0x37/0x59
[<c023bed3>] ? kthread+0x0/0x59
[<c0203d23>] kernel_thread_helper+0x7/0x10
modprobe D f6a43ca0 6104 1403 1401
f6765d48 00000046 c0207ff6 f6a43ca0 c2059740 c079cba0 00201488 00000000
f6652500 00000000 f6764000 c2059740 f6a3eac8 f6a3e860 f6a43ca0 00004792
c1f0e2bc f6765d40 c023e48b c2059740 c2059740 00000000 c1f0e2bc f6765d54
Call Trace:
[<c0207ff6>] ? read_tsc+0x9/0x26
[<c023e48b>] ? ktime_get_ts+0x43/0x47
[<c05b5a3e>] io_schedule+0x47/0x79
[<c0279c46>] sync_page+0x45/0x49
[<c05b5cd2>] __wait_on_bit_lock+0x30/0x57
[<c0279c01>] ? sync_page+0x0/0x49
[<c027a2f6>] __lock_page+0x62/0x6a
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c027a4a4>] ? find_get_page+0xcf/0xd9
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c027a4d9>] find_lock_page+0x2b/0x50
[<c027b1ab>] filemap_fault+0x86/0x29d
[<c028ab8b>] ? __do_fault+0x2df/0x325
[<c028a8fc>] __do_fault+0x50/0x325
[<c024bd04>] ? __lock_acquire+0x72a/0x77c
[<c028abfd>] do_linear_fault+0x2c/0x33
[<c028ad66>] handle_mm_fault+0xf9/0x1d9
[<c05b9586>] do_page_fault+0x1b6/0x427
[<c05b93d0>] ? do_page_fault+0x0/0x427
[<c05b7b17>] error_code+0x77/0x7c
[<c033e62a>] ? __copy_from_user_ll+0x35/0x4c
[<c033e787>] copy_from_user+0x31/0x56
[<c0253bec>] load_module+0x72/0x9d7
[<c0254577>] ? sys_init_module+0x26/0x18c
[<c03181c0>] ? security_capable+0x12/0x15
[<c0254592>] sys_init_module+0x41/0x18c
[<c0202fae>] syscall_call+0x7/0xb
Sched Debug Version: v0.07, 2.6.28-rc5-mm1 #1
now at 31572.891772 msecs
.sysctl_sched_latency : 40.000000
.sysctl_sched_min_granularity : 8.000000
.sysctl_sched_wakeup_granularity : 10.000000
.sysctl_sched_child_runs_first : 0.000001
.sysctl_sched_features : 24191
----- Quoting from http://I-love.SAKURA.ne.jp/tmp/messages5.txt (2008-11-19-02-19 + TOMOYO)-----
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
umount: devfs: not mounted
Calling /sbin/tomoyo-init to load policy. Please wait.
SysRq : Changing Loglevel
Loglevel set to 8
SysRq : Show Blocked State
task PC stack pid father
chroot D 0000004d 5904 1 0
f704fe38 00000046 00000000 0000004d c21a4740 f6a229c0 7d2e883b 0000000c
f65ee500 00000000 f704e000 c21a4740 f7050268 f7050000 c0247f75 f70504f0
00000063 00000000 00000063 00000001 7fffffff f704fea0 f704fec0 f704fe78
Call Trace:
[<c0247f75>] ? lock_release_holdtime+0x91/0x97
[<c05ba9dc>] schedule_timeout+0x16/0x92
[<c024a9ae>] ? mark_held_locks+0x4d/0x6c
[<c05bc739>] ? _spin_unlock_irq+0x22/0x26
[<c024ab0b>] ? trace_hardirqs_on_caller+0x120/0x148
[<c024ab3e>] ? trace_hardirqs_on+0xb/0xd
[<c05ba699>] wait_for_common+0xcd/0x121
[<c022518f>] ? default_wake_function+0x0/0xd
[<c022518f>] ? default_wake_function+0x0/0xd
[<c0239034>] ? queue_work_on+0x35/0x41
[<c05ba6ff>] wait_for_completion+0x12/0x14
[<c0238ed5>] call_usermodehelper_exec+0x76/0x8f
[<c05ba5fc>] ? wait_for_common+0x30/0x121
[<c031c1cf>] tomoyo_load_policy+0x9a/0x100
[<c031cca0>] tomoyo_bprm_set_creds+0x23/0x35
[<c03184a7>] security_bprm_set_creds+0xc/0xe
[<c02a4d47>] prepare_binprm+0xac/0xe1
[<c02a5180>] do_execve+0x10f/0x232
[<c0201f61>] sys_execve+0x29/0x4c
[<c0202fae>] syscall_call+0x7/0xb
cat D f6b463b8 6124 823 815
f6741da8 00000046 c0207ff6 f6b463b8 c2059740 c07a3ba0 00201487 00000000
f6726500 00000000 f6740000 c2059740 f62a1748 f62a14e0 f6b463b8 000021f2
c1f0edbc f6741da0 c023e48b c2059740 c2059740 00000000 c1f0edbc f6741db4
Call Trace:
[<c0207ff6>] ? read_tsc+0x9/0x26
[<c023e48b>] ? ktime_get_ts+0x43/0x47
[<c05ba8fe>] io_schedule+0x47/0x79
[<c0279c46>] sync_page+0x45/0x49
[<c0279c52>] sync_page_killable+0x8/0x30
[<c05bab92>] __wait_on_bit_lock+0x30/0x57
[<c0279c4a>] ? sync_page_killable+0x0/0x30
[<c027a363>] __lock_page_killable+0x65/0x6d
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c027a4a4>] ? find_get_page+0xcf/0xd9
[<c023c3f3>] ? wake_bit_function+0x0/0x42
[<c027ab96>] do_generic_file_read+0x244/0x3c6
[<c027afa3>] generic_file_aio_read+0x132/0x15a
[<c027ad18>] ? file_read_actor+0x0/0xe8
[<c02a04d2>] do_sync_read+0xb7/0xf5
[<c020777a>] ? native_sched_clock+0x42/0x92
[<c023c3c0>] ? autoremove_wake_function+0x0/0x33
[<c024c6d7>] ? __lock_release+0x4b/0x51
[<c02d4b47>] ? dnotify_parent+0x64/0x6d
[<c0318b51>] ? security_file_permission+0xf/0x11
[<c02a03cc>] ? rw_verify_area+0x96/0xb2
[<c02a059c>] vfs_read+0x8c/0x12e
[<c02a089f>] sys_read+0x3b/0x60
[<c0202fae>] syscall_call+0x7/0xb
Sched Debug Version: v0.07, 2.6.28-rc5-mm1 #2
now at 25613.058324 msecs
.sysctl_sched_latency : 40.000000
.sysctl_sched_min_granularity : 8.000000
.sysctl_sched_wakeup_granularity : 10.000000
.sysctl_sched_child_runs_first : 0.000001
.sysctl_sched_features : 24191
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random freeze (Re: mmotm 2008-11-19-02-19 uploaded)
2008-11-20 6:09 Random freeze (Re: mmotm 2008-11-19-02-19 uploaded) Tetsuo Handa
@ 2008-11-20 23:05 ` Valdis.Kletnieks
2008-11-20 23:20 ` Andrew Morton
0 siblings, 1 reply; 7+ messages in thread
From: Valdis.Kletnieks @ 2008-11-20 23:05 UTC (permalink / raw)
To: Tetsuo Handa; +Cc: linux-kernel, akpm
[-- Attachment #1: Type: text/plain, Size: 28887 bytes --]
On Thu, 20 Nov 2008 15:09:35 +0900, Tetsuo Handa said:
> Hello.
>
> > The mm-of-the-moment snapshot 2008-11-19-02-19 has been uploaded to
> Recent mmotm randomly freezes on /sbin/modprobe and read(). 2.6.28-rc2-mm1 was OK.
I'm seeing very similar hangs on -mmotm-11-17 as well. I've hit it 3 times
today, all while disk activity was moderately heavy (things like 'yum update',
or a 'find . | xargs grep', and so on).
Managed to catch one while netconsole was active - I didn't have any messages
for bugs/warns/oopsen. Apparently, somebody is holding a lock. (I also have an
alt-sysrq-t from this incident, but that's about 10 times as big, didn't want
to abuse vger too much.. ;)
[ 3932.912494] SysRq : Show Blocked State
[ 3932.913465] task PC stack pid father
[ 3932.913465] pdflush D ffff88007e247cf0 5776 303 2
[ 3932.913465] ffff88007e247c50 0000000000000002 ffff88007e247bb0 ffff88007dcf54d8
[ 3932.913465] ffff88007e247c00 ffffffff8081c780 ffffffff8081c780 ffff88007f269040
[ 3932.913465] ffff88007f232040 ffff88007f269398 000000007e247be0 ffff88007f269398
[ 3932.913465] Call Trace:
[ 3932.913465] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3932.913465] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3932.913465] [<ffffffff8027e44d>] sync_page+0x78/0x7f
[ 3932.913465] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
[ 3932.913465] [<ffffffff8027e3d5>] ? sync_page+0x0/0x7f
[ 3932.913465] [<ffffffff8027e5ed>] wait_on_page_bit+0x6e/0x75
[ 3932.913465] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3932.913465] [<ffffffff80286343>] ? pagevec_lookup_tag+0x22/0x2b
[ 3932.913465] [<ffffffff8027eed9>] wait_on_page_writeback_range+0x75/0x13d
[ 3932.913465] [<ffffffff8027efc1>] filemap_fdatawait+0x20/0x22
[ 3932.913465] [<ffffffff8027f0cc>] filemap_write_and_wait+0x27/0x33
[ 3932.913465] [<ffffffff802c7c32>] sync_blockdev+0x1b/0x1d
[ 3932.913465] [<ffffffff802c2242>] __sync_inodes+0x74/0xbf
[ 3932.913465] [<ffffffff802c22a6>] sync_inodes+0x19/0x33
[ 3932.913465] [<ffffffff802c5378>] do_sync+0x1a/0x77
[ 3932.913465] [<ffffffff80285a4c>] pdflush+0x145/0x1f8
[ 3932.913465] [<ffffffff802c535e>] ? do_sync+0x0/0x77
[ 3932.913465] [<ffffffff80285907>] ? pdflush+0x0/0x1f8
[ 3932.913465] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3932.913465] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3932.913465] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3932.913465] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3932.913465] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3932.913465] pdflush D 0000000000000246 5056 304 2
[ 3932.913465] ffff88007e275d50 0000000000000002 ffffe200016b1698 ffffe20001612f48
[ 3932.913465] ffffe20000d66ea8 ffffffff8081c780 ffffffff8081c780 ffff88007f25c040
[ 3932.913465] ffff88007f2697f0 ffff88007f25c398 000000007e275cf0 ffff88007f25c398
[ 3932.913465] Call Trace:
[ 3932.913465] [<ffffffff8022f968>] ? account_group_exec_runtime+0x26/0x64
[ 3932.913465] [<ffffffff8022fa79>] ? update_curr+0xd3/0xdc
[ 3932.913465] [<ffffffff8056883e>] __mutex_lock_common+0x280/0x406
[ 3932.913465] [<ffffffff80569f44>] ? _spin_unlock_irq+0x47/0x52
[ 3932.913465] [<ffffffff80568a56>] __mutex_lock_slowpath+0x14/0x16
[ 3932.913465] [<ffffffff80568a7d>] mutex_lock+0x25/0x29
[ 3932.913465] [<ffffffff802a8172>] lock_super+0x27/0x29
[ 3932.913465] [<ffffffff802a85db>] sync_supers+0x48/0xb6
[ 3932.913465] [<ffffffff80284ebe>] wb_kupdate+0x35/0x128
[ 3932.913465] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3932.913465] [<ffffffff80285a4c>] pdflush+0x145/0x1f8
[ 3932.913465] [<ffffffff80284e89>] ? wb_kupdate+0x0/0x128
[ 3932.913465] [<ffffffff80285907>] ? pdflush+0x0/0x1f8
[ 3932.913465] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3932.913465] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3932.913465] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3932.913465] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3932.913465] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3932.913465] kjournald D ffff88007e08c480 5696 1508 2
[ 3932.913465] ffff88007dc47c60 0000000000000002 ffff88007dcf54d8 ffff88007dcf54d8
[ 3932.913465] ffff88007dc47bd0 ffffffff8081c780 ffffffff8081c780 ffff88007e1a2040
[ 3932.913465] ffff88007906d040 ffff88007e1a2398 000000017dc47c00 ffff88007e1a2398
[ 3932.913465] Call Trace:
[ 3932.913465] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3932.913465] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3932.913465] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3932.913465] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
[ 3932.913465] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3932.913465] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3932.913465] [<ffffffff8056841c>] out_of_line_wait_on_bit+0x6a/0x77
[ 3932.913465] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3932.913465] [<ffffffff803636fb>] ? submit_bio+0xc4/0xcd
[ 3932.913465] [<ffffffff802c7dc0>] __wait_on_buffer+0x36/0x3a
[ 3932.913465] [<ffffffff802c7e01>] wait_on_buffer+0x3d/0x41
[ 3932.913465] [<ffffffff802c8477>] sync_dirty_buffer+0x84/0xd3
[ 3932.913465] [<ffffffff80315357>] journal_commit_transaction+0xd22/0x11cb
[ 3932.913465] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3932.913465] [<ffffffff80241178>] ? try_to_del_timer_sync+0x5e/0x6a
[ 3932.913465] [<ffffffff80318cf4>] kjournald+0xe2/0x23c
[ 3932.913465] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff80318c12>] ? kjournald+0x0/0x23c
[ 3933.879285] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3933.879285] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3933.879285] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3933.879285] [<ffffffff805680f2>] ? preempt_schedule+0x43/0x81
[ 3933.879285] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3933.879285] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3933.879285] kjournald D ffff880077c51d10 5264 1509 2
[ 3933.879285] ffff880077c51c80 0000000000000002 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff880077c51bf0 ffffffff8081c780 ffffffff8081c780 ffff88007e2dc7f0
[ 3933.879285] ffff88007f30a7f0 ffff88007e2dcb48 0000000177c51c20 ffff88007e2dcb48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056841c>] out_of_line_wait_on_bit+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff803636fb>] ? submit_bio+0xc4/0xcd
[ 3933.879285] [<ffffffff802c7dc0>] __wait_on_buffer+0x36/0x3a
[ 3933.879285] [<ffffffff80314536>] wait_on_buffer+0x3d/0x41
[ 3933.879285] [<ffffffff80314c27>] journal_commit_transaction+0x5f2/0x11cb
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff80241178>] ? try_to_del_timer_sync+0x5e/0x6a
[ 3933.879285] [<ffffffff80318cf4>] kjournald+0xe2/0x23c
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff80318c12>] ? kjournald+0x0/0x23c
[ 3933.879285] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3933.879285] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3933.879285] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3933.879285] [<ffffffff805680f2>] ? preempt_schedule+0x43/0x81
[ 3933.879285] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3933.879285] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3933.879285] kjournald D ffff88007e41b080 5264 1511 2
[ 3933.879285] ffff8800770adc80 0000000000000002 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff8800770adbf0 ffffffff8081c780 ffffffff8081c780 ffff88007e1a4040
[ 3933.879285] ffff88007d5897f0 ffff88007e1a4398 00000000770adc20 ffff88007e1a4398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056841c>] out_of_line_wait_on_bit+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff803636fb>] ? submit_bio+0xc4/0xcd
[ 3933.879285] [<ffffffff802c7dc0>] __wait_on_buffer+0x36/0x3a
[ 3933.879285] [<ffffffff80314536>] wait_on_buffer+0x3d/0x41
[ 3933.879285] [<ffffffff80315125>] journal_commit_transaction+0xaf0/0x11cb
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff80241178>] ? try_to_del_timer_sync+0x5e/0x6a
[ 3933.879285] [<ffffffff80318cf4>] kjournald+0xe2/0x23c
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff80318c12>] ? kjournald+0x0/0x23c
[ 3933.879285] [<ffffffff8024ba10>] kthread+0x49/0x76
[ 3933.879285] [<ffffffff8020cb79>] child_rip+0xa/0x11
[ 3933.879285] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
[ 3933.879285] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
[ 3933.879285] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
[ 3933.879285] syslog-ng D 00000000ffffffff 4616 2021 1
[ 3933.879285] ffff88007e67fae8 0000000000000046 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff88007e67fa58 ffffffff8081c780 ffffffff8081c780 ffff88007e0d57f0
[ 3933.879285] ffffffff807093b0 ffff88007e0d5b48 000000007e67fa88 ffff88007e0d5b48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056832c>] out_of_line_wait_on_bit_lock+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff802c7001>] ? __find_get_block+0x1c7/0x1d9
[ 3933.879285] [<ffffffff802c8364>] __lock_buffer+0x25/0x27
[ 3933.879285] [<ffffffff8031283d>] lock_buffer+0x44/0x48
[ 3933.879285] [<ffffffff80313511>] do_get_write_access+0x72/0x4c7
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80317a12>] ? bit_spin_unlock+0x6b/0x79
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80302ca4>] ext3_reserve_inode_write+0x3f/0x76
[ 3933.879285] [<ffffffff80302d11>] ext3_mark_inode_dirty+0x36/0x53
[ 3933.879285] [<ffffffff80302e7c>] ext3_dirty_inode+0x71/0x88
[ 3933.879285] [<ffffffff802c23e7>] __mark_inode_dirty+0x33/0x1a4
[ 3933.879285] [<ffffffff802ba2f0>] inode_setattr+0x11b/0x125
[ 3933.879285] [<ffffffff8030302d>] ext3_setattr+0x19a/0x201
[ 3933.879285] [<ffffffff802ba510>] notify_change+0x216/0x35e
[ 3933.879285] [<ffffffff802a52dc>] chown_common+0x69/0x80
[ 3933.879285] [<ffffffff802a525f>] ? audit_inode+0x43/0x57
[ 3933.879285] [<ffffffff802a5342>] sys_fchown+0x4f/0x6f
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] console-kit-d D ffff88007e08c480 4952 2109 1
[ 3933.879285] ffff88007d505a78 0000000000000046 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff88007d5059e8 ffffffff8081c780 ffffffff8081c780 ffff88007e1a37f0
[ 3933.879285] ffff88007906d040 ffff88007e1a3b48 000000007d505a18 ffff88007e1a3b48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056832c>] out_of_line_wait_on_bit_lock+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff802c667f>] ? put_bh+0x49/0x56
[ 3933.879285] [<ffffffff802c8364>] __lock_buffer+0x25/0x27
[ 3933.879285] [<ffffffff8031283d>] lock_buffer+0x44/0x48
[ 3933.879285] [<ffffffff80313511>] do_get_write_access+0x72/0x4c7
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80317a12>] ? bit_spin_unlock+0x6b/0x79
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80301676>] ext3_new_inode+0x4d9/0x9fd
[ 3933.879285] [<ffffffff803088a4>] ext3_create+0x99/0x11b
[ 3933.879285] [<ffffffff802b0cbf>] vfs_create+0x79/0xa0
[ 3933.879285] [<ffffffff802b10c6>] do_filp_open+0x27c/0x806
[ 3933.879285] [<ffffffff802aa156>] ? cp_new_stat+0xe2/0xef
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] e16 D 00000000ffffffff 3752 2699 2640
[ 3933.879285] ffff880077173b88 0000000000000046 ffff88007f4a2da8 0000000000000001
[ 3933.879285] ffff880077173b08 ffffffff8081c780 ffffffff8081c780 ffff880077d6a040
[ 3933.879285] ffffffff807093b0 ffff880077d6a398 0000000077173b38 ffff880077d6a398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff8031371b>] do_get_write_access+0x27c/0x4c7
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff8030177d>] ext3_new_inode+0x5e0/0x9fd
[ 3933.879285] [<ffffffff803088a4>] ext3_create+0x99/0x11b
[ 3933.879285] [<ffffffff802b0cbf>] vfs_create+0x79/0xa0
[ 3933.879285] [<ffffffff802b10c6>] do_filp_open+0x27c/0x806
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff8024f26a>] ? ktime_get_ts+0x59/0x5e
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] festival_serv D 00000000ffffffff 4920 2918 2699
[ 3933.879285] ffff880077dddb68 0000000000000046 000000030006dac8 0000000000000001
[ 3933.879285] ffff880077dddad8 ffffffff8081c780 ffffffff8081c780 ffff88007f2617f0
[ 3933.879285] ffff88007f2327f0 ffff88007f261b48 0000000100000000 ffff88007f261b48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8034092d>] ? avc_has_perm+0x4e/0x60
[ 3933.879285] [<ffffffff8056883e>] __mutex_lock_common+0x280/0x406
[ 3933.879285] [<ffffffff80314062>] ? start_this_handle+0x3b3/0x3d0
[ 3933.879285] [<ffffffff80568a56>] __mutex_lock_slowpath+0x14/0x16
[ 3933.879285] [<ffffffff80568a7d>] mutex_lock+0x25/0x29
[ 3933.879285] [<ffffffff802a8172>] lock_super+0x27/0x29
[ 3933.879285] [<ffffffff80308d65>] ext3_orphan_add+0x28/0x17d
[ 3933.879285] [<ffffffff8031423e>] ? journal_start+0xb7/0xef
[ 3933.879285] [<ffffffff80302ff5>] ext3_setattr+0x162/0x201
[ 3933.879285] [<ffffffff802ba510>] notify_change+0x216/0x35e
[ 3933.879285] [<ffffffff802a5ae0>] do_truncate+0x63/0x81
[ 3933.879285] [<ffffffff802ae39d>] ? vfs_dq_init+0x5e/0x68
[ 3933.879285] [<ffffffff802b0c0b>] may_open+0x19f/0x1da
[ 3933.879285] [<ffffffff802b1259>] do_filp_open+0x40f/0x806
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] firefox-bin D 00000000ffffffff 4888 4737 4680
[ 3933.879285] ffff880079c09668 0000000000000046 0000000000000000 0002005079c095e8
[ 3933.879285] ffff880079c095d8 ffffffff8081c780 ffffffff8081c780 ffff88007afc4040
[ 3933.879285] ffffffff807093b0 ffff88007afc4398 0000000079c09618 ffff88007afc4398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff8031371b>] do_get_write_access+0x27c/0x4c7
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80300063>] ext3_new_blocks+0x33f/0x5b9
[ 3933.879285] [<ffffffff805689b3>] ? __mutex_lock_common+0x3f5/0x406
[ 3933.879285] [<ffffffff803044fc>] ext3_get_blocks_handle+0x3bc/0x891
[ 3933.879285] [<ffffffff8036867f>] ? blk_rq_map_sg+0x1cf/0x2ed
[ 3933.879285] [<ffffffff80367cdb>] ? blk_recalc_rq_segments+0x36/0x1a3
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff80304a94>] ext3_get_block+0xc3/0x101
[ 3933.879285] [<ffffffff802c8be5>] __block_prepare_write+0x21b/0x3ca
[ 3933.879285] [<ffffffff803049d1>] ? ext3_get_block+0x0/0x101
[ 3933.879285] [<ffffffff802c8f06>] block_write_begin+0x9e/0x102
[ 3933.879285] [<ffffffff80303f96>] ext3_write_begin+0xeb/0x1e1
[ 3933.879285] [<ffffffff803049d1>] ? ext3_get_block+0x0/0x101
[ 3933.879285] [<ffffffff8027d13e>] ? iov_iter_fault_in_readable+0x3d/0x83
[ 3933.879285] [<ffffffff8027f21b>] generic_file_buffered_write+0x143/0x35b
[ 3933.879285] [<ffffffff803817a1>] ? _raw_spin_lock+0xcf/0x18a
[ 3933.879285] [<ffffffff8027f860>] __generic_file_aio_write_nolock+0x274/0x2a8
[ 3933.879285] [<ffffffff802b8ab7>] ? file_update_time+0xc6/0x101
[ 3933.879285] [<ffffffff802800f9>] generic_file_aio_write+0x78/0xd4
[ 3933.879285] [<ffffffff80300cd4>] ext3_file_write+0x1e/0x9e
[ 3933.879285] [<ffffffff802a6714>] do_sync_write+0xe7/0x12d
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff8033c97f>] ? security_file_permission+0x11/0x13
[ 3933.879285] [<ffffffff802a696b>] ? rw_verify_area+0xe4/0x10c
[ 3933.879285] [<ffffffff802a7123>] vfs_write+0xd9/0x137
[ 3933.879285] [<ffffffff802a7242>] sys_write+0x47/0x6c
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] yum-updatesd- D 00000000ffffffff 4536 6218 2539
[ 3933.879285] ffff880079e23bb8 0000000000000046 ffff880079e23b18 ffff88007dcf54d8
[ 3933.879285] ffff880079e23b68 ffffffff8081c780 ffffffff8081c780 ffff8800784f97f0
[ 3933.879285] ffffffff807093b0 ffff8800784f9b48 0000000079e23b48 ffff8800784f9b48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff8027e44d>] sync_page+0x78/0x7f
[ 3933.879285] [<ffffffff8027e462>] sync_page_killable+0xe/0x56
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff8027e454>] ? sync_page_killable+0x0/0x56
[ 3933.879285] [<ffffffff8027e2d2>] __lock_page_killable+0x63/0x6a
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff8027e1ec>] ? find_get_page+0xe8/0x101
[ 3933.879285] [<ffffffff8027e31f>] lock_page_killable+0x46/0x4e
[ 3933.879285] [<ffffffff8027fc77>] generic_file_aio_read+0x3e3/0x605
[ 3933.879285] [<ffffffff80257b2e>] ? debug_mutex_free_waiter+0x2a/0x64
[ 3933.879285] [<ffffffff802a6841>] do_sync_read+0xe7/0x12d
[ 3933.879285] [<ffffffff80376b5e>] ? _atomic_dec_and_lock+0x5a/0x98
[ 3933.879285] [<ffffffff8024be3f>] ? autoremove_wake_function+0x0/0x38
[ 3933.879285] [<ffffffff8033c97f>] ? security_file_permission+0x11/0x13
[ 3933.879285] [<ffffffff802a696b>] ? rw_verify_area+0xe4/0x10c
[ 3933.879285] [<ffffffff802a733d>] vfs_read+0xd6/0x133
[ 3933.879285] [<ffffffff802a73f6>] sys_pread64+0x5c/0x7a
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] GrabWeather D 00000000ffffffff 5624 6239 3250
[ 3933.879285] ffff880079e2fb28 0000000000000046 ffff88007f2e3cc0 00000000000006d0
[ 3933.879285] ffff880079e2fb38 ffffffff8081c780 ffffffff8081c780 ffff880079dbf040
[ 3933.879285] ffffffff807093b0 ffff880079dbf398 0000000079e2fad8 ffff880079dbf398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569fad>] ? _spin_unlock_irqrestore+0x5e/0x6c
[ 3933.879285] [<ffffffff8031371b>] do_get_write_access+0x27c/0x4c7
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80308dde>] ext3_orphan_add+0xa1/0x17d
[ 3933.879285] [<ffffffff8031423e>] ? journal_start+0xb7/0xef
[ 3933.879285] [<ffffffff80302ff5>] ext3_setattr+0x162/0x201
[ 3933.879285] [<ffffffff802ba510>] notify_change+0x216/0x35e
[ 3933.879285] [<ffffffff802a5ae0>] do_truncate+0x63/0x81
[ 3933.879285] [<ffffffff802ae39d>] ? vfs_dq_init+0x5e/0x68
[ 3933.879285] [<ffffffff802b0c0b>] may_open+0x19f/0x1da
[ 3933.879285] [<ffffffff802b1259>] do_filp_open+0x40f/0x806
[ 3933.879285] [<ffffffff802b5dc6>] ? __d_free+0x76/0x7b
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] refile D 00000000ffffffff 5992 6248 6247
[ 3933.879285] ffff880079d3bb68 0000000000000046 000000030006bac8 0000000000000001
[ 3933.879285] ffff880079d3bad8 ffffffff8081c780 ffffffff8081c780 ffff880079dbf7f0
[ 3933.879285] ffff88007f2327f0 ffff880079dbfb48 0000000100000000 ffff880079dbfb48
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff8034092d>] ? avc_has_perm+0x4e/0x60
[ 3933.879285] [<ffffffff8056883e>] __mutex_lock_common+0x280/0x406
[ 3933.879285] [<ffffffff80314062>] ? start_this_handle+0x3b3/0x3d0
[ 3933.879285] [<ffffffff80568a56>] __mutex_lock_slowpath+0x14/0x16
[ 3933.879285] [<ffffffff80568a7d>] mutex_lock+0x25/0x29
[ 3933.879285] [<ffffffff802a8172>] lock_super+0x27/0x29
[ 3933.879285] [<ffffffff80308d65>] ext3_orphan_add+0x28/0x17d
[ 3933.879285] [<ffffffff8031423e>] ? journal_start+0xb7/0xef
[ 3933.879285] [<ffffffff80302ff5>] ext3_setattr+0x162/0x201
[ 3933.879285] [<ffffffff802ba510>] notify_change+0x216/0x35e
[ 3933.879285] [<ffffffff802a5ae0>] do_truncate+0x63/0x81
[ 3933.879285] [<ffffffff802ae39d>] ? vfs_dq_init+0x5e/0x68
[ 3933.879285] [<ffffffff802b0c0b>] may_open+0x19f/0x1da
[ 3933.879285] [<ffffffff802b1259>] do_filp_open+0x40f/0x806
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] sendmail D ffff88007e755c00 5624 6266 2416
[ 3933.879285] ffff880079e5fa78 0000000000000046 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff880079e5f9e8 ffffffff8081c780 ffffffff8081c780 ffff880077bb9040
[ 3933.879285] ffff8800790a3040 ffff880077bb9398 0000000079e5fa18 ffff880077bb9398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056832c>] out_of_line_wait_on_bit_lock+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff802c8364>] __lock_buffer+0x25/0x27
[ 3933.879285] [<ffffffff8031283d>] lock_buffer+0x44/0x48
[ 3933.879285] [<ffffffff80313511>] do_get_write_access+0x72/0x4c7
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80317a12>] ? bit_spin_unlock+0x6b/0x79
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff8030177d>] ext3_new_inode+0x5e0/0x9fd
[ 3933.879285] [<ffffffff803088a4>] ext3_create+0x99/0x11b
[ 3933.879285] [<ffffffff802b0cbf>] vfs_create+0x79/0xa0
[ 3933.879285] [<ffffffff802b10c6>] do_filp_open+0x27c/0x806
[ 3933.879285] [<ffffffff802aa156>] ? cp_new_stat+0xe2/0xef
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[ 3933.879285] hal-acl-tool D 00000000ffffffff 5896 6295 2110
[ 3933.879285] ffff8800773fda78 0000000000000046 ffff88007dcf54d8 ffff88007dcf54d8
[ 3933.879285] ffff8800773fd9e8 ffffffff8081c780 ffffffff8081c780 ffff880079118040
[ 3933.879285] ffff88007f2327f0 ffff880079118398 00000001773fda18 ffff880079118398
[ 3933.879285] Call Trace:
[ 3933.879285] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
[ 3933.879285] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
[ 3933.879285] [<ffffffff802c7e91>] sync_buffer+0x3e/0x42
[ 3933.879285] [<ffffffff8056828e>] __wait_on_bit_lock+0x45/0x79
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff802c7e53>] ? sync_buffer+0x0/0x42
[ 3933.879285] [<ffffffff8056832c>] out_of_line_wait_on_bit_lock+0x6a/0x77
[ 3933.879285] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
[ 3933.879285] [<ffffffff802c667f>] ? put_bh+0x49/0x56
[ 3933.879285] [<ffffffff802c8364>] __lock_buffer+0x25/0x27
[ 3933.879285] [<ffffffff8031283d>] lock_buffer+0x44/0x48
[ 3933.879285] [<ffffffff80313511>] do_get_write_access+0x72/0x4c7
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80317a12>] ? bit_spin_unlock+0x6b/0x79
[ 3933.879285] [<ffffffff80313988>] journal_get_write_access+0x22/0x33
[ 3933.879285] [<ffffffff8030f77e>] __ext3_journal_get_write_access+0x1f/0x48
[ 3933.879285] [<ffffffff80301676>] ext3_new_inode+0x4d9/0x9fd
[ 3933.879285] [<ffffffff803088a4>] ext3_create+0x99/0x11b
[ 3933.879285] [<ffffffff802b0cbf>] vfs_create+0x79/0xa0
[ 3933.879285] [<ffffffff802b10c6>] do_filp_open+0x27c/0x806
[ 3933.879285] [<ffffffff8023a6ea>] ? do_wait+0x33c/0x464
[ 3933.879285] [<ffffffff8022f7fc>] ? get_parent_ip+0x11/0x41
[ 3933.879285] [<ffffffff8056cfc9>] ? sub_preempt_count+0x35/0x49
[ 3933.879285] [<ffffffff80569ffc>] ? _spin_unlock+0x41/0x4c
[ 3933.879285] [<ffffffff802bb067>] ? alloc_fd+0x11d/0x12e
[ 3933.879285] [<ffffffff802a4bf6>] do_sys_open+0x6b/0xff
[ 3933.879285] [<ffffffff802a4cb3>] sys_open+0x1b/0x1d
[ 3933.879285] [<ffffffff8020b9db>] system_call_fastpath+0x16/0x1b
[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random freeze (Re: mmotm 2008-11-19-02-19 uploaded)
2008-11-20 23:05 ` Valdis.Kletnieks
@ 2008-11-20 23:20 ` Andrew Morton
2008-11-21 5:23 ` Valdis.Kletnieks
2008-11-21 18:12 ` Valdis.Kletnieks
0 siblings, 2 replies; 7+ messages in thread
From: Andrew Morton @ 2008-11-20 23:20 UTC (permalink / raw)
To: Valdis.Kletnieks; +Cc: penguin-kernel, linux-kernel
On Thu, 20 Nov 2008 18:05:47 -0500
Valdis.Kletnieks@vt.edu wrote:
> On Thu, 20 Nov 2008 15:09:35 +0900, Tetsuo Handa said:
> > Hello.
> >
> > > The mm-of-the-moment snapshot 2008-11-19-02-19 has been uploaded to
> > Recent mmotm randomly freezes on /sbin/modprobe and read(). 2.6.28-rc2-mm1 was OK.
>
> I'm seeing very similar hangs on -mmotm-11-17 as well. I've hit it 3 times
> today, all while disk activity was moderately heavy (things like 'yum update',
> or a 'find . | xargs grep', and so on).
>
> Managed to catch one while netconsole was active - I didn't have any messages
> for bugs/warns/oopsen. Apparently, somebody is holding a lock. (I also have an
> alt-sysrq-t from this incident, but that's about 10 times as big, didn't want
> to abuse vger too much.. ;)
>
> [ 3932.912494] SysRq : Show Blocked State
> [ 3932.913465] task PC stack pid father
> [ 3932.913465] pdflush D ffff88007e247cf0 5776 303 2
> [ 3932.913465] ffff88007e247c50 0000000000000002 ffff88007e247bb0 ffff88007dcf54d8
> [ 3932.913465] ffff88007e247c00 ffffffff8081c780 ffffffff8081c780 ffff88007f269040
> [ 3932.913465] ffff88007f232040 ffff88007f269398 000000007e247be0 ffff88007f269398
> [ 3932.913465] Call Trace:
> [ 3932.913465] [<ffffffff80252a1a>] ? getnstimeofday+0x4a/0xa6
> [ 3932.913465] [<ffffffff80567f2e>] io_schedule+0x63/0xa5
> [ 3932.913465] [<ffffffff8027e44d>] sync_page+0x78/0x7f
> [ 3932.913465] [<ffffffff80568380>] __wait_on_bit+0x47/0x79
> [ 3932.913465] [<ffffffff8027e3d5>] ? sync_page+0x0/0x7f
> [ 3932.913465] [<ffffffff8027e5ed>] wait_on_page_bit+0x6e/0x75
> [ 3932.913465] [<ffffffff8024be77>] ? wake_bit_function+0x0/0x2a
> [ 3932.913465] [<ffffffff80286343>] ? pagevec_lookup_tag+0x22/0x2b
> [ 3932.913465] [<ffffffff8027eed9>] wait_on_page_writeback_range+0x75/0x13d
> [ 3932.913465] [<ffffffff8027efc1>] filemap_fdatawait+0x20/0x22
> [ 3932.913465] [<ffffffff8027f0cc>] filemap_write_and_wait+0x27/0x33
> [ 3932.913465] [<ffffffff802c7c32>] sync_blockdev+0x1b/0x1d
> [ 3932.913465] [<ffffffff802c2242>] __sync_inodes+0x74/0xbf
> [ 3932.913465] [<ffffffff802c22a6>] sync_inodes+0x19/0x33
> [ 3932.913465] [<ffffffff802c5378>] do_sync+0x1a/0x77
> [ 3932.913465] [<ffffffff80285a4c>] pdflush+0x145/0x1f8
> [ 3932.913465] [<ffffffff802c535e>] ? do_sync+0x0/0x77
> [ 3932.913465] [<ffffffff80285907>] ? pdflush+0x0/0x1f8
> [ 3932.913465] [<ffffffff8024ba10>] kthread+0x49/0x76
> [ 3932.913465] [<ffffffff8020cb79>] child_rip+0xa/0x11
> [ 3932.913465] [<ffffffff8020bfe5>] ? restore_args+0x0/0x30
> [ 3932.913465] [<ffffffff8024b9c7>] ? kthread+0x0/0x76
> [ 3932.913465] [<ffffffff8020cb6f>] ? child_rip+0x0/0x11
The traditional cause of the above trace is that someone mucked up the
block/driver/irq-routing layer and we lost an IO completion.
It's also of course possible (but less common) that someone mucked up
the VFS. It would be interesting to revert
do_mpage_readpage-dont-submit-lots-of-small-bios-on-boundary.patch.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random freeze (Re: mmotm 2008-11-19-02-19 uploaded)
2008-11-20 23:20 ` Andrew Morton
@ 2008-11-21 5:23 ` Valdis.Kletnieks
2008-11-21 6:54 ` Daisuke Nishimura
2008-11-21 18:12 ` Valdis.Kletnieks
1 sibling, 1 reply; 7+ messages in thread
From: Valdis.Kletnieks @ 2008-11-21 5:23 UTC (permalink / raw)
To: Andrew Morton; +Cc: penguin-kernel, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 807 bytes --]
On Thu, 20 Nov 2008 15:20:54 PST, Andrew Morton said:
> The traditional cause of the above trace is that someone mucked up the
> block/driver/irq-routing layer and we lost an IO completion.
Yes, that would explain all the symptoms and tracebacks - everybody comes
to a screeching halt the next time they try to go to disk, while the actual
disk drive is showing zero activity.
> It's also of course possible (but less common) that someone mucked up
> the VFS. It would be interesting to revert
> do_mpage_readpage-dont-submit-lots-of-small-bios-on-boundary.patch.
I'm seeing an MTBF of about 2-3 hours when actually applying an I/O load to the
system. I'll try reverting that patch, and if it survives an entire day or
two it will be pretty strong circumstantial evidence that patch is the culprit...
[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random freeze (Re: mmotm 2008-11-19-02-19 uploaded)
2008-11-21 5:23 ` Valdis.Kletnieks
@ 2008-11-21 6:54 ` Daisuke Nishimura
0 siblings, 0 replies; 7+ messages in thread
From: Daisuke Nishimura @ 2008-11-21 6:54 UTC (permalink / raw)
To: Valdis.Kletnieks; +Cc: nishimura, Andrew Morton, penguin-kernel, linux-kernel
Hi.
On Fri, 21 Nov 2008 00:23:18 -0500, Valdis.Kletnieks@vt.edu wrote:
> On Thu, 20 Nov 2008 15:20:54 PST, Andrew Morton said:
>
> > The traditional cause of the above trace is that someone mucked up the
> > block/driver/irq-routing layer and we lost an IO completion.
>
> Yes, that would explain all the symptoms and tracebacks - everybody comes
> to a screeching halt the next time they try to go to disk, while the actual
> disk drive is showing zero activity.
>
> > It's also of course possible (but less common) that someone mucked up
> > the VFS. It would be interesting to revert
> > do_mpage_readpage-dont-submit-lots-of-small-bios-on-boundary.patch.
>
> I'm seeing an MTBF of about 2-3 hours when actually applying an I/O load to the
> system. I'll try reverting that patch, and if it survives an entire day or
> two it will be pretty strong circumstantial evidence that patch is the culprit...
>
Just FYI, I had seen similar errors with recent mmotms,
but current mmotm(2008-11-20-17-03) seems more stable in my environment.
Thanks,
Daisuke Nishimura.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random freeze (Re: mmotm 2008-11-19-02-19 uploaded)
2008-11-20 23:20 ` Andrew Morton
2008-11-21 5:23 ` Valdis.Kletnieks
@ 2008-11-21 18:12 ` Valdis.Kletnieks
1 sibling, 0 replies; 7+ messages in thread
From: Valdis.Kletnieks @ 2008-11-21 18:12 UTC (permalink / raw)
To: Andrew Morton; +Cc: penguin-kernel, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 419 bytes --]
On Thu, 20 Nov 2008 15:20:54 PST, Andrew Morton said:
> It's also of course possible (but less common) that someone mucked up
> the VFS. It would be interesting to revert
> do_mpage_readpage-dont-submit-lots-of-small-bios-on-boundary.patch.
Reverted that one, managed to provoke a wedge-up this morning anyhow.
So it's something else.
I may have to go and find a reproducer good enough to make bisection plausible.
[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Random freeze (Re: mmotm 2008-11-19-02-19 uploaded)
@ 2008-11-25 2:32 Tetsuo Handa
0 siblings, 0 replies; 7+ messages in thread
From: Tetsuo Handa @ 2008-11-25 2:32 UTC (permalink / raw)
To: akpm; +Cc: linux-kernel
Hello.
mmotm 2008-11-24-16-52 booted fine. But a patch for below BUG is missing.
http://kerneltrap.org/mailarchive/linux-netdev/2008/11/22/4204804
----------
INIT: version 2.86 booting
BUG: using smp_processor_id() in preemptible [00000000] code: hotplug/1159
caller is sock_prot_inuse_add+0x1c/0x40
Pid: 1159, comm: hotplug Not tainted 2.6.28-rc6-mm1 #1
Call Trace:
[<c0200ba4>] debug_smp_processor_id+0xc4/0xe0
[<c027cb7c>] sock_prot_inuse_add+0x1c/0x40
[<f8047821>] unix_create1+0x101/0x120 [unix]
[<f8047875>] unix_create+0x35/0x70 [unix]
[<c0279077>] __sock_create+0xf7/0x250
[<c0279208>] sock_create+0x38/0x50
[<c0279298>] sys_socket+0x38/0x70
[<c027a1fd>] sys_socketcall+0x5d/0x270
[<c0191fec>] sys_getcwd+0x11c/0x150
[<c011af30>] do_page_fault+0x0/0x450
[<c01034e6>] syscall_call+0x7/0xb
BUG: using smp_processor_id() in preemptible [00000000] code: hotplug/1159
caller is sock_prot_inuse_add+0x1c/0x40
BUG: using smp_processor_id() in preemptible [00000000] code: hotplug/1156
caller is sock_prot_inuse_add+0x1c/0x40
Pid: 1156, comm: hotplug Not tainted 2.6.28-rc6-mm1 #1
Call Trace:
[<c0200ba4>] debug_smp_processor_id+0xc4/0xe0
[<c027cb7c>] sock_prot_inuse_add+0x1c/0x40
[<f8047821>] unix_create1+0x101/0x120 [unix]
[<f8047875>] unix_create+0x35/0x70 [unix]
[<c0279077>] __sock_create+0xf7/0x250
[<c0279208>] sock_create+0x38/0x50
[<c0279298>] sys_socket+0x38/0x70
[<c027a1fd>] sys_socketcall+0x5d/0x270
[<c0191fec>] sys_getcwd+0x11c/0x150
[<c011af30>] do_page_fault+0x0/0x450
[<c01034e6>] syscall_call+0x7/0xb
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2008-11-25 2:33 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-11-20 6:09 Random freeze (Re: mmotm 2008-11-19-02-19 uploaded) Tetsuo Handa
2008-11-20 23:05 ` Valdis.Kletnieks
2008-11-20 23:20 ` Andrew Morton
2008-11-21 5:23 ` Valdis.Kletnieks
2008-11-21 6:54 ` Daisuke Nishimura
2008-11-21 18:12 ` Valdis.Kletnieks
-- strict thread matches above, loose matches on Subject: below --
2008-11-25 2:32 Tetsuo Handa
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.