Re: Hangs

xming
2008-12-09T00:34:10Z

updated to kvm-80 and vanilla kernel 2.6.27.8.

This is still happening after a few hours on a idle host.

# uptime
01:19:43 up 8:02, 1 user, load average: 0.08, 0.03, 0.01
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 68 7516 4084 379072 0 0 41 15 27 83 0 0 99 0
1 0 68 7516 4084 379072 0 0 0 0 970 681963 0 0 100 0
1 0 68 7516 4084 379072 0 0 0 0 991 708242 0 0 100 0
0 0 68 7516 4084 379072 0 0 0 0 975 710206 0 0 100 0
0 0 68 7308 4084 379072 0 0 0 0 1026 674657 1 1 98 0
0 0 68 7308 4084 379072 0 0 0 0 1000 650320 0 0 100 0
1 0 68 7308 4084 379072 0 0 0 0 995 696319 0 0 100 0
1 0 68 7308 4084 379072 0 0 0 0 985 707663 0 0 100 0
1 0 68 7308 4084 379072 0 0 0 0 987 706937 0 0 100 0
0 0 68 7308 4084 379072 0 0 0 0 989 706222 0 0 100 0
1 0 68 7308 4084 379072 0 0 0 0 978 700453 0 0 100 0
^C
# uptime
01:19:43 up 8:02, 1 user, load average: 0.08, 0.03, 0.01

here are the outputs of sysrq-t, sysreq-m and sysrq-w

Dec 9 01:19:43 builder SysRq : Show State
Dec 9 01:19:43 builder task PC stack pid father
Dec 9 01:19:43 builder init S df82bb3c 0 1 0
Dec 9 01:19:43 builder df830000 00000086 00000002 df82bb3c df82bb44
00000000 df8310e0 df830154
Dec 9 01:19:43 builder df830154 c1413500 c03ce080 c03ce080 c03ce080
df82bb64 01b3bb70 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f df82bb64
01b48dd0 00000000 00001388
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02ef71b>] schedule_timeout+0x4b/0xd0
Dec 9 01:19:43 builder [<c013546d>] add_wait_queue+0x1d/0x50
Dec 9 01:19:43 builder [<c012b5d0>] process_timeout+0x0/0x10
Dec 9 01:19:43 builder [<c017d50e>] do_select+0x3ae/0x4a0
Dec 9 01:19:43 builder [<c017db60>] __pollwait+0x0/0x100
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c013a66d>] getnstimeofday+0x3d/0xe0
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c01e2f82>] __next_cpu+0x12/0x20
Dec 9 01:19:43 builder [<c01e2f82>] __next_cpu+0x12/0x20
Dec 9 01:19:43 builder [<c011b335>] find_busiest_group+0x195/0x560
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c01e8303>] number+0x2d3/0x2e0
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c01e8ba7>] vsnprintf+0x307/0x5e0
Dec 9 01:19:43 builder [<c02f0b73>] _spin_lock_irq+0x13/0x20
Dec 9 01:19:43 builder [<c012b426>] run_timer_softirq+0x166/0x1a0
Dec 9 01:19:43 builder [<c013ebba>] tick_program_event+0x2a/0x40
Dec 9 01:19:43 builder [<c0127118>] __do_softirq+0x88/0x100
Dec 9 01:19:43 builder [<c0183af0>] destroy_inode+0x20/0x40
Dec 9 01:19:43 builder [<c0183af0>] destroy_inode+0x20/0x40
Dec 9 01:19:43 builder [<c02f0a85>] _spin_lock+0x5/0x10
Dec 9 01:19:43 builder [<c0182103>] __d_lookup+0xe3/0x110
Dec 9 01:19:43 builder [<c0183af0>] destroy_inode+0x20/0x40
Dec 9 01:19:43 builder [<c018177c>] dput+0x1c/0x120
Dec 9 01:19:43 builder [<c017d7c8>] core_sys_select+0x1c8/0x2f0
Dec 9 01:19:43 builder [<c017a097>] __link_path_walk+0xa27/0xba0
Dec 9 01:19:43 builder [<c01a60e9>] proc_flush_task+0x159/0x280
Dec 9 01:19:43 builder [<c0186de5>] mntput_no_expire+0x15/0xe0
Dec 9 01:19:43 builder [<c017a25f>] path_walk+0x4f/0x90
Dec 9 01:19:43 builder [<c017a35a>] do_path_lookup+0x6a/0x120
Dec 9 01:19:43 builder [<c0179513>] getname+0xb3/0xe0
Dec 9 01:19:43 builder [<c0173c98>] cp_new_stat64+0xf8/0x110
Dec 9 01:19:43 builder [<c017dd42>] sys_select+0xe2/0x1b0
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder kthreadd S df839fa8 0 2 0
Dec 9 01:19:43 builder df830360 00000046 00000002 df839fa8 df839fb0
00000000 deddde60 df8304b4
Dec 9 01:19:43 builder df8304b4 c1409500 c03ce080 c03ce080 c03ce080
dedddeb0 fffc4930 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f c03751b4
00001010 deddde8c c03eb684
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c0135029>] kthreadd+0x169/0x170
Dec 9 01:19:43 builder [<c0134ec0>] kthreadd+0x0/0x170
Dec 9 01:19:43 builder [<c0104fc7>] kernel_thread_helper+0x7/0x10
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder migration/0 S df83bf90 0 3 2
Dec 9 01:19:43 builder df8306c0 00000046 00000002 df83bf90 df83bf98
00000000 d219bf2c df830814
Dec 9 01:19:43 builder df830814 c1409500 c03ce080 c03ce080 c03ce080
d219bf6c 01ab9906 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f c1409918
c140fdc0 c1409500 00000000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c011ee48>] migration_thread+0x158/0x250
Dec 9 01:19:43 builder [<c011ecf0>] migration_thread+0x0/0x250
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder [<c0134e50>] kthread+0x0/0x70
Dec 9 01:19:43 builder [<c0104fc7>] kernel_thread_helper+0x7/0x10
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder [<c01351b0>] autoremove_wake_function+0x0/0x50
Dec 9 01:19:43 builder [<c02443f0>] serio_thread+0x0/0x300
Dec 9 01:19:43 builder [<c0134e92>] /0x200
Dec 9 01:19:43 builder [<c0126ab3>] current_fs_time+0x13/0x20
Dec 9 01:19:43 builder [<c014dbc5>] __generic_file_aio_write_nolock+0x255/0x520
Dec 9 01:19:43 builder [<c02580aa>] verify_iovec+0x2a/0x90
Dec 9 01:19:43 builder [<c0250674>] sys_sendmsg+0x164/0x280
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c013a66d>] getnstimeofday+0x3d/0xe0
Dec 9 01:19:43 builder [<c010f75c>] lapic_next_event+0xc/0x10
Dec 9 01:19:43 builder [<c013d8d8>] clockevents_program_event+0xa8/0x120
Dec 9 01:19:43 builder [<c02f0b73>] _spin_lock_irq+0x13/0x20
Dec 9 01:19:43 builder [<c0251abb>] sys_socketcall+0x25b/0x2b0
Dec 9 01:19:43 builder [<c017d0a5>] sys_poll+0x35/0x80
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f 0000000a
00000000 00000000 ffffffff
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02ef747>] schedule_timeout+0x77/0xd0
Dec 9 01:19:43 builder [<c013546d>] add_wait_queue+0x1d/0x50
Dec 9 01:19:43 builder [<c02844c4>] tcp_poll+0x14/0x160
Dec 9 01:19:43 builder [<c017d50e>] do_select+0x3ae/0x4a0
Dec 9 01:19:43 builder [<c017db60>] __pollwait+0x0/0x100
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c027e08f>] ip_finish_output+0x14f/0x2d0
Dec 9 01:19:43 builder [<c027c9c8>] ip_cork_release+0x28/0x40
Dec 9 01:19:43 builder [<c027de64>] ip_push_pending_frames+0x274/0x350
Dec 9 01:19:43 builder [<c0298c97>] udp_push_pending_frames+0x167/0x3a0
Dec 9 01:19:43 builder [<c02f0b2f>] _spin_lock_bh+0xf/0x20
Dec 9 01:19:43 builder [<c0252783>] release_sock+0x13/0xa0
Dec 9 01:19:43 builder [<c029a0eb>] udp_sendmsg+0x34b/0x690
Dec 9 01:19:43 builder [<c0258057>] memcpy_toiovec+0x37/0x60
Dec 9 01:19:43 builder [<c02f0b2f>] _spin_lock_bh+0xf/0x20
Dec 9 01:19:43 builder [<c02528b5>] lock_sock_nested+0xa5/0xb0
Dec 9 01:19:43 builder [<c02f0b2f>] _spin_lock_bh+0xf/0x20
Dec 9 01:19:43 builder [<c0252783>] release_sock+0x13/0xa0
Dec 9 01:19:43 builder [<c029adb9>] udp_recvmsg+0x1a9/0x2e0
Dec 9 01:19:43 builder [<c02520f5>] sock_common_recvmsg+0x45/0x70
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c0192228>] __find_get_block+0x88/0x190
Dec 9 01:19:43 builder [<c011a19c>] resched_task+0x1c/0x60
Dec 9 01:19:43 builder dfb9a314 c1409500 c03ce080 c03ce080 c03ce080
00000000 00000082 df832880
Dec 9 01:19:43 builder dfb9a314 c02f0bc3 00000282 c0135360 dede3680
dede368c 00000000 00000000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c011da4d>] finish_task_switch+0x2d/0xb0
Dec 9 01:19:43 builder [<c02f0bc3>] _spin_lock_irqsave+0x23/0x30
Dec 9 01:19:43 builder [<c0135360>] prepare_to_wait+0x20/0x70
Dec 9 01:19:43 builder [<c0132655>] worker_thread+0x95/0xb0
Dec 9 01:19:43 builder [<c01351b0>] autoremove_wake_function+0x0/0x50
Dec 9 01:19:43 builder [<c01325c0>] worker_thread+0x0/0xb0
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder df89ed34 c1413500 c03ce080 c03ce080 c03ce080
c02f0b73 c0131ce7 00000000
Dec 9 01:19:43 builder 00000000 c02f0bc3 00000282 c0135360 dede3780
dede378c 00000000 00000000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<e08a668b>] rpc_wake_up+0xb/0x60 [sunrpc]
Dec 9 01:19:43 builder [<c02f0b73>] _spin_lock_irq+0x13/0x20
Dec 9 01:19:43 builder [<c0131ce7>] run_workqueue+0xb7/0x120
Dec 9 01:19:43 builder [<c02f0bc3>] _spin_lock_irqsave+0x23/0x30
Dec 9 01:19:43 builder [<c0135360>] prepare_to_wait+0x20/0x70
Dec 9 01:19:43 builder [<c0132655>] worker_thread+0x95/0xb0
Dec 9 01:19:43 builder rpc.mountd S dededb3c 0 4102 1
Dec 9 01:19:43 builder df89c6c0 00000086 00000002 dededb3c dededb44
00000000 df565000 df89c814
Dec 9 01:19:43 builder df89c814 c1413500 c03ce080 c03ce080 c03ce080
00000007 fffc4918 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f 00000008
00000000 00000000 ffffffff
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02ef747>] schedule_timeout+0x77/0xd0
Dec 9 01:19:43 builder [<c013546d>] add_wait_queue+0x1d/0x50
Dec 9 01:19:43 builder [<c02844c4>] tcp_poll+0x14/0x160
Dec 9 01:19:43 builder [<c017d50e>] do_select+0x3ae/0x4a0
Dec 9 01:19:43 builder [<c017db60>] __pollwait+0x0/0x100
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c01a3bc7>] proc_alloc_inode+0x47/0x70
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c014c245>] find_get_page+0x25/0xa0
Dec 9 01:19:43 builder [<c02f0a85>] _spin_lock+0x5/0x10
Dec 9 01:19:43 builder [<c0182103>] __d_lookup+0xe3/0x110
Dec 9 01:19:43 builder [<c0154e1b>] mark_page_accessed+0x2b/0x40
Dec 9 01:19:43 builder [<c018177c>] dput+0x1c/0x120
Dec 9 01:19:43 builder [<c01783b5>] do_lookup+0x65/0x1a0
Dec 9 01:19:43 builder [<c018177c>] dput+0x1c/0x120
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder lockd S deddbee0 0 4104 2
Dec 9 01:19:43 builder df89f600 00000046 00000002 deddbee0 deddbee8
00000000 c0119d9d df89f754
Dec 9 01:19:43 builder df89f754 c1413500 c03ce080 c03ce080 c03ce080
c1409500 fffc4918 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f df8edd90
deddbf50 ded9ce04 df8ed000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c0119d9d>] update_curr+0x4d/0x70
Dec 9 01:19:43 builder [<c02ef747>] schedule_timeout+0x77/0xd0
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c02f0bc3>] _spin_lock_irqsave+0x23/0x30
Dec 9 01:19:43 builder [<e0880797>] lockd+0x97/0x1a0 [lockd]
Dec 9 01:19:43 builder [<c02ef1bc>] schedule+0x21c/0x620
Dec 9 01:19:43 builder [<e0880700>] lockd+0x0/0x1a0 [lockd]
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder [<c0134e50>] kthread+0x0/0x70
Dec 9 01:19:43 builder [<c0104fc7>] kernel_thread_helper+0x7/0x10
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder nfsd S df833b64 0 4105 2
Dec 9 01:19:43 builder df89d7a0 00000046 df833960 df833b64 df89d7a0
c011da4d df833960 df89d8f4
Dec 9 01:19:43 builder df89d8f4 c1409500 c03ce080 c03ce080 c03ce080
deddff44 deddff44 00000282
Dec 9 01:19:43 builder [<e08b42af>] svc_recv+0x26f/0x770 [sunrpc]
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<e08ec7dc>] nfsd+0xbc/0x270 [nfsd]
Dec 9 01:19:43 builder [<c011a28d>] complete+0x3d/0x60
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder [<c0134e50>] kthread+0x0/0x70
Dec 9 01:19:43 builder [<c0104fc7>] kernel_thread_helper+0x7/0x10
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder [<c012b5d0>] process_timeout+0x0/0x10
Dec 9 01:19:43 builder [<e08b42af>] svc_recv+0x26f/0x770 [sunrpc]
Dec 9 01:19:43 builder [<c0119282>] enqueue_task+0x12/0x30
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<e08ec7dc>] nfsd+0xbc/0x270 [nfsd]
Dec 9 01:19:43 builder [<c011a28d>] complete+0x3d/0x60
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder [<c0134e50>] kthread+0x0/0x70
Dec 9 01:19:43 builder df832f40 00000046 df832f6c c0119d9d c140953c
df832f40 df8321c0 df833094
Dec 9 01:19:43 builder df833094 c1409500 c03ce080 c03ce080 c03ce080
ded2bf44 ded2bf44 00000282
Dec 9 01:19:43 builder c03ea300 c012bac7 00000000 00000282 ded2bf44
01eaabbc df8aff84 ded08000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c0119d9d>] update_curr+0x4d/0x70
Dec 9 01:19:43 builder [<c012bac7>] __mod_timer+0x97/0xb0
Dec 9 01:19:43 builder [<c02ef71b>] schedule_timeout+0x4b/0xd0
Dec 9 01:19:43 builder [<c012b5d0>] process_timeout+0x0/0x10
Dec 9 01:19:43 builder [<e08b42af>] svc_recv+0x26f/0x770 [sunrpc]
Dec 9 01:19:43 builder [<c0119282>] enqueue_task+0x12/0x30
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder nfsd S c0119d9d 0 4109 2
Dec 9 01:19:43 builder df832520 00000046 df83254c c0119d9d c140953c
df832520 df832f40 df832674
Dec 9 01:19:43 builder df832674 c1409500 c03ce080 c03ce080 c03ce080
ded2df44 ded2df44 00000282
Dec 9 01:19:43 builder c03ea300 c012bac7 00000000 00000282 ded2df44
01eaabbc df8aff84 ded09000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c0119d9d>] update_curr+0x4d/0x70
Dec 9 01:19:43 builder [<c012bac7>] __mod_timer+0x97/0xb0
Dec 9 01:19:43 builder [<c02ef71b>] schedule_timeout+0x4b/0xd0
Dec 9 01:19:43 builder [<c012b5d0>] process_timeout+0x0/0x10
Dec 9 01:19:43 builder [<e08b42af>] svc_recv+0x26f/0x770 [sunrpc]
Dec 9 01:19:43 builder [<c0119282>] enqueue_task+0x12/0x30
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<e08ec7dc>] nfsd+0xbc/0x270 [nfsd]
Dec 9 01:19:43 builder [<c011a28d>] complete+0x3d/0x60
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder [<c0134e50>] kthread+0x0/0x70
Dec 9 01:19:43 builder [<c0104fc7>] kernel_thread_helper+0x7/0x10
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder nfsd S c0119d9d 0 4110 2
Dec 9 01:19:43 builder df833600 00000046 df83362c c0119d9d c140953c
df833600 df832520 df833754
Dec 9 01:19:43 builder df833754 c1409500 c03ce080 c03ce080 c03ce080
dee8df44 dee8df44 00000282
Dec 9 01:19:43 builder c03ea300 c012bac7 00000000 00000282 dee8df44
01eaabbc df8aff84 ded0a000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c0119d9d>] update_curr+0x4d/0x70
Dec 9 01:19:43 builder [<c012bac7>] __mod_timer+0x97/0xb0
Dec 9 01:19:43 builder [<c02ef71b>] schedule_timeout+0x4b/0xd0
Dec 9 01:19:43 builder [<c012b5d0>] process_timeout+0x0/0x10
Dec 9 01:19:43 builder [<e08b42af>] svc_recv+0x26f/0x770 [sunrpc]
Dec 9 01:19:43 builder [<c0119282>] enqueue_task+0x12/0x30
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<e08ec7dc>] nfsd+0xbc/0x270 [nfsd]
Dec 9 01:19:43 builder [<c011a28d>] complete+0x3d/0x60
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder [<c0134e50>] kthread+0x0/0x70
Dec 9 01:19:43 builder [<c0104fc7>] kernel_thread_helper+0x7/0x10
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder nfsd S c0119d9d 0 4111 2
Dec 9 01:19:43 builder df8332a0 00000046 df8332cc c0119d9d c140953c
df8332a0 df833600 df8333f4
Dec 9 01:19:43 builder df8333f4 c1409500 c03ce080 c03ce080 c03ce080
dee8ff44 dee8ff44 00000282
Dec 9 01:19:43 builder c03ea300 c012bac7 00000000 00000282 dee8ff44
01eaabbc df8aff84 ded0b000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c0119d9d>] update_curr+0x4d/0x70
Dec 9 01:19:43 builder [<c012bac7>] __mod_timer+0x97/0xb0
Dec 9 01:19:43 builder [<c02ef71b>] schedule_timeout+0x4b/0xd0
Dec 9 01:19:43 builder [<c012b5d0>] process_timeout+0x0/0x10
Dec 9 01:19:43 builder [<e08b42af>] svc_recv+0x26f/0x770 [sunrpc]
Dec 9 01:19:43 builder [<c02ef1bc>] schedule+0x21c/0x620
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<e08ec7dc>] nfsd+0xbc/0x270 [nfsd]
Dec 9 01:19:43 builder [<c011a28d>] complete+0x3d/0x60
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder [<c0134e50>] kthread+0x0/0x70
Dec 9 01:19:43 builder [<c0104fc7>] kernel_thread_helper+0x7/0x10
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder nfsd S deeb9f1c 0 4112 2
Dec 9 01:19:43 builder df832be0 00000046 00000002 deeb9f1c deeb9f24
00000000 df89d7a0 df832d34
Dec 9 01:19:43 builder df832d34 c1409500 c03ce080 c03ce080 c03ce080
deeb9f44 deeb9f44 00000282
Dec 9 01:19:43 builder c03ea300 c012bac7 00000000 00000282 deeb9f44
01eaabbc df8aff84 ded0c000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c012bac7>] __mod_timer+0x97/0xb0
Dec 9 01:19:43 builder [<c02ef71b>] schedule_timeout+0x4b/0xd0
Dec 9 01:19:43 builder [<c012b5d0>] process_timeout+0x0/0x10
Dec 9 01:19:43 builder [<e08b42af>] svc_recv+0x26f/0x770 [sunrpc]
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<e08ec7dc>] nfsd+0xbc/0x270 [nfsd]
Dec 9 01:19:43 builder [<c011a28d>] complete+0x3d/0x60
Dec 9 01:19:43 builder [<e08ec720>] nfsd+0x0/0x270 [nfsd]
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder [<c0134e50>] kthread+0x0/0x70
Dec 9 01:19:43 builder [<c0104fc7>] kernel_thread_helper+0x7/0x10
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder smbd S deeabb3c 0 4216 1
Dec 9 01:19:43 builder dfb98000 00000086 00000002 deeabb3c deeabb44
00000000 dfb9af40 dfb98154
Dec 9 01:19:43 builder dfb98154 c1409500 c03ce080 c03ce080 c03ce080
00000000 01b47a4c 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f 00000015
00000000 00000000 ffffffff
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02ef747>] schedule_timeout+0x77/0xd0
Dec 9 01:19:43 builder [<c013546d>] add_wait_queue+0x1d/0x50
Dec 9 01:19:43 builder [<c0176712>] pipe_poll+0x32/0xb0
Dec 9 01:19:43 builder [<c017d50e>] do_select+0x3ae/0x4a0
Dec 9 01:19:43 builder [<c017db60>] __pollwait+0x0/0x100
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c0254499>] sock_def_readable+0x69/0x70
Dec 9 01:19:43 builder [<c012b987>] lock_timer_base+0x27/0x60
Dec 9 01:19:43 builder [<c012bac7>] __mod_timer+0x97/0xb0
Dec 9 01:19:43 builder [<c02f0bc3>] _spin_lock_irqsave+0x23/0x30
Dec 9 01:19:43 builder [<c01351b0>] autoremove_wake_function+0x0/0x50
Dec 9 01:19:43 builder [<c017dcad>] sys_select+0x4d/0x1b0
Dec 9 01:19:43 builder [<c0171781>] sys_read+0x41/0x70
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder smbd S deeddf94 0 4225 4216
Dec 9 01:19:43 builder deee06c0 00000082 00000002 deeddf94 deeddf9c
00000000 00000000 deee0814
Dec 9 01:19:43 builder deee0814 c1413500 c03ce080 c03ce080 c03ce080
00000000 fffc508e 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f b7f63ff4
b7f6bc80 00000000 deedc000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f deedfb64
01b4a158 00000000 00002710
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02ef71b>] schedule_timeout+0x4b/0xd0
Dec 9 01:19:43 builder [<c012b5d0>] process_timeout+0x0/0x10
Dec 9 01:19:43 builder [<c017d50e>] do_select+0x3ae/0x4a0
Dec 9 01:19:43 builder [<c017db60>] __pollwait+0x0/0x100
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c027e08f>] ip_finish_output+0x14f/0x2d0
Dec 9 01:19:43 builder [<c02a0827>] inet_sendmsg+0x37/0x70
Dec 9 01:19:43 builder [<c02504df>] sock_sendmsg+0xbf/0xf0
Dec 9 01:19:43 builder [<c01351b0>] autoremove_wake_function+0x0/0x50
Dec 9 01:19:43 builder [<c017d7c8>] core_sys_select+0x1c8/0x2f0
Dec 9 01:19:43 builder [<c0250897>] sys_sendto+0x107/0x160
Dec 9 01:19:43 builder [<c013a66d>] getnstimeofday+0x3d/0xe0
Dec 9 01:19:43 builder [<c010f75c>] lapic_next_event+0xc/0x10
Dec 9 01:19:43 builder [<c013d8d8>] clockevents_program_event+0xa8/0x120
Dec 9 01:19:43 builder sshd S deef7b3c 0 4284 1
Dec 9 01:19:43 builder deee1e60 00000086 00000002 deef7b3c deef7b44
00000000 dfb9b960 deee1fb4
Dec 9 01:19:43 builder deee1fb4 c1413500 c03ce080 c03ce080 c03ce080
00000046 01a678d7 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f 00000005
00000000 00000000 ffffffff
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02ef747>] schedule_timeout+0x77/0xd0
Dec 9 01:19:43 builder [<c013546d>] add_wait_queue+0x1d/0x50
Dec 9 01:19:43 builder [<c0113535>] handle_vm86_fault+0x285/0x810
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c01e8ba7>] vsnprintf+0x307/0x5e0
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c01244bf>] release_task+0x1cf/0x300
Dec 9 01:19:43 builder [<c01244bf>] release_task+0x1cf/0x300
Dec 9 01:19:43 builder [<c01244bf>] release_task+0x1cf/0x300
Dec 9 01:19:43 builder [<c0124507>] release_task+0x217/0x300
Dec 9 01:19:43 builder [<c010b5ee>] restore_i387+0xee/0x100
Dec 9 01:19:43 builder [<c017dcad>] sys_select+0x4d/0x1b0
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder zabbix_agentd S deec7f0c 0 4399 1
Dec 9 01:19:43 builder dfae5b00 00000082 00000002 deec7f0c deec7f14
00000000 c02f0b73 dfae5c54
Dec 9 01:19:43 builder dfae5c54 c1409500 c03ce080 c03ce080 c03ce080
c015b836 fffc5cb9 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f 00000000
dfae5b00 dfae5bf0 00000000
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02f0b73>] _spin_lock_irq+0x13/0x20
Dec 9 01:19:43 builder [<c015b836>] handle_mm_fault+0x4c6/0x6a0
Dec 9 01:19:43 builder [<c01250fc>] do_wait+0x29c/0x350
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c0138390>] hrtimer_wakeup+0x0/0x20
Dec 9 01:19:43 builder [<c0138988>] sys_nanosleep+0x58/0x60
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder [<c02528b5>] lock_sock_nested+0xa5/0xb0
Dec 9 01:19:43 builder [<c02f0b2f>] _spin_lock_bh+0xf/0x20
Dec 9 01:19:43 builder [<c0252783>] release_sock+0x13/0xa0
Dec 9 01:19:43 builder [<c0194265>] invalidate_inode_buffers+0x15/0x80
Dec 9 01:19:43 builder [<c01812d7>] d_kill+0x37/0x50
Dec 9 01:19:43 builder [<c0251932>] sys_socketcall+0xd2/0x2b0
Dec 9 01:19:43 builder [<c016ef67>] filp_close+0x47/0x80
Dec 9 01:19:43 builder [<c0282b82>] inet_csk_accept+0x132/0x230
Dec 9 01:19:43 builder [<c01351b0>] autoremove_wake_function+0x0/0x50
Dec 9 01:19:43 builder [<c02a1445>] inet_accept+0x25/0xa0
Dec 9 01:19:43 builder [<c025175e>] do_accept+0x12e/0x230
Dec 9 01:19:43 builder [<c0290a08>] __tcp_push_pending_frames+0x108/0x760
Dec 9 01:19:43 builder [<c02f0b2f>] _spin_lock_bh+0xf/0x20
Dec 9 01:19:43 builder [<c02528b5>] lock_sock_nested+0xa5/0xb0
Dec 9 01:19:43 builder [<c02f0b2f>] _spin_lock_bh+0xf/0x20
Dec 9 01:19:43 builder [<c0252783>] release_sock+0x13/0xa0
Dec 9 01:19:43 builder [<c0194265>] invalidate_inode_buffers+0x15/0x80
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02ef747>] schedule_timeout+0x77/0xd0
Dec 9 01:19:43 builder [<c02528b5>] lock_sock_nested+0xa5/0xb0
Dec 9 01:19:43 builder [<c02f0b2f>] _spin_lock_bh+0xf/0x20
Dec 9 01:19:43 builder [<c0252783>] release_sock+0x13/0xa0
Dec 9 01:19:43 builder [<c01352f0>] prepare_to_wait_exclusive+0x20/0x70
Dec 9 01:19:43 builder [<c0282b82>] inet_csk_accept+0x132/0x230
Dec 9 01:19:43 builder [<c0194265>] invalidate_inode_buffers+0x15/0x80
Dec 9 01:19:43 builder [<c01812d7>] d_kill+0x37/0x50
Dec 9 01:19:43 builder [<c0251932>] sys_socketcall+0xd2/0x2b0
Dec 9 01:19:43 builder [<c016ef67>] filp_close+0x47/0x80
Dec 9 01:19:43 builder [<c01703ee>] sys_close+0x6e/0xc0
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder zabbix_agentd S def2df30 0 4407 4399
Dec 9 01:19:43 builder deee2520 00000086 00000002 def2df30 def2df38
00000000 00000001 deee2674
Dec 9 01:19:43 builder deee2674 c1409500 c03ce080 c03ce080 c03ce080
297d5df0 01b47a4c 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f def2df60
deee2520 00000000 bfbd9c54
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02efd6f>] do_nanosleep+0x5f/0x90
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder agetty S deef1eb0 0 4469 1
Dec 9 01:19:43 builder dfae7960 00000082 00000002 deef1eb0 deef1eb8
00000000 00000007 dfae7ab4
Dec 9 01:19:43 builder dfae7ab4 c1409500 c03ce080 c03ce080 c03ce080
c038750c fffc656a 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f 00000001
dfbad800 7fffffff 7fffffff
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02ef747>] schedule_timeout+0x77/0xd0
Dec 9 01:19:43 builder [<c02f0bc3>] _spin_lock_irqsave+0x23/0x30
Dec 9 01:19:43 builder [<c0201c85>] tty_read+0x85/0xc0
Dec 9 01:19:43 builder [<c0170b17>] do_sync_read+0xc7/0x110
Dec 9 01:19:43 builder [<c01351b0>] autoremove_wake_function+0x0/0x50
Dec 9 01:19:43 builder [<c0171453>] vfs_read+0x133/0x140
Dec 9 01:19:43 builder [<c0171781>] sys_read+0x41/0x70
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder sshd S 00000000 0 4519 4516
Dec 9 01:19:43 builder deee2f40 00000086 00000000 00000000 00000000
00000000 00000000 deee3094
Dec 9 01:19:43 builder deee3094 c1409500 c03ce080 c03ce080 c03ce080
00000000 00000001 00000082
Dec 9 01:19:43 builder c011a35e 00000000 00000000 00000003 0000000b
00000000 00000000 ffffffff
Dec 9 01:19:43 builder [<c02640bb>] neigh_resolve_output+0xdb/0x280

Dec 9 01:19:43 builder SysRq : Show Blocked State
Dec 9 01:19:43 builder task PC stack pid father
Dec 9 01:19:43 builder pdflush D c140fdc0 0 94 2
Dec 9 01:19:43 builder df89c000 00000046 00000818 c140fdc0 00000002
df87def8 df87df34 df89c154
Dec 9 01:19:43 builder df89c154 c1413500 c03ce080 c03ce080 c03ce080
df87df24 01b3bb70 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000818 0000000f df87df24
01b47a49 df567198 00000001
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c02ef71b>] schedule_timeout+0x4b/0xd0
Dec 9 01:19:43 builder [<c012b5d0>] process_timeout+0x0/0x10
Dec 9 01:19:43 builder [<c01c46b1>] journal_stop+0x141/0x1d0
Dec 9 01:19:43 builder [<c0153db0>] pdflush+0x0/0x1e0
Dec 9 01:19:43 builder [<c01bd36d>] ext3_sync_fs+0xd/0x30
Dec 9 01:19:43 builder [<c017267a>] __fsync_super+0x4a/0x70
Dec 9 01:19:43 builder [<c01726a8>] fsync_super+0x8/0x20
Dec 9 01:19:43 builder [<c01726e9>] do_remount_sb+0x29/0x170
Dec 9 01:19:43 builder [<c0153db0>] pdflush+0x0/0x1e0
Dec 9 01:19:43 builder [<c0172c35>] do_emergency_remount+0x95/0xc0
Dec 9 01:19:43 builder [<c0153eb6>] pdflush+0x106/0x1e0
Dec 9 01:19:43 builder [<c0172ba0>] do_emergency_remount+0x0/0xc0
Dec 9 01:19:43 builder [<c0134e92>] kthread+0x42/0x70
Dec 9 01:19:43 builder [<c0134e50>] kthread+0x0/0x70
Dec 9 01:19:43 builder [<c0104fc7>] kernel_thread_helper+0x7/0x10
Dec 9 01:19:43 builder =======================

Dec 9 01:19:43 builder [<c027e08f>] ip_finish_output+0x14f/0x2d0
Dec 9 01:19:43 builder [<c027dbe5>] ip_local_out+0x15/0x20
Dec 9 01:19:43 builder [<c027e725>] ip_queue_xmit+0x145/0x310
Dec 9 01:19:43 builder [<c0294613>] tcp_v4_send_check+0x43/0xf0
Dec 9 01:19:43 builder [<c028f366>] tcp_transmit_skb+0x386/0x690
Dec 9 01:19:43 builder [<c01155de>] pvclock_clocksource_read+0x4e/0xe0
Dec 9 01:19:43 builder [<c0290a08>] __tcp_push_pending_frames+0x108/0x760
Dec 9 01:19:43 builder [<c0283fe5>] sk_stream_alloc_skb+0x35/0xe0
Dec 9 01:19:43 builder [<c017dcad>] sys_select+0x4d/0x1b0
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f 00000000
deee32a0 deee3390 00000003
Dec 9 01:19:43 builder Call Trace:
Dec 9 01:19:43 builder [<c015b836>] handle_mm_fault+0x4c6/0x6a0
Dec 9 01:19:43 builder [<c01250fc>] do_wait+0x29c/0x350
Dec 9 01:19:43 builder [<c011ca60>] default_wake_function+0x0/0x10
Dec 9 01:19:43 builder [<c0125225>] sys_wait4+0x75/0xc0
Dec 9 01:19:43 builder [<c0125295>] sys_waitpid+0x25/0x30
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder bash S def33eb0 0 4526 4525
Dec 9 01:19:43 builder [<c0205d10>] read_chan+0x0/0x650
Dec 9 01:19:43 builder [<c01713c1>] vfs_read+0xa1/0x140
Dec 9 01:19:43 builder [<c0201c00>] tty_read+0x0/0xc0
Dec 9 01:19:43 builder [<c0171781>] sys_read+0x41/0x70
Dec 9 01:19:43 builder [<c0103d1f>] sysenter_do_call+0x12/0x33
Dec 9 01:19:43 builder =======================
Dec 9 01:19:43 builder smbd S def53b3c 0 5148 4216
Dec 9 01:19:43 builder deee0000 00000086 00000002 def53b3c def53b44
00000000 c0371340 deee0154
Dec 9 01:19:43 builder deee0154 c1409500 c03ce080 c03ce080 c03ce080
def53b64 01b3f66d 00000000
Dec 9 01:19:43 builder 00000000 00000000 00000000 0000000f def53b64
01b4edf2 00000000 0000ea60
Dec 9 01:19:43 builder [<c02640bb>] neigh_resolve_output+0xdb/0x280
Dec 9 01:19:43 builder [<c027e08f>] ip_finish_output+0x14f/0x2d0
Dec 9 01:19:43 builder [<c027dbe5>] ip_local_out+0x15/0x20
Dec 9 01:19:43 builder [<c027e725>] ip_queue_xmit+0x145/0x310
Dec 9 01:19:43 builder [<c0294613>] tcp_v4_send_check+0x43/0xf0
Dec 9 01:19:43 builder [<c028f366>] tcp_transmit_skb+0x386/0x690
Dec 9 01:19:43 builder [<c02f0bc3>] _spin_lock_irqsave+0x23/0x30
Dec 9 01:19:43 builder [<c012b987>] lock_timer_base+0x27/0x60
Dec 9 01:19:43 builder [<c012bac7>] __mod_timer+0x97/0xb0
Dec 9 01:19:43 builder [<c02528fc>] sk_reset_timer+0xc/0x20

Dec 9 01:19:43 builder SysRq : Show Memory
Dec 9 01:19:43 builder Mem-Info:
Dec 9 01:19:43 builder DMA per-cpu:
Dec 9 01:19:43 builder CPU 0: hi: 0, btch: 1 usd: 0
Dec 9 01:19:43 builder CPU 1: hi: 0, btch: 1 usd: 0
Dec 9 01:19:43 builder Normal per-cpu:
Dec 9 01:19:43 builder CPU 0: hi: 186, btch: 31 usd: 171
Dec 9 01:19:43 builder CPU 1: hi: 186, btch: 31 usd: 75
Dec 9 01:19:43 builder Active:34994 inactive:89089 dirty:19
writeback:0 unstable:0
Dec 9 01:19:43 builder free:1827 slab:2161 mapped:1690 pagetables:144 bounce:0
Dec 9 01:19:43 builder DMA free:2324kB min:84kB low:104kB high:124kB
active:1964kB inactive:8580kB present:15868kB pages_scanned:0
all_unreclaimable? no
Dec 9 01:19:43 builder lowmem_reserve[]: 0 492 492
Dec 9 01:19:43 builder Normal free:4984kB min:2792kB low:3488kB
high:4188kB active:138012kB inactive:347776kB present:503872kB
pages_scanned:0 all_unreclaimable?
no
Dec 9 01:19:43 builder lowmem_reserve[]: 0 0 0
Dec 9 01:19:43 builder DMA: 3*4kB 11*8kB 1*16kB 5*32kB 8*64kB 2*128kB
1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2324kB
Dec 9 01:19:43 builder Normal: 230*4kB 33*8kB 8*16kB 18*32kB 3*64kB
3*128kB 0*256kB 1*512kB 2*1024kB 0*2048kB 0*4096kB = 5024kB
Dec 9 01:19:43 builder 95789 total pagecache pages
Dec 9 01:19:43 builder 0 pages in swap cache
Dec 9 01:19:43 builder Swap cache stats: add 17, delete 17, find 0/0
Dec 9 01:19:43 builder Free swap = 262068kB
Dec 9 01:19:43 builder Total swap = 262136kB
Dec 9 01:19:43 builder 131056 pages RAM
Dec 9 01:19:43 builder 1975 pages reserved
Dec 9 01:19:43 builder 7105 pages shared
Dec 9 01:19:43 builder 123482 pages non-shared