From: Andrew Lutomirski <luto@mit.edu>
To: Andi Kleen <andi@firstfloor.org>
Cc: linux-kernel@vger.kernel.org
Subject: Re: Kernel falls apart under light memory pressure (i.e. linking vmlinux)
Date: Thu, 12 May 2011 07:54:26 -0400 [thread overview]
Message-ID: <BANLkTi=fk3DUT9cYd2gAzC98c69F6HXX7g@mail.gmail.com> (raw)
In-Reply-To: <20110512054631.GI6008@one.firstfloor.org>
[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]
On Thu, May 12, 2011 at 1:46 AM, Andi Kleen <andi@firstfloor.org> wrote:
>> Here's a nice picture of alt-sysrq-m with lots of memory free but the
>> system mostly hung. I can still switch VTs.
>
> Would rather need backtraces. Try setting up netconsole or crashdump
> first.
Here are some logs for two different failure mores.
incorrect_oom_kill.txt is an OOM kill when there was lots of available
swap to use. AFAICT the kernel should not have OOM killed at all.
stuck_xyz is when the system is wedged with plenty (~300MB) free
memory but no swap. The sysrq files are self-explanatory.
stuck-sysrq-f.txt is after the others so that it won't have corrupted
the output. After taking all that data, I waited awhile and started
getting soft lockup messges.
I'm having trouble reproducing the "stuck" failure mode on my
lockdep-enabled kernel right now (the OOM kill is easy), so no lock
state trace. But I got one yesterday and IIRC it showed a few tty
locks and either kworker or kcryptd holding (kqueue) and
((&io->work)).
I compressed the larger files.
--Andy
>
> -Andi
>
> --
> ak@linux.intel.com -- Speaking for myself only.
>
[-- Attachment #2: stuck-sysrq-m.txt --]
[-- Type: text/plain, Size: 3349 bytes --]
[ 302.500408] SysRq : Show Memory
[ 302.500967] Mem-Info:
[ 302.501502] Node 0 DMA per-cpu:
[ 302.502050] CPU 0: hi: 0, btch: 1 usd: 0
[ 302.502597] CPU 1: hi: 0, btch: 1 usd: 0
[ 302.503140] CPU 2: hi: 0, btch: 1 usd: 0
[ 302.503674] CPU 3: hi: 0, btch: 1 usd: 0
[ 302.504201] Node 0 DMA32 per-cpu:
[ 302.504742] CPU 0: hi: 186, btch: 31 usd: 103
[ 302.505290] CPU 1: hi: 186, btch: 31 usd: 49
[ 302.505841] CPU 2: hi: 186, btch: 31 usd: 179
[ 302.506393] CPU 3: hi: 186, btch: 31 usd: 191
[ 302.506945] Node 0 Normal per-cpu:
[ 302.507500] CPU 0: hi: 0, btch: 1 usd: 0
[ 302.508069] CPU 1: hi: 0, btch: 1 usd: 0
[ 302.508623] CPU 2: hi: 0, btch: 1 usd: 0
[ 302.509179] CPU 3: hi: 0, btch: 1 usd: 0
[ 302.509728] active_anon:21828 inactive_anon:7989 isolated_anon:0
[ 302.509729] active_file:5958 inactive_file:554 isolated_file:0
[ 302.509730] unevictable:330922 dirty:2 writeback:1 unstable:0
[ 302.509731] free:85704 slab_reclaimable:3513 slab_unreclaimable:8839
[ 302.509731] mapped:4156 shmem:8110 pagetables:2962 bounce:0
[ 302.512624] Node 0 DMA free:8116kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:0kB active_file:7512kB inactive_file:0kB unevictable:256kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no
[ 302.514706] lowmem_reserve[]: 0 1853 1859 1859
[ 302.515446] Node 0 DMA32 free:334684kB min:66816kB low:83520kB high:100224kB active_anon:87312kB inactive_anon:31956kB active_file:16320kB inactive_file:2216kB unevictable:1323432kB isolated(anon):0kB isolated(file):0kB present:1898336kB mlocked:0kB dirty:8kB writeback:4kB mapped:16620kB shmem:32440kB slab_reclaimable:13948kB slab_unreclaimable:33660kB kernel_stack:1672kB pagetables:11848kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 302.517960] lowmem_reserve[]: 0 0 5 5
[ 302.518848] Node 0 Normal free:16kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:88kB slab_unreclaimable:1696kB kernel_stack:264kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 302.521819] lowmem_reserve[]: 0 0 0 0
[ 302.522858] Node 0 DMA: 1*4kB 0*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8116kB
[ 302.523968] Node 0 DMA32: 6673*4kB 3706*8kB 2320*16kB 1414*32kB 768*64kB 269*128kB 85*256kB 37*512kB 10*1024kB 6*2048kB 12*4096kB = 334676kB
[ 302.525120] Node 0 Normal: 4*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16kB
[ 302.526273] 345568 total pagecache pages
[ 302.527369] 0 pages in swap cache
[ 302.528431] Swap cache stats: add 0, delete 0, find 0/0
[ 302.529516] Free swap = 0kB
[ 302.530608] Total swap = 0kB
[ 302.538578] 525808 pages RAM
[ 302.539703] 41300 pages reserved
[ 302.540829] 30451 pages shared
[ 302.541964] 384267 pages non-shared
[-- Attachment #3: incorrect_oom_kill.txt.xz --]
[-- Type: application/x-xz, Size: 25112 bytes --]
[-- Attachment #4: stuck-sysrq-t.txt.xz --]
[-- Type: application/x-xz, Size: 31008 bytes --]
[-- Attachment #5: stuck-sysrq-w.txt --]
[-- Type: text/plain, Size: 28818 bytes --]
[ 343.295265] SysRq : Show Blocked State
[ 343.295972] task PC stack pid father
[ 343.296680] jbd2/dm-2-8 D ffff8800754a5bc0 0 372 2 0x00000000
[ 343.297390] ffff8800751f3b70 0000000000000046 ffff88006ca644d0 ffffffff810088f4
[ 343.298096] ffff8800754a5bc0 ffff8800751f3fd8 ffff8800751f3fd8 0000000000011a40
[ 343.298797] 0000000000011a40 ffff8800754a5bc0 ffff880078391a40 ffff88007858ce08
[ 343.299505] Call Trace:
[ 343.300202] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 343.300915] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.301629] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.302339] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 343.303049] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 343.303756] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 343.304468] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 343.305179] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 343.305887] [<ffffffff810b1b1f>] filemap_fdatawait_range+0x84/0x163
[ 343.306610] [<ffffffff811e4709>] ? submit_bio+0xde/0xfd
[ 343.307344] [<ffffffff8111e1ca>] ? bio_alloc_bioset+0x4c/0xc3
[ 343.308089] [<ffffffff810b1c22>] filemap_fdatawait+0x24/0x26
[ 343.308843] [<ffffffff8119df33>] jbd2_journal_commit_transaction+0x9eb/0x13c0
[ 343.309610] [<ffffffff81058dbc>] ? autoremove_wake_function+0x0/0x37
[ 343.310385] [<ffffffff811a274a>] kjournald2+0x13d/0x36a
[ 343.311166] [<ffffffff81058dbc>] ? autoremove_wake_function+0x0/0x37
[ 343.311959] [<ffffffff811a260d>] ? kjournald2+0x0/0x36a
[ 343.312750] [<ffffffff810586f0>] kthread+0x84/0x8c
[ 343.313542] [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
[ 343.314350] [<ffffffff814485ad>] ? restore_args+0x0/0x30
[ 343.315159] [<ffffffff8105866c>] ? kthread+0x0/0x8c
[ 343.315972] [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
[ 343.316796] NetworkManager D ffffffff810b1879 0 794 1 0x00000000
[ 343.317633] ffff88006cf61b48 0000000000000082 ffff88006dfa0000 ffffffff810088f4
[ 343.318489] ffff88006f3344d0 ffff88006cf61fd8 ffff88006cf61fd8 0000000000011a40
[ 343.319356] 0000000000011a40 ffff88006f3344d0 ffff880078291a40 ffff88007858c3e8
[ 343.320231] Call Trace:
[ 343.321090] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 343.321969] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.322844] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.323717] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 343.324601] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 343.325484] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 343.326377] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 343.327277] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 343.328180] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 343.329092] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 343.330010] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 343.330919] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 343.331808] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 343.332680] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 343.333546] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 343.334413] [<ffffffff810430dd>] ? timespec_add_safe+0x37/0x66
[ 343.335268] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 343.336113] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 343.336962] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 343.337801] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 343.338632] ntpd D ffff88007828d980 0 796 1 0x00000000
[ 343.339466] ffff88006f391b48 0000000000000086 ffff8801005c96f0 ffffffffffffffff
[ 343.340332] ffff88006f335bc0 ffff88006f391fd8 ffff88006f391fd8 0000000000011a40
[ 343.341211] 0000000000011a40 ffff88006f335bc0 ffff880078291a40 0000000178583148
[ 343.342099] Call Trace:
[ 343.342966] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.343850] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.344719] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 343.345586] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 343.346451] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 343.347309] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 343.348176] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 343.349043] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 343.349905] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 343.350766] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 343.351622] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 343.352472] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 343.353328] [<ffffffff811058eb>] ? set_fd_set+0x3c/0x46
[ 343.354178] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 343.355023] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 343.355868] [<ffffffff8105fbeb>] ? timekeeping_get_ns+0x18/0x3a
[ 343.356715] [<ffffffff8102ab0d>] ? should_resched+0xe/0x2d
[ 343.357562] [<ffffffff8144677c>] ? _cond_resched+0xe/0x22
[ 343.358403] [<ffffffff8110587c>] ? might_fault+0x21/0x23
[ 343.359230] [<ffffffff811059d1>] ? poll_select_copy_remaining+0xdc/0x100
[ 343.360062] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 343.360891] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 343.361705] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 343.362510] crond D ffffffff810b1879 0 827 1 0x00000000
[ 343.363341] ffff88006dd2fb48 0000000000000086 ffff88006d402de0 ffffffff810088f4
[ 343.364198] ffff88006d4016f0 ffff88006dd2ffd8 ffff88006dd2ffd8 0000000000011a40
[ 343.365061] 0000000000011a40 ffff88006d4016f0 ffff880078291a40 ffff880078583c88
[ 343.365932] Call Trace:
[ 343.366785] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 343.367646] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.368509] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.369358] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 343.370210] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 343.371051] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 343.371897] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 343.372740] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 343.373591] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 343.374436] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 343.375286] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 343.376131] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 343.376975] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 343.377824] [<ffffffff811058eb>] ? set_fd_set+0x3c/0x46
[ 343.378671] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 343.379512] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 343.380340] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 343.381167] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 343.381999] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 343.382809] rs:main Q:Reg D ffff88007828d980 0 942 1 0x00000000
[ 343.383639] ffff88006d51bb48 0000000000000086 ffff8801005c96f0 ffffffffffffffff
[ 343.384491] ffff88006d400000 ffff88006d51bfd8 ffff88006d51bfd8 0000000000011a40
[ 343.385358] 0000000000011a40 ffff88006d400000 ffff880078291a40 00000001785929e8
[ 343.386223] Call Trace:
[ 343.387068] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.387928] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.388771] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 343.389611] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 343.390451] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 343.391290] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 343.392131] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 343.392972] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 343.393809] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 343.394652] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 343.395496] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 343.396334] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 343.397179] [<ffffffff81166466>] ? ext4_file_write+0x1dc/0x232
[ 343.398023] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 343.398869] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 343.399705] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 343.400549] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 343.401398] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 343.402240] auditd D ffff88007838d980 0 982 1 0x00000000
[ 343.403098] ffff88006de2fb48 0000000000000082 ffff8801004c96f0 ffffffffffffffff
[ 343.403959] ffff88006cceade0 ffff88006de2ffd8 ffff88006de2ffd8 0000000000011a40
[ 343.404820] 0000000000011a40 ffff88006cceade0 ffff880078391a40 000000017858bc08
[ 343.405676] Call Trace:
[ 343.406505] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.407344] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.408178] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 343.409022] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 343.409865] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 343.410710] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 343.411558] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 343.412404] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 343.413250] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 343.414097] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 343.414944] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 343.415784] [<ffffffff8102c72b>] ? set_next_entity+0x46/0x98
[ 343.416629] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 343.417474] [<ffffffff8109e4d6>] ? trace_hardirqs_on+0x9/0x20
[ 343.418318] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 343.419162] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 343.420009] [<ffffffff81127b07>] ? sys_epoll_wait+0x2e4/0x302
[ 343.420851] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 343.421865] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 343.422710] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 343.423550] console-kit-dae D ffff88007828d980 0 1228 1 0x00000080
[ 343.424398] ffff88007414bb48 0000000000000086 ffff8801005c96f0 ffffffffffffffff
[ 343.425259] ffff88006d9c44d0 ffff88007414bfd8 ffff88007414bfd8 0000000000011a40
[ 343.426124] 0000000000011a40 ffff88006d9c44d0 ffff880078291a40 0000000178598d48
[ 343.426982] Call Trace:
[ 343.427814] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.428670] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.429520] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 343.430368] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 343.431218] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 343.432066] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 343.432919] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 343.433773] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 343.434626] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 343.435478] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 343.436326] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 343.437174] [<ffffffff810cbbd9>] ? do_wp_page+0x258/0x554
[ 343.438024] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 343.438870] [<ffffffff811c17ca>] ? kzalloc+0x14/0x16
[ 343.439717] [<ffffffff811c17ca>] ? kzalloc+0x14/0x16
[ 343.440548] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 343.441385] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 343.442220] [<ffffffff81127d6f>] ? anon_inode_getfile+0x117/0x189
[ 343.443052] [<ffffffff81447f56>] ? _raw_spin_lock+0xe/0x10
[ 343.443870] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 343.444691] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 343.445508] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 343.446317] packagekitd D ffff88007828d980 0 1882 1 0x00000080
[ 343.447133] ffff8800567c3b48 0000000000000082 ffff8801005c96f0 ffffffffffffffff
[ 343.447978] ffff88006d5c44d0 ffff8800567c3fd8 ffff8800567c3fd8 0000000000011a40
[ 343.448831] 0000000000011a40 ffff88006d5c44d0 ffff880078291a40 0000000178592c28
[ 343.449690] Call Trace:
[ 343.450535] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.451387] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.452231] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 343.453073] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 343.453912] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 343.454746] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 343.455586] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 343.456429] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 343.457263] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 343.458102] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 343.458945] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 343.459782] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 343.460621] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 343.461455] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 343.462292] [<ffffffff810430dd>] ? timespec_add_safe+0x37/0x66
[ 343.463126] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 343.463953] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 343.464777] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 343.465587] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 343.466398] gdm-simple-gree D ffff8800521d96f0 0 2282 2254 0x00000080
[ 343.467211] ffff880051a7db48 0000000000000086 ffff8800754edbc0 ffffffff810088f4
[ 343.468047] ffff8800521d96f0 ffff880051a7dfd8 ffff880051a7dfd8 0000000000011a40
[ 343.468897] 0000000000011a40 ffff8800521d96f0 ffff880078211a40 ffff880078594e48
[ 343.469747] Call Trace:
[ 343.470580] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 343.471424] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.472270] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 343.473108] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 343.473947] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 343.474780] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 343.475608] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 343.476439] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 343.477272] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 343.478109] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 343.478949] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 343.479795] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 343.480640] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 343.481488] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 343.482330] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 343.483174] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 343.484020] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 343.484870] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 343.485709] umount D ffff88007830d860 0 2364 2322 0x00000080
[ 343.486565] ffff8800655f3d58 0000000000000082 ffff88006f83dbc0 0000000000000001
[ 343.487423] ffff880064f416f0 ffff8800655f3fd8 ffff8800655f3fd8 0000000000011a40
[ 343.488280] 0000000000011a40 ffff880064f416f0 ffff8800655f3de8 00000001810bba0e
[ 343.489127] Call Trace:
[ 343.489947] [<ffffffff81446a7f>] schedule_timeout+0x34/0xde
[ 343.490785] [<ffffffff8108ce96>] ? rcu_barrier_func+0x0/0x38
[ 343.491633] [<ffffffff8109e496>] ? trace_hardirqs_off+0x9/0x20
[ 343.492481] [<ffffffff81446846>] wait_for_common+0xb6/0x12d
[ 343.493329] [<ffffffff8108ce96>] ? rcu_barrier_func+0x0/0x38
[ 343.494185] [<ffffffff81037363>] ? default_wake_function+0x0/0x14
[ 343.495039] [<ffffffff8108d86b>] ? call_rcu_sched+0x0/0x17
[ 343.495894] [<ffffffff81446971>] wait_for_completion+0x1d/0x1f
[ 343.496750] [<ffffffff8108dbba>] _rcu_barrier+0x94/0xa4
[ 343.497603] [<ffffffff8108dbe6>] rcu_barrier_sched+0x1c/0x1e
[ 343.498455] [<ffffffff8108dbf6>] rcu_barrier+0xe/0x10
[ 343.499300] [<ffffffff810f90e4>] deactivate_locked_super+0x2b/0x4b
[ 343.500156] [<ffffffff810f9c5c>] deactivate_super+0x37/0x3b
[ 343.501010] [<ffffffff8110e60f>] mntput_no_expire+0xcc/0xd1
[ 343.501863] [<ffffffff8110f08c>] sys_umount+0x2b7/0x2e5
[ 343.502712] [<ffffffff8109e3a0>] ? trace_hardirqs_on_caller+0xf/0x23
[ 343.503570] [<ffffffff81002a3b>] system_call_fastpath+0x16/0x1b
[ 343.504425] Sched Debug Version: v0.10, 2.6.38.6-luto+ #1
[ 343.505281] ktime : 343780.077864
[ 343.506147] sched_clk : 343504.424586
[ 343.507005] cpu_clk : 343504.424621
[ 343.507851] jiffies : 4295010866
[ 343.508680] sched_clock_stable : 1
[ 343.509504]
[ 343.510304] sysctl_sched
[ 343.511100] .sysctl_sched_latency : 18.000000
[ 343.511908] .sysctl_sched_min_granularity : 2.250000
[ 343.512722] .sysctl_sched_wakeup_granularity : 3.000000
[ 343.513534] .sysctl_sched_child_runs_first : 0
[ 343.514352] .sysctl_sched_features : 7279
[ 343.515162] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 343.515983]
[ 343.515983] cpu#0, 2691.244 MHz
[ 343.517583] .nr_running : 0
[ 343.518383] .load : 0
[ 343.519180] .nr_switches : 680560
[ 343.519975] .nr_load_updates : 123585
[ 343.520762] .nr_uninterruptible : 1
[ 343.521544] .next_balance : 4295.010867
[ 343.522326] .curr->pid : 0
[ 343.523102] .clock : 343294.616974
[ 343.523881] .cpu_load[0] : 0
[ 343.524656] .cpu_load[1] : 0
[ 343.525412] .cpu_load[2] : 0
[ 343.526145] .cpu_load[3] : 0
[ 343.526870] .cpu_load[4] : 0
[ 343.527583] .yld_count : 5876
[ 343.528280] .sched_switch : 0
[ 343.528970] .sched_count : 689739
[ 343.529658] .sched_goidle : 140522
[ 343.530336] .avg_idle : 1000000
[ 343.531015] .ttwu_count : 344064
[ 343.531686] .ttwu_local : 165965
[ 343.532350] .bkl_count : 0
[ 343.533001]
[ 343.533001] cfs_rq[0]:/autogroup-0
[ 343.534278] .exec_clock : 31906.911768
[ 343.534929] .MIN_vruntime : 0.000001
[ 343.535575] .min_vruntime : 31190.284361
[ 343.536219] .max_vruntime : 0.000001
[ 343.536857] .spread : 0.000000
[ 343.537490] .spread0 : 0.000000
[ 343.538111] .nr_spread_over : 49
[ 343.538726] .nr_running : 0
[ 343.539333] .load : 0
[ 343.539925] .load_avg : 0.000000
[ 343.540507] .load_period : 0.000000
[ 343.541087] .load_contrib : 0
[ 343.541663] .load_tg : 0
[ 343.542237]
[ 343.542237] runnable tasks:
[ 343.542237] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 343.542238] ----------------------------------------------------------------------------------------------------------
[ 343.544572]
[ 343.544573] cpu#1, 2691.244 MHz
[ 343.545796] .nr_running : 0
[ 343.546414] .load : 0
[ 343.547028] .nr_switches : 754129
[ 343.547640] .nr_load_updates : 125065
[ 343.548242] .nr_uninterruptible : 6
[ 343.548853] .next_balance : 4295.010867
[ 343.549464] .curr->pid : 0
[ 343.550075] .clock : 343549.291505
[ 343.550694] .cpu_load[0] : 0
[ 343.551305] .cpu_load[1] : 0
[ 343.551913] .cpu_load[2] : 0
[ 343.552514] .cpu_load[3] : 0
[ 343.553111] .cpu_load[4] : 0
[ 343.553700] .yld_count : 29555
[ 343.554289] .sched_switch : 0
[ 343.554877] .sched_count : 785479
[ 343.555469] .sched_goidle : 102675
[ 343.556066] .avg_idle : 1000000
[ 343.556670] .ttwu_count : 379567
[ 343.557275] .ttwu_local : 262152
[ 343.557878] .bkl_count : 0
[ 343.558485]
[ 343.558485] cfs_rq[1]:/autogroup-0
[ 343.559681] .exec_clock : 27988.720915
[ 343.560492] .MIN_vruntime : 0.000001
[ 343.561108] .min_vruntime : 30180.328331
[ 343.561727] .max_vruntime : 0.000001
[ 343.562343] .spread : 0.000000
[ 343.562962] .spread0 : -1009.956030
[ 343.563575] .nr_spread_over : 99
[ 343.564183] .nr_running : 0
[ 343.564792] .load : 0
[ 343.565395] .load_avg : 0.000000
[ 343.565998] .load_period : 0.000000
[ 343.566607] .load_contrib : 0
[ 343.567210] .load_tg : 0
[ 343.567815]
[ 343.567815] runnable tasks:
[ 343.567816] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 343.567817] ----------------------------------------------------------------------------------------------------------
[ 343.570335]
[ 343.570336] cpu#2, 2691.244 MHz
[ 343.571669] .nr_running : 4
[ 343.572357] .load : 2048
[ 343.573029] .nr_switches : 645587
[ 343.573678] .nr_load_updates : 114758
[ 343.574298] .nr_uninterruptible : 1
[ 343.574917] .next_balance : 4295.010989
[ 343.575539] .curr->pid : 35
[ 343.576162] .clock : 287653.957942
[ 343.576793] .cpu_load[0] : 2048
[ 343.577420] .cpu_load[1] : 2048
[ 343.578042] .cpu_load[2] : 2048
[ 343.578659] .cpu_load[3] : 2048
[ 343.579267] .cpu_load[4] : 2048
[ 343.579866] .yld_count : 87800
[ 343.580465] .sched_switch : 0
[ 343.581068] .sched_count : 734865
[ 343.581677] .sched_goidle : 177805
[ 343.582288] .avg_idle : 1000000
[ 343.582906] .ttwu_count : 327160
[ 343.583526] .ttwu_local : 119891
[ 343.584141] .bkl_count : 0
[ 343.584759]
[ 343.584759] cfs_rq[2]:/autogroup-101
[ 343.585973] .exec_clock : 1403.158882
[ 343.586597] .MIN_vruntime : 0.000001
[ 343.587223] .min_vruntime : 1422.111430
[ 343.587849] .max_vruntime : 0.000001
[ 343.588470] .spread : 0.000000
[ 343.589097] .spread0 : -29768.172931
[ 343.589718] .nr_spread_over : 12
[ 343.590339] .nr_running : 0
[ 343.590954] .load : 0
[ 343.591569] .load_avg : 59.337728
[ 343.592185] .load_period : 3.260946
[ 343.592803] .load_contrib : 18
[ 343.593417] .load_tg : 18
[ 343.594024] .se->exec_start : 287650.761121
[ 343.594645] .se->vruntime : 19050.162029
[ 343.595267] .se->sum_exec_runtime : 1403.158882
[ 343.595897] .se->statistics.wait_start : 0.000000
[ 343.596527] .se->statistics.sleep_start : 0.000000
[ 343.597156] .se->statistics.block_start : 0.000000
[ 343.597783] .se->statistics.sleep_max : 0.000000
[ 343.598408] .se->statistics.block_max : 0.000000
[ 343.599018] .se->statistics.exec_max : 1.004017
[ 343.599602] .se->statistics.slice_max : 7.980791
[ 343.600157] .se->statistics.wait_max : 20.231126
[ 343.600719] .se->statistics.wait_sum : 518.802429
[ 343.601276] .se->statistics.wait_count : 23055
[ 343.601831] .se->load.weight : 2
[ 343.602392]
[ 343.602392] cfs_rq[2]:/autogroup-0
[ 343.603489] .exec_clock : 17597.540310
[ 343.604049] .MIN_vruntime : 19056.995325
[ 343.604617] .min_vruntime : 19057.193589
[ 343.605170] .max_vruntime : 19056.995325
[ 343.605730] .spread : 0.000000
[ 343.606292] .spread0 : -12133.090772
[ 343.606858] .nr_spread_over : 70
[ 343.607428] .nr_running : 2
[ 343.608000] .load : 2048
[ 343.608571] .load_avg : 0.000000
[ 343.609148] .load_period : 0.000000
[ 343.609720] .load_contrib : 0
[ 343.610288] .load_tg : 0
[ 343.610848]
[ 343.610848] rt_rq[2]:/autogroup-0
[ 343.611946] .rt_nr_running : 1
[ 343.612510] .rt_throttled : 0
[ 343.613076] .rt_time : 0.000000
[ 343.613644] .rt_runtime : 950.000000
[ 343.614204]
[ 343.614205] runnable tasks:
[ 343.614205] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 343.614206] ----------------------------------------------------------------------------------------------------------
[ 343.616506] migration/2 13 233.919157 204 0 233.919157 0.000748 0.000000 /autogroup-0
[ 343.617172] watchdog/2 16 0.000000 25 0 0.000000 0.405837 0.002261 /autogroup-0
[ 343.617847] R kswapd0 35 19060.192146 13301 120 19060.192146 10653.322143 240965.615309 /autogroup-0
[ 343.618539] kworker/2:2 452 19056.995325 64837 120 19056.995325 1980.111250 275438.926005 /autogroup-0
[ 343.619251]
[ 343.619252] cpu#3, 2691.244 MHz
[ 343.620622] .nr_running : 0
[ 343.621322] .load : 0
[ 343.622012] .nr_switches : 611779
[ 343.622706] .nr_load_updates : 114438
[ 343.623396] .nr_uninterruptible : 2
[ 343.624081] .next_balance : 4295.010868
[ 343.624775] .curr->pid : 0
[ 343.625470] .clock : 343625.193174
[ 343.626172] .cpu_load[0] : 0
[ 343.626874] .cpu_load[1] : 0
[ 343.627577] .cpu_load[2] : 0
[ 343.628274] .cpu_load[3] : 0
[ 343.628970] .cpu_load[4] : 0
[ 343.629660] .yld_count : 68568
[ 343.630358] .sched_switch : 0
[ 343.631050] .sched_count : 681648
[ 343.631741] .sched_goidle : 133540
[ 343.632424] .avg_idle : 1000000
[ 343.633113] .ttwu_count : 309555
[ 343.633799] .ttwu_local : 164424
[ 343.634479] .bkl_count : 0
[ 343.635156]
[ 343.635156] cfs_rq[3]:/autogroup-0
[ 343.636491] .exec_clock : 17594.691867
[ 343.637176] .MIN_vruntime : 0.000001
[ 343.637865] .min_vruntime : 24200.609785
[ 343.638545] .max_vruntime : 0.000001
[ 343.639233] .spread : 0.000000
[ 343.639921] .spread0 : -6989.674576
[ 343.640609] .nr_spread_over : 144
[ 343.641301] .nr_running : 0
[ 343.641997] .load : 0
[ 343.642693] .load_avg : 0.000000
[ 343.643388] .load_period : 0.000000
[ 343.644077] .load_contrib : 0
[ 343.644771] .load_tg : 0
[ 343.645458]
[ 343.645459] runnable tasks:
[ 343.645459] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 343.645460] ----------------------------------------------------------------------------------------------------------
[ 343.648279]
[-- Attachment #6: stuck-sysrq-f.txt --]
[-- Type: text/plain, Size: 9768 bytes --]
[ 352.913556] SysRq : Manual OOM execution
[ 352.918668] kworker/0:2 invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
[ 352.919347] kworker/0:2 cpuset=/ mems_allowed=0
[ 352.920006] Pid: 348, comm: kworker/0:2 Not tainted 2.6.38.6-luto+ #1
[ 352.920683] Call Trace:
[ 352.921365] [<ffffffff8107f9c5>] ? cpuset_print_task_mems_allowed+0x91/0x9c
[ 352.922062] [<ffffffff810b3ef1>] ? dump_header+0x7f/0x1ba
[ 352.922772] [<ffffffff8109e4d6>] ? trace_hardirqs_on+0x9/0x20
[ 352.923494] [<ffffffff810b42ba>] ? oom_kill_process+0x50/0x24e
[ 352.924204] [<ffffffff810b4961>] ? out_of_memory+0x2e4/0x359
[ 352.924921] [<ffffffff812835ce>] ? moom_callback+0x0/0x27
[ 352.925635] [<ffffffff812835f3>] ? moom_callback+0x25/0x27
[ 352.926346] [<ffffffff810547d3>] ? process_one_work+0x16f/0x281
[ 352.927060] [<ffffffff8105533b>] ? worker_thread+0xd2/0x155
[ 352.927784] [<ffffffff81055269>] ? worker_thread+0x0/0x155
[ 352.928513] [<ffffffff810586f0>] ? kthread+0x84/0x8c
[ 352.929236] [<ffffffff81003814>] ? kernel_thread_helper+0x4/0x10
[ 352.929975] [<ffffffff814485ad>] ? restore_args+0x0/0x30
[ 352.930723] [<ffffffff8105866c>] ? kthread+0x0/0x8c
[ 352.931466] [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
[ 352.932215] Mem-Info:
[ 352.932970] Node 0 DMA per-cpu:
[ 352.933732] CPU 0: hi: 0, btch: 1 usd: 0
[ 352.934503] CPU 1: hi: 0, btch: 1 usd: 0
[ 352.935260] CPU 2: hi: 0, btch: 1 usd: 0
[ 352.936021] CPU 3: hi: 0, btch: 1 usd: 0
[ 352.936777] Node 0 DMA32 per-cpu:
[ 352.937543] CPU 0: hi: 186, btch: 31 usd: 40
[ 352.938310] CPU 1: hi: 186, btch: 31 usd: 52
[ 352.939067] CPU 2: hi: 186, btch: 31 usd: 179
[ 352.939830] CPU 3: hi: 186, btch: 31 usd: 191
[ 352.940593] Node 0 Normal per-cpu:
[ 352.941353] CPU 0: hi: 0, btch: 1 usd: 0
[ 352.942118] CPU 1: hi: 0, btch: 1 usd: 0
[ 352.942886] CPU 2: hi: 0, btch: 1 usd: 0
[ 352.943649] CPU 3: hi: 0, btch: 1 usd: 0
[ 352.944415] active_anon:22164 inactive_anon:7989 isolated_anon:0
[ 352.944416] active_file:5958 inactive_file:596 isolated_file:0
[ 352.944416] unevictable:330922 dirty:2 writeback:1 unstable:0
[ 352.944417] free:85392 slab_reclaimable:3513 slab_unreclaimable:8843
[ 352.944418] mapped:4156 shmem:8110 pagetables:2963 bounce:0
[ 352.948336] Node 0 DMA free:8116kB min:548kB low:684kB high:820kB active_anon:0kB inactive_anon:0kB active_file:7512kB inactive_file:0kB unevictable:256kB isolated(anon):0kB isolated(file):0kB present:15676kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:16kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no
[ 352.951034] lowmem_reserve[]: 0 1853 1859 1859
[ 352.951937] Node 0 DMA32 free:333436kB min:66816kB low:83520kB high:100224kB active_anon:88656kB inactive_anon:31956kB active_file:16320kB inactive_file:2384kB unevictable:1323432kB isolated(anon):0kB isolated(file):0kB present:1898336kB mlocked:0kB dirty:8kB writeback:4kB mapped:16620kB shmem:32440kB slab_reclaimable:13948kB slab_unreclaimable:33676kB kernel_stack:1672kB pagetables:11852kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 352.954939] lowmem_reserve[]: 0 0 5 5
[ 352.955985] Node 0 Normal free:16kB min:212kB low:264kB high:316kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6060kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:88kB slab_unreclaimable:1696kB kernel_stack:264kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 352.959275] lowmem_reserve[]: 0 0 0 0
[ 352.960407] Node 0 DMA: 1*4kB 0*8kB 1*16kB 1*32kB 2*64kB 2*128kB 2*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8116kB
[ 352.961598] Node 0 DMA32: 6363*4kB 3706*8kB 2320*16kB 1414*32kB 768*64kB 269*128kB 85*256kB 37*512kB 10*1024kB 6*2048kB 12*4096kB = 333436kB
[ 352.962830] Node 0 Normal: 4*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16kB
[ 352.964079] 345599 total pagecache pages
[ 352.965284] 0 pages in swap cache
[ 352.966476] Swap cache stats: add 0, delete 0, find 0/0
[ 352.967698] Free swap = 0kB
[ 352.968899] Total swap = 0kB
[ 352.977043] 525808 pages RAM
[ 352.978257] 41300 pages reserved
[ 352.979468] 30446 pages shared
[ 352.980659] 384641 pages non-shared
[ 352.981856] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
[ 352.983083] [ 423] 0 423 5066 1147 3 -17 -1000 udevd
[ 352.984318] [ 446] 0 446 5257 178 1 0 0 systemd-logger
[ 352.985551] [ 794] 0 794 39629 823 1 0 0 NetworkManager
[ 352.986780] [ 796] 38 796 8152 432 1 0 0 ntpd
[ 352.988017] [ 797] 0 797 4204 188 3 0 0 atd
[ 352.989255] [ 803] 0 803 1580 166 1 0 0 acpid
[ 352.990499] [ 804] 0 804 29790 313 0 0 0 abrtd
[ 352.991748] [ 822] 0 822 1708 52 1 0 0 gpm
[ 352.992992] [ 827] 0 827 28130 338 1 0 0 crond
[ 352.994233] [ 831] 0 831 1715 183 1 0 0 mcelog
[ 352.995475] [ 835] 81 835 24097 619 1 -13 -900 dbus-daemon
[ 352.996736] [ 841] 0 841 46757 824 1 -13 -900 polkitd
[ 352.997994] [ 867] 0 867 60752 865 1 0 0 rsyslogd
[ 352.999255] [ 872] 0 872 17191 492 1 -13 -900 modem-manager
[ 353.000506] [ 906] 0 906 6262 367 0 0 0 bluetoothd
[ 353.001788] [ 982] 0 982 6417 134 3 -17 -1000 auditd
[ 353.003066] [ 984] 0 984 20031 195 3 0 0 audispd
[ 353.004363] [ 985] 0 985 27199 271 3 0 0 sedispatch
[ 353.005644] [ 1057] 0 1057 47724 1087 3 0 0 cupsd
[ 353.006930] [ 1060] 0 1060 109370 1233 1 -13 -900 colord
[ 353.008215] [ 1074] 0 1074 21491 1394 1 0 0 dhclient
[ 353.009501] [ 1128] 0 1128 33012 443 1 0 0 gdm-binary
[ 353.010770] [ 1216] 0 1216 1043 139 1 0 0 agetty
[ 353.012008] [ 1218] 0 1218 18553 565 0 0 0 login
[ 353.013201] [ 1220] 0 1220 1043 140 1 0 0 agetty
[ 353.014366] [ 1223] 0 1223 1043 140 1 0 0 agetty
[ 353.015501] [ 1227] 0 1227 1043 139 0 0 0 agetty
[ 353.016600] [ 1228] 0 1228 539206 734 1 0 0 console-kit-dae
[ 353.017672] [ 1324] 487 1324 41115 281 3 0 0 rtkit-daemon
[ 353.018707] [ 1342] 0 1342 35816 720 0 -13 -900 upowerd
[ 353.019722] [ 1345] 0 1345 31871 712 0 -13 -900 udisks-daemon
[ 353.020698] [ 1346] 0 1346 11294 112 0 -13 -900 udisks-daemon
[ 353.021639] [ 1408] 0 1408 52335 607 2 0 0 accounts-daemon
[ 353.022544] [ 1507] 0 1507 5065 1018 2 -17 -1000 udevd
[ 353.023437] [ 1882] 0 1882 37333 734 1 -13 -900 packagekitd
[ 353.024315] [ 2116] 0 2116 28505 1270 1 0 0 bash
[ 353.025176] [ 2229] 0 2229 37573 562 1 0 0 gdm-simple-slav
[ 353.026003] [ 2231] 0 2231 30347 1999 1 0 0 Xorg
[ 353.026811] [ 2252] 42 2252 7541 340 1 0 0 dbus-daemon
[ 353.027626] [ 2254] 42 2254 102191 1050 1 0 0 gnome-session
[ 353.028429] [ 2260] 42 2260 36045 1463 1 0 0 gconfd-2
[ 353.029238] [ 2263] 42 2263 214606 2759 1 0 0 gnome-settings-
[ 353.030043] [ 2270] 42 2270 35570 375 3 0 0 gvfsd
[ 353.030853] [ 2275] 42 2275 89803 861 3 0 0 pulseaudio
[ 353.031670] [ 2279] 42 2279 146393 1887 1 0 0 metacity
[ 353.032489] [ 2282] 42 2282 123869 2011 0 0 0 gdm-simple-gree
[ 353.033313] [ 2283] 42 2283 59245 668 1 0 0 gnome-session-c
[ 353.034122] [ 2284] 42 2284 140919 1271 0 0 0 gnome-power-man
[ 353.034920] [ 2285] 42 2285 75907 981 2 0 0 polkit-gnome-au
[ 353.035736] [ 2289] 42 2289 11784 207 1 0 0 gnome-session-c
[ 353.036536] [ 2299] 42 2299 44851 461 0 0 0 dconf-service
[ 353.037347] [ 2313] 0 2313 33188 388 0 0 0 gdm-session-wor
[ 353.038157] [ 2322] 0 2322 27092 265 0 0 0 test_mempressur
[ 353.038965] [ 2335] 0 2335 5065 996 2 -17 -1000 udevd
[ 353.039781] [ 2364] 0 2364 28500 210 2 0 0 umount
[ 353.040597] Out of memory: Kill process 2263 (gnome-settings-) score 6 or sacrifice child
[ 353.041432] Killed process 2263 (gnome-settings-) total-vm:858424kB, anon-rss:7192kB, file-rss:3844kB
[-- Attachment #7: stuck-softlockup.txt --]
[-- Type: text/plain, Size: 25678 bytes --]
[ 419.796520] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:35]
[ 419.797364] Modules linked in: netconsole configfs tcp_lp fuse tun cpufreq_ondemand rfcomm sco bnep l2cap ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ipt_MASQUERADE ip6_tables iptable_nat nf_nat coretemp snd_hda_codec_hdmi snd_hda_codec_conexant btusb bluetooth arc4 ecb snd_hda_intel iwlagn snd_hda_codec i2c_i801 iwlcore thinkpad_acpi iTCO_wdt mac80211 snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer iTCO_vendor_support cfg80211 snd_page_alloc tpm_tis hwmon tpm tpm_bios wmi microcode snd soundcore kvm_intel kvm ipv6 aes_x86_64 aes_generic xts dm_crypt sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 419.802332] CPU 2
[ 419.802342] Modules linked in: netconsole configfs tcp_lp fuse tun cpufreq_ondemand rfcomm sco bnep l2cap ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ipt_MASQUERADE ip6_tables iptable_nat nf_nat coretemp snd_hda_codec_hdmi snd_hda_codec_conexant btusb bluetooth arc4 ecb snd_hda_intel iwlagn snd_hda_codec i2c_i801 iwlcore thinkpad_acpi iTCO_wdt mac80211 snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer iTCO_vendor_support cfg80211 snd_page_alloc tpm_tis hwmon tpm tpm_bios wmi microcode snd soundcore kvm_intel kvm ipv6 aes_x86_64 aes_generic xts dm_crypt sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 419.809095]
[ 419.810252] Pid: 35, comm: kswapd0 Not tainted 2.6.38.6-luto+ #1 LENOVO 4286CTO/4286CTO
[ 419.811466] RIP: 0010:[<ffffffff810bdd27>] [<ffffffff810bdd27>] shrink_slab+0x86/0x166
[ 419.812678] RSP: 0018:ffff88006c885da0 EFLAGS: 00000246
[ 419.813876] RAX: 0000000000000000 RBX: ffff88006c885de0 RCX: 0000000000000002
[ 419.815089] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88006caf1820
[ 419.816303] RBP: ffff88006c885de0 R08: ffff88006c885d60 R09: 0000000000000002
[ 419.817526] R10: 0000000000000002 R11: 0000000000000278 R12: ffffffff810033ce
[ 419.818756] R13: ffff88006c885e58 R14: 0000000000000002 R15: 00000000000050db
[ 419.819982] FS: 0000000000000000(0000) GS:ffff880078300000(0000) knlGS:0000000000000000
[ 419.821225] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 419.822461] CR2: 00007fed187b3000 CR3: 000000005422f000 CR4: 00000000000406e0
[ 419.823712] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 419.824967] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 419.826216] Process kswapd0 (pid: 35, threadinfo ffff88006c884000, task ffff88006f83dbc0)
[ 419.827480] Stack:
[ 419.828737] 0000000000000000 0000000000000080 ffff880000000002 ffff8801005fee00
[ 419.830036] ffff8801005fe000 0000000000000002 0000000000000000 000000000000000c
[ 419.831343] ffff88006c885ee0 ffffffff810c0959 ffff88006c885e40 ffff88006f83dbc0
[ 419.832652] Call Trace:
[ 419.833948] [<ffffffff810c0959>] kswapd+0x50a/0x76f
[ 419.835252] [<ffffffff810c044f>] ? kswapd+0x0/0x76f
[ 419.836551] [<ffffffff810586f0>] kthread+0x84/0x8c
[ 419.837852] [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
[ 419.839155] [<ffffffff814485ad>] ? restore_args+0x0/0x30
[ 419.840454] [<ffffffff8105866c>] ? kthread+0x0/0x8c
[ 419.841757] [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
[ 419.843054] Code: 83 eb 10 e9 ce 00 00 00 44 89 f2 31 f6 48 89 df ff 13 48 63 4b 08 31 d2 4c 63 e8 48 8b 45 c8 48 f7 f1 49 0f af c5 31 d2 49 f7 f7 <48> 03 43 20 48 85 c0 48 89 43 20 79 18 48 8b 33 48 89 c2 48 c7
[ 419.846008] Call Trace:
[ 419.847394] [<ffffffff810c0959>] kswapd+0x50a/0x76f
[ 419.848780] [<ffffffff810c044f>] ? kswapd+0x0/0x76f
[ 419.850140] [<ffffffff810586f0>] kthread+0x84/0x8c
[ 419.851482] [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
[ 419.852840] [<ffffffff814485ad>] ? restore_args+0x0/0x30
[ 419.854211] [<ffffffff8105866c>] ? kthread+0x0/0x8c
[ 419.855571] [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
[ 480.119201] INFO: task systemd:1 blocked for more than 120 seconds.
[ 480.120983] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.122740] systemd D 0000000000000000 0 1 0 0x00000000
[ 480.124447] ffff8801002e9b48 0000000000000086 ffffffff81a0b020 ffffffff00000000
[ 480.126126] ffff8801002e0000 ffff8801002e9fd8 ffff8801002e9fd8 0000000000011a40
[ 480.127814] 0000000000011a40 ffff8801002e0000 ffff880078211a40 000000017858e7e8
[ 480.129461] Call Trace:
[ 480.131026] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.132555] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.134010] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 480.135410] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 480.136807] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 480.138192] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 480.139577] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 480.140931] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 480.142270] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 480.143620] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 480.144957] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 480.146285] [<ffffffff8102c72b>] ? set_next_entity+0x46/0x98
[ 480.147612] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 480.148914] [<ffffffff814466b2>] ? schedule+0x63d/0x654
[ 480.150214] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 480.151495] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 480.152781] [<ffffffff81127b07>] ? sys_epoll_wait+0x2e4/0x302
[ 480.154077] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 480.155355] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 480.156609] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 480.157847] INFO: task fsnotify_mark:38 blocked for more than 120 seconds.
[ 480.159099] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.160359] fsnotify_mark D 0000000000000002 0 38 2 0x00000000
[ 480.161642] ffff88006c8bbd50 0000000000000046 ffff88006d402de0 ffffffff810ab657
[ 480.162936] ffff88006c88ade0 ffff88006c8bbfd8 ffff88006c8bbfd8 0000000000011a40
[ 480.164239] 0000000000011a40 ffff88006c88ade0 0000000000000082 ffff880078311a40
[ 480.165558] Call Trace:
[ 480.166840] [<ffffffff810ab657>] ? __perf_event_task_sched_out+0x27/0x2c
[ 480.168159] [<ffffffff81446f0b>] __mutex_lock_common+0x129/0x18e
[ 480.169484] [<ffffffff8108dc6d>] ? synchronize_sched+0x0/0x57
[ 480.170812] [<ffffffff814470ee>] __mutex_lock_slowpath+0x19/0x1b
[ 480.172075] [<ffffffff8144707f>] mutex_lock+0x36/0x50
[ 480.173272] [<ffffffff8109e4d6>] ? trace_hardirqs_on+0x9/0x20
[ 480.174480] [<ffffffff8105cc68>] __synchronize_srcu+0x25/0x89
[ 480.175686] [<ffffffff8105ccf8>] synchronize_srcu+0x15/0x17
[ 480.176901] [<ffffffff81124c78>] fsnotify_mark_destroy+0x8b/0x14d
[ 480.178138] [<ffffffff81058dbc>] ? autoremove_wake_function+0x0/0x37
[ 480.179369] [<ffffffff81124bed>] ? fsnotify_mark_destroy+0x0/0x14d
[ 480.180605] [<ffffffff810586f0>] kthread+0x84/0x8c
[ 480.181834] [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
[ 480.183074] [<ffffffff814485ad>] ? restore_args+0x0/0x30
[ 480.184299] [<ffffffff8105866c>] ? kthread+0x0/0x8c
[ 480.185500] [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
[ 480.186715] INFO: task jbd2/dm-2-8:372 blocked for more than 120 seconds.
[ 480.187910] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.189126] jbd2/dm-2-8 D ffff8800754a5bc0 0 372 2 0x00000000
[ 480.190357] ffff8800751f3b70 0000000000000046 ffff88006ca644d0 ffffffff810088f4
[ 480.191600] ffff8800754a5bc0 ffff8800751f3fd8 ffff8800751f3fd8 0000000000011a40
[ 480.192837] 0000000000011a40 ffff8800754a5bc0 ffff880078391a40 ffff88007858ce08
[ 480.194116] Call Trace:
[ 480.195359] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 480.196622] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.197875] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.199110] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 480.200359] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 480.201599] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 480.202842] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 480.204088] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 480.205330] [<ffffffff810b1b1f>] filemap_fdatawait_range+0x84/0x163
[ 480.206576] [<ffffffff811e4709>] ? submit_bio+0xde/0xfd
[ 480.207814] [<ffffffff8111e1ca>] ? bio_alloc_bioset+0x4c/0xc3
[ 480.209061] [<ffffffff810b1c22>] filemap_fdatawait+0x24/0x26
[ 480.210316] [<ffffffff8119df33>] jbd2_journal_commit_transaction+0x9eb/0x13c0
[ 480.211584] [<ffffffff81058dbc>] ? autoremove_wake_function+0x0/0x37
[ 480.212848] [<ffffffff811a274a>] kjournald2+0x13d/0x36a
[ 480.214109] [<ffffffff81058dbc>] ? autoremove_wake_function+0x0/0x37
[ 480.215393] [<ffffffff811a260d>] ? kjournald2+0x0/0x36a
[ 480.216655] [<ffffffff810586f0>] kthread+0x84/0x8c
[ 480.217898] [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
[ 480.219135] [<ffffffff814485ad>] ? restore_args+0x0/0x30
[ 480.220355] [<ffffffff8105866c>] ? kthread+0x0/0x8c
[ 480.221560] [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
[ 480.222777] INFO: task NetworkManager:794 blocked for more than 120 seconds.
[ 480.223997] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.225218] NetworkManager D ffffffff810b1879 0 794 1 0x00000000
[ 480.226450] ffff88006cf61b48 0000000000000082 ffff88006dfa0000 ffffffff810088f4
[ 480.227696] ffff88006f3344d0 ffff88006cf61fd8 ffff88006cf61fd8 0000000000011a40
[ 480.228960] 0000000000011a40 ffff88006f3344d0 ffff880078291a40 ffff88007858c3e8
[ 480.230221] Call Trace:
[ 480.231449] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 480.232686] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.233928] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.235159] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 480.236398] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 480.237612] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 480.238823] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 480.240049] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 480.241263] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 480.242487] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 480.243717] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 480.244927] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 480.246123] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 480.247313] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 480.248479] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 480.249631] [<ffffffff810430dd>] ? timespec_add_safe+0x37/0x66
[ 480.250784] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 480.251920] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 480.253065] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 480.254185] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 480.255306] INFO: task ntpd:796 blocked for more than 120 seconds.
[ 480.256456] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.257631] ntpd D ffff88007828d980 0 796 1 0x00000000
[ 480.258833] ffff88006f391b48 0000000000000086 ffff8801005c96f0 ffffffffffffffff
[ 480.260075] ffff88006f335bc0 ffff88006f391fd8 ffff88006f391fd8 0000000000011a40
[ 480.261310] 0000000000011a40 ffff88006f335bc0 ffff880078291a40 0000000178583148
[ 480.262540] Call Trace:
[ 480.263756] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.264980] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.266202] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 480.267404] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 480.268599] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 480.269788] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 480.270982] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 480.272156] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 480.273322] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 480.274492] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 480.275674] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 480.276866] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 480.278042] [<ffffffff811058eb>] ? set_fd_set+0x3c/0x46
[ 480.279222] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 480.280388] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 480.281543] [<ffffffff8105fbeb>] ? timekeeping_get_ns+0x18/0x3a
[ 480.282680] [<ffffffff8102ab0d>] ? should_resched+0xe/0x2d
[ 480.283803] [<ffffffff8144677c>] ? _cond_resched+0xe/0x22
[ 480.284929] [<ffffffff8110587c>] ? might_fault+0x21/0x23
[ 480.286041] [<ffffffff811059d1>] ? poll_select_copy_remaining+0xdc/0x100
[ 480.287151] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 480.288251] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 480.289356] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 480.290469] INFO: task crond:827 blocked for more than 120 seconds.
[ 480.291590] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.292752] crond D ffffffff810b1879 0 827 1 0x00000000
[ 480.293927] ffff88006dd2fb48 0000000000000086 ffff88006d402de0 ffffffff810088f4
[ 480.295136] ffff88006d4016f0 ffff88006dd2ffd8 ffff88006dd2ffd8 0000000000011a40
[ 480.296342] 0000000000011a40 ffff88006d4016f0 ffff880078291a40 ffff880078583c88
[ 480.297555] Call Trace:
[ 480.298737] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 480.299932] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.301141] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.302332] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 480.303529] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 480.304715] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 480.305893] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 480.307094] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 480.308291] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 480.309486] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 480.310675] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 480.311851] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 480.313020] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 480.314178] [<ffffffff811058eb>] ? set_fd_set+0x3c/0x46
[ 480.315315] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 480.316431] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 480.317555] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 480.318679] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 480.319792] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 480.320910] INFO: task polkitd:841 blocked for more than 120 seconds.
[ 480.322049] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.323215] polkitd D 0000000000000000 0 841 1 0x00000000
[ 480.324409] ffff880064c2bb48 0000000000000082 ffff8801005c96f0 ffffffff00000000
[ 480.325615] ffff88006ddadbc0 ffff880064c2bfd8 ffff880064c2bfd8 0000000000011a40
[ 480.326838] 0000000000011a40 ffff88006ddadbc0 ffff880078291a40 0000000178593108
[ 480.328058] Call Trace:
[ 480.329250] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.330463] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.331650] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 480.332825] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 480.334028] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 480.335221] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 480.336405] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 480.337590] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 480.338786] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 480.339975] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 480.341174] [<ffffffff81105c2d>] ? pollwake+0x0/0x54
[ 480.342363] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 480.343548] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 480.344738] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 480.345930] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 480.347116] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 480.348283] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 480.349424] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 480.350558] INFO: task rs:main Q:Reg:942 blocked for more than 120 seconds.
[ 480.351695] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.352853] rs:main Q:Reg D ffff88007828d980 0 942 1 0x00000000
[ 480.354012] ffff88006d51bb48 0000000000000086 ffff8801005c96f0 ffffffffffffffff
[ 480.355192] ffff88006d400000 ffff88006d51bfd8 ffff88006d51bfd8 0000000000011a40
[ 480.356385] 0000000000011a40 ffff88006d400000 ffff880078291a40 00000001785929e8
[ 480.357591] Call Trace:
[ 480.358771] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.359975] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.361157] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 480.362348] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 480.363529] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 480.364708] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 480.365891] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 480.367073] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 480.368282] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 480.369474] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 480.370654] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 480.371823] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 480.373002] [<ffffffff81166466>] ? ext4_file_write+0x1dc/0x232
[ 480.374184] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 480.375377] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 480.376552] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 480.377727] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 480.378883] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 480.380022] INFO: task modem-manager:872 blocked for more than 120 seconds.
[ 480.381163] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.382329] modem-manager D ffffffff810b1879 0 872 1 0x00000000
[ 480.383513] ffff88006d96fb48 0000000000000086 ffff880064cbdbc0 ffffffff810088f4
[ 480.384710] ffff88006f3316f0 ffff88006d96ffd8 ffff88006d96ffd8 0000000000011a40
[ 480.385906] 0000000000011a40 ffff88006f3316f0 ffff880078391a40 ffff880078588248
[ 480.387132] Call Trace:
[ 480.388328] [<ffffffff810088f4>] ? read_tsc+0x9/0x19
[ 480.389556] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.390760] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.391954] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 480.393162] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 480.394355] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 480.395558] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 480.396755] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 480.397944] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 480.399143] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 480.400346] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 480.401571] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 480.402774] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 480.403975] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 480.405187] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 480.406388] [<ffffffff8102f3c3>] ? finish_task_switch+0x89/0xb4
[ 480.407583] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 480.408761] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 480.409920] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 480.411054] INFO: task auditd:982 blocked for more than 120 seconds.
[ 480.412192] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 480.413371] auditd D ffff88007838d980 0 982 1 0x00000000
[ 480.414541] ffff88006de2fb48 0000000000000082 ffff8801004c96f0 ffffffffffffffff
[ 480.415739] ffff88006cceade0 ffff88006de2ffd8 ffff88006de2ffd8 0000000000011a40
[ 480.416962] 0000000000011a40 ffff88006cceade0 ffff880078391a40 000000017858bc08
[ 480.418191] Call Trace:
[ 480.419383] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.420617] [<ffffffff810b1879>] ? sync_page+0x0/0x4f
[ 480.421826] [<ffffffff81446710>] io_schedule+0x47/0x62
[ 480.423017] [<ffffffff810b18c4>] sync_page+0x4b/0x4f
[ 480.424224] [<ffffffff81446cdd>] __wait_on_bit+0x48/0x7b
[ 480.425424] [<ffffffff810b1a41>] wait_on_page_bit+0x72/0x74
[ 480.426621] [<ffffffff81058df3>] ? wake_bit_function+0x0/0x31
[ 480.427839] [<ffffffff810b1a5a>] wait_on_page_locked+0x17/0x19
[ 480.429042] [<ffffffff810b330e>] __lock_page_or_retry+0x30/0x36
[ 480.430240] [<ffffffff810b34fb>] filemap_fault+0x1e7/0x36c
[ 480.431438] [<ffffffff810caf47>] __do_fault+0x56/0x342
[ 480.432623] [<ffffffff8102c72b>] ? set_next_entity+0x46/0x98
[ 480.433817] [<ffffffff810cd41f>] handle_pte_fault+0x29f/0x765
[ 480.435016] [<ffffffff8109e4d6>] ? trace_hardirqs_on+0x9/0x20
[ 480.436217] [<ffffffff810cdbe4>] handle_mm_fault+0x18e/0x1a1
[ 480.437409] [<ffffffff8144b490>] do_page_fault+0x34a/0x36c
[ 480.438581] [<ffffffff81127b07>] ? sys_epoll_wait+0x2e4/0x302
[ 480.439737] [<ffffffff8109e183>] ? trace_hardirqs_off_caller+0xe/0x22
[ 480.440889] [<ffffffff81447e6e>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[ 480.442022] [<ffffffff8144878f>] page_fault+0x1f/0x30
[ 503.688962] BUG: soft lockup - CPU#2 stuck for 67s! [kswapd0:35]
[ 503.689824] Modules linked in: netconsole configfs tcp_lp fuse tun cpufreq_ondemand rfcomm sco bnep l2cap ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ipt_MASQUERADE ip6_tables iptable_nat nf_nat coretemp snd_hda_codec_hdmi snd_hda_codec_conexant btusb bluetooth arc4 ecb snd_hda_intel iwlagn snd_hda_codec i2c_i801 iwlcore thinkpad_acpi iTCO_wdt mac80211 snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer iTCO_vendor_support cfg80211 snd_page_alloc tpm_tis hwmon tpm tpm_bios wmi microcode snd soundcore kvm_intel kvm ipv6 aes_x86_64 aes_generic xts dm_crypt sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 503.694990] CPU 2
[ 503.695002] Modules linked in: netconsole configfs tcp_lp fuse tun cpufreq_ondemand rfcomm sco bnep l2cap ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ipt_MASQUERADE ip6_tables iptable_nat nf_nat coretemp snd_hda_codec_hdmi snd_hda_codec_conexant btusb bluetooth arc4 ecb snd_hda_intel iwlagn snd_hda_codec i2c_i801 iwlcore thinkpad_acpi iTCO_wdt mac80211 snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer iTCO_vendor_support cfg80211 snd_page_alloc tpm_tis hwmon tpm tpm_bios wmi microcode snd soundcore kvm_intel kvm ipv6 aes_x86_64 aes_generic xts dm_crypt sdhci_pci sdhci mmc_core i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 503.702037]
[ 503.703226] Pid: 35, comm: kswapd0 Not tainted 2.6.38.6-luto+ #1 LENOVO 4286CTO/4286CTO
[ 503.704457] RIP: 0010:[<ffffffff810b58cc>] [<ffffffff810b58cc>] zone_watermark_ok_safe+0x61/0xae
[ 503.705695] RSP: 0018:ffff88006c885da0 EFLAGS: 00000297
[ 503.706935] RAX: 0000000000000003 RBX: ffff88006c885de0 RCX: 0000000000000000
[ 503.708189] RDX: 000000000000004f RSI: 0000000000000000 RDI: ffff880078380000
[ 503.709432] RBP: ffff88006c885de0 R08: 0000000000000000 R09: ffffffff81a7d140
[ 503.710681] R10: 0000000000000002 R11: ffff8801005ffeb8 R12: ffffffff810033ce
[ 503.711931] R13: ffff88006c885e58 R14: ffff88006c884000 R15: 00000000000050d5
[ 503.713188] FS: 0000000000000000(0000) GS:ffff880078300000(0000) knlGS:0000000000000000
[ 503.714439] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 503.715679] CR2: 00007fed187b3000 CR3: 000000005422f000 CR4: 00000000000406e0
[ 503.716918] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 503.718158] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 503.719391] Process kswapd0 (pid: 35, threadinfo ffff88006c884000, task ffff88006f83dbc0)
[ 503.720632] Stack:
[ 503.721858] ffff8801005ffeb8 ffff880100000000 ffff880000000000 000000000000004f
[ 503.723141] ffff8801005fee00 ffff8801005fe000 0000000000000002 0000000000000000
[ 503.724419] ffff88006c885ee0 ffffffff810c0845 ffff88006c885e40 ffff88006f83dbc0
[ 503.725707] Call Trace:
[ 503.726974] [<ffffffff810c0845>] kswapd+0x3f6/0x76f
[ 503.728253] [<ffffffff810c044f>] ? kswapd+0x0/0x76f
[ 503.729522] [<ffffffff810586f0>] kthread+0x84/0x8c
[ 503.730793] [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
[ 503.732065] [<ffffffff814485ad>] ? restore_args+0x0/0x30
[ 503.733336] [<ffffffff8105866c>] ? kthread+0x0/0x8c
[ 503.734601] [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
[ 503.735875] Code: c1 73 5d 4c 8b af 30 05 00 00 83 c8 ff 4c 8b 35 4b 56 55 00 eb 18 48 63 f8 48 8b 73 58 48 8b 3c fd d0 cc a7 81 48 0f be 74 3e 42 <49> 01 f5 4c 89 f6 89 c7 48 89 55 d8 89 4d d0 44 89 45 c8 e8 cd
[ 503.738776] Call Trace:
[ 503.740126] [<ffffffff810c0845>] kswapd+0x3f6/0x76f
[ 503.741490] [<ffffffff810c044f>] ? kswapd+0x0/0x76f
[ 503.742853] [<ffffffff810586f0>] kthread+0x84/0x8c
[ 503.744208] [<ffffffff81003814>] kernel_thread_helper+0x4/0x10
[ 503.745564] [<ffffffff814485ad>] ? restore_args+0x0/0x30
[ 503.746919] [<ffffffff8105866c>] ? kthread+0x0/0x8c
[ 503.748267] [<ffffffff81003810>] ? kernel_thread_helper+0x0/0x10
next prev parent reply other threads:[~2011-05-12 11:54 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-05-11 22:42 Kernel falls apart under light memory pressure (i.e. linking vmlinux) Andrew Lutomirski
2011-05-11 23:07 ` Andi Kleen
2011-05-11 23:28 ` Andrew Lutomirski
2011-05-12 5:46 ` Andi Kleen
2011-05-12 11:54 ` Andrew Lutomirski [this message]
2011-05-14 15:46 ` Andrew Lutomirski
2011-05-14 16:53 ` Andi Kleen
[not found] ` <BANLkTik6SS9NH7XVSRBoCR16_5veY0MKBw@mail.gmail.com>
[not found] ` <20110514174333.GW6008@one.firstfloor.org>
[not found] ` <BANLkTinst+Ryox9VZ-s7gdXKa574XXqt5w@mail.gmail.com>
2011-05-15 15:27 ` Wu Fengguang
2011-05-15 15:59 ` Andrew Lutomirski
2011-05-15 22:58 ` Minchan Kim
2011-05-16 8:51 ` Mel Gorman
2011-05-15 16:12 ` Andrew Lutomirski
2011-05-17 6:00 ` Wu Fengguang
2011-05-17 6:35 ` Minchan Kim
2011-05-17 19:22 ` Andrew Lutomirski
2011-05-18 5:17 ` Minchan Kim
2011-05-19 2:15 ` Andrew Lutomirski
2011-05-19 2:30 ` KAMEZAWA Hiroyuki
2011-05-19 2:41 ` Andrew Lutomirski
2011-05-19 2:54 ` Minchan Kim
2011-05-19 14:16 ` Andrew Lutomirski
2011-05-20 0:17 ` Minchan Kim
2011-05-20 2:58 ` Andrew Lutomirski
2011-05-20 3:12 ` KOSAKI Motohiro
2011-05-20 3:38 ` Andrew Lutomirski
2011-05-20 4:20 ` Minchan Kim
2011-05-20 5:08 ` KAMEZAWA Hiroyuki
2011-05-20 5:36 ` Minchan Kim
2011-05-20 7:43 ` KAMEZAWA Hiroyuki
2011-05-20 10:11 ` Andrea Arcangeli
2011-05-20 14:11 ` Andrew Lutomirski
2011-05-20 15:33 ` Minchan Kim
2011-05-20 16:01 ` Andrew Lutomirski
2011-05-20 16:19 ` Minchan Kim
2011-05-20 18:09 ` Andrew Lutomirski
2011-05-20 18:40 ` Andrew Lutomirski
2011-05-21 12:04 ` KOSAKI Motohiro
2011-05-21 13:34 ` Andrew Lutomirski
2011-05-21 14:14 ` KOSAKI Motohiro
2011-05-21 14:44 ` Minchan Kim
2011-05-22 12:22 ` Andrew Lutomirski
2011-05-22 23:12 ` Minchan Kim
2011-05-23 16:42 ` Andrea Arcangeli
2011-05-23 17:35 ` Mel Gorman
2011-05-24 1:19 ` Andrew Lutomirski
2011-05-24 1:34 ` Minchan Kim
2011-05-24 11:24 ` Andrew Lutomirski
2011-05-24 11:55 ` Andrew Lutomirski
2011-05-25 0:43 ` KOSAKI Motohiro
2011-05-21 14:31 ` Minchan Kim
2011-05-19 14:51 ` Wu Fengguang
2011-05-19 15:00 ` Andrew Lutomirski
2011-05-20 0:20 ` Minchan Kim
2011-05-15 22:40 ` Minchan Kim
2011-05-17 5:52 ` Wu Fengguang
2011-05-17 6:26 ` Minchan Kim
2011-05-20 10:40 ` Andrea Arcangeli
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='BANLkTi=fk3DUT9cYd2gAzC98c69F6HXX7g@mail.gmail.com' \
--to=luto@mit.edu \
--cc=andi@firstfloor.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).