* sync-Regression in 2.6.28.2? @ 2009-01-27 9:35 Ralf Hildebrandt 2009-01-27 21:09 ` Federico Cuello 0 siblings, 1 reply; 14+ messages in thread From: Ralf Hildebrandt @ 2009-01-27 9:35 UTC (permalink / raw) To: linux-kernel; +Cc: Udo Wolter, cf I recently installed 2.6.28.2 on our postfix/dovecot-based mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there without a hitch. Now with 2.6.28.2 I had two major lockups: All writes to the users' Maildirs (on ext4) would stall, the load would rise, "sync" would never return. I had to "reboot -f -n" to get the machine back. All hanging processes were unkillable, even with kill -9. There was NOTHING in dmesg, nothing in the log either. No indication of filesystem errors or anything. Lookin at the http://www.eu.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.28.2 I see changes to fs: sys_sync fix fs: sync_sb_inodes fix fs: remove WB_SYNC_HOLD Since the mailserver uses sync() extensively (both dovecot and postfix), maybe that's the culprit? Note: I'm really just guessing here! Back to 2.6.28.1 for now. -- Ralf Hildebrandt Ralf.Hildebrandt@charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962 Hindenburgdamm 30 | 12200 Berlin ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-01-27 9:35 sync-Regression in 2.6.28.2? Ralf Hildebrandt @ 2009-01-27 21:09 ` Federico Cuello 2009-02-03 1:09 ` Nick Piggin 0 siblings, 1 reply; 14+ messages in thread From: Federico Cuello @ 2009-01-27 21:09 UTC (permalink / raw) To: linux-kernel, Udo Wolter, cf Ralf Hildebrandt escribió: > I recently installed 2.6.28.2 on our postfix/dovecot-based > mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there > without a hitch. > > Now with 2.6.28.2 I had two major lockups: All writes to the users' > Maildirs (on ext4) would stall, the load would rise, "sync" would never > return. > > I had to "reboot -f -n" to get the machine back. All hanging processes > were unkillable, even with kill -9. > [...] The same is happening to me, but I have some logs taken with sysrq. Here is my vmstat output: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 2 99028 46536 26356 1569400 0 0 0 0 1400 437 0 3 0 96 0 2 99028 46536 26356 1569400 0 0 0 0 1344 355 0 6 0 94 0 2 99028 46536 26356 1569400 0 0 0 0 1373 387 0 0 0 100 0 2 99028 46536 26364 1569400 0 0 0 12 1403 384 1 0 0 99 0 2 99028 46536 26364 1569400 0 0 0 0 1370 378 0 0 0 100 0 2 99028 46536 26364 1569400 0 0 0 0 1351 346 0 0 0 100 0 2 99028 46536 26364 1569400 0 0 0 0 1395 412 0 0 0 100 0 2 99028 46536 26364 1569400 0 0 0 0 1349 332 0 0 0 100 0 2 99028 46536 26368 1569400 0 0 4 0 1407 387 0 0 0 100 Notice the 100% iowait. I also managed to reproduce it doing a rsync from one partition to a USB drive. After the lockup I can't read any file from the source partition, but the other partitions can be accessed normally. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-01-27 21:09 ` Federico Cuello @ 2009-02-03 1:09 ` Nick Piggin 2009-02-03 19:51 ` Federico Cuello 0 siblings, 1 reply; 14+ messages in thread From: Nick Piggin @ 2009-02-03 1:09 UTC (permalink / raw) To: Federico Cuello; +Cc: linux-kernel, Udo Wolter, cf On Wednesday 28 January 2009 08:09:10 Federico Cuello wrote: > Ralf Hildebrandt escribió: > > I recently installed 2.6.28.2 on our postfix/dovecot-based > > mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there > > without a hitch. > > > > Now with 2.6.28.2 I had two major lockups: All writes to the users' > > Maildirs (on ext4) would stall, the load would rise, "sync" would never > > return. > > > > I had to "reboot -f -n" to get the machine back. All hanging processes > > were unkillable, even with kill -9. > > [...] > > The same is happening to me, but I have some logs taken with sysrq. > > Here is my vmstat output: > > procs -----------memory---------- ---swap-- -----io---- -system-- > ----cpu---- > r b swpd free buff cache si so bi bo in cs us sy > id wa > 0 2 99028 46536 26356 1569400 0 0 0 0 1400 437 0 > 3 0 96 > 0 2 99028 46536 26356 1569400 0 0 0 0 1344 355 0 > 6 0 94 > 0 2 99028 46536 26356 1569400 0 0 0 0 1373 387 0 > 0 0 100 > 0 2 99028 46536 26364 1569400 0 0 0 12 1403 384 1 > 0 0 99 > 0 2 99028 46536 26364 1569400 0 0 0 0 1370 378 0 > 0 0 100 > 0 2 99028 46536 26364 1569400 0 0 0 0 1351 346 0 > 0 0 100 > 0 2 99028 46536 26364 1569400 0 0 0 0 1395 412 0 > 0 0 100 > 0 2 99028 46536 26364 1569400 0 0 0 0 1349 332 0 > 0 0 100 > 0 2 99028 46536 26368 1569400 0 0 4 0 1407 387 0 > 0 0 100 > > Notice the 100% iowait. > > I also managed to reproduce it doing a rsync from one partition to a USB > drive. After the lockup I can't read any file from the source partition, > but the other partitions can be accessed normally. Hm, thanks for reporting, can you guys get a sysrq+W trace when the system reaches this state? Thanks, Nick ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-03 1:09 ` Nick Piggin @ 2009-02-03 19:51 ` Federico Cuello 2009-02-04 6:17 ` Nick Piggin 0 siblings, 1 reply; 14+ messages in thread From: Federico Cuello @ 2009-02-03 19:51 UTC (permalink / raw) To: linux-kernel Nick Piggin wrote: > On Wednesday 28 January 2009 08:09:10 Federico Cuello wrote: > >> Ralf Hildebrandt escribió: >> >>> I recently installed 2.6.28.2 on our postfix/dovecot-based >>> mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there >>> without a hitch. >>> >>> Now with 2.6.28.2 I had two major lockups: All writes to the users' >>> Maildirs (on ext4) would stall, the load would rise, "sync" would never >>> return. >>> >>> I had to "reboot -f -n" to get the machine back. All hanging processes >>> were unkillable, even with kill -9. >>> [...] >>> >> The same is happening to me, but I have some logs taken with sysrq. >> >> Here is my vmstat output: >> >> procs -----------memory---------- ---swap-- -----io---- -system-- >> ----cpu---- >> r b swpd free buff cache si so bi bo in cs us sy >> id wa >> 0 2 99028 46536 26356 1569400 0 0 0 0 1400 437 0 >> 3 0 96 >> 0 2 99028 46536 26356 1569400 0 0 0 0 1344 355 0 >> 6 0 94 >> 0 2 99028 46536 26356 1569400 0 0 0 0 1373 387 0 >> 0 0 100 >> 0 2 99028 46536 26364 1569400 0 0 0 12 1403 384 1 >> 0 0 99 >> 0 2 99028 46536 26364 1569400 0 0 0 0 1370 378 0 >> 0 0 100 >> 0 2 99028 46536 26364 1569400 0 0 0 0 1351 346 0 >> 0 0 100 >> 0 2 99028 46536 26364 1569400 0 0 0 0 1395 412 0 >> 0 0 100 >> 0 2 99028 46536 26364 1569400 0 0 0 0 1349 332 0 >> 0 0 100 >> 0 2 99028 46536 26368 1569400 0 0 4 0 1407 387 0 >> 0 0 100 >> >> Notice the 100% iowait. >> >> I also managed to reproduce it doing a rsync from one partition to a USB >> drive. After the lockup I can't read any file from the source partition, >> but the other partitions can be accessed normally. >> > > Hm, thanks for reporting, can you guys get a sysrq+W trace when the system > reaches this state? Here it is, if you need something else please ask: Jan 26 23:44:38 kaoz [10117.763822] SysRq : Show Memory Jan 26 23:44:38 kaoz [10117.765692] Mem-Info: Jan 26 23:44:38 kaoz [10117.767141] DMA per-cpu: Jan 26 23:44:38 kaoz [10117.769184] CPU 0: hi: 0, btch: 1 usd: 0 Jan 26 23:44:38 kaoz [10117.769184] CPU 1: hi: 0, btch: 1 usd: 0 Jan 26 23:44:38 kaoz [10117.769184] DMA32 per-cpu: Jan 26 23:44:38 kaoz [10117.769184] CPU 0: hi: 186, btch: 31 usd: 42 Jan 26 23:44:38 kaoz [10117.769184] CPU 1: hi: 186, btch: 31 usd: 31 Jan 26 23:44:38 kaoz [10117.769184] Active_anon:24752 active_file:11041 inactive_anon:10001 Jan 26 23:44:38 kaoz [10117.769184] inactive_file:396899 unevictable:0 dirty:70 writeback:0 unstable:0 Jan 26 23:44:38 kaoz [10117.769184] free:43547 slab:15243 mapped:7057 pagetables:1711 bounce:0 Jan 26 23:44:38 kaoz [10117.769184] DMA free:8092kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:12kB active_file:2360kB inactive_file:1324kB unevictable:0kB present:11100kB pages_scanned:0 all_unreclaimable? no Jan 26 23:44:38 kaoz [10117.785214] lowmem_reserve[]: 0 2004 2004 2004 Jan 26 23:44:38 kaoz [10117.785214] DMA32 free:165928kB min:5712kB low:7140kB high:8568kB active_anon:99008kB inactive_anon:39992kB active_file:41804kB inactive_file:1586396kB unevictable:0kB present:2052128kB pages_scanned:0 all_unreclaimable? no Jan 26 23:44:38 kaoz [10117.785214] lowmem_reserve[]: 0 0 0 0 Jan 26 23:44:38 kaoz [10117.785214] DMA: 15*4kB 24*8kB 20*16kB 21*32kB 19*64kB 10*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 8092kB Jan 26 23:44:38 kaoz [10117.785214] DMA32: 5284*4kB 5258*8kB 995*16kB 486*32kB 336*64kB 127*128kB 40*256kB 19*512kB 9*1024kB 2*2048kB 0*4096kB = 165712kB Jan 26 23:44:38 kaoz [10117.785214] 409973 total pagecache pages Jan 26 23:44:38 kaoz [10117.785214] 1917 pages in swap cache Jan 26 23:44:38 kaoz [10117.785214] Swap cache stats: add 38607, delete 36690, find 9693/11511 Jan 26 23:44:38 kaoz [10117.785214] Free swap = 1037804kB Jan 26 23:44:38 kaoz [10117.785214] Total swap = 1048568kB Jan 26 23:44:38 kaoz [10117.785214] 524240 pages RAM Jan 26 23:44:38 kaoz [10117.785214] 9080 pages reserved Jan 26 23:44:38 kaoz [10117.785214] 192696 pages shared Jan 26 23:44:38 kaoz [10117.785214] 286249 pages non-shared Jan 26 23:44:38 kaoz [10123.434993] SysRq : Emergency Remount R/O Jan 26 23:44:38 kaoz [10127.036454] SysRq : HELP : loglevel0-8 reBoot tErm Full kIll saK aLlcpus showMem Nice powerOff showPc show-all-timers(Q) unRaw Sync showTasks Unmount shoW-blocked-tasks Jan 26 23:44:38 kaoz [10136.390066] SysRq : Show Blocked State Jan 26 23:44:38 kaoz [10136.391430] task PC stack pid father Jan 26 23:44:38 kaoz [10136.392801] kjournald2 D ffff880001014c20 0 4635 2 Jan 26 23:44:38 kaoz [10136.393391] ffff88007e5e7d50 0000000000000046 ffff8800581e9d00 ffff88007e5e38a0 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88007ea6ad80 ffffffff80544340 ffff88007ea6afe8 Jan 26 23:44:38 kaoz [10136.393391] 0000000000000000 000000010023bf96 ffff88007ea6afe8 ffff88007c5df090 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022e140>] update_curr+0xd0/0x140 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d561>] jbd2_journal_commit_transaction+0x171/0x1240 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80474ef7>] thread_return+0x3d/0x5e6 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80246204>] lock_timer_base+0x34/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80246294>] try_to_del_timer_sync+0x54/0x60 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80322548>] kjournald2+0xd8/0x230 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80322470>] kjournald2+0x0/0x230 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802519b7>] kthread+0x47/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802386c7>] schedule_tail+0x27/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7e9>] child_rip+0xa/0x11 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251970>] kthread+0x0/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7df>] child_rip+0x0/0x11 Jan 26 23:44:38 kaoz [10136.393391] quasselcore D ffff88000100ad98 0 6054 1 Jan 26 23:44:38 kaoz [10136.393391] ffff88006d2e5bb8 0000000000000082 ffff88007c5dec00 ffffffff80496fd0 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88007c52b8e0 ffff88005597c9f0 ffff88007c52bb48 Jan 26 23:44:38 kaoz [10136.393391] 0000000100000004 000000010023b9b0 ffff88007c52bb48 ffffffff802fa6a1 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fa6a1>] ext4_da_write_end+0x141/0x280 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod] Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f940>] sync_page+0x0/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff804754d7>] io_schedule+0x37/0x50 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f975>] sync_page+0x35/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8047585a>] __wait_on_bit_lock+0x4a/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f91e>] __lock_page+0x5e/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251db0>] wake_bit_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276d1d>] write_cache_pages+0x1ed/0x3d0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fbea0>] __mpage_da_writepage+0x0/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fbc57>] ext4_da_writepages+0x237/0x3b0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fd1b0>] ext4_da_get_block_write+0x0/0x160 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276f50>] do_writepages+0x20/0x40 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8027077d>] __filemap_fdatawrite_range+0x4d/0x60 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c12c5>] do_fsync+0x45/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c138e>] __do_fsync+0x2e/0x50 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] klauncher D ffff880001020c20 0 7760 1 Jan 26 23:44:38 kaoz [10136.393391] ffff880060b65cf0 0000000000000086 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8cc9f0 ffff88007f862220 ffff88005b8ccc58 Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 00000001002ce821 ffff88005b8ccc58 ffff880060b65cc8 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f783>] __wake_up+0x43/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802476c0>] do_sigaction+0x130/0x1c0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] kded4 D ffff880001020c20 0 7762 1 Jan 26 23:44:38 kaoz [10136.393391] ffff880060999cf0 0000000000000086 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8b8000 ffff88007f862220 ffff88005b8b8268 Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 00000001002c4165 ffff88005b8b8268 ffff880060999e08 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8028eda5>] free_pages_and_swap_cache+0x85/0xb0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802867a0>] unmap_region+0x180/0x1a0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] ksmserver D ffff8800583319c8 0 7773 1 Jan 26 23:44:38 kaoz [10136.393391] ffff88005bb99cf0 0000000000000086 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8bb8e0 ffff88007c52e0b0 ffff88005b8bbb48 Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 ffffe200006b6fe0 ffff88005b8bbb48 ffff88005bb99cc8 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f783>] __wake_up+0x43/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f783>] __wake_up+0x43/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251c60>] bit_waitqueue+0x10/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b0673>] d_kill+0x53/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] kwin D ffff8800583319c8 0 7775 7773 Jan 26 23:44:38 kaoz [10136.393391] ffff88005ba6bcf0 0000000000000082 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8bf1c0 ffff88007c52e660 ffff88005b8bf428 Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 ffffe200008c8618 ffff88005b8bf428 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f783>] __wake_up+0x43/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] nepomukserver D ffff8800583319c8 0 7778 1 Jan 26 23:44:38 kaoz [10136.393391] ffff88007c0bfcf0 0000000000000082 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8b9110 ffff88007c52e660 ffff88005b8b9378 Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 ffffe200005c0370 ffff88005b8b9378 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] plasma D ffff880001020c20 0 7781 1 Jan 26 23:44:38 kaoz [10136.393391] ffff88005b8ddcf0 0000000000000082 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8be660 ffff88007f862220 ffff88005b8be8c8 Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 00000001002c24a1 ffff88005b8be8c8 ffff88005b8dde08 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a3350>] vfs_lstat_fd+0x20/0x60 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] mysqld D 0000000000000002 0 7820 7804 Jan 26 23:44:38 kaoz [10136.393391] ffff880063a39b58 0000000000000082 ffff880063a39ad8 ffffffff80236cbb Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8cb8e0 ffff88005597f770 ffff88005b8cbb48 Jan 26 23:44:38 kaoz [10136.393391] 0000000163a39b98 ffffffff80474ef7 ffff88005b8cbb48 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80236cbb>] finish_task_switch+0x2b/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80474ef7>] thread_return+0x3d/0x5e6 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031cebb>] start_this_handle+0x33b/0x450 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1a6>] jbd2_journal_start+0x96/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f9bc6>] ext4_dirty_inode+0x36/0xb0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802be294>] __mark_inode_dirty+0x34/0x1d0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b33a0>] file_update_time+0x110/0x130 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80270fd6>] __generic_file_aio_write_nolock+0x246/0x470 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80280382>] __do_fault+0x202/0x4c0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b64>] generic_file_aio_write+0x64/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80223b50>] flat_send_IPI_mask+0x0/0x60 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] pdflush D ffff880001014c20 0 9198 2 Jan 26 23:44:38 kaoz [10136.393391] ffff88003c311a20 0000000000000046 ffff88007ea00000 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005597d550 ffffffff80544340 ffff88005597d7b8 Jan 26 23:44:38 kaoz [10136.393391] 0000000000000000 000000010023b9b6 ffff88005597d7b8 ffff88005591d900 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8032f3c7>] generic_make_request+0x207/0x340 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod] Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f940>] sync_page+0x0/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff804754d7>] io_schedule+0x37/0x50 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f975>] sync_page+0x35/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8047585a>] __wait_on_bit_lock+0x4a/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f91e>] __lock_page+0x5e/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251db0>] wake_bit_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802786ca>] pagevec_lookup_tag+0x1a/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276d1d>] write_cache_pages+0x1ed/0x3d0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fbea0>] __mpage_da_writepage+0x0/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031bb5f>] jbd2_journal_stop+0x17f/0x200 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fbc57>] ext4_da_writepages+0x237/0x3b0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022e6a7>] dequeue_task+0x57/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80236cbb>] finish_task_switch+0x2b/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802fd1b0>] ext4_da_get_block_write+0x0/0x160 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276f50>] do_writepages+0x20/0x40 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802bd526>] __writeback_single_inode+0xa6/0x470 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802bdeb0>] generic_sync_sb_inodes+0x370/0x4d0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802be1cd>] writeback_inodes+0x4d/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802775e3>] background_writeout+0xb3/0xf0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277d1b>] pdflush+0x15b/0x240 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277530>] background_writeout+0x0/0xf0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277bc0>] pdflush+0x0/0x240 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802519b7>] kthread+0x47/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802386c7>] schedule_tail+0x27/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7e9>] child_rip+0xa/0x11 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251970>] kthread+0x0/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7df>] child_rip+0x0/0x11 Jan 26 23:44:38 kaoz [10136.393391] krunner_lock D 0000000000000002 0 9309 1 Jan 26 23:44:38 kaoz [10136.393391] ffff88005b843b58 0000000000000082 ffff88005b843ad8 ffffffff8022e140 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88006d3eec10 ffff88007c52a7d0 ffff88006d3eee78 Jan 26 23:44:38 kaoz [10136.393391] 000000015b843b08 ffffffff8022f9f8 ffff88006d3eee78 ffff8800010252f0 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022e140>] update_curr+0xd0/0x140 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f9f8>] dequeue_entity+0x18/0x190 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022f958>] set_next_entity+0x38/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031cebb>] start_this_handle+0x33b/0x450 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1a6>] jbd2_journal_start+0x96/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f9bc6>] ext4_dirty_inode+0x36/0xb0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802be294>] __mark_inode_dirty+0x34/0x1d0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b33a0>] file_update_time+0x110/0x130 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80270fd6>] __generic_file_aio_write_nolock+0x246/0x470 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80280382>] __do_fault+0x202/0x4c0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b64>] generic_file_aio_write+0x64/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8028eda5>] free_pages_and_swap_cache+0x85/0xb0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802867a0>] unmap_region+0x180/0x1a0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] krunner_lock D ffff8800583319c8 0 9310 9309 Jan 26 23:44:38 kaoz [10136.393391] ffff88007fa9bcf0 0000000000000082 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88006d3ee660 ffff8800708d9c70 ffff88006d3ee8c8 Jan 26 23:44:38 kaoz [10136.393391] 00000001805e19e0 ffffe200010d3fb0 ffff88006d3ee8c8 ffff88007fa9bed8 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8024940f>] kill_pid_info+0x4f/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802496c2>] sys_kill+0xa2/0x1c0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] kblankscrn.ks D ffff880001020c20 0 9313 9310 Jan 26 23:44:38 kaoz [10136.393391] ffff88004c291cf0 0000000000000082 ffffe200010d3fb0 ffffffff805e19e0 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88006d3eb8e0 ffff88007f862220 ffff88006d3ebb48 Jan 26 23:44:38 kaoz [10136.393391] 0000000180279245 000000010024fa44 ffff88006d3ebb48 ffffffff80570008 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80286726>] unmap_region+0x106/0x1a0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] less D ffff880001014c20 0 9633 1 Jan 26 23:44:38 kaoz [10136.393391] ffff88003ba2bec0 0000000000000086 ffff88003ba2be68 0000000000008000 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff8800708d96c0 ffffffff80544340 ffff8800708d9928 Jan 26 23:44:38 kaoz [10136.393391] 0000000000000001 0000000100282d66 ffff8800708d9928 000000000000fe03 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a06ff>] generic_file_llseek+0x2f/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029f31e>] sys_lseek+0x7e/0x90 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029da9b>] do_sys_open+0xeb/0x110 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] pdflush D ffffffff80552468 0 9675 2 Jan 26 23:44:38 kaoz [10136.393391] ffff880041ac9e38 0000000000000046 ffffffff80270658 ffff880041ac9dc0 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8bbe90 ffff88007f863330 ffff88005b8bc0f8 Jan 26 23:44:38 kaoz [10136.393391] 000000017f85ff00 ffff880041ac9de0 ffff88005b8bc0f8 ffffffff805e5b80 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80270658>] wait_on_page_writeback_range+0x78/0x130 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80276f50>] do_writepages+0x20/0x40 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a15c3>] sync_filesystems+0x13/0x130 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c145e>] do_sync+0x1e/0x60 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277d1b>] pdflush+0x15b/0x240 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c1440>] do_sync+0x0/0x60 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80277bc0>] pdflush+0x0/0x240 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802519b7>] kthread+0x47/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802386c7>] schedule_tail+0x27/0x70 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7e9>] child_rip+0xa/0x11 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251970>] kthread+0x0/0x80 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020c7df>] child_rip+0x0/0x11 Jan 26 23:44:38 kaoz [10136.393391] shutdown D ffff880001020c20 0 10071 1 Jan 26 23:44:38 kaoz [10136.393391] ffff88003ac5de68 0000000000000086 ffff88003ac5de78 ffffffff80474ef7 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005b8bad80 ffff88007f862220 ffff88005b8bafe8 Jan 26 23:44:38 kaoz [10136.393391] 000000017f862220 00000001002c243e ffff88005b8bafe8 00000001002c2435 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80474ef7>] thread_return+0x3d/0x5e6 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031cebb>] start_this_handle+0x33b/0x450 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1a6>] jbd2_journal_start+0x96/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1fe>] jbd2_journal_force_commit+0xe/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a1687>] sync_filesystems+0xd7/0x130 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c1465>] do_sync+0x25/0x60 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802c14ae>] sys_sync+0xe/0x20 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10136.393391] kdm D ffff880001014c20 0 10137 6943 Jan 26 23:44:38 kaoz [10136.393391] ffff88005b953cd8 0000000000000086 ffffffff80570008 ffff880062e76a28 Jan 26 23:44:38 kaoz [10136.393391] ffffffff805de000 ffff88005bbf5b00 ffffffff80544340 ffff88005bbf5d68 Jan 26 23:44:38 kaoz [10136.393391] 0000000000000012 00000001002ce9e5 ffff88005bbf5d68 ffffffff805e19e0 Jan 26 23:44:38 kaoz [10136.393391] Call Trace: Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b2557>] __d_lookup+0xc7/0x140 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031cebb>] start_this_handle+0x33b/0x450 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8031d1a6>] jbd2_journal_start+0x96/0xe0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff80301cf5>] ext4_create+0x85/0x120 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a904e>] vfs_create+0xae/0xf0 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802ac836>] do_filp_open+0x856/0x960 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8022e140>] update_curr+0xd0/0x140 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802a33c3>] vfs_stat_fd+0x23/0x60 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff802b5c42>] alloc_fd+0x122/0x150 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8029da30>] do_sys_open+0x80/0x110 Jan 26 23:44:38 kaoz [10136.393391] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.340218] SysRq : Show Blocked State Jan 26 23:44:38 kaoz [10141.341232] task PC stack pid father Jan 26 23:44:38 kaoz [10141.342274] kjournald2 D ffff880001014c20 0 4635 2 Jan 26 23:44:38 kaoz [10141.343331] ffff88007e5e7d50 0000000000000046 ffff8800581e9d00 ffff88007e5e38a0 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88007ea6ad80 ffffffff80544340 ffff88007ea6afe8 Jan 26 23:44:38 kaoz [10141.343544] 0000000000000000 000000010023bf96 ffff88007ea6afe8 ffff88007c5df090 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022e140>] update_curr+0xd0/0x140 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d561>] jbd2_journal_commit_transaction+0x171/0x1240 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80474ef7>] thread_return+0x3d/0x5e6 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80246204>] lock_timer_base+0x34/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80246294>] try_to_del_timer_sync+0x54/0x60 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80322548>] kjournald2+0xd8/0x230 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80322470>] kjournald2+0x0/0x230 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802519b7>] kthread+0x47/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802386c7>] schedule_tail+0x27/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7e9>] child_rip+0xa/0x11 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251970>] kthread+0x0/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7df>] child_rip+0x0/0x11 Jan 26 23:44:38 kaoz [10141.343544] quasselcore D ffff88000100ad98 0 6054 1 Jan 26 23:44:38 kaoz [10141.343544] ffff88006d2e5bb8 0000000000000082 ffff88007c5dec00 ffffffff80496fd0 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88007c52b8e0 ffff88005597c9f0 ffff88007c52bb48 Jan 26 23:44:38 kaoz [10141.343544] 0000000100000004 000000010023b9b0 ffff88007c52bb48 ffffffff802fa6a1 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fa6a1>] ext4_da_write_end+0x141/0x280 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod] Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f940>] sync_page+0x0/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff804754d7>] io_schedule+0x37/0x50 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f975>] sync_page+0x35/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8047585a>] __wait_on_bit_lock+0x4a/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f91e>] __lock_page+0x5e/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251db0>] wake_bit_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276d1d>] write_cache_pages+0x1ed/0x3d0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fbea0>] __mpage_da_writepage+0x0/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fbc57>] ext4_da_writepages+0x237/0x3b0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fd1b0>] ext4_da_get_block_write+0x0/0x160 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276f50>] do_writepages+0x20/0x40 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8027077d>] __filemap_fdatawrite_range+0x4d/0x60 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c12c5>] do_fsync+0x45/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c138e>] __do_fsync+0x2e/0x50 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] klauncher D ffff880001020c20 0 7760 1 Jan 26 23:44:38 kaoz [10141.343544] ffff880060b65cf0 0000000000000086 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8cc9f0 ffff88007f862220 ffff88005b8ccc58 Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 00000001002ce821 ffff88005b8ccc58 ffff880060b65cc8 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f783>] __wake_up+0x43/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802476c0>] do_sigaction+0x130/0x1c0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] kded4 D ffff880001020c20 0 7762 1 Jan 26 23:44:38 kaoz [10141.343544] ffff880060999cf0 0000000000000086 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8b8000 ffff88007f862220 ffff88005b8b8268 Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 00000001002c4165 ffff88005b8b8268 ffff880060999e08 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8028eda5>] free_pages_and_swap_cache+0x85/0xb0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802867a0>] unmap_region+0x180/0x1a0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] ksmserver D ffff8800583319c8 0 7773 1 Jan 26 23:44:38 kaoz [10141.343544] ffff88005bb99cf0 0000000000000086 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8bb8e0 ffff88007c52e0b0 ffff88005b8bbb48 Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 ffffe200006b6fe0 ffff88005b8bbb48 ffff88005bb99cc8 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f783>] __wake_up+0x43/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f783>] __wake_up+0x43/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251c60>] bit_waitqueue+0x10/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b0673>] d_kill+0x53/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] kwin D ffff8800583319c8 0 7775 7773 Jan 26 23:44:38 kaoz [10141.343544] ffff88005ba6bcf0 0000000000000082 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8bf1c0 ffff88007c52e660 ffff88005b8bf428 Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 ffffe200008c8618 ffff88005b8bf428 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f783>] __wake_up+0x43/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] nepomukserver D ffff8800583319c8 0 7778 1 Jan 26 23:44:38 kaoz [10141.343544] ffff88007c0bfcf0 0000000000000082 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8b9110 ffff88007c52e660 ffff88005b8b9378 Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 ffffe200005c0370 ffff88005b8b9378 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] plasma D ffff880001020c20 0 7781 1 Jan 26 23:44:38 kaoz [10141.343544] ffff88005b8ddcf0 0000000000000082 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8be660 ffff88007f862220 ffff88005b8be8c8 Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 00000001002c24a1 ffff88005b8be8c8 ffff88005b8dde08 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a3350>] vfs_lstat_fd+0x20/0x60 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] mysqld D 0000000000000002 0 7820 7804 Jan 26 23:44:38 kaoz [10141.343544] ffff880063a39b58 0000000000000082 ffff880063a39ad8 ffffffff80236cbb Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8cb8e0 ffff88005597f770 ffff88005b8cbb48 Jan 26 23:44:38 kaoz [10141.343544] 0000000163a39b98 ffffffff80474ef7 ffff88005b8cbb48 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80236cbb>] finish_task_switch+0x2b/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80474ef7>] thread_return+0x3d/0x5e6 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031cebb>] start_this_handle+0x33b/0x450 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1a6>] jbd2_journal_start+0x96/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f9bc6>] ext4_dirty_inode+0x36/0xb0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802be294>] __mark_inode_dirty+0x34/0x1d0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b33a0>] file_update_time+0x110/0x130 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80270fd6>] __generic_file_aio_write_nolock+0x246/0x470 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80280382>] __do_fault+0x202/0x4c0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b64>] generic_file_aio_write+0x64/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80223b50>] flat_send_IPI_mask+0x0/0x60 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] pdflush D ffff880001014c20 0 9198 2 Jan 26 23:44:38 kaoz [10141.343544] ffff88003c311a20 0000000000000046 ffff88007ea00000 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005597d550 ffffffff80544340 ffff88005597d7b8 Jan 26 23:44:38 kaoz [10141.343544] 0000000000000000 000000010023b9b6 ffff88005597d7b8 ffff88005591d900 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8032f3c7>] generic_make_request+0x207/0x340 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod] Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f940>] sync_page+0x0/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff804754d7>] io_schedule+0x37/0x50 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f975>] sync_page+0x35/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8047585a>] __wait_on_bit_lock+0x4a/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f91e>] __lock_page+0x5e/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251db0>] wake_bit_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802786ca>] pagevec_lookup_tag+0x1a/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276d1d>] write_cache_pages+0x1ed/0x3d0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fbea0>] __mpage_da_writepage+0x0/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031bb5f>] jbd2_journal_stop+0x17f/0x200 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fbc57>] ext4_da_writepages+0x237/0x3b0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022e6a7>] dequeue_task+0x57/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80236cbb>] finish_task_switch+0x2b/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802fd1b0>] ext4_da_get_block_write+0x0/0x160 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276f50>] do_writepages+0x20/0x40 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802bd526>] __writeback_single_inode+0xa6/0x470 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802bdeb0>] generic_sync_sb_inodes+0x370/0x4d0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802be1cd>] writeback_inodes+0x4d/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802775e3>] background_writeout+0xb3/0xf0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277d1b>] pdflush+0x15b/0x240 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277530>] background_writeout+0x0/0xf0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277bc0>] pdflush+0x0/0x240 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802519b7>] kthread+0x47/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802386c7>] schedule_tail+0x27/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7e9>] child_rip+0xa/0x11 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251970>] kthread+0x0/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7df>] child_rip+0x0/0x11 Jan 26 23:44:38 kaoz [10141.343544] krunner_lock D 0000000000000002 0 9309 1 Jan 26 23:44:38 kaoz [10141.343544] ffff88005b843b58 0000000000000082 ffff88005b843ad8 ffffffff8022e140 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88006d3eec10 ffff88007c52a7d0 ffff88006d3eee78 Jan 26 23:44:38 kaoz [10141.343544] 000000015b843b08 ffffffff8022f9f8 ffff88006d3eee78 ffff8800010252f0 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022e140>] update_curr+0xd0/0x140 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f9f8>] dequeue_entity+0x18/0x190 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022f958>] set_next_entity+0x38/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031cebb>] start_this_handle+0x33b/0x450 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1a6>] jbd2_journal_start+0x96/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f9bc6>] ext4_dirty_inode+0x36/0xb0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802be294>] __mark_inode_dirty+0x34/0x1d0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b33a0>] file_update_time+0x110/0x130 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80270fd6>] __generic_file_aio_write_nolock+0x246/0x470 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80280382>] __do_fault+0x202/0x4c0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b64>] generic_file_aio_write+0x64/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8028eda5>] free_pages_and_swap_cache+0x85/0xb0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802867a0>] unmap_region+0x180/0x1a0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] krunner_lock D ffff8800583319c8 0 9310 9309 Jan 26 23:44:38 kaoz [10141.343544] ffff88007fa9bcf0 0000000000000082 ffffffff8026f7ad 0000000000000000 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88006d3ee660 ffff8800708d9c70 ffff88006d3ee8c8 Jan 26 23:44:38 kaoz [10141.343544] 00000001805e19e0 ffffe200010d3fb0 ffff88006d3ee8c8 ffff88007fa9bed8 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8026f7ad>] find_get_page+0x8d/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8024940f>] kill_pid_info+0x4f/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802496c2>] sys_kill+0xa2/0x1c0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] kblankscrn.ks D ffff880001020c20 0 9313 9310 Jan 26 23:44:38 kaoz [10141.343544] ffff88004c291cf0 0000000000000082 ffffe200010d3fb0 ffffffff805e19e0 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88006d3eb8e0 ffff88007f862220 ffff88006d3ebb48 Jan 26 23:44:38 kaoz [10141.343544] 0000000180279245 000000010024fa44 ffff88006d3ebb48 ffffffff80570008 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80271b4f>] generic_file_aio_write+0x4f/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802f67e8>] ext4_file_write+0x58/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f69b>] do_sync_write+0xdb/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80227320>] do_page_fault+0x300/0x970 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80286726>] unmap_region+0x106/0x1a0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a00bb>] vfs_write+0xcb/0x170 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a0263>] sys_write+0x53/0xa0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] less D ffff880001014c20 0 9633 1 Jan 26 23:44:38 kaoz [10141.343544] ffff88003ba2bec0 0000000000000086 ffff88003ba2be68 0000000000008000 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff8800708d96c0 ffffffff80544340 ffff8800708d9928 Jan 26 23:44:38 kaoz [10141.343544] 0000000000000001 0000000100282d66 ffff8800708d9928 000000000000fe03 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a06ff>] generic_file_llseek+0x2f/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029f31e>] sys_lseek+0x7e/0x90 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029da9b>] do_sys_open+0xeb/0x110 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] pdflush D ffffffff80552468 0 9675 2 Jan 26 23:44:38 kaoz [10141.343544] ffff880041ac9e38 0000000000000046 ffffffff80270658 ffff880041ac9dc0 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8bbe90 ffff88007f863330 ffff88005b8bc0f8 Jan 26 23:44:38 kaoz [10141.343544] 000000017f85ff00 ffff880041ac9de0 ffff88005b8bc0f8 ffffffff805e5b80 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80270658>] wait_on_page_writeback_range+0x78/0x130 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80276f50>] do_writepages+0x20/0x40 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475c25>] __mutex_lock_slowpath+0x75/0xc0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80475a5a>] mutex_lock+0xa/0x10 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a15c3>] sync_filesystems+0x13/0x130 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c145e>] do_sync+0x1e/0x60 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277d1b>] pdflush+0x15b/0x240 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c1440>] do_sync+0x0/0x60 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80277bc0>] pdflush+0x0/0x240 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802519b7>] kthread+0x47/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802386c7>] schedule_tail+0x27/0x70 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7e9>] child_rip+0xa/0x11 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251970>] kthread+0x0/0x80 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020c7df>] child_rip+0x0/0x11 Jan 26 23:44:38 kaoz [10141.343544] shutdown D ffff880001020c20 0 10071 1 Jan 26 23:44:38 kaoz [10141.343544] ffff88003ac5de68 0000000000000086 ffff88003ac5de78 ffffffff80474ef7 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005b8bad80 ffff88007f862220 ffff88005b8bafe8 Jan 26 23:44:38 kaoz [10141.343544] 000000017f862220 00000001002c243e ffff88005b8bafe8 00000001002c2435 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80474ef7>] thread_return+0x3d/0x5e6 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031cebb>] start_this_handle+0x33b/0x450 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1a6>] jbd2_journal_start+0x96/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1fe>] jbd2_journal_force_commit+0xe/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a1687>] sync_filesystems+0xd7/0x130 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c1465>] do_sync+0x25/0x60 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802c14ae>] sys_sync+0xe/0x20 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10141.343544] kdm D ffff880001014c20 0 10137 6943 Jan 26 23:44:38 kaoz [10141.343544] ffff88005b953cd8 0000000000000086 ffffffff80570008 ffff880062e76a28 Jan 26 23:44:38 kaoz [10141.343544] ffffffff805de000 ffff88005bbf5b00 ffffffff80544340 ffff88005bbf5d68 Jan 26 23:44:38 kaoz [10141.343544] 0000000000000012 00000001002ce9e5 ffff88005bbf5d68 ffffffff805e19e0 Jan 26 23:44:38 kaoz [10141.343544] Call Trace: Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b2557>] __d_lookup+0xc7/0x140 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031cebb>] start_this_handle+0x33b/0x450 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80251d80>] autoremove_wake_function+0x0/0x30 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8031d1a6>] jbd2_journal_start+0x96/0xe0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff80301cf5>] ext4_create+0x85/0x120 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a904e>] vfs_create+0xae/0xf0 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802ac836>] do_filp_open+0x856/0x960 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8022e140>] update_curr+0xd0/0x140 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802a33c3>] vfs_stat_fd+0x23/0x60 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff802b5c42>] alloc_fd+0x122/0x150 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8029da30>] do_sys_open+0x80/0x110 Jan 26 23:44:38 kaoz [10141.343544] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Jan 26 23:44:38 kaoz [10155.129152] SysRq : HELP : loglevel0-8 reBoot tErm Full kIll saK aLlcpus showMem Nice powerOff showPc show-all-timers(Q) unRaw Sync showTasks Unmount shoW-blocked-tasks Jan 26 23:44:38 kaoz [10161.699932] SysRq : Show clockevent devices & pending hrtimers (no others) Jan 26 23:44:38 kaoz [10161.701338] Timer List Version: v0.4 Jan 26 23:44:38 kaoz [10161.702747] HRTIMER_MAX_CLOCK_BASES: 2 Jan 26 23:44:38 kaoz [10161.703258] now at 10161701339810 nsecs Jan 26 23:44:38 kaoz [10161.703258] Jan 26 23:44:38 kaoz [10161.703258] cpu: 0 Jan 26 23:44:38 kaoz [10161.703258] clock 0: Jan 26 23:44:38 kaoz [10161.703258] .base: ffff880001015428 Jan 26 23:44:38 kaoz [10161.703258] .index: 0 Jan 26 23:44:38 kaoz [10161.703258] .resolution: 1 nsecs Jan 26 23:44:38 kaoz [10161.703258] .get_time: ktime_get_real Jan 26 23:44:38 kaoz [10161.703258] .offset: 1233010481853079954 nsecs Jan 26 23:44:38 kaoz [10161.703258] active timers: Jan 26 23:44:38 kaoz [10161.703258] clock 1: Jan 26 23:44:38 kaoz [10161.703258] .base: ffff880001015468 Jan 26 23:44:38 kaoz [10161.703258] .index: 1 Jan 26 23:44:38 kaoz [10161.703258] .resolution: 1 nsecs Jan 26 23:44:38 kaoz [10161.703258] .get_time: ktime_get Jan 26 23:44:38 kaoz [10161.703258] .offset: 0 nsecs Jan 26 23:44:38 kaoz [10161.703258] active timers: Jan 26 23:44:38 kaoz [10161.703258] #0: <ffff880001015500>, tick_sched_timer, S:01 Jan 26 23:44:38 kaoz [10161.703258] # expires at 10162345650432-10162345650432 nsecs [in 644310622 to 644310622 nsecs] Jan 26 23:44:38 kaoz [10161.703258] #1: <ffff88005bb220e8>, it_real_fn, S:01 Jan 26 23:44:38 kaoz [10161.703258] # expires at 32702039507086-32702039507086 nsecs [in 22540338167276 to 22540338167276 nsecs] Jan 26 23:44:38 kaoz [10161.703258] .expires_next : 10162345650432 nsecs Jan 26 23:44:38 kaoz [10161.703258] .hres_active : 1 Jan 26 23:44:38 kaoz [10161.703258] .nr_events : 1715455 Jan 26 23:44:38 kaoz [10161.703258] .nohz_mode : 2 Jan 26 23:44:38 kaoz [10161.703258] .idle_tick : 10161348983865 nsecs Jan 26 23:44:38 kaoz [10161.703258] .tick_stopped : 1 Jan 26 23:44:38 kaoz [10161.703258] .idle_jiffies : 4297925700 Jan 26 23:44:38 kaoz [10161.703258] .idle_calls : 2284355 Jan 26 23:44:38 kaoz [10161.703258] .idle_sleeps : 164609 Jan 26 23:44:38 kaoz [10161.703258] .idle_entrytime : 10161737675287 nsecs Jan 26 23:44:38 kaoz [10161.703258] .idle_waketime : 10161737671047 nsecs Jan 26 23:44:38 kaoz [10161.703258] .idle_exittime : 10161345670207 nsecs Jan 26 23:44:38 kaoz [10161.703258] .idle_sleeptime : 8728154262894 nsecs Jan 26 23:44:38 kaoz [10161.703258] .last_jiffies : 4297925817 Jan 26 23:44:38 kaoz [10161.703258] .next_jiffies : 4297926000 Jan 26 23:44:38 kaoz [10161.703258] .idle_expires : 10162345650432 nsecs Jan 26 23:44:38 kaoz [10161.703258] jiffies: 4297925817 Jan 26 23:44:38 kaoz [10161.703258] Jan 26 23:44:38 kaoz [10161.703258] cpu: 1 Jan 26 23:44:38 kaoz [10161.703258] clock 0: Jan 26 23:44:38 kaoz [10161.703258] .base: ffff880001021428 Jan 26 23:44:38 kaoz [10161.703258] .index: 0 Jan 26 23:44:38 kaoz [10161.703258] .resolution: 1 nsecs Jan 26 23:44:38 kaoz [10161.703258] .get_time: ktime_get_real Jan 26 23:44:38 kaoz [10161.703258] .offset: 1233010481853079954 nsecs Jan 26 23:44:38 kaoz [10161.703258] active timers: Jan 26 23:44:38 kaoz [10161.703258] clock 1: Jan 26 23:44:38 kaoz [10161.703258] .base: ffff880001021468 Jan 26 23:44:38 kaoz [10161.703258] .index: 1 Jan 26 23:44:38 kaoz [10161.703258] .resolution: 1 nsecs Jan 26 23:44:38 kaoz [10161.703258] .get_time: ktime_get Jan 26 23:44:38 kaoz [10161.703258] .offset: 0 nsecs Jan 26 23:44:38 kaoz [10161.703258] active timers: Jan 26 23:44:38 kaoz [10161.703258] #0: <ffff88007f85d9a8>, hrtimer_wakeup, S:01 Jan 26 23:44:38 kaoz [10161.703258] # expires at 10161860916333-10161865916331 nsecs [in 159576523 to 164576521 nsecs] Jan 26 23:44:38 kaoz [10161.703258] #1: <ffff880001021500>, tick_sched_timer, S:01 Jan 26 23:44:38 kaoz [10161.703258] # expires at 10161878983812-10161878983812 nsecs [in 177644002 to 177644002 nsecs] Jan 26 23:44:38 kaoz [10161.703258] .expires_next : 10161865916331 nsecs Jan 26 23:44:38 kaoz [10161.703258] .hres_active : 1 Jan 26 23:44:38 kaoz [10161.703258] .nr_events : 2322638 Jan 26 23:44:38 kaoz [10161.703258] .nohz_mode : 2 Jan 26 23:44:38 kaoz [10161.703258] .idle_tick : 10161683150498 nsecs Jan 26 23:44:38 kaoz [10161.703258] .tick_stopped : 1 Jan 26 23:44:38 kaoz [10161.703258] .idle_jiffies : 4297925800 Jan 26 23:44:38 kaoz [10161.703258] .idle_calls : 13381452 Jan 26 23:44:38 kaoz [10161.703258] .idle_sleeps : 5700675 Jan 26 23:44:38 kaoz [10161.703258] .idle_entrytime : 10161699594129 nsecs Jan 26 23:44:38 kaoz [10161.703258] .idle_waketime : 10161699926354 nsecs Jan 26 23:44:38 kaoz [10161.703258] .idle_exittime : 10161676129546 nsecs Jan 26 23:44:38 kaoz [10161.703258] .idle_sleeptime : 7638933463214 nsecs Jan 26 23:44:38 kaoz [10161.703258] .last_jiffies : 4297925806 Jan 26 23:44:38 kaoz [10161.703258] .next_jiffies : 4297925860 Jan 26 23:44:38 kaoz [10161.703258] .idle_expires : 10161878983812 nsecs Jan 26 23:44:38 kaoz [10161.703258] jiffies: 4297925826 Jan 26 23:44:38 kaoz [10161.703258] Jan 26 23:44:38 kaoz [10161.703258] Jan 26 23:44:38 kaoz [10161.703258] Tick Device: mode: 1 Jan 26 23:44:38 kaoz [10161.703258] Broadcast device Jan 26 23:44:38 kaoz [10161.703258] Clock Event Device: hpet Jan 26 23:44:38 kaoz [10161.703258] max_delta_ns: 85899346200 Jan 26 23:44:38 kaoz [10161.703258] min_delta_ns: 5000 Jan 26 23:44:38 kaoz [10161.703258] mult: 107374182 Jan 26 23:44:38 kaoz [10161.703258] shift: 32 Jan 26 23:44:38 kaoz [10161.703258] mode: 3 Jan 26 23:44:38 kaoz [10161.703258] next_event: 9223372036854775807 nsecs Jan 26 23:44:38 kaoz [10161.703258] set_next_event: hpet_legacy_next_event Jan 26 23:44:38 kaoz [10161.703258] set_mode: hpet_legacy_set_mode Jan 26 23:44:38 kaoz [10161.703258] event_handler: tick_handle_oneshot_broadcast Jan 26 23:44:38 kaoz [10161.703258] tick_broadcast_mask: 00000000 Jan 26 23:44:38 kaoz [10161.703258] tick_broadcast_oneshot_mask: 00000000 Jan 26 23:44:38 kaoz [10161.703258] Jan 26 23:44:38 kaoz [10161.703258] Jan 26 23:44:38 kaoz [10161.703258] Tick Device: mode: 1 Jan 26 23:44:38 kaoz [10161.703258] Per CPU device: 0 Jan 26 23:44:38 kaoz [10161.703258] Clock Event Device: lapic Jan 26 23:44:38 kaoz [10161.703258] max_delta_ns: 671049102 Jan 26 23:44:38 kaoz [10161.703258] min_delta_ns: 1199 Jan 26 23:44:38 kaoz [10161.703258] mult: 53690248 Jan 26 23:44:38 kaoz [10161.703258] shift: 32 Jan 26 23:44:38 kaoz [10161.703258] mode: 3 Jan 26 23:44:38 kaoz [10161.703258] next_event: 10162345650432 nsecs Jan 26 23:44:38 kaoz [10161.703258] set_next_event: lapic_next_event Jan 26 23:44:38 kaoz [10161.703258] set_mode: lapic_timer_setup Jan 26 23:44:38 kaoz [10161.703258] event_handler: hrtimer_interrupt Jan 26 23:44:38 kaoz [10161.703258] Jan 26 23:44:38 kaoz [10161.703258] Tick Device: mode: 1 Jan 26 23:44:38 kaoz [10161.703258] Per CPU device: 1 Jan 26 23:44:38 kaoz [10161.703258] Clock Event Device: lapic Jan 26 23:44:38 kaoz [10161.703258] max_delta_ns: 671049102 Jan 26 23:44:38 kaoz [10161.703258] min_delta_ns: 1199 Jan 26 23:44:38 kaoz [10161.703258] mult: 53690248 Jan 26 23:44:38 kaoz [10161.703258] shift: 32 Jan 26 23:44:38 kaoz [10161.703258] mode: 3 Jan 26 23:44:38 kaoz [10161.703258] next_event: 10161865916331 nsecs Jan 26 23:44:38 kaoz [10161.703258] set_next_event: lapic_next_event Jan 26 23:44:38 kaoz [10161.703258] set_mode: lapic_timer_setup Jan 26 23:44:38 kaoz [10161.703258] event_handler: hrtimer_interrupt Jan 26 23:44:38 kaoz [10161.703258] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-03 19:51 ` Federico Cuello @ 2009-02-04 6:17 ` Nick Piggin 2009-02-04 17:31 ` Federico Cuello 0 siblings, 1 reply; 14+ messages in thread From: Nick Piggin @ 2009-02-04 6:17 UTC (permalink / raw) To: Federico Cuello, Ralf Hildebrandt, Artem Bityutskiy; +Cc: linux-kernel [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset="utf-8", Size: 4237 bytes --] On Wednesday 04 February 2009 06:51:16 Federico Cuello wrote:> Nick Piggin wrote:> > On Wednesday 28 January 2009 08:09:10 Federico Cuello wrote:> >> Ralf Hildebrandt escribió:> >>> I recently installed 2.6.28.2 on our postfix/dovecot-based> >>> mailboxserver. Previously, 2.6.28 and 2.6.28.1 have been running there> >>> without a hitch.> >>>> >>> Now with 2.6.28.2S I had two major lockups: All writes to the users'> >>> Maildirs (on ext4) would stall, the load would rise, "sync" would never> >>> return.> >>>> >>> I had to "reboot -f -n" to get the machine back. All hanging processes> >>> were unkillable, even with kill -9.> >>> [...]> >>> >> The same is happening to me, but I have some logs taken with sysrq.[...]> >> 0 2 99028 46536 26368 1569400 0 0 4 0 1407 387 0> >> 0 0 100> >>> >> Notice the 100% iowait.> >>> >> I also managed to reproduce it doing a rsync from one partition to a USB> >> drive. After the lockup I can't read any file from the source partition,> >> but the other partitions can be accessed normally.> >> > Hm, thanks for reporting, can you guys get a sysrq+W trace when the> > system reaches this state?>> Here it is, if you need something else please ask: Thanks, could you reply-to-all when replying to retain ccs please? Common theme is ext4, which uses no_nrwrite_index_update, and I introduceda bug in there which could possibly cause ext4 to go into a loop... Would it be possible if you can test the following patch? Thanks,Nick --- Commit 05fe478dd04e02fa230c305ab9b5616669821dd3 introduced some@wbc->nr_to_write breakage. Here is the change from the commit: >--- a/mm/page-writeback.c>+++ b/mm/page-writeback.c>@@ -963,8 +963,10 @@ retry:> }> }>> if (--nr_to_write <= 0)> done = 1;> if (wbc->sync_mode == WB_SYNC_NONE) {> if (--wbc->nr_to_write <= 0)> done = 1;> }> if (wbc->nonblocking && bdi_write_congested(bdi)) {> wbc->encountered_congestion = 1;> done = 1> } It makes the following changes:1. Decrement wbc->nr_to_write instead of nr_to_write2. Decrement wbc->nr_to_write _only_ if wbc->sync_mode == WB_SYNC_NONE3. If synced nr_to_write pages, stop only if if wbc->sync_mode ==WB_SYNC_NONE, otherwise keep going. However, according to the commit message, the intention was toonly make change 3. Change 1 is a bug. Change 2 does not seem to benecessary, and it breaks UBIFS expectations, so if needed, itshould be done separately later. And change 2 does not seem tobe documented in the commit message. This patch does the following:1. Undo changes 1 and 22. Add a comment explaining change 3 (it very useful to have comments in_code_, not only in the commit). Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>Acked-by: Nick Piggin <npiggin@suse.de>Cc: Andrew Morton <akpm@linux-foundation.org>--- mm/page-writeback.c | 21 +++++++++++++++------ 1 files changed, 15 insertions(+), 6 deletions(-) diff --git a/mm/page-writeback.c b/mm/page-writeback.cindex b493db7..dc32dae 100644--- a/mm/page-writeback.c+++ b/mm/page-writeback.c@@ -1051,13 +1051,22 @@ continue_unlock: } } - if (wbc->sync_mode == WB_SYNC_NONE) {- wbc->nr_to_write--;- if (wbc->nr_to_write <= 0) {- done = 1;- break;- }+ if (nr_to_write > 0)+ nr_to_write--;+ else if (wbc->sync_mode == WB_SYNC_NONE) {+ /*+ * We stop writing back only if we are not+ * doing integrity sync. In case of integrity+ * sync we have to keep going because someone+ * may be concurrently dirtying pages, and we+ * might have synced a lot of newly appeared+ * dirty pages, but have not synced all of the+ * old dirty pages.+ */+ done = 1;+ break; }+ if (wbc->nonblocking && bdi_write_congested(bdi)) { wbc->encountered_congestion = 1; done = 1;\0ÿôèº{.nÇ+‰·Ÿ®‰†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a¢f£¢·hšïêÿ‘êçz_è®\x03(階ŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨èÚ&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥ ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-04 6:17 ` Nick Piggin @ 2009-02-04 17:31 ` Federico Cuello 2009-02-05 3:25 ` Nick Piggin 2009-02-05 10:19 ` Ralf Hildebrandt 0 siblings, 2 replies; 14+ messages in thread From: Federico Cuello @ 2009-02-04 17:31 UTC (permalink / raw) To: Nick Piggin; +Cc: Ralf Hildebrandt, Artem Bityutskiy, linux-kernel Nick Piggin wrote: > [...] > Thanks, could you reply-to-all when replying to retain ccs please? > > Common theme is ext4, which uses no_nrwrite_index_update, and I introduced > a bug in there which could possibly cause ext4 to go into a loop... > > Would it be possible if you can test the following patch? > I'll test it as soon as I get home. Meanwhile, I think the new patch may be slightly wrong. If I understand correctly PageWriteback(page) is called before nr_to_write is tested for being > 0 and then decremented if true, but "done" is not set to 1 until the next iteration. So another call to PageWriteback(page) while take place and then "done" will be set to true (if wbc->sync_mode == WB_SYNC_NONE). If nr_to_write == 1 at the beginning of the loop then two pages will be written. I think the test condition should something like: if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) { done = 1; break; } . . > Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com> > Acked-by: Nick Piggin <npiggin@suse.de> > Cc: Andrew Morton <akpm@linux-foundation.org> > --- > mm/page-writeback.c | 21 +++++++++++++++------ > 1 files changed, 15 insertions(+), 6 deletions(-) > > diff --git a/mm/page-writeback.c b/mm/page-writeback.c > index b493db7..dc32dae 100644 > --- a/mm/page-writeback.c > +++ b/mm/page-writeback.c > @@ -1051,13 +1051,22 @@ continue_unlock: > } > } > > - if (wbc->sync_mode == WB_SYNC_NONE) { > - wbc->nr_to_write--; > - if (wbc->nr_to_write <= 0) { > - done = 1; > - break; > - } > + if (nr_to_write > 0) > + nr_to_write--; > + else if (wbc->sync_mode == WB_SYNC_NONE) { > + /* > + * We stop writing back only if we are not > + * doing integrity sync. In case of integrity > + * sync we have to keep going because someone > + * may be concurrently dirtying pages, and we > + * might have synced a lot of newly appeared > + * dirty pages, but have not synced all of the > + * old dirty pages. > + */ > + done = 1; > + break; > } > + > if (wbc->nonblocking && bdi_write_congested(bdi)) { > wbc->encountered_congestion = 1; > done = 1; > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-04 17:31 ` Federico Cuello @ 2009-02-05 3:25 ` Nick Piggin 2009-02-05 11:54 ` Federico Cuello 2009-02-05 10:19 ` Ralf Hildebrandt 1 sibling, 1 reply; 14+ messages in thread From: Nick Piggin @ 2009-02-05 3:25 UTC (permalink / raw) To: Federico Cuello; +Cc: Ralf Hildebrandt, Artem Bityutskiy, linux-kernel On Thursday 05 February 2009 04:31:00 Federico Cuello wrote: > Nick Piggin wrote: > > [...] > > Thanks, could you reply-to-all when replying to retain ccs please? > > > > Common theme is ext4, which uses no_nrwrite_index_update, and I > > introduced a bug in there which could possibly cause ext4 to go into a > > loop... > > > > Would it be possible if you can test the following patch? > > I'll test it as soon as I get home. Thanks. > Meanwhile, I think the new patch may be slightly wrong. If I understand > correctly PageWriteback(page) is called before nr_to_write is tested for > being > 0 and then decremented if true, but "done" is not set to 1 > until the next iteration. So another call to PageWriteback(page) while > take place and then "done" will be set to true (if wbc->sync_mode == > WB_SYNC_NONE). > > If nr_to_write == 1 at the beginning of the loop then two pages will be > written. > > I think the test condition should something like: > > if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) { > done = 1; > break; > } I think you're quite right. Good catch. We probably want to prevent nr_to_write from going -ve, though. I think something like this if (nr_to_write > 0) nr_to_write--; if (!nr_to_write && wbc->sync_mode == WB_SYNC_NONE) { ... Would you care to send a patch? Thanks, Nick ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-05 3:25 ` Nick Piggin @ 2009-02-05 11:54 ` Federico Cuello 2009-02-09 13:45 ` Nick Piggin 0 siblings, 1 reply; 14+ messages in thread From: Federico Cuello @ 2009-02-05 11:54 UTC (permalink / raw) To: Nick Piggin; +Cc: Ralf Hildebrandt, Artem Bityutskiy, linux-kernel Nick Piggin wrote: > On Thursday 05 February 2009 04:31:00 Federico Cuello wrote: > >> Nick Piggin wrote: >> >>> [...] >>> Thanks, could you reply-to-all when replying to retain ccs please? >>> >>> Common theme is ext4, which uses no_nrwrite_index_update, and I >>> introduced a bug in there which could possibly cause ext4 to go into a >>> loop... >>> >>> Would it be possible if you can test the following patch? >>> >> I'll test it as soon as I get home. >> > > Thanks. > > > >> Meanwhile, I think the new patch may be slightly wrong. If I understand >> correctly PageWriteback(page) is called before nr_to_write is tested for >> being > 0 and then decremented if true, but "done" is not set to 1 >> until the next iteration. So another call to PageWriteback(page) while >> take place and then "done" will be set to true (if wbc->sync_mode == >> WB_SYNC_NONE). >> >> If nr_to_write == 1 at the beginning of the loop then two pages will be >> written. >> >> I think the test condition should something like: >> >> if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) { >> done = 1; >> break; >> } >> > > I think you're quite right. Good catch. We probably want to prevent > nr_to_write from going -ve, though. > > I think something like this > > if (nr_to_write > 0) > nr_to_write--; > if (!nr_to_write && wbc->sync_mode == WB_SYNC_NONE) { > ... > > Would you care to send a patch? > Ok, after extensive testing I haven't been able to solve the problem. I'm posting below the patch I used. I tried 3 different patches with one successful test run with the one you sent me. I don't know if it was just a coincidence as I had no time to test it again. Now, with the patch below, it stalls with 50% IO-wait (dual core, one core at 100%). Perhaps the patch is part of the solution, I don't know. I also have the sysrq-W logs and I'm also posting them below. Thanks, diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 08d2b96..9e2ae50 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -981,13 +981,23 @@ continue_unlock: } } - if (wbc->sync_mode == WB_SYNC_NONE) { - wbc->nr_to_write--; - if (wbc->nr_to_write <= 0) { + if (nr_to_write > 0) { + nr_to_write--; + if (nr_to_write == 0 && wbc->sync_mode == WB_SYNC_NONE) { + /* + * We stop writing back only if we are not + * doing integrity sync. In case of integrity + * sync we have to keep going because someone + * may be concurrently dirtying pages, and we + * might have synced a lot of newly appeared + * dirty pages, but have not synced all of the + * old dirty pages. + */ done = 1; break; } } + if (wbc->nonblocking && bdi_write_congested(bdi)) { wbc->encountered_congestion = 1; done = 1; Feb 5 03:15:19 kaoz [ 1953.633858] SysRq : Show Blocked State Feb 5 03:15:19 kaoz [ 1953.634882] task PC stack pid father Feb 5 03:15:19 kaoz [ 1953.634896] kjournald2 D ffff880001020c20 0 4713 2 Feb 5 03:15:19 kaoz [ 1953.634904] ffff88007e4edd50 0000000000000046 ffff8800582e2900 ffff88007e64a840 Feb 5 03:15:19 kaoz [ 1953.634912] ffffffff805de000 ffff88007eb4ec10 ffff88007f862220 ffff88007eb4ee78 Feb 5 03:15:19 kaoz [ 1953.634919] 0000000100000000 0000000100057f3b ffff88007eb4ee78 ffff88007edc0090 Feb 5 03:15:19 kaoz [ 1953.634926] Call Trace: Feb 5 03:15:19 kaoz [ 1953.634938] [<ffffffff8022e110>] update_curr+0xd0/0x140 Feb 5 03:15:19 kaoz [ 1953.634949] [<ffffffff8031d5e1>] jbd2_journal_commit_transaction+0x171/0x1240 Feb 5 03:15:19 kaoz [ 1953.634957] [<ffffffff8023048f>] dequeue_task_fair+0x3f/0xc0 Feb 5 03:15:19 kaoz [ 1953.634963] [<ffffffff8022e677>] dequeue_task+0x57/0x70 Feb 5 03:15:19 kaoz [ 1953.634972] [<ffffffff80474d27>] thread_return+0x3d/0x5e6 Feb 5 03:15:19 kaoz [ 1953.634979] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.634986] [<ffffffff802461a4>] lock_timer_base+0x34/0x70 Feb 5 03:15:19 kaoz [ 1953.634991] [<ffffffff80246234>] try_to_del_timer_sync+0x54/0x60 Feb 5 03:15:19 kaoz [ 1953.634998] [<ffffffff80322598>] kjournald2+0xd8/0x230 Feb 5 03:15:19 kaoz [ 1953.635004] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.635009] [<ffffffff803224c0>] kjournald2+0x0/0x230 Feb 5 03:15:19 kaoz [ 1953.635014] [<ffffffff80251957>] kthread+0x47/0x80 Feb 5 03:15:19 kaoz [ 1953.635020] [<ffffffff80238697>] schedule_tail+0x27/0x70 Feb 5 03:15:19 kaoz [ 1953.635026] [<ffffffff8020c7e9>] child_rip+0xa/0x11 Feb 5 03:15:19 kaoz [ 1953.635031] [<ffffffff80251910>] kthread+0x0/0x80 Feb 5 03:15:19 kaoz [ 1953.635036] [<ffffffff8020c7df>] child_rip+0x0/0x11 Feb 5 03:15:19 kaoz [ 1953.635068] kdeinit4 D ffff88007c760870 0 6084 1 Feb 5 03:15:19 kaoz [ 1953.635075] ffff88006e83bcf0 0000000000000086 ffffffff8026f77d 0000000000000000 Feb 5 03:15:19 kaoz [ 1953.635082] ffffffff805de000 ffff880070ae4440 ffff880070ae3330 ffff880070ae46a8 Feb 5 03:15:19 kaoz [ 1953.635089] 00000001805e19e0 ffffe20001a79688 ffff880070ae46a8 0000000000000000 Feb 5 03:15:19 kaoz [ 1953.635095] Call Trace: Feb 5 03:15:19 kaoz [ 1953.635103] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0 Feb 5 03:15:19 kaoz [ 1953.635109] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0 Feb 5 03:15:19 kaoz [ 1953.635114] [<ffffffff8047588a>] mutex_lock+0xa/0x10 Feb 5 03:15:19 kaoz [ 1953.635119] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0 Feb 5 03:15:19 kaoz [ 1953.635126] [<ffffffff802f6848>] ext4_file_write+0x58/0x170 Feb 5 03:15:19 kaoz [ 1953.635134] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120 Feb 5 03:15:19 kaoz [ 1953.635142] [<ffffffff80227210>] do_page_fault+0x300/0x970 Feb 5 03:15:19 kaoz [ 1953.635148] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.635155] [<ffffffff802a008b>] vfs_write+0xcb/0x170 Feb 5 03:15:19 kaoz [ 1953.635161] [<ffffffff802a0233>] sys_write+0x53/0xa0 Feb 5 03:15:19 kaoz [ 1953.635166] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Feb 5 03:15:19 kaoz [ 1953.635173] quasselcore D ffff880001006b08 0 6204 1 Feb 5 03:15:19 kaoz [ 1953.635179] ffff88006ae09bb8 0000000000000086 ffff88007e91d800 ffffffff80496fd0 Feb 5 03:15:19 kaoz [ 1953.635186] ffffffff805de000 ffff88006ad685b0 ffff880055a605b0 ffff88006ad68818 Feb 5 03:15:19 kaoz [ 1953.635192] 0000000100000004 0000000000000000 ffff88006ad68818 ffffffff802fa711 Feb 5 03:15:19 kaoz [ 1953.635199] Call Trace: Feb 5 03:15:19 kaoz [ 1953.635205] [<ffffffff802fa711>] ext4_da_write_end+0x141/0x280 Feb 5 03:15:19 kaoz [ 1953.635259] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod] Feb 5 03:15:19 kaoz [ 1953.635266] [<ffffffff8026f910>] sync_page+0x0/0x70 Feb 5 03:15:19 kaoz [ 1953.635273] [<ffffffff80475307>] io_schedule+0x37/0x50 Feb 5 03:15:19 kaoz [ 1953.635279] [<ffffffff8026f945>] sync_page+0x35/0x70 Feb 5 03:15:19 kaoz [ 1953.635283] [<ffffffff8047568a>] __wait_on_bit_lock+0x4a/0x80 Feb 5 03:15:19 kaoz [ 1953.635289] [<ffffffff8026f8ee>] __lock_page+0x5e/0x70 Feb 5 03:15:19 kaoz [ 1953.635295] [<ffffffff80251d50>] wake_bit_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.635302] [<ffffffff80276cdd>] write_cache_pages+0x1ed/0x3d0 Feb 5 03:15:19 kaoz [ 1953.635308] [<ffffffff802fbf10>] __mpage_da_writepage+0x0/0x170 Feb 5 03:15:19 kaoz [ 1953.635315] [<ffffffff802fbcc7>] ext4_da_writepages+0x237/0x3b0 Feb 5 03:15:19 kaoz [ 1953.635321] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.635328] [<ffffffff802fd240>] ext4_da_get_block_write+0x0/0x160 Feb 5 03:15:19 kaoz [ 1953.635333] [<ffffffff80276f10>] do_writepages+0x20/0x40 Feb 5 03:15:19 kaoz [ 1953.635338] [<ffffffff8027074d>] __filemap_fdatawrite_range+0x4d/0x60 Feb 5 03:15:19 kaoz [ 1953.635345] [<ffffffff802c12b5>] do_fsync+0x45/0xe0 Feb 5 03:15:19 kaoz [ 1953.635350] [<ffffffff802c137e>] __do_fsync+0x2e/0x50 Feb 5 03:15:19 kaoz [ 1953.635355] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Feb 5 03:15:19 kaoz [ 1953.635361] kwin D ffff88007c760870 0 6307 6283 Feb 5 03:15:19 kaoz [ 1953.635366] ffff88006af0fcf0 0000000000000082 ffffffff8026f77d 0000000000000000 Feb 5 03:15:19 kaoz [ 1953.635373] ffffffff805de000 ffff88007e7ea7d0 ffff8800709d96c0 ffff88007e7eaa38 Feb 5 03:15:19 kaoz [ 1953.635380] 00000000805e19e0 ffffe20000664a20 ffff88007e7eaa38 ffff88006af0fcc8 Feb 5 03:15:19 kaoz [ 1953.635386] Call Trace: Feb 5 03:15:19 kaoz [ 1953.635392] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0 Feb 5 03:15:19 kaoz [ 1953.635398] [<ffffffff8022f753>] __wake_up+0x43/0x70 Feb 5 03:15:19 kaoz [ 1953.635403] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0 Feb 5 03:15:19 kaoz [ 1953.635408] [<ffffffff8047588a>] mutex_lock+0xa/0x10 Feb 5 03:15:19 kaoz [ 1953.635413] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0 Feb 5 03:15:19 kaoz [ 1953.635419] [<ffffffff802f6848>] ext4_file_write+0x58/0x170 Feb 5 03:15:19 kaoz [ 1953.635425] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120 Feb 5 03:15:19 kaoz [ 1953.635431] [<ffffffff80227210>] do_page_fault+0x300/0x970 Feb 5 03:15:19 kaoz [ 1953.635437] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.635443] [<ffffffff803fc4d6>] sock_ioctl+0x56/0x260 Feb 5 03:15:19 kaoz [ 1953.635450] [<ffffffff802ad76f>] vfs_ioctl+0x2f/0xa0 Feb 5 03:15:19 kaoz [ 1953.635456] [<ffffffff802a008b>] vfs_write+0xcb/0x170 Feb 5 03:15:19 kaoz [ 1953.635462] [<ffffffff802a0233>] sys_write+0x53/0xa0 Feb 5 03:15:19 kaoz [ 1953.635467] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Feb 5 03:15:19 kaoz [ 1953.635479] mysqld D 0000000000000002 0 6688 6674 Feb 5 03:15:19 kaoz [ 1953.635485] ffff88005c58db58 0000000000000086 ffff88005c58dad8 ffffffff80230285 Feb 5 03:15:19 kaoz [ 1953.635491] ffffffff805de000 ffff88007f95ec10 ffff8800636ca7d0 ffff88007f95ee78 Feb 5 03:15:19 kaoz [ 1953.635498] 0000000100000001 ffff880070bdd840 ffff88007f95ee78 ffffffff8023384b Feb 5 03:15:19 kaoz [ 1953.635504] Call Trace: Feb 5 03:15:19 kaoz [ 1953.635510] [<ffffffff80230285>] enqueue_entity+0xf5/0x160 Feb 5 03:15:19 kaoz [ 1953.635517] [<ffffffff8023384b>] check_preempt_wakeup+0x15b/0x180 Feb 5 03:15:19 kaoz [ 1953.635522] [<ffffffff802356ae>] try_to_wake_up+0xee/0x1a0 Feb 5 03:15:19 kaoz [ 1953.635528] [<ffffffff8031cf3b>] start_this_handle+0x33b/0x450 Feb 5 03:15:19 kaoz [ 1953.635534] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.635541] [<ffffffff8031d226>] jbd2_journal_start+0x96/0xe0 Feb 5 03:15:19 kaoz [ 1953.635547] [<ffffffff802f9c36>] ext4_dirty_inode+0x36/0xb0 Feb 5 03:15:19 kaoz [ 1953.635552] [<ffffffff802be284>] __mark_inode_dirty+0x34/0x1d0 Feb 5 03:15:19 kaoz [ 1953.635558] [<ffffffff802b3380>] file_update_time+0x110/0x130 Feb 5 03:15:19 kaoz [ 1953.635564] [<ffffffff80270fa6>] __generic_file_aio_write_nolock+0x246/0x470 Feb 5 03:15:19 kaoz [ 1953.635569] [<ffffffff80271b34>] generic_file_aio_write+0x64/0xe0 Feb 5 03:15:19 kaoz [ 1953.635575] [<ffffffff802f6848>] ext4_file_write+0x58/0x170 Feb 5 03:15:19 kaoz [ 1953.635582] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120 Feb 5 03:15:19 kaoz [ 1953.635588] [<ffffffff80227210>] do_page_fault+0x300/0x970 Feb 5 03:15:19 kaoz [ 1953.635594] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.635600] [<ffffffff802a008b>] vfs_write+0xcb/0x170 Feb 5 03:15:19 kaoz [ 1953.635606] [<ffffffff802a0233>] sys_write+0x53/0xa0 Feb 5 03:15:19 kaoz [ 1953.635611] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Feb 5 03:15:19 kaoz [ 1953.635616] konsole D ffff88007c760870 0 6722 1 Feb 5 03:15:19 kaoz [ 1953.635622] ffff88006ae49cf0 0000000000000082 ffffffff8026f77d 0000000000000000 Feb 5 03:15:19 kaoz [ 1953.635628] ffffffff805de000 ffff88005c594fa0 ffff88007c0b2220 ffff88005c595208 Feb 5 03:15:19 kaoz [ 1953.635635] 00000001805e19e0 ffffe20001a79688 ffff88005c595208 0000000000000000 Feb 5 03:15:19 kaoz [ 1953.635641] Call Trace: Feb 5 03:15:19 kaoz [ 1953.635647] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0 Feb 5 03:15:19 kaoz [ 1953.635652] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0 Feb 5 03:15:19 kaoz [ 1953.635657] [<ffffffff8047588a>] mutex_lock+0xa/0x10 Feb 5 03:15:19 kaoz [ 1953.635662] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0 Feb 5 03:15:19 kaoz [ 1953.635668] [<ffffffff802f6848>] ext4_file_write+0x58/0x170 Feb 5 03:15:19 kaoz [ 1953.635674] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120 Feb 5 03:15:19 kaoz [ 1953.635680] [<ffffffff80227210>] do_page_fault+0x300/0x970 Feb 5 03:15:19 kaoz [ 1953.635686] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.635692] [<ffffffff802ad76f>] vfs_ioctl+0x2f/0xa0 Feb 5 03:15:19 kaoz [ 1953.635698] [<ffffffff802a008b>] vfs_write+0xcb/0x170 Feb 5 03:15:19 kaoz [ 1953.635704] [<ffffffff802a0233>] sys_write+0x53/0xa0 Feb 5 03:15:19 kaoz [ 1953.635709] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Feb 5 03:15:19 kaoz [ 1953.635715] pdflush D ffff880001003178 0 7292 2 Feb 5 03:15:19 kaoz [ 1953.635720] ffff88007c09ba20 0000000000000046 ffff880024e2b870 ffff88007ebd32d0 Feb 5 03:15:19 kaoz [ 1953.635727] ffffffff805de000 ffff880055a638e0 ffff88007ea7c9f0 ffff880055a63b48 Feb 5 03:15:19 kaoz [ 1953.635734] 000000017c09b9c0 ffff880024e2b870 ffff880055a63b48 ffff88007edd1d80 Feb 5 03:15:19 kaoz [ 1953.635740] Call Trace: Feb 5 03:15:19 kaoz [ 1953.635746] [<ffffffff8032f437>] generic_make_request+0x207/0x340 Feb 5 03:15:19 kaoz [ 1953.635767] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod] Feb 5 03:15:19 kaoz [ 1953.635775] [<ffffffff80341b27>] radix_tree_gang_lookup_slot+0x77/0xc0 Feb 5 03:15:19 kaoz [ 1953.635781] [<ffffffff8026f910>] sync_page+0x0/0x70 Feb 5 03:15:19 kaoz [ 1953.635788] [<ffffffff80475307>] io_schedule+0x37/0x50 Feb 5 03:15:19 kaoz [ 1953.635794] [<ffffffff8026f945>] sync_page+0x35/0x70 Feb 5 03:15:19 kaoz [ 1953.635798] [<ffffffff8047568a>] __wait_on_bit_lock+0x4a/0x80 Feb 5 03:15:19 kaoz [ 1953.635805] [<ffffffff8026f8ee>] __lock_page+0x5e/0x70 Feb 5 03:15:19 kaoz [ 1953.635810] [<ffffffff80251d50>] wake_bit_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.635816] [<ffffffff8027867a>] pagevec_lookup_tag+0x1a/0x30 Feb 5 03:15:19 kaoz [ 1953.635822] [<ffffffff80276cdd>] write_cache_pages+0x1ed/0x3d0 Feb 5 03:15:19 kaoz [ 1953.635828] [<ffffffff802fbf10>] __mpage_da_writepage+0x0/0x170 Feb 5 03:15:19 kaoz [ 1953.635835] [<ffffffff8031bbdf>] jbd2_journal_stop+0x17f/0x200 Feb 5 03:15:19 kaoz [ 1953.635842] [<ffffffff802fbcc7>] ext4_da_writepages+0x237/0x3b0 Feb 5 03:15:19 kaoz [ 1953.635848] [<ffffffff80476f6c>] _spin_unlock_irq+0xc/0x30 Feb 5 03:15:19 kaoz [ 1953.635853] [<ffffffff80236c8b>] finish_task_switch+0x2b/0xe0 Feb 5 03:15:19 kaoz [ 1953.635860] [<ffffffff802fd240>] ext4_da_get_block_write+0x0/0x160 Feb 5 03:15:19 kaoz [ 1953.635865] [<ffffffff80276f10>] do_writepages+0x20/0x40 Feb 5 03:15:19 kaoz [ 1953.635871] [<ffffffff802bd516>] __writeback_single_inode+0xa6/0x470 Feb 5 03:15:19 kaoz [ 1953.635876] [<ffffffff802bdea0>] generic_sync_sb_inodes+0x370/0x4d0 Feb 5 03:15:19 kaoz [ 1953.635882] [<ffffffff802be1bd>] writeback_inodes+0x4d/0xe0 Feb 5 03:15:19 kaoz [ 1953.635887] [<ffffffff8027759d>] background_writeout+0xad/0xe0 Feb 5 03:15:19 kaoz [ 1953.635893] [<ffffffff80277ccb>] pdflush+0x15b/0x240 Feb 5 03:15:19 kaoz [ 1953.635898] [<ffffffff802774f0>] background_writeout+0x0/0xe0 Feb 5 03:15:19 kaoz [ 1953.635904] [<ffffffff80277b70>] pdflush+0x0/0x240 Feb 5 03:15:19 kaoz [ 1953.635909] [<ffffffff80251957>] kthread+0x47/0x80 Feb 5 03:15:19 kaoz [ 1953.635914] [<ffffffff80238697>] schedule_tail+0x27/0x70 Feb 5 03:15:19 kaoz [ 1953.635919] [<ffffffff8020c7e9>] child_rip+0xa/0x11 Feb 5 03:15:19 kaoz [ 1953.635924] [<ffffffff80251910>] kthread+0x0/0x80 Feb 5 03:15:19 kaoz [ 1953.635929] [<ffffffff8020c7df>] child_rip+0x0/0x11 Feb 5 03:15:19 kaoz [ 1953.635932] krunner_lock D ffff88007c760870 0 7306 6670 Feb 5 03:15:19 kaoz [ 1953.635938] ffff880063e77cf0 0000000000000082 ffffffff8026f77d 0000000000000000 Feb 5 03:15:19 kaoz [ 1953.635945] ffffffff805de000 ffff88000395b330 ffff880055a63e90 ffff88000395b598 Feb 5 03:15:19 kaoz [ 1953.635952] 00000001805e19e0 ffffe200000b0d88 ffff88000395b598 0000000000000086 Feb 5 03:15:19 kaoz [ 1953.635958] Call Trace: Feb 5 03:15:19 kaoz [ 1953.635964] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0 Feb 5 03:15:19 kaoz [ 1953.635969] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0 Feb 5 03:15:19 kaoz [ 1953.635974] [<ffffffff8047588a>] mutex_lock+0xa/0x10 Feb 5 03:15:19 kaoz [ 1953.635979] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0 Feb 5 03:15:19 kaoz [ 1953.635985] [<ffffffff802f6848>] ext4_file_write+0x58/0x170 Feb 5 03:15:19 kaoz [ 1953.635991] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120 Feb 5 03:15:19 kaoz [ 1953.635997] [<ffffffff80227210>] do_page_fault+0x300/0x970 Feb 5 03:15:19 kaoz [ 1953.636003] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.636010] [<ffffffff802493af>] kill_pid_info+0x4f/0x80 Feb 5 03:15:19 kaoz [ 1953.636016] [<ffffffff802a008b>] vfs_write+0xcb/0x170 Feb 5 03:15:19 kaoz [ 1953.636022] [<ffffffff802a0233>] sys_write+0x53/0xa0 Feb 5 03:15:19 kaoz [ 1953.636027] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Feb 5 03:15:19 kaoz [ 1953.636031] krunner_lock D ffff88007c760870 0 7307 7306 Feb 5 03:15:19 kaoz [ 1953.636037] ffff88006364bcf0 0000000000000082 ffffe200000b0d88 ffffffff805e19e0 Feb 5 03:15:19 kaoz [ 1953.636044] ffffffff805de000 ffff880055a63e90 ffff88007c0b2220 ffff880055a640f8 Feb 5 03:15:19 kaoz [ 1953.636050] 00000001802791f5 ffffe200000b0d88 ffff880055a640f8 0000000000000000 Feb 5 03:15:19 kaoz [ 1953.636057] Call Trace: Feb 5 03:15:19 kaoz [ 1953.636062] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0 Feb 5 03:15:19 kaoz [ 1953.636066] [<ffffffff8047588a>] mutex_lock+0xa/0x10 Feb 5 03:15:19 kaoz [ 1953.636071] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0 Feb 5 03:15:19 kaoz [ 1953.636077] [<ffffffff802f6848>] ext4_file_write+0x58/0x170 Feb 5 03:15:19 kaoz [ 1953.636084] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120 Feb 5 03:15:19 kaoz [ 1953.636090] [<ffffffff80227210>] do_page_fault+0x300/0x970 Feb 5 03:15:19 kaoz [ 1953.636095] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.636101] [<ffffffff802493af>] kill_pid_info+0x4f/0x80 Feb 5 03:15:19 kaoz [ 1953.636108] [<ffffffff802a008b>] vfs_write+0xcb/0x170 Feb 5 03:15:19 kaoz [ 1953.636113] [<ffffffff802a0233>] sys_write+0x53/0xa0 Feb 5 03:15:19 kaoz [ 1953.636119] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Feb 5 03:15:19 kaoz [ 1953.636122] kblankscrn.ks D ffff880001020c20 0 7309 7306 Feb 5 03:15:19 kaoz [ 1953.636128] ffff8800635e3cf0 0000000000000086 ffffffff8026f77d 0000000000000000 Feb 5 03:15:19 kaoz [ 1953.636135] ffffffff805de000 ffff8800039585b0 ffff88007f862220 ffff880003958818 Feb 5 03:15:19 kaoz [ 1953.636141] 00000001805e19e0 000000010005efb2 ffff880003958818 ffffffff8056ffc8 Feb 5 03:15:19 kaoz [ 1953.636148] Call Trace: Feb 5 03:15:19 kaoz [ 1953.636154] [<ffffffff8026f77d>] find_get_page+0x8d/0xc0 Feb 5 03:15:19 kaoz [ 1953.636159] [<ffffffff80475a55>] __mutex_lock_slowpath+0x75/0xc0 Feb 5 03:15:19 kaoz [ 1953.636164] [<ffffffff8047588a>] mutex_lock+0xa/0x10 Feb 5 03:15:19 kaoz [ 1953.636169] [<ffffffff80271b1f>] generic_file_aio_write+0x4f/0xe0 Feb 5 03:15:19 kaoz [ 1953.636175] [<ffffffff802f6848>] ext4_file_write+0x58/0x170 Feb 5 03:15:19 kaoz [ 1953.636181] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120 Feb 5 03:15:19 kaoz [ 1953.636187] [<ffffffff80227210>] do_page_fault+0x300/0x970 Feb 5 03:15:19 kaoz [ 1953.636193] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.636200] [<ffffffff802866e6>] unmap_region+0x106/0x1a0 Feb 5 03:15:19 kaoz [ 1953.636206] [<ffffffff802a008b>] vfs_write+0xcb/0x170 Feb 5 03:15:19 kaoz [ 1953.636212] [<ffffffff802a0233>] sys_write+0x53/0xa0 Feb 5 03:15:19 kaoz [ 1953.636217] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Feb 5 03:15:19 kaoz [ 1953.636221] kblankscrn.ks D 0000000000000002 0 7311 7307 Feb 5 03:15:19 kaoz [ 1953.636227] ffff880042a2bb58 0000000000000086 ffff880024edb150 0000000000b29b6f Feb 5 03:15:19 kaoz [ 1953.636233] ffffffff805de000 ffff88000395b8e0 ffff8800039585b0 ffff88000395bb48 Feb 5 03:15:19 kaoz [ 1953.636240] 000000010395b8e0 000300018022f9c8 ffff88000395bb48 ffff880070bdd900 Feb 5 03:15:19 kaoz [ 1953.636246] Call Trace: Feb 5 03:15:19 kaoz [ 1953.636252] [<ffffffff8022f928>] set_next_entity+0x38/0x80 Feb 5 03:15:19 kaoz [ 1953.636259] [<ffffffff8031cf3b>] start_this_handle+0x33b/0x450 Feb 5 03:15:19 kaoz [ 1953.636265] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.636271] [<ffffffff8031d226>] jbd2_journal_start+0x96/0xe0 Feb 5 03:15:19 kaoz [ 1953.636277] [<ffffffff802f9c36>] ext4_dirty_inode+0x36/0xb0 Feb 5 03:15:19 kaoz [ 1953.636282] [<ffffffff802be284>] __mark_inode_dirty+0x34/0x1d0 Feb 5 03:15:19 kaoz [ 1953.636288] [<ffffffff802b3380>] file_update_time+0x110/0x130 Feb 5 03:15:19 kaoz [ 1953.636293] [<ffffffff80270fa6>] __generic_file_aio_write_nolock+0x246/0x470 Feb 5 03:15:19 kaoz [ 1953.636299] [<ffffffff80280332>] __do_fault+0x202/0x4c0 Feb 5 03:15:19 kaoz [ 1953.636304] [<ffffffff80271b34>] generic_file_aio_write+0x64/0xe0 Feb 5 03:15:19 kaoz [ 1953.636310] [<ffffffff802f6848>] ext4_file_write+0x58/0x170 Feb 5 03:15:19 kaoz [ 1953.636316] [<ffffffff8029f66b>] do_sync_write+0xdb/0x120 Feb 5 03:15:19 kaoz [ 1953.636322] [<ffffffff80227210>] do_page_fault+0x300/0x970 Feb 5 03:15:19 kaoz [ 1953.636328] [<ffffffff80251d20>] autoremove_wake_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.636333] [<ffffffff802866e6>] unmap_region+0x106/0x1a0 Feb 5 03:15:19 kaoz [ 1953.636340] [<ffffffff802a008b>] vfs_write+0xcb/0x170 Feb 5 03:15:19 kaoz [ 1953.636346] [<ffffffff802a0233>] sys_write+0x53/0xa0 Feb 5 03:15:19 kaoz [ 1953.636351] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b Feb 5 03:15:19 kaoz [ 1953.636355] updatedb D ffff880001020c20 0 7511 7506 Feb 5 03:15:19 kaoz [ 1953.636361] ffff8800372fdbf8 0000000000000086 ffff88007ea30000 0000000000000000 Feb 5 03:15:19 kaoz [ 1953.636367] ffffffff805de000 ffff88000395e660 ffff88007f862220 ffff88000395e8c8 Feb 5 03:15:19 kaoz [ 1953.636374] 000000017fa93a50 00000001000791d8 ffff88000395e8c8 ffff88007eb544f0 Feb 5 03:15:19 kaoz [ 1953.636381] Call Trace: Feb 5 03:15:19 kaoz [ 1953.636386] [<ffffffff802c3880>] sync_buffer+0x0/0x50 Feb 5 03:15:19 kaoz [ 1953.636392] [<ffffffff80330564>] generic_unplug_device+0x24/0x40 Feb 5 03:15:19 kaoz [ 1953.636412] [<ffffffffa008c6cc>] dm_table_unplug_all+0x3c/0x90 [dm_mod] Feb 5 03:15:19 kaoz [ 1953.636418] [<ffffffff802c3880>] sync_buffer+0x0/0x50 Feb 5 03:15:19 kaoz [ 1953.636424] [<ffffffff80475307>] io_schedule+0x37/0x50 Feb 5 03:15:19 kaoz [ 1953.636429] [<ffffffff802c38bd>] sync_buffer+0x3d/0x50 Feb 5 03:15:19 kaoz [ 1953.636433] [<ffffffff804757b0>] __wait_on_bit+0x50/0x80 Feb 5 03:15:19 kaoz [ 1953.636439] [<ffffffff802c3880>] sync_buffer+0x0/0x50 Feb 5 03:15:19 kaoz [ 1953.636444] [<ffffffff80475859>] out_of_line_wait_on_bit+0x79/0xa0 Feb 5 03:15:19 kaoz [ 1953.636449] [<ffffffff80251d50>] wake_bit_function+0x0/0x30 Feb 5 03:15:19 kaoz [ 1953.636455] [<ffffffff802fd5ba>] ext4_bread+0x5a/0x80 Feb 5 03:15:19 kaoz [ 1953.636461] [<ffffffff803032b3>] htree_dirblock_to_tree+0x33/0x1b0 Feb 5 03:15:19 kaoz [ 1953.636466] [<ffffffff803035a2>] ext4_htree_fill_tree+0x172/0x210 Feb 5 03:15:19 kaoz [ 1953.636472] [<ffffffff802aaf4e>] do_path_lookup+0x9e/0x1b0 Feb 5 03:15:19 kaoz [ 1953.636477] [<ffffffff802a0805>] file_move+0x25/0x70 Feb 5 03:15:19 kaoz [ 1953.636483] [<ffffffff802f6492>] ext4_readdir+0x4a2/0x600 Feb 5 03:15:19 kaoz [ 1953.636489] [<ffffffff802ae0b0>] filldir+0x0/0xe0 Feb 5 03:15:19 kaoz [ 1953.636494] [<ffffffff8029829f>] add_partial+0x1f/0x80 Feb 5 03:15:19 kaoz [ 1953.636500] [<ffffffff802a34c4>] cp_new_stat+0xe4/0x100 Feb 5 03:15:19 kaoz [ 1953.636506] [<ffffffff802ae0b0>] filldir+0x0/0xe0 Feb 5 03:15:19 kaoz [ 1953.636511] [<ffffffff802ae320>] vfs_readdir+0xb0/0xd0 Feb 5 03:15:19 kaoz [ 1953.636518] [<ffffffff802ae4b4>] sys_getdents+0x84/0xe0 Feb 5 03:15:19 kaoz [ 1953.636524] [<ffffffff8029e6ca>] sys_fchdir+0x4a/0x90 Feb 5 03:15:19 kaoz [ 1953.636529] [<ffffffff8020b8cb>] system_call_fastpath+0x16/0x1b ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-05 11:54 ` Federico Cuello @ 2009-02-09 13:45 ` Nick Piggin 2009-02-09 13:49 ` Ralf Hildebrandt 0 siblings, 1 reply; 14+ messages in thread From: Nick Piggin @ 2009-02-09 13:45 UTC (permalink / raw) To: Federico Cuello; +Cc: Ralf Hildebrandt, Artem Bityutskiy, linux-kernel On Thursday 05 February 2009 22:54:16 Federico Cuello wrote: > Nick Piggin wrote: > > On Thursday 05 February 2009 04:31:00 Federico Cuello wrote: > >> Nick Piggin wrote: > >>> [...] > >>> Thanks, could you reply-to-all when replying to retain ccs please? > >>> > >>> Common theme is ext4, which uses no_nrwrite_index_update, and I > >>> introduced a bug in there which could possibly cause ext4 to go into a > >>> loop... > >>> > >>> Would it be possible if you can test the following patch? > >> > >> I'll test it as soon as I get home. > > > > Thanks. > > > >> Meanwhile, I think the new patch may be slightly wrong. If I understand > >> correctly PageWriteback(page) is called before nr_to_write is tested for > >> being > 0 and then decremented if true, but "done" is not set to 1 > >> until the next iteration. So another call to PageWriteback(page) while > >> take place and then "done" will be set to true (if wbc->sync_mode == > >> WB_SYNC_NONE). > >> > >> If nr_to_write == 1 at the beginning of the loop then two pages will be > >> written. > >> > >> I think the test condition should something like: > >> > >> if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) { > >> done = 1; > >> break; > >> } > > > > I think you're quite right. Good catch. We probably want to prevent > > nr_to_write from going -ve, though. > > > > I think something like this > > > > if (nr_to_write > 0) > > nr_to_write--; > > if (!nr_to_write && wbc->sync_mode == WB_SYNC_NONE) { > > ... > > > > Would you care to send a patch? > > Ok, after extensive testing I haven't been able to solve the problem. > > I'm posting below the patch I used. I tried 3 different patches with one > successful test run with the one you sent me. I don't know if it was > just a coincidence as I had no time to test it again. > > Now, with the patch below, it stalls with 50% IO-wait (dual core, one > core at 100%). Perhaps the patch is part of the solution, I don't know. > > I also have the sysrq-W logs and I'm also posting them below. > > Thanks, > > > > diff --git a/mm/page-writeback.c b/mm/page-writeback.c > index 08d2b96..9e2ae50 100644 > --- a/mm/page-writeback.c > +++ b/mm/page-writeback.c > @@ -981,13 +981,23 @@ continue_unlock: > } > } > > - if (wbc->sync_mode == WB_SYNC_NONE) { > - wbc->nr_to_write--; > - if (wbc->nr_to_write <= 0) { > + if (nr_to_write > 0) { > + nr_to_write--; > + if (nr_to_write == 0 && wbc->sync_mode > == WB_SYNC_NONE) { > + /* > + * We stop writing back only if > we are not > + * doing integrity sync. In case > of integrity > + * sync we have to keep going > because someone > + * may be concurrently dirtying > pages, and we > + * might have synced a lot of > newly appeared > + * dirty pages, but have not > synced all of the > + * old dirty pages. > + */ > done = 1; > break; > } > } > + > if (wbc->nonblocking && bdi_write_congested(bdi)) { > wbc->encountered_congestion = 1; > done = 1; This patch seems good to me. If you would care to add a changelog and Signed-off-by: line, then we could get it merged? I am not too sure about this bug. I have reproduced a strange hang with ext4 (which does include sys_sync and write_cache_pages traces), and also turned up a lockdep report. Also, we haven't seen any reports of this problem on other filesystems. So it could be an ext4 bug. Your traces also have lots of tasks hung waiting for page lock. It is possible that wakeups get lost, which is fixed by this commit in mainline 777c6c5f1f6e757ae49ecca2ed72d6b1f523c007 Which might also be your bug. Any chance you can test this patch (as well as the existing patches you are using to fix write_cache_pages?). Thanks, Nick ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-09 13:45 ` Nick Piggin @ 2009-02-09 13:49 ` Ralf Hildebrandt 2009-02-15 13:42 ` Ralf Hildebrandt 0 siblings, 1 reply; 14+ messages in thread From: Ralf Hildebrandt @ 2009-02-09 13:49 UTC (permalink / raw) To: Nick Piggin; +Cc: Federico Cuello, Artem Bityutskiy, linux-kernel > This patch seems good to me. If you would care to add a changelog and > Signed-off-by: line, then we could get it merged? > > I am not too sure about this bug. I have reproduced a strange hang with > ext4 (which does include sys_sync and write_cache_pages traces), and > also turned up a lockdep report. Also, we haven't seen any reports of > this problem on other filesystems. So it could be an ext4 bug. > > Your traces also have lots of tasks hung waiting for page lock. It is > possible that wakeups get lost, which is fixed by this commit in > mainline > 777c6c5f1f6e757ae49ecca2ed72d6b1f523c007 > > Which might also be your bug. > > > Any chance you can test this patch (as well as the existing patches > you are using to fix write_cache_pages?). I could test 2.6.28.4 -- Ralf Hildebrandt Ralf.Hildebrandt@charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962 Hindenburgdamm 30 | 12200 Berlin ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-09 13:49 ` Ralf Hildebrandt @ 2009-02-15 13:42 ` Ralf Hildebrandt 2009-02-17 4:17 ` Nick Piggin 0 siblings, 1 reply; 14+ messages in thread From: Ralf Hildebrandt @ 2009-02-15 13:42 UTC (permalink / raw) To: linux-kernel; +Cc: Nick Piggin, Federico Cuello, Artem Bityutskiy * Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>: > > This patch seems good to me. If you would care to add a changelog and > > Signed-off-by: line, then we could get it merged? > > > > I am not too sure about this bug. I have reproduced a strange hang with > > ext4 (which does include sys_sync and write_cache_pages traces), and > > also turned up a lockdep report. Also, we haven't seen any reports of > > this problem on other filesystems. So it could be an ext4 bug. > > > > Your traces also have lots of tasks hung waiting for page lock. It is > > possible that wakeups get lost, which is fixed by this commit in > > mainline > > 777c6c5f1f6e757ae49ecca2ed72d6b1f523c007 > > > > Which might also be your bug. > > > > > > Any chance you can test this patch (as well as the existing patches > > you are using to fix write_cache_pages?). > > I could test 2.6.28.4 Still there in 2.6.28.5 :( -- Ralf Hildebrandt Ralf.Hildebrandt@charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962 Hindenburgdamm 30 | 12200 Berlin ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-15 13:42 ` Ralf Hildebrandt @ 2009-02-17 4:17 ` Nick Piggin 2009-02-17 15:15 ` Ralf Hildebrandt 0 siblings, 1 reply; 14+ messages in thread From: Nick Piggin @ 2009-02-17 4:17 UTC (permalink / raw) To: Ralf Hildebrandt, stable, linux-ext4 Cc: linux-kernel, Federico Cuello, Artem Bityutskiy On Monday 16 February 2009 00:42:35 Ralf Hildebrandt wrote: > * Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>: > > > This patch seems good to me. If you would care to add a changelog and > > > Signed-off-by: line, then we could get it merged? > > > > > > I am not too sure about this bug. I have reproduced a strange hang with > > > ext4 (which does include sys_sync and write_cache_pages traces), and > > > also turned up a lockdep report. Also, we haven't seen any reports of > > > this problem on other filesystems. So it could be an ext4 bug. > > > > > > Your traces also have lots of tasks hung waiting for page lock. It is > > > possible that wakeups get lost, which is fixed by this commit in > > > mainline > > > 777c6c5f1f6e757ae49ecca2ed72d6b1f523c007 > > > > > > Which might also be your bug. > > > > > > > > > Any chance you can test this patch (as well as the existing patches > > > you are using to fix write_cache_pages?). > > > > I could test 2.6.28.4 > > Still there in 2.6.28.5 :( I believe it is basically a deadlock in ext4 due to the cyclic writeback change. http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=2acf2c261b823d9d9ed954f348b97620297a36b5 ext4 guys, is this correct? Is there a minimal fix that can be made for .stable kernels? ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-17 4:17 ` Nick Piggin @ 2009-02-17 15:15 ` Ralf Hildebrandt 0 siblings, 0 replies; 14+ messages in thread From: Ralf Hildebrandt @ 2009-02-17 15:15 UTC (permalink / raw) To: linux-kernel * Nick Piggin <nickpiggin@yahoo.com.au>: > > Still there in 2.6.28.5 :( > > I believe it is basically a deadlock in ext4 due to the cyclic writeback > change. That could be. It happens during time of big write activity here. > http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commitdiff;h=2acf2c261b823d9d9ed954f348b97620297a36b5 > > ext4 guys, is this correct? Is there a minimal fix that can be made for > .stable kernels? > -- Ralf Hildebrandt Ralf.Hildebrandt@charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962 Hindenburgdamm 30 | 12200 Berlin ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: sync-Regression in 2.6.28.2? 2009-02-04 17:31 ` Federico Cuello 2009-02-05 3:25 ` Nick Piggin @ 2009-02-05 10:19 ` Ralf Hildebrandt 1 sibling, 0 replies; 14+ messages in thread From: Ralf Hildebrandt @ 2009-02-05 10:19 UTC (permalink / raw) To: Federico Cuello; +Cc: Nick Piggin, Artem Bityutskiy, linux-kernel * Federico Cuello <fedux@lugmen.org.ar>: > Nick Piggin wrote: > > [...] > > Thanks, could you reply-to-all when replying to retain ccs please? > > > > Common theme is ext4, which uses no_nrwrite_index_update, and I introduced > > a bug in there which could possibly cause ext4 to go into a loop... > > > > Would it be possible if you can test the following patch? > > > > I'll test it as soon as I get home. > > Meanwhile, I think the new patch may be slightly wrong. If I understand > correctly PageWriteback(page) is called before nr_to_write is tested for > being > 0 and then decremented if true, but "done" is not set to 1 > until the next iteration. So another call to PageWriteback(page) while > take place and then "done" will be set to true (if wbc->sync_mode == > WB_SYNC_NONE). > > If nr_to_write == 1 at the beginning of the loop then two pages will be > written. > > I think the test condition should something like: > > if (--nr_to_write <= 0 && wbc->sync_mode == WB_SYNC_NONE) { > done = 1; > break; > } > > . > . > > > Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com> > > Acked-by: Nick Piggin <npiggin@suse.de> > > Cc: Andrew Morton <akpm@linux-foundation.org> > > --- > > mm/page-writeback.c | 21 +++++++++++++++------ > > 1 files changed, 15 insertions(+), 6 deletions(-) > > > > diff --git a/mm/page-writeback.c b/mm/page-writeback.c > > index b493db7..dc32dae 100644 > > --- a/mm/page-writeback.c > > +++ b/mm/page-writeback.c > > @@ -1051,13 +1051,22 @@ continue_unlock: > > } > > } > > > > - if (wbc->sync_mode == WB_SYNC_NONE) { > > - wbc->nr_to_write--; > > - if (wbc->nr_to_write <= 0) { > > - done = 1; > > - break; > > - } > > + if (nr_to_write > 0) > > + nr_to_write--; > > + else if (wbc->sync_mode == WB_SYNC_NONE) { > > + /* > > + * We stop writing back only if we are not > > + * doing integrity sync. In case of integrity > > + * sync we have to keep going because someone > > + * may be concurrently dirtying pages, and we > > + * might have synced a lot of newly appeared > > + * dirty pages, but have not synced all of the > > + * old dirty pages. > > + */ > > + done = 1; > > + break; > > } > > + > > if (wbc->nonblocking && bdi_write_congested(bdi)) { > > wbc->encountered_congestion = 1; > > done = 1; > > > While I didn't test the patch, I can confirm that the problem is also in 2.6.28.3 -- Ralf Hildebrandt Ralf.Hildebrandt@charite.de Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155 Geschäftsbereich IT | Abt. Netzwerk Fax. +49 (0)30-450 570-962 Hindenburgdamm 30 | 12200 Berlin ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2009-02-17 15:15 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2009-01-27 9:35 sync-Regression in 2.6.28.2? Ralf Hildebrandt 2009-01-27 21:09 ` Federico Cuello 2009-02-03 1:09 ` Nick Piggin 2009-02-03 19:51 ` Federico Cuello 2009-02-04 6:17 ` Nick Piggin 2009-02-04 17:31 ` Federico Cuello 2009-02-05 3:25 ` Nick Piggin 2009-02-05 11:54 ` Federico Cuello 2009-02-09 13:45 ` Nick Piggin 2009-02-09 13:49 ` Ralf Hildebrandt 2009-02-15 13:42 ` Ralf Hildebrandt 2009-02-17 4:17 ` Nick Piggin 2009-02-17 15:15 ` Ralf Hildebrandt 2009-02-05 10:19 ` Ralf Hildebrandt
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox