* Kernel (2.6.11) deadlock in user mode when writing data through mmap on large files (64-bit systems, xfs or ext3)
@ 2005-05-09 13:18 Xavier Roche
2005-05-10 15:26 ` [XFS] Kernel (2.6.11) deadlock in user mode when writing data through mmap on large files (64-bit systems) Xavier Roche
0 siblings, 1 reply; 5+ messages in thread
From: Xavier Roche @ 2005-05-09 13:18 UTC (permalink / raw)
To: linux-kernel
Hi folks,
We managed to get a deadlock while writing data through mmap in a
context where memory is highly stressed.
If you launch the program below with a size far greater than your RAM
size (let's say 10 times), chances are that you will reproduce this
deadlock.
This deadlock happens on xfs and on ext3fs, using Linux 2.6.11, on an
EMT64 system.
$ uname -s -r -m -p -i -o
Linux 2.6.11-gentoo-r6 x86_64 Intel(R) Xeon(TM) CPU 3.20GHz GenuineIntel
GNU/Linux
Crash test:
========================================================================
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <stdio.h>
#include <string.h>
#define MEGA (1024 * 1024)
/**
* Usage : fname size (Mo)
**/
int main(int argc, char* argv[]) {
int fd0 = open(argv[1], O_RDWR | O_CREAT, S_IREAD | S_IWUSR);
size_t size0 = ((size_t) atoi(argv[2])) * MEGA;
char* ptr0;
if (ftruncate(fd0, size0) != 0) {
printf("error truncating file");
return 1;
}
ptr0 = mmap(NULL,
size0,
(PROT_READ | PROT_WRITE),
(MAP_FILE | MAP_SHARED),
fd0,
0);
{
size_t i;
for(i = 0; i < size0; i += MEGA) {
(void) memset(ptr0 + i, 0, MEGA);
if (i % ((100 * MEGA)) == 0) {
int m = (int) (i / MEGA);
int t = (int) (size0 / MEGA);
printf("Creating : %d/%d\n", m, t);
}
}
}
if (msync((void*) ptr0, size0, MS_SYNC) != 0) {
printf("error syncing file");
return 1;
}
if (munmap(ptr0, size0) != 0) {
printf("error closing file");
return 1;
}
return 0;
}
========================================================================
Below, the SysRq output (For more lisibility, in the Show State section,
I moved on top the output of the incriminated program (ngtest))
========================================================================
telnet> send break
SysRq : Changing Loglevel
Loglevel set to 8
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
[...]
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
telnet> send break
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
SysRq : Show State
sibling
task PC pid father child younger older
ngtest D ffff810078303468 0 20514 20505
(NOTLB)
ffff810078302eb8 0000000000000086 00000001013820a7 0000000000000008
0000000000000086 0000006e7c5cfe88 ffffffff805915c0 ffff81007fe095a0
000000000000006e ffff810002c1a780
Call Trace:<ffffffff80139d89>{__mod_timer+318}
<ffffffff803782c2>{schedule_timeout+165}
<ffffffff8013a8d0>{process_timeout+0}
<ffffffff803781ff>{io_schedule_timeout+49}
<ffffffff802b94cf>{blk_congestion_wait+151}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8022982e>{kmem_alloc+210}
<ffffffff802298cd>{kmem_realloc+43}
<ffffffff8020d181>{xfs_iext_realloc+245}
<ffffffff801e62a8>{xfs_bmap_insert_exlist+53}
<ffffffff801e7194>{xfs_bmap_add_extent_hole_real+949}
<ffffffff801e84df>{xfs_bmap_add_extent+4759}
<ffffffff8021f92e>{xfs_trans_brelse+84}
<ffffffff8021faa8>{xfs_trans_log_buf+107}
<ffffffff801dbb04>{xfs_alloc_ag_vextent+4013}
<ffffffff8022995c>{kmem_zone_alloc+70}
<ffffffff801f215f>{xfs_btree_init_cursor+72}
<ffffffff801eb7b7>{xfs_bmapi+6221}
<ffffffff801e8be8>{xfs_bmap_do_search_extents+778}
<ffffffff80210443>{xfs_iomap_write_direct+646}
<ffffffff803786c7>{__down_write+129}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8020ff2d>{xfs_iomap+569}
<ffffffff802ba0ad>{submit_bio+221}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff801653c5>{page_referenced_file+209}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000}
<ffffffff8022ab3d>{__linvfs_get_block+136}
<ffffffff80241fc2>{__memset+50}
<ffffffff80192ed3>{do_mpage_readpage+949}
<ffffffff80157148>{do_drain+0} <ffffffff8012b801>{try_to_wake_up+755}
<ffffffff8023e937>{radix_tree_node_alloc+19}
<ffffffff8023eb29>{radix_tree_insert+291}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff8015575e>{__do_page_cache_readahead+215}
<ffffffff8014fe07>{filemap_nopage+347}
<ffffffff8015f511>{do_no_page+984}
<ffffffff8015f8e4>{handle_mm_fault+419}
<ffffffff80378992>{_spin_unlock_irqrestore+5}
<ffffffff8011d090>{do_page_fault+1185}
<ffffffff8010dd9d>{error_exit+0}
init D 0000000000000000 0 1 0 2
(NOTLB)
ffff81007ff13728 0000000000000082 0000000000000000 0000000000000000
ffff81007f8d1e00 ffffffff802bec15 0000000100000000 ffff81007958dbe8
ffff81007fd2daa0 0000000000000003
Call Trace:<ffffffff802bec15>{as_insert_request+1073}
<ffffffff802b9ba8>{__make_request+1270}
<ffffffff803786c7>{__down_write+129}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8020a319>{xfs_ilock+96} <ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff802ba0ad>{submit_bio+221} <ffffffff80173352>{submit_bh+295}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff801653c5>{page_referenced_file+209}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000}
<ffffffff80157148>{do_drain+0} <ffffffff80157148>{do_drain+0}
<ffffffff8011978b>{flat_send_IPI_allbutself+20}
<ffffffff801176c4>{__smp_call_function+111}
<ffffffff80157148>{do_drain+0}
<ffffffff8012b2d9>{recalc_task_prio+317}
<ffffffff8012b801>{try_to_wake_up+755}
<ffffffff8019beb8>{mb_cache_shrink_fn+100}
<ffffffff8012bbae>{finish_task_switch+64}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff801536e7>{__get_free_pages+14}
<ffffffff80182e2c>{__pollwait+75} <ffffffff8017c8d5>{pipe_poll+67}
<ffffffff801831a8>{do_select+727} <ffffffff80182de1>{__pollwait+0}
<ffffffff80183694>{sys_select+910}
<ffffffff8010d39a>{system_call+126}
migration/0 S ffff810002c12760 0 2 1 3
(L-TLB)
ffff8100032bdeb8 0000000000000046 0000000000000086 0000000000000004
0000000000000003 0000007500000000 ffffffff805915c0 ffff81007ec19820
0000000000000075 ffff810002c12780
Call Trace:<ffffffff8012e2f4>{migration_thread+518}
<ffffffff8012e0ee>{migration_thread+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
ksoftirqd/0 S ffffffff80565fb0 0 3 1 4 2
(L-TLB)
ffff8100032c1f08 0000000000000046 ffff8100790b1210 0000000000000005
ffffffff80594a00 000000000000000a 0000000000000000 0000000000000292
0000000000000292 0000000000000000
Call Trace:<ffffffff80136883>{tasklet_action+107}
<ffffffff801364b6>{__do_softirq+110}
<ffffffff801369ea>{ksoftirqd+0} <ffffffff801369ea>{ksoftirqd+0}
<ffffffff80136a2a>{ksoftirqd+64} <ffffffff801369ea>{ksoftirqd+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
migration/1 S ffff810002c1a760 0 4 1 5 3
(L-TLB)
ffff8100032c3eb8 0000000000000046 0000000000000086 0000000000000008
0000000000000003 0000007600000000 ffffffff805915c0 ffff81007ec19820
0000000000000076 ffff810002c1a780
Call Trace:<ffffffff8012e2f4>{migration_thread+518}
<ffffffff8012e0ee>{migration_thread+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
ksoftirqd/1 S ffffffff80565fb0 0 5 1 6 4
(L-TLB)
ffff8100032d5f08 0000000000000046 ffff8100781a9870 0000000000000001
ffffffff80594a00 0000000000000008 0000000000000001 0000000000000292
0000000000000292 0000000000000000
Call Trace:<ffffffff80136883>{tasklet_action+107}
<ffffffff801364b6>{__do_softirq+110}
<ffffffff801369ea>{ksoftirqd+0} <ffffffff801369ea>{ksoftirqd+0}
<ffffffff80136a2a>{ksoftirqd+64} <ffffffff801369ea>{ksoftirqd+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
migration/2 S ffff810002c22760 0 6 1 7 5
(L-TLB)
ffff8100032dbeb8 0000000000000046 0000000000000086 0000000000000001
0000000000000003 0000007400000000 ffffffff805915c0 ffff81007ec19820
0000000000000074 ffff810002c22780
Call Trace:<ffffffff8012e2f4>{migration_thread+518}
<ffffffff8012e0ee>{migration_thread+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
ksoftirqd/2 S ffffffff80565fb0 0 7 1 8 6
(L-TLB)
ffff8100032ddf08 0000000000000046 ffff8100795a1990 0000000000000009
ffffffff80594a00 000000000000000a 0000000000000002 0000000000000292
0000000000000292 0000000000000000
Call Trace:<ffffffff80136883>{tasklet_action+107}
<ffffffff801364b6>{__do_softirq+110}
<ffffffff801369ea>{ksoftirqd+0} <ffffffff801369ea>{ksoftirqd+0}
<ffffffff80136a2a>{ksoftirqd+64} <ffffffff801369ea>{ksoftirqd+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
migration/3 S ffff810002c2a760 0 8 1 9 7
(L-TLB)
ffff8100032efeb8 0000000000000046 0000000000000086 0000000000000002
0000000000000003 0000007400000000 ffffffff805915c0 ffff810029b97660
0000000000000074 ffff810002c2a780
Call Trace:<ffffffff8012e2f4>{migration_thread+518}
<ffffffff8012e0ee>{migration_thread+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
ksoftirqd/3 S ffffffff80565fb0 0 9 1 10 8
(L-TLB)
ffff8100032f3f08 0000000000000046 0000000000018000 0000000000018000
0000000000000000 ffffffff802f017f 0000000000000001 ffffffff802e2dcf
ffff81007f89c800 0000000000002002
Call Trace:<ffffffff802f017f>{sd_rw_intr+704}
<ffffffff802e2dcf>{scsi_device_unbusy+106}
<ffffffff801364b6>{__do_softirq+110} <ffffffff801369ea>{ksoftirqd+0}
<ffffffff801369ea>{ksoftirqd+0} <ffffffff80136a2a>{ksoftirqd+64}
<ffffffff801369ea>{ksoftirqd+0} <ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8} <ffffffff80145907>{kthread+0}
<ffffffff8010df4b>{child_rip+0}
events/0 S ffffffff80157f98 0 10 1 11 9
(L-TLB)
ffff8100032ffe58 0000000000000046 ffff81000143e848 ffff81000000c590
ffff81000000c500 0000000000000292 0000000000000292 0000000000000286
0000000000000286 0000000000000001
Call Trace:<ffffffff8012d15d>{__wake_up+67} <ffffffff80157f98>{cache_reap+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80141857>{worker_thread+0} <ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8} <ffffffff80145907>{kthread+0}
<ffffffff8010df4b>{child_rip+0}
events/1 S ffffffff80157f98 0 11 1 12 10
(L-TLB)
ffff81007fe0be58 0000000000000046 ffff810003291cc0 0000000000000008
ffff8100127fa000 0000007680156df0 ffffffff805915c0 ffff81007f0e5110
0000000000000076 ffff810002c1a780
Call Trace:<ffffffff8012d15d>{__wake_up+67} <ffffffff80157f98>{cache_reap+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80141857>{worker_thread+0} <ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8} <ffffffff80145907>{kthread+0}
<ffffffff8010df4b>{child_rip+0}
events/2 S ffffffff80157f98 0 12 1 13 11
(L-TLB)
ffff81007fe0fe58 0000000000000046 ffff810003291cc0 0000000000000001
ffff810005d94000 0000007380156df0 ffffffff805915c0 ffff81007f871760
0000000000000073 ffff810002c22780
Call Trace:<ffffffff8012d15d>{__wake_up+67} <ffffffff80157f98>{cache_reap+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80141857>{worker_thread+0} <ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8} <ffffffff80145907>{kthread+0}
<ffffffff8010df4b>{child_rip+0}
events/3 S ffffffff8024f486 0 13 1 14 12
(L-TLB)
ffff81007fe13e58 0000000000000046 0000000000000001 0000000000000002
ffff810002c2b600 0000007300000292 ffffffff805915c0 ffff81007ecc2b00
0000000000000073 ffff810002c2a780
Call Trace:<ffffffff8012d15d>{__wake_up+67}
<ffffffff8024f486>{flush_to_ldisc+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80141857>{worker_thread+0} <ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8} <ffffffff80145907>{kthread+0}
<ffffffff8010df4b>{child_rip+0}
khelper S ffff81005db1bcf8 0 14 1 19 13
(L-TLB)
ffff81007fe15e58 0000000000000046 ffff81005db1bd00 0000000000000008
0000000000000292 000000705db1bd68 ffffffff805915c0 ffff81007ecc38a0
0000000000000070 ffff810002c1a780
Call Trace:<ffffffff8012d15d>{__wake_up+67} <ffffffff8010df4b>{child_rip+0}
<ffffffff8014141f>{__call_usermodehelper+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
kthread S ffffffff80145a1c 0 19 1 33 136 14
(L-TLB)
ffff81007fe2be58 0000000000000046 0000000000000082 ffffffff8012b801
0000000000000003 000000017fe2bde8 ffff810002c19520 0000000000000002
000000017fe2be38 0000000000000002
Call Trace:<ffffffff8012b801>{try_to_wake_up+755}
<ffffffff8012d15d>{__wake_up+67}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
kacpid S ffffffff80145a1c 0 33 19 120
(L-TLB)
ffff81007fd2fe58 0000000000000046 ffffffff804f4160 0000000000000118
000000000000003a 0000000000000000 ffffffff804f4160 0000000000000000
0000000202c127e0 ffff81007fd2fe80
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
kblockd/0 S ffffffff802b84a3 0 120 19 121 33
(L-TLB)
ffff81007fda5e58 0000000000000046 0000000000000000 ffffffff802bdf81
ffffffff04000000 0000002300000060 ffff81007f89cb28 ffff81007f8d2280
000000007fda5da0 ffff81007f8d2280
Call Trace:<ffffffff802bdf81>{as_remove_queued_request+278}
<ffffffff8012d15d>{__wake_up+67}
<ffffffff802b84a3>{blk_unplug_work+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0}
<ffffffff8010df4b>{child_rip+0}
kblockd/1 S ffffffff802bf4cb 0 121 19 122 120
(L-TLB)
ffff81007fda9e58 0000000000000046 ffffffff00000000 000000bf00000060
ffff81007f89cb28 ffff81006b19a500 000000006b19b780 ffff81006b19a500
0000000000000292 ffff81007f89c800
Call Trace:<ffffffff8012d15d>{__wake_up+67}
<ffffffff802bf4cb>{as_work_handler+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
kblockd/2 S ffffffff802b84a3 0 122 19 123 121
(L-TLB)
ffff81007fdabe58 0000000000000046 0000000000000001 00002002802bdf81
0000022b04000000 000000ec00000060 ffff81007f89cb28 ffff81007db3ac80
0000022b7fdabda0 ffff81007db3ac80
Call Trace:<ffffffff8012d15d>{__wake_up+67}
<ffffffff802b84a3>{blk_unplug_work+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
kblockd/3 S ffffffff802b84a3 0 123 19 194 122
(L-TLB)
ffff81007fdade58 0000000000000046 0000000000000000 ffff81007f812200
ffffffff04000000 0000001e00000060 ffff81007f89cb28 ffff81007f8d2280
00000000032ca780 ffff81007f8d2280
Call Trace:<ffffffff8012d15d>{__wake_up+67}
<ffffffff802b84a3>{blk_unplug_work+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
khubd S ffffffff80565fb0 0 136 1 196 19
(L-TLB)
ffff81007f803e38 0000000000000046 0000000000000000 ffffffffffffffff
0000003000000008 ffff81007f803e48 ffff81007f803d88 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80309d69>{hub_thread+2583}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8010df53>{child_rip+8}
<ffffffff80309352>{hub_thread+0} <ffffffff8010df4b>{child_rip+0}
pdflush D ffffffff8014e993 0 194 19 197 123
(L-TLB)
ffff81007f8159c8 0000000000000046 ffffffff80145f95 ffff81007f815920
ffff81007f815920 ffffffff802bdf81 0000000000000000 ffff81007fda2880
ffffffff80145f95 ffff81007f815920
Call Trace:<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff802bdf81>{as_remove_queued_request+278}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff802e3f3d>{scsi_request_fn+1143}
<ffffffff8014e993>{sync_page+0} <ffffffff803781c0>{io_schedule+49}
<ffffffff8014e9d5>{sync_page+66}
<ffffffff80378527>{__wait_on_bit_lock+66}
<ffffffff8014f0b1>{__lock_page+164}
<ffffffff80145fc3>{wake_bit_function+0}
<ffffffff8014f3e4>{find_get_pages_tag+154}
<ffffffff80145fc3>{wake_bit_function+0}
<ffffffff801590ce>{pagevec_lookup_tag+26}
<ffffffff801931e3>{mpage_writepages+409}
<ffffffff8011ab7e>{dma_map_sg+660}
<ffffffff8022aee2>{linvfs_writepage+0}
<ffffffff802b6772>{elv_next_request+260}
<ffffffff802e3f3d>{scsi_request_fn+1143}
<ffffffff8022c94e>{pagebuf_iorequest+993}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff802158cf>{xfs_log_force+148}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80191f5d>{__writeback_single_inode+486}
<ffffffff8021ed33>{xfs_trans_first_ail+28}
<ffffffff80213084>{xfs_log_need_covered+168}
<ffffffff80222546>{xfs_syncsub+2710}
<ffffffff8019267d>{sync_sb_inodes+505}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff8019292c>{writeback_inodes+146}
<ffffffff8015497a>{wb_kupdate+158}
<ffffffff8015546d>{pdflush+323} <ffffffff801548dc>{wb_kupdate+0}
<ffffffff8015532a>{pdflush+0} <ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
kswapd0 D 0000000000000000 0 196 1 209 136
(L-TLB)
ffff81007f8198a8 0000000000000046 0000000100000000 0000000100000001
ffff81007fd2daa0 ffff81000001e238 ffff810033dc0580 ffffffff802bf06f
0000000000000000 0000000000000008
Call Trace:<ffffffff802bf06f>{as_merged_request+86}
<ffffffff802b9ba8>{__make_request+1270}
<ffffffff803786c7>{__down_write+129}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8020a319>{xfs_ilock+96} <ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff802ba0ad>{submit_bio+221} <ffffffff80173352>{submit_bh+295}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff801653c5>{page_referenced_file+209}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000}
<ffffffff8015ad0a>{balance_pgdat+540} <ffffffff8015af9d>{kswapd+315}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8010df53>{child_rip+8} <ffffffff8015ae62>{kswapd+0}
<ffffffff8010df4b>{child_rip+0}
aio/0 S ffffffff80145a1c 0 197 19 198 194
(L-TLB)
ffff81007f82be58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
aio/1 S ffffffff80145a1c 0 198 19 199 197
(L-TLB)
ffff81007f82fe58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
aio/2 S ffffffff80145a1c 0 199 19 200 198
(L-TLB)
ffff81007f833e58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
aio/3 S ffffffff80145a1c 0 200 19 201 199
(L-TLB)
ffff81007f835e58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
xfslogd/0 S ffffffff80145a1c 0 201 19 202 200
(L-TLB)
ffff81007f859e58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
xfslogd/1 S ffffffff80145a1c 0 202 19 203 201
(L-TLB)
ffff81007f85de58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
xfslogd/2 S ffffffff80145a1c 0 203 19 204 202
(L-TLB)
ffff81007f85fe58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
xfslogd/3 S ffff81004fb89730 0 204 19 205 203
(L-TLB)
ffff81007f863e58 0000000000000046 ffff81007d485cc0 0000000000000002
0000000000000000 000000767fe07c50 ffffffff805915c0 ffff81007f0e5110
0000000000000076 ffff810002c2a780
Call Trace:<ffffffff8012d15d>{__wake_up+67}
<ffffffff8022c32c>{pagebuf_iodone_work+0}
<ffffffff8022c32c>{pagebuf_iodone_work+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218} <ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0}
<ffffffff8010df4b>{child_rip+0}
xfsdatad/0 S ffffffff80145a1c 0 205 19 206 204
(L-TLB)
ffff81007f865e58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
xfsdatad/1 S ffffffff80145a1c 0 206 19 207 205
(L-TLB)
ffff81007f869e58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
xfsdatad/2 S ffffffff80145a1c 0 207 19 208 206
(L-TLB)
ffff81007f86de58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
xfsdatad/3 S ffffffff80145a1c 0 208 19 20564 207
(L-TLB)
ffff81007f86fe58 0000000000000046 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80141989>{worker_thread+306}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0f3>{__wake_up_common+64}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80141857>{worker_thread+0}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
xfsbufd S ffffffff80565fb0 0 209 1 284 196
(L-TLB)
ffff81007f873e98 0000000000000046 ffffffff80145f95 0000000000000001
0000000100000001 0000000000000060 ffff81007f902750 ffff81007f89c000
ffff81007f902720 0000000000000000
Call Trace:<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80139d89>{__mod_timer+318}
<ffffffff803782c2>{schedule_timeout+165}
<ffffffff8013a8d0>{process_timeout+0}
<ffffffff8022d292>{pagebuf_daemon+144}
<ffffffff8010df53>{child_rip+8}
<ffffffff8022d202>{pagebuf_daemon+0} <ffffffff8010df4b>{child_rip+0}
kseriod S ffffffff80565fb0 0 284 1 351 209
(L-TLB)
ffff81007f8b9eb8 0000000000000046 0000000000000000 ffffffffffffffff
0000003000000008 ffff81007f8b9ec8 ffff81007f8b9e08 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff802a4aa4>{serio_thread+443}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8010df53>{child_rip+8}
<ffffffff802a48e9>{serio_thread+0} <ffffffff8010df4b>{child_rip+0}
scsi_eh_0 S 0000000000000000 0 351 1 370 284
(L-TLB)
ffff81007f8fbde8 0000000000000046 0000000401c000e1 ffff81007ff0b001
ffff810003256b00 ffff8100032524b0 0000000100000000 ffff81007f86b344
0000000000000002 ffff81007f86b34b
Call Trace:<ffffffff8012b36c>{activate_task+141}
<ffffffff80376ac0>{__down_interruptible+208}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff80378824>{__down_failed_interruptible+53}
<ffffffff802e2ace>{.text.lock.scsi_error+45}
<ffffffff8014141f>{__call_usermodehelper+0}
<ffffffff801342b8>{do_exit+2917} <ffffffff8010df53>{child_rip+8}
<ffffffff802e1bba>{scsi_error_handler+0}
<ffffffff8010df4b>{child_rip+0}
kjournald S 0000000000000000 0 370 1 389 351
(L-TLB)
ffff81007fb5fe58 0000000000000046 ffff810078b84978 ffff810078b84978
ffff810078b84978 ffff810078b84978 ffff810078b84978 ffff810078b84978
ffff810078b84978 ffff810078b84978
Call Trace:<ffffffff8012d15d>{__wake_up+67}
<ffffffff801bfe70>{kjournald+534}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff801bfc54>{commit_timeout+0} <ffffffff8010df53>{child_rip+8}
<ffffffff801bfc5a>{kjournald+0} <ffffffff8010df4b>{child_rip+0}
udevd S 0000000000000028 0 389 1 3616 370
(NOTLB)
ffff81007ecadd78 0000000000000082 0000000000000292 ffffffff80153152
0000000000000292 0000007480153152 ffffffff805915c0 ffff81007c0e5110
0000000000000074 000000d002c22780
Call Trace:<ffffffff80153152>{buffered_rmqueue+524}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff801832b9>{do_select+1000}
<ffffffff80182de1>{__pollwait+0} <ffffffff80183694>{sys_select+910}
<ffffffff8010d39a>{system_call+126}
kjournald S 0000000000000000 0 3616 1 3617 389
(L-TLB)
ffff81007cd57e58 0000000000000046 ffff81007efc6400 ffffffff80131bb4
0000003000000010 ffff81007cd57e68 ffff81007cd57da8 ffff81007ce33b48
ffff81007efc64a0 0000000000000005
Call Trace:<ffffffff80131bb4>{printk+141} <ffffffff8012d15d>{__wake_up+67}
<ffffffff801bfe70>{kjournald+534}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff801bfc54>{commit_timeout+0}
<ffffffff8010df53>{child_rip+8} <ffffffff801bfc5a>{kjournald+0}
<ffffffff8010df4b>{child_rip+0}
kjournald S ffff81007efc6200 0 3617 1 3618 3616
(L-TLB)
ffff81007d289e58 0000000000000046 ffff81001d68d660 0000000000000002
ffff81001d68d660 000000761d68d660 ffffffff805915c0 ffff81007f0e5110
0000000000000076 ffff810002c2a780
Call Trace:<ffffffff8012d15d>{__wake_up+67}
<ffffffff801bfe70>{kjournald+534}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff801bfc54>{commit_timeout+0} <ffffffff8010df53>{child_rip+8}
<ffffffff801bfc5a>{kjournald+0} <ffffffff8010df4b>{child_rip+0}
kjournald S ffff81007d682200 0 3618 1 3619 3617
(L-TLB)
ffff81007d397e58 0000000000000046 ffff81005bc965b0 0000000000000002
ffff81005bc965b0 000000765bc965b0 ffffffff805915c0 ffff81007f0e5110
0000000000000076 ffff810002c2a780
Call Trace:<ffffffff8012d15d>{__wake_up+67}
<ffffffff801bfe70>{kjournald+534}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff801bfc54>{commit_timeout+0} <ffffffff8010df53>{child_rip+8}
<ffffffff801bfc5a>{kjournald+0} <ffffffff8010df4b>{child_rip+0}
xfssyncd S 0000000000007530 0 3619 1 4821 3618
(L-TLB)
ffff81007d413e88 0000000000000046 0000000000000031 0000000000000002
ffffffff8037885e 0000007680222664 ffffffff805915c0 ffff81007f0e5110
ffffffff8021ed33 ffff81007f8a3800
Call Trace:<ffffffff8037885e>{__down_failed_trylock+53}
<ffffffff8021ed33>{xfs_trans_first_ail+28}
<ffffffff80222546>{xfs_syncsub+2710}
<ffffffff80139d89>{__mod_timer+318}
<ffffffff803782c2>{schedule_timeout+165}
<ffffffff8013a8d0>{process_timeout+0}
<ffffffff80232105>{xfssyncd+150}
<ffffffff80232621>{linvfs_fill_super+0}
<ffffffff8010df53>{child_rip+8}
<ffffffff80232621>{linvfs_fill_super+0}
<ffffffff80221950>{xfs_root+0} <ffffffff8023206f>{xfssyncd+0}
<ffffffff8010df4b>{child_rip+0}
syslogd D 0000000000000000 0 4821 1 4887 3619
(NOTLB)
ffff81007c7ab738 0000000000000086 0000000000000000 ffffffff801bf934
ffff81001d68d608 ffff81007efc635c ffff81001d68d608 ffff81007b435710
0000000000000000 ffffffff801bbd3c
Call Trace:<ffffffff801bf934>{journal_cancel_revoke+214}
<ffffffff801bbd3c>{do_get_write_access+1622}
<ffffffff803786c7>{__down_write+129} <ffffffff8020a319>{xfs_ilock+96}
<ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff801bf934>{journal_cancel_revoke+214}
<ffffffff8017516d>{__getblk+31} <ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff801653c5>{page_referenced_file+209}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000}
<ffffffff801b3b24>{__ext3_journal_stop+45}
<ffffffff801ab82f>{ext3_ordered_commit_write+183}
<ffffffff80150ab9>{generic_file_buffered_write+1162}
<ffffffff80322190>{skb_dequeue+95}
<ffffffff80322a71>{skb_recv_datagram+141}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff801536e7>{__get_free_pages+14}
<ffffffff80182e2c>{__pollwait+75}
<ffffffff80323217>{datagram_poll+36}
<ffffffff801831a8>{do_select+727}
<ffffffff80182de1>{__pollwait+0} <ffffffff80183694>{sys_select+910}
<ffffffff8010d39a>{system_call+126}
klogd S ffff81007c5cfd68 0 4887 1 5002 4821
(NOTLB)
ffff81007c5cfbe8 0000000000000082 ffff81007c5cfb30 ffff81007ede2470
0000000000000000 ffff81007ede2470 ffffffff80145f95 0000000000000092
0000000000000092 ffffffff8011a00e
Call Trace:<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8011a00e>{flush_gart+225}
<ffffffff8012b36c>{activate_task+141}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80145eb9>{prepare_to_wait_exclusive+35}
<ffffffff8036d39e>{unix_wait_for_peer+217}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8012d15d>{__wake_up+67}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80322773>{memcpy_fromiovec+55}
<ffffffff8036df2c>{unix_dgram_sendmsg+960}
<ffffffff8031cf65>{sock_aio_write+287}
<ffffffff80170bbd>{do_sync_write+173}
<ffffffff80131394>{do_syslog+536}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8019d9ad>{dnotify_parent+51}
<ffffffff80170cbd>{vfs_write+211} <ffffffff80170df7>{sys_write+83}
<ffffffff8010d39a>{system_call+126}
gpm S 0000000000000018 0 5002 1 5447 4887
(NOTLB)
ffff81007c3e7d78 0000000000000086 0000000000000086 ffffffff8012b801
0000000000000003 0000000000000296 0000000000000296 ffffffff80153152
000000017fd8c528 0000000000000002
Call Trace:<ffffffff8012b801>{try_to_wake_up+755}
<ffffffff80153152>{buffered_rmqueue+524}
<ffffffff80139d89>{__mod_timer+318}
<ffffffff803782c2>{schedule_timeout+165}
<ffffffff8013a8d0>{process_timeout+0}
<ffffffff801832b9>{do_select+1000}
<ffffffff80182de1>{__pollwait+0} <ffffffff80183694>{sys_select+910}
<ffffffff8010d39a>{system_call+126}
ntpd D 0000000000000000 0 5447 1 5499 5002
(NOTLB)
ffff81007bd23718 0000000000000086 ffff810054d0ab88 ffffffff80172d56
ffff81007eadd6c0 0000000000000296 0000000000000003 ffff81007eadd6c0
ffff81007bd23758 ffffffff803777de
Call Trace:<ffffffff80172d56>{__find_get_block+400}
<ffffffff803777de>{thread_return+42}
<ffffffff801bf934>{journal_cancel_revoke+214}
<ffffffff803786c7>{__down_write+129}
<ffffffff8020a319>{xfs_ilock+96} <ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff801bbd3c>{do_get_write_access+1622}
<ffffffff801bbd3c>{do_get_write_access+1622}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000}
<ffffffff8032b34f>{neigh_resolve_output+622}
<ffffffff8033a645>{ip_finish_output+404}
<ffffffff8033c65d>{ip_push_pending_frames+821}
<ffffffff80356e32>{udp_push_pending_frames+530}
<ffffffff8031f705>{release_sock+25}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff801536e7>{__get_free_pages+14}
<ffffffff80182e2c>{__pollwait+75}
<ffffffff80323217>{datagram_poll+36} <ffffffff803587b1>{udp_poll+15}
<ffffffff801831a8>{do_select+727} <ffffffff80182de1>{__pollwait+0}
<ffffffff80183694>{sys_select+910}
<ffffffff8010d39a>{system_call+126}
portmap S 0000000000000000 0 5499 1 5550 5447
(NOTLB)
ffff81007c60fe88 0000000000000086 0000000000000292 0000000000000002
00002aaaaae797f4 000000767c4a64e8 ffffffff805915c0 ffff81007c442980
0000000000000076 ffff810002c2a780
Call Trace:<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff80183ab0>{sys_poll+620}
<ffffffff80182de1>{__pollwait+0} <ffffffff8010d39a>{system_call+126}
snmpd S 0000000000000f50 0 5550 1 5601 5499
(NOTLB)
ffff81007b83bd78 0000000000000082 ffff81007d779460 0000000000000292
0000000000000292 ffffffff80153152 ffff81007b83be18 ffffffff80151155
0000000000000015 ffff81007cb14aa0
Call Trace:<ffffffff80153152>{buffered_rmqueue+524}
<ffffffff80151155>{__generic_file_aio_write_nolock+964}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff80323217>{datagram_poll+36} <ffffffff803587b1>{udp_poll+15}
<ffffffff801832b9>{do_select+1000} <ffffffff80182de1>{__pollwait+0}
<ffffffff80183694>{sys_select+910}
<ffffffff8010d39a>{system_call+126}
sshd D 0000000000000000 0 5601 1 20493 5644 5550
(NOTLB)
ffff81007ba4d7d8 0000000000000086 0000000000001fc9 000171324f6ebb1b
ffff81007ff10030 ffff81007f8709c0 ffff81007f870c38 ffff81007f810000
0000000000000002 0000000000000000
Call Trace:<ffffffff8011ab7e>{dma_map_sg+660}
<ffffffff803786c7>{__down_write+129}
<ffffffff8020a319>{xfs_ilock+96} <ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff801ab3b6>{ext3_get_branch+96}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000}
<ffffffff8012bbae>{finish_task_switch+64}
<ffffffff803777de>{thread_return+42}
<ffffffff80167a23>{swap_info_get+189}
<ffffffff80167a86>{swap_info_put+13}
<ffffffff80167c1b>{can_share_swap_page+233}
<ffffffff8015fbaf>{handle_mm_fault+1134}
<ffffffff8023fa03>{__up_read+33}
<ffffffff8011d0d3>{do_page_fault+1252}
<ffffffff8014f10f>{find_get_page+88}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff80156758>{cache_alloc_refill+414}
<ffffffff801536e7>{__get_free_pages+14}
<ffffffff8012f72f>{copy_process+209}
<ffffffff8011d0d3>{do_page_fault+1252}
<ffffffff80130966>{do_fork+228}
<ffffffff8031c871>{sock_map_fd+336}
<ffffffff8031d7b2>{__sock_create+329}
<ffffffff8010d39a>{system_call+126}
<ffffffff8010d70f>{ptregscall_common+103}
cron D 0000000000000000 0 5644 1 20627 5697 5601
(NOTLB)
ffff81007ba796e8 0000000000000082 0000000100000000 0000000100000001
ffff81007fd2daa0 ffff810077c14710 ffff81000501b400 ffffffff802bf06f
0000000000000000 0000000000000008
Call Trace:<ffffffff802bf06f>{as_merged_request+86}
<ffffffff802b9ba8>{__make_request+1270}
<ffffffff803786c7>{__down_write+129}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8020a319>{xfs_ilock+96} <ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff802ba0ad>{submit_bio+221} <ffffffff80173352>{submit_bh+295}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000} <ffffffff80157148>{do_drain+0}
<ffffffff80157148>{do_drain+0}
<ffffffff8011978b>{flat_send_IPI_allbutself+20}
<ffffffff801176c4>{__smp_call_function+111}
<ffffffff80157148>{do_drain+0}
<ffffffff8012b2d9>{recalc_task_prio+317}
<ffffffff8012b801>{try_to_wake_up+755}
<ffffffff8019beb8>{mb_cache_shrink_fn+100}
<ffffffff80153152>{buffered_rmqueue+524}
<ffffffff8023fa03>{__up_read+33}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff8015375c>{get_zeroed_page+36}
<ffffffff8015c407>{__pud_alloc+55}
<ffffffff8015ccfa>{copy_page_range+370}
<ffffffff80130224>{copy_process+3014} <ffffffff80130966>{do_fork+228}
<ffffffff80139ff8>{del_singleshot_timer_sync+22}
<ffffffff8010d39a>{system_call+126}
<ffffffff8010d70f>{ptregscall_common+103}
xinetd S 0000000000000068 0 5697 1 5712 5644
(NOTLB)
ffff81007ba8fd78 0000000000000086 0000000000000000 0000000000000292
0000000000000292 ffffffff80153152 0000000000000000 0000000000524f70
0000000000000000 0000000000000000
Call Trace:<ffffffff80153152>{buffered_rmqueue+524}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff801832b9>{do_select+1000}
<ffffffff80182de1>{__pollwait+0} <ffffffff80183694>{sys_select+910}
<ffffffff8010d39a>{system_call+126}
agetty S 0000000a6c99d880 0 5712 1 5713 5697
(NOTLB)
ffff81007f251d78 0000000000000086 0000000000000292 0000000000000008
ffff81007f251cc8 000000747c614018 ffffffff805915c0 ffff81007d6d1660
0000000000000074 ffff810002c1a780
Call Trace:<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80131920>{release_console_sem+395}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff80251af8>{read_chan+1149}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8024c173>{tty_read+216} <ffffffff80170aa2>{vfs_read+189}
<ffffffff80170d6b>{sys_read+83} <ffffffff8010d39a>{system_call+126}
agetty S 0000000000000001 0 5713 1 5714 5712
(NOTLB)
ffff81007cb4fd78 0000000000000082 0000000000000292 ffffffff8024bd51
ffff81007cb4fcc8 ffff81007d25d018 0000000300000000 0000000000000292
0000000000000292 ffffffff80131920
Call Trace:<ffffffff8024bd51>{tty_ldisc_deref+47}
<ffffffff80131920>{release_console_sem+395}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80131920>{release_console_sem+395}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff80251af8>{read_chan+1149}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8024c173>{tty_read+216} <ffffffff80170aa2>{vfs_read+189}
<ffffffff80170d6b>{sys_read+83} <ffffffff8010d39a>{system_call+126}
agetty S 0000000a6c938c16 0 5714 1 5715 5713
(NOTLB)
ffff81007c3c9d78 0000000000000086 0000000000000292 0000000000000008
ffff81007c3c9cc8 000000747ba53018 ffffffff805915c0 ffff81007f8b70d0
0000000000000074 ffff810002c1a780
Call Trace:<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80131920>{release_console_sem+395}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff80251af8>{read_chan+1149}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8024c173>{tty_read+216} <ffffffff80170aa2>{vfs_read+189}
<ffffffff80170d6b>{sys_read+83} <ffffffff8010d39a>{system_call+126}
agetty S 0000000000000001 0 5715 1 5716 5714
(NOTLB)
ffff81007bb1fd78 0000000000000086 0000000000000292 ffffffff8024bd51
ffff81007bb1fcc8 ffff81007ba66018 0000000300000000 0000000000000292
0000000000000292 ffffffff80131920
Call Trace:<ffffffff8024bd51>{tty_ldisc_deref+47}
<ffffffff80131920>{release_console_sem+395}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80131920>{release_console_sem+395}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff80251af8>{read_chan+1149}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8024c173>{tty_read+216} <ffffffff80170aa2>{vfs_read+189}
<ffffffff80170d6b>{sys_read+83} <ffffffff8010d39a>{system_call+126}
agetty S 0000000000000001 0 5716 1 5717 5715
(NOTLB)
ffff81007bf67d78 0000000000000082 0000000000000292 ffffffff8024bd51
ffff81007bf67cc8 ffff81007c07a018 0000000300000000 0000000000000292
0000000000000292 ffffffff80131920
Call Trace:<ffffffff8024bd51>{tty_ldisc_deref+47}
<ffffffff80131920>{release_console_sem+395}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80131920>{release_console_sem+395}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff80251af8>{read_chan+1149}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8024c173>{tty_read+216} <ffffffff80170aa2>{vfs_read+189}
<ffffffff80170d6b>{sys_read+83} <ffffffff8010d39a>{system_call+126}
agetty S ffff81007cb80000 0 5717 1 5718 5716
(NOTLB)
ffff81007ca87d78 0000000000000082 0000000000000292 0000000000000008
ffff81007ca87cc8 000000747cb80018 ffffffff805915c0 ffff81007d6d1660
0000000000000074 ffff810002c1a780
Call Trace:<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80131920>{release_console_sem+395}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff80251af8>{read_chan+1149}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8024c173>{tty_read+216} <ffffffff80170aa2>{vfs_read+189}
<ffffffff80170d6b>{sys_read+83} <ffffffff8010d39a>{system_call+126}
agetty D 0000000000000000 0 5718 1 5717
(NOTLB)
ffff81007cae54d8 0000000000000086 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff803786c7>{__down_write+129}
<ffffffff8020a319>{xfs_ilock+96}
<ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff801653c5>{page_referenced_file+209}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000} <ffffffff80157148>{do_drain+0}
<ffffffff80157148>{do_drain+0}
<ffffffff8011978b>{flat_send_IPI_allbutself+20}
<ffffffff8012b2d9>{recalc_task_prio+317}
<ffffffff8012b36c>{activate_task+141}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff80167591>{read_swap_cache_async+125}
<ffffffff8015f0b7>{swapin_readahead+100}
<ffffffff8015f999>{handle_mm_fault+600}
<ffffffff8011d090>{do_page_fault+1185}
<ffffffff80377806>{thread_return+82}
<ffffffff8024bd51>{tty_ldisc_deref+47}
<ffffffff8010dd9d>{error_exit+0}
<ffffffff80241572>{copy_user_generic+178}
<ffffffff80251cfc>{read_chan+1665}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff8024c173>{tty_read+216} <ffffffff80170aa2>{vfs_read+189}
<ffffffff80170d6b>{sys_read+83} <ffffffff8010d39a>{system_call+126}
sshd S ffff8100780afd68 0 20493 5601 20498 20525
(NOTLB)
ffff8100780afbd8 0000000000000082 ffff81007f211580 ffff8100780afb48
0000000000000001 ffffffff8031f48e ffff8100780afe28 ffffffff80323d65
ffff810078abaaa0 ffffffff00000000
Call Trace:<ffffffff8031f48e>{sock_alloc_send_skb+102}
<ffffffff80323d65>{__scm_send+294}
<ffffffff8036e318>{unix_stream_sendmsg+709}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80145e41>{prepare_to_wait+35}
<ffffffff8036e95d>{unix_stream_recvmsg+615}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8036e318>{unix_stream_sendmsg+709}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8031ce24>{sock_aio_read+278}
<ffffffff801709b8>{do_sync_read+173}
<ffffffff80178e0c>{chrdev_open+391}
<ffffffff8016ffac>{dentry_open+284}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff801700d2>{filp_open+63} <ffffffff80170ab5>{vfs_read+208}
<ffffffff80170d6b>{sys_read+83} <ffffffff8010d39a>{system_call+126}
sshd D 0000000000000000 0 20498 20493 20505
(NOTLB)
ffff8100787bd708 0000000000000086 0000000000000000 0000000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff803786c7>{__down_write+129}
<ffffffff8020a319>{xfs_ilock+96}
<ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff8014f10f>{find_get_page+88}
<ffffffff80172d56>{__find_get_block+400}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000}
<ffffffff80156758>{cache_alloc_refill+414}
<ffffffff80151d86>{mempool_alloc+164}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8011a00e>{flush_gart+225}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff80167591>{read_swap_cache_async+125}
<ffffffff8015f0b7>{swapin_readahead+100}
<ffffffff8015f999>{handle_mm_fault+600}
<ffffffff8011d090>{do_page_fault+1185}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8018380c>{sys_select+1286} <ffffffff80170ab5>{vfs_read+208}
<ffffffff8010dd9d>{error_exit+0}
bash S ffff81007c443720 0 20505 20498 20514
(NOTLB)
ffff81007814de58 0000000000000082 ffff810002a4a640 0000000000000296
0000000000000296 ffffffff8015ec10 ffff81007eadc0a8 ffff81007806a010
ffff81007c57f378 800000007819e065
Call Trace:<ffffffff8015ec10>{do_wp_page+989}
<ffffffff8013543e>{do_wait+3483}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8010d39a>{system_call+126}
sshd S ffff81006d477d68 0 20525 5601 20530 20493
(NOTLB)
ffff81006d477bd8 0000000000000086 ffff81006d4e5a80 ffff81006d477b48
0000000000000001 ffffffff8031f48e ffff81006d477e28 ffffffff80323d65
ffff810078abaaa0 ffffffff00000000
Call Trace:<ffffffff8031f48e>{sock_alloc_send_skb+102}
<ffffffff80323d65>{__scm_send+294}
<ffffffff8036e318>{unix_stream_sendmsg+709}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff80145e41>{prepare_to_wait+35}
<ffffffff8036e95d>{unix_stream_recvmsg+615}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8036e318>{unix_stream_sendmsg+709}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8031ce24>{sock_aio_read+278}
<ffffffff801709b8>{do_sync_read+173}
<ffffffff80178e0c>{chrdev_open+391}
<ffffffff8016ffac>{dentry_open+284}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff801700d2>{filp_open+63} <ffffffff80170ab5>{vfs_read+208}
<ffffffff80170d6b>{sys_read+83} <ffffffff8010d39a>{system_call+126}
sshd S 0000000000000328 0 20530 20525 20543
(NOTLB)
ffff81005db1bd78 0000000000000086 0000000100000246 ffffffff8031f705
0000000000000005 0000000000000296 0000000000000296 ffffffff80153152
0000000000000292 ffffffff8034045b
Call Trace:<ffffffff8031f705>{release_sock+25}
<ffffffff80153152>{buffered_rmqueue+524}
<ffffffff8034045b>{tcp_sendmsg+3985}
<ffffffff8024bc2f>{tty_ldisc_try+71}
<ffffffff8037823c>{schedule_timeout+31}
<ffffffff8024bd99>{tty_ldisc_deref+119}
<ffffffff80145d40>{add_wait_queue+28}
<ffffffff801832b9>{do_select+1000}
<ffffffff80182de1>{__pollwait+0}
<ffffffff80136417>{current_fs_time+82}
<ffffffff80183694>{sys_select+910}
<ffffffff8010d39a>{system_call+126}
bash S ffff81007ecc38a0 0 20543 20530 20550
(NOTLB)
ffff81005bd1de58 0000000000000086 ffff810002416980 0000000000000296
0000000000000296 ffff81005bd1dda8 ffffffff8012b2d9 0000000000000000
ffff81005bd1ddd8 ffffffff8012b36c
Call Trace:<ffffffff8012b2d9>{recalc_task_prio+317}
<ffffffff8012b36c>{activate_task+141}
<ffffffff8013543e>{do_wait+3483}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8012d0a5>{default_wake_function+0}
<ffffffff8010d39a>{system_call+126}
iostat D 0000000000000000 0 20550 20543
(NOTLB)
ffff8100316eb5c8 0000000000000082 0000000100000000 0000000100000001
ffff81007fd2daa0 ffff81007a573140 ffff810066c66d00 ffffffff802bf06f
0000000000000000 0000000000000008
Call Trace:<ffffffff802bf06f>{as_merged_request+86}
<ffffffff802b9ba8>{__make_request+1270}
<ffffffff803786c7>{__down_write+129}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8020a319>{xfs_ilock+96} <ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff802ba0ad>{submit_bio+221} <ffffffff80173352>{submit_bh+295}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000} <ffffffff80157148>{do_drain+0}
<ffffffff80157148>{do_drain+0}
<ffffffff8011978b>{flat_send_IPI_allbutself+20}
<ffffffff801176c4>{__smp_call_function+111}
<ffffffff80157148>{do_drain+0}
<ffffffff8012b2d9>{recalc_task_prio+317}
<ffffffff8012b801>{try_to_wake_up+755}
<ffffffff8019beb8>{mb_cache_shrink_fn+100}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff8015f3c5>{do_no_page+652}
<ffffffff8015f8e4>{handle_mm_fault+419}
<ffffffff8011d090>{do_page_fault+1185}
<ffffffff801a0eaf>{proc_lookup+135}
<ffffffff8019e3c2>{proc_root_lookup+64}
<ffffffff8017d81c>{do_lookup+216} <ffffffff8010dd9d>{error_exit+0}
<ffffffff802414ff>{copy_user_generic+63}
<ffffffff8018f8e9>{seq_read+507}
<ffffffff80170aa2>{vfs_read+189} <ffffffff80170d6b>{sys_read+83}
<ffffffff8010d39a>{system_call+126}
pdflush S ffff810049617ef0 0 20564 19 208
(L-TLB)
ffff810049617ec8 0000000000000046 ffff810049617e10 0000000000000002
0000000000000000 000000767ecc31d0 ffffffff805915c0 ffff81007f0e5110
0000000000000076 ffff810002c2a780
Call Trace:<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff801553e6>{pdflush+188}
<ffffffff8015532a>{pdflush+0} <ffffffff801459e1>{kthread+218}
<ffffffff8010df53>{child_rip+8}
<ffffffff80145a1c>{keventd_create_kthread+0}
<ffffffff80145907>{kthread+0} <ffffffff8010df4b>{child_rip+0}
cron D 0000000000000000 0 20627 5644
(NOTLB)
ffff81005e90d738 0000000000000082 0000000005000000 ffff81002f5d41c8
0000000006000000 ffff810030846110 0000000007010003 ffff81006635c560
0000000008010002 0000000000000000
Call Trace:<ffffffff803786c7>{__down_write+129}
<ffffffff8020a319>{xfs_ilock+96}
<ffffffff8020fdfc>{xfs_iomap+264}
<ffffffff8021e462>{xfs_trans_committed+0}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff801653c5>{page_referenced_file+209}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000}
<ffffffff8010dd9d>{error_exit+0}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff80167a86>{swap_info_put+13}
<ffffffff8015ea7f>{do_wp_page+588}
<ffffffff8015fbd7>{handle_mm_fault+1174}
<ffffffff8011d090>{do_page_fault+1185}
<ffffffff8016101e>{remove_vm_struct+128}
<ffffffff801624a1>{do_munmap+906} <ffffffff80378712>{__down_read+51}
<ffffffff8023f8ce>{__up_write+49} <ffffffff8010dd9d>{error_exit+0}
^ permalink raw reply [flat|nested] 5+ messages in thread
* [XFS] Kernel (2.6.11) deadlock in user mode when writing data through mmap on large files (64-bit systems)
2005-05-09 13:18 Kernel (2.6.11) deadlock in user mode when writing data through mmap on large files (64-bit systems, xfs or ext3) Xavier Roche
@ 2005-05-10 15:26 ` Xavier Roche
2005-05-10 17:01 ` Christoph Hellwig
0 siblings, 1 reply; 5+ messages in thread
From: Xavier Roche @ 2005-05-10 15:26 UTC (permalink / raw)
To: linux-kernel
Hi again,
It seems that this bug is _only_ related to XFS - EXT3 tests were
successfull, but the same test run on an XFS filesystem crashed.
ngtest D ffff810078303468 0 20514 20505 (NOTLB)
ffff810078302eb8 0000000000000086 00000001013820a7 0000000000000008
0000000000000086 0000006e7c5cfe88 ffffffff805915c0 ffff81007fe095a0
000000000000006e ffff810002c1a780
Call Trace:<ffffffff80139d89>{__mod_timer+318}
<ffffffff803782c2>{schedule_timeout+165}
<ffffffff8013a8d0>{process_timeout+0}
<ffffffff803781ff>{io_schedule_timeout+49}
<ffffffff802b94cf>{blk_congestion_wait+151}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8022982e>{kmem_alloc+210}
<ffffffff802298cd>{kmem_realloc+43}
<ffffffff8020d181>{xfs_iext_realloc+245}
<ffffffff801e62a8>{xfs_bmap_insert_exlist+53}
<ffffffff801e7194>{xfs_bmap_add_extent_hole_real+949}
<ffffffff801e84df>{xfs_bmap_add_extent+4759}
<ffffffff8021f92e>{xfs_trans_brelse+84}
<ffffffff8021faa8>{xfs_trans_log_buf+107}
<ffffffff801dbb04>{xfs_alloc_ag_vextent+4013}
<ffffffff8022995c>{kmem_zone_alloc+70}
<ffffffff801f215f>{xfs_btree_init_cursor+72}
<ffffffff801eb7b7>{xfs_bmapi+6221}
<ffffffff801e8be8>{xfs_bmap_do_search_extents+778}
<ffffffff80210443>{xfs_iomap_write_direct+646}
<ffffffff803786c7>{__down_write+129}
<ffffffff80145f95>{autoremove_wake_function+0}
<ffffffff8020ff2d>{xfs_iomap+569}
<ffffffff802ba0ad>{submit_bio+221}
<ffffffff80229b83>{xfs_map_blocks+66}
<ffffffff8022a844>{xfs_page_state_convert+996}
<ffffffff8017281c>{__set_page_dirty_buffers+191}
<ffffffff801653c5>{page_referenced_file+209}
<ffffffff80173958>{alloc_buffer_head+50}
<ffffffff80173fa5>{alloc_page_buffers+99}
<ffffffff8022af95>{linvfs_writepage+179}
<ffffffff8015a457>{shrink_zone+3000}
<ffffffff8022ab3d>{__linvfs_get_block+136}
<ffffffff80241fc2>{__memset+50}
<ffffffff80192ed3>{do_mpage_readpage+949}
<ffffffff80157148>{do_drain+0} <ffffffff8012b801>{try_to_wake_up+755}
<ffffffff8023e937>{radix_tree_node_alloc+19}
<ffffffff8023eb29>{radix_tree_insert+291}
<ffffffff8015aa0c>{try_to_free_pages+278}
<ffffffff801534d8>{__alloc_pages+531}
<ffffffff8015575e>{__do_page_cache_readahead+215}
<ffffffff8014fe07>{filemap_nopage+347}
<ffffffff8015f511>{do_no_page+984}
<ffffffff8015f8e4>{handle_mm_fault+419}
<ffffffff80378992>{_spin_unlock_irqrestore+5}
<ffffffff8011d090>{do_page_fault+1185}
<ffffffff8010dd9d>{error_exit+0}
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [XFS] Kernel (2.6.11) deadlock in user mode when writing data through mmap on large files (64-bit systems)
2005-05-10 15:26 ` [XFS] Kernel (2.6.11) deadlock in user mode when writing data through mmap on large files (64-bit systems) Xavier Roche
@ 2005-05-10 17:01 ` Christoph Hellwig
2005-05-12 9:19 ` [XFS] Kernel (2.6.11) deadlock (kernel hang) " Xavier Roche
0 siblings, 1 reply; 5+ messages in thread
From: Christoph Hellwig @ 2005-05-10 17:01 UTC (permalink / raw)
To: Xavier Roche; +Cc: linux-kernel
This is a known problem, or rather two of them:
1) when dirtying lots of memory via mmap writeout happens pretty randmomly
inside the file and the filesystem has problems creating nice clusters.
In the case of extent based filesystems that's really bad as the extent
map is fragmented now
2) XFS keeps each inode's extent map in one single memory block.
You're seeing allocation errors where we are trying to realloc that memory
block.
Could you try the patches that Nikita posted to -mm that should improve
this behaviour?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [XFS] Kernel (2.6.11) deadlock (kernel hang) in user mode when writing data through mmap on large files (64-bit systems)
2005-05-10 17:01 ` Christoph Hellwig
@ 2005-05-12 9:19 ` Xavier Roche
2005-05-16 9:40 ` Christoph Hellwig
0 siblings, 1 reply; 5+ messages in thread
From: Xavier Roche @ 2005-05-12 9:19 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: linux-kernel
Hi,
Christoph Hellwig wrote:
> This is a known problem, or rather two of them:
> 1) when dirtying lots of memory via mmap writeout happens pretty randmomly
> inside the file and the filesystem has problems creating nice clusters.
> In the case of extent based filesystems that's really bad as the extent
> map is fragmented now
> 2) XFS keeps each inode's extent map in one single memory block.
It seems that the file was really *badly* fragmented. The reason, as far
as we understand the problem, was:
- a file "truncated" to _expand_ its size (using ftruncate() with a size
MUCH larger that the current size, which is == 0), leading to create a
"big sparse file" area
- sequential write in this file (_NOT_ random) using the corresponding
mmapp'ed data segment
- random (!) flush from kswapd leading to allocate fragmented pages
(sparse file)
The file appears to have over one million fragments:
$ xfs_bmap ./data | wc -l
1051397
$ xfs_bmap ./data | head -n 20
./data:
0: [0..7]: 15103824..15103831
1: [8..15]: 15103840..15103847
2: [16..23]: 15103856..15103863
3: [24..31]: 15103872..15103879
4: [32..39]: 15103888..15103895
5: [40..47]: 15103904..15103911
6: [48..55]: 15103920..15103927
7: [56..63]: 15103936..15103943
8: [64..71]: 15103952..15103959
9: [72..79]: 15103968..15103975
10: [80..87]: 15104104..15104111
11: [88..95]: 15104120..15104127
12: [96..103]: 15104136..15104143
13: [104..111]: 15104152..15104159
14: [112..119]: 15104168..15104175
15: [120..127]: 15104184..15104191
16: [128..135]: 15104200..15104207
17: [136..143]: 15104216..15104223
18: [144..151]: 15104232..15104239
..
> You're seeing allocation errors where we are trying to realloc that memory
> block.
> Could you try the patches that Nikita posted to -mm that should improve
> this behaviour?
Well, the reasons seems to clearly be this anormal number fo fragments -
is there any potential solution (in the kernel/mm), or the olny solution
is a patch to ensure that ftruncate() is replaced by regulars
fwrite()-zero calls ?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [XFS] Kernel (2.6.11) deadlock (kernel hang) in user mode when writing data through mmap on large files (64-bit systems)
2005-05-12 9:19 ` [XFS] Kernel (2.6.11) deadlock (kernel hang) " Xavier Roche
@ 2005-05-16 9:40 ` Christoph Hellwig
0 siblings, 0 replies; 5+ messages in thread
From: Christoph Hellwig @ 2005-05-16 9:40 UTC (permalink / raw)
To: Xavier Roche; +Cc: linux-kernel
On Thu, May 12, 2005 at 11:19:01AM +0200, Xavier Roche wrote:
> It seems that the file was really *badly* fragmented. The reason, as far
> as we understand the problem, was:
>
> - a file "truncated" to _expand_ its size (using ftruncate() with a size
> MUCH larger that the current size, which is == 0), leading to create a
> "big sparse file" area
> - sequential write in this file (_NOT_ random) using the corresponding
> mmapp'ed data segment
> - random (!) flush from kswapd leading to allocate fragmented pages
> (sparse file)
..
> >You're seeing allocation errors where we are trying to realloc that memory
> >block.
> >Could you try the patches that Nikita posted to -mm that should improve
> >this behaviour?
>
> Well, the reasons seems to clearly be this anormal number fo fragments -
> is there any potential solution (in the kernel/mm), or the olny solution
> is a patch to ensure that ftruncate() is replaced by regulars
> fwrite()-zero calls ?
>
Yes. Currently the kernel is doing very badly about clustering these
kinds of allocations. Can you test whether the patches at:
http://marc.theaimsgroup.com/?l=linux-mm&m=111375946911468&w=2
http://marc.theaimsgroup.com/?l=linux-mm&m=111375947014227&w=2
http://marc.theaimsgroup.com/?l=linux-mm&m=111375947006819&w=2
http://marc.theaimsgroup.com/?l=linux-mm&m=111375947029723&w=2
http://marc.theaimsgroup.com/?l=linux-mm&m=111375947010814&w=2
http://marc.theaimsgroup.com/?l=linux-mm&m=111375990000352&w=2
http://marc.theaimsgroup.com/?l=linux-mm&m=111375990030384&w=2
http://marc.theaimsgroup.com/?l=linux-mm&m=111375990032680&w=2
http://marc.theaimsgroup.com/?l=linux-mm&m=111375990019453&w=2
make any difference to you?
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2005-05-16 9:41 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-05-09 13:18 Kernel (2.6.11) deadlock in user mode when writing data through mmap on large files (64-bit systems, xfs or ext3) Xavier Roche
2005-05-10 15:26 ` [XFS] Kernel (2.6.11) deadlock in user mode when writing data through mmap on large files (64-bit systems) Xavier Roche
2005-05-10 17:01 ` Christoph Hellwig
2005-05-12 9:19 ` [XFS] Kernel (2.6.11) deadlock (kernel hang) " Xavier Roche
2005-05-16 9:40 ` Christoph Hellwig
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).