* File system remain unresponsive until the system is rebooted.
@ 2012-01-30 9:18 Supratik Goswami
2012-01-30 17:23 ` Peter Grandi
2012-01-31 1:31 ` Dave Chinner
0 siblings, 2 replies; 18+ messages in thread
From: Supratik Goswami @ 2012-01-30 9:18 UTC (permalink / raw)
To: xfs
Hi
We are using RAID-0 volumes as PV's in our LVM stack and XFS as the filesystem.
The kernel logged the below call trace when the filesystem was being
expanded using "xfs_growfs" command.
We have used xfs_grofs at least 3 times earlier but did not cam across
this situation.
The files system remained unresponsive until we rebooted the system
and again increased the size of the filesystem.
This time it worked fine. Can you please tell us why xfs_grofs hanged suddenly ?
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550724] flush-254:0 D
00000001016ff564 0 31679 2 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550728]
ffff8801de5df388 0000000000000246 ffff88002a3f57d0 ffff8801de5df308
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550732]
ffff8801deb59a10 ffff8801de5df350 ffff8801def9ec70 ffff8801de5dffd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550734]
ffff8801def9e8c0 ffff8801def9e8c0 ffff8801def9e8c0 ffff8801de5dffd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550737] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550748]
[<ffffffff812604c7>] ? xfs_btree_insert+0x67/0x180
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550754]
[<ffffffff814b172d>] rwsem_down_failed_common+0xbd/0x240
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550757]
[<ffffffff814b1906>] rwsem_down_read_failed+0x26/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550761]
[<ffffffff813424b4>] call_rwsem_down_read_failed+0x14/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550764]
[<ffffffff814b0ab2>] ? down_read+0x12/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550770]
[<ffffffff81257f20>] xfs_bmap_btalloc+0x300/0xa90
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550774]
[<ffffffff812513c5>] ? xfs_bmap_search_multi_extents+0xa5/0x110
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550784]
[<ffffffff81251609>] ? xfs_bmap_search_extents+0x69/0xf0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550787]
[<ffffffff812586cc>] xfs_bmap_alloc+0x1c/0x40
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550790]
[<ffffffff8125928f>] xfs_bmapi+0xb9f/0x1290
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550796]
[<ffffffff8127a205>] xfs_iomap_write_allocate+0x1c5/0x3c0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550799]
[<ffffffff8127af46>] xfs_iomap+0x2a6/0x2e0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550804]
[<ffffffff81293998>] xfs_map_blocks+0x28/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550807]
[<ffffffff81294d7a>] xfs_page_state_convert+0x3da/0x720
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550809]
[<ffffffff81295211>] xfs_vm_writepage+0x71/0x120
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550815]
[<ffffffff810b5b62>] __writepage+0x12/0x40
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550817]
[<ffffffff810b6917>] write_cache_pages+0x1d7/0x3d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550820]
[<ffffffff810b5b50>] ? __writepage+0x0/0x40
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550823]
[<ffffffff810b6b2f>] generic_writepages+0x1f/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550826]
[<ffffffff81294020>] xfs_vm_writepages+0x70/0x90
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550829]
[<ffffffff810b6b5c>] do_writepages+0x1c/0x40
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550834]
[<ffffffff81111693>] writeback_single_inode+0x103/0x3f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550837]
[<ffffffff81111c64>] writeback_sb_inodes+0x184/0x2a0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550840]
[<ffffffff81111deb>] writeback_inodes_wb+0x6b/0x1f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550843]
[<ffffffff811121db>] wb_writeback+0x26b/0x2d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550847]
[<ffffffff8104c73a>] ? del_timer_sync+0x1a/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550850]
[<ffffffff811123bf>] wb_do_writeback+0x17f/0x190
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550853]
[<ffffffff8111241b>] bdi_writeback_task+0x4b/0xe0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550858]
[<ffffffff810c72f0>] ? bdi_start_fn+0x0/0x110
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550861]
[<ffffffff810c7371>] bdi_start_fn+0x81/0x110
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550863]
[<ffffffff810c72f0>] ? bdi_start_fn+0x0/0x110
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550868]
[<ffffffff81059e2e>] kthread+0x8e/0xa0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550873]
[<ffffffff8100a71a>] child_rip+0xa/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550876]
[<ffffffff81059da0>] ? kthread+0x0/0xa0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550878]
[<ffffffff8100a710>] ? child_rip+0x0/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550897] cm-httpserver D
00000001016fed9a 0 16726 1 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550900]
ffff8801dd869d28 0000000000000286 ffff8801dca1a8c0 ffff8801dd869ca8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550903]
ffff8801b1583032 ffff8801dd869cf0 ffff8801dd7f6870 ffff8801dd869fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550905]
ffff8801dd7f64c0 ffff8801dd7f64c0 ffff8801dd7f64c0 ffff8801dd869fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550908] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550916]
[<ffffffff812de22a>] ? security_inode_permission+0x1a/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550920]
[<ffffffff810fa91d>] ? __link_path_walk+0xed/0xf90
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550922]
[<ffffffff814b026c>] __mutex_lock_slowpath+0x11c/0x1b0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550928]
[<ffffffff8130e314>] ? apparmor_file_alloc_security+0x24/0x90
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550930]
[<ffffffff814b005e>] mutex_lock+0x1e/0x40
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550933]
[<ffffffff810fce3d>] do_filp_open+0x3cd/0xba0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550936]
[<ffffffff810f8fdc>] ? path_put+0x2c/0x40
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550939]
[<ffffffff811089bb>] ? alloc_fd+0x4b/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550944]
[<ffffffff810ec404>] do_sys_open+0x64/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550947]
[<ffffffff810ec52b>] sys_open+0x1b/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550950]
[<ffffffff81009bb8>] system_call_fastpath+0x16/0x1b
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550953]
[<ffffffff81009b50>] ? system_call+0x0/0x52
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550977] cm-gdoc D
0000000000000002 0 16809 1 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550980]
ffff8801dcd65810 0000000000000286 000000008102a929 ffff8801dcd65790
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550983]
ffff880002c67bd0 ffff8801dcd657d8 ffff88012679e8f0 ffff8801dcd65fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550985]
ffff88012679e540 ffff88012679e540 ffff88012679e540 ffff8801dcd65fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550990] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550995]
[<ffffffff81038c0f>] ? find_busiest_group+0x5f/0x440
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550997]
[<ffffffff814b172d>] rwsem_down_failed_common+0xbd/0x240
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551003]
[<ffffffff813a37ee>] ? notify_remote_via_irq+0x5e/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551006]
[<ffffffff814b1906>] rwsem_down_read_failed+0x26/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551008]
[<ffffffff813424b4>] call_rwsem_down_read_failed+0x14/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551011]
[<ffffffff814b0ab2>] ? down_read+0x12/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551013]
[<ffffffff81271882>] xfs_ialloc_ag_select+0x92/0x370
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551017]
[<ffffffff81421a95>] ? sch_direct_xmit+0x95/0x200
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551020]
[<ffffffff81272465>] xfs_dialloc+0x415/0x940
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551026]
[<ffffffff8102cddd>] ? update_sd_lb_stats+0x1fd/0x5f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551029]
[<ffffffff81275ead>] xfs_ialloc+0x5d/0x700
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551032]
[<ffffffff812806ec>] ? xlog_grant_log_space+0x3fc/0x450
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551035]
[<ffffffff8128dded>] xfs_dir_ialloc+0x7d/0x2d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551038]
[<ffffffff81280814>] ? xfs_log_reserve+0xd4/0xe0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551041]
[<ffffffff8128f9d3>] xfs_create+0x3e3/0x5f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551044]
[<ffffffff81103361>] ? __d_lookup+0xb1/0x180
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551047]
[<ffffffff8129afd2>] xfs_vn_mknod+0xa2/0x1b0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551050]
[<ffffffff8129b0fb>] xfs_vn_create+0xb/0x10
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551052]
[<ffffffff810f966f>] vfs_create+0xaf/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551054]
[<ffffffff810fa20c>] __open_namei_create+0xbc/0x100
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551057]
[<ffffffff810fd4d6>] do_filp_open+0xa66/0xba0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551059]
[<ffffffff811089bb>] ? alloc_fd+0x4b/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551062]
[<ffffffff810ec404>] do_sys_open+0x64/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551065]
[<ffffffff810ec52b>] sys_open+0x1b/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551067]
[<ffffffff81009bb8>] system_call_fastpath+0x16/0x1b
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551070]
[<ffffffff81009b50>] ? system_call+0x0/0x52
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551078] cm-gdoc D
0000000000000001 0 16816 1 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551081]
ffff8801dea61a58 0000000000000286 ffff8800087f1078 ffff8801dea619d8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551083]
ffff880004a73000 ffff8801dea61a20 ffff8801dcc3ca70 ffff8801dea61fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551086]
ffff8801dcc3c6c0 ffff8801dcc3c6c0 ffff8801dcc3c6c0 ffff8801dea61fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551088] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551094]
[<ffffffff8127efc0>] ? xlog_state_get_iclog_space+0x60/0x370
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551097]
[<ffffffff814afcb5>] schedule_timeout+0x1e5/0x2c0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551099]
[<ffffffff8127efc0>] ? xlog_state_get_iclog_space+0x60/0x370
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551102]
[<ffffffff8127f8a8>] ? xlog_write+0x5d8/0x690
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551104]
[<ffffffff814b0cc8>] __down+0x78/0xf0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551109]
[<ffffffff8105eb6c>] down+0x3c/0x50
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551112]
[<ffffffff81296e0e>] xfs_buf_lock+0x1e/0x60
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551114]
[<ffffffff81297699>] _xfs_buf_find+0x149/0x270
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551117]
[<ffffffff8129781b>] xfs_buf_get_flags+0x5b/0x170
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551119]
[<ffffffff81297943>] xfs_buf_read_flags+0x13/0xa0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551122]
[<ffffffff8128cad9>] xfs_trans_read_buf+0x1c9/0x300
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551126]
[<ffffffff8126d3c1>] ? xfs_dir2_sf_removename+0x121/0x190
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551128]
[<ffffffff812708ef>] xfs_read_agi+0x6f/0x100
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551131]
[<ffffffff81276736>] xfs_iunlink+0x46/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551134]
[<ffffffff81042582>] ? current_fs_time+0x22/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551137]
[<ffffffff811113cd>] ? __mark_inode_dirty+0x3d/0x200
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551139]
[<ffffffff8129b90d>] ? xfs_ichgtime+0x1d/0xc0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551142]
[<ffffffff8128dd19>] xfs_droplink+0x59/0x70
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551144]
[<ffffffff8128f364>] xfs_remove+0x294/0x350
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551148]
[<ffffffff810f821e>] ? generic_permission+0x1e/0xc0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551150]
[<ffffffff8129b213>] xfs_vn_unlink+0x43/0x90
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551152]
[<ffffffff810f98c0>] vfs_unlink+0xa0/0xf0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551154]
[<ffffffff810fa755>] ? lookup_hash+0x35/0x50
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551156]
[<ffffffff810fc16e>] do_unlinkat+0x19e/0x1d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551159]
[<ffffffff810efe6d>] ? fput+0x1d/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551162]
[<ffffffff810ec218>] ? filp_close+0x58/0x90
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551164]
[<ffffffff810fc1b1>] sys_unlink+0x11/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551167]
[<ffffffff81009bb8>] system_call_fastpath+0x16/0x1b
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551169]
[<ffffffff81009b50>] ? system_call+0x0/0x52
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551177] cm-gdoc D
00000001016fea9d 0 16817 1 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551180]
ffff8801ddbad810 0000000000000286 0000000000000000 ffff8801ddbad790
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551183]
ffff880002c67bd0 ffff8801ddbad7d8 ffff8801de886ab0 ffff8801ddbadfd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551185]
ffff8801de886700 ffff8801de886700 ffff8801de886700 ffff8801ddbadfd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551190] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551192]
[<ffffffff81038c0f>] ? find_busiest_group+0x5f/0x440
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551195]
[<ffffffff814b172d>] rwsem_down_failed_common+0xbd/0x240
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551198]
[<ffffffff813a37ee>] ? notify_remote_via_irq+0x5e/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551200]
[<ffffffff814b1906>] rwsem_down_read_failed+0x26/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551203]
[<ffffffff813424b4>] call_rwsem_down_read_failed+0x14/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551205]
[<ffffffff814b0ab2>] ? down_read+0x12/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551208]
[<ffffffff81271882>] xfs_ialloc_ag_select+0x92/0x370
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551210]
[<ffffffff81421a95>] ? sch_direct_xmit+0x95/0x200
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551212]
[<ffffffff81272465>] xfs_dialloc+0x415/0x940
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551217]
[<ffffffff8143ccc4>] ? ip_finish_output+0x134/0x310
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551220]
[<ffffffff8143cf58>] ? ip_output+0xb8/0xc0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551222]
[<ffffffff814b08ad>] ? schedule_hrtimeout_range+0xcd/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551225]
[<ffffffff8143bf10>] ? ip_local_out+0x20/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551227]
[<ffffffff81275ead>] xfs_ialloc+0x5d/0x700
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551230]
[<ffffffff812806ec>] ? xlog_grant_log_space+0x3fc/0x450
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551233]
[<ffffffff8128dded>] xfs_dir_ialloc+0x7d/0x2d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551236]
[<ffffffff81280814>] ? xfs_log_reserve+0xd4/0xe0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551239]
[<ffffffff8128f9d3>] xfs_create+0x3e3/0x5f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551241]
[<ffffffff81103361>] ? __d_lookup+0xb1/0x180
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551244]
[<ffffffff8129afd2>] xfs_vn_mknod+0xa2/0x1b0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551246]
[<ffffffff8129b0fb>] xfs_vn_create+0xb/0x10
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551249]
[<ffffffff810f966f>] vfs_create+0xaf/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551251]
[<ffffffff810fa20c>] __open_namei_create+0xbc/0x100
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551253]
[<ffffffff810fd4d6>] do_filp_open+0xa66/0xba0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551256]
[<ffffffff811089bb>] ? alloc_fd+0x4b/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551259]
[<ffffffff810ec404>] do_sys_open+0x64/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551261]
[<ffffffff810ec52b>] sys_open+0x1b/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551264]
[<ffffffff81009bb8>] system_call_fastpath+0x16/0x1b
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551267]
[<ffffffff81009b50>] ? system_call+0x0/0x52
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551274] cm-gdoc D
00000001016fea61 0 16818 1 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551277]
ffff8801dd593810 0000000000000286 0000000000000000 ffff8801dd593790
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551280]
ffff880002c67bd0 ffff8801dd5937d8 ffff8801ded64af0 ffff8801dd593fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551284]
ffff8801ded64740 ffff8801ded64740 ffff8801ded64740 ffff8801dd593fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551287] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551290]
[<ffffffff8100a505>] ? hypervisor_callback+0x25/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551293]
[<ffffffff814b172d>] rwsem_down_failed_common+0xbd/0x240
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551295]
[<ffffffff814b1906>] rwsem_down_read_failed+0x26/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551298]
[<ffffffff813424b4>] call_rwsem_down_read_failed+0x14/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551300]
[<ffffffff814b0ab2>] ? down_read+0x12/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551302]
[<ffffffff81271882>] xfs_ialloc_ag_select+0x92/0x370
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551305]
[<ffffffff81421a95>] ? sch_direct_xmit+0x95/0x200
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551307]
[<ffffffff81272465>] xfs_dialloc+0x415/0x940
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551310]
[<ffffffff8102cddd>] ? update_sd_lb_stats+0x1fd/0x5f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551312]
[<ffffffff81275ead>] xfs_ialloc+0x5d/0x700
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551315]
[<ffffffff812806ec>] ? xlog_grant_log_space+0x3fc/0x450
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551318]
[<ffffffff8128dded>] xfs_dir_ialloc+0x7d/0x2d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551321]
[<ffffffff81280814>] ? xfs_log_reserve+0xd4/0xe0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551324]
[<ffffffff8128f9d3>] xfs_create+0x3e3/0x5f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551326]
[<ffffffff81103361>] ? __d_lookup+0xb1/0x180
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551329]
[<ffffffff8129afd2>] xfs_vn_mknod+0xa2/0x1b0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551331]
[<ffffffff8129b0fb>] xfs_vn_create+0xb/0x10
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551333]
[<ffffffff810f966f>] vfs_create+0xaf/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551336]
[<ffffffff810fa20c>] __open_namei_create+0xbc/0x100
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551338]
[<ffffffff810fd4d6>] do_filp_open+0xa66/0xba0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551340]
[<ffffffff811089bb>] ? alloc_fd+0x4b/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551343]
[<ffffffff810ec404>] do_sys_open+0x64/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551346]
[<ffffffff810ec52b>] sys_open+0x1b/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551348]
[<ffffffff81009bb8>] system_call_fastpath+0x16/0x1b
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551351]
[<ffffffff81009b50>] ? system_call+0x0/0x52
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551359] cm-gdoc D
00000001016fea55 0 16831 1 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551362]
ffff8801dde39810 0000000000000286 0000000000000000 ffff8801dde39790
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551364]
ffff880002c67bd0 ffff8801dde397d8 ffff8801de888cb0 ffff8801dde39fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551367]
ffff8801de888900 ffff8801de888900 ffff8801de888900 ffff8801dde39fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551369] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551374]
[<ffffffff81038c0f>] ? find_busiest_group+0x5f/0x440
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551377]
[<ffffffff814b172d>] rwsem_down_failed_common+0xbd/0x240
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551379]
[<ffffffff813a37ee>] ? notify_remote_via_irq+0x5e/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551382]
[<ffffffff814b1906>] rwsem_down_read_failed+0x26/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551384]
[<ffffffff813424b4>] call_rwsem_down_read_failed+0x14/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551387]
[<ffffffff814b0ab2>] ? down_read+0x12/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551389]
[<ffffffff81271882>] xfs_ialloc_ag_select+0x92/0x370
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551391]
[<ffffffff81421a95>] ? sch_direct_xmit+0x95/0x200
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551394]
[<ffffffff81272465>] xfs_dialloc+0x415/0x940
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551396]
[<ffffffff8143ccc4>] ? ip_finish_output+0x134/0x310
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551399]
[<ffffffff8143cf58>] ? ip_output+0xb8/0xc0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551401]
[<ffffffff814b08ad>] ? schedule_hrtimeout_range+0xcd/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551404]
[<ffffffff8143bf10>] ? ip_local_out+0x20/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551406]
[<ffffffff81275ead>] xfs_ialloc+0x5d/0x700
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551409]
[<ffffffff812806ec>] ? xlog_grant_log_space+0x3fc/0x450
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551412]
[<ffffffff8128dded>] xfs_dir_ialloc+0x7d/0x2d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551415]
[<ffffffff81280814>] ? xfs_log_reserve+0xd4/0xe0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551417]
[<ffffffff8128f9d3>] xfs_create+0x3e3/0x5f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551420]
[<ffffffff81103361>] ? __d_lookup+0xb1/0x180
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551423]
[<ffffffff8129afd2>] xfs_vn_mknod+0xa2/0x1b0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551426]
[<ffffffff8129b0fb>] xfs_vn_create+0xb/0x10
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551428]
[<ffffffff810f966f>] vfs_create+0xaf/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551430]
[<ffffffff810fa20c>] __open_namei_create+0xbc/0x100
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551433]
[<ffffffff810fd4d6>] do_filp_open+0xa66/0xba0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551435]
[<ffffffff811089bb>] ? alloc_fd+0x4b/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551438]
[<ffffffff810ec404>] do_sys_open+0x64/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551441]
[<ffffffff810ec52b>] sys_open+0x1b/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551443]
[<ffffffff81009bb8>] system_call_fastpath+0x16/0x1b
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551446]
[<ffffffff81009b50>] ? system_call+0x0/0x52
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551453] cm-gdoc D
0000000000000001 0 16837 1 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551456]
ffff8801dd5a7810 0000000000000286 0000000000000000 ffff8801dd5a7790
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551459]
ffff880004a73000 ffff8801dd5a77d8 ffff8801dd59e4f0 ffff8801dd5a7fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551461]
ffff8801dd59e140 ffff8801dd59e140 ffff8801dd59e140 ffff8801dd5a7fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551464] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551469]
[<ffffffff81038c0f>] ? find_busiest_group+0x5f/0x440
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551472]
[<ffffffff814b172d>] rwsem_down_failed_common+0xbd/0x240
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551474]
[<ffffffff813a37ee>] ? notify_remote_via_irq+0x5e/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551477]
[<ffffffff814b1906>] rwsem_down_read_failed+0x26/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551479]
[<ffffffff813424b4>] call_rwsem_down_read_failed+0x14/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551482]
[<ffffffff814b0ab2>] ? down_read+0x12/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551484]
[<ffffffff81271882>] xfs_ialloc_ag_select+0x92/0x370
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551486]
[<ffffffff81421a95>] ? sch_direct_xmit+0x95/0x200
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551489]
[<ffffffff81272465>] xfs_dialloc+0x415/0x940
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551491]
[<ffffffff8143ccc4>] ? ip_finish_output+0x134/0x310
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551494]
[<ffffffff8143cf58>] ? ip_output+0xb8/0xc0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551496]
[<ffffffff814b08ad>] ? schedule_hrtimeout_range+0xcd/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551499]
[<ffffffff8143bf10>] ? ip_local_out+0x20/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551501]
[<ffffffff81275ead>] xfs_ialloc+0x5d/0x700
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551504]
[<ffffffff812806ec>] ? xlog_grant_log_space+0x3fc/0x450
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551507]
[<ffffffff8128dded>] xfs_dir_ialloc+0x7d/0x2d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551509]
[<ffffffff81280814>] ? xfs_log_reserve+0xd4/0xe0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551512]
[<ffffffff8128f9d3>] xfs_create+0x3e3/0x5f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551515]
[<ffffffff81103361>] ? __d_lookup+0xb1/0x180
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551518]
[<ffffffff8129afd2>] xfs_vn_mknod+0xa2/0x1b0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551520]
[<ffffffff8129b0fb>] xfs_vn_create+0xb/0x10
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551522]
[<ffffffff810f966f>] vfs_create+0xaf/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551525]
[<ffffffff810fa20c>] __open_namei_create+0xbc/0x100
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551527]
[<ffffffff810fd4d6>] do_filp_open+0xa66/0xba0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551530]
[<ffffffff811089bb>] ? alloc_fd+0x4b/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551533]
[<ffffffff810ec404>] do_sys_open+0x64/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551535]
[<ffffffff810ec52b>] sys_open+0x1b/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551538]
[<ffffffff81009bb8>] system_call_fastpath+0x16/0x1b
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551541]
[<ffffffff81009b50>] ? system_call+0x0/0x52
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551548] cm-gdoc D
0000000000000001 0 16838 1 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551551]
ffff8801b15e9810 0000000000000286 0000000000000000 ffff8801b15e9790
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551554]
ffff880004a73000 ffff8801b15e97d8 ffff8801deb9e530 ffff8801b15e9fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551556]
ffff8801deb9e180 ffff8801deb9e180 ffff8801deb9e180 ffff8801b15e9fd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551558] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551563]
[<ffffffff81038c0f>] ? find_busiest_group+0x5f/0x440
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551566]
[<ffffffff814b172d>] rwsem_down_failed_common+0xbd/0x240
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551569]
[<ffffffff813a37ee>] ? notify_remote_via_irq+0x5e/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551571]
[<ffffffff814b1906>] rwsem_down_read_failed+0x26/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551574]
[<ffffffff813424b4>] call_rwsem_down_read_failed+0x14/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551576]
[<ffffffff814b0ab2>] ? down_read+0x12/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551578]
[<ffffffff81271882>] xfs_ialloc_ag_select+0x92/0x370
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551581]
[<ffffffff81421a95>] ? sch_direct_xmit+0x95/0x200
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551583]
[<ffffffff81272465>] xfs_dialloc+0x415/0x940
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551586]
[<ffffffff8143ccc4>] ? ip_finish_output+0x134/0x310
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551588]
[<ffffffff8143cf58>] ? ip_output+0xb8/0xc0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551591]
[<ffffffff814b08ad>] ? schedule_hrtimeout_range+0xcd/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551593]
[<ffffffff8143bf10>] ? ip_local_out+0x20/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551596]
[<ffffffff81275ead>] xfs_ialloc+0x5d/0x700
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551598]
[<ffffffff812806ec>] ? xlog_grant_log_space+0x3fc/0x450
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551601]
[<ffffffff8128dded>] xfs_dir_ialloc+0x7d/0x2d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551604]
[<ffffffff81280814>] ? xfs_log_reserve+0xd4/0xe0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551606]
[<ffffffff8128f9d3>] xfs_create+0x3e3/0x5f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551609]
[<ffffffff81103361>] ? __d_lookup+0xb1/0x180
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551612]
[<ffffffff8129afd2>] xfs_vn_mknod+0xa2/0x1b0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551614]
[<ffffffff8129b0fb>] xfs_vn_create+0xb/0x10
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551616]
[<ffffffff810f966f>] vfs_create+0xaf/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551618]
[<ffffffff810fa20c>] __open_namei_create+0xbc/0x100
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551621]
[<ffffffff810fd4d6>] do_filp_open+0xa66/0xba0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551623]
[<ffffffff811089bb>] ? alloc_fd+0x4b/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551626]
[<ffffffff810ec404>] do_sys_open+0x64/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551629]
[<ffffffff810ec52b>] sys_open+0x1b/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551631]
[<ffffffff81009bb8>] system_call_fastpath+0x16/0x1b
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551634]
[<ffffffff81009b50>] ? system_call+0x0/0x52
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551641] cm-gdoc D
0000000000000002 0 16859 1 0x00000000
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551644]
ffff88012646b810 0000000000000286 0000000000000000 ffff88012646b790
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551647]
ffff880002c67bd0 ffff88012646b7d8 ffff8801b1690a70 ffff88012646bfd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551649]
ffff8801b16906c0 ffff8801b16906c0 ffff8801b16906c0 ffff88012646bfd8
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551652] Call Trace:
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551656]
[<ffffffff81038c0f>] ? find_busiest_group+0x5f/0x440
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551659]
[<ffffffff814b172d>] rwsem_down_failed_common+0xbd/0x240
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551662]
[<ffffffff813a37ee>] ? notify_remote_via_irq+0x5e/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551664]
[<ffffffff814b1906>] rwsem_down_read_failed+0x26/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551667]
[<ffffffff813424b4>] call_rwsem_down_read_failed+0x14/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551669]
[<ffffffff814b0ab2>] ? down_read+0x12/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551671]
[<ffffffff81271882>] xfs_ialloc_ag_select+0x92/0x370
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551674]
[<ffffffff81421a95>] ? sch_direct_xmit+0x95/0x200
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551676]
[<ffffffff81272465>] xfs_dialloc+0x415/0x940
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551679]
[<ffffffff8143ccc4>] ? ip_finish_output+0x134/0x310
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551682]
[<ffffffff8143cf58>] ? ip_output+0xb8/0xc0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551684]
[<ffffffff814b08ad>] ? schedule_hrtimeout_range+0xcd/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551687]
[<ffffffff8143bf10>] ? ip_local_out+0x20/0x30
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551689]
[<ffffffff81275ead>] xfs_ialloc+0x5d/0x700
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551692]
[<ffffffff812806ec>] ? xlog_grant_log_space+0x3fc/0x450
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551695]
[<ffffffff8128dded>] xfs_dir_ialloc+0x7d/0x2d0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551697]
[<ffffffff81280814>] ? xfs_log_reserve+0xd4/0xe0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551700]
[<ffffffff8128f9d3>] xfs_create+0x3e3/0x5f0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551703]
[<ffffffff81103361>] ? __d_lookup+0xb1/0x180
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551705]
[<ffffffff8129afd2>] xfs_vn_mknod+0xa2/0x1b0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551708]
[<ffffffff8129b0fb>] xfs_vn_create+0xb/0x10
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551710]
[<ffffffff810f966f>] vfs_create+0xaf/0xd0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551712]
[<ffffffff810fa20c>] __open_namei_create+0xbc/0x100
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551715]
[<ffffffff810fd4d6>] do_filp_open+0xa66/0xba0
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551717]
[<ffffffff811089bb>] ? alloc_fd+0x4b/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551720]
[<ffffffff810ec404>] do_sys_open+0x64/0x160
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551723]
[<ffffffff810ec52b>] sys_open+0x1b/0x20
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551726]
[<ffffffff81009bb8>] system_call_fastpath+0x16/0x1b
Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.551728]
[<ffffffff81009b50>] ? system_call+0x0/0x52
--
Warm Regards
Supratik
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-30 9:18 File system remain unresponsive until the system is rebooted Supratik Goswami
@ 2012-01-30 17:23 ` Peter Grandi
2012-01-31 1:31 ` Dave Chinner
1 sibling, 0 replies; 18+ messages in thread
From: Peter Grandi @ 2012-01-30 17:23 UTC (permalink / raw)
To: Linux fs XFS
> We are using RAID-0 volumes as PV's in our LVM stack and XFS
> as the filesystem.
LVM is in general a bad idea, and I have found that it
occasionally interacts not so well with XFS and other
filesystems under resource pressure.
It also seems from one of the backtraces that you are
complicating all this further by running under Xen,
perhaps on sparsely allocated virtual disks.
> [ ... ] The files system remained unresponsive until we
> rebooted the system and again increased the size of the
> filesystem. [ ... ]
Good luck. I know some people who also went the whole VM/LVM/XFS
stack way and had lots of problems. It is what I call the
"syntactic" approach: expecting that every syntactically valid
combination of features is going to work, and work well. Sure it
should :-).
Most of the hangs seem to happen during resource allocation,
and at least one is triggered by the flusher:
> Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550853] [<ffffffff8111241b>] bdi_writeback_task+0x4b/0xe0
> Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550858] [<ffffffff810c72f0>] ? bdi_start_fn+0x0/0x110
> Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550861] [<ffffffff810c7371>] bdi_start_fn+0x81/0x110
> Jan 26 03:05:47 ip-10-0-1-153 kernel: [241565.550863] [<ffffffff810c72f0>] ? bdi_start_fn+0x0/0x110
It could be that there is intense pressure on kernel memory,
often due to excessively loose flusher parameters.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-30 9:18 File system remain unresponsive until the system is rebooted Supratik Goswami
2012-01-30 17:23 ` Peter Grandi
@ 2012-01-31 1:31 ` Dave Chinner
2012-01-31 5:04 ` Supratik Goswami
1 sibling, 1 reply; 18+ messages in thread
From: Dave Chinner @ 2012-01-31 1:31 UTC (permalink / raw)
To: Supratik Goswami; +Cc: xfs
On Mon, Jan 30, 2012 at 02:48:58PM +0530, Supratik Goswami wrote:
> Hi
>
> We are using RAID-0 volumes as PV's in our LVM stack and XFS as the filesystem.
>
> The kernel logged the below call trace when the filesystem was being
> expanded using "xfs_growfs" command.
> We have used xfs_grofs at least 3 times earlier but did not cam across
> this situation.
>
> The files system remained unresponsive until we rebooted the system
> and again increased the size of the filesystem.
> This time it worked fine. Can you please tell us why xfs_grofs hanged suddenly ?
What kernel?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-31 1:31 ` Dave Chinner
@ 2012-01-31 5:04 ` Supratik Goswami
2012-01-31 7:19 ` Emmanuel Florac
` (2 more replies)
0 siblings, 3 replies; 18+ messages in thread
From: Supratik Goswami @ 2012-01-31 5:04 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
We are using Amazon EC2 instances.
ubuntu@ip-10-0-0-10:~$ uname -aLinux ip-10-0-0-10 2.6.32-318-ec2
#38-Ubuntu SMP Thu Sep 1 18:09:30 UTC 2011 x86_64 GNU/Linux
On Tue, Jan 31, 2012 at 7:01 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Mon, Jan 30, 2012 at 02:48:58PM +0530, Supratik Goswami wrote:
>> Hi
>>
>> We are using RAID-0 volumes as PV's in our LVM stack and XFS as the filesystem.
>>
>> The kernel logged the below call trace when the filesystem was being
>> expanded using "xfs_growfs" command.
>> We have used xfs_grofs at least 3 times earlier but did not cam across
>> this situation.
>>
>> The files system remained unresponsive until we rebooted the system
>> and again increased the size of the filesystem.
>> This time it worked fine. Can you please tell us why xfs_grofs hanged suddenly ?
>
> What kernel?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
--
Warm Regards
Supratik
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-31 5:04 ` Supratik Goswami
@ 2012-01-31 7:19 ` Emmanuel Florac
2012-01-31 9:04 ` Stan Hoeppner
2012-01-31 19:44 ` Dave Chinner
2 siblings, 0 replies; 18+ messages in thread
From: Emmanuel Florac @ 2012-01-31 7:19 UTC (permalink / raw)
To: Supratik Goswami; +Cc: xfs
Le Tue, 31 Jan 2012 10:34:10 +0530 vous écriviez:
> We are using Amazon EC2 instances.
>
You can't know for sure what's happening behind the scenes. The most
common problem of EC2 instances is IO starving, so this is hardly
surprising.
--
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <eflorac@intellique.com>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-31 5:04 ` Supratik Goswami
2012-01-31 7:19 ` Emmanuel Florac
@ 2012-01-31 9:04 ` Stan Hoeppner
2012-01-31 11:08 ` Emmanuel Florac
2012-01-31 20:50 ` Dave Chinner
2012-01-31 19:44 ` Dave Chinner
2 siblings, 2 replies; 18+ messages in thread
From: Stan Hoeppner @ 2012-01-31 9:04 UTC (permalink / raw)
To: xfs
On 1/30/2012 11:04 PM, Supratik Goswami wrote:
> We are using Amazon EC2 instances.
^^^^^^^^^^
I'd have never thought I would see those words on this list, except
maybe as a joke, or as an example of one of the the worst possible
platforms for XFS.
I wish EC2 had been asked about during the QA session after Dave's
presentation. I'm guessing some laughter would have been involved. ;)
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-31 9:04 ` Stan Hoeppner
@ 2012-01-31 11:08 ` Emmanuel Florac
2012-02-01 12:31 ` Peter Grandi
2012-01-31 20:50 ` Dave Chinner
1 sibling, 1 reply; 18+ messages in thread
From: Emmanuel Florac @ 2012-01-31 11:08 UTC (permalink / raw)
To: stan; +Cc: xfs
Le Tue, 31 Jan 2012 03:04:18 -0600
Stan Hoeppner <stan@hardwarefreak.com> écrivait:
> I'd have never thought I would see those words on this list, except
> maybe as a joke, or as an example of one of the the worst possible
> platforms for XFS.
Oh come on, be nice for once :) People are constantly brainwashed about
how "cloud computing" will solve financial crisis and world hunger,
and didn't you notice that Amazon now sells an over-the-top storage
platform that'll cover everyone needs real soon now? If you believe
their marketing, of course. Don't forget to check:
http://aws.amazon.com/storagegateway/
> I wish EC2 had been asked about during the QA session after Dave's
> presentation. I'm guessing some laughter would have been involved. ;)
Is there a filesystem that's really suitable for EC2? What about
workloads? my impression is that EC2 is fine for whatever doesn't need
any QoS. Prototyping, for instance.
--
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <eflorac@intellique.com>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-31 5:04 ` Supratik Goswami
2012-01-31 7:19 ` Emmanuel Florac
2012-01-31 9:04 ` Stan Hoeppner
@ 2012-01-31 19:44 ` Dave Chinner
2 siblings, 0 replies; 18+ messages in thread
From: Dave Chinner @ 2012-01-31 19:44 UTC (permalink / raw)
To: Supratik Goswami; +Cc: xfs
On Tue, Jan 31, 2012 at 10:34:10AM +0530, Supratik Goswami wrote:
> We are using Amazon EC2 instances.
>
> ubuntu@ip-10-0-0-10:~$ uname -aLinux ip-10-0-0-10 2.6.32-318-ec2
> #38-Ubuntu SMP Thu Sep 1 18:09:30 UTC 2011 x86_64 GNU/Linux
The growfs hang problem was fixed in 2.6.34.
On earlier kernels, if you do a grow while the system is under
allocation load it could deadlock. growing on a mostly idle
filesystem was generally OK, but under heavy load problems could
occur. This hang is what xfstests 104 exercises...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-31 9:04 ` Stan Hoeppner
2012-01-31 11:08 ` Emmanuel Florac
@ 2012-01-31 20:50 ` Dave Chinner
2012-02-01 0:20 ` Stan Hoeppner
1 sibling, 1 reply; 18+ messages in thread
From: Dave Chinner @ 2012-01-31 20:50 UTC (permalink / raw)
To: Stan Hoeppner; +Cc: xfs
On Tue, Jan 31, 2012 at 03:04:18AM -0600, Stan Hoeppner wrote:
> On 1/30/2012 11:04 PM, Supratik Goswami wrote:
> > We are using Amazon EC2 instances.
>
> ^^^^^^^^^^
> I'd have never thought I would see those words on this list, except
> maybe as a joke, or as an example of one of the the worst possible
> platforms for XFS.
I don't agree with you there. If the workload works best on XFs, it
doesn't matter what the underlying storage device is. e.g. if it's a
fsync heavy workload, it will still perform better on XFS on EC2
than btrfs on EC2...
> I wish EC2 had been asked about during the QA session after Dave's
> presentation. I'm guessing some laughter would have been involved. ;)
You'd be wrong about that. There are as many good uses of cloud
services as there are bad ones, yet the same decisions about storage
need to be made even when services are remotely hosted....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-31 20:50 ` Dave Chinner
@ 2012-02-01 0:20 ` Stan Hoeppner
2012-02-01 11:40 ` Peter Grandi
0 siblings, 1 reply; 18+ messages in thread
From: Stan Hoeppner @ 2012-02-01 0:20 UTC (permalink / raw)
To: Dave Chinner; +Cc: xfs
On 1/31/2012 2:50 PM, Dave Chinner wrote:
> On Tue, Jan 31, 2012 at 03:04:18AM -0600, Stan Hoeppner wrote:
>> On 1/30/2012 11:04 PM, Supratik Goswami wrote:
>>> We are using Amazon EC2 instances.
>>
>> ^^^^^^^^^^
>> I'd have never thought I would see those words on this list, except
>> maybe as a joke, or as an example of one of the the worst possible
>> platforms for XFS.
>
> I don't agree with you there. If the workload works best on XFs, it
> doesn't matter what the underlying storage device is. e.g. if it's a
> fsync heavy workload, it will still perform better on XFS on EC2
> than btrfs on EC2...
>
>> I wish EC2 had been asked about during the QA session after Dave's
>> presentation. I'm guessing some laughter would have been involved. ;)
>
> You'd be wrong about that. There are as many good uses of cloud
> services as there are bad ones, yet the same decisions about storage
> need to be made even when services are remotely hosted....
Maybe I should have elaborated a bit. My thinking is that workloads
that would require XFS, or benefit most from it, are probably going to
need more guarantees WRT bandwidth and IOPS being available
consistently, vs sharing said resources with other systems in the cloud
infrastructure. Additionally, you have driven the point home many times
WRT tuning XFS to the underlying hardware, specifically stripe
alignment. I'd bet alignment would be a bit tricky to achieve in a
cloud environment such as EC2.
In summary, I wasn't saying XFS is bad on EC2. I was simply saying EC2
is probably bad for the typical workloads where XFS best flexes its muscles.
--
Stan
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-02-01 0:20 ` Stan Hoeppner
@ 2012-02-01 11:40 ` Peter Grandi
2012-02-01 23:55 ` Dave Chinner
2012-02-02 22:54 ` Peter Grandi
0 siblings, 2 replies; 18+ messages in thread
From: Peter Grandi @ 2012-02-01 11:40 UTC (permalink / raw)
To: Linux fs XFS
[ ... ]
>>> We are using Amazon EC2 instances.
>>> [ ... ] one of the the worst possible platforms for XFS.
>> I don't agree with you there. If the workload works best on
>> XFs, it doesn't matter what the underlying storage device is.
>> e.g. if it's a fsync heavy workload, it will still perform
>> better on XFS on EC2 than btrfs on EC2...
There are special cases, but «fsync heavy» is a bit of bad
example.
In general file system designs are not at all independent of the
expected storage platform, and some designs are far better than
others for specific storage platforms, and viceversa. This goes
all the way back to the 4BSD filesystem being specifically
optimized for rotational latency.
[ ... ]
>> You'd be wrong about that. There are as many good uses of
>> cloud services as there are bad ones,
VMs are not "cloud" services, those are more like remotely
hosted services, used via SOAP/REST. VMs are more like
colocation on the cheap.
>> yet the same decisions about storage need to be made even
>> when services are remotely hosted....
The basic problem with VM platforms is that they have completely
different latency (and somewhat different bandwidth) and
scheduling characteristics from "real" hardware, in particular
the relative costs of several operations are very different than
on "real" hardware, and the design tradeoffs that are good for
"real" hardware may not be relevant or may even be bad for VMs.
In addition VM "disks" can be implemented in crazy ways, like
with sparse files, and those impact severely achievable
performance levels.
> [ ... ] workloads that would require XFS, or benefit most from
> it, are probably going to need more guarantees WRT bandwidth
> and IOPS being available consistently, vs sharing said
> resources with other systems in the cloud infrastructure.
This is almost there, but «consistently» is a bit of an
understatement. It is not just that in VMs resources are
shared and subject to externally induced loads.
What matters is that the storage layer performance envelope have
roughly the same tradeoffs as those for which a certain design
has been aimed at. Even differently shaped hardware, like flash
SSD, can have very different performance envelopes than rotating
disks, or sets of rotating disks. A VM running on its own on
a certain platform still has different latencies and tradeoffs
than the underlying platform.
> Additionally, you have driven the point home many times WRT
> tuning XFS to the underlying hardware, specifically stripe
> alignment.
That as usual only matters for RMW-oriented storage layers, and
we don't really know what storage layer EC2 uses (hopefully not
one with RMW problems as parity RAID is known to be quite ill
suited to VM disks).
[ ... ]
> [ ... ] EC2 is probably bad for the typical workloads where
> XFS best flexes its muscles.
That's probably a good point but not quite the apposite one
here.
In the case raised by the OP, he had a large delay and "forgot"
to say he was running the system under layers (of unknown
structure) of virtualization.
In that case the latency (and bandwidth) profiles of both the
computing and the storage platforms can be very different from
those XFS has been aimed at, and I would not be surprised by
starvation or locking problems. Eventually DaveC pointed out a
known locking one during 'growfs', so not dependent on the
latency profile of the platform.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-01-31 11:08 ` Emmanuel Florac
@ 2012-02-01 12:31 ` Peter Grandi
2012-02-01 14:31 ` Emmanuel Florac
0 siblings, 1 reply; 18+ messages in thread
From: Peter Grandi @ 2012-02-01 12:31 UTC (permalink / raw)
To: Linux fs XFS
[ ... ]
> Is there a filesystem that's really suitable for EC2? What
> about workloads? my impression is that EC2 is fine for
> whatever doesn't need any QoS. Prototyping, for instance.
That's one use, but it is wider than that. Services like that
are good for "emarassingly parallel" workloads, where the QoS of
*a single* element does not matter, or even the *performance*
(or the *reliability*) of a single element is less important, at
least compared to the ability to throw a lot of cheap ones at a
problem.
Largely the same domain of application as the Google platform,
where their "embarassingly parallel" workload is log generation
and analysis.
Which advises that on EC2 simpler is better, and 'ext2' might be
most appropriate for non-shared applications. XFS, being like JFS
a rather general purpose design, looks appropriate, even if as
mentioned in another reply, it is aimed at massive and highly
parallel storage layers with highly threaded applications.
Aside note: I think that on most VM systems using "virtual disks"
of any sort except to store he OS filetree which is mostly RO is a
bad idea, and I suffered a lot last year dealing with a rather
hastily thrown together setup of that sort.
In that case I eliminated all but the root filetree VM disks and
replaced them with filetrees exported via NFS from XFS on the
underlying VM host itself (that is not over the network).
This improved performance tremendously (in part because in most VM
layers virtual NICs are more efficient than virtual disk adapters)
but in particular much faster check/repair and much reduced crazy
latencies during backups, because I could run check/repair and the
backups *on the real machine*, where XFS performed a lot better
without the VM overheads and "skewed" latencies.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-02-01 12:31 ` Peter Grandi
@ 2012-02-01 14:31 ` Emmanuel Florac
2012-02-01 22:20 ` Peter Grandi
0 siblings, 1 reply; 18+ messages in thread
From: Emmanuel Florac @ 2012-02-01 14:31 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux fs XFS
Le Wed, 1 Feb 2012 12:31:53 +0000
pg_xf2@xf2.for.sabi.co.UK (Peter Grandi) écrivait:
> In that case I eliminated all but the root filetree VM disks and
> replaced them with filetrees exported via NFS from XFS on the
> underlying VM host itself (that is not over the network).
>
> This improved performance tremendously (in part because in most VM
> layers virtual NICs are more efficient than virtual disk adapters)
> but in particular much faster check/repair and much reduced crazy
> latencies during backups, because I could run check/repair and the
> backups *on the real machine*, where XFS performed a lot better
> without the VM overheads and "skewed" latencies.
>
Thank you for all the good info. To add a last note, I use iSCSI to
export lvm LVs to VMs from the host, and it works fine. Exporting
files living on an XFS works well enough, too, though slightly slower.
It can be useful particularly for windows VM, because many windows app
really behave poorly with network shares (or refuse to use them
altogether).
--
------------------------------------------------------------------------
Emmanuel Florac | Direction technique
| Intellique
| <eflorac@intellique.com>
| +33 1 78 94 84 02
------------------------------------------------------------------------
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-02-01 14:31 ` Emmanuel Florac
@ 2012-02-01 22:20 ` Peter Grandi
0 siblings, 0 replies; 18+ messages in thread
From: Peter Grandi @ 2012-02-01 22:20 UTC (permalink / raw)
To: Linux fs XFS
>>> [ ... ] my impression is that EC2 is fine for whatever
>>> doesn't need any QoS. Prototyping, for instance. [ ... ]
>> [ ... ] *performance* (or the *reliability*) of a single
>> element is less important, at least compared to the ability
>> to throw a lot of cheap ones at a problem.
BTW, here I am not implying that EC2 allows one to «throw a lot
of cheap ones at a problem», because the published "retail" price
list is fairly expensive. But I guess that if one wants to buy «a
lot» of VMs as bulk purchase Amazon can do a deal.
>> In that case I eliminated all but the root filetree VM disks
>> and replaced them with filetrees exported via NFS from XFS on
>> the underlying VM host itself (that is not over the network).
>> [ ... ] because I could run check/repair and the backups *on
>> the real machine*, where XFS performed a lot better without
>> the VM overheads and "skewed" latencies.
> [ ... ] iSCSI to export lvm LVs to VMs from the host, and it
> works fine. Exporting files living on an XFS works well
> enough, too, though slightly slower.
iSCSI is a good alternative because it uses the better NIC
emulation in most VM layers, but I think that NFS is really a
better alternative overall, if suitable, because it gives the
inestimable option of running all the heavy hitting "maintenance"
stuff on the server itself, without any overheads, while
otherwise you must run it inside each VM.
Even if NFS has three problems that iSCSI does not have:
* It is a bit of a not awesome network filesystem, with a number
of limitations, but NFSv4 seems Oki-ish.
* It has a reputation of not playing that well with NFS, but
IIRC the stack issues happen only on 32b systems.
* While the server side is fairly good performance in Linux,
the NFS client in Linux has some non trivial performance
issues.
The problem is that there aren't much better network filesystems
around. Samba/SMB have a particularly rich and well done Linux
implementation, and are fully POSIX compatible, but performance
can be disappointing with the client in older kernels. A number
of sires have been discovering Gluster, and now that it is a Red
Hat product I guess we will here more of it especially in
relation to XFS.
BTW an attractive alternative to my usual favorite filesystems,
JFS and XFS, is the somewhat underestimated OCFS2, which is
well-maintained, and which can work pretty well in standalone
more, but also in share-disk mode, and that might be useful with
iSCSI to do backups etc. on another system than the client VM,
for example the server itself.
Also, an alternative to VMs is often using the pretty good
Linux-VServer.org "containers" (extended 'chroot's in effect),
which have zero overheads and where the only limitation is that
all "containers" must share the same running kernel, and can
share the same filesystem like exporting from NFS but without the
networking overhead. Xen (or UML) style paravirtualization is
next best (no need to emulate complicated "real" devices).
> It can be useful particularly for windows VM, because many
> windows app really behave poorly with network shares (or
> refuse to use them altogether).
That's a good point, and then one can also use the iSCSI daemon
on Linux to turn into a SAN server, but I guess you been there
and done that.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-02-01 11:40 ` Peter Grandi
@ 2012-02-01 23:55 ` Dave Chinner
2012-02-02 22:54 ` Peter Grandi
1 sibling, 0 replies; 18+ messages in thread
From: Dave Chinner @ 2012-02-01 23:55 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux fs XFS
On Wed, Feb 01, 2012 at 11:40:19AM +0000, Peter Grandi wrote:
> [ ... ]
>
> >>> We are using Amazon EC2 instances.
>
> >>> [ ... ] one of the the worst possible platforms for XFS.
>
> >> I don't agree with you there. If the workload works best on
> >> XFs, it doesn't matter what the underlying storage device is.
> >> e.g. if it's a fsync heavy workload, it will still perform
> >> better on XFS on EC2 than btrfs on EC2...
>
> There are special cases, but «fsync heavy» is a bit of bad
> example.
It's actually a really good example of where XFS will be better
than other filesystems. Why? Because XFS does less log IO due to
aggregation of log writes during concurrent fsyncs. The more latency
there is on a log write, the more aggregation that occurs. On a
platform where the IO subsystem is going to give you unpredictable
IO latencies, that's exactly what want.
Sure, it was designed to optimise spinning rust performance, but
that same design is also optimal for virtual devices with
unpredictable IO latency...
> In general file system designs are not at all independent of the
> expected storage platform, and some designs are far better than
> others for specific storage platforms, and viceversa.
Sure, but filesystems also have inherent capabilities that are
independent of the underlying storage. In these cases, the
underlying storage really doesn't matter if the filesystem can't do
what the application needs. Allocation parallelism, CPU
parallelism, minimal concurrent fsync latency, etc are all
characteristics of filesystems that are independent of the
underlying storage. If you need those characteristics in your
remotely hosted VMs, then XFS is what you want regardless of how
much storage capability you buy for those VMs....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-02-01 11:40 ` Peter Grandi
2012-02-01 23:55 ` Dave Chinner
@ 2012-02-02 22:54 ` Peter Grandi
2012-02-03 1:32 ` Dave Chinner
2012-02-15 11:38 ` Michael Monnerie
1 sibling, 2 replies; 18+ messages in thread
From: Peter Grandi @ 2012-02-02 22:54 UTC (permalink / raw)
To: Linux fs XFS
[ ... ]
>>>>> We are using Amazon EC2 instances.
>>>>> [ ... ] one of the the worst possible platforms for XFS.
>>>> I don't agree with you there. If the workload works best on
>>>> XFs, it doesn't matter what the underlying storage device
>>>> is. e.g. if it's a fsync heavy workload, it will still
>>>> perform better on XFS on EC2 than btrfs on EC2...
>> There are special cases, but «fsync heavy» is a bit of bad
>> example.
> It's actually a really good example of where XFS will be
> better than other filesystems.
But this is better at being less bad. Because we are talking here
about «fsync heavy» workloads on a VM, and these should not be
run on a VM if performance matters. That's why I wrote about a
«bad example» on which to discuss XFS for a VM.
But even with «fsync heavy» workloads in general your argument is
not exactly appropriate:
> Why? Because XFS does less log IO due to aggregation of log
> writes during concurrent fsyncs.
But «fsync heavy» does not necessarily means «concurrent fsyncs»,
for me it typically means logging or database apps where every
'write' is 'fsync'ed, even if there is a single thread. But let's
imagine for a moment we were talking about the special case where
«fsync heavy» involves a high degree of concurrency.
> The more latency there is on a log write, the more aggregation
> that occurs.
This seems to describe hardcoding in XFS a decision to trade
worse latency for better throughput, understandable as XFS was
after all quite clearly aimed at high throughput (or isochronous
throughput), rather than low latency (except for metadata, and
that has been "fixed" with 'delaylog').
Unless you mean that if the latency is low, then aggregation does
not take place, but then it is hard for me to see how that can be
*predicted*. I am assuming that in the above you refer to:
https://lwn.net/Articles/476267/
the XFS transaction subsystem is that most transactions are
asynchronous. That is, they don't commit to disk until either a
log buffer is filled (a log buffer can hold multiple transactions)
or a synchronous operation forces the log buffers holding the
transactions to disk. This means that XFS is doing aggregation of
transactions in memory - batching them, if you like - to minimise
the impact of the log IO on transaction throughput.
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/ch07s04s02.html
The delaylog mount option also improves sustained metadata
modification performance by reducing the number of changes to the
log. It achieves this by aggregating individual changes in memory
before writing them to the log: frequently modified metadata is
written to the log periodically instead of on every modification.
This option increases the memory usage of tracking dirty metadata
and increases the potential lost operations when a crash occurs,
but can improve metadata modification speed and scalability by an
order of magnitude or more. Use of this option does not reduce
data or metadata integrity when fsync, fdatasync or sync are used
to ensure data and metadata is written to disk.
BTW curious note in the latter:
However, under fsync-heavy workloads, small log buffers can be
noticeably faster than large buffers with a large stripe unit
alignment.
> On a platform where the IO subsystem is going to give you
> unpredictable IO latencies, that's exactly what want.
This then the argument that on platforms with bad latency that
decision works still works well because then you might as well go
for throughput.
But if someone really aims to run some kind of «fsync heavy»
workload on a high-latency and highly-variable latency VM,
usually their aim is to *minimize* the additional latency the
filesystem imposes, because «fsync heavy» workloads tend to be
transactional, and persisting data without delay is part of their
goal.
> Sure, it was designed to optimise spinning rust performance,
> but that same design is also optimal for virtual devices with
> unpredictable IO latency...
Ahhhh, now the «bad example» has become a worse one :-).
The argument you are making here is one for crass layering
violation: that the filesystem code should embed storage-layer
specific optimizations within it, and then one might get lucky
with other storage layers of similar profile. Tsk tsk :-). At
least it is not as breathtakingly inane as putting plug/unplug
block io subsystem.
But even on spinning rust, and on real host, and even forgiving
the layering violation, I question the aim to get better
throughput at the expense of worse latency for «fsync heavy»
loads, and even for the type of workloads for which this tradeoff
is good.
Because *my* argument is that how often 'fsync' "happens" should
be a decision by the application programmer; if they want higher
throughput at the cost of higher latency, they should issue it
less frequently, as 'fsync' should be executed with as low a
latency as possible.
Your underlying argument for XFS and its handling of «fsync
heavy» workloads (and it is the same argument for 'delaylog' I
guess) seems to me that applications issue 'fsync' too often, and
thus we can briefly hold them back to bunch them up, and people
like the extra throughput more than they dislike the extra latency.
Which reminds me of a discussions I had some time ago with some
misguided person who argued that 'fsync' and Linux barriers only
require ordering constraints, and don't imply any actual writing
to persistent storage, or within any specific timeframe, where
instead I was persuaded that their main purpose (no matter what
POSIX says :->) is to commit to persistent storage as quickly as
possible.
It looks like that XFS has gone more the way of something like
his position, because admittedly in practice keeping commits a
bit looser does deliver better throughput (hints of O_PONIES
here).
But again, that's not what should be happening. Perhaps POSIX
should have provided :-) two barrier operations, a purely
ordering one, and a commit-now one. And application writers would
use them at the right times. And ponies for everybody :-).
>> In general file system designs are not at all independent of
>> the expected storage platform, and some designs are far better
>> than others for specific storage platforms, and viceversa.
> Sure, but filesystems also have inherent capabilities that are
> independent of the underlying storage.
But the example you make is not a «capability», it is the
hardcoded assumption that it is better to trade worse latency for
better throughput, which only makes sense for workloads that
don't want tight latency, or else storage layers that don't
support it.
> In these cases, the underlying storage really doesn't matter if
> the filesystem can't do what the application needs. Allocation
> parallelism, CPU parallelism, minimal concurrent fsync latency,
But you seemed to be describing above that XFS good at "maximal
concurrent fsync throughput" by disregarding «minimal concurrent
fsync latency» (as in «less log IO due to aggregation of log
writes during concurrent fsyncs. The more latency there is on a
log write, the more aggregation»).
> etc are all characteristics of filesystems that are independent
> of the underlying storage.
Ahhhh, but this is a totally different argument from embedding
specific latency/throughput tradeoffs in the storage layer.
This is an argument that a well designed filesystem that does
have bottlenecks on any aspect of the performance envelope is a
good general purpose one. Well, you can try to design one :-).
XFS comes close, like JFS and OCFS2, but it does have, as you
have pointed out above, workload-specific (which can turn into
storage-friendly) tradeoffs. And since Red Hat's acquisition of
GlusterFS I guess (or at least I hope) that XFS will be even more
central to their strategy.
BTW as to that, did a brief search and found this amusing
article, yet another proof that reality surpasses imagination:
http://bioteam.net/2010/07/playing-with-nfs-glusterfs-on-amazon-cc1-4xlarge-ec2-instance-types/
Ah I was totally unaware of the AWS Compute Cluster service.
> If you need those characteristics in your remotely hosted VMs,
> then XFS is what you want regardless of how much storage
> capability you buy for those VMs....
Possibly, but from also a practical viewpoint that is again a
moderately bizarre argument, because workloads requiring high
levels of «Allocation parallelism, CPU parallelism, minimal
concurrent fsync latency» beg to be run on an Altix, or similar,
not on a bunch of random EC2 shared hosts running Xen VMs.
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-02-02 22:54 ` Peter Grandi
@ 2012-02-03 1:32 ` Dave Chinner
2012-02-15 11:38 ` Michael Monnerie
1 sibling, 0 replies; 18+ messages in thread
From: Dave Chinner @ 2012-02-03 1:32 UTC (permalink / raw)
To: Peter Grandi; +Cc: Linux fs XFS
On Thu, Feb 02, 2012 at 10:54:09PM +0000, Peter Grandi wrote:
> [ ... ]
>
> >>>>> We are using Amazon EC2 instances.
>
> >>>>> [ ... ] one of the the worst possible platforms for XFS.
>
> >>>> I don't agree with you there. If the workload works best on
> >>>> XFs, it doesn't matter what the underlying storage device
> >>>> is. e.g. if it's a fsync heavy workload, it will still
> >>>> perform better on XFS on EC2 than btrfs on EC2...
>
> >> There are special cases, but «fsync heavy» is a bit of bad
> >> example.
>
> > It's actually a really good example of where XFS will be
> > better than other filesystems.
>
> But this is better at being less bad. Because we are talking here
> about «fsync heavy» workloads on a VM, and these should not be
> run on a VM if performance matters. That's why I wrote about a
> «bad example» on which to discuss XFS for a VM.
Whether or not you should put a workload that does fsyncs in a VM is
a completely different argument altogether. It's not a meaningful
argument to make when we are talking about how filesystems deal with
unpredictable storage latencies or what filesystem to use in a
virtualised environment.
> But even with «fsync heavy» workloads in general your argument is
> not exactly appropriate:
>
> > Why? Because XFS does less log IO due to aggregation of log
> > writes during concurrent fsyncs.
>
> But «fsync heavy» does not necessarily means «concurrent fsyncs»,
> for me it typically means logging or database apps where every
> 'write' is 'fsync'ed, even if there is a single thread.
Doesn't matter if there's concurrent fsyncs - XFS will aggregreate all
transactions while there is one fsync or anything else that triggers
log forces in progress. It's a generic solution to the "we're doing
too many synchronous transactions really close together" problem.
> But let's
> imagine for a moment we were talking about the special case where
> «fsync heavy» involves a high degree of concurrency.
>
> > The more latency there is on a log write, the more aggregation
> > that occurs.
>
> This seems to describe hardcoding in XFS a decision to trade
> worse latency for better throughput,
Except it doesn't. XFS's mechanism is well known to -minimise-
journal latency without increasing individual or maximum latencies
as load increases. This then translates directly into higher
sustained throughputs because less time is spent by applications
waiting for IO completions because there is less IO being done.
Yes, you can trade off latency for throughput - that's easy to do -
but a well designed system acheives high throughput by minimising
the impact unavoidable latencies. That's what the XFS journal does.
And quite frankly, it does't matter what the source of the latency
is or whether it is unpredictable. If you can't avoid it, you have
to design to minimise the impact.
> understandable as XFS was
> after all quite clearly aimed at high throughput (or isochronous
> throughput), rather than low latency (except for metadata, and
> that has been "fixed" with 'delaylog').
I like how you say "fixed" in a way that implies you don't beleive
that it is fixed...
> Unless you mean that if the latency is low, then aggregation does
> not take place,
That's exactly what I'm saying.
> but then it is hard for me to see how that can be
> *predicted*.
That's because it doesn't need to be predicted. We *know* if a
journal write is currently in progress or not and we can wait on it
to complete. It doesn't matter how long it takes to complete - if it
is instantenous, then aggregation does not occur simply due to the
very short wait time. If the IO takes a long time to complete, then
lots of aggregation of transaction commits will occur before we
submit the next IO.
Smarter people than me designed this stuff - I've just learnt from
what they've done and built on top of it....
> I am assuming that in the above you refer to:
>
> https://lwn.net/Articles/476267/
Documentation/filesystems/xfs-delayed-logging-design.txt is a better
reference to use.
> the XFS transaction subsystem is
> that most transactions are asynchronous. That is, they don't
> commit to disk until either a log buffer is filled (a log buffer
> can hold multiple transactions) or a synchronous operation forces
> the log buffers holding the transactions to disk. This means that
> XFS is doing aggregation of transactions in memory - batching
> them, if you like - to minimise the impact of the log IO on
> transaction throughput.
That's part of it. This describes the pre-delaylog method of
aggregation, but even delaylog relies on this mechanism because
checkpoints are a journalled transaction just like all transactions
were pre-delaylog.
The point about fsync is that it is just an asynchronous transaction
as well. It is made synchronous by then pushing the log buffer to
disk. But it will only do that immeidately if the previous log
buffer is idle. If the previous log buffer is under IO, then it will
wait to start the IO on the current log buffer, allowing further
aggregation to occur.
> BTW curious note in the latter:
>
> However, under fsync-heavy workloads, small log buffers can be
> noticeably faster than large buffers with a large stripe unit
> alignment.
Because setting a log stripe unit (LSU) mean the size of the log IO is
padded. A 32k LSU means the minimum log IO size is 32k, while an
fsync transaciton is usually only a couple of hundred bytes. Without
an LSU, than means a solitary fsync transaction being written to disk
will be 512 bytes vs 32kB with a LSU and that means the non LSU-log will
complete IO faster. Same goes for LSU=32k vs LSU=256k.
> > On a platform where the IO subsystem is going to give you
> > unpredictable IO latencies, that's exactly what want.
>
> This then the argument that on platforms with bad latency that
> decision works still works well because then you might as well go
> for throughput.
If one fsync takes X, and you can make 10 concurrent fsyncs take X,
why wouldn't you optimise to enable the latter case? It doesn't
matter if X is 10us, 1ms or even 1s - having an algorithm that works
independently of the magnitude of the storage latency will result in
good throughput no matter the storage characteristics. That's what
users want - something that just works without needing to tweak it
differently to perform optimally on all their different systems...
> But if someone really aims to run some kind of «fsync heavy»
> workload on a high-latency and highly-variable latency VM, usually
> their aim is to *minimize* the additional latency the filesystem
> imposes, because «fsync heavy» workloads tend to be transactional,
> and persisting data without delay is part of their goal.
I still don't understand what part of "use XFS for this workload"
you are saying is wrong?
> > Sure, it was designed to optimise spinning rust performance, but
> > that same design is also optimal for virtual devices with
> > unpredictable IO latency...
>
> Ahhhh, now the «bad example» has become a worse one :-).
>
> The argument you are making here is one for crass layering
> violation: that the filesystem code should embed storage-layer
> specific optimizations within it, and then one might get lucky
> with other storage layers of similar profile. Tsk tsk :-). At
> least it is not as breathtakingly inane as putting plug/unplug
> block io subsystem.
Filesystems are nothing but a dense concentration algorithms that
are optimal for as wide a range of known storage behaviours as
possible.
> XFS comes close, like JFS and OCFS2, but it does have, as you have
> pointed out above, workload-specific (which can turn into
> storage-friendly) tradeoffs. And since Red Hat's acquisition of
> GlusterFS I guess (or at least I hope) that XFS will be even more
> central to their strategy.
http://docs.redhat.com/docs/en-US/Red_Hat_Storage_Software_Appliance/3.2/html-single/User_Guide/index.html#sect-User_Guide-gssa_prepare-chec_min_req
"File System Requirements
Red Hat recommends XFS when formatting the disk sub-system. ..."
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: File system remain unresponsive until the system is rebooted.
2012-02-02 22:54 ` Peter Grandi
2012-02-03 1:32 ` Dave Chinner
@ 2012-02-15 11:38 ` Michael Monnerie
1 sibling, 0 replies; 18+ messages in thread
From: Michael Monnerie @ 2012-02-15 11:38 UTC (permalink / raw)
To: xfs
[-- Attachment #1.1: Type: text/plain, Size: 1998 bytes --]
Am Donnerstag, 2. Februar 2012, 22:54:09 schrieb Peter Grandi:
> This then the argument that on platforms with bad latency that
> decision works still works well because then you might as well go
> for throughput.
Hi, I just took these lines to reply to your whole mail. I guess that
the advantage of XFS will grow on a shared storage type like you
typically have on a VM environment. The aggregation XFS does can result
in a more bursty type of I/O, with larger I/Os happening at once. That
always is better for RAID storage - which you normally have in a VM
environment. Also, all better RAID controllers, and especially
enterprise RAIDs, have large write buffers, so even more aggregation
occurs at the storage itself, helping throughput maximisation.
I don't know of any scientific investigation of "which filesystem is
better in a VM environment" that could be referenced in a generic way,
mostly because there are so many variables there that it doesn't
neccessarily fit your own use case. Maybe someone can point me to such
research material.
My hope is - and that is what Dave is arguing - that minimising I/O
"disturbances" by metadata work like log file handling helps keeping
overall throughput on a shared storage type in a VM environment high.
And that seems very reasonable.
I don't really understand your argument about delay for a single thread
fsync. First, XFS should do this quicker by "batching" transactions, and
second, overall storage throughput is usually much more important than
that of a single server performance - at least in a VM environment. I
need to run 50 servers on a storage with acceptable performance, and if
one server needs more performance than is available, you need to do
something else - there are lots of options then.
--
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc
it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531
[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
[-- Attachment #2: Type: text/plain, Size: 121 bytes --]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2012-02-15 11:39 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-30 9:18 File system remain unresponsive until the system is rebooted Supratik Goswami
2012-01-30 17:23 ` Peter Grandi
2012-01-31 1:31 ` Dave Chinner
2012-01-31 5:04 ` Supratik Goswami
2012-01-31 7:19 ` Emmanuel Florac
2012-01-31 9:04 ` Stan Hoeppner
2012-01-31 11:08 ` Emmanuel Florac
2012-02-01 12:31 ` Peter Grandi
2012-02-01 14:31 ` Emmanuel Florac
2012-02-01 22:20 ` Peter Grandi
2012-01-31 20:50 ` Dave Chinner
2012-02-01 0:20 ` Stan Hoeppner
2012-02-01 11:40 ` Peter Grandi
2012-02-01 23:55 ` Dave Chinner
2012-02-02 22:54 ` Peter Grandi
2012-02-03 1:32 ` Dave Chinner
2012-02-15 11:38 ` Michael Monnerie
2012-01-31 19:44 ` Dave Chinner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox