Hangs in balance_dirty_pages with arm-32 LPAE + highmem

Hangs in balance_dirty_pages with arm-32 LPAE + highmem

[Laura Abbott]

Hi,

The Fedora arm-32 build VMs have a somewhat long standing problem
of hanging when running mkfs.ext4 with a bunch of processes stuck
in D state. This has been seen as far back as 4.13 but is still
present on 4.14:

sysrq: SysRq : Show Blocked State                                                [255/1885]
   task                PC stack   pid father
auditd          D    0   377      1 0x00000020
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224ef0>] (__sys_trace_return+0x0/0x10)
rs:main Q:Reg   D    0   441      1 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
ntpd            D    0  1453      1 0x00000001
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)  [203/1885]
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
kojid           D    0  4616      1 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
kworker/u8:0    D    0 28525      2 0x00000000
Workqueue: writeback wb_workfn (flush-7:0)
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c0281c34>] (io_schedule+0x1c/0x2c)
[<c0281c34>] (io_schedule) from [<c055ff2c>] (wbt_wait+0x21c/0x300)
[<c055ff2c>] (wbt_wait) from [<c053ba14>] (blk_mq_make_request+0xac/0x560)
[<c053ba14>] (blk_mq_make_request) from [<c0530034>] (generic_make_request+0xd0/0x214)
[<c0530034>] (generic_make_request) from [<c053028c>] (submit_bio+0x114/0x16c)
[<c053028c>] (submit_bio) from [<c0412b98>] (submit_bh_wbc+0x190/0x1a0)
[<c0412b98>] (submit_bh_wbc) from [<c0412e90>] (__block_write_full_page+0x2e8/0x43c)
[<c0412e90>] (__block_write_full_page) from [<c0413150>] (block_write_full_page+0x80/0xec)
[<c0413150>] (block_write_full_page) from [<c0370380>] (__writepage+0x1c/0x4c)
[<c0370380>] (__writepage) from [<c0370d30>] (write_cache_pages+0x350/0x3f0)
[<c0370d30>] (write_cache_pages) from [<c03715d8>] (generic_writepages+0x44/0x60)
[<c03715d8>] (generic_writepages) from [<c0372eb4>] (do_writepages+0x3c/0x74)
[<c0372eb4>] (do_writepages) from [<c04098a8>] (__writeback_single_inode+0xb4/0x404)
[<c04098a8>] (__writeback_single_inode) from [<c040a118>] (writeback_sb_inodes+0x258/0x438)
[<c040a118>] (writeback_sb_inodes) from [<c040a364>] (__writeback_inodes_wb+0x6c/0xa8)
[<c040a364>] (__writeback_inodes_wb) from [<c040a564>] (wb_writeback+0x1c4/0x30c)
[<c040a564>] (wb_writeback) from [<c040ae98>] (wb_workfn+0x130/0x450)
[<c040ae98>] (wb_workfn) from [<c026f488>] (process_one_work+0x254/0x42c)
[<c026f488>] (process_one_work) from [<c02705c4>] (worker_thread+0x2d0/0x450)
[<c02705c4>] (worker_thread) from [<c02750cc>] (kthread+0x13c/0x154)
[<c02750cc>] (kthread) from [<c0224e18>] (ret_from_fork+0x14/0x3c)
kworker/u8:1    D    0 16594      2 0x00000000
Workqueue: writeback wb_workfn (flush-252:0)
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c0281c34>] (io_schedule+0x1c/0x2c)
[<c0281c34>] (io_schedule) from [<c055ff2c>] (wbt_wait+0x21c/0x300)
[<c055ff2c>] (wbt_wait) from [<c053ba14>] (blk_mq_make_request+0xac/0x560)
[<c053ba14>] (blk_mq_make_request) from [<c0530034>] (generic_make_request+0xd0/0x214)
[<c0530034>] (generic_make_request) from [<c053028c>] (submit_bio+0x114/0x16c)
[<c053ba14>] (blk_mq_make_request) from [<c0530034>] (generic_make_request+0xd0/0[151/1885]
[<c0530034>] (generic_make_request) from [<c053028c>] (submit_bio+0x114/0x16c)
[<c053028c>] (submit_bio) from [<c0412b98>] (submit_bh_wbc+0x190/0x1a0)
[<c0412b98>] (submit_bh_wbc) from [<c0412e90>] (__block_write_full_page+0x2e8/0x43c)
[<c0412e90>] (__block_write_full_page) from [<c0413150>] (block_write_full_page+0x80/0xec)
[<c0413150>] (block_write_full_page) from [<c0370380>] (__writepage+0x1c/0x4c)
[<c0370380>] (__writepage) from [<c0370d30>] (write_cache_pages+0x350/0x3f0)
[<c0370d30>] (write_cache_pages) from [<c03715d8>] (generic_writepages+0x44/0x60)
[<c03715d8>] (generic_writepages) from [<c0372eb4>] (do_writepages+0x3c/0x74)
[<c0372eb4>] (do_writepages) from [<c04098a8>] (__writeback_single_inode+0xb4/0x404)
[<c04098a8>] (__writeback_single_inode) from [<c040a118>] (writeback_sb_inodes+0x258/0x438)
[<c040a118>] (writeback_sb_inodes) from [<c040a364>] (__writeback_inodes_wb+0x6c/0xa8)
[<c040a364>] (__writeback_inodes_wb) from [<c040a564>] (wb_writeback+0x1c4/0x30c)
[<c040a564>] (wb_writeback) from [<c040ae98>] (wb_workfn+0x130/0x450)
[<c040ae98>] (wb_workfn) from [<c026f488>] (process_one_work+0x254/0x42c)
[<c026f488>] (process_one_work) from [<c02705c4>] (worker_thread+0x2d0/0x450)
[<c02705c4>] (worker_thread) from [<c02750cc>] (kthread+0x13c/0x154)
[<c02750cc>] (kthread) from [<c0224e18>] (ret_from_fork+0x14/0x3c)
loop0           D    0  9138      2 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03dbccc>] (do_iter_readv_writev+0x118/0x140)
[<c03dbccc>] (do_iter_readv_writev) from [<c03dd024>] (do_iter_write+0x84/0xf8)
[<c03dd024>] (do_iter_write) from [<bf38b19c>] (lo_write_bvec+0x70/0xec [loop])
[<bf38b19c>] (lo_write_bvec [loop]) from [<bf38c04c>] (loop_queue_work+0x3b4/0x92c [loop])
[<bf38c04c>] (loop_queue_work [loop]) from [<c0275234>] (kthread_worker_fn+0x114/0x1c8)
[<c0275234>] (kthread_worker_fn) from [<c02750cc>] (kthread+0x13c/0x154)
[<c02750cc>] (kthread) from [<c0224e18>] (ret_from_fork+0x14/0x3c)
mkfs.ext4       D    0  9142   1535 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c0281c34>] (io_schedule+0x1c/0x2c)
[<c0281c34>] (io_schedule) from [<c0362080>] (__lock_page+0x10c/0x144)
[<c0362080>] (__lock_page) from [<c0370bb8>] (write_cache_pages+0x1d8/0x3f0)
[<c0370bb8>] (write_cache_pages) from [<c03715d8>] (generic_writepages+0x44/0x60)
[<c03715d8>] (generic_writepages) from [<c0372eb4>] (do_writepages+0x3c/0x74)
[<c0372eb4>] (do_writepages) from [<c0364264>] (__filemap_fdatawrite_range+0xc0/0xe0)
[<c0364264>] (__filemap_fdatawrite_range) from [<c0364454>] (file_write_and_wait_range+0x40
/0x78)
[<c0364454>] (file_write_and_wait_range) from [<c0415868>] (blkdev_fsync+0x20/0x50)
[<c0415868>] (blkdev_fsync) from [<c040e26c>] (vfs_fsync+0x28/0x30)
[<c040e26c>] (vfs_fsync) from [<c040e2a4>] (do_fsync+0x30/0x4c)
[<c040e2a4>] (do_fsync) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
python          D    0  9167   9165 0x00000000
Sun Feb 18 18:17:58 2018] [<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c0397724>] (fault_dirty_shared_page+0
x9c/0xb4)
[<c0397724>] (fault_dirty_shared_page) from [<c0399c48>] (do_wp_page+0x628/0x688)
[<c0399c48>] (do_wp_page) from [<c039cd0c>] (handle_mm_fault+0xd5c/0xe08)
[<c039cd0c>] (handle_mm_fault) from [<c095c7d8>] (do_page_fault+0x1f0/0x360)
[<c095c7d8>] (do_page_fault) from [<c0201384>] (do_DataAbort+0x34/0xb4)
[<c0201384>] (do_DataAbort) from [<c095c19c>] (__dabt_usr+0x3c/0x40)
Exception stack(0xc69dbfb0 to 0xc69dbff8)
bfa0:                                     019fec50 00000002 cc684d00 cc684d00
bfc0: 00000001 b4214cf4 019fec50 00000002 000587b8 00000000 00000001 b4214cf0
bfe0: b4f39584 bec32990 b4eee238 b4e799a0 600f0010 ffffffff
python          D    0  9313   9304 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
python          D    0  9326   9317 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
python          D    0  9351   9342 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
python          D    0  9361   9352 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
python          D    0  9374   9365 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
python          D    0  9385   9376 0x00000000
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c03637f8>] (generic_perform_write+0x1
74/0x1a4)
[<c03637f8>] (generic_perform_write) from [<c0365e98>] (__generic_file_write_iter+0x16c/0x1
98)
[<c0365e98>] (__generic_file_write_iter) from [<c046b398>] (ext4_file_write_iter+0x314/0x41
4)
[<c046b398>] (ext4_file_write_iter) from [<c03ddd68>] (__vfs_write+0x100/0x128)
[<c03ddd68>] (__vfs_write) from [<c03ddf5c>] (vfs_write+0xc0/0x194)
[<c03ddf5c>] (vfs_write) from [<c03de12c>] (SyS_write+0x44/0x7c)
[<c03de12c>] (SyS_write) from [<c0224d40>] (ret_fast_syscall+0x0/0x4c)
systemd-journal D    0  9678      1 0x00000080
[<c0955e68>] (__schedule) from [<c0956070>] (schedule+0x98/0xbc)
[<c0956070>] (schedule) from [<c09598e4>] (schedule_timeout+0x328/0x3ac)
[<c09598e4>] (schedule_timeout) from [<c09564f8>] (io_schedule_timeout+0x24/0x38)
[<c09564f8>] (io_schedule_timeout) from [<c03720bc>] (balance_dirty_pages.constprop.6+0xac8
/0xc5c)
[<c03720bc>] (balance_dirty_pages.constprop.6) from [<c0372508>] (balance_dirty_pages_ratel
imited+0x2b8/0x43c)
[<c0372508>] (balance_dirty_pages_ratelimited) from [<c0397724>] (fault_dirty_shared_page+0
x9c/0xb4)
[<c0397724>] (fault_dirty_shared_page) from [<c039cc34>] (handle_mm_fault+0xc84/0xe08)
[<c039cc34>] (handle_mm_fault) from [<c095c7d8>] (do_page_fault+0x1f0/0x360)
[<c095c7d8>] (do_page_fault) from [<c0201384>] (do_DataAbort+0x34/0xb4)
[<c0201384>] (do_DataAbort) from [<c095c19c>] (__dabt_usr+0x3c/0x40)
Exception stack(0xc40d9fb0 to 0xc40d9ff8)
9fa0:                                     b5e605e0 00000000 001c2b28 b5e64000
9fc0: 01b60ff0 00000000 be94e444 be94e448 001c6550 00000000 be94e660 be94e450
9fe0: 00000000 be94e400 b6cfffcc b6e94a50 20000010 ffffffff

This looks like everything is blocked on the writeback completing but
the writeback has been throttled. According to the infra team, this problem
is _not_ seen without LPAE (i.e. only 4G of RAM). I did see
https://patchwork.kernel.org/patch/10201593/ but that doesn't seem to
quite match since this seems to be completely stuck. Any suggestions to
narrow the problem down?

Thanks,
Laura

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hangs in balance_dirty_pages with arm-32 LPAE + highmem

[Michal Hocko]

On Fri 23-02-18 11:51:41, Laura Abbott wrote:
> Hi,
> 
> The Fedora arm-32 build VMs have a somewhat long standing problem
> of hanging when running mkfs.ext4 with a bunch of processes stuck
> in D state. This has been seen as far back as 4.13 but is still
> present on 4.14:
> 
[...]
> This looks like everything is blocked on the writeback completing but
> the writeback has been throttled. According to the infra team, this problem
> is _not_ seen without LPAE (i.e. only 4G of RAM). I did see
> https://patchwork.kernel.org/patch/10201593/ but that doesn't seem to
> quite match since this seems to be completely stuck. Any suggestions to
> narrow the problem down?

How much dirtyable memory does the system have? We do allow only lowmem
to be dirtyable by default on 32b highmem systems. Maybe you have the
lowmem mostly consumed by the kernel memory. Have you tried to enable
highmem_is_dirtyable?
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hangs in balance_dirty_pages with arm-32 LPAE + highmem

[Laura Abbott]

On 02/26/2018 06:28 AM, Michal Hocko wrote:
> On Fri 23-02-18 11:51:41, Laura Abbott wrote:
>> Hi,
>>
>> The Fedora arm-32 build VMs have a somewhat long standing problem
>> of hanging when running mkfs.ext4 with a bunch of processes stuck
>> in D state. This has been seen as far back as 4.13 but is still
>> present on 4.14:
>>
> [...]
>> This looks like everything is blocked on the writeback completing but
>> the writeback has been throttled. According to the infra team, this problem
>> is _not_ seen without LPAE (i.e. only 4G of RAM). I did see
>> https://patchwork.kernel.org/patch/10201593/ but that doesn't seem to
>> quite match since this seems to be completely stuck. Any suggestions to
>> narrow the problem down?
> 
> How much dirtyable memory does the system have? We do allow only lowmem
> to be dirtyable by default on 32b highmem systems. Maybe you have the
> lowmem mostly consumed by the kernel memory. Have you tried to enable
> highmem_is_dirtyable?
> 

Setting highmem_is_dirtyable did fix the problem. The infrastructure
people seemed satisfied enough with this (and are happy to have the
machines back). I'll see if they are willing to run a few more tests
to get some more state information.

Thanks,
Laura

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hangs in balance_dirty_pages with arm-32 LPAE + highmem

[Tetsuo Handa]

Laura Abbott wrote:
> On 02/26/2018 06:28 AM, Michal Hocko wrote:
> > On Fri 23-02-18 11:51:41, Laura Abbott wrote:
> >> Hi,
> >>
> >> The Fedora arm-32 build VMs have a somewhat long standing problem
> >> of hanging when running mkfs.ext4 with a bunch of processes stuck
> >> in D state. This has been seen as far back as 4.13 but is still
> >> present on 4.14:
> >>
> > [...]
> >> This looks like everything is blocked on the writeback completing but
> >> the writeback has been throttled. According to the infra team, this problem
> >> is _not_ seen without LPAE (i.e. only 4G of RAM). I did see
> >> https://patchwork.kernel.org/patch/10201593/ but that doesn't seem to
> >> quite match since this seems to be completely stuck. Any suggestions to
> >> narrow the problem down?
> > 
> > How much dirtyable memory does the system have? We do allow only lowmem
> > to be dirtyable by default on 32b highmem systems. Maybe you have the
> > lowmem mostly consumed by the kernel memory. Have you tried to enable
> > highmem_is_dirtyable?
> > 
> 
> Setting highmem_is_dirtyable did fix the problem. The infrastructure
> people seemed satisfied enough with this (and are happy to have the
> machines back).

That's good.

>                 I'll see if they are willing to run a few more tests
> to get some more state information.

Well, I'm far from understanding what is happening in your case, but I'm
interested in other threads which were trying to allocate memory. Therefore,
I appreciate if they can take SysRq-m + SysRq-t than SysRq-w (as described
at http://akari.osdn.jp/capturing-kernel-messages.html ).

Code which assumes that kswapd can make progress can get stuck when kswapd
is blocked somewhere. And wbt_wait() seems to change behavior based on
current_is_kswapd(). If everyone is waiting for kswapd but kswapd cannot
make progress, I worry that it leads to hangups like your case.



Below is a totally different case which I got today, but an example of
whether SysRq-m + SysRq-t can give us some clues.

Running below program on CPU 0 (using "taskset -c 0") on 4.16-rc4 against XFS
can trigger OOM lockups (hangup without being able to invoke the OOM killer).

----------
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(int argc, char *argv[])
{
	static char buffer[4096] = { };
	char *buf = NULL;
	unsigned long size;
	unsigned long i;
	for (i = 0; i < 1024; i++) {
		if (fork() == 0) {
			int fd;
			snprintf(buffer, sizeof(buffer), "/tmp/file.%u", getpid());
			fd = open(buffer, O_WRONLY | O_CREAT | O_APPEND, 0600);
			memset(buffer, 0, sizeof(buffer));
			sleep(1);
			while (write(fd, buffer, sizeof(buffer)) == sizeof(buffer));
			_exit(0);
		}
	}
	for (size = 1048576; size < 512UL * (1 << 30); size <<= 1) {
		char *cp = realloc(buf, size);
		if (!cp) {
			size >>= 1;
			break;
		}
		buf = cp;
	}
	sleep(2);
	/* Will cause OOM due to overcommit */
	for (i = 0; i < size; i += 4096)
		buf[i] = 0;
	return 0;
}
----------

MM people love to ignore such kind of problem with "It is a DoS attack", but
only one CPU out of 8 CPUs is occupied by this program, which means that other
threads (including kernel threads doing memory reclaim activities) are free to
use idle CPUs 1-7 as they need. Also, while CPU 0 was really busy processing
hundreds of threads doing direct reclaim, idle CPUs 1-7 should be able to invoke
the OOM killer shortly because there should be already little to reclaim. Also,
writepending: did not decrease (and no disk I/O was observed) during the OOM
lockup. Thus, I don't know whether this is just an overloaded.

[  660.035957] Node 0 Normal free:17056kB min:17320kB low:21648kB high:25976kB active_anon:570132kB inactive_anon:13452kB active_file:15136kB inactive_file:13296kB unevictable:0kB writepending:42320kB present:1048576kB managed:951188kB mlocked:0kB kernel_stack:22448kB pagetables:37304kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[  709.498421] Node 0 Normal free:16920kB min:17320kB low:21648kB high:25976kB active_anon:570132kB inactive_anon:13452kB active_file:19180kB inactive_file:17640kB unevictable:0kB writepending:42740kB present:1048576kB managed:951188kB mlocked:0kB kernel_stack:22400kB pagetables:37304kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB
[  751.290146] Node 0 Normal free:16920kB min:17320kB low:21648kB high:25976kB active_anon:570132kB inactive_anon:13452kB active_file:14556kB inactive_file:14452kB unevictable:0kB writepending:42740kB present:1048576kB managed:951188kB mlocked:0kB kernel_stack:22400kB pagetables:37304kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB
[  783.437211] Node 0 Normal free:16920kB min:17320kB low:21648kB high:25976kB active_anon:570132kB inactive_anon:13452kB active_file:14756kB inactive_file:13888kB unevictable:0kB writepending:42740kB present:1048576kB managed:951188kB mlocked:0kB kernel_stack:22304kB pagetables:37304kB bounce:0kB free_pcp:312kB local_pcp:32kB free_cma:0kB
[ 1242.729271] Node 0 Normal free:16920kB min:17320kB low:21648kB high:25976kB active_anon:570132kB inactive_anon:13452kB active_file:14072kB inactive_file:14304kB unevictable:0kB writepending:42740kB present:1048576kB managed:951188kB mlocked:0kB kernel_stack:22128kB pagetables:37304kB bounce:0kB free_pcp:440kB local_pcp:48kB free_cma:0kB
[ 1412.248884] Node 0 Normal free:16920kB min:17320kB low:21648kB high:25976kB active_anon:570132kB inactive_anon:13452kB active_file:14332kB inactive_file:14280kB unevictable:0kB writepending:42740kB present:1048576kB managed:951188kB mlocked:0kB kernel_stack:22128kB pagetables:37304kB bounce:0kB free_pcp:440kB local_pcp:48kB free_cma:0kB
[ 1549.795514] Node 0 Normal free:16920kB min:17320kB low:21648kB high:25976kB active_anon:570132kB inactive_anon:13452kB active_file:14416kB inactive_file:14272kB unevictable:0kB writepending:42740kB present:1048576kB managed:951188kB mlocked:0kB kernel_stack:22128kB pagetables:37304kB bounce:0kB free_pcp:440kB local_pcp:48kB free_cma:0kB

Complete log is http://I-love.SAKURA.ne.jp/tmp/serial-20180306.txt.xz .
Config is http://I-love.SAKURA.ne.jp/tmp/config-4.16-rc4 .

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Hangs in balance_dirty_pages with arm-32 LPAE + highmem

[Michal Hocko]

On Mon 05-03-18 13:04:24, Laura Abbott wrote:
> On 02/26/2018 06:28 AM, Michal Hocko wrote:
> > On Fri 23-02-18 11:51:41, Laura Abbott wrote:
> > > Hi,
> > > 
> > > The Fedora arm-32 build VMs have a somewhat long standing problem
> > > of hanging when running mkfs.ext4 with a bunch of processes stuck
> > > in D state. This has been seen as far back as 4.13 but is still
> > > present on 4.14:
> > > 
> > [...]
> > > This looks like everything is blocked on the writeback completing but
> > > the writeback has been throttled. According to the infra team, this problem
> > > is _not_ seen without LPAE (i.e. only 4G of RAM). I did see
> > > https://patchwork.kernel.org/patch/10201593/ but that doesn't seem to
> > > quite match since this seems to be completely stuck. Any suggestions to
> > > narrow the problem down?
> > 
> > How much dirtyable memory does the system have? We do allow only lowmem
> > to be dirtyable by default on 32b highmem systems. Maybe you have the
> > lowmem mostly consumed by the kernel memory. Have you tried to enable
> > highmem_is_dirtyable?
> > 
> 
> Setting highmem_is_dirtyable did fix the problem. The infrastructure
> people seemed satisfied enough with this (and are happy to have the
> machines back). I'll see if they are willing to run a few more tests
> to get some more state information.

Please be aware that highmem_is_dirtyable is not for free. There are
some code paths which can only allocate from lowmem (e.g. block device
AFAIR) and those could fill up the whole lowmem without any throttling.
-- 
Michal Hocko
SUSE Labs

Re: Hangs in balance_dirty_pages with arm-32 LPAE + highmem

[Michal Hocko]

On Tue 06-03-18 20:28:59, Tetsuo Handa wrote:
> Laura Abbott wrote:
> > On 02/26/2018 06:28 AM, Michal Hocko wrote:
> > > On Fri 23-02-18 11:51:41, Laura Abbott wrote:
> > >> Hi,
> > >>
> > >> The Fedora arm-32 build VMs have a somewhat long standing problem
> > >> of hanging when running mkfs.ext4 with a bunch of processes stuck
> > >> in D state. This has been seen as far back as 4.13 but is still
> > >> present on 4.14:
> > >>
> > > [...]
> > >> This looks like everything is blocked on the writeback completing but
> > >> the writeback has been throttled. According to the infra team, this problem
> > >> is _not_ seen without LPAE (i.e. only 4G of RAM). I did see
> > >> https://patchwork.kernel.org/patch/10201593/ but that doesn't seem to
> > >> quite match since this seems to be completely stuck. Any suggestions to
> > >> narrow the problem down?
> > > 
> > > How much dirtyable memory does the system have? We do allow only lowmem
> > > to be dirtyable by default on 32b highmem systems. Maybe you have the
> > > lowmem mostly consumed by the kernel memory. Have you tried to enable
> > > highmem_is_dirtyable?
> > > 
> > 
> > Setting highmem_is_dirtyable did fix the problem. The infrastructure
> > people seemed satisfied enough with this (and are happy to have the
> > machines back).
> 
> That's good.
> 
> >                 I'll see if they are willing to run a few more tests
> > to get some more state information.
> 
> Well, I'm far from understanding what is happening in your case, but I'm
> interested in other threads which were trying to allocate memory. Therefore,
> I appreciate if they can take SysRq-m + SysRq-t than SysRq-w (as described
> at http://akari.osdn.jp/capturing-kernel-messages.html ).
> 
> Code which assumes that kswapd can make progress can get stuck when kswapd
> is blocked somewhere. And wbt_wait() seems to change behavior based on
> current_is_kswapd(). If everyone is waiting for kswapd but kswapd cannot
> make progress, I worry that it leads to hangups like your case.

Tetsuo, could you stop this finally, pretty please? This is a
well known limitation of 32b architectures with more than 4G. The lowmem
can only handle 896MB of memory and that can be filled up with other
kernel allocations. Stalled writeback is _usually_ a result of only
little dirtyable memory which is left in the lowmem. We cannot simply
allow highmem to be dirtyable by default due to reasons explained in
other email.

I can imagine that it is hard for you to grasp that not everything is
"silent hang during OOM" but there are other things going on in the VM.
-- 
Michal Hocko
SUSE Labs

Re: Hangs in balance_dirty_pages with arm-32 LPAE + highmem

[Laura Abbott]

On 03/14/2018 02:08 AM, Michal Hocko wrote:
> On Mon 05-03-18 13:04:24, Laura Abbott wrote:
>> On 02/26/2018 06:28 AM, Michal Hocko wrote:
>>> On Fri 23-02-18 11:51:41, Laura Abbott wrote:
>>>> Hi,
>>>>
>>>> The Fedora arm-32 build VMs have a somewhat long standing problem
>>>> of hanging when running mkfs.ext4 with a bunch of processes stuck
>>>> in D state. This has been seen as far back as 4.13 but is still
>>>> present on 4.14:
>>>>
>>> [...]
>>>> This looks like everything is blocked on the writeback completing but
>>>> the writeback has been throttled. According to the infra team, this problem
>>>> is _not_ seen without LPAE (i.e. only 4G of RAM). I did see
>>>> https://patchwork.kernel.org/patch/10201593/ but that doesn't seem to
>>>> quite match since this seems to be completely stuck. Any suggestions to
>>>> narrow the problem down?
>>>
>>> How much dirtyable memory does the system have? We do allow only lowmem
>>> to be dirtyable by default on 32b highmem systems. Maybe you have the
>>> lowmem mostly consumed by the kernel memory. Have you tried to enable
>>> highmem_is_dirtyable?
>>>
>>
>> Setting highmem_is_dirtyable did fix the problem. The infrastructure
>> people seemed satisfied enough with this (and are happy to have the
>> machines back). I'll see if they are willing to run a few more tests
>> to get some more state information.
> 
> Please be aware that highmem_is_dirtyable is not for free. There are
> some code paths which can only allocate from lowmem (e.g. block device
> AFAIR) and those could fill up the whole lowmem without any throttling.
> 

Good to note. This particular setup is one basically everyone dislikes
so I think this is only encouragement to move to something else.

Thanks,
Laura