[f2fs-dev] IO hang due to f2fs checkpoint and writeback stuck

[Sahitya Tummala <stummala@codeaurora.org>]

Hi Chao, Jaegeuk,

I have received an issue report that indicates that system is stuck
on IO due to f2fs checkpoint and writeback stuck waiting on each other
as explained below.

WB thread -
----------

io_schedule
wait_on_page_bit
f2fs_wait_on_page_writeback -> It is waiting for node
			node page writeback whose bio is in the
			plug list of CP thread below.
f2fs_update_data_blkaddr
f2fs_outplace_write_data
f2fs_do_write_data_page
__write_data_page
__f2fs_write_data_pages
f2fs_write_data_pages
do_writepages

CP thread -
-----------

__f2fs_write_data_pages -> It is for the same inode above that is under WB (which
	is waiting for node page writeback). In this context, there is nothing to
	be written as the data is already under WB. 
filemap_fdatawrite
f2fs_sync_dirty_inodes -> It just loops here in f2fs_sync_dirty_inodes() until
			f2fs_remove_dirty_inode() has been done by the WB thread above.
block_operations
f2fs_write_checkpoint

The CP thread somehow has the node page bio in its plug list that cannot be submitted 
until end of block_operations() and CP thread is blocked on WB of an inode who is again
waiting for io pending in CP plug list. Both the stacks are stuck on for each other.

The below patch helped to solve the issue, please review and suggest if this seems to 
be okay. Since anyways we are doing cond_resched(), I thought it will be good to flush
the plug list as well (in this issue case, it will loop for the same inode again and again).

diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
index e460d90..152df48 100644
--- a/fs/f2fs/checkpoint.c
+++ b/fs/f2fs/checkpoint.c
@@ -1071,10 +1071,12 @@ int f2fs_sync_dirty_inodes(struct f2fs_sb_info *sbi, enum inode_type type)

                iput(inode);
                /* We need to give cpu to another writers. */
-               if (ino == cur_ino)
+               if (ino == cur_ino) {
+                       blk_flush_plug(current);
                        cond_resched();
-               else
+                } else {
                        ino = cur_ino;
+                }
        } else {
                /*
                 * We should submit bio, since it exists several

Thanks,

-- 
--
Sent by a consultant of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel