Re: IO less throttling and cgroup aware writeback (Was: Re: [Lsf] Preliminary Agenda and Activities for LSF)

Vivek Goyal
2011-03-31T22:26:30Z

On Thu, Mar 31, 2011 at 10:34:03AM -0400, Chris Mason wrote:

[..]
> >
> > For me knowing this is also important in another context of absolute IO
> > throttling.
> >
> > - If a fsync is in progress and gets throttled at device, what impact it
> > has on other file system operations. What gets serialized behind it.
>
> It depends. atime updates log inodes and logging needs a transaction
> and transactions sometimes need to wait for the last transaction to
> finish. So its very possible you'll make anything using the FS appear
> to stop.

I think I have run into this. I created a cgroup and gave ridiculously
low limit of 1bytes/sec and did a fsync. This process blocks. Later
I did "ls" in the directory where fsync process is blocked and ls
also hangs. Following is backtrace. Looks like atime update led to
some kind of blocking in do_get_write_access().

ls D ffffffff8160b060 0 5936 5192 0x00000000
ffff880138729c48 0000000000000086 0000000000000000 0000000100000010
0000000000000000 ffff88013fc40100 ffff88013ac7ac00 000000012e5d70f3
ffff8801353d7af8 ffff880138729fd8 000000000000f558 ffff8801353d7af8
Call Trace:
[<ffffffffa035b0dd>] do_get_write_access+0x29d/0x500 [jbd2]
[<ffffffff8108e150>] ? wake_bit_function+0x0/0x50
[<ffffffffa035b491>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
[<ffffffffa03a7328>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
[<ffffffffa0383843>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
[<ffffffffa037c618>] ? call_filldir+0x78/0xe0 [ext4]
[<ffffffffa03838bc>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
[<ffffffff81041594>] ? __do_page_fault+0x1e4/0x480
[<ffffffffa0383bb0>] ext4_dirty_inode+0x40/0x60 [ext4]
[<ffffffff8119a21b>] __mark_inode_dirty+0x3b/0x160
[<ffffffff8118acad>] touch_atime+0x12d/0x170
[<ffffffff81184c00>] ? filldir+0x0/0xe0
[<ffffffff81184e96>] vfs_readdir+0xd6/0xe0
[<ffffffff81185009>] sys_getdents+0x89/0xf0
[<ffffffff814dc635>] ? page_fault+0x25/0x30
[<ffffffff8100b172>] system_call_fastpath+0x16/0x1b

The vim process doing fsync trace is as follows. This is waiting for
some IO to finish which has been throttled at the device.

vim D ffffffff8110d1f0 0 5934 4452 0x00000000
ffff880107e2dcc8 0000000000000086 0000000100000000 0000000000000003
ffff8801351f4538 0000000000000000 ffff880107e2dc68 ffffffff810e7da2
ffff8801353d70b8 ffff880107e2dfd8 000000000000f558 ffff8801353d70b8
Call Trace:
[<ffffffff810e7da2>] ? ring_buffer_lock_reserve+0xa2/0x160
[<ffffffff81098cb9>] ? ktime_get_ts+0xa9/0xe0
[<ffffffff8110d1f0>] ? sync_page+0x0/0x50
[<ffffffff814da123>] io_schedule+0x73/0xc0
[<ffffffff8110d22d>] sync_page+0x3d/0x50
[<ffffffff814da98f>] __wait_on_bit+0x5f/0x90
[<ffffffff8110d3e3>] wait_on_page_bit+0x73/0x80
[<ffffffff8108e150>] ? wake_bit_function+0x0/0x50
[<ffffffff81123195>] ? pagevec_lookup_tag+0x25/0x40
[<ffffffff8110d7fb>] wait_on_page_writeback_range+0xfb/0x190
[<ffffffff81122341>] ? do_writepages+0x21/0x40
[<ffffffff8110d94b>] ? __filemap_fdatawrite_range+0x5b/0x60
[<ffffffff8110d9c8>] filemap_write_and_wait_range+0x78/0x90
[<ffffffff8119f5fe>] vfs_fsync_range+0x7e/0xe0
[<ffffffff8119f6cd>] vfs_fsync+0x1d/0x20
[<ffffffff8119f70e>] do_fsync+0x3e/0x60
[<ffffffff8119f760>] sys_fsync+0x10/0x20
[<ffffffff8100b172>] system_call_fastpath+0x16/0x1b

Thanks
Vivek