From: Jan Kara <jack@suse.cz>
To: Liu Bo <bo.liu@linux.alibaba.com>
Cc: linux-ext4@vger.kernel.org, fengguang.wu@intel.com,
tj@kernel.org, jack@suse.cz, cgroups@vger.kernel.org,
gthelen@google.com, linux-mm@kvack.org,
yang.shi@linux.alibaba.com
Subject: Re: ext4 hang and per-memcg dirty throttling
Date: Wed, 12 Sep 2018 14:11:30 +0200 [thread overview]
Message-ID: <20180912121130.GF7782@quack2.suse.cz> (raw)
In-Reply-To: <20180912001054.bu3x3xwukusnsa26@US-160370MP2.local>
Hi!
On Tue 11-09-18 17:10:55, Liu Bo wrote:
> With ext4's data=ordered mode and the underlying blk throttle setting, we
> can easily run to hang,
>
> 1.
> mount /dev/sdc /mnt -odata=ordered
> 2.
> mkdir /sys/fs/cgroup/unified/cg
> 3.
> echo "+io" > /sys/fs/cgroup/unified/cgroup.subtree_control
> 4.
> echo "`cat /sys/block/sdc/dev` wbps=$((1 << 20))" > /sys/fs/cgroup/unified/cg/io.max
> 5.
> echo $$ > /sys/fs/cgroup/unified/cg/cgroup.procs
> 6.
> // background dirtier
> xfs_io -f -c "pwrite 0 1G" $M/dummy &
> 7.
> echo $$ > /sys/fs/cgroup/unified/cgroup.procs
> 8.
> // issue synchronous IO
> for i in `seq 1 100`;
> do
> xfs_io -f -s -c "pwrite 0 4k" $M/foo > /dev/null
> done
>
>
> And the hang is like
>
> [jbd2-sdc]
> jbd2_journal_commit_transaction
> journal_submit_data_buffers
> # file 'dummy' has been written by writeback kthread
> journal_finish_inode_data_buffers
> # wait on page's writeback
Yes, I guess you're speaking about the one Chris Mason mentioned [1].
Essentially it's a priority inversion where jbd2 thread gets blocked behind
writeback done on behalf of a heavily restricted process. It actually is
not related to dirty throttling or anything like that. And the solution for
this priority inversion is to use unwritten extents for writeback
unconditionally as I wrote in that thread. The core of this is implemented
and hidden behind dioread_nolock mount option but it needs some serious
polishing work and testing...
[1] https://marc.info/?l=linux-fsdevel&m=151688776319077
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
next prev parent reply other threads:[~2018-09-12 12:11 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-12 0:10 ext4 hang and per-memcg dirty throttling Liu Bo
2018-09-12 12:11 ` Jan Kara [this message]
2018-09-12 15:07 ` Theodore Y. Ts'o
2018-09-12 19:22 ` Liu Bo
2018-09-12 19:19 ` Liu Bo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180912121130.GF7782@quack2.suse.cz \
--to=jack@suse.cz \
--cc=bo.liu@linux.alibaba.com \
--cc=cgroups@vger.kernel.org \
--cc=fengguang.wu@intel.com \
--cc=gthelen@google.com \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=tj@kernel.org \
--cc=yang.shi@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).