From: Jaegeuk Kim <jaegeuk@kernel.org>
To: Christoph Hellwig <hch@infradead.org>
Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org,
linux-f2fs-devel@lists.sourceforge.net, axboe@kernel.dk
Subject: Re: [PATCH 5/6] f2fs: add a kernel thread to issue discard commands asynchronously
Date: Mon, 6 Feb 2017 19:44:03 -0800 [thread overview]
Message-ID: <20170207034403.GA24935@jaegeuk.local> (raw)
In-Reply-To: <20170205085947.GA25319@infradead.org>
On 02/05, Christoph Hellwig wrote:
> On Mon, Jan 16, 2017 at 09:32:20AM -0800, Christoph Hellwig wrote:
> > On Fri, Jan 13, 2017 at 11:12:11AM -0800, Jaegeuk Kim wrote:
> > > Previously, I've done to issue discard bios asynchronously. But the problem that
> > > I've got is that was not enough. When testing nvme SSD with noop IO scheduler,
> > > submit_bio() was blocked at every 8 async discard bios, resulting in very slow
> > > checkpoint process which blocks most of other FS operations.
> >
> > Where does it block? Are you running out of request? What driver is
> > this on top of?
>
> Ping? I'm currently spending a lot of effort on fs and block dіscard
> code, and I'd like to make sure we get common infrastructure instead
> of local hacks.
Sorry for the late response due to the travel.
When doing fstrim with a fresh f2fs image fomatted on Intel NVMe SSD whose
model name is SSDPE2MW012T4, I've got the following trace.
...
fstrim-12620 [000] .... 334572.907534: f2fs_issue_discard: dev = (259,1), blkstart = 0x902900, blklen = 0x400
fstrim-12620 [000] .... 334572.907535: block_bio_remap: 259,0 D 75583488 + 8192 <- (259,1) 75581440
fstrim-12620 [000] .... 334572.907535: block_bio_queue: 259,0 D 75583488 + 8192 [fstrim]
fstrim-12620 [000] .... 334572.907535: block_getrq: 259,0 D 75583488 + 8192 [fstrim]
fstrim-12620 [000] .... 334572.907536: block_unplug: [fstrim] 1
fstrim-12620 [000] .... 334572.907536: block_rq_insert: 259,0 D 0 () 75583488 + 8192 [fstrim]
fstrim-12620 [000] .... 334572.907536: block_rq_issue: 259,0 D 0 () 75583488 + 8192 [fstrim]
< repeat 6 times >
fstrim-12620 [000] .... 334572.907620: f2fs_issue_discard: dev = (259,1), blkstart = 0x904500, blklen = 0x400
fstrim-12620 [000] .... 334572.907620: block_bio_remap: 259,0 D 75640832 + 8192 <- (259,1) 75638784
fstrim-12620 [000] .... 334572.907620: block_bio_queue: 259,0 D 75640832 + 8192 [fstrim]
fstrim-12620 [000] .... 334572.907621: block_getrq: 259,0 D 75640832 + 8192 [fstrim]
<idle>-0 [000] d.h. 334572.907723: block_rq_complete: 259,0 D () 67260416 + 8192 [0]
<idle>-0 [000] d.h. 334572.907942: block_rq_complete: 259,0 D () 67268608 + 8192 [0]
<idle>-0 [000] d.h. 334572.908155: block_rq_complete: 259,0 D () 67276800 + 8192 [0]
<idle>-0 [000] d.h. 334572.908374: block_rq_complete: 259,0 D () 67284992 + 8192 [0]
<idle>-0 [000] d.h. 334572.908597: block_rq_complete: 259,0 D () 67293184 + 8192 [0]
<idle>-0 [000] d.h. 334572.908823: block_rq_complete: 259,0 D () 67301376 + 8192 [0]
<idle>-0 [000] d.h. 334572.909033: block_rq_complete: 259,0 D () 67309568 + 8192 [0]
<idle>-0 [000] d.h. 334572.909216: block_rq_complete: 259,0 D () 67317760 + 8192 [0]
fstrim-12620 [000] .... 334572.909222: block_unplug: [fstrim] 1
fstrim-12620 [000] .... 334572.909223: block_rq_insert: 259,0 D 0 () 75640832 + 8192 [fstrim]
fstrim-12620 [000] .... 334572.909224: block_rq_issue: 259,0 D 0 () 75640832 + 8192 [fstrim]
fstrim-12620 [000] .... 334572.909240: f2fs_issue_discard: dev = (259,1), blkstart = 0x904900, blklen = 0x400
fstrim-12620 [000] .... 334572.909241: block_bio_remap: 259,0 D 75649024 + 8192 <- (259,1) 75646976
fstrim-12620 [000] .... 334572.909241: block_bio_queue: 259,0 D 75649024 + 8192 [fstrim]
fstrim-12620 [000] .... 334572.909241: block_getrq: 259,0 D 75649024 + 8192 [fstrim]
fstrim-12620 [000] .... 334572.909242: block_unplug: [fstrim] 1
fstrim-12620 [000] .... 334572.909242: block_rq_insert: 259,0 D 0 () 75649024 + 8192 [fstrim]
fstrim-12620 [000] .... 334572.909242: block_rq_issue: 259,0 D 0 () 75649024 + 8192 [fstrim]
< repeat >
So, I investigated why block_rq_complete() happened in more detail.
The root-caused call path looks like:
- submit_bio
- generic_make_request
- q->make_request_fn
- blk_mq_make_request
- blk_mq_map_request
- blk_mq_alloc_request
- blk_mq_get_tag
- __blk_mq_get_tag
- bt_get
- blk_mq_run_hw_queue
- finish_wait
--> this waits for pending 8 discard bios!
It seems the problem comes from the storage processing discard commands too
slowly comparing to normal read/write IOs.
Any thoughts?
Thanks,
next prev parent reply other threads:[~2017-02-07 3:44 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-12 22:44 [PATCH 1/6] f2fs: clean up flush/discard command namings Jaegeuk Kim
2017-01-12 22:44 ` [PATCH 2/6] f2fs: reorganize stat information Jaegeuk Kim
2017-02-22 9:40 ` Chao Yu
2017-01-12 22:44 ` [PATCH 3/6] f2fs: remove batched discard in f2fs_trim_fs Jaegeuk Kim
2017-02-22 9:40 ` Chao Yu
2017-02-22 21:55 ` Jaegeuk Kim
2017-02-23 2:19 ` Chao Yu
2017-02-23 4:25 ` Jaegeuk Kim
2017-01-12 22:44 ` [PATCH 4/6] f2fs: factor out discard command info into discard_cmd_control Jaegeuk Kim
2017-02-22 9:40 ` Chao Yu
2017-01-12 22:44 ` [PATCH 5/6] f2fs: add a kernel thread to issue discard commands asynchronously Jaegeuk Kim
2017-01-13 8:01 ` Christoph Hellwig
2017-01-13 19:12 ` Jaegeuk Kim
2017-01-16 17:32 ` Christoph Hellwig
2017-02-05 8:59 ` Christoph Hellwig
2017-02-07 3:44 ` Jaegeuk Kim [this message]
2017-02-08 16:02 ` Christoph Hellwig
2017-02-08 22:05 ` Jaegeuk Kim
2017-02-22 9:40 ` Chao Yu
2017-01-12 22:44 ` [PATCH 6/6] f2fs: show # of on-going flush and discard bios Jaegeuk Kim
2017-01-14 2:26 ` heyunlei
2017-01-14 7:27 ` [f2fs-dev] " heyunlei
2017-01-14 23:32 ` [PATCH 6/6 v2] " Jaegeuk Kim
2017-02-22 9:40 ` [PATCH 6/6] " Chao Yu
2017-02-22 9:40 ` [PATCH 1/6] f2fs: clean up flush/discard command namings Chao Yu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170207034403.GA24935@jaegeuk.local \
--to=jaegeuk@kernel.org \
--cc=axboe@kernel.dk \
--cc=hch@infradead.org \
--cc=linux-f2fs-devel@lists.sourceforge.net \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).