From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoph Hellwig Subject: Re: [PATCH 5/6] f2fs: add a kernel thread to issue discard commands asynchronously Date: Wed, 8 Feb 2017 08:02:07 -0800 Message-ID: <20170208160207.GA14621@infradead.org> References: <20170112224407.54026-1-jaegeuk@kernel.org> <20170112224407.54026-5-jaegeuk@kernel.org> <20170113080159.GB31777@infradead.org> <20170113191211.GB1301@jaegeuk.local> <20170116173220.GA2199@infradead.org> <20170205085947.GA25319@infradead.org> <20170207034403.GA24935@jaegeuk.local> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from sog-mx-4.v43.ch3.sourceforge.com ([172.29.43.194] helo=mx.sourceforge.net) by sfs-ml-2.v29.ch3.sourceforge.com with esmtp (Exim 4.76) (envelope-from ) id 1cbUhI-0000D1-49 for linux-f2fs-devel@lists.sourceforge.net; Wed, 08 Feb 2017 16:02:20 +0000 Received: from bombadil.infradead.org ([65.50.211.133]) by sog-mx-4.v43.ch3.sourceforge.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.76) id 1cbUhF-0000ii-L4 for linux-f2fs-devel@lists.sourceforge.net; Wed, 08 Feb 2017 16:02:20 +0000 Content-Disposition: inline In-Reply-To: <20170207034403.GA24935@jaegeuk.local> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-f2fs-devel-bounces@lists.sourceforge.net To: Jaegeuk Kim Cc: linux-fsdevel@vger.kernel.org, axboe@kernel.dk, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-f2fs-devel@lists.sourceforge.net T24gTW9uLCBGZWIgMDYsIDIwMTcgYXQgMDc6NDQ6MDNQTSAtMDgwMCwgSmFlZ2V1ayBLaW0gd3Jv dGU6Cj4gU29ycnkgZm9yIHRoZSBsYXRlIHJlc3BvbnNlIGR1ZSB0byB0aGUgdHJhdmVsLgo+IAo+ IFdoZW4gZG9pbmcgZnN0cmltIHdpdGggYSBmcmVzaCBmMmZzIGltYWdlIGZvbWF0dGVkIG9uIElu dGVsIE5WTWUgU1NEIHdob3NlCj4gbW9kZWwgbmFtZSBpcyBTU0RQRTJNVzAxMlQ0LCBJJ3ZlIGdv dCB0aGUgZm9sbG93aW5nIHRyYWNlLgoKPHNuaXA+Cgo+IFNvLCBJIGludmVzdGlnYXRlZCB3aHkg YmxvY2tfcnFfY29tcGxldGUoKSBoYXBwZW5lZCBpbiBtb3JlIGRldGFpbC4KPiAKPiBUaGUgcm9v dC1jYXVzZWQgY2FsbCBwYXRoIGxvb2tzIGxpa2U6Cj4gIC0gc3VibWl0X2Jpbwo+ICAgLSBnZW5l cmljX21ha2VfcmVxdWVzdAo+ICAgIC0gcS0+bWFrZV9yZXF1ZXN0X2ZuCj4gICAgIC0gYmxrX21x X21ha2VfcmVxdWVzdAo+ICAgICAgLSBibGtfbXFfbWFwX3JlcXVlc3QKPiAgICAgICAtIGJsa19t cV9hbGxvY19yZXF1ZXN0Cj4gICAgICAgIC0gYmxrX21xX2dldF90YWcKPiAgICAgICAgIC0gX19i bGtfbXFfZ2V0X3RhZwo+ICAgICAgICAgIC0gYnRfZ2V0Cj4gICAgICAgICAgIC0gYmxrX21xX3J1 bl9od19xdWV1ZQo+ICAgICAgICAgICAtIGZpbmlzaF93YWl0Cj4gICAgICAgICAgIC0tPiB0aGlz IHdhaXRzIGZvciBwZW5kaW5nIDggZGlzY2FyZCBiaW9zIQoKWW91J3JlIGJsb2NraW5nIG9uIHRh ZyBhbGxvY2F0aW9uLiAgSG93IG1hbnkgdGFncyBwZXIgcXVldWUgZG9lcwp5b3VyIGRldmljZSBo YXZlPywgZS5nLiBkbyBhCgpjYXQgL3N5cy9ibG9jay9udm1lMG4xL21xLzAvbnJfdGFncwoKPiBJ dCBzZWVtcyB0aGUgcHJvYmxlbSBjb21lcyBmcm9tIHRoZSBzdG9yYWdlIHByb2Nlc3NpbmcgZGlz Y2FyZCBjb21tYW5kcyB0b28KPiBzbG93bHkgY29tcGFyaW5nIHRvIG5vcm1hbCByZWFkL3dyaXRl IElPcy4KPiAKPiBBbnkgdGhvdWdodHM/CgpEZWFsbG9jYXRlIGlzIGFsd2F5cyBnb2luZyB0byBi ZSBhbiBleGNlcHRpb24gcGF0aCBjb21wYXJlZCB0byBub3JtYWwKcmVhZC93cml0ZeKApiBidXQg anVzdCBob3cgbXVjaCBzbG93ZXIgaXMgZ29pbmcgdG8gYmUgZGV2aWNlCmRlcGVuZGVudC4KCk9u ZSBvcHRpb24gd291bGQgYmUgdG8gcmV1c2UgdGhlIG51bWJlciBvZiBkaXNjYXJkcywgZm9yIHRo YXQgY2FuIHlvdQp0cnkgdGhlIHNlcmllcyBoZXJlIHRvIHN1cHBvcnQgdmVjdG9yZWQgZGlzY2Fy ZHM6CgpodHRwOi8vZ2l0LmluZnJhZGVhZC5vcmcvdXNlcnMvaGNoL2Jsb2NrLmdpdC9zaG9ydGxv Zy9yZWZzL2hlYWRzL3ZlY3RvcmVkLWRpc2NhcmQtZm9yLWF4Ym9lCgotLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0KQ2hlY2sgb3V0IHRoZSB2aWJyYW50IHRlY2ggY29tbXVuaXR5IG9uIG9uZSBvZiB0aGUg d29ybGQncyBtb3N0CmVuZ2FnaW5nIHRlY2ggc2l0ZXMsIFNsYXNoRG90Lm9yZyEgaHR0cDovL3Nk bS5saW5rL3NsYXNoZG90Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fCkxpbnV4LWYyZnMtZGV2ZWwgbWFpbGluZyBsaXN0CkxpbnV4LWYyZnMtZGV2ZWxAbGlz dHMuc291cmNlZm9yZ2UubmV0Cmh0dHBzOi8vbGlzdHMuc291cmNlZm9yZ2UubmV0L2xpc3RzL2xp c3RpbmZvL2xpbnV4LWYyZnMtZGV2ZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 From: hch@infradead.org (Christoph Hellwig) Date: Wed, 8 Feb 2017 08:02:07 -0800 Subject: [PATCH 5/6] f2fs: add a kernel thread to issue discard commands asynchronously In-Reply-To: <20170207034403.GA24935@jaegeuk.local> References: <20170112224407.54026-1-jaegeuk@kernel.org> <20170112224407.54026-5-jaegeuk@kernel.org> <20170113080159.GB31777@infradead.org> <20170113191211.GB1301@jaegeuk.local> <20170116173220.GA2199@infradead.org> <20170205085947.GA25319@infradead.org> <20170207034403.GA24935@jaegeuk.local> Message-ID: <20170208160207.GA14621@infradead.org> On Mon, Feb 06, 2017@07:44:03PM -0800, Jaegeuk Kim wrote: > Sorry for the late response due to the travel. > > When doing fstrim with a fresh f2fs image fomatted on Intel NVMe SSD whose > model name is SSDPE2MW012T4, I've got the following trace. > So, I investigated why block_rq_complete() happened in more detail. > > The root-caused call path looks like: > - submit_bio > - generic_make_request > - q->make_request_fn > - blk_mq_make_request > - blk_mq_map_request > - blk_mq_alloc_request > - blk_mq_get_tag > - __blk_mq_get_tag > - bt_get > - blk_mq_run_hw_queue > - finish_wait > --> this waits for pending 8 discard bios! You're blocking on tag allocation. How many tags per queue does your device have?, e.g. do a cat /sys/block/nvme0n1/mq/0/nr_tags > It seems the problem comes from the storage processing discard commands too > slowly comparing to normal read/write IOs. > > Any thoughts? Deallocate is always going to be an exception path compared to normal read/write? but just how much slower is going to be device dependent. One option would be to reuse the number of discards, for that can you try the series here to support vectored discards: http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/vectored-discard-for-axboe From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bombadil.infradead.org ([65.50.211.133]:60776 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751060AbdBHQlB (ORCPT ); Wed, 8 Feb 2017 11:41:01 -0500 Date: Wed, 8 Feb 2017 08:02:07 -0800 From: Christoph Hellwig To: Jaegeuk Kim Cc: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-f2fs-devel@lists.sourceforge.net, axboe@kernel.dk, linux-nvme@lists.infradead.org Subject: Re: [PATCH 5/6] f2fs: add a kernel thread to issue discard commands asynchronously Message-ID: <20170208160207.GA14621@infradead.org> References: <20170112224407.54026-1-jaegeuk@kernel.org> <20170112224407.54026-5-jaegeuk@kernel.org> <20170113080159.GB31777@infradead.org> <20170113191211.GB1301@jaegeuk.local> <20170116173220.GA2199@infradead.org> <20170205085947.GA25319@infradead.org> <20170207034403.GA24935@jaegeuk.local> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20170207034403.GA24935@jaegeuk.local> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Mon, Feb 06, 2017 at 07:44:03PM -0800, Jaegeuk Kim wrote: > Sorry for the late response due to the travel. > > When doing fstrim with a fresh f2fs image fomatted on Intel NVMe SSD whose > model name is SSDPE2MW012T4, I've got the following trace. > So, I investigated why block_rq_complete() happened in more detail. > > The root-caused call path looks like: > - submit_bio > - generic_make_request > - q->make_request_fn > - blk_mq_make_request > - blk_mq_map_request > - blk_mq_alloc_request > - blk_mq_get_tag > - __blk_mq_get_tag > - bt_get > - blk_mq_run_hw_queue > - finish_wait > --> this waits for pending 8 discard bios! You're blocking on tag allocation. How many tags per queue does your device have?, e.g. do a cat /sys/block/nvme0n1/mq/0/nr_tags > It seems the problem comes from the storage processing discard commands too > slowly comparing to normal read/write IOs. > > Any thoughts? Deallocate is always going to be an exception path compared to normal read/write… but just how much slower is going to be device dependent. One option would be to reuse the number of discards, for that can you try the series here to support vectored discards: http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/vectored-discard-for-axboe