From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bart Van Assche Subject: Re: [RFC PATCH] dm: fix excessive dm-mq context switching Date: Sun, 7 Feb 2016 08:37:42 -0800 Message-ID: <56B772D6.2090403@sandisk.com> References: <20160129233504.GA13661@redhat.com> <56AC79D0.5060104@suse.de> <20160130191238.GA18686@redhat.com> <56AEFF63.7050606@suse.de> <20160203180406.GA11591@redhat.com> <20160203182423.GA12913@redhat.com> <56B2F5BC.1010700@suse.de> <20160204135420.GA18227@redhat.com> <20160205151334.GA82754@redhat.com> <20160205180515.GA25808@redhat.com> <20160205191909.GA25982@redhat.com> <56B7659C.8040601@dev.mellanox.co.il> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <56B7659C.8040601@dev.mellanox.co.il> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: Sagi Grimberg , Mike Snitzer , "axboe@kernel.dk" , Hannes Reinecke , Christoph Hellwig Cc: "keith.busch@intel.com" , "linux-block@vger.kernel.org" , device-mapper development , "linux-nvme@lists.infradead.org" List-Id: dm-devel.ids On 02/07/16 07:41, Sagi Grimberg wrote: > >> FYI, here is the latest revised patch: >> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.6&id=a5b835282422ec41991c1dbdb88daa4af7d166d2 >> >> (revised patch header and fixed a thinko in the dm.c:rq_completed() >> change from the RFC patch I posted earlier) > > Hi Mike, > > So I gave your patches a go (dm-4.6) but I still don't see the > improvement you reported (while I do see a minor improvement). > > null_blk queue_mode=2 submit_queues=24 > dm_mod blk_mq_nr_hw_queues=24 blk_mq_queue_depth=4096 use_blk_mq=Y > > I see 620K IOPs on dm_mq vs. 1750K IOPs on raw nullb0. > > Is there something I'm missing? Hello Sagi, Did you run your test on a NUMA system ? If so, can you check with e.g. perf record -ags -e LLC-load-misses sleep 10 && perf report whether this workload triggers perhaps lock contention ? What you need to look for in the perf output is whether any functions occupy more than 10% CPU time. Bart. From mboxrd@z Thu Jan 1 00:00:00 1970 From: bart.vanassche@sandisk.com (Bart Van Assche) Date: Sun, 7 Feb 2016 08:37:42 -0800 Subject: [RFC PATCH] dm: fix excessive dm-mq context switching In-Reply-To: <56B7659C.8040601@dev.mellanox.co.il> References: <20160129233504.GA13661@redhat.com> <56AC79D0.5060104@suse.de> <20160130191238.GA18686@redhat.com> <56AEFF63.7050606@suse.de> <20160203180406.GA11591@redhat.com> <20160203182423.GA12913@redhat.com> <56B2F5BC.1010700@suse.de> <20160204135420.GA18227@redhat.com> <20160205151334.GA82754@redhat.com> <20160205180515.GA25808@redhat.com> <20160205191909.GA25982@redhat.com> <56B7659C.8040601@dev.mellanox.co.il> Message-ID: <56B772D6.2090403@sandisk.com> On 02/07/16 07:41, Sagi Grimberg wrote: > >> FYI, here is the latest revised patch: >> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=dm-4.6&id=a5b835282422ec41991c1dbdb88daa4af7d166d2 >> >> (revised patch header and fixed a thinko in the dm.c:rq_completed() >> change from the RFC patch I posted earlier) > > Hi Mike, > > So I gave your patches a go (dm-4.6) but I still don't see the > improvement you reported (while I do see a minor improvement). > > null_blk queue_mode=2 submit_queues=24 > dm_mod blk_mq_nr_hw_queues=24 blk_mq_queue_depth=4096 use_blk_mq=Y > > I see 620K IOPs on dm_mq vs. 1750K IOPs on raw nullb0. > > Is there something I'm missing? Hello Sagi, Did you run your test on a NUMA system ? If so, can you check with e.g. perf record -ags -e LLC-load-misses sleep 10 && perf report whether this workload triggers perhaps lock contention ? What you need to look for in the perf output is whether any functions occupy more than 10% CPU time. Bart.