From mboxrd@z Thu Jan 1 00:00:00 1970 From: Bart Van Assche Subject: Re: split scsi passthrough fields out of struct request V2 Date: Thu, 26 Jan 2017 23:47:33 +0000 Message-ID: <1485474426.2540.25.camel@sandisk.com> References: <1485365126-23210-1-git-send-email-hch@lst.de> <1485455329.2540.7.camel@sandisk.com> <1485456745.2540.9.camel@sandisk.com> <20170126185924.GA25289@lst.de> <71e22257-0592-fdd3-25e5-a78ceced2ab9@sandisk.com> <4054e944-b28d-1cd6-574f-6cd90e28c301@fb.com> <1485464486.2540.12.camel@sandisk.com> <6995c991-65a4-8dca-c36e-fb2eff277ca9@fb.com> <1485467235.2540.14.camel@sandisk.com> <1485472465.2540.19.camel@sandisk.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Content-Language: en-US Content-ID: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com To: "hch@lst.de" , "axboe@fb.com" Cc: "linux-block@vger.kernel.org" , "linux-scsi@vger.kernel.org" , "snitzer@redhat.com" , "linux-raid@vger.kernel.org" , "dm-devel@redhat.com" , "j-nomura@ce.jp.nec.com" List-Id: dm-devel.ids On Thu, 2017-01-26 at 16:26 -0700, Jens Axboe wrote: > What device is stuck? Is it running with an mq scheduler attached, or > with "none"? > > Would also be great to see the output of /sys/block/*/mq/*/tags and > sched_tags so we can see if they have anything pending. > > From a quick look at the below, it looks like a request leak. Bisection > would most likely be very helpful. Hello Jens, This happens with and without scheduler attached. The most recent test I ran was with the deadline scheduler configured as default scheduler for all blk-mq devices (CONFIG_DEFAULT_SQ_IOSCHED="mq-deadline" and CONFIG_DEFAULT_MQ_IOSCHED="mq-deadline"). The block devices that hang are /dev/dm-0 and /dev/dm-1. The tags and sched_tags data is as follows: # (cd /sys/class/block && grep -aH '' dm*/mq/*/tags) dm-0/mq/0/tags:nr_tags=2048, reserved_tags=0, bits_per_word=64 dm-0/mq/0/tags:nr_free=1795, nr_reserved=0 dm-0/mq/0/tags:active_queues=0 dm-1/mq/0/tags:nr_tags=2048, reserved_tags=0, bits_per_word=64 dm-1/mq/0/tags:nr_free=2047, nr_reserved=0 dm-1/mq/0/tags:active_queues=0 # (cd /sys/class/block && grep -aH '' dm*/mq/*/sched_tags) dm-0/mq/0/sched_tags:nr_tags=256, reserved_tags=0, bits_per_word=64 dm-0/mq/0/sched_tags:nr_free=0, nr_reserved=0 dm-0/mq/0/sched_tags:active_queues=0 dm-1/mq/0/sched_tags:nr_tags=256, reserved_tags=0, bits_per_word=64 dm-1/mq/0/sched_tags:nr_free=254, nr_reserved=0 dm-1/mq/0/sched_tags:active_queues=0 Bart. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-dm3nam03on0068.outbound.protection.outlook.com ([104.47.41.68]:29785 "EHLO NAM03-DM3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752456AbdAZXsW (ORCPT ); Thu, 26 Jan 2017 18:48:22 -0500 From: Bart Van Assche To: "hch@lst.de" , "axboe@fb.com" CC: "linux-scsi@vger.kernel.org" , "linux-raid@vger.kernel.org" , "dm-devel@redhat.com" , "linux-block@vger.kernel.org" , "snitzer@redhat.com" , "j-nomura@ce.jp.nec.com" Subject: Re: [dm-devel] split scsi passthrough fields out of struct request V2 Date: Thu, 26 Jan 2017 23:47:33 +0000 Message-ID: <1485474426.2540.25.camel@sandisk.com> References: <1485365126-23210-1-git-send-email-hch@lst.de> <1485455329.2540.7.camel@sandisk.com> <1485456745.2540.9.camel@sandisk.com> <20170126185924.GA25289@lst.de> <71e22257-0592-fdd3-25e5-a78ceced2ab9@sandisk.com> <4054e944-b28d-1cd6-574f-6cd90e28c301@fb.com> <1485464486.2540.12.camel@sandisk.com> <6995c991-65a4-8dca-c36e-fb2eff277ca9@fb.com> <1485467235.2540.14.camel@sandisk.com> <1485472465.2540.19.camel@sandisk.com> In-Reply-To: Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org List-Id: linux-block@vger.kernel.org On Thu, 2017-01-26 at 16:26 -0700, Jens Axboe wrote: > What device is stuck? Is it running with an mq scheduler attached, or > with "none"? >=20 > Would also be great to see the output of /sys/block/*/mq/*/tags and > sched_tags so we can see if they have anything pending. >=20 > From a quick look at the below, it looks like a request leak. Bisection > would most likely be very helpful. Hello Jens, This happens with and without scheduler attached. The most recent test I ra= n was with the deadline scheduler configured as default scheduler for all blk= -mq devices (CONFIG_DEFAULT_SQ_IOSCHED=3D"mq-deadline" and CONFIG_DEFAULT_MQ_IOSCHED=3D"mq-deadline"). The block devices that hang are /dev/dm-0 and /dev/dm-1. The tags and sched_tags data is as follows: # (cd /sys/class/block && grep -aH '' dm*/mq/*/tags) dm-0/mq/0/tags:nr_tags=3D2048, reserved_tags=3D0, bits_per_word=3D64 dm-0/mq/0/tags:nr_free=3D1795, nr_reserved=3D0 dm-0/mq/0/tags:active_queues=3D0 dm-1/mq/0/tags:nr_tags=3D2048, reserved_tags=3D0, bits_per_word=3D64 dm-1/mq/0/tags:nr_free=3D2047, nr_reserved=3D0 dm-1/mq/0/tags:active_queues=3D0 # (cd /sys/class/block && grep -aH '' dm*/mq/*/sched_tags) dm-0/mq/0/sched_tags:nr_tags=3D256, reserved_tags=3D0, bits_per_word=3D64 dm-0/mq/0/sched_tags:nr_free=3D0, nr_reserved=3D0 dm-0/mq/0/sched_tags:active_queues=3D0 dm-1/mq/0/sched_tags:nr_tags=3D256, reserved_tags=3D0, bits_per_word=3D64 dm-1/mq/0/sched_tags:nr_free=3D254, nr_reserved=3D0 dm-1/mq/0/sched_tags:active_queues=3D0 Bart.=