From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from esa1.hgst.iphmx.com ([68.232.141.245]:65455 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751116AbdCQRe2 (ORCPT ); Fri, 17 Mar 2017 13:34:28 -0400 From: Bart Van Assche To: "linux-kernel@vger.kernel.org" , "hch@infradead.org" , "linux-block@vger.kernel.org" , "tom.leiming@gmail.com" , "axboe@fb.com" CC: "yizhan@redhat.com" , "tj@kernel.org" Subject: Re: [PATCH v1 3/3] blk-mq: start to freeze queue just after setting dying Date: Fri, 17 Mar 2017 17:32:09 +0000 Message-ID: <1489771915.2826.4.camel@sandisk.com> References: <20170317095711.5819-1-tom.leiming@gmail.com> <20170317095711.5819-4-tom.leiming@gmail.com> In-Reply-To: <20170317095711.5819-4-tom.leiming@gmail.com> Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org List-Id: linux-block@vger.kernel.org On Fri, 2017-03-17 at 17:57 +0800, Ming Lei wrote: > Before commit 780db2071a(blk-mq: decouble blk-mq freezing > from generic bypassing), the dying flag is checked before > entering queue, and Tejun converts the checking into .mq_freeze_depth, > and assumes the counter is increased just after dying flag > is set. Unfortunately we doesn't do that in blk_set_queue_dying(). >=20 > This patch calls blk_mq_freeze_queue_start() for blk-mq in > blk_set_queue_dying(), so that we can block new I/O coming > once the queue is set as dying. >=20 > Given blk_set_queue_dying() is always called in remove path > of block device, and queue will be cleaned up later, we don't > need to worry about undoing the counter. >=20 > Cc: Bart Van Assche > Cc: Tejun Heo > Signed-off-by: Ming Lei > --- > block/blk-core.c | 7 +++++-- > 1 file changed, 5 insertions(+), 2 deletions(-) >=20 > diff --git a/block/blk-core.c b/block/blk-core.c > index d772c221cc17..62d4967c369f 100644 > --- a/block/blk-core.c > +++ b/block/blk-core.c > @@ -500,9 +500,12 @@ void blk_set_queue_dying(struct request_queue *q) > queue_flag_set(QUEUE_FLAG_DYING, q); > spin_unlock_irq(q->queue_lock); > =20 > - if (q->mq_ops) > + if (q->mq_ops) { > blk_mq_wake_waiters(q); > - else { > + > + /* block new I/O coming */ > + blk_mq_freeze_queue_start(q); > + } else { > struct request_list *rl; > =20 > spin_lock_irq(q->queue_lock); Hello Ming, I think we need the queue freezing not only for blk-mq but also for blk-sq. Since the queue flags and the mq_freeze_depth are stored in different variables we need to prevent that the CPU reorders the stores to these variables. The comment about=A0blk_mq_freeze_queue_start() should be more clear. How about something like the patch below? [PATCH] blk-mq: Force block layer users to check the "dying" flag=A0after i= t has been set Commit 780db2071ac4 removed the blk_queue_dying() check from the hot path of blk_mq_queue_enter() although that check is necessary when cleaning up a queue. Hence make sure that blk_queue_enter() and blk_mq_queue_enter() check the dying flag after it has been set. Because blk_set_queue_dying() is only called from the remove path of a block device we don't need to worry about unfreezing the queue. Fixes: commit 780db2071ac4 ("blk-mq: decouble blk-mq freezing from generic = bypassing") --- =A0block/blk-core.c | 13 +++++++++++++ =A01 file changed, 13 insertions(+) diff --git a/block/blk-core.c b/block/blk-core.c index d772c221cc17..730f715b72ff 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -500,6 +500,19 @@ void blk_set_queue_dying(struct request_queue *q) =A0 queue_flag_set(QUEUE_FLAG_DYING, q); =A0 spin_unlock_irq(q->queue_lock); =A0 + /* + =A0* Avoid that the updates of the queue flags and q_usage_counter + =A0* are reordered. + =A0*/ + smp_wmb(); + + /* + =A0* Force blk_queue_enter() and blk_mq_queue_enter() to check the + =A0* "dying" flag. Despite its name, blk_mq_freeze_queue_start() + =A0* affects blk-sq and blk-mq queues. + =A0*/ + blk_mq_freeze_queue_start(q); + =A0 if (q->mq_ops) =A0 blk_mq_wake_waiters(q); =A0 else { Thanks, Bart. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751332AbdCQReg (ORCPT ); Fri, 17 Mar 2017 13:34:36 -0400 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:65455 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751116AbdCQRe2 (ORCPT ); Fri, 17 Mar 2017 13:34:28 -0400 X-IronPort-AV: E=Sophos;i="5.35,258,1483977600"; d="scan'208";a="106280142" Authentication-Results: spf=pass (sender IP is 74.221.232.54) smtp.mailfrom=sandisk.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=bestguesspass action=none header.from=sandisk.com; X-AuditID: ac1c2133-99bff7000000c960-56-58cc1d9a90d7 From: Bart Van Assche To: "linux-kernel@vger.kernel.org" , "hch@infradead.org" , "linux-block@vger.kernel.org" , "tom.leiming@gmail.com" , "axboe@fb.com" CC: "yizhan@redhat.com" , "tj@kernel.org" Subject: Re: [PATCH v1 3/3] blk-mq: start to freeze queue just after setting dying Thread-Topic: [PATCH v1 3/3] blk-mq: start to freeze queue just after setting dying Thread-Index: AQHSn0RncJFE51k7/EmcSzzCzSQeiw== Date: Fri, 17 Mar 2017 17:32:09 +0000 Message-ID: <1489771915.2826.4.camel@sandisk.com> References: <20170317095711.5819-1-tom.leiming@gmail.com> <20170317095711.5819-4-tom.leiming@gmail.com> In-Reply-To: <20170317095711.5819-4-tom.leiming@gmail.com> Accept-Language: en-US, nl-NL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.28.1.254] Content-Type: text/plain; charset="iso-8859-1" Content-ID: <23C3DBC5A6F43C4595F85C929435AFE2@sandisk.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrPIsWRmVeSWpSXmKPExsWyRobxn+5c2TMRBvNnmFv833OMzeL0hEVM FntvaVtc3jWHzeLX8qOMFu9/XGe32HLlLpsDu8fE5nfsHjtn3WX32LxCy2PTqk42j/f7rrJ5 fN4kF8AWxWWTkpqTWZZapG+XwJVxeL5kwRPJivl3DjI1ME4T7WLk5JAQMJG4sn8KexcjF4eQ wBImiQ1zXrKCJIQEjjFKvNloDWKzCRhJzJ6whwWkSETgL6PE5O4bzCAJZgEfiYnNE5hAbGGB EIm7t/+zgdgiAqES0+9MZIWw9ST2fTnKDmKzCKhK/Dx2CMzmFTCUWL71CdSyLIkna9aBzeEU sJJY+fc3WA2jgKzE4uktTBC7xCVuPZnPBHG1gMSSPeeZIWxRiZeP/7FC2AoSn1f8Y4Oo15O4 MXUKlG0lcf/gN6ibtSWWLXzNDHGDoMTJmU9YJjCKzUKyYhaS9llI2mchaZ+FpH0BI+sqRrHi xOTi3PTUAkMTveLEvJTM4my95PzcTYzgaFU03sH4b4P7IUYBDkYlHt4bT05HCLEmlhVX5h5i lOBgVhLh3S14JkKINyWxsiq1KD++qDQntfgQozQHi5I4b8zsqRFCAumJJanZqakFqUUwWSYO TqkGxh0fjk8ILntQvkdk46x935xubQq7tMqsQPbR74Zk/zpV9SX/69imME3w0FtlotZTc8Ls brTo/uBZtU3hsd7e/K9TXzD6vOH3YfykvfrITPZpz7Uf9/nflChsWJFxZsZkM7Z3XfGGe5fd rLKJTY0yltyuaf21rjBB/P2D9/PnCJx6+fyYi9be10osxRmJhlrMRcWJAOMKO0jSAgAA X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:74.221.232.54;IPV:NLI;CTRY:US;EFV:NLI;SFV:NSPM;SFS:(10019020)(6009001)(39840400002)(39450400003)(39850400002)(39860400002)(39410400002)(2980300002)(438002)(199003)(189002)(24454002)(377424004)(9170700003)(54906002)(2906002)(102836003)(23756003)(356003)(6246003)(229853002)(39060400002)(53936002)(2900100001)(106466001)(5660300001)(2950100002)(33646002)(81166006)(2201001)(5250100002)(86362001)(50466002)(2501003)(189998001)(8746002)(4326008)(36756003)(76176999)(6116002)(3846002)(38730400002)(8676002)(7736002)(8936002)(47776003)(305945005)(54356999)(103116003)(50986999);DIR:OUT;SFP:1102;SCL:1;SRVR:DM2PR04MB830;H:sacsmgep14.sandisk.com;FPR:;SPF:Pass;MLV:ovrnspm;A:1;MX:1;PTR:InfoDomainNonexistent;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;CO1NAM04FT051;1:U0rnwK54eo6podr58TR3y7GIBkoz7ggrQ0Xg1wFPD7AYJOap32tXOVsTMDeMaoTO6D3+0IeDsVQvfs4QMVUjv/SgzYZ4W5n9bqiXgA8gCmS9xT/haVK349NwqolapIwzRNQfmyEBJhRMajMMCl7yXtKDjrIENH8x0jIdykYawKT1fwy3EzRa3Ui/G41M6qz/FeHTcfel8stW3AmgDhfFnO6UrTfuulH1Us5F65eEZtRMSggwyHp32YsHOdxn1CGnQf+rfR0OG8szFm/o5CBdzxQOWsxieNLEh5iVkswlb1D6EukCTl1zgFJ87NewcATufjgnBVQTe6m+ga3UWCozlTls/D9V8NrqWAbbZZ7j1oYApm4Yry3YsrnSBjbS3SLU6AHu8OX1WMiBlfyrCyJkE3WVOBRtvRoavs0bxdWar8FmklvGdgnaLFE9gce6h6JzOYXes4yiv4I1s2qOJseeJN/I4sRKUIUp7+GbgBcyWpYLM9CFtRowGhqoWGSb25X7VhPiouI+QyDn2I3VUiCykA== X-MS-Office365-Filtering-Correlation-Id: 95487912-6610-49e9-e1af-08d46d5b8d4b X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(8251501002)(2017030254051);SRVR:DM2PR04MB830; X-Microsoft-Exchange-Diagnostics: 1;DM2PR04MB830;3:EFbvumfKQntgGRHqxSDNOAEvWoqXJG8tQdCQ5n53BgTBnK3SuN+IPS2dfcqr8p6LDxBAckrqh/nYpEwb1MGFS5ob2vEewMQ5DDnd3zHEOTMeQZ7x3bSseEl87u7ETtpOmFzNZgU1n8cKFF1POCImORk+Qzw78VzLBrynUCn5h/pgy/xjrsVRrTeeLwmZL8yWLZ0zZD4F3yacJs4i3QAdI0cCZZo1N0qlzRBnWj395q2LLjbtKGCuIxPfYn39be3oBMuO2+ddtmJBLQByOyBPyjIFzpWwABE6VoZ9eUp7pJWY2V0ID6UEevZzc11OvyCpCMRESbggNRSZTg4PzQ5A82u/IEp0aVgZVDeZIkjA+BTWgUAUL9CGRCDDk9eTJc3ntqgv8crljT6tpYvkTvT4vfrwHkaj4FHMq4q8fNy2H4s=;25:BGFMENvQzihnaCI/ORkvNfB6TusKG50w+IwzQ3PWV6WlwHPwgMgy04LLt/1cSD5gUwByzdaHbovXyX3GSWgPZqeMxpYsdgxDXf9CSNXGYc+D4Xg/A6VSBVczcUTj1q0mG6AWw2lLVjs4Yt1xMKUBD7xEUuQa9YNd3Tq7DGewO6d/Dba+X91e7yrf85vaG5k/G1W6Lkd4tI2tkNHcAGHUfhoxR4h70lZFNmZyYwf9MK4xyPtlg79nEEiagy68tR9NiAKWO9o5mwyLfMDXdwihcaAYnbv+zSm5vjR5p5uXbAlO9vy4+QQe9ZstWsJjsjR1dCOb7/vgkltrurYbrIG8T+X/1i61qeIL9OBLuZouX/HdMKtLU4m7Ufruoem/edWMxPy+2MYNo1IPpG4c+xPDPMRmun5zSa0qZhauDtGeM9Qb1ETNm3DkvfnGNek7bvBlGcmPYMyORH0F4hmjiSiOsQ== X-Microsoft-Exchange-Diagnostics: 1;DM2PR04MB830;31:3I4n7b6H3UoTncOdrc5SUn6fkdK2fzmKuilBztc+YMo2Y/5bi198H42ikKe3jO6NhOI/APArFheaCFObAH/7DAojzbQjy+scFCjj1FwuLiFgndKOHXVx3UzdKTM7uyb2rKWSS2PVPF1H3+f/oOyYQeYO4HDCUphnUQvgrHoqZTn2BN1zhlo5ZZ4XZdH/rV2RWatSxANr36T/Nrpe9Hh+EVHDQEjZQdC3uPCPwwtwlYyd06fP1S120jTqqploGnedOzPIeWEI7LfiEl1kCWDGxmxQCQxw3DXUMbJAxqy27gs=;20:+XsnTyVA5ybQamnjyeoVOdzT31WO4bU7dbI7szybslbQQJdIRp1ytY70YyYQjPbbcXGRTrYNsYLDCNf24ii/1MDKRTRK+ViWF0KgzzbsdEcklmR5Nh+9OV2j6LRaYUMZznwI2IGkL9Ke3FhGNSR9GJTxnuDhVY3XN8vsKWVwMsZaPWrL7H9W4wvPhnubpzc/s0CH/XVL0G+3kmkcW+GEW8XPdmQlrzNXjMDL0zepEwCnbIc1kMMKFHKUhi6x4ba3KMO0rCr0jW9OgvW+ipGxJo5FCkr/xH7tK3LyUggI287RR0Mygq44cZseJsUcRonHNY7WmscF+dCK8iMOBOYkBHXeGvIP/u1zqL1Dhk3nA8VfUMsVCIC5S4DTBIMhl3EkBh8Jq7c7jti0rvW1sNOC+dQ6g8cM+BmwWLqFJpYpp+SYNy9mfrjMK6qoq+V/dK/3QKnRdQ0n/uUQGSXD+HmbjFqwD6RMvEIktWIXIzbP83Gsrx4brPZK8/s4Ya/b4kab WDCIPOUTBOUND: EOP-TRUE X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(42932892334569); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(6040375)(601004)(2401047)(13024025)(13023025)(5005006)(8121501046)(13016025)(13018025)(10201501046)(3002001)(6055026)(6041248)(20161123564025)(20161123562025)(20161123558025)(20161123555025)(20161123560025)(6072148);SRVR:DM2PR04MB830;BCL:0;PCL:0;RULEID:;SRVR:DM2PR04MB830; X-Microsoft-Exchange-Diagnostics: 1;DM2PR04MB830;4:WKrTqeYR40Ti3HPdkR57l31vODeATdwRDVedUZdOAdxFdUndNTUPKVcYA+EqdzOsnMcwUD1xrGW9PjR6ek64b5jGSGS5vozIg1iszBNGTzHqx59FyRphzAHLmYq6ZL0Rr2/Mdw+rAtGUDygNre03oZ3muxj8tlm0uiHlZ9tzX4dsde9AFs1jyKpc6SUgIXEzCrTtgWGGHzl0p1/YfY9mIcYe9ZeV4mZDh4hHtpzkz6EnUp2FwzAvQjAHxl+cj2mnEu9mVUhgkuNjb3bpN5iQSfT7JT92nNwLfPKPJ2tC9h5HjEiF7epmZTOVH7/0qhRiE6AI/qO+lqDdjJNm1RvyrSLFX1lkX7hI2yfCqLkW9BtJHokgrOgIJ99jjJndEpQDvzo120qw4ghLaUhZeCN0CROZj3Fn904dzGTE21XpUnetw+hhpwgW3l4TMHmepjJeONw+anlnuDZqJrQSVd9qLynzAmEhKZ4WakKrIXQrM0878B1XTxVc3lH8xuvm/rYwDSjXdGjCtDdFqqOLZOilTBfZJqkq5nICnXbpyLfktrHoOxpeWTFZo6UxNn55Dhh54prMhwrkWZgoygV27MqZ4TD349qLivytGSC/E/VbtXQTUgKYC3T9zk/Dr5GNElfTmScr+Z1WX0IpFxMTvds+WH4Yv09i2MB2afUH4+sNzmn7u5rAmwWkLJRwtu4P5wy2CRhutBV3JRfrB+6gwRul1BeQtceNvdxbULFsYfa80dqx0qZswnjK+EFq9l6Ils7M X-Forefront-PRVS: 0249EFCB0B X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1;DM2PR04MB830;23:OMi8TX8wgIfkFBy9/JkVpFWbXZiIhr34eNY/8NvJ?= =?iso-8859-1?Q?CJ+4NpueWH2wzmJfIRhJMT+9XfLxwEqeEtZqpbWjs65lU5stBaUlwevmn5?= =?iso-8859-1?Q?iIVTgUUgpfkEI7IfKyEYKeLR6LKg9ZEaVUB8/NVkD3rhhfIJS518E3h0wc?= =?iso-8859-1?Q?g3LQBok+vqrt7fYMtMZzPEl8oQH3aoRCEFYpaZd3ZNjq6MgQNhmXSUKSiz?= =?iso-8859-1?Q?HfmxvCSCsXQn0jGYRJsbpsNzKE5RXd2aMjBdLP/0JK/Jci1RrDxztTDZMq?= =?iso-8859-1?Q?sXUAho363ZsvVER7/2cz9ES+kWwkJwwP9f9nE6aJlKejqMK1W9CLfTv2Uc?= =?iso-8859-1?Q?XWomaRs6KtzWM1+L52/QkbTln1JgRUedugJXkknqRN18RwTp6oTfxURhiN?= =?iso-8859-1?Q?nu/11YKwOFa/mCjbakGMRQj0u2PRbL4ZHxVPxmgSz/RHWrzhgRhRW36kB/?= =?iso-8859-1?Q?GiyThrxETMkWWutdQlZ+ojwi76PjIoRspHzG8stFvdPkEk4Zt4G0Zzhu0M?= =?iso-8859-1?Q?19KBVjIMpN8OM8joCq6GIhw4hnL8s+C+MgZeIJvnOcfsCgOqNq/h2i0Zid?= =?iso-8859-1?Q?51JUu+0xPThm3wUXMRe+DTKvh4WwOZnsF/qcZGe1qVgsTsUcUuXf1/UNhl?= =?iso-8859-1?Q?2JXWFIIVDjIUrs2nWgsMW245bMFBmY48kcfbEqyQo8B9dE6yXLeyQc7UUa?= =?iso-8859-1?Q?sgwviwKpibd+ExNdKlyKIzV6KyZuuUyOiWVccqHBq4owvIOPh7rT5NyMd9?= =?iso-8859-1?Q?eGd7tgpb+/HpqbEO3vxwkvs1/haXYg13Zk1Y15VZo3OpVZ4yE0ECt0r2lz?= =?iso-8859-1?Q?pOXMAiw2LVBGUSzK0ZLueipATJIGPKqqZGAwHjcdpfanEms0avbmHp6Aoy?= =?iso-8859-1?Q?sifYOXJDixHc2851tFZVwxBYHOMtOSrAj+hkCozGcTSx7tOAjtJDXESnCs?= =?iso-8859-1?Q?DHGjMDoHzLyrfiYWZhNtXZSKiMsOPJQ6BpCDa7Rka+D858jEg23zS/Z9PG?= =?iso-8859-1?Q?cYbbfvbeBSyr5g9YPja/+NQ2m5BBrnXa2VMqpDK13tU4bYku4rdDg0TcdR?= =?iso-8859-1?Q?A+HjCKsxNeSgv0hxfCjnwVpjz6iRaVwIqZHKyv61k3evhXQrLqaY8Pt3hw?= =?iso-8859-1?Q?f+g1u5nkFimvmSlypk6KqPaBxgbFr6chgXDUKYuFlA9yVD477FpJof3Zhr?= =?iso-8859-1?Q?v55jiS/3WB9UMhEw1Ypo9KID8SPppck01KuKxLN3MJajwt/+7G1R6ZC9z8?= =?iso-8859-1?Q?fCuUKyQnR9lVmzrC?= X-Microsoft-Exchange-Diagnostics: 1;DM2PR04MB830;6:a/kqFEp1TF87iImfccoJTWQF516XPpYfoSjDZiEi52jnONTUn31YdOZO4cK2okAOeH4Hcp6BMJuNVNlaGZYR4+z1++IK+r6uiLi+qSn2Z2T8PsYylGLO0pMHetZG2EzcsB4mRAeMaU3+3FlnHzADHam/XL10C4hSzotEdIRzMTNE/RVE3m6ubPuclqByygubFis2pjBHilsnOzB51SoEPzoGbREAr5cChLwPStU6f7tfnzuaUJBkIamm9qDi0VEl1SmQHopHEPpU2c8+1sLjIifv+V5/U/ODcCMPSiq2nIYS8sjey+a60Skih2Fz4oZMf+Q4BA/ZvD3DxmAO8rzerkp2uRJDSZnge5yecQjorHHiUW87FuQKUgj7zIo4XVcon1IrC03QW27iEqUTnzdgq1iS9ez9SLtNeIfi5v1qJHF5mUe6h4g6hJtWPu2hWQ2r;5:Uha9LHYHppLKzjtO2RnX56v4I/bgpKNaxGQrnnejHmLEg9EuqYyHnK1cd467il77Qcsm0vCcgYT7z/IQRZ5ddLiPoshyJEMze/QanSMb3JiVyWj4ksYDhYNdKsuez7BpIzufHxd07hhHQnjzn3Hqag==;24:8pmUqGxl8+zDQp4FqyPDvjVbQWPgoU1lBYXQ6tqnMgKkyREHmX7PxJT/xcA2HMvQuYDdwbpxcdNlJONq7dUrpg7Asxov0mabocYwWZ2kM4A= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;DM2PR04MB830;7:ezRd8CQqt/n/C4W7lQOxcfydy4clptLSGZW/xfv5OrC2a+xkkWMzx137eh25kREg0kN4OGMUhQDQ7wa4srs2kWT28i97OpAAy3tOplrs29H8ohMOtv+J15LWXAYKRCi/Uno+oBgTMoji0VoGc5++FgUCtjR7L8zsztz5LdaOKC5QHUScVS3QWmnlJ1ye+ieL0nzRlA93nHsRMehgAjKCGQpSOhUXTa+LpvdnUS+b1dr5Ug5k9//5E23fcy73OvhmFhzTK6DOwojbYX1XN6PFxN+flpZZg9Z8L+Vf/u5bE43sP2yiBCAlRURCkZ9ulXAPBsZ5kZyFIn031DIIp2+7dg==;20:yl+acd3n6ASOaKDAT5Z93zeBtGgf39GZ46iXo3nLCHe8SGLFhuQKcsvyNQkKYLbdBeUZc3/kidSYw4JlwbJj9YDsomD1NGKdJTHukBgdNu6nL1rNpQATiCBUlhujhvKguo/Q9ief45XaGUsnInAOEjENtTI2Z8U+0667i4Z4ZTU= X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Mar 2017 17:32:14.1406 (UTC) X-MS-Exchange-CrossTenant-Id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=b61c8803-16f3-4c35-9b17-6f65f441df86;Ip=[74.221.232.54];Helo=[sacsmgep14.sandisk.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM2PR04MB830 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by mail.home.local id v2HHYlGg004165 On Fri, 2017-03-17 at 17:57 +0800, Ming Lei wrote: > Before commit 780db2071a(blk-mq: decouble blk-mq freezing > from generic bypassing), the dying flag is checked before > entering queue, and Tejun converts the checking into .mq_freeze_depth, > and assumes the counter is increased just after dying flag > is set. Unfortunately we doesn't do that in blk_set_queue_dying(). > > This patch calls blk_mq_freeze_queue_start() for blk-mq in > blk_set_queue_dying(), so that we can block new I/O coming > once the queue is set as dying. > > Given blk_set_queue_dying() is always called in remove path > of block device, and queue will be cleaned up later, we don't > need to worry about undoing the counter. > > Cc: Bart Van Assche > Cc: Tejun Heo > Signed-off-by: Ming Lei > --- > block/blk-core.c | 7 +++++-- > 1 file changed, 5 insertions(+), 2 deletions(-) > > diff --git a/block/blk-core.c b/block/blk-core.c > index d772c221cc17..62d4967c369f 100644 > --- a/block/blk-core.c > +++ b/block/blk-core.c > @@ -500,9 +500,12 @@ void blk_set_queue_dying(struct request_queue *q) > queue_flag_set(QUEUE_FLAG_DYING, q); > spin_unlock_irq(q->queue_lock); > > - if (q->mq_ops) > + if (q->mq_ops) { > blk_mq_wake_waiters(q); > - else { > + > + /* block new I/O coming */ > + blk_mq_freeze_queue_start(q); > + } else { > struct request_list *rl; > > spin_lock_irq(q->queue_lock); Hello Ming, I think we need the queue freezing not only for blk-mq but also for blk-sq. Since the queue flags and the mq_freeze_depth are stored in different variables we need to prevent that the CPU reorders the stores to these variables. The comment about blk_mq_freeze_queue_start() should be more clear. How about something like the patch below? [PATCH] blk-mq: Force block layer users to check the "dying" flag after it has been set Commit 780db2071ac4 removed the blk_queue_dying() check from the hot path of blk_mq_queue_enter() although that check is necessary when cleaning up a queue. Hence make sure that blk_queue_enter() and blk_mq_queue_enter() check the dying flag after it has been set. Because blk_set_queue_dying() is only called from the remove path of a block device we don't need to worry about unfreezing the queue. Fixes: commit 780db2071ac4 ("blk-mq: decouble blk-mq freezing from generic bypassing") ---  block/blk-core.c | 13 +++++++++++++  1 file changed, 13 insertions(+) diff --git a/block/blk-core.c b/block/blk-core.c index d772c221cc17..730f715b72ff 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -500,6 +500,19 @@ void blk_set_queue_dying(struct request_queue *q)   queue_flag_set(QUEUE_FLAG_DYING, q);   spin_unlock_irq(q->queue_lock);   + /* +  * Avoid that the updates of the queue flags and q_usage_counter +  * are reordered. +  */ + smp_wmb(); + + /* +  * Force blk_queue_enter() and blk_mq_queue_enter() to check the +  * "dying" flag. Despite its name, blk_mq_freeze_queue_start() +  * affects blk-sq and blk-mq queues. +  */ + blk_mq_freeze_queue_start(q); +   if (q->mq_ops)   blk_mq_wake_waiters(q);   else { Thanks, Bart.