From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 114D1C4332F for ; Fri, 3 Nov 2023 16:16:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 424A08D00C4; Fri, 3 Nov 2023 12:16:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3D4758D000C; Fri, 3 Nov 2023 12:16:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29C4B8D00C4; Fri, 3 Nov 2023 12:16:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 196F58D000C for ; Fri, 3 Nov 2023 12:16:02 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D060F161074 for ; Fri, 3 Nov 2023 16:16:01 +0000 (UTC) X-FDA: 81417144522.16.F19B914 Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by imf28.hostedemail.com (Postfix) with ESMTP id 82DB4C001E for ; Fri, 3 Nov 2023 16:15:59 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=invisiblethingslab.com header.s=fm3 header.b=r2rmGoKn; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="r/MxdHIR"; spf=none (imf28.hostedemail.com: domain of marmarek@invisiblethingslab.com has no SPF policy when checking 64.147.123.24) smtp.mailfrom=marmarek@invisiblethingslab.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699028159; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dHOdN0Q4arzDi1FH7Q2SMau1Y7NQ5Lx4NHg9xz7kfwE=; b=Nh8ABCRXeJbwMDE4A+Rhvs2RMTHQHB0lsALd+Gr+2qm3w4htLos7HDB1sbxk3oJxLexL2s X65hBxxyAo6m9vhEofSve9VFprm/7GJOJba4YnN2Iv33iFrVPDU1e0ig84qPjTA7pR71EP ZeDmS17TG+nb0gFSEAXNIyF+08Q78lk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699028159; a=rsa-sha256; cv=none; b=Xp+W0fBlouetfJivGuQmgJEf5sGoltQqXNpKENW6bqLYN3Uv4378PXV/pYQ/6uphLxemAF AdGHWpngvGy4oSZftVR7kpXCjtTmwavEiY2qz8IkxzoOTwFADG5z0mCw5XHqvMu9qUB2M2 CWDknLz2StXI6wdegCqTJKt6V9aX8s0= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=invisiblethingslab.com header.s=fm3 header.b=r2rmGoKn; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="r/MxdHIR"; spf=none (imf28.hostedemail.com: domain of marmarek@invisiblethingslab.com has no SPF policy when checking 64.147.123.24) smtp.mailfrom=marmarek@invisiblethingslab.com; dmarc=none Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id A2D313200935; Fri, 3 Nov 2023 12:15:56 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Fri, 03 Nov 2023 12:15:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1699028156; x=1699114556; bh=dHOdN0Q4arzDi1FH7Q2SMau1Y7NQ5Lx4NHg 9xz7kfwE=; b=r2rmGoKn0sF3EVywj9Gn+J350GUEURJq4B1HpQA0HdI5GDSL5WH CbCvH8u/HzCP0rGfL1RaF7G/EpvTKYsvFEtQxSOkUvPJCv8VFgCm9ig8GkXuRY4g u2bN4/T1+U+CUlO7bfytE87EVXvJb9I9rucQB/hHIiT3Yirdi5YjKHSaWpaOEUqW x64hgecvajSI29oDfO5Yk4vpkWfML5rjVRV9vM2mllpBtLah/3ft/3Oqv5t8jubk Tp0cyvOh/bSo2Aj3oC/RXuTX5WHtvnbLPiq8W6leJfMxRZf7LPwjOph9Lc5s8suj 5tG9AAQ+/vfleLv/0Opdq9t/rgryZPMDmoQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1699028156; x=1699114556; bh=dHOdN0Q4arzDi 1FH7Q2SMau1Y7NQ5Lx4NHg9xz7kfwE=; b=r/MxdHIRgTd/0YVIADBIJWDu7kO+a EUuJsLTVGQDrUK+KK8TqVOkjI1Z7esYwfMGGVYFwhWkqZjgrAHqXLYCxwuO1qgw6 KMFVlHlCKqCRSfIbJZ2ETnj7WP77O4Z/ZLhSrVCx/VvFpaLrQwC1WZA4WFNErQuJ mSQzWmD+ftgKgo5kKF3mTaSx8xcrty727PsoiayzNwxcwM5Frb0dlcYlqycuMZCL FJkpPipfvLYcGo+AQCXo/cDHDENJp3ruUq60aleG33UX9AueJQLQ6fIjpbQX+MLy sOprqEFtrWSyvlLYHgaDeSg+ALukbSOnDl7CnSb7Utge1vImgGL08piyA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtkedgkeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtroertddtjeenucfhrhhomhepofgrrhgv khcuofgrrhgtiiihkhhofihskhhiqdfikphrvggtkhhiuceomhgrrhhmrghrvghksehinh hvihhsihgslhgvthhhihhnghhslhgrsgdrtghomheqnecuggftrfgrthhtvghrnheptdet vdfhkedutedvleffgeeutdektefhtefhfffhfeetgefhieegledvtddtkedtnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghk sehinhhvihhsihgslhgvthhhihhnghhslhgrsgdrtghomh X-ME-Proxy: Feedback-ID: i1568416f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 3 Nov 2023 12:15:51 -0400 (EDT) Date: Fri, 3 Nov 2023 17:15:49 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Mikulas Patocka Cc: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Message-ID: References: <98aefaa9-1ac-a0e4-fb9a-89ded456750@redhat.com> <20231031140136.25bio5wajc5pmdtl@quack3> <8a35cdea-3a1a-e859-1f7c-55d1c864a48@redhat.com> <3cb4133c-b6db-9187-a678-11ed8c9456e@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="FVS16JFHJ7btn9Q1" Content-Disposition: inline In-Reply-To: <3cb4133c-b6db-9187-a678-11ed8c9456e@redhat.com> X-Rspamd-Queue-Id: 82DB4C001E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: wjkj8ua9w6r8rz6kaurco4ymn5kuiqch X-HE-Tag: 1699028159-820459 X-HE-Meta: U2FsdGVkX1+EW5B2ybwYfl1eAxbVHmL1M9EUb/s3TV2q/UUwumXR7y/w4H2lB9RSppW+8vcL9jEqqJxQxLi1lxAPY71opxrotZ5auDHrrYPDojO7EfMC3VIY7jhigBF83TKtTXpyz4ssNF5tJwCr1wuvIJPHmTArgo2f79JfKbLkMuat7GPQ088hD88Oa3EIuJ148YB/lz0Me0EyGXcY1SxAS4osOjLz7LcznBn/gfagXGYzHiw3rpXXtM17iTLpNgKDDt+NR5CjiGw1igZ2Vlphdea25XpSjD7gGwENmvrYku08dtl2uOp8bSbVZ/T+PXxYHWLBmBNrjvtxnlhjpwN5gbouhg4VUka5NthhMZ7vY+LB3eslwq1/exV94+fyslwx/LiqruuEwnD9pugCB7SAJsUk+4M7j7OJzhWQBVJXrhziiYNZjZSrMCujW/2zJaZTlACIUG2XowoL7swD+3mEFNtfWOO+AftsQoT2mh1EHw0wY6QOD6oD69dLKGuYriW/N89XDEAAkEKp2NYTWp/+KzqJ53RCNxH98Nu6c0siDK1DaYgoe6NkNGdY1Y2OgmWZ+g/m7TDYXipV+7mA9hEGUYaHhZQ1kIST0Xph53HAERBs8hHrZ5xUIYnTEgRz6dtW/vvSlGK3QhdWHZuAjCQ1vJA0EzNroqrA46TJ4qUA72ml5wtX+NCXxvisnPVfjpHUnYNLAoJ/LzgAA4iuI3ljBrJHb7lo0qcML3uFLKM0Q5/Ix7FDB7+zDrZf64JCGxX8amtsIGi8dI7S52ZSe22wEJzUAajpOYwG0osBtacBlxsVMr6pH7LnpAj/SbDg8kDUQ0NVPEGluVISx6P5vhRtvNw1TjVEOhgP4w2zbqgVUb/a0mkWoFqYjGr1I9sXrMphYZjhRTtqNFXM2EiuFVzPICQ9zWcmkqmyvySGD7F85hmpe6akbmHWwlW/WdaUQ3N+kpm9FP92xedq++A 0J1rVlK7 H7qxH271waS5sOhPJZKqX9JZKEi8KUEUXdP9KIZkRSwaD27q/KtnvRReuYLKAp2Lu3m6KXqcBHw4Dz0FTGd4iuIsJ0A2OsSbWZRllXXRInuuz0y/Y4egCCPNkh9omd3sV+Ana37lxsi33ea4EuEWh5y6Y/YEqHbZzmEIyjTPhZStfP1CbBM6vO99IH2peCcwLbSEV6PnonQwTQ47ppedCJh5B0ZFzGJbCI7PyYZERViFnvClO+TC1UEBSRvXFMeb1kHXmMJi7XwN7HMBUCAdzxRdrvn8R5/YtRlBvYuCqiXxDU007lBOp7wAcRwPjD5yAkvGASkZocm/hJ1NNJWnj7tgtbhktHcssVdS3QeeqCS6CfJAUVXlfQ5V81/pbd20avmzNC+f2xQyLZ+5bZ1+INRZDpN9Va3MZ+t/7jBFKDej8OcoMZS3jvDr6nq8rfCCtlvtwTlpxrLythvI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: --FVS16JFHJ7btn9Q1 Content-Type: text/plain; protected-headers=v1; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Date: Fri, 3 Nov 2023 17:15:49 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Mikulas Patocka Cc: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 On Thu, Nov 02, 2023 at 06:06:33PM +0100, Mikulas Patocka wrote: > Then, try this patch (without "iommu=3Dpanic"), reproduce the deadlock an= d=20 > tell us which one of the "printk" statements is triggered during the=20 > deadlock. The "821" one - see below. > Mikulas >=20 > --- > drivers/nvme/host/core.c | 8 ++++++-- > drivers/nvme/host/pci.c | 27 ++++++++++++++++++++++----- > 2 files changed, 28 insertions(+), 7 deletions(-) >=20 > Index: linux-stable/drivers/nvme/host/pci.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- linux-stable.orig/drivers/nvme/host/pci.c 2023-10-31 15:35:38.0000000= 00 +0100 > +++ linux-stable/drivers/nvme/host/pci.c 2023-11-02 17:38:20.000000000 +0= 100 > @@ -622,6 +622,10 @@ static blk_status_t nvme_pci_setup_prps( > prp_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); > if (!prp_list) { > iod->nr_allocations =3D -1; > + if (nprps <=3D (256 / 8)) > + printk("allocation failure at %d\n", __LINE__); > + else > + printk("allocation failure at %d\n", __LINE__); > return BLK_STS_RESOURCE; > } > iod->list[0].prp_list =3D prp_list; > @@ -631,8 +635,10 @@ static blk_status_t nvme_pci_setup_prps( > if (i =3D=3D NVME_CTRL_PAGE_SIZE >> 3) { > __le64 *old_prp_list =3D prp_list; > prp_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); > - if (!prp_list) > + if (!prp_list) { > + printk("allocation failure at %d\n", __LINE__); > goto free_prps; > + } > iod->list[iod->nr_allocations++].prp_list =3D prp_list; > prp_list[0] =3D old_prp_list[i - 1]; > old_prp_list[i - 1] =3D cpu_to_le64(prp_dma); > @@ -712,6 +718,7 @@ static blk_status_t nvme_pci_setup_sgls( > sg_list =3D dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma); > if (!sg_list) { > iod->nr_allocations =3D -1; > + printk("allocation failure at %d\n", __LINE__); > return BLK_STS_RESOURCE; > } > =20 > @@ -736,8 +743,10 @@ static blk_status_t nvme_setup_prp_simpl > unsigned int first_prp_len =3D NVME_CTRL_PAGE_SIZE - offset; > =20 > iod->first_dma =3D dma_map_bvec(dev->dev, bv, rq_dma_dir(req), 0); > - if (dma_mapping_error(dev->dev, iod->first_dma)) > + if (dma_mapping_error(dev->dev, iod->first_dma)) { > + printk("allocation failure at %d\n", __LINE__); > return BLK_STS_RESOURCE; > + } > iod->dma_len =3D bv->bv_len; > =20 > cmnd->dptr.prp1 =3D cpu_to_le64(iod->first_dma); > @@ -755,8 +764,10 @@ static blk_status_t nvme_setup_sgl_simpl > struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); > =20 > iod->first_dma =3D dma_map_bvec(dev->dev, bv, rq_dma_dir(req), 0); > - if (dma_mapping_error(dev->dev, iod->first_dma)) > + if (dma_mapping_error(dev->dev, iod->first_dma)) { > + printk("allocation failure at %d\n", __LINE__); > return BLK_STS_RESOURCE; > + } > iod->dma_len =3D bv->bv_len; > =20 > cmnd->flags =3D NVME_CMD_SGL_METABUF; > @@ -791,8 +802,10 @@ static blk_status_t nvme_map_data(struct > =20 > iod->dma_len =3D 0; > iod->sgt.sgl =3D mempool_alloc(dev->iod_mempool, GFP_ATOMIC); > - if (!iod->sgt.sgl) > + if (!iod->sgt.sgl) { > + printk("allocation failure at %d\n", __LINE__); > return BLK_STS_RESOURCE; > + } > sg_init_table(iod->sgt.sgl, blk_rq_nr_phys_segments(req)); > iod->sgt.orig_nents =3D blk_rq_map_sg(req->q, req, iod->sgt.sgl); > if (!iod->sgt.orig_nents) > @@ -801,8 +814,12 @@ static blk_status_t nvme_map_data(struct > rc =3D dma_map_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), > DMA_ATTR_NO_WARN); > if (rc) { > - if (rc =3D=3D -EREMOTEIO) > + if (rc =3D=3D -EREMOTEIO) { > + printk("allocation failure at %d\n", __LINE__); > ret =3D BLK_STS_TARGET; > + } else { > + printk("allocation failure at %d\n", __LINE__); I get a lot of this one. > + } > goto out_free_sg; > } > =20 > Index: linux-stable/drivers/nvme/host/core.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- linux-stable.orig/drivers/nvme/host/core.c 2023-10-31 15:35:38.000000= 000 +0100 > +++ linux-stable/drivers/nvme/host/core.c 2023-11-02 17:12:39.000000000 += 0100 > @@ -708,8 +708,10 @@ blk_status_t nvme_fail_nonready_command( > ctrl->state !=3D NVME_CTRL_DELETING && > ctrl->state !=3D NVME_CTRL_DEAD && > !test_bit(NVME_CTRL_FAILFAST_EXPIRED, &ctrl->flags) && > - !blk_noretry_request(rq) && !(rq->cmd_flags & REQ_NVME_MPATH)) > + !blk_noretry_request(rq) && !(rq->cmd_flags & REQ_NVME_MPATH)) { > + printk("allocation failure at %d\n", __LINE__); > return BLK_STS_RESOURCE; > + } > return nvme_host_path_error(rq); > } > EXPORT_SYMBOL_GPL(nvme_fail_nonready_command); > @@ -784,8 +786,10 @@ static blk_status_t nvme_setup_discard(s > * discard page. If that's also busy, it's safe to return > * busy, as we know we can make progress once that's freed. > */ > - if (test_and_set_bit_lock(0, &ns->ctrl->discard_page_busy)) > + if (test_and_set_bit_lock(0, &ns->ctrl->discard_page_busy)) { > + printk("allocation failure at %d\n", __LINE__); > return BLK_STS_RESOURCE; > + } > =20 > range =3D page_address(ns->ctrl->discard_page); > } --=20 Best Regards, Marek Marczykowski-G=C3=B3recki Invisible Things Lab --FVS16JFHJ7btn9Q1 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmVFHLUACgkQ24/THMrX 1yzUvAf8CIbTqn3u4kztOYavM8CP6Oyv22Wl/+svhHeItHvMBisOG0F6t0TvJgaW 8LVB8g/fZgqxW99/X5D8V8V9I2qJXZlD7N0LDv9b625ay8PR/VvESTiOJbVd+d6s RQGCeF6fazD2efOeUb7HDzfiLW6vedQk9EkfhEyjf7Ey7TtlqvHpa9DiFIoBTPnq xBIxq8q2gorWBwUDkXpgnQt78Res60TlpKa/epDVW9ZY+cHEL/gas3rUi/r2zgvy w2AiTx4aTQr/769TvGdDPk+dK3V5+y3WbfWdB0nM6lj7Sb/LlN6n2Qn29CD2Drvh kVo6d9JNXGcfvc/yQmBuKASUp0UThg== =7MuL -----END PGP SIGNATURE----- --FVS16JFHJ7btn9Q1--