From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DDD3C4332F for ; Thu, 2 Nov 2023 00:38:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A9BD980011; Wed, 1 Nov 2023 20:38:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A4B8180009; Wed, 1 Nov 2023 20:38:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8ED2680011; Wed, 1 Nov 2023 20:38:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7FF2880009 for ; Wed, 1 Nov 2023 20:38:58 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4F4F9C096B for ; Thu, 2 Nov 2023 00:38:58 +0000 (UTC) X-FDA: 81411154356.27.55D3194 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) by imf15.hostedemail.com (Postfix) with ESMTP id 40B0CA0008 for ; Thu, 2 Nov 2023 00:38:56 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=invisiblethingslab.com header.s=fm3 header.b=PT0LNGhD; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=mbZXg7UJ; dmarc=none; spf=none (imf15.hostedemail.com: domain of marmarek@invisiblethingslab.com has no SPF policy when checking 66.111.4.26) smtp.mailfrom=marmarek@invisiblethingslab.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698885536; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XfCwqkrmHX39PAF8DgFnwRKfMHvhyDtQ9mQnwUGuGdk=; b=KUP9Y65Jj8tg8LdcRZfU/1f/M9IBtTbGcVJjVsbzzfttD/LhSWT9/FTezn8q3WEqn3DwhQ EglwiY1y3Do7hGFQOYgnoDGDFNx7tk4gHenSMarPBu2SfRyftEGgCpyDuVsFHKYwGceU17 IBvth8tvry4urMnsUGSI4yNf3NCR4jM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=invisiblethingslab.com header.s=fm3 header.b=PT0LNGhD; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=mbZXg7UJ; dmarc=none; spf=none (imf15.hostedemail.com: domain of marmarek@invisiblethingslab.com has no SPF policy when checking 66.111.4.26) smtp.mailfrom=marmarek@invisiblethingslab.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698885536; a=rsa-sha256; cv=none; b=a7Z20iv8gHfSF2B66G8MvHQtwAIaRVOiWyNmfVkkp+030xbjN1RAvyXHfuTgMHU9BRo+8W 98MRUMAI62uxUwNxjUKmldjPSKnwCe3KnOyS464tSoj+4WvfBRNlS9YBQcCmw2sVI+zyRB INY1Xxa01zdiJ4A0ISiRG9ZFWXpigAk= Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 8699A5C01BF; Wed, 1 Nov 2023 20:38:55 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Wed, 01 Nov 2023 20:38:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1698885535; x=1698971935; bh=XfCwqkrmHX39PAF8DgFnwRKfMHvhyDtQ9mQ nwUGuGdk=; b=PT0LNGhDj+vbfNHoMmL2qJ1AuJAWVHGJCctzbVQqfIEQuHvA5B7 lFevgrg85IyGyvH4TBivJ30VIxMHlpLICzqJyhi23cTG18/2zl1dWY3xh2zygZ7k Lrjo+IGCRanEw1SiUYD9XOeccVO6bp2wmrnnPwu7KvHI4++/N4H3uZaGV0nOtwUU j1aasyAgjZa9iNtcb9WTf76NKC8EPJY7txIfPO81ieZcHdQlysdr+bFN/Bok+gym c9f1e9oh5OPLTVFVUiBpjT4Pr5yRdknVkplRrieIR2xkfiw5BAKnZX9SkEjxr9Mx 6EGs5lLxvjQzkpQYw/6q3rogweNkOPAKy7w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1698885535; x=1698971935; bh=XfCwqkrmHX39P AF8DgFnwRKfMHvhyDtQ9mQnwUGuGdk=; b=mbZXg7UJ3EtBNWfqqbqJ8fKK2T4xf XZCzpWLT/P0bAwS2bHZvyMUBJSqImup9XkzFdTrzCRr0UHr2FAihjCFobrnSwJmo c4VLDXr3+lRFItdq+yVw2lZw4VWCYUosMGbD43VCwKSH9gMtleMOF0m4yMUfVk6A a60qWGxadK3mPzto7LVL9UCvBjpDSi/OYHp2fTlrs8y58d0AYet4pcIwsG2/pc3J BfIiTCBUl6q18ieV7tuvsbh+I9AcGLse2rR2YqVXRUJ2rY+fxtXZBOxCTSHvQ5+r zpdPifR8jr8mXT4/A+JfJFiwLVu0JI9lg5setnL1Bi9ts6kPLTAoQXDcA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddthedgvdegucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtjeenucfhrhhomhepofgrrhgv khcuofgrrhgtiiihkhhofihskhhiqdfikphrvggtkhhiuceomhgrrhhmrghrvghksehinh hvihhsihgslhgvthhhihhnghhslhgrsgdrtghomheqnecuggftrfgrthhtvghrnhepgfdu leetfeevhfefheeiteeliefhjefhleduveetteekveettddvgeeuteefjedunecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghk sehinhhvihhsihgslhgvthhhihhnghhslhgrsgdrtghomh X-ME-Proxy: Feedback-ID: i1568416f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 1 Nov 2023 20:38:49 -0400 (EDT) Date: Thu, 2 Nov 2023 01:38:40 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Mikulas Patocka Cc: Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Message-ID: References: <20231030155603.k3kejytq2e4vnp7z@quack3> <98aefaa9-1ac-a0e4-fb9a-89ded456750@redhat.com> <20231031140136.25bio5wajc5pmdtl@quack3> <8a35cdea-3a1a-e859-1f7c-55d1c864a48@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="EW5Jd6ZInl+5b4q+" Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 40B0CA0008 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 63gbwurtpyeqzqttrrren4nm8wudzcbs X-HE-Tag: 1698885536-923396 X-HE-Meta: U2FsdGVkX18dw6QJxWo0Y6/zozE8/VGLWllbwxXrj448kKE7U7qlIzgiw6Z2lLpKLotAVqp+Cey3mbwyRXc0VmcPffDWdkgZd9soKgdkeWUDaR49VRVN2KdA+X8e1UZLXq5+9RV1L56Hp/ruGxCuTwVQo8Ov865efA2ouuw2XzC++3an2olQr566hryRR7vYCyphnkvkm3fH+evSYTUZwZym7H54c5ROfl1ni05Ko/W4SdOVAiKr+eRoGBzCOQzBF6jTj+Q9DrCqzg54FxgIk7Be0tHmyVUk/ONEPXk3n8vLZkGqWGzxYLVyB6HHTL8y/nLzHoq+MJQzAoYSOE8Jinf2mjAugJMCbnMRjvkJNFPJMs22hPDqB24gy2wUCV8Hl6a5Lz+ePuE23TvSWrbNr/TjMErerJhh86JCnUvb24BQRXMnrhDtmorXqlHgtUMt2w4T8Mx4ukEX3jpnWvC7X0rZ6yoLYfvayxP3apUQmrtnMV4nwkFnUzCOO7ONGsWOtFUkhJ2V8lzia0AmpahYhFXPnEGp5ZbfFTLKPqFwD8nooF68uptgwlCWAWGeVACKH9wb2xnApKaRNfrdZtqWqhSIHl4thloiQQiM7eTX684MbufSpMnY8iAncJu8t89W7w1l/6dHoBJ4zQF98YHgskdQDUvtKbXuqUw/8AMiDKKkcBdOskkBaBcB1ONO/GICZFSbYc/5G/wyIzNNI761Dq+Z4nIH9YcT5U7WIkgqM3MnyB/hliDquUY+cYAAHBO34VEXVZR5MMX6x1JK+56uliaJTB0gR+x34DsqPwhln0PTuvAkY2uxL4WkC2ENEbPVSXZTJ3xzNJbONcUMpNqmid0s3Wv4M7rIB1TXdnVfhfrqvqxBtHI8q1wxDd5j75aVQBv7/9/vVCeV4W/FBhWgV4EQSB8RITvKnFiS7JAUHlcYUfu1u6bt53+DnEfjKe+m3SNtNGYPjYICAs8gvVE THFluAgF dCk5d3ceAkGSTeetxH/bufbgmPabEbgYb/e0x+BRujhcJZ8O2LFBDgDf2e9FZ/+DnOO64HOw1/5JNfEgqCGFRJ+UUgoD+9ETqZW3frIR8C1S0fvB3/vzL2oy5SiHrpTXoLHP18/OhhzBlJ0l7YLXHAbcnCIcCBDPLCntEpRWhF9I0qVdWLSQs1Avm0HrBBO6JZVm3y8UTdqQuogXrHeipr6yLAC2xgzLcSSq2auTFNT3Dgs9MH22YXHZHOjZ8hTnB+63jXDHJKwEfEuL9NgAPtVqlE+sNQdkZHi3UzTtekawKTY8oydDf+GIhgbK0cm7Uy6fi8E1gXSmhZ7eFebFAex1J68z1QYJEJlsQAUDRQ/AyN84OI7FHNlrdHg4JAiEUCFTd65Byp4+IKPv4KDyB0uMm/PeIGdoPkYhgzrOhVOycwlvHR+BASMsoHAz/h56SNJMIj4yshqK3jfs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: --EW5Jd6ZInl+5b4q+ Content-Type: text/plain; protected-headers=v1; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Date: Thu, 2 Nov 2023 01:38:40 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Mikulas Patocka Cc: Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 On Tue, Oct 31, 2023 at 06:24:19PM +0100, Mikulas Patocka wrote: >=20 >=20 > On Tue, 31 Oct 2023, Mikulas Patocka wrote: >=20 > >=20 > >=20 > > On Tue, 31 Oct 2023, Marek Marczykowski-G=C3=B3recki wrote: > >=20 > > > On Tue, Oct 31, 2023 at 03:01:36PM +0100, Jan Kara wrote: > > > > On Tue 31-10-23 04:48:44, Marek Marczykowski-G=C3=B3recki wrote: > > > > > Then tried: > > > > > - PAGE_ALLOC_COSTLY_ORDER=3D4, order=3D4 - cannot reproduce, > > > > > - PAGE_ALLOC_COSTLY_ORDER=3D4, order=3D5 - cannot reproduce, > > > > > - PAGE_ALLOC_COSTLY_ORDER=3D4, order=3D6 - freeze rather quickly > > > > >=20 > > > > > I've retried the PAGE_ALLOC_COSTLY_ORDER=3D4,order=3D5 case sever= al times > > > > > and I can't reproduce the issue there. I'm confused... > > > >=20 > > > > And this kind of confirms that allocations > PAGE_ALLOC_COSTLY_ORDER > > > > causing hangs is most likely just a coincidence. Rather something e= ither in > > > > the block layer or in the storage driver has problems with handling= bios > > > > with sufficiently high order pages attached. This is going to be a = bit > > > > painful to debug I'm afraid. How long does it take for you trigger = the > > > > hang? I'm asking to get rough estimate how heavy tracing we can aff= ord so > > > > that we don't overwhelm the system... > > >=20 > > > Sometimes it freezes just after logging in, but in worst case it takes > > > me about 10min of more or less `tar xz` + `dd`. > >=20 > > Hi > >=20 > > I would like to ask you to try this patch. Revert the changes to "order= "=20 > > and "PAGE_ALLOC_COSTLY_ORDER" back to normal and apply this patch on a= =20 > > clean upstream kernel. > >=20 > > Does it deadlock? > >=20 > > There is a bug in dm-crypt that it doesn't account large pages in=20 > > cc->n_allocated_pages, this patch fixes the bug. This patch did not help. > If the previous patch didn't fix it, try this patch (on a clean upstream= =20 > kernel). > > This patch allocates large pages, but it breaks them up into single-page= =20 > entries when adding them to the bio. But this does help. > If this patch deadlocks, it is a sign that allocating large pages causes= =20 > the problem. >=20 > If this patch doesn't deadlock, it is a sign that processing a bio with= =20 > large pages is the problem. >=20 > Mikulas >=20 > --- > drivers/md/dm-crypt.c | 26 ++++++++++++++++++++++---- > 1 file changed, 22 insertions(+), 4 deletions(-) >=20 > Index: linux-stable/drivers/md/dm-crypt.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- linux-stable.orig/drivers/md/dm-crypt.c 2023-10-31 17:16:03.000000000= +0100 > +++ linux-stable/drivers/md/dm-crypt.c 2023-10-31 17:21:14.000000000 +0100 > @@ -1700,11 +1700,16 @@ retry: > order =3D min(order, remaining_order); > =20 > while (order > 0) { > + if (unlikely(percpu_counter_read_positive(&cc->n_allocated_pages) + (= 1 << order) > dm_crypt_pages_per_client)) > + goto decrease_order; > pages =3D alloc_pages(gfp_mask > | __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN | __GFP_COMP, > order); > - if (likely(pages !=3D NULL)) > + if (likely(pages !=3D NULL)) { > + percpu_counter_add(&cc->n_allocated_pages, 1 << order); > goto have_pages; > + } > +decrease_order: > order--; > } > =20 > @@ -1719,8 +1724,13 @@ retry: > =20 > have_pages: > size_to_add =3D min((unsigned)PAGE_SIZE << order, remaining_size); > - __bio_add_page(clone, pages, size_to_add, 0); > remaining_size -=3D size_to_add; > + while (size_to_add) { > + unsigned this_step =3D min(size_to_add, (unsigned)PAGE_SIZE); > + __bio_add_page(clone, pages, this_step, 0); > + size_to_add -=3D this_step; > + pages++; > + } > } > =20 > /* Allocate space for integrity tags */ > @@ -1739,13 +1749,21 @@ have_pages: > static void crypt_free_buffer_pages(struct crypt_config *cc, struct bio = *clone) > { > struct folio_iter fi; > + unsigned skip =3D 0; > =20 > if (clone->bi_vcnt > 0) { /* bio_for_each_folio_all crashes with an emp= ty bio */ > bio_for_each_folio_all(fi, clone) { > - if (folio_test_large(fi.folio)) > + if (skip) { > + skip--; > + continue; > + } > + if (folio_test_large(fi.folio)) { > + skip =3D (1 << folio_order(fi.folio)) - 1; > + percpu_counter_sub(&cc->n_allocated_pages, 1 << folio_order(fi.folio= )); > folio_put(fi.folio); > - else > + } else { > mempool_free(&fi.folio->page, &cc->page_pool); > + } > } > } > } --=20 Best Regards, Marek Marczykowski-G=C3=B3recki Invisible Things Lab --EW5Jd6ZInl+5b4q+ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmVC75AACgkQ24/THMrX 1yzj5gf/ToKBkxnC0lV27lc38Y5dkZb8PL1uH65jV1eoAUkXZxyaIBWniybQ8/Sk z2AnmGLZQ4B+3cPVGRVsRxdVyZKbj4/RLdZNJX9vYZgz6lLrM6/uFqqTeEetoDwP zMyiy/ZlA43O83rpnIqOCGxnuKxJAlLBPHPxcAUtHZNNFOT8nCs3Zc7sl8pSYcwp R5+u+lOVJVMTMkjOYLLHeighXpjROK62QHl36L/09op7yX+22A6QhNFgLJRlLxbm snQ4wRhJ3JlNDZMHm+7W5iXm/+TNzZJHCnAUY+ddSF57FGCHM8vnb8skkfHEKKUt VtkX13+bgQ6fJKTyDhGAGMxNFnWpmw== =DePr -----END PGP SIGNATURE----- --EW5Jd6ZInl+5b4q+--