From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79A1FC4167D for ; Fri, 3 Nov 2023 15:02:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA8538D00BE; Fri, 3 Nov 2023 11:02:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A0B608D000C; Fri, 3 Nov 2023 11:02:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E9578D00BE; Fri, 3 Nov 2023 11:02:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 612A68D000C for ; Fri, 3 Nov 2023 11:02:06 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0E8FB160F72 for ; Fri, 3 Nov 2023 15:02:06 +0000 (UTC) X-FDA: 81416958252.21.38C1C6D Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) by imf07.hostedemail.com (Postfix) with ESMTP id 7E605400FD for ; Fri, 3 Nov 2023 15:01:29 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=invisiblethingslab.com header.s=fm3 header.b=w96ZssLt; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="Tc3yw/8i"; spf=none (imf07.hostedemail.com: domain of marmarek@invisiblethingslab.com has no SPF policy when checking 64.147.123.24) smtp.mailfrom=marmarek@invisiblethingslab.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1699023690; a=rsa-sha256; cv=none; b=MWLN5sJoJegaKZrWV7xrYmsbavepWHvv0yEN954t5objIz55LwYXPzSTH/3fipyWPEwOYU 77VIsGJWAOBUCCj/LTyj3M5lQTFRozumNvN8JG0aOMFx6Ggrl7gkdpo3GX8TJHow7m2Qxx uoZxbJH1UG1q5UUqbWZtmgv2znfyCp4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=invisiblethingslab.com header.s=fm3 header.b=w96ZssLt; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="Tc3yw/8i"; spf=none (imf07.hostedemail.com: domain of marmarek@invisiblethingslab.com has no SPF policy when checking 64.147.123.24) smtp.mailfrom=marmarek@invisiblethingslab.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1699023690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8rI9gXS7PRI2oTRO29ALzS6qReZNHb1zj/hti9DlwAo=; b=4fiLQq0c9O+KAdys14iNFjSiMAt8BCITBu0IRO+XKKJlpODnXIyZ4Qgulki5NvjA3yokTS pEP/mjk8eT34ztvCStX1IbIfixyoxnRfCfsO2b7aF1ez0fyx+Ww9eZhew0luIVSx4Z2qtC qwx+hWA3SuxrrpkI6Tp+WFe7E65qCDc= Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id A1DF33200921; Fri, 3 Nov 2023 11:01:26 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 03 Nov 2023 11:01:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1699023686; x=1699110086; bh=8rI9gXS7PRI2oTRO29ALzS6qReZNHb1zj/h ti9DlwAo=; b=w96ZssLttYEYRJHkH7AbUgFmzxSm9R0UX6HonBkBV4JTc09juJ6 zOs4kpvDt0Rjllhy8Mg/lp/LedBBMS0wa4Hx33ybYAw2Uq8RIztol1nuJ3GL7qch SKhUIIzo3/uUbuOhuPz8tqQ2v/NRIH4eBmKJCQrWCh1NTaa+Y0z7cRikixE8R4dl XBz50Uvu3TD6qtDQ7gxkYbUp+gd9nRhDI3GO0XNMXNFX+CLmzKukfO1dlfa5R4OK zUT8JQe1fhX8tVgoEZjlhDRSoPTZ4Co/Q33zIGXT8N9A+YFmfXwblI4VptLvv+VT DMFwJQnzbxxY4CR8kYLQc+Yw5pux+IIH7Ag== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1699023686; x=1699110086; bh=8rI9gXS7PRI2o TRO29ALzS6qReZNHb1zj/hti9DlwAo=; b=Tc3yw/8iSXzh40pTuZcHpImoly+ew 7HN9xOokJDzzlbawsfta1TmGebJlQdCwJcxPuPPR9Utqt4rTwET6GYSHyguvR1er kftDET/nagoXCad56oCIQLVXQWPuoPfdI05VFnQDAVnqKZixUEz+rwib2jngjURu HkZS3ZpMjbRJNoyUl3TN26KD6S6UZ+0EYwDkjqp6DyaX0bt3TFiL5n0g/6DvfH/L 1xaiyTYNPoX+tvWb6dVK3XRPooCkeBqK8G29bljJvMTN2R7WwuD/WeJ3DMo9AYM0 Q1mOeQ+ucdeWPqW9pYcjOg7QbtL4pX2hEEZhudUeI5HkbULGjlnWGyG7Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedruddtkedgjedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtjeenucfhrhhomhepofgrrhgv khcuofgrrhgtiiihkhhofihskhhiqdfikphrvggtkhhiuceomhgrrhhmrghrvghksehinh hvihhsihgslhgvthhhihhnghhslhgrsgdrtghomheqnecuggftrfgrthhtvghrnhepgfdu leetfeevhfefheeiteeliefhjefhleduveetteekveettddvgeeuteefjedunecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghk sehinhhvihhsihgslhgvthhhihhnghhslhgrsgdrtghomh X-ME-Proxy: Feedback-ID: i1568416f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 3 Nov 2023 11:01:22 -0400 (EDT) Date: Fri, 3 Nov 2023 16:01:19 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Mikulas Patocka Cc: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 Message-ID: References: <98aefaa9-1ac-a0e4-fb9a-89ded456750@redhat.com> <20231031140136.25bio5wajc5pmdtl@quack3> <8a35cdea-3a1a-e859-1f7c-55d1c864a48@redhat.com> <3cb4133c-b6db-9187-a678-11ed8c9456e@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="jJXFS76+C6nnDWam" Content-Disposition: inline In-Reply-To: <3cb4133c-b6db-9187-a678-11ed8c9456e@redhat.com> X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 7E605400FD X-Stat-Signature: dssg6cr4jmt3aqyxq1fithhg6ptzgfk5 X-Rspam-User: X-HE-Tag: 1699023689-242211 X-HE-Meta: U2FsdGVkX19EpG0rtbdd0Z0DA9jxiGM3rpJmYu2FuMpgheOKconK/P3jqg+BtxygUtrxXl1Uy9YaXr01RljAF1/1/yMTLB++5FRqMGC7o5g94RFaInAGDfRg++4maftKo2Pff3ljUU4MdJINpYWRgWMxMttYGieLY+8K0UAv66jawwlrkDP/AfFr5zJ+a+oHCw5gkJX8ebD8AyXGpKhd7acArygrtpwbccnRV88+LAvTykFJwYEqH12OYbi4TElOBP0WimcRB/K0hdhCspVw8g5eUfxqs4X2qwQItoxTvkeP+d9tEJ0Lc/W2M/5OucNNfykiUbEx0eFOe7MVn3eFldJLhnGGfp6jR4vA7Ix3VXgwEKyAr2rLojGtL5s7UWmiMKE6CHuZrNHpN26T7rQ54yQj6Nz4+Oq7Hz5g7evgjpzzJjWz2QUVzZpRWHKc7WT+3HqVBdnQovZBRsbSljp3V+eZBGlY8uCWKlK5ynHWzwKpWpktcTz6LS71Ih1jnMyCMnzPVyIzQVusg1J3kDDdqny09ojg/3tuuL5HGlZsrCyznn0rIniprBwMvR4PabU7TVgGYbnXLXhQlY1Bb4n/ZWP3Q8miHV/p7M5zHpueTojJzwFf0sO8BYTwhi6oXjqx4fqcnZRLa46pLlBdSbMp0PwOxJjqV3hFvsbld6yz7jOKSyAxX8T+zfAzk5nQkSMgnLf4Qnf2B72BoNxIjHy7GZrg8Y69Z0GQVrFo6UIDAa2Ro6pFPQ76qVVphJAvInCoJAiIot6EQTGfL9RoUWGvENZFb/hKK5Ge6JUsRNJlTeI0fyWAr7r76njIM3nHMveBm05pvbH22cBt4cba7Tn/A9g5c8VzjcqFxN+AY7crGUHnUREyg1SzKDqq3nO+JaWqw3lSAZ09eRwGaI7LpKimhNE11fONcX29PGoi9cP8KnzS6cOhYZyLhk4QvJ4PJIsaOYPKdQ3GFxPvGjCQZad XkU1mOP6 /Y/eUPTw2IYyWe7ZDVhWMczTKOZZe/gH+S+aqqRTHqiHEn+eicut1gVgndnRWKSQ25FTvnH+tCFBkUZSP101fFp2ofZIRGZ6h2fW5STSUqfzShxDgw4vdbzIgL9wq+z4WXT2leKQAI2yEi6Vf+g67ggkYkDbCQ0lpGQSyGBMVEGEnKBHiTL1b7aKZxCyU6HlsSUuV9Qe/oYNqdBcfrYNjGw+jrfwcWJ8fROJ4V5spJeA7KwPthcQaFV9r6IuZ19zhdrrsbiU2gzbBPRMJhlxABOrkwghWhuDN4v698QwKwLYpQsUbVXdvf2tGO/jFj49/aDIhauA7j8h/QQorGMeWwtuWH69z2e3RKeftVnPqaR6MXymamcQq+TmzK2yDX63WCkw3+Fo4YVBQLYviCtpkLSI5vyj88PhniOgGRPo9ilT453zNOKABDfveCT62ztUgAIEV6Z5aj+5CorE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: --jJXFS76+C6nnDWam Content-Type: text/plain; protected-headers=v1; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Date: Fri, 3 Nov 2023 16:01:19 +0100 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Mikulas Patocka Cc: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 On Thu, Nov 02, 2023 at 06:06:33PM +0100, Mikulas Patocka wrote: >=20 >=20 > On Thu, 2 Nov 2023, Marek Marczykowski-G=C3=B3recki wrote: >=20 > > On Thu, Nov 02, 2023 at 10:28:57AM +0100, Mikulas Patocka wrote: > >=20 > > > Try lowring /sys/block/nvme0n1/queue/max_sectors_kb to some small val= ue=20 > > > (for example 64) and test if it helps. > >=20 > > Yes, this helps too. >=20 > On a plain upstream kernel with no other modifications (and with default= =20 > max_sectors_kb), set the value /sys/module/nvme/parameters/sgl_threshold= =20 > to "0" and test it if it deadlocks. Then, set this value to "1" and test= =20 > it again. Got deadlock wit both values. > Revert sgl_threshold back to the default (32768). Boot the kernel with th= e=20 > option "iommu=3Dpanic". Reproduce the deadlock and if you get a kernel=20 > panic, send us the panic log. This is a Xen PV, so Linux is not in charge of IOMMU here. And there is SWIOTLB involved (64MB of it), I'm not sure if for every DMA, but definitely for some.=20 > Then, try this patch (without "iommu=3Dpanic"), reproduce the deadlock an= d=20 > tell us which one of the "printk" statements is triggered during the=20 > deadlock. I'll try this next. --=20 Best Regards, Marek Marczykowski-G=C3=B3recki Invisible Things Lab --jJXFS76+C6nnDWam Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAmVFC0AACgkQ24/THMrX 1yxTwAf/SFlkRMbe2ZO3nNL4wgyZQkhMn5+snIYJwfkUhabco4QTc1GHGfsaMD7N fjghgVc3TYnCGTU/vIMZlZg+zv+6Ol2Sp2XIoglAX7EhpHuHdgb6X+RnOk0jNvBr BqvMILFZdElEuB260lKCrH1xV2CGDo+MmvU18UszgOmy3uMR8Pb6b493ckklabyC w+p80dzKcjNSKzSpMtjU/w21s0dWgFGoSouz9xqtQFWQEHjLw8TptA3+qSSFl/Yq VlTy+zGB+Buj3qMOzXW97roTJH8KxI7I9wdbKboZWq3EXH7eFD9t6k000i72z/6T qTVgLfU0itg1QJHAFMcAV0LsZKkEJA== =bk7Z -----END PGP SIGNATURE----- --jJXFS76+C6nnDWam--