From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8513C4167B for ; Thu, 2 Nov 2023 17:06:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D78168002F; Thu, 2 Nov 2023 13:06:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D27FD8D000F; Thu, 2 Nov 2023 13:06:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF0168002F; Thu, 2 Nov 2023 13:06:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AD1FC8D000F for ; Thu, 2 Nov 2023 13:06:43 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 828FC120DFD for ; Thu, 2 Nov 2023 17:06:43 +0000 (UTC) X-FDA: 81413643486.03.482F9CD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 86C8C180009 for ; Thu, 2 Nov 2023 17:06:40 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XkKUwTVZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of mpatocka@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mpatocka@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698944800; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0UeuJsv4XGxr7QngFqmHvnq1CZ9BvM92/cz1tGmQL4A=; b=eWkWsss4bOnXTfYzXqCMQJCkbYxIdlXOh2IymMRktlUyv3nssmWnHRJHW8MSvRRzp0TpN6 8jfsIVnQSMSrKIhJxWJYAhPyUIS4tyeF6tEIhPQSMvR+ST3Gss7pig0oYpC9PENin7ig2n EhJDPEawrZf2BgY3/PGJmYwv7W2MJJc= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XkKUwTVZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of mpatocka@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mpatocka@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698944800; a=rsa-sha256; cv=none; b=bI4dWcr42KwLTQyWYatZUs2dm5YIQGr4Zfksgos/XqhQqEnTSrD10dwaZcrXjmu1B/tICP HwyRc/BbimqTBbTAEaI8kpo2Fw3hdye8f8OuOkGKvorpNYmDfZnypJ3Q/P8+z0uE994I3I A1cauen9rzO1YkHE2Nzgmx50GlX7JAU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1698944799; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=0UeuJsv4XGxr7QngFqmHvnq1CZ9BvM92/cz1tGmQL4A=; b=XkKUwTVZ+Q+6JlPS9l55siEryOlFqqRyF01TspKL+wfw5ySLnevYKpzAUj7cQBXeAOeHR5 g7PBPJSqeuoSyFVMIzAMTLhQtYMaTL0HKfpxHCRNF8jKTLwRlEYzoYNOIZZScI4qXVJj9j MvtxOQEaG+9GmQka7dv2lVkn8B3HwCs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-573-8By9kwQ-OE-0s75BmH0nag-1; Thu, 02 Nov 2023 13:06:34 -0400 X-MC-Unique: 8By9kwQ-OE-0s75BmH0nag-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 51E9A85A58C; Thu, 2 Nov 2023 17:06:33 +0000 (UTC) Received: from file1-rdu.file-001.prod.rdu2.dc.redhat.com (unknown [10.11.5.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2B13B1C060BA; Thu, 2 Nov 2023 17:06:33 +0000 (UTC) Received: by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix, from userid 12668) id 1369D30C72A4; Thu, 2 Nov 2023 17:06:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by file1-rdu.file-001.prod.rdu2.dc.redhat.com (Postfix) with ESMTP id 0ED763FD17; Thu, 2 Nov 2023 18:06:33 +0100 (CET) Date: Thu, 2 Nov 2023 18:06:33 +0100 (CET) From: Mikulas Patocka To: =?ISO-8859-15?Q?Marek_Marczykowski-G=F3recki?= cc: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Jan Kara , Vlastimil Babka , Andrew Morton , Matthew Wilcox , Michal Hocko , stable@vger.kernel.org, regressions@lists.linux.dev, Alasdair Kergon , Mike Snitzer , dm-devel@lists.linux.dev, linux-mm@kvack.org Subject: Re: Intermittent storage (dm-crypt?) freeze - regression 6.4->6.5 In-Reply-To: Message-ID: <3cb4133c-b6db-9187-a678-11ed8c9456e@redhat.com> References: <20231030155603.k3kejytq2e4vnp7z@quack3> <98aefaa9-1ac-a0e4-fb9a-89ded456750@redhat.com> <20231031140136.25bio5wajc5pmdtl@quack3> <8a35cdea-3a1a-e859-1f7c-55d1c864a48@redhat.com> MIME-Version: 1.0 Content-Type: multipart/mixed; BOUNDARY="185210117-1922301100-1698942373=:543920" Content-ID: <752ad75f-d113-13e4-4fc9-bfe050665852@redhat.com> X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-Rspamd-Queue-Id: 86C8C180009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: a39qo38q6ghcugm1bahukb6qjw36z3xk X-HE-Tag: 1698944800-839157 X-HE-Meta: U2FsdGVkX197qfozQlDeoIYDWXd8XvIE3pv4eqaE9vx+cEZ2MVrZiC1m7L8wRzil2ukRpYVzoLOC7NOoLnSB6X0k+n/Lp67+Uq740bRRhF7v8y/D5wvywu8NpR1uz3WmnwXnW8xNrovfoKsjx3viob6sr4N8yMy91bN56VjeP4qBLWvcBUYzJT+l/QBiaGHUHBQPjYBNLJ1uVzaezrtJ2QSpAhar9IuOC3tRb9qJpTlzGNRfDejtNbj7pFJ+0c7zaVU/FJyGnST5rmoToKe3ni316EzD/JJxCr4VPpms5HfN0jB3EdZuOyYjSQl+P3lTmxK14+Om16ltXEum8clS7a14Z+1pKn1TBK4YRZL9/87q3nl+OkRpa9ud6vG0wF6H2bbmHlyo9gdUVJVUW63u+WiwumifMKzNArsHHf1qs3q+Od2v5IoGV7Pl6NWrhxnNG5mbnsmM68mmgK8g7paxYFUEcFTTOGakBiW4IYD4MkdpMjOssqMfVydzemMwOWa0PeG84+85V5lc0kMQgLV16U2aNnHMLN9haGlNivTKbtejcSuyFBs+uDwnDDvtLduiHhVT5jCyNtw3XeufbxBg5zeKvDTFBoiqoOlNH+MIQUOSJ8cH2f9nWSkAMUdDX8gvP2usOBxK3vkOJB8Zn99G2Xrz6/NPNS+axSX5uaIG0AyY0CHA+0MmftatPuyP4np4yMBFLzW1fpTqAweIJFnG5zuXkZ65G00BbZgMC+2/eo9Fum4n9ljCYjOYkCftRXqAfD6oGrOT+QUUKN4YjimeHe6K7iX4NkRqbRXsSD6q4RvEoYiPqJ9PTD5fpmX/yS1P38p7QY+LieBsQDTf5XTBtc5/iqujUdfVxOV0mb0R7jOfIXSq7lkdeScMBNjba9H/SfirnSEjFYbqCUCFMBaWHFRKgxH1QYJZiYixuSWCLIrMRkJ/yJhbdsV6d2osYW9kV2LZxhCH5P6wlFjJZCN 4j+qh6pT teAhFMzavYP0E1G8Of0GfesQTazSo45FAylla0zFpp5s3ZZKZxEhMTDi4eZzxaUtiCiatq7z7EVXwIiV1FDZCD3ZQ1THQMbPNXv+D4iAfljxiJoVmkFcX9K9qRj8oXQFcnQjIsMud+DviWqqeqhk4IpoUAbpRcseRm9oXG3tyj0IqyIxAAcoGyxsBQ5MPONOBVDRrNYvpB4sTr7UG9bjARZqVFAc7fteRslIrAYjhQZZ/hxa9a83Gzd6aSGJtOBBd08iJIV+BjqoPddNhpxSykBnjGg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --185210117-1922301100-1698942373=:543920 Content-Type: text/plain; CHARSET=ISO-8859-15 Content-Transfer-Encoding: 8BIT Content-ID: <84311bc9-9f3-6568-af7e-549ca0e53215@redhat.com> On Thu, 2 Nov 2023, Marek Marczykowski-Górecki wrote: > On Thu, Nov 02, 2023 at 10:28:57AM +0100, Mikulas Patocka wrote: > > > Try lowring /sys/block/nvme0n1/queue/max_sectors_kb to some small value > > (for example 64) and test if it helps. > > Yes, this helps too. On a plain upstream kernel with no other modifications (and with default max_sectors_kb), set the value /sys/module/nvme/parameters/sgl_threshold to "0" and test it if it deadlocks. Then, set this value to "1" and test it again. Revert sgl_threshold back to the default (32768). Boot the kernel with the option "iommu=panic". Reproduce the deadlock and if you get a kernel panic, send us the panic log. Then, try this patch (without "iommu=panic"), reproduce the deadlock and tell us which one of the "printk" statements is triggered during the deadlock. Mikulas --- drivers/nvme/host/core.c | 8 ++++++-- drivers/nvme/host/pci.c | 27 ++++++++++++++++++++++----- 2 files changed, 28 insertions(+), 7 deletions(-) Index: linux-stable/drivers/nvme/host/pci.c =================================================================== --- linux-stable.orig/drivers/nvme/host/pci.c 2023-10-31 15:35:38.000000000 +0100 +++ linux-stable/drivers/nvme/host/pci.c 2023-11-02 17:38:20.000000000 +0100 @@ -622,6 +622,10 @@ static blk_status_t nvme_pci_setup_prps( prp_list = dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); if (!prp_list) { iod->nr_allocations = -1; + if (nprps <= (256 / 8)) + printk("allocation failure at %d\n", __LINE__); + else + printk("allocation failure at %d\n", __LINE__); return BLK_STS_RESOURCE; } iod->list[0].prp_list = prp_list; @@ -631,8 +635,10 @@ static blk_status_t nvme_pci_setup_prps( if (i == NVME_CTRL_PAGE_SIZE >> 3) { __le64 *old_prp_list = prp_list; prp_list = dma_pool_alloc(pool, GFP_ATOMIC, &prp_dma); - if (!prp_list) + if (!prp_list) { + printk("allocation failure at %d\n", __LINE__); goto free_prps; + } iod->list[iod->nr_allocations++].prp_list = prp_list; prp_list[0] = old_prp_list[i - 1]; old_prp_list[i - 1] = cpu_to_le64(prp_dma); @@ -712,6 +718,7 @@ static blk_status_t nvme_pci_setup_sgls( sg_list = dma_pool_alloc(pool, GFP_ATOMIC, &sgl_dma); if (!sg_list) { iod->nr_allocations = -1; + printk("allocation failure at %d\n", __LINE__); return BLK_STS_RESOURCE; } @@ -736,8 +743,10 @@ static blk_status_t nvme_setup_prp_simpl unsigned int first_prp_len = NVME_CTRL_PAGE_SIZE - offset; iod->first_dma = dma_map_bvec(dev->dev, bv, rq_dma_dir(req), 0); - if (dma_mapping_error(dev->dev, iod->first_dma)) + if (dma_mapping_error(dev->dev, iod->first_dma)) { + printk("allocation failure at %d\n", __LINE__); return BLK_STS_RESOURCE; + } iod->dma_len = bv->bv_len; cmnd->dptr.prp1 = cpu_to_le64(iod->first_dma); @@ -755,8 +764,10 @@ static blk_status_t nvme_setup_sgl_simpl struct nvme_iod *iod = blk_mq_rq_to_pdu(req); iod->first_dma = dma_map_bvec(dev->dev, bv, rq_dma_dir(req), 0); - if (dma_mapping_error(dev->dev, iod->first_dma)) + if (dma_mapping_error(dev->dev, iod->first_dma)) { + printk("allocation failure at %d\n", __LINE__); return BLK_STS_RESOURCE; + } iod->dma_len = bv->bv_len; cmnd->flags = NVME_CMD_SGL_METABUF; @@ -791,8 +802,10 @@ static blk_status_t nvme_map_data(struct iod->dma_len = 0; iod->sgt.sgl = mempool_alloc(dev->iod_mempool, GFP_ATOMIC); - if (!iod->sgt.sgl) + if (!iod->sgt.sgl) { + printk("allocation failure at %d\n", __LINE__); return BLK_STS_RESOURCE; + } sg_init_table(iod->sgt.sgl, blk_rq_nr_phys_segments(req)); iod->sgt.orig_nents = blk_rq_map_sg(req->q, req, iod->sgt.sgl); if (!iod->sgt.orig_nents) @@ -801,8 +814,12 @@ static blk_status_t nvme_map_data(struct rc = dma_map_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), DMA_ATTR_NO_WARN); if (rc) { - if (rc == -EREMOTEIO) + if (rc == -EREMOTEIO) { + printk("allocation failure at %d\n", __LINE__); ret = BLK_STS_TARGET; + } else { + printk("allocation failure at %d\n", __LINE__); + } goto out_free_sg; } Index: linux-stable/drivers/nvme/host/core.c =================================================================== --- linux-stable.orig/drivers/nvme/host/core.c 2023-10-31 15:35:38.000000000 +0100 +++ linux-stable/drivers/nvme/host/core.c 2023-11-02 17:12:39.000000000 +0100 @@ -708,8 +708,10 @@ blk_status_t nvme_fail_nonready_command( ctrl->state != NVME_CTRL_DELETING && ctrl->state != NVME_CTRL_DEAD && !test_bit(NVME_CTRL_FAILFAST_EXPIRED, &ctrl->flags) && - !blk_noretry_request(rq) && !(rq->cmd_flags & REQ_NVME_MPATH)) + !blk_noretry_request(rq) && !(rq->cmd_flags & REQ_NVME_MPATH)) { + printk("allocation failure at %d\n", __LINE__); return BLK_STS_RESOURCE; + } return nvme_host_path_error(rq); } EXPORT_SYMBOL_GPL(nvme_fail_nonready_command); @@ -784,8 +786,10 @@ static blk_status_t nvme_setup_discard(s * discard page. If that's also busy, it's safe to return * busy, as we know we can make progress once that's freed. */ - if (test_and_set_bit_lock(0, &ns->ctrl->discard_page_busy)) + if (test_and_set_bit_lock(0, &ns->ctrl->discard_page_busy)) { + printk("allocation failure at %d\n", __LINE__); return BLK_STS_RESOURCE; + } range = page_address(ns->ctrl->discard_page); } --185210117-1922301100-1698942373=:543920--