From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f45.google.com (mail-ej1-f45.google.com [209.85.218.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 889F721FF5F for ; Thu, 4 Dec 2025 19:58:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764878286; cv=none; b=Do28shmlxDCkY+IChDtiPGe4b1n6fgoKZ8OmwdTu8ZFVN0/+24WEh9OUDs6p9XLK0Fj08w24KQ7j/sJ1lYqCe38h/ueNEFXU7IWt9yKxbDD4dvVp5Qwbupeksy3AIMQ27NOkxbVbUbDFjlel05/LrmLP9l2njjMLEtuAUDPSNYY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764878286; c=relaxed/simple; bh=A6Ci7mAh2AbmOa9EDiuaP1L8vpxkl+Y7UtFtcBqG99M=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=G+pVUwo2d+WB8yJ5w0UDzZTPUzrwTQ5P9pPZ8V3R2TGnfVVoi1Zuls52unjYTot+l30u16y8aNIeRrV1ho/XIctT/F1Tb+gXSl/uGACl60qxtLYSADNhNhRLJe460XH8qjAshDMZW5xz4OWnzpAJDEqOJRel047zkNFkftHHpHM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=A6FzshQ3; arc=none smtp.client-ip=209.85.218.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="A6FzshQ3" Received: by mail-ej1-f45.google.com with SMTP id a640c23a62f3a-b79af62d36bso234699966b.3 for ; Thu, 04 Dec 2025 11:58:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1764878283; x=1765483083; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=K85hLvlaLvTs7QWXRfbHiJadW8G+Kx4QGeXhEF8abl0=; b=A6FzshQ3oBiIAG5NpYTfI7c9E1Abn8PkC90gunJV6uxGAYn6BwgXVnTEJGwha3c99h Rr/NIYUsZT8htXzZ3NBZlMg/oM5tFMzUOcfChVwZwU5vAojZ5pu13ge69eqaOQx9hhz4 fDcj70QfcBRmJlKb5KYZIsarxx92BT0Lnv3/PWnRN532aHFk2bl5FVfSl4p/SRwPPVzt OGxkyeP97+3Yr/dpyR86lPOhyo0AxuwGIRUs0qZiPrcqZ6qq2b079mUKU5K6njS8X+g2 kzp9eR5lOZBWBEM79RbrhoMEyAjX74dS9BS3FXncBbQvgjU0gSAUtNQRTK8lQVAxyn6I umTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764878283; x=1765483083; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-gg:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K85hLvlaLvTs7QWXRfbHiJadW8G+Kx4QGeXhEF8abl0=; b=CllPNnhppPus8vMeXBQxygI6JkxfxlBFVR14UK6CtaNJeFAIQ8GtLV1iI088Z+sV6j 68I6ALmOZWA+zk1LqBuNJufg7RwMTtrTmGtKClveXZIaaFO9lc4K9UFOZB/eI9u6NVTr uvuwx9diT51Ql+/pc5S+ZPZYQ2+TDDgFEPD3S1MW28n1Lc/XwQbLuNPoZxWuWQ4feWkG cmsIMGOXP1sN9u0xvnAdR4p8VWoweqtIx8I6fpNYUNbDdEXc16mv6ttchIPO8s+WGXRU fPXJWr3ygYh7xc6a64h7KkBTLSa9nBjZAC/9n+R22LoAWZkbmqaphGmWsyzrcVfV2cRi GMyg== X-Forwarded-Encrypted: i=1; AJvYcCUo1IiM7Z96cXdMp4c9pNo99vUthtmfSp4HigjHqOlCt8EJkRPhArQyFSo8OQAqefuKNdIr2I9sz0pFMO4=@vger.kernel.org X-Gm-Message-State: AOJu0Yzm4E/eQhkv7pfloZfqcBd1UkzrN+tX0JSYyvtTrrxTGqIE7tfN V8KT91zi5ef6mk0NSwdnlrBeZwXLWS8eQpCdDLQxbXMUW4dYXRrE1XJ4qjm/qx5H1RA= X-Gm-Gg: ASbGncvdFNT2jXNbpwvX6JTtx5AE6OYQKMZVTIlQRpIn3ylWTrOmO+TNgaCoOdrSemX 64UrK3bVvOMIY5N4BucKsJ6/svjKcMlxjfSvG2e5xENukMJIceXQZ1vcZoii5pTLn2C43btNCx1 gHCN8Z9dD/q+6z9gACBW/RjJzNBJ/XQFzYceI5W16DaNOTZWOELIqq+iWSsYL0TEtlj6iFaTDrz XxpjIIfFXIjmBazkjL+onOIeXR9c6bf1NWBsABR7XIA18jWsszQ/oaQkILXrx49M4xlZk4eR+gA enAgUnyS+qH+w3LsJhiAFrXSB8VnWmmSlvgZ+Ycp4KJK0M0xsuptEBp6BqugOpXY+fZqzk3Dfw3 wN+fOeEMIavjqUexZmYXNHcxgCuXys4eFi8EsQ7i83zrjxFJ2uNP6Cn0EgfA8smg02vZ3JA7qDJ 7Z+X7L3PSR3nvhEYAqInvV3l0POqUiy4k= X-Google-Smtp-Source: AGHT+IHUPzhRdk1tJWg+WYux5jYGrf/7CZPDqfa+5xnHk5ZwVdIiMDev2CwK7tTMHnl9R6P38XGmgw== X-Received: by 2002:a17:907:3d8f:b0:b73:9a71:13bb with SMTP id a640c23a62f3a-b79ec67421bmr451758966b.32.1764878282667; Thu, 04 Dec 2025 11:58:02 -0800 (PST) Received: from medusa.lab.kspace.sh ([208.88.152.253]) by smtp.googlemail.com with UTF8SMTPSA id a640c23a62f3a-b79f4976027sm193456466b.39.2025.12.04.11.58.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Dec 2025 11:58:02 -0800 (PST) Date: Thu, 4 Dec 2025 11:57:59 -0800 From: Mohamed Khalfella To: Bart Van Assche Cc: Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Casey Chen , Yuanyuan Zhong , Hannes Reinecke , Ming Lei , Waiman Long , Hillf Danton , linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 1/1] block: Use RCU in blk_mq_[un]quiesce_tagset() instead of set->tag_list_lock Message-ID: <20251204195759.GC337106-mkhalfella@purestorage.com> References: <20251204181212.1484066-1-mkhalfella@purestorage.com> <20251204181212.1484066-2-mkhalfella@purestorage.com> <5450d3fa-3f00-40ae-ac95-1f08886de3b6@acm.org> <20251204184243.GZ337106-mkhalfella@purestorage.com> <71e9950f-ace7-4570-a604-ceca347eea20@acm.org> <20251204191555.GB337106-mkhalfella@purestorage.com> <77c5c064-2539-4ad9-8657-8a1db487522f@acm.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <77c5c064-2539-4ad9-8657-8a1db487522f@acm.org> On Thu 2025-12-04 09:31:55 -1000, Bart Van Assche wrote: > On 12/4/25 9:15 AM, Mohamed Khalfella wrote: > > The stacktraces are from old 6.6.9 kernel. > > Please always include stack traces from a recent upstream kernel in > patch descriptions. > Good point. Will do that in next version of the patch. > > However, the issue is still > > applicable to recent kernels. This is an example from 6.13 kernel. > > Thanks, these stack traces make it clear what is causing the deadlock. > > From nvme_timeout(): > > /* > * Reset immediately if the controller is failed > */ > if (nvme_should_reset(dev, csts)) { > nvme_warn_reset(dev, csts); > nvme_dev_disable(dev, false); > nvme_reset_ctrl(&dev->ctrl); > return BLK_EH_DONE; > } > > Is my understanding correct that the above code is involved in the > reported deadlock? If so, has it been considered to run the code inside > the if-statement asynchronously (queue_work()) instead of calling it > synchronously? Would this be sufficient to fix the deadlock? > Yes, the above code is involved in the deadlock. I do not see how running this code in another thread will solve the problem. It will still cause a deadlock between blk_mq_quiesce_tagset() and blk_mq_del_queue_tag_set(). The later is holding the mutex and while waiting for the queue to be frozen. The former wants the mutex in order to make progress and cancel inflight requests to let the queue to be frozen. I do not see how this will make a difference. > Thanks, > > Bart.