From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DD561DD877 for ; Fri, 20 Feb 2026 01:22:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.46 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771550558; cv=none; b=WSGl2rmUX0LhBinfKkt3/o7r35lKvITQPB+NuEHlYiqcnxHY5P01S2hk6fK3mnzQQN33qtb+a40ogh6oRigP3VMv2lXE18SZAfFUrzVV5tvhgSiIgOTJ8gis+Tcy4u3dEzP63pJWX+wD29/5WXZo7WCdSF29T7noW3m5kq39VU4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771550558; c=relaxed/simple; bh=XdjGpx0Qf8T5eToMFvZb9NQzj30KOKeQT9Xa7s0aFaQ=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=aj41sCqSwMUxYC+gZ3/zS5BOE6yri6Sm0iv7uRTbxsYwmc6oSHF9lt7kL1FDVRQwC4y/5rVBv0HR3ih3bZTEXfkO/jLqOQkYaVyq61yfXO/X/SBGwlQjjS+iyEoUipoB9YgvpMkfHP3uoY/I754KHgNHnO9CHyNkpKAv4nbe9Jk= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dVM7dEok; arc=none smtp.client-ip=209.85.219.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dVM7dEok" Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-896f95e07f5so12400696d6.3 for ; Thu, 19 Feb 2026 17:22:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1771550556; x=1772155356; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=6ivmmj3rmqHiPEXxOwvvtMYJmo8FpOUGoOJc5c2N5Y4=; b=dVM7dEokZDm77+aEa4x2BRUsghFH5stFIfAMcV+Z7o2EeZxIl/YwuV+esbXML3oUaz NGBhdQ6LeRfXcRnpjjsWszIgNbubNUIR53ARX221cdGBGflYqnvl3v9divFzEIEJv0ot T5DYfdhG8E9Z/Zp3VycTHdz6wBolthT1ntNDObzGoYB5Yeo8Fny1eZFfTTx6oSvHHsma RnKDlWXeLVPijS3IXn7wIKGrDMIePImz355GDqrFSRBr21XqI+8WRDVnp4UuzE0bZ74/ 1ZRASFn7HamesbWqyJ8Gkf0daBpZEXKXh9Ozx3Nowyj2YYJRYG4EaEdWiDjhv5XUmCxU RshA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771550556; x=1772155356; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=6ivmmj3rmqHiPEXxOwvvtMYJmo8FpOUGoOJc5c2N5Y4=; b=M+ztd+LsXk2lSnVTa54m8Ke3UdjDGJnaWfpx3t1GGCaDYh7sXUL27mpetFNEbMr43Q UYtMoPBy1rGlg+kt0r5NXzUWUWnXiIXdb01RksDqwWfDU1qSdlQY98S2JnXU8XEAbmf8 jM9MY0ef0fLqycMYZEODwe6167nJIEUS8FzA056onKmcB3AEwJ+1Um9unX4WSGG2TdZh Jbhsyy21qKOAnOx/bUoDjxYeLr+B7U6bJFJlFPG0X9q2p/LiGQlLtD570ofViua6lKDI 6BEAqFssgL1gotwQ0URFPpAGj4kwiBexMoHP8255kQeKc11tey+mKkzsX33av4Cqt7j5 4X5A== X-Forwarded-Encrypted: i=1; AJvYcCXglt0Av0KXeDyeJsrvzjYa1sUNeSSl63DUfmaY9J+7SRSOTho6WLg9GjGOio38bHMB6uzrKqF8j73GMQE=@vger.kernel.org X-Gm-Message-State: AOJu0Yxdp0czstTRfXJ8AG1iuglBu7cl920kNVim/5iJUKgebemnVN0A ncW3Uo3D1d5qPaONsNz6kjsn6Hz1NSSZsJgoRnHjbbaE+rzxdy6npPAZ X-Gm-Gg: AZuq6aL75Uu7lTie1qH5lSnHvkBy1Ki0dNqxq4EwMdYvx+mtZX/ePPSCVDp7BSzfcXP zA+i/ZehNPj0q4lNY3kCPXvkfQr4Jxi8S45k58l2jQe5TzeJJcbxVS5nKZ2EXSJHBy8hKfy05SB T7sZdMo2E90LeaiMhi578teS3n1b2DFzhJt7MftYAgbqVxdOp83E6uI89yhLP3lJWjRDtUPXVDu dpcnZEHF8oUC5IWB+nhpUBlt9lqnOGzJyHQem/fzz+hLFmtr5BuBz+Ea5eprG31eTxOGv58kQcC wl2zopsixi9X5g2JmwJ39wz9oy5Kn99Zx4CmWJJCAK7T+DBpCu0R+Un1Eglnz+ti5dQ/eDyUJAc 8uNblwFeygNGlw4yWqgW/NDR+4i+PYzxNd9VJVY8LkFHd5lP8r1JQjKh6XTN31EFIpLkfpQSeu8 WGxhG22JT9OrBgsK1wlKVB2FYB/0cT0SqnYKTkB1IzUKhy X-Received: by 2002:a05:6214:d43:b0:894:64c1:6643 with SMTP id 6a1803df08f44-89971996be8mr965616d6.19.1771550556131; Thu, 19 Feb 2026 17:22:36 -0800 (PST) Received: from [10.69.55.177] ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8971cc95aa7sm276265386d6.17.2026.02.19.17.22.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 19 Feb 2026 17:22:35 -0800 (PST) Message-ID: <96ea8e82-ead3-498f-bb87-5b2809089950@gmail.com> Date: Thu, 19 Feb 2026 17:22:32 -0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 18/21] nvme: Update CCR completion wait timeout to consider CQT To: Mohamed Khalfella , Hannes Reinecke Cc: Justin Tee , Naresh Gottumukkala , Paul Ely , Chaitanya Kulkarni , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Aaron Dailey , Randy Jennings , Dhaval Giani , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, jsmart833426@gmail.com References: <20260214042753.4073668-1-mkhalfella@purestorage.com> <20260214042753.4073668-19-mkhalfella@purestorage.com> <9d6cf4f9-37d8-4704-bcc1-0b849ad28955@suse.de> <20260216184515.GH2392949-mkhalfella@purestorage.com> <3bad149e-a0fa-4377-9701-7b35ef6b5b88@suse.de> <20260217153530.GI2392949-mkhalfella@purestorage.com> Content-Language: en-US From: James Smart In-Reply-To: <20260217153530.GI2392949-mkhalfella@purestorage.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 2/17/2026 7:35 AM, Mohamed Khalfella wrote: > On Tue 2026-02-17 08:09:33 +0100, Hannes Reinecke wrote: >> On 2/16/26 19:45, Mohamed Khalfella wrote: >>> On Mon 2026-02-16 13:54:18 +0100, Hannes Reinecke wrote: >>>> On 2/14/26 05:25, Mohamed Khalfella wrote: >>>>> TP8028 Rapid Path Failure Recovery does not define how much time the >>>>> host should wait for CCR operation to complete. It is reasonable to >>>>> assume that CCR operation can take up to ctrl->cqt. Update wait time for >>>>> CCR operation to be max(ctrl->cqt, ctrl->kato). >>>>> >>>>> Signed-off-by: Mohamed Khalfella >>>>> --- >>>>> drivers/nvme/host/core.c | 2 +- >>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>> >>>>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c >>>>> index 0680d05900c1..ff479c0263ab 100644 >>>>> --- a/drivers/nvme/host/core.c >>>>> +++ b/drivers/nvme/host/core.c >>>>> @@ -631,7 +631,7 @@ static int nvme_issue_wait_ccr(struct nvme_ctrl *sctrl, struct nvme_ctrl *ictrl) >>>>> if (result & 0x01) /* Immediate Reset Successful */ >>>>> goto out; >>>>> >>>>> - tmo = secs_to_jiffies(ictrl->kato); >>>>> + tmo = msecs_to_jiffies(max(ictrl->cqt, ictrl->kato * 1000)); >>>>> if (!wait_for_completion_timeout(&ccr.complete, tmo)) { >>>>> ret = -ETIMEDOUT; >>>>> goto out; >>>> >>>> That is not my understanding. I was under the impression that CQT is the >>>> _additional_ time a controller requires to clear out outstanding >>>> commands once it detected a loss of communication (ie _after_ KATO). >>>> Which would mean we have to wait for up to >>>> (ctrl->kato * 1000) + ctrl->cqt. >>> >>> At this point the source controller knows about communication loss. We >>> do not need kato wait. In theory we should just wait for CQT. >>> max(cqt, kato) is a conservative guess I made. >>> >> Not quite. The source controller (on the host!) knows about the >> communication loss. But the target might not, as the keep-alive >> command might have arrived at the target _just_ before KATO >> triggered on the host. So the target is still good, and will >> be waiting for _another_ KATO interval before declaring >> a loss of communication. >> And only then will the CQT period start at the target. >> >> Randy, please correct me if I'm wrong ... >> > > wait_for_completion_timeout(&ccr.complete, tmo)) waits for CCR operation > to complete. The wait starts after CCR command completed successfully. > IOW, it starts after the host received a CQE from source controller on > the target telling us all is good. If the source controller on the target > already know about loss of communication then there is no need to wait > for KATO. We just need to wait for CCR operation to finish because we > know it has been started successfully. > > The specs does not tell us how much time to wait for CCR operation to > complete. max(cqt, kato) is an estimate I think reasonable to make. So, we've sent CCR, received a CQE for the CCR within KATO (timeout in nvme_issue_wait_ccr()), then are waiting another max(KATO, CQT) for the io to die. As CQT is the time to wait once the ctrl is killing the io, and as the response indicated it certainly passed that point, a minimum of CQT should be all that is needed. Why are we bringing KATO into the picture? -- this takes me over to patch 8 and the timeout on CCR response being KATO: Why is KATO being used ? nothing about getting the response says it is related to the keep alive. Keepalive can move along happily while CCR hangs out and really has nothing to do with KATO. If using the rationale of a keepalive cmd processing - has roundtrip time and minimal and prioritized processing, as CCR needs to do more and as the spec allows holding on to always return 1, it should be KATO+, where is no more than CQT. But given that KATO can be really long as its trying to catch communication failures, and as our ccr controller should not have comm issues, it should be fairly quick. So rather than a 2min KATO, why not 10-15s ? This gets a little crazy as it takes me down paths of why not fire off multiple CCRs (via different ctlrs) to the subsystem at short intervals (the timeout) to finally find one that completes quickly and then start CQT. And if nothing completes quickly bound the whole thing to fencing start+KATO+CQT ? -- james