From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 767CCCD11C2 for ; Wed, 10 Apr 2024 06:02:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:In-Reply-To:From:References:Cc:To:Subject:MIME-Version:Date: Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=d0M7jKhmdMOsWzkxHcH3wZyl/PZBBuzlnSutY/U1M5s=; b=4DCCQk9rIQsTFNY3rirPsh6VT2 6ixwG+1/pXKMaFG3dsfAtb72MCRO02a3CTUGs/jzaS9UCcQUD7cpgWJkJabGpVyMYVU3XmJP1HMIs 2MDA7jp+AQKLCr8ejGbrv1E2MsL8z/SZeJpDxbAMe3NSbkM9vefGk3pX1OysKK4RNlHzMgll0KEDX MALqGdLAl5IqMB9nQBY2QkPKhSlwCyILfNi3Z6Hjpuz+Nv7i4PDoRdFDR/Abbo0F9sI+Pp1/DB74z xckVbJEBOHQw5I0iarQ/RXwpxqDsgcAWeu3M4E2bj0s0sl7/RGRQlL4t51kSQn+0iuMMLltl33A79 4nUyNC6w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruR20-00000005I5c-3Hsp; Wed, 10 Apr 2024 06:02:00 +0000 Received: from smtp-out2.suse.de ([2a07:de40:b251:101:10:150:64:2]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1ruR1v-00000005I42-0uYm for linux-nvme@lists.infradead.org; Wed, 10 Apr 2024 06:01:58 +0000 Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id D0EE15C556; Wed, 10 Apr 2024 06:01:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1712728910; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0M7jKhmdMOsWzkxHcH3wZyl/PZBBuzlnSutY/U1M5s=; b=DrnA2H3ghV+heTuVaOtxI73/hJf8WSaFu97lNbJLTH0t97GQlwfbT0YmlFfXs3wldzpp8y zLDI68+iNlaQPggs2yt0ze31CBhbjKAfNnHEC289RyDQ3DRqqFBo2Pjd1HMB/mqATVBiWV WZsUKofuu+GJ4wfzxyRcj+cz/mQ5ixc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1712728910; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0M7jKhmdMOsWzkxHcH3wZyl/PZBBuzlnSutY/U1M5s=; b=x59rp1/Zq4R3FFovqSee2qBOr4y9xzyLtOuM/4eBWHigwhjL/At1aVhcTYkBba7RoO4KCM GSKE7w/kQGNRFhAw== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1712728910; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0M7jKhmdMOsWzkxHcH3wZyl/PZBBuzlnSutY/U1M5s=; b=DrnA2H3ghV+heTuVaOtxI73/hJf8WSaFu97lNbJLTH0t97GQlwfbT0YmlFfXs3wldzpp8y zLDI68+iNlaQPggs2yt0ze31CBhbjKAfNnHEC289RyDQ3DRqqFBo2Pjd1HMB/mqATVBiWV WZsUKofuu+GJ4wfzxyRcj+cz/mQ5ixc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1712728910; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d0M7jKhmdMOsWzkxHcH3wZyl/PZBBuzlnSutY/U1M5s=; b=x59rp1/Zq4R3FFovqSee2qBOr4y9xzyLtOuM/4eBWHigwhjL/At1aVhcTYkBba7RoO4KCM GSKE7w/kQGNRFhAw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 1965513691; Wed, 10 Apr 2024 06:01:49 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id 8/i3Ok0rFmYZFQAAD6G6ig (envelope-from ); Wed, 10 Apr 2024 06:01:49 +0000 Message-ID: Date: Wed, 10 Apr 2024 08:01:49 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v5 3/6] nvme-tcp: short-circuit reconnect retries Content-Language: en-US To: Sagi Grimberg , Daniel Wagner , Christoph Hellwig Cc: Keith Busch , James Smart , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org References: <20240409093510.12321-1-dwagner@suse.de> <20240409093510.12321-4-dwagner@suse.de> From: Hannes Reinecke In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-4.29 / 50.00]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; RCPT_COUNT_SEVEN(0.00)[7]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns] X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240409_230155_507177_EA9B8ECF X-CRM114-Status: GOOD ( 23.21 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org On 4/9/24 22:20, Sagi Grimberg wrote: > > > On 09/04/2024 12:35, Daniel Wagner wrote: >> From: Hannes Reinecke >> >> Returning an nvme status from nvme_tcp_setup_ctrl() indicates that the >> association was established and we have received a status from the >> controller; consequently we should honour the DNR bit. If not any future >> reconnect attempts will just return the same error, so we can >> short-circuit the reconnect attempts and fail the connection directly. >> >> Signed-off-by: Hannes Reinecke >> [dwagner: add helper to decide to reconnect] >> Signed-off-by: Daniel Wagner >> --- >>   drivers/nvme/host/nvme.h | 24 ++++++++++++++++++++++++ >>   drivers/nvme/host/tcp.c  | 23 +++++++++++++++-------- >>   2 files changed, 39 insertions(+), 8 deletions(-) >> >> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h >> index 9b8904a476b8..dfe103283a3d 100644 >> --- a/drivers/nvme/host/nvme.h >> +++ b/drivers/nvme/host/nvme.h >> @@ -701,6 +701,30 @@ static inline bool nvme_is_path_error(u16 status) >>       return (status & 0x700) == 0x300; >>   } >> +/* >> + * Evaluate the status information returned by the LLDD in order to >> + * decided if a reconnect attempt should be scheduled. >> + * >> + * There are two cases where no reconnect attempt should be attempted: >> + * >> + * 1) The LLDD reports an negative status. There was an error (e.g. no >> + *    memory) on the host side and thus abort the operation. >> + *    Note, there are exception such as ENOTCONN which is >> + *    not an internal driver error, thus we filter these errors >> + *    out and retry later. >> + * 2) The DNR bit is set and the specification states no further >> + *    connect attempts with the same set of paramenters should be >> + *    attempted. >> + */ >> +static inline bool nvme_ctrl_reconnect(int status) >> +{ >> +    if (status < 0 && status != -ENOTCONN) >> +        return false; > > So if the host failed to allocate a buffer it will never attempt > another reconnect? doesn't sound right to me.., Memory allocation errors are always tricky. If a system is under memory pressure yours won't be the only process which will exhibit issues, so it's questionable whether you should retry, or whether it wouldn't be safer to just abort the operation, and retry manually once the system has recovered. Also we just have the interesting case where RDMA goes haywire and reports a log buffer size of several TB. No amount of retries will be fixing that. Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich