From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6025B3B29E; Mon, 11 Aug 2025 13:03:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754917397; cv=none; b=WSq6EQEg02RpxETVbsvj92c5I61LjS5TAcS9xDuElWSNiVmhizP27gvRPkIhSWrVPmztL3EWt2ys90pXoVIEYQhvyrJxjU5c5ikcJKVNIxWucJlhs0bmKlhM86rzO9HvuWC2ApTRCbHURelYXynCRAaBQI9XwssiTjzGjWGqMn0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754917397; c=relaxed/simple; bh=nyWhbSykx6E52BnfjzxRnFJKTUek7padXw4kN3T5ym8=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=MJEWTseOudrCpUh9A46wMNizWghGIabd/he/oXnHqWY4I5PKy68jCxVJsQwaFjdKCoWQqCeKlSo6YGJi72NEndqz1i/SdK1Sb9vm4u2oxQD9oXQJeH9HfOFuiPPgaMUBtMt0e3VG8kl+Jysjk9R6nV+HUJBQA7H7bNHQRkM14aE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QjzbsPNf; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QjzbsPNf" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 844EBC4CEED; Mon, 11 Aug 2025 13:03:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1754917396; bh=nyWhbSykx6E52BnfjzxRnFJKTUek7padXw4kN3T5ym8=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=QjzbsPNf1wio9uRRvLbpLgGktMQfnLj0TQy71UCADJ0JG6eT+XCRJRXR0OLMC1x7S yy3wyzeRan7BLKu65C3kGdxdefFhElEuUNY04nE+UqtC1FOWd/1uv/BLo4OT9m9xMn kJ+tp0aaaOwzXiSFBE92uSfcxilU0ceTZwSi2wH57eqDBVBkAQ+qO6E1NorpMWBPVq RuUtPhsPUZOGMZ3B3iPCe4RK20XGFYFedhJJJaPnmAOfILKqIcYBD8M2D/z7yy6JOo A454lgBIqSBRgv93taZ34mZAuNLTIwnGV3i6sVG7n2gjLxfhlDg/05nzr87PYHWsYD cW68dKFvRlL6Q== Message-ID: Subject: Re: [Question]nfs: never returned delegation From: Trond Myklebust To: "zhangjian (CG)" , anna@kernel.org Cc: linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org Date: Mon, 11 Aug 2025 06:03:14 -0700 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.2 (3.56.2-1.fc42) Precedence: bulk X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 On Mon, 2025-08-11 at 20:48 +0800, zhangjian (CG) wrote: > Recently, we meet a NFS problem in 5.10. There are so many > test_state_id request after a non-privilaged request in tcpdump > result. There are 40w+ delegations in client (I read the delegation > list from /proc/kcore). > Firstly, I think state manager cost a lot in > nfs_server_reap_expired_delegations. But I see they are all in > NFS_DELEGATION_REVOKED state except 6 in NFS_DELEGATION_REFERENCED (I > read this from /proc/kcore too).=20 > I analyze NFS code and find if NFSPROC4_CLNT_DELEGRETURN procedure > meet ETIMEOUT, delegation will be marked as NFS4ERR_DELEG_REVOKED and > never return it again. NFS server will keep the revoked delegation in > clp->cl_revoked forever. This will result in following sequence > response with RECALLABLE_STATE_REVOKED flag. Client will send > test_state_id request for all non-revoked delegation. > This can only be solved by restarting NFS server. > I think ETIMEOUT in NFSPROC4_CLNT_DELEGRETURN procedure may be not > the only case that cause lots of non-terminable test_state_id > requests after any non-privilaged request.=20 > Wish NFS experts give some advices on this problem. >=20 You have the following options: 1. Don't ever use "soft" or "softerr" on the NFS client. 2. Reboot your server every now and again. 3. Change the server code to not bother caching revoked state. Doing so is rather pointless, since there is nothing a client can do differently when presented with NFS4ERR_DELEG_REVOKED vs. NFS4ERR_BAD_STATEID. 4. Change the server code to garbage collect revoked stateids after a while. --=20 Trond Myklebust Linux NFS client maintainer, Hammerspace trondmy@kernel.org, trond.myklebust@hammerspace.com