From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B28CC3A4F51 for ; Wed, 19 Aug 2026 21:50:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787176243; cv=none; b=q0kPZt7OIT2C0c8Iqd8vK+mIZIOzBclS3ib8EYNKeV5i3/stKh0QFi8vpKjBiPXqe3cIHTwYEfLPGUQe5CystnPibD0kT4su/dyYdegS1xlSB8arKhtA2GlGb8GU02xsT0cLxmf6ldotZ6YyvqsXwQ9rawaD/q7jApnxxuxlX14= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787176243; c=relaxed/simple; bh=2Am3osv0wvO7os0MJJsASZd1ZN25DhzEOV9MFm5mr78=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=Q3Y1VOVGYwfGuADGmqGcNmI1U9MJH6sqkBGPGEtNF6m0g7DxvKrc0XzaCBrHSWmFzhGJMc4xOALaxCrFQTL5qNtP48GJCqRJHzKyKjbcJU1ynmQ58tWheBCUqHU9nnTZheq1kdPgWnHJKG4QxikbIIVEGcw7lA0/RWHr07cPQac= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=bU+0ty+6; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bU+0ty+6" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1787176240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+TzkIkXJy7hGwyp9eR+gqjG98fv/6iQ4VO/4i7ImQRQ=; b=bU+0ty+6IzUyc+acphoUEgMQ5RD166g5mfAnFhZ+L9XE18sq1xXMAbCcNcCLBYR2NYnOmE 1sXDNcuYYqhXuUwIUsqy5xUx4KfVMg4gp/lFiEYDGgXWysxreho/ccks1TTtudSgTmx1Sh 2LNW/KI4+OTh4WPsVJUw3thec5zk/po= Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-673-fUnlG3QFNg6XXCeSxXtZ2g-1; Wed, 19 Aug 2026 17:50:39 -0400 X-MC-Unique: fUnlG3QFNg6XXCeSxXtZ2g-1 X-Mimecast-MFC-AGG-ID: fUnlG3QFNg6XXCeSxXtZ2g_1787176238 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D00E0192053E; Wed, 19 Aug 2026 21:49:57 +0000 (UTC) Received: from [10.22.80.117] (unknown [10.22.80.117]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C65CB18005B1; Wed, 19 Aug 2026 21:49:54 +0000 (UTC) Message-ID: <0d3cb83e-a4b6-414d-8756-fafd4a8238bc@redhat.com> Date: Wed, 19 Aug 2026 17:49:53 -0400 Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2 0/2] Test multipath and marginal ports To: Jesse Taube , linux-block@vger.kernel.org Cc: linux-nvme@lists.infradead.org, shinichiro.kawasaki@wdc.com, Daniel Wagner References: <20260819200425.194743-1-jtaubepe@redhat.com> Content-Language: en-US From: John Meneghini Organization: RHEL Core Storge Team In-Reply-To: <20260819200425.194743-1-jtaubepe@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 This is a great improvement and we now have a functioning blkstest 070 to test the FPIN LI kernel patches. The problem is: I don't think we are done yet. In my own private testing and development with these patches I've been working on a next-version test that combines the ANA states from test/nvme/057 with test/nvme/070. So I am working on a test 071. The good news is: everything now works with test/nvme/070.  The bad news is: the kernel patches are not done. What I've found is: a long as the ANA states are all optimized or non-optimized everything works.  However, once we throw in an inaccessible state to the mix, we run into serious problems. At this point in development I don't care about the test failures in my test/nvme/071 script. I expect the script to bug out because it doesn't understand the inaccessible state. The test sill continues flipping rports in and out of the marginal state and keeps going. That's what it is designed to do. That means it is testing all of the code paths in the kernel patches. The problem is: when turning marginal paths on and off with controllers that are in the inaccessible state, the path selection algorithm in the kernel fails and we end up with the following: [Wed Aug 19 16:25:42 2026] nvme_ns_head_submit_bio: 6 callbacks suppressed [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O [Wed Aug 19 16:25:42 2026] block nvme2n1: no usable path - requeuing I/O At this point the fio jobs are still running but there is no progress.  This means no path was found and ALL of the IOs got re-queued. And if we flip the marginal state off on all of the controllers IO continues to be hung.  The IO scheduler is hung and there is no possibility of getting it restarted again. So this is a really serious bug in the kernel patches and we can't ship this stuff until we fix the problem. This IO re-requeing problem should NEVER happen - no matter what the state of the marginal paths. So I can recommend that Shinichiro test these patches with the current upstream kernel patches: https://lore.kernel.org/linux-nvme/20260812181300.3712426-1-jtaubepe@redhat.com/ But there will be another version of Kernel patches and a V3 of this patch set will be forth coming. John A. Meneghini Senior Principal Platform Storage Engineer RHEL SST - Platform Storage Group jmeneghi@redhat.com On 8/19/26 16:04, Jesse Taube wrote: > Tests for the upcoming nvme-fc: FPIN link integrity handling set. > It tests for various multipath and marginal port > scenarios, while confirming the port usage and state. The test is > intended to emulate receiving an FPIN event in a multipath environment. > > Link: https://bugzilla.kernel.org/show_bug.cgi?id=220329 > Link: https://github.com/linux-blktests/blktests/pull/264 > > Jesse Taube (2): > nvme: Add _setup_nvmet_port_marginal > nvme/070: Test multipath and marginal ports > > common/nvme | 31 +++ > tests/nvme/070 | 613 +++++++++++++++++++++++++++++++++++++++++++++ > tests/nvme/070.out | 43 ++++ > 3 files changed, 687 insertions(+) > create mode 100755 tests/nvme/070 > create mode 100644 tests/nvme/070.out >