[PATCH] nvme: Don't fail to resume if NSIDs change

[PATCH] nvme: Don't fail to resume if NSIDs change

[Mario Limonciello]

Samsung PM9B1 has problems after resume because NSID has changed.
This has been reported in the past on OEM varities of PM9B1 parts
and fixed by firmware updates on 'some' of those parts.

However this same issue also happens on 'retail' PM9B1 parts which
Samsung has not released firmware updates for.

As the check has been relaxed at startup for multiple disks with
duplicate NSIDs with commit ac522fc6c3165 ("nvme: don't reject
probe due to duplicate IDs for single-ported PCIe devices") also
relax the check that runs on resume for NSIDs and mark them bogus
if this occurs on resume.

Fixes: 1d5df6af8c74 ("nvme: don't blindly overwrite identifiers on disk revalidate")
Cc: stable@vger.kernel.org # 6.1+
Cc: Nils Kruse <nilskruse97@gmail.com>
Cc: August Wikerfors <git@augustwikerfors.se>
Cc: David Chang <David.Chang@amd.com>
Link: https://github.com/tomsom/yoga-linux/issues/9
Link: https://lore.kernel.org/linux-nvme/b99a5149-c3d6-2a9b-1298-576a1b4b22c1@gmail.com/
Link: https://lore.kernel.org/all/20221116171727.4083-1-git@augustwikerfors.se/t/
Link: https://lore.kernel.org/all/d0ce0f3b-9407-9207-73a4-3536f0948653@augustwikerfors.se/
Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
---
 drivers/nvme/host/core.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 37b6fa7466620..fc85b4cd11fa2 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -3423,6 +3423,16 @@ static int nvme_global_check_duplicate_ids(struct nvme_subsystem *this,
 	return ret;
 }
 
+static void nvme_mark_nid_bogus(struct nvme_ns *ns, struct nvme_ns_info *info)
+{
+	dev_warn(ns->ctrl->device,
+		 "use of /dev/disk/by-id/ may cause data corruption\n");
+	memset(&info->ids.nguid, 0, sizeof(info->ids.nguid));
+	memset(&info->ids.uuid, 0, sizeof(info->ids.uuid));
+	memset(&info->ids.eui64, 0, sizeof(info->ids.eui64));
+	ns->ctrl->quirks |= NVME_QUIRK_BOGUS_NID;
+}
+
 static int nvme_init_ns_head(struct nvme_ns *ns, struct nvme_ns_info *info)
 {
 	struct nvme_ctrl *ctrl = ns->ctrl;
@@ -3459,12 +3469,7 @@ static int nvme_init_ns_head(struct nvme_ns *ns, struct nvme_ns_info *info)
 
 		dev_err(ctrl->device,
 			"clearing duplicate IDs for nsid %d\n", info->nsid);
-		dev_err(ctrl->device,
-			"use of /dev/disk/by-id/ may cause data corruption\n");
-		memset(&info->ids.nguid, 0, sizeof(info->ids.nguid));
-		memset(&info->ids.uuid, 0, sizeof(info->ids.uuid));
-		memset(&info->ids.eui64, 0, sizeof(info->ids.eui64));
-		ctrl->quirks |= NVME_QUIRK_BOGUS_NID;
+		nvme_mark_nid_bogus(ns, info);
 	}
 
 	mutex_lock(&ctrl->subsys->lock);
@@ -3706,14 +3711,14 @@ static void nvme_validate_ns(struct nvme_ns *ns, struct nvme_ns_info *info)
 {
 	int ret = NVME_SC_INVALID_NS | NVME_SC_DNR;
 
-	if (!nvme_ns_ids_equal(&ns->head->ids, &info->ids)) {
+	if (!nvme_ns_ids_equal(&ns->head->ids, &info->ids) &&
+	    !(ns->ctrl->quirks & NVME_QUIRK_BOGUS_NID)) {
 		dev_err(ns->ctrl->device,
 			"identifiers changed for nsid %d\n", ns->head->ns_id);
-		goto out;
+		nvme_mark_nid_bogus(ns, info);
 	}
 
 	ret = nvme_update_ns_info(ns, info);
-out:
 	/*
 	 * Only remove the namespace if we got a fatal error back from the
 	 * device, otherwise ignore the error and just move on.
-- 
2.34.1

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Keith Busch]

On Mon, Jul 31, 2023 at 01:51:03PM -0500, Mario Limonciello wrote:
> Samsung PM9B1 has problems after resume because NSID has changed.
> This has been reported in the past on OEM varities of PM9B1 parts
> and fixed by firmware updates on 'some' of those parts.
> 
> However this same issue also happens on 'retail' PM9B1 parts which
> Samsung has not released firmware updates for.
> 
> As the check has been relaxed at startup for multiple disks with
> duplicate NSIDs with commit ac522fc6c3165 ("nvme: don't reject
> probe due to duplicate IDs for single-ported PCIe devices") also
> relax the check that runs on resume for NSIDs and mark them bogus
> if this occurs on resume.

How could the driver tell the difference between the device needing a
quirk compared to a rapid delete-create-attach namespace sequence?
Proceeding with the namespace now may get dirty writes intended for the
previous namespace, corrupting the new one.

The commit you mentioned tries to constrain allowing duplication where
we can reasonably assume the quirk is needed. If we need to do similiar
for this condition, one possible constraint might be that the device
doesn't report OACS bit 3 (Namespace Management).

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Christoph Hellwig]

On Mon, Jul 31, 2023 at 01:10:11PM -0600, Keith Busch wrote:
> > As the check has been relaxed at startup for multiple disks with
> > duplicate NSIDs with commit ac522fc6c3165 ("nvme: don't reject
> > probe due to duplicate IDs for single-ported PCIe devices") also
> > relax the check that runs on resume for NSIDs and mark them bogus
> > if this occurs on resume.
> 
> How could the driver tell the difference between the device needing a
> quirk compared to a rapid delete-create-attach namespace sequence?
> Proceeding with the namespace now may get dirty writes intended for the
> previous namespace, corrupting the new one.
> 
> The commit you mentioned tries to constrain allowing duplication where
> we can reasonably assume the quirk is needed. If we need to do similiar
> for this condition, one possible constraint might be that the device
> doesn't report OACS bit 3 (Namespace Management).

Yes, this patch as-is looks really dangerous.  I don't think we should
just ignore the fact that IDs change when queried again.

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[August Wikerfors]

On 2023-07-31 21:10, Keith Busch wrote:
> On Mon, Jul 31, 2023 at 01:51:03PM -0500, Mario Limonciello wrote:
>> Samsung PM9B1 has problems after resume because NSID has changed.
>> This has been reported in the past on OEM varities of PM9B1 parts
>> and fixed by firmware updates on 'some' of those parts.
>>
>> However this same issue also happens on 'retail' PM9B1 parts which
>> Samsung has not released firmware updates for.
>>
>> As the check has been relaxed at startup for multiple disks with
>> duplicate NSIDs with commit ac522fc6c3165 ("nvme: don't reject
>> probe due to duplicate IDs for single-ported PCIe devices") also
>> relax the check that runs on resume for NSIDs and mark them bogus
>> if this occurs on resume.
> 
> How could the driver tell the difference between the device needing a
> quirk compared to a rapid delete-create-attach namespace sequence?
> Proceeding with the namespace now may get dirty writes intended for the
> previous namespace, corrupting the new one.
> 
> The commit you mentioned tries to constrain allowing duplication where
> we can reasonably assume the quirk is needed. If we need to do similiar
> for this condition, one possible constraint might be that the device
> doesn't report OACS bit 3 (Namespace Management).

It looks like that would work for the PM9B1:
> $ sudo nvme id-ctrl -H /dev/nvme0
> [...] > oacs      : 0x17
>   [10:10] : 0   Lockdown Command and Feature Not Supported
>   [9:9] : 0     Get LBA Status Capability Not Supported
>   [8:8] : 0     Doorbell Buffer Config Not Supported
>   [7:7] : 0     Virtualization Management Not Supported
>   [6:6] : 0     NVMe-MI Send and Receive Not Supported
>   [5:5] : 0     Directives Not Supported
>   [4:4] : 0x1   Device Self-test Supported
>   [3:3] : 0     NS Management and Attachment Not Supported
>   [2:2] : 0x1   FW Commit and Download Supported
>   [1:1] : 0x1   Format NVM Supported
>   [0:0] : 0x1   Security Send and Receive Supported

Regards,
August Wikerfors

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Limonciello, Mario]

On 7/31/2023 2:54 PM, August Wikerfors wrote:
> On 2023-07-31 21:10, Keith Busch wrote:
>> On Mon, Jul 31, 2023 at 01:51:03PM -0500, Mario Limonciello wrote:
>>> Samsung PM9B1 has problems after resume because NSID has changed.
>>> This has been reported in the past on OEM varities of PM9B1 parts
>>> and fixed by firmware updates on 'some' of those parts.
>>>
>>> However this same issue also happens on 'retail' PM9B1 parts which
>>> Samsung has not released firmware updates for.
>>>
>>> As the check has been relaxed at startup for multiple disks with
>>> duplicate NSIDs with commit ac522fc6c3165 ("nvme: don't reject
>>> probe due to duplicate IDs for single-ported PCIe devices") also
>>> relax the check that runs on resume for NSIDs and mark them bogus
>>> if this occurs on resume.
>>
>> How could the driver tell the difference between the device needing a
>> quirk compared to a rapid delete-create-attach namespace sequence?
>> Proceeding with the namespace now may get dirty writes intended for the
>> previous namespace, corrupting the new one.
>>
>> The commit you mentioned tries to constrain allowing duplication where
>> we can reasonably assume the quirk is needed. If we need to do similiar
>> for this condition, one possible constraint might be that the device
>> doesn't report OACS bit 3 (Namespace Management).
> 
> It looks like that would work for the PM9B1:
>> $ sudo nvme id-ctrl -H /dev/nvme0
>> [...] > oacs      : 0x17
>>   [10:10] : 0   Lockdown Command and Feature Not Supported
>>   [9:9] : 0     Get LBA Status Capability Not Supported
>>   [8:8] : 0     Doorbell Buffer Config Not Supported
>>   [7:7] : 0     Virtualization Management Not Supported
>>   [6:6] : 0     NVMe-MI Send and Receive Not Supported
>>   [5:5] : 0     Directives Not Supported
>>   [4:4] : 0x1   Device Self-test Supported
>>   [3:3] : 0     NS Management and Attachment Not Supported
>>   [2:2] : 0x1   FW Commit and Download Supported
>>   [1:1] : 0x1   Format NVM Supported
>>   [0:0] : 0x1   Security Send and Receive Supported
> 
> Regards,
> August Wikerfors

So is it reasonable to just add a check for

ctrl->oacs & NVME_CTRL_OACS_NS_MNGT_SUPP

In the same error handling path as this patch?

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Christoph Hellwig]

On Mon, Jul 31, 2023 at 03:09:08PM -0500, Limonciello, Mario wrote:
> So is it reasonable to just add a check for
>
> ctrl->oacs & NVME_CTRL_OACS_NS_MNGT_SUPP
>
> In the same error handling path as this patch?

No.  There are tons of NVMe devices that only support creating and
deleting namespace out of band, especially in virtualized and cloud
setups.

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Limonciello, Mario]

On 7/31/2023 3:10 PM, Christoph Hellwig wrote:
> On Mon, Jul 31, 2023 at 03:09:08PM -0500, Limonciello, Mario wrote:
>> So is it reasonable to just add a check for
>>
>> ctrl->oacs & NVME_CTRL_OACS_NS_MNGT_SUPP
>>
>> In the same error handling path as this patch?
> 
> No.  There are tons of NVMe devices that only support creating and
> deleting namespace out of band, especially in virtualized and cloud
> setups.

Even if it's only the error handling path only that it's checked?

If you don't want more changes or heuristics on the error handling path 
for this case, I think the best solution is probably to pick up

https://lore.kernel.org/all/20221116171727.4083-1-git@augustwikerfors.se/t/

instead then and hopefully we don't end up with more disks like this.

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Christoph Hellwig]

On Mon, Jul 31, 2023 at 03:14:54PM -0500, Limonciello, Mario wrote:
>> No.  There are tons of NVMe devices that only support creating and
>> deleting namespace out of band, especially in virtualized and cloud
>> setups.
>
> Even if it's only the error handling path only that it's checked?

Do you mean nvme_validate_ns with the error code?  I wouldn't really call
that an error case, that's the function called to check namespaces are
still the same after we did a rescan (either manually or triggered by the
AEN).

> If you don't want more changes or heuristics on the error handling path for 
> this case, I think the best solution is probably to pick up
>
> https://lore.kernel.org/all/20221116171727.4083-1-git@augustwikerfors.se/t/
>
> instead then and hopefully we don't end up with more disks like this.

That's probably the better idea.  I know at least one of the early
quirked devices also IDs that changed for subsequent identify calls.

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Mario Limonciello]

On 8/1/23 06:24, Christoph Hellwig wrote:
> On Mon, Jul 31, 2023 at 03:14:54PM -0500, Limonciello, Mario wrote:
>>> No.  There are tons of NVMe devices that only support creating and
>>> deleting namespace out of band, especially in virtualized and cloud
>>> setups.
>>
>> Even if it's only the error handling path only that it's checked?
> 
> Do you mean nvme_validate_ns with the error code?  I wouldn't really call
> that an error case, that's the function called to check namespaces are
> still the same after we did a rescan (either manually or triggered by the
> AEN).
> 
>> If you don't want more changes or heuristics on the error handling path for
>> this case, I think the best solution is probably to pick up
>>
>> https://lore.kernel.org/all/20221116171727.4083-1-git@augustwikerfors.se/t/
>>
>> instead then and hopefully we don't end up with more disks like this.
> 
> That's probably the better idea.  I know at least one of the early
> quirked devices also IDs that changed for subsequent identify calls.

Do you want that re-sent?  Or can you just pick up from that lore link?

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Keith Busch]

On Tue, Aug 01, 2023 at 06:30:52AM -0500, Mario Limonciello wrote:
> 
> Do you want that re-sent?  Or can you just pick up from that lore link?

I got it, and applied to nvme-6.5 now.

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Mario Limonciello]

On 8/1/2023 15:29, Keith Busch wrote:
> On Tue, Aug 01, 2023 at 06:30:52AM -0500, Mario Limonciello wrote:
>>
>> Do you want that re-sent?  Or can you just pick up from that lore link?
> 
> I got it, and applied to nvme-6.5 now.

Thanks!

If you can still change it before sending out can you add a stable tag 
as well?

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[August Wikerfors]

On 2023-08-01 22:34, Mario Limonciello wrote:
> If you can still change it before sending out can you add a stable tag 
> as well?

This didn't get added in time, so, stable team, please backport:

688b419c57c1 ("nvme-pci: add NVME_QUIRK_BOGUS_NID for Samsung PM9B1 256G and 512G")

Regards,
August Wikerfors

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Keith Busch]

On Fri, Aug 11, 2023 at 10:19:35PM +0200, August Wikerfors wrote:
> On 2023-08-01 22:34, Mario Limonciello wrote:
> > If you can still change it before sending out can you add a stable tag
> > as well?
> 
> This didn't get added in time, so, stable team, please backport:
> 
> 688b419c57c1 ("nvme-pci: add NVME_QUIRK_BOGUS_NID for Samsung PM9B1 256G and 512G")

Perhaps bad form on my end for relying on it, but in my experience, the
stable bot has a great record on auto selecting nvme quirks.

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Greg KH]

On Fri, Aug 11, 2023 at 02:59:33PM -0600, Keith Busch wrote:
> On Fri, Aug 11, 2023 at 10:19:35PM +0200, August Wikerfors wrote:
> > On 2023-08-01 22:34, Mario Limonciello wrote:
> > > If you can still change it before sending out can you add a stable tag
> > > as well?
> > 
> > This didn't get added in time, so, stable team, please backport:
> > 
> > 688b419c57c1 ("nvme-pci: add NVME_QUIRK_BOGUS_NID for Samsung PM9B1 256G and 512G")
> 
> Perhaps bad form on my end for relying on it, but in my experience, the
> stable bot has a great record on auto selecting nvme quirks.

It's better for everyone if you mark it for stable, so you know it will
get reviewed, otherwise you are at the mercy of our scripts and free
time to dig for patches :)

thanks,

greg k-h

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[Greg KH]

On Fri, Aug 11, 2023 at 10:19:35PM +0200, August Wikerfors wrote:
> On 2023-08-01 22:34, Mario Limonciello wrote:
> > If you can still change it before sending out can you add a stable tag
> > as well?
> 
> This didn't get added in time, so, stable team, please backport:
> 
> 688b419c57c1 ("nvme-pci: add NVME_QUIRK_BOGUS_NID for Samsung PM9B1 256G and 512G")

Now queued up to 6.4.y and 6.1.y, thanks.

greg k-h

Re: [PATCH] nvme: Don't fail to resume if NSIDs change

[August Wikerfors]

On 2023-07-31 20:51, Mario Limonciello wrote:
> Samsung PM9B1 has problems after resume because NSID has changed.
> This has been reported in the past on OEM varities of PM9B1 parts
> and fixed by firmware updates on 'some' of those parts.
> 
> However this same issue also happens on 'retail' PM9B1 parts which
> Samsung has not released firmware updates for.
> 
> As the check has been relaxed at startup for multiple disks with
> duplicate NSIDs with commit ac522fc6c3165 ("nvme: don't reject
> probe due to duplicate IDs for single-ported PCIe devices") also
> relax the check that runs on resume for NSIDs and mark them bogus
> if this occurs on resume.
> 
> Fixes: 1d5df6af8c74 ("nvme: don't blindly overwrite identifiers on disk revalidate")
> Cc: stable@vger.kernel.org # 6.1+
> Cc: Nils Kruse <nilskruse97@gmail.com>
> Cc: August Wikerfors <git@augustwikerfors.se>
> Cc: David Chang <David.Chang@amd.com>
> Link: https://github.com/tomsom/yoga-linux/issues/9
> Link: https://lore.kernel.org/linux-nvme/b99a5149-c3d6-2a9b-1298-576a1b4b22c1@gmail.com/
> Link: https://lore.kernel.org/all/20221116171727.4083-1-git@augustwikerfors.se/t/
> Link: https://lore.kernel.org/all/d0ce0f3b-9407-9207-73a4-3536f0948653@augustwikerfors.se/
> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
Tested-by: August Wikerfors <git@augustwikerfors.se>

Thanks!