[bug report] nvme4: inconsistent AWUPF, controller not added (0/7).

[bug report] nvme4: inconsistent AWUPF, controller not added (0/7).

[Yi Zhang]

Hi Christoph

I found this failure on one Samsung NVMe disk[1] with the latest
linux-block/for-next. Here is the reproducer and dmesg log.
Please help check it and let me know if you need any info/test. Thanks.

[1]
SAMSUNG MZQL2960HCJR-00A07 (PM9A3)
[2]
+ nvme format -l1 -f /dev/nvme4n1
Success formatting namespace:1
+ nvme reset /dev/nvme4
Reset: Network dropped connection on reset

dmesg:
[  751.872864] nvme nvme4: rescanning namespaces.
[  752.177475] nvme nvme4: resetting controller
[  752.221030] nvme nvme4: inconsistent AWUPF, controller not added (0/7).
[  752.227653] nvme nvme4: Disabling device after reset failure: -22

-- 
Best Regards,
  Yi Zhang

Re: [bug report] nvme4: inconsistent AWUPF, controller not added (0/7).

[alan.adamson]

On 7/2/25 4:13 AM, Yi Zhang wrote:
> Hi Christoph
>
> I found this failure on one Samsung NVMe disk[1] with the latest
> linux-block/for-next. Here is the reproducer and dmesg log.
> Please help check it and let me know if you need any info/test. Thanks.
>
> [1]
> SAMSUNG MZQL2960HCJR-00A07 (PM9A3)
> [2]
> + nvme format -l1 -f /dev/nvme4n1
> Success formatting namespace:1
> + nvme reset /dev/nvme4
> Reset: Network dropped connection on reset
>
> dmesg:
> [  751.872864] nvme nvme4: rescanning namespaces.
> [  752.177475] nvme nvme4: resetting controller
> [  752.221030] nvme nvme4: inconsistent AWUPF, controller not added (0/7).
> [  752.227653] nvme nvme4: Disabling device after reset failure: -22
>
Looks like the device isn't reporting AWUPF after the format/reset.

Can you try:

nvme id-ctrl /dev/nvme4 | grep awupf
nvme id-ns  /dev/nvme4n1 | grep nawupf
nvme format -l1 -f /dev/nvme4n1
nvme id-ctrl /dev/nvme4 | grep awupf
nvme id-ns  /dev/nvme4n1 | grep nawupf
nvme reset /dev/nvme4
nvme id-ctrl /dev/nvme4 | grep awupf


Alan

Re: [bug report] nvme4: inconsistent AWUPF, controller not added (0/7).

[Christoph Hellwig]

On Wed, Jul 02, 2025 at 09:33:32AM -0700, alan.adamson@oracle.com wrote:
> Looks like the device isn't reporting AWUPF after the format/reset.

The other option would be that the format changed the value.

The mess NVMe creasted with the totally un-thought out atomics is
beyond belive :(

I wonder if we should just back out the whole thing and wait for the
working group to come up with something that can actually safely work.

Re: [bug report] nvme4: inconsistent AWUPF, controller not added (0/7).

[Yi Zhang]

On Thu, Jul 3, 2025 at 4:04 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Wed, Jul 02, 2025 at 09:33:32AM -0700, alan.adamson@oracle.com wrote:
> > Looks like the device isn't reporting AWUPF after the format/reset.
>
> The other option would be that the format changed the value.
>
> The mess NVMe creasted with the totally un-thought out atomics is
> beyond belive :(
>
> I wonder if we should just back out the whole thing and wait for the
> working group to come up with something that can actually safely work.
>

Yeah, the format operation will change the awupf value.
Here is the reset operation pass[1] and fail[2] log
[1]
+ nvme format -l0 -f /dev/nvme3n1
Success formatting namespace:1
+ nvme id-ctrl /dev/nvme3
+ grep awupf
awupf     : 7
+ grep nawupf
+ nvme id-ns /dev/nvme3n1
nawupf  : 7
+ nvme format -l1 -f /dev/nvme3n1
Success formatting namespace:1
+ nvme id-ctrl /dev/nvme3
+ grep awupf
awupf     : 0
+ nvme id-ns /dev/nvme3n1
+ grep nawupf
nawupf  : 0
+ nvme reset /dev/nvme3
+ nvme id-ctrl /dev/nvme3
+ grep awupf
awupf     : 0

[2]
+ nvme format -l0 -f /dev/nvme5n1
Success formatting namespace:1
+ nvme id-ctrl /dev/nvme5
+ grep awupf
awupf     : 7
+ nvme id-ns /dev/nvme5n1
+ grep nawupf
nawupf  : 7
+ nvme format -l1 -f /dev/nvme5n1
Success formatting namespace:1
+ nvme id-ctrl /dev/nvme5
+ grep awupf
awupf     : 0
+ nvme id-ns /dev/nvme5n1
+ grep nawupf
nawupf  : 0
+ nvme reset /dev/nvme5
Reset: Network dropped connection on reset
# dmesg | tail -5
[  597.973393] nvme nvme5: rescanning namespaces.
[  598.292285] nvme nvme5: rescanning namespaces.
[  598.584937] nvme nvme5: resetting controller
[  598.626440] nvme nvme5: inconsistent AWUPF, controller not added (0/7).
[  598.633064] nvme nvme5: Disabling device after reset failure: -22


-- 
Best Regards,
  Yi Zhang

Re: [bug report] nvme4: inconsistent AWUPF, controller not added (0/7).

[Ming Lei]

On Fri, Jul 04, 2025 at 01:47:50AM +0800, Yi Zhang wrote:
> On Thu, Jul 3, 2025 at 4:04 PM Christoph Hellwig <hch@lst.de> wrote:
> >
> > On Wed, Jul 02, 2025 at 09:33:32AM -0700, alan.adamson@oracle.com wrote:
> > > Looks like the device isn't reporting AWUPF after the format/reset.
> >
> > The other option would be that the format changed the value.
> >
> > The mess NVMe creasted with the totally un-thought out atomics is
> > beyond belive :(
> >
> > I wonder if we should just back out the whole thing and wait for the
> > working group to come up with something that can actually safely work.
> >
> 
> Yeah, the format operation will change the awupf value.
> Here is the reset operation pass[1] and fail[2] log
> [1]
> + nvme format -l0 -f /dev/nvme3n1
> Success formatting namespace:1
> + nvme id-ctrl /dev/nvme3
> + grep awupf
> awupf     : 7
> + grep nawupf
> + nvme id-ns /dev/nvme3n1
> nawupf  : 7
> + nvme format -l1 -f /dev/nvme3n1
> Success formatting namespace:1
> + nvme id-ctrl /dev/nvme3
> + grep awupf
> awupf     : 0
> + nvme id-ns /dev/nvme3n1
> + grep nawupf
> nawupf  : 0
> + nvme reset /dev/nvme3
> + nvme id-ctrl /dev/nvme3
> + grep awupf
> awupf     : 0
> 
> [2]
> + nvme format -l0 -f /dev/nvme5n1
> Success formatting namespace:1
> + nvme id-ctrl /dev/nvme5
> + grep awupf
> awupf     : 7
> + nvme id-ns /dev/nvme5n1
> + grep nawupf
> nawupf  : 7
> + nvme format -l1 -f /dev/nvme5n1
> Success formatting namespace:1
> + nvme id-ctrl /dev/nvme5
> + grep awupf
> awupf     : 0

Per NVMe spec, AWUPF unit is 'logical blocks', and logical block size is changed
by 'nvme format', so AWUPF value retuned from `Identify command` can be changed
because the controller implements fixed-length atomic write size(512*8, 4096 * 1)?


Thanks,
Ming

Re: [bug report] nvme4: inconsistent AWUPF, controller not added (0/7).

[Christoph Hellwig]

On Fri, Jul 04, 2025 at 10:17:38AM +0800, Ming Lei wrote:
> Per NVMe spec, AWUPF unit is 'logical blocks', and logical block size is changed
> by 'nvme format', so AWUPF value retuned from `Identify command` can be changed
> because the controller implements fixed-length atomic write size(512*8, 4096 * 1)?
> 

Yes.  And that's an issue because NVMe doesn't have a controller-level
concept of a logical bloc ksize, the logical block size is per-namespace.

Or in other words, AWUPF is a mess.