Re: Issue with AWUPF when using multiple controllers in a subsystem

[Chaitanya Kulkarni <chaitanyak@nvidia.com>]

On 4/4/25 15:42, alan.adamson@oracle.com wrote:
> While testing atomic write parameters with multiple controllers within 
> a subsystem, I found an issue:
>
> QEMU-NVME Configuartion
> =================
> -device nvme-subsys,id=subsys0 \
> -device 
> nvme,serial=deadbeef,id=nvme0,subsys=subsys0,atomic.dn=off,atomic.awun=31,atomic.awupf=15 
> \
> -device 
> nvme,serial=deadbeef,id=nvme1,subsys=subsys0,atomic.dn=off,atomic.awun=127,atomic.awupf=63 
> \
> -drive id=ns1,file=/dev/nullb3,if=none \
> -drive id=ns2,file=/dev/nullb2,if=none \
> -device nvme-ns,drive=ns1,bus=nvme0,nsid=1,shared=false \
> -device nvme-ns,drive=ns2,bus=nvme1,nsid=2,shared=false \
>
> [root@localhost ~]# nvme id-ctrl /dev/nvme1n2 | grep awupf
> awupf     : 63
> [root@localhost ~]# cat /sys/block/nvme1n2/queue/atomic_write_max_bytes
> 32768
> [root@localhost ~]# nvme id-ctrl /dev/nvme1n1 | grep awupf
> awupf     : 15
> [root@localhost ~]# cat /sys/block/nvme1n1/queue/atomic_write_max_bytes
> 32768
> [root@localhost ~]#
>
> When dumping the awupf value with nvme-cli, the values match what was 
> setup in qemu, but when each devices atomic queue limits was 
> displayed, it didn't match its awupf. Currently the awupf is saved in 
> the nvme_subsystem, but the awupf is specific to the controller, not 
> the subsystem. Two controllers in a subsystem can have different 
> atomic parameters.
>
> The commit 81adb8633491 ("nvme: set physical block size and optimal 
> I/O size") added subsys->awupf.
>
> I think the awupf belongs in the nvme_ctrl structure. 

AWUPF is a part of Identify controller data structure so Ideally
it should be part of the controller attribute.

 From NVM Express® Base Specification, Revision 2.2, March 11th,
2025 Page 319 :-

Figure 313: Identify – Identify Controller Data Structure,
I/O Command Set Independent

529:528

Atomic Write Unit Power Fail (AWUPF): This field is specific
to namespaces that are associated with command sets that
specify logical blocks (i.e., Command Set Identifier 0h or 2h),
and shall be cleared to 0h for namespaces that are not associated
with command sets that specify logical blocks. Refer to the
applicable I/O Command Set specification (e.g., the Atomic Operation
section of the NVM Command Set Specification).

-ck