sysfs integrity fields use

sysfs integrity fields use

[Milan Broz]

Hi,

I tried to add some support for using devices with PI/DIF metadata
and checked (through sysfs) how large metadata space per sector
is available.

The problem is that some values behave differently than I expected.

For an NVMe drive, reformatted to 4096 + 64 profile, I see this:

- /sys/block/<disk>/integrity/device_is_integrity_capable
   Contains 0 (?)
   According to docs, this field
  "Indicates whether a storage device is capable of storing integrity metadata.
  Set if the device is T10 PI-capable."

- /sys/block/<disk>/integrity/format
  Contains expected "nop" (not "none")

- /sys/block/<disk>/integrity/tag_size
   Contains 0 (?)
   According to docs, this is "Number of bytes of integrity tag space
   available per 512 bytes of data."
   (I think 512 bytes is incorrect; it should be sector size, or perhaps
    value in protection_interval_bytes, though.)

Then we have new (undocumented) value for NVMe in
- /sys/block/<nvme>/integrity/metadata_bytes
   This contains the correct 64.

Anyway, when I try to use it (for authentication tags in dm-crypt), it works.

Should tag_size and device_is_integrity_capable be set even for the "nop" format?
Is it a bug or a feature? :-)

If not, what is the correct way to read per-sector metadata size (not only for NVMe
as metadata_bytes is not available for other block devices)?

(This is on recent Linus' master - 6.14.0-rc4)

Thanks,
Milan

Re: sysfs integrity fields use

[Kanchan Joshi]

On 2/25/2025 2:53 PM, Milan Broz wrote:
> Hi,
> 
> I tried to add some support for using devices with PI/DIF metadata
> and checked (through sysfs) how large metadata space per sector
> is available.
> 
> The problem is that some values behave differently than I expected.
> 
> For an NVMe drive, reformatted to 4096 + 64 profile, I see this:
> 
> - /sys/block/<disk>/integrity/device_is_integrity_capable
>    Contains 0 (?)
>    According to docs, this field
>   "Indicates whether a storage device is capable of storing integrity 
> metadata.
>   Set if the device is T10 PI-capable."
> 
> - /sys/block/<disk>/integrity/format
>   Contains expected "nop" (not "none")
> 
> - /sys/block/<disk>/integrity/tag_size
>    Contains 0 (?)

This and "nop" indicates that pi-type was configured to be 0?
Maybe you can share the nvme format command as well.

>    According to docs, this is "Number of bytes of integrity tag space
>    available per 512 bytes of data."
>    (I think 512 bytes is incorrect; it should be sector size, or perhaps
>     value in protection_interval_bytes, though.)
> 
> Then we have new (undocumented) value for NVMe in
> - /sys/block/<nvme>/integrity/metadata_bytes
>    This contains the correct 64.

Maybe you mean "/sys/block/>/metadata_bytes"?

Re: sysfs integrity fields use

[Kanchan Joshi]

On 2/25/2025 3:40 PM, Kanchan Joshi wrote:
>>     According to docs, this is "Number of bytes of integrity tag space
>>     available per 512 bytes of data."
>>     (I think 512 bytes is incorrect; it should be sector size, or perhaps
>>      value in protection_interval_bytes, though.)
>>
>> Then we have new (undocumented) value for NVMe in
>> -/sys/block/<nvme>/integrity/metadata_bytes
>>     This contains the correct 64.
> Maybe you mean "/sys/block/>/metadata_bytes"?

/sys/block/<nvme>/metadata_bytes.

Re: sysfs integrity fields use

[Milan Broz]

On 2/25/25 11:10 AM, Kanchan Joshi wrote:
> On 2/25/2025 2:53 PM, Milan Broz wrote:
>> Hi,
>>
>> I tried to add some support for using devices with PI/DIF metadata
>> and checked (through sysfs) how large metadata space per sector
>> is available.
>>
>> The problem is that some values behave differently than I expected.
>>
>> For an NVMe drive, reformatted to 4096 + 64 profile, I see this:
>>
>> - /sys/block/<disk>/integrity/device_is_integrity_capable
>>     Contains 0 (?)
>>     According to docs, this field
>>    "Indicates whether a storage device is capable of storing integrity
>> metadata.
>>    Set if the device is T10 PI-capable."
>>
>> - /sys/block/<disk>/integrity/format
>>    Contains expected "nop" (not "none")
>>
>> - /sys/block/<disk>/integrity/tag_size
>>     Contains 0 (?)
> 
> This and "nop" indicates that pi-type was configured to be 0?
> Maybe you can share the nvme format command as well.

Sure, it is formatted to 4k data + 64 bytes metadata profile:

# nvme id-ns -H /dev/nvme0n1
...

LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0 Best
LBA Format  1 : Metadata Size: 8   bytes - Data Size: 512 bytes - Relative Performance: 0 Best
LBA Format  2 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0 Best
LBA Format  3 : Metadata Size: 64  bytes - Data Size: 4096 bytes - Relative Performance: 0 Best (in use)

formatted with
# nvme format --lbaf=3 --force /dev/nvme0n1

>>     According to docs, this is "Number of bytes of integrity tag space
>>     available per 512 bytes of data."
>>     (I think 512 bytes is incorrect; it should be sector size, or perhaps
>>      value in protection_interval_bytes, though.)
>>
>> Then we have new (undocumented) value for NVMe in
>> - /sys/block/<nvme>/integrity/metadata_bytes
>>     This contains the correct 64.
> 
> Maybe you mean "/sys/block/>/metadata_bytes"?

Yes, it is not under integrity subdir, just copy& paste error, sorry.

Just to add - I would like to add these integrity fields also to lsblk, but there we need exact
specification where to get integrity info.

Milan

Re: sysfs integrity fields use

[Kanchan Joshi]

On 2/25/2025 4:33 PM, Milan Broz wrote:
> Sure, it is formatted to 4k data + 64 bytes metadata profile:
> 
> # nvme id-ns -H /dev/nvme0n1
> ...
> 
> LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - 
> Relative Performance: 0 Best
> LBA Format  1 : Metadata Size: 8   bytes - Data Size: 512 bytes - 
> Relative Performance: 0 Best
> LBA Format  2 : Metadata Size: 0   bytes - Data Size: 4096 bytes - 
> Relative Performance: 0 Best
> LBA Format  3 : Metadata Size: 64  bytes - Data Size: 4096 bytes - 
> Relative Performance: 0 Best (in use)
> 
> formatted with
> # nvme format --lbaf=3 --force /dev/nvme0n1

Since you're not passing "--pi=[1-3]", it takes 0 as default, which 
means protection-info is off.

Re: sysfs integrity fields use

[Christoph Hellwig]

On Tue, Feb 25, 2025 at 10:23:49AM +0100, Milan Broz wrote:
> Hi,
> 
> I tried to add some support for using devices with PI/DIF metadata
> For an NVMe drive, reformatted to 4096 + 64 profile, I see this:

Based on everything below you have no formatted it with PI and you
also don't plan to use PI, just plain non-integrity metadata.

> 
> - /sys/block/<disk>/integrity/device_is_integrity_capable
>   Contains 0 (?)
>   According to docs, this field
>  "Indicates whether a storage device is capable of storing integrity metadata.
>  Set if the device is T10 PI-capable."

this is only 1 if protection information is enabled, and 0
for non-PI metadata.

> - /sys/block/<disk>/integrity/tag_size
>   Contains 0 (?)
>   According to docs, this is "Number of bytes of integrity tag space
>   available per 512 bytes of data."
>   (I think 512 bytes is incorrect; it should be sector size, or perhaps
>    value in protection_interval_bytes, though.)

Yes, it should be an integrity interval.  Which for all current drivers
is the LBA size.  And the tag_size is the size of the PI metadata, so
0 is expected.

> Then we have new (undocumented) value for NVMe in
> - /sys/block/<nvme>/integrity/metadata_bytes
>   This contains the correct 64.

Yes, this contains the full size of the metadata.  And besides
documenting it we should probably also lift it to the block layer.

> Anyway, when I try to use it (for authentication tags in dm-crypt), it works.
> 
> Should tag_size and device_is_integrity_capable be set even for the "nop" format?
> Is it a bug or a feature? :-)

It is expected.  The only issue is that the block support for metadata
is called integrity all over because it was initially added for PI only
and then extended for non-PI metadata, which makes things a little
confusing.

> If not, what is the correct way to read per-sector metadata size (not only for NVMe
> as metadata_bytes is not available for other block devices)?

Right now the nvme specific attribute is the only way.

Re: sysfs integrity fields use

[Milan Broz]

Hi,

On 2/25/25 4:38 PM, Christoph Hellwig wrote:
...
> 
>> Then we have new (undocumented) value for NVMe in
>> - /sys/block/<nvme>/integrity/metadata_bytes
>>    This contains the correct 64.
> 
> Yes, this contains the full size of the metadata.  And besides
> documenting it we should probably also lift it to the block layer.

yes, this is exactly what I need, just it should be present
for all block devices.
  
>> Anyway, when I try to use it (for authentication tags in dm-crypt), it works.
>>
>> Should tag_size and device_is_integrity_capable be set even for the "nop" format?
>> Is it a bug or a feature? :-)
> 
> It is expected.  The only issue is that the block support for metadata
> is called integrity all over because it was initially added for PI only
> and then extended for non-PI metadata, which makes things a little
> confusing.

ok, that make sense. Maybe the note should be added to sysfs doc too.

Thanks,
Milan

[PATCH] docs: sysfs-block: Clarify integrity sysfs attributes for non-PI metadata

[Milan Broz]

The /sys/block/<disk>/integrity fields are historically set if PI
is enabled. It is not set if some upper layer uses integrity metadata
for non-PI format.
Document it.

Signed-off-by: Milan Broz <gmazyland@gmail.com>
---
 Documentation/ABI/stable/sysfs-block | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block
index 0cceb2badc83..f67fd46f15b6 100644
--- a/Documentation/ABI/stable/sysfs-block
+++ b/Documentation/ABI/stable/sysfs-block
@@ -109,6 +109,8 @@ Contact:	Martin K. Petersen <martin.petersen@oracle.com>
 Description:
 		Indicates whether a storage device is capable of storing
 		integrity metadata. Set if the device is T10 PI-capable.
+		This flag is set if a PI profile is enabled.
+		It is not set when non-PI metadata are used.
 
 
 What:		/sys/block/<disk>/integrity/format
@@ -117,6 +119,8 @@ Contact:	Martin K. Petersen <martin.petersen@oracle.com>
 Description:
 		Metadata format for integrity capable block device.
 		E.g. T10-DIF-TYPE1-CRC.
+		If the storage device supports metadata but no PI
+		is used, this field will contain "nop".
 
 
 What:		/sys/block/<disk>/integrity/protection_interval_bytes
@@ -142,7 +146,10 @@ Date:		June 2008
 Contact:	Martin K. Petersen <martin.petersen@oracle.com>
 Description:
 		Number of bytes of integrity tag space available per
-		512 bytes of data.
+		protection_interval_bytes, which is typically
+		the device's logical block size.
+		If the storage device supports metadata but no PI
+		is used, this field will contain 0.
 
 
 What:		/sys/block/<disk>/integrity/write_generate
-- 
2.47.2

Re: [PATCH] docs: sysfs-block: Clarify integrity sysfs attributes for non-PI metadata

[Martin K. Petersen]

Milan,

> +		This flag is set if a PI profile is enabled.
> +		It is not set when non-PI metadata are used.

This flag is set to 1 if the storage media is formatted with T10
Protection Information. If the storage media is not formatted with T10
Protection Information, this flag is set to 0.

>  What:		/sys/block/<disk>/integrity/format
> @@ -117,6 +119,8 @@ Contact:	Martin K. Petersen <martin.petersen@oracle.com>
>  Description:
>  		Metadata format for integrity capable block device.
>  		E.g. T10-DIF-TYPE1-CRC.
> +		If the storage device supports metadata but no PI
> +		is used, this field will contain "nop".

This field describes the type of T10 Protection Information that the
block device is capable of sending and receiving. If the device does not
support sending and receiving T10 Protection Information, this field
contains "nop".

> -		512 bytes of data.
> +		protection_interval_bytes, which is typically
> +		the device's logical block size.
> +		If the storage device supports metadata but no PI
> +		is used, this field will contain 0.

This field describes the size of the application tag if the storage
device is formatted with T10 Protection Information and permits use of
the application tag. tag_size is reported in bytes and indicates the
space available for adding an opaque tag to each block
(protection_interval_bytes). If the device does not support T10
Protection Information or the device uses the application tag space
internally, this field is set to 0.


Wrt. the size of opaque (non-PI) metadata, I think it would good to have
a dedicated field to describe it.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH] docs: sysfs-block: Clarify integrity sysfs attributes for non-PI metadata

[Milan Broz]

Hi Martin,

thanks for the clarification, should I just update the
patch and add your signed-off line?

Anyway, some questions below:

On 3/3/25 6:25 PM, Martin K. Petersen wrote:

>>   What:		/sys/block/<disk>/integrity/format
>> @@ -117,6 +119,8 @@ Contact:	Martin K. Petersen <martin.petersen@oracle.com>
>>   Description:
>>   		Metadata format for integrity capable block device.
>>   		E.g. T10-DIF-TYPE1-CRC.
>> +		If the storage device supports metadata but no PI
>> +		is used, this field will contain "nop".
> 
> This field describes the type of T10 Protection Information that the
> block device is capable of sending and receiving. If the device does not
> support sending and receiving T10 Protection Information, this field
> contains "nop".

There can be "none" and "nop" here. My understanding was that "none"
means the device does not support any metadata space (no T10 profile possible)
while "nop" is that there is a metadata space but it is not used by
any known T10 profile. Is it correct?

What I need (in userspace):

  - a flag that device supports non-PI metadata (is it that "nop" above?)
    If not, then there is no way to check for non-PI metadata for non-NVMe
    devices (as metadata_bytes is present on NVMe only)

  -  maximal size of usable metadata (currently NVMe metadata_bytes field).

...

> Wrt. the size of opaque (non-PI) metadata, I think it would good to have
> a dedicated field to describe it.

I think that metadata_bytes would be enough, if supported for all block devices
(note we emulate metadata in dm-integrity, so it should be set there too).

Thanks,
Milan

Re: [PATCH] docs: sysfs-block: Clarify integrity sysfs attributes for non-PI metadata

[Martin K. Petersen]

Milan,

> thanks for the clarification, should I just update the
> patch and add your signed-off line?

Feel free to tweak and submit. You can add a Co-developed-by tag, if you
wish.

> There can be "none" and "nop" here. My understanding was that "none"
> means the device does not support any metadata space (no T10 profile
> possible) while "nop" is that there is a metadata space but it is not
> used by any known T10 profile. Is it correct?

Looks like it, yes. We didn't originally register a profile unless the
device was capable. So the "nop" vs. "none" distinction is a recent
addition.

>  - a flag that device supports non-PI metadata (is it that "nop"
>  above?) If not, then there is no way to check for non-PI metadata for
>  non-NVMe devices (as metadata_bytes is present on NVMe only)

Currently, yes.

>  -  maximal size of usable metadata (currently NVMe metadata_bytes
>  field).

Aside from PI, SCSI doesn't support separate metadata. It does, however,
support larger logical block sizes. So 520, 524, 528, etc. And those
block sizes may be accompanied by 8 bytes of PI. But the non-PI metadata
is considered part of the logical block data and not a separate entity
like in NVMe.

So until NVMe happened, we didn't have a situation where we could
actually have non-PI metadata in a buffer separate from the data. And
the block integrity interface still reflects that.

> I think that metadata_bytes would be enough, if supported for all
> block devices (note we emulate metadata in dm-integrity, so it should
> be set there too).

That's fine with me. I do prefer to distinguish between PI and non-PI
metadata. Even though they may be sharing a buffer in the NVMe case.

-- 
Martin K. Petersen	Oracle Linux Engineering

[PATCH] docs: sysfs-block: Clarify integrity sysfs attributes

[Milan Broz]

The /sys/block/<disk>/integrity fields are historically set
if T10 protection Information is enabled.

It is not set if some upper layer uses integrity metadata.
Document it.

Signed-off-by: Milan Broz <gmazyland@gmail.com>
Co-developed-by: Martin K. Petersen <martin.petersen@oracle.com>
---
 Documentation/ABI/stable/sysfs-block | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block
index 0cceb2badc83..16e485c05d36 100644
--- a/Documentation/ABI/stable/sysfs-block
+++ b/Documentation/ABI/stable/sysfs-block
@@ -109,6 +109,10 @@ Contact:	Martin K. Petersen <martin.petersen@oracle.com>
 Description:
 		Indicates whether a storage device is capable of storing
 		integrity metadata. Set if the device is T10 PI-capable.
+		This flag is set to 1 if the storage media is formatted
+		with T10 Protection Information. If the storage media is
+		not formatted with T10 Protection Information, this flag
+		is set to 0.
 
 
 What:		/sys/block/<disk>/integrity/format
@@ -117,6 +121,13 @@ Contact:	Martin K. Petersen <martin.petersen@oracle.com>
 Description:
 		Metadata format for integrity capable block device.
 		E.g. T10-DIF-TYPE1-CRC.
+		This field describes the type of T10 Protection Information
+		that the block device can send and receive.
+		If the device can store application integrity metadata but
+		no T10 Protection Information profile is used, this field
+		contains "nop".
+		If the device does not support integrity metadata, this
+		field contains "none".
 
 
 What:		/sys/block/<disk>/integrity/protection_interval_bytes
@@ -142,7 +153,17 @@ Date:		June 2008
 Contact:	Martin K. Petersen <martin.petersen@oracle.com>
 Description:
 		Number of bytes of integrity tag space available per
-		512 bytes of data.
+		protection_interval_bytes, which is typically
+		the device's logical block size.
+		This field describes the size of the application tag
+		if the storage device is formatted with T10 Protection
+		Information and permits use of the application tag.
+		The tag_size is reported in bytes and indicates the
+		space available for adding an opaque tag to each block
+		(protection_interval_bytes).
+		If the device does not support T10 Protection Information
+		(even if the device provides application integrity
+		metadata space), this field is set to 0.
 
 
 What:		/sys/block/<disk>/integrity/write_generate
-- 
2.47.2

Re: [PATCH] docs: sysfs-block: Clarify integrity sysfs attributes

[Christoph Hellwig]

Looks good:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: [PATCH] docs: sysfs-block: Clarify integrity sysfs attributes

[Jens Axboe]

On Tue, 18 Mar 2025 16:44:47 +0100, Milan Broz wrote:
> The /sys/block/<disk>/integrity fields are historically set
> if T10 protection Information is enabled.
> 
> It is not set if some upper layer uses integrity metadata.
> Document it.
> 
> 
> [...]

Applied, thanks!

[1/1] docs: sysfs-block: Clarify integrity sysfs attributes
      commit: fc22b34e95ce0a294c797c397a9db671e6ff4448

Best regards,
-- 
Jens Axboe