Re: [Lsf-pc] [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] Meta/Integrity/PI improvements

["Martin K. Petersen" <martin.petersen@oracle.com>]

Kanchan,

> - Generic user interface that user-space can use to exchange meta. A
> new io_uring opcode IORING_OP_READ/WRITE_META - seems feasible for
> direct IO.

Yep. I'm interested in this too. Reviving this effort is near the top of
my todo list so I'm happy to collaborate.

> NVMe SSD can do the offload when the host sends the PRACT bit. But in
> the driver, this is tied to global integrity disablement using
> CONFIG_BLK_DEV_INTEGRITY.

> So, the idea is to introduce a bio flag REQ_INTEGRITY_OFFLOAD
> that the filesystem can send. The block-integrity and NVMe driver do
> the rest to make the offload work.

Whether to have a block device do this is currently controlled by the
/sys/block/foo/integrity/{read_verify,write_generate} knobs. At least
for SCSI, protected transfers are always enabled between HBA and target
if both support it. If no integrity has been attached to an I/O by the
application/filesystem, the block layer will do so controlled by the
sysfs knobs above. IOW, if the hardware is capable, protected transfers
should always be enabled, at least from the block layer down.

It's possible that things don't work quite that way with NVMe since, at
least for PCIe, the drive is both initiator and target. And NVMe also
missed quite a few DIX details in its PI implementation. It's been a
while since I messed with PI on NVMe, I'll have a look.

But in any case the intent for the Linux code was for protected
transfers to be enabled automatically when possible. If the block layer
protection is explicitly disabled, a filesystem can still trigger
protected transfers via the bip flags. So that capability should
definitely be exposed via io_uring.

> "Work is in progress to implement support for the data integrity
> extensions in btrfs, enabling the filesystem to use the application
> tag."

This didn't go anywhere for a couple of reasons:

 - Individual disk drives supported ATO but every storage array we
   worked with used the app tag space internally. And thus there were
   very few real-life situations where it would be possible to store
   additional information in each block.

   Back in the mid-2000s, putting enterprise data on individual disk
   drives was not considered acceptable. So implementing filesystem
   support that would only be usable on individual disk drives didn't
   seem worth the investment. Especially when the PI-for-ATA efforts
   were abandoned.

   Wrt. the app tag ownership situation in SCSI, the storage tag in NVMe
   spec is a remedy for this, allowing the application to own part of
   the extra tag space and the storage device itself another.

 - Our proposed use case for the app tag was to provide filesystems with
   back pointers without having to change the on-disk format.

   The use of 0xFFFF as escape check in PI meant that the caller had to
   be very careful about what to store in the app tag. Our prototype
   attached structs of metadata to each filesystem block (8 512-byte
   sectors * 2 bytes of PI, so 16 bytes of metadata per filesystem
   block). But none of those 2-byte blobs could contain the value
   0xFFFF. Wasn't really a great interface for filesystems that wanted
   to be able to attach whatever data structure was important to them.

So between a very limited selection of hardware actually providing the
app tag space and a clunky interface for filesystems, the app tag just
never really took off. We ended up modifying it to be an access control
instead, see the app tag control mode page in SCSI.

Databases and many filesystems have means to protect blocks or extents.
And these means are often better at identifying the nature of read-time
problems than a CRC over each 512-byte LBA would be. So what made PI
interesting was the ability to catch problems at write time in case of a
bad partition remap, wrong buffer pointer, misordered blocks, etc. Once
the data is on media, the drive ECC is superior. And again, at read time
the database or application is often better equipped to identify
corruption than PI.

And consequently our interest focused on treating PI something more akin
to a network checksum than a facility to protect data at rest on media.

-- 
Martin K. Petersen	Oracle Linux Engineering