dm-integrity: 4k devices report 512b discard alignment

dm-integrity: 4k devices report 512b discard alignment

[Eric Wheeler]

Hello all,

I'm not sure if this is a bug or a feature, so thought I'd report it.
These are all for Linux 6.6.26:

If you start with a 512-byte backing device (logical and physical):
  # blockdev --getss --getpbsz  /dev/nvme9n9
  512
  512

then format+open with 4k sectors:
  # integritysetup format -s 4096 /dev/nvme9n9
  # integritysetup open /dev/nvme9n9 integrity-nvme9n9

and find the dm-X device:
  # ls -l /dev/mapper/integrity-nvme9n9
  lrwxrwxrwx. 1 root root 9 Apr 13 17:09 /dev/mapper/integrity-nvme9n9 -> ../dm-189

then check it's discard_granularity, it shows 512:
  # tail /sys/block/dm-189/queue/discard_*
  ==> /sys/block/dm-189/queue/discard_granularity <==
  512

The sector size is 4k, so supporting unaligned discards seems at least
strange, if not invalid, and dmesg complains too:
  # blockdev --getss /dev/mapper/integrity-nvme9n9
  4096

Relatedly, should the physical size of an integrity volume "properly"
report 512 to match the backing device, or is 4k correct in this case?
Presently it reports 4k:
  # blockdev --getpbsz /dev/mapper/integrity-nvme2n1
  4096

Finally, this is what made me go investiage: when you issue `blkdiscard`
on the integrity volume:
  # blkdiscard /dev/mapper/integrity-nvme9n9

it causes dmesg spews the following:
[ 2508.364637] device-mapper: integrity: Bio not aligned on 8 sectors: 0x74ff8000, 0x7f0f
[ 2508.364856] device-mapper: integrity: Bio not aligned on 8 sectors: 0x74ffff16, 0xea
[ 2508.365070] device-mapper: integrity: Bio not aligned on 8 sectors: 0x757fff0e, 0x7
[ 2508.365973] device-mapper: integrity: Bio not aligned on 8 sectors: 0x757f8000, 0x7f0e

What do you think?


--
Eric Wheeler

Re: dm-integrity: 4k devices report 512b discard alignment

[Milan Broz]

On 4/14/24 3:05 AM, Eric Wheeler wrote:
> Hello all,
> 
> I'm not sure if this is a bug or a feature, so thought I'd report it.
> These are all for Linux 6.6.26:
> 
> If you start with a 512-byte backing device (logical and physical):
>    # blockdev --getss --getpbsz  /dev/nvme9n9
>    512
>    512

Hi,

this looks wrong. But you should send it to dm-devel list (added to cc) ...

Maybe also add lsblk -D (and lsblk -t for block sizes)
(where you can see inherited values for all layers).

And I see similar problem in dm-crypt with 4k sector but 512B discard granularity
(on 6.8.4 - LUKS with forced 4k sector and alllow_discards).

What is interesting, for current Linus' tree I see discard granularity 0 (?), but
with Mike's DM for-next (with use queue_limits_set fixes) it correctly shows 4k.

(Ditto for your integrity case. So perhaps linux-next is fixed.)

Milan



> 
> then format+open with 4k sectors:
>    # integritysetup format -s 4096 /dev/nvme9n9
>    # integritysetup open /dev/nvme9n9 integrity-nvme9n9
> 
> and find the dm-X device:
>    # ls -l /dev/mapper/integrity-nvme9n9
>    lrwxrwxrwx. 1 root root 9 Apr 13 17:09 /dev/mapper/integrity-nvme9n9 -> ../dm-189
> 
> then check it's discard_granularity, it shows 512:
>    # tail /sys/block/dm-189/queue/discard_*
>    ==> /sys/block/dm-189/queue/discard_granularity <==
>    512
> 
> The sector size is 4k, so supporting unaligned discards seems at least
> strange, if not invalid, and dmesg complains too:
>    # blockdev --getss /dev/mapper/integrity-nvme9n9
>    4096
> 
> Relatedly, should the physical size of an integrity volume "properly"
> report 512 to match the backing device, or is 4k correct in this case?
> Presently it reports 4k:
>    # blockdev --getpbsz /dev/mapper/integrity-nvme2n1
>    4096
> 
> Finally, this is what made me go investiage: when you issue `blkdiscard`
> on the integrity volume:
>    # blkdiscard /dev/mapper/integrity-nvme9n9
> 
> it causes dmesg spews the following:
> [ 2508.364637] device-mapper: integrity: Bio not aligned on 8 sectors: 0x74ff8000, 0x7f0f
> [ 2508.364856] device-mapper: integrity: Bio not aligned on 8 sectors: 0x74ffff16, 0xea
> [ 2508.365070] device-mapper: integrity: Bio not aligned on 8 sectors: 0x757fff0e, 0x7
> [ 2508.365973] device-mapper: integrity: Bio not aligned on 8 sectors: 0x757f8000, 0x7f0e
> 
> What do you think?
> 
> 
> --
> Eric Wheeler

Re: dm-integrity: 4k devices report 512b discard alignment

[Eric Wheeler]

[-- Attachment #1: Type: text/plain, Size: 3367 bytes --]

On Sun, 14 Apr 2024, Milan Broz wrote:
> On 4/14/24 3:05 AM, Eric Wheeler wrote:
> > Hello all,
> > 
> > I'm not sure if this is a bug or a feature, so thought I'd report it.
> > These are all for Linux 6.6.26:
> > 
> > If you start with a 512-byte backing device (logical and physical):
> >    # blockdev --getss --getpbsz  /dev/nvme9n9
> >    512
> >    512
> 
> Hi,
> 
> this looks wrong. But you should send it to dm-devel list (added to cc) ...
> 
> Maybe also add lsblk -D (and lsblk -t for block sizes)
> (where you can see inherited values for all layers).

ok here it is for the same 6.6.27. 

# lsblk -o NAME,MIN-IO,PHY-SEC,LOG-SEC,DISC-GRAN,DISC-MAX /dev/nvme[12]n1
NAME                           MIN-IO PHY-SEC LOG-SEC DISC-GRAN DISC-MAX
nvme2n1                           512     512     512      512B       2T
└─integrity-0025385681b3003f     4096    4096    4096      512B       2T
  └─md2                      16777216    4096    4096      512B       2T

nvme1n1                           512     512     512      512B       2T
└─integrity-0025385681b30055     4096    4096    4096      512B       2T
  └─md2                      16777216    4096    4096      512B       2T


> And I see similar problem in dm-crypt with 4k sector but 512B discard
> granularity
> (on 6.8.4 - LUKS with forced 4k sector and alllow_discards).
> 
> What is interesting, for current Linus' tree I see discard granularity 0 (?),
> but
> with Mike's DM for-next (with use queue_limits_set fixes) it correctly shows
> 4k.
> 
> (Ditto for your integrity case. So perhaps linux-next is fixed.)

Good to know.

Do you know which commit might fix this for stable, and was it marked `Cc: stable@`?


--
Eric Wheeler



> 
> Milan
> 
> 
> 
> > 
> > then format+open with 4k sectors:
> >    # integritysetup format -s 4096 /dev/nvme9n9
> >    # integritysetup open /dev/nvme9n9 integrity-nvme9n9
> > 
> > and find the dm-X device:
> >    # ls -l /dev/mapper/integrity-nvme9n9
> >    lrwxrwxrwx. 1 root root 9 Apr 13 17:09 /dev/mapper/integrity-nvme9n9 ->
> >    ../dm-189
> > 
> > then check it's discard_granularity, it shows 512:
> >    # tail /sys/block/dm-189/queue/discard_*
> >    ==> /sys/block/dm-189/queue/discard_granularity <==
> >    512
> > 
> > The sector size is 4k, so supporting unaligned discards seems at least
> > strange, if not invalid, and dmesg complains too:
> >    # blockdev --getss /dev/mapper/integrity-nvme9n9
> >    4096
> > 
> > Relatedly, should the physical size of an integrity volume "properly"
> > report 512 to match the backing device, or is 4k correct in this case?
> > Presently it reports 4k:
> >    # blockdev --getpbsz /dev/mapper/integrity-nvme2n1
> >    4096
> > 
> > Finally, this is what made me go investiage: when you issue `blkdiscard`
> > on the integrity volume:
> >    # blkdiscard /dev/mapper/integrity-nvme9n9
> > 
> > it causes dmesg spews the following:
> > [ 2508.364637] device-mapper: integrity: Bio not aligned on 8 sectors:
> > 0x74ff8000, 0x7f0f
> > [ 2508.364856] device-mapper: integrity: Bio not aligned on 8 sectors:
> > 0x74ffff16, 0xea
> > [ 2508.365070] device-mapper: integrity: Bio not aligned on 8 sectors:
> > 0x757fff0e, 0x7
> > [ 2508.365973] device-mapper: integrity: Bio not aligned on 8 sectors:
> > 0x757f8000, 0x7f0e
> > 
> > What do you think?
> > 
> > 
> > --
> > Eric Wheeler
> 
>