kernel 6.12-rc7 device-mapper inconsistency

kernel 6.12-rc7 device-mapper inconsistency

[dap581]

Hello,
Starting with kernel 6.11.x series up to current 6.12-rc7
when I unlock my encrypted hdd (storage, not root) with cryptsetup:

cryptsetup luksOpen /dev/sda1 hdd5 -d keyfile_hdd5

it triggers an alignment inconsistency:
In dmesg log, I read:
----
[  105.841278] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
[  105.841292] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
[  105.841601] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
[  105.841607] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
----

Also, starting with Linux >6.11.x up to current 6.12-rc7, "lsblk -t" returns different values than before in OPT-IO & RA column.

----
With kernel >6.11.x
With device closed:
NAME         ALIGNMENT MIN-IO   OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE    RA WSAME
sda                  0   4096 16776704    4096     512    1 bfq       256 32764    0B
└─sda1               0   4096 16776704    4096     4096    1 bfq       256 32764    0B
After luksOpen:
sda                  0   4096 16776704    4096     512    1 bfq       256 32764    0B
└─sda1               0   4096 16776704    4096     512    1 bfq       256 32764    0B
  └─hdd5            -1   4096        0    4096    4096    1               32764    0B


I tried to track down when this started exactly, to do so, I compiled kernel:
6.7.12, 6.8.12, 6.9.9, 6.10.14
and none of them returns this device-mapper inconsistency.
lsblk -t with kernel 6.10.14 and earlier:
NAME         ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE  RA WSAME
sda                  0   4096      0    4096     512    1 bfq       256 128    0B
└─sda1               0   4096      0    4096     512    1 bfq       256 128    0B
  └─hdd5             0   4096      0    4096    4096    1               128    0B

So far, I have been able to locate this commit (part of Linux 6.11.x)
a23634644afc2f7c1bac98776440a1f3b161819e (block: take io_opt and io_min into account for max_sectors )
that seems to trigger the change in lsblk output and introduces opt-io & ra value
but as a casual user, I can't say for sure, I'm not a developer either.
On the other hand, if I compile a kernel using one or two commit before a23634644afc2f7c1bac98776440a1f3b161819e then the lsblk -t output is not changed and no "device-mapper inconsistency" appears so it seems at least related somehow.
By the way, after enteting my password (or using keyfile) cryptsetup returns no error and the filesystem (btrfs) mounts correctly.

Let me know if you need other info.
See also: https://gitlab.com/cryptsetup/cryptsetup/-/issues/919

System used: Gentoo

The HDD is connected to an SAS HBA (Dell Perc H310 with IT firmware)
Actually, there are also 4 other HDD (same brand, model, capacity, same configuration used) connected to it and they all show the exact same message "device-mapper inconsistency" as soon as there are unlocked.

gdisk -l /dev/sda
----
Disk /dev/sda: 5860533168 sectors, 2.7 TiB
Model: WDC WD30EZRX-00D
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): 4D31A7E7-A2A6-4A5A-8FE1-8368581B78DC
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 5860533134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2924 sectors (1.4 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      5860532224   2.7 TiB     8300  Linux filesystem
----

luksDump /dev/sda1
----
LUKS header information
Version:       2
Epoch:         5
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID:           d1ff1c64-a294-4287-ab33-215327716e96
Label:         hdd5
Subsystem:     (no subsystem)
Flags:       (no flags)

Data segments:
  0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: serpent-xts-plain64
sector: 4096 [bytes]
----

Re: kernel 6.12-rc7 device-mapper inconsistency

[Mikulas Patocka]

[-- Attachment #1: Type: text/plain, Size: 7001 bytes --]

Hi

Please apply this patch and report what it prints. So, that we can find 
out which limits cause the problem.

Mikulas



On Sat, 16 Nov 2024, dap581 wrote:

> Hello,
> Starting with kernel 6.11.x series up to current 6.12-rc7
> when I unlock my encrypted hdd (storage, not root) with cryptsetup:
> 
> cryptsetup luksOpen /dev/sda1 hdd5 -d keyfile_hdd5
> 
> it triggers an alignment inconsistency:
> In dmesg log, I read:
> ----
> [  105.841278] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> [  105.841292] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> [  105.841601] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> [  105.841607] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> ----
> 
> Also, starting with Linux >6.11.x up to current 6.12-rc7, "lsblk -t" 
> returns different values than before in OPT-IO & RA column.
> 
> ----
> With kernel >6.11.x
> With device closed:
> NAME         ALIGNMENT MIN-IO   OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE    RA WSAME
> sda                  0   4096 16776704    4096     512    1 bfq       256 32764    0B
> └─sda1               0   4096 16776704    4096     4096    1 bfq       256 32764    0B
> After luksOpen:
> sda                  0   4096 16776704    4096     512    1 bfq       256 32764    0B
> └─sda1               0   4096 16776704    4096     512    1 bfq       256 32764    0B
>   └─hdd5            -1   4096        0    4096    4096    1               32764    0B
> 
> 
> I tried to track down when this started exactly, to do so, I compiled kernel:
> 6.7.12, 6.8.12, 6.9.9, 6.10.14
> and none of them returns this device-mapper inconsistency.
> lsblk -t with kernel 6.10.14 and earlier:
> NAME         ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE  RA WSAME
> sda                  0   4096      0    4096     512    1 bfq       256 128    0B
> └─sda1               0   4096      0    4096     512    1 bfq       256 128    0B
>   └─hdd5             0   4096      0    4096    4096    1               128    0B
> 
> So far, I have been able to locate this commit (part of Linux 6.11.x) 
> a23634644afc2f7c1bac98776440a1f3b161819e (block: take io_opt and io_min 
> into account for max_sectors ) that seems to trigger the change in lsblk 
> output and introduces opt-io & ra value but as a casual user, I can't 
> say for sure, I'm not a developer either. On the other hand, if I 
> compile a kernel using one or two commit before 
> a23634644afc2f7c1bac98776440a1f3b161819e then the lsblk -t output is not 
> changed and no "device-mapper inconsistency" appears so it seems at 
> least related somehow. By the way, after enteting my password (or using 
> keyfile) cryptsetup returns no error and the filesystem (btrfs) mounts 
> correctly.
> 
> Let me know if you need other info.
> See also: https://gitlab.com/cryptsetup/cryptsetup/-/issues/919
> 
> System used: Gentoo
> 
> The HDD is connected to an SAS HBA (Dell Perc H310 with IT firmware) 
> Actually, there are also 4 other HDD (same brand, model, capacity, same 
> configuration used) connected to it and they all show the exact same 
> message "device-mapper inconsistency" as soon as there are unlocked.
> 
> gdisk -l /dev/sda
> ----
> Disk /dev/sda: 5860533168 sectors, 2.7 TiB
> Model: WDC WD30EZRX-00D
> Sector size (logical/physical): 512/4096 bytes
> Disk identifier (GUID): 4D31A7E7-A2A6-4A5A-8FE1-8368581B78DC
> Partition table holds up to 128 entries
> Main partition table begins at sector 2 and ends at sector 33
> First usable sector is 34, last usable sector is 5860533134
> Partitions will be aligned on 2048-sector boundaries
> Total free space is 2924 sectors (1.4 MiB)
> 
> Number  Start (sector)    End (sector)  Size       Code  Name
>    1            2048      5860532224   2.7 TiB     8300  Linux filesystem
> ----
> 
> luksDump /dev/sda1
> ----
> LUKS header information
> Version:       2
> Epoch:         5
> Metadata area: 16384 [bytes]
> Keyslots area: 16744448 [bytes]
> UUID:           d1ff1c64-a294-4287-ab33-215327716e96
> Label:         hdd5
> Subsystem:     (no subsystem)
> Flags:       (no flags)
> 
> Data segments:
>   0: crypt
> offset: 16777216 [bytes]
> length: (whole device)
> cipher: serpent-xts-plain64
> sector: 4096 [bytes]
> ----
> 
> 
---
 block/blk-settings.c |    6 ++++++
 1 file changed, 6 insertions(+)

Index: linux-2.6/block/blk-settings.c
===================================================================
--- linux-2.6.orig/block/blk-settings.c	2024-09-30 16:40:54.000000000 +0200
+++ linux-2.6/block/blk-settings.c	2024-11-18 14:25:19.000000000 +0100
@@ -557,6 +557,7 @@ int blk_stack_limits(struct queue_limits
 
 		/* Verify that top and bottom intervals line up */
 		if (max(top, bottom) % min(top, bottom)) {
+			printk("failure 1: %x, %x\n", top, bottom);
 			t->flags |= BLK_FLAG_MISALIGNED;
 			ret = -1;
 		}
@@ -578,6 +579,7 @@ int blk_stack_limits(struct queue_limits
 
 	/* Physical block size a multiple of the logical block size? */
 	if (t->physical_block_size & (t->logical_block_size - 1)) {
+		printk("failure 2: %x, %x\n", t->physical_block_size, t->logical_block_size);
 		t->physical_block_size = t->logical_block_size;
 		t->flags |= BLK_FLAG_MISALIGNED;
 		ret = -1;
@@ -585,6 +587,7 @@ int blk_stack_limits(struct queue_limits
 
 	/* Minimum I/O a multiple of the physical block size? */
 	if (t->io_min & (t->physical_block_size - 1)) {
+		printk("failure 3: %x, %x\n", t->io_min, t->physical_block_size);
 		t->io_min = t->physical_block_size;
 		t->flags |= BLK_FLAG_MISALIGNED;
 		ret = -1;
@@ -592,6 +595,7 @@ int blk_stack_limits(struct queue_limits
 
 	/* Optimal I/O a multiple of the physical block size? */
 	if (t->io_opt & (t->physical_block_size - 1)) {
+		printk("failure 4: %x, %x\n", t->io_opt, t->physical_block_size);
 		t->io_opt = 0;
 		t->flags |= BLK_FLAG_MISALIGNED;
 		ret = -1;
@@ -599,6 +603,7 @@ int blk_stack_limits(struct queue_limits
 
 	/* chunk_sectors a multiple of the physical block size? */
 	if ((t->chunk_sectors << 9) & (t->physical_block_size - 1)) {
+		printk("failure 5: %x, %x\n", t->chunk_sectors, t->physical_block_size);
 		t->chunk_sectors = 0;
 		t->flags |= BLK_FLAG_MISALIGNED;
 		ret = -1;
@@ -610,6 +615,7 @@ int blk_stack_limits(struct queue_limits
 
 	/* Verify that new alignment_offset is on a logical block boundary */
 	if (t->alignment_offset & (t->logical_block_size - 1)) {
+		printk("failure 6: %x, %x\n", t->alignment_offset, t->logical_block_size);
 		t->flags |= BLK_FLAG_MISALIGNED;
 		ret = -1;
 	}

[PATCH] blk-settings: round down io_opt to physical_block_size

[Mikulas Patocka]

On Mon, 18 Nov 2024, Mikulas Patocka wrote:

> On Sat, 16 Nov 2024, dap581 wrote:
> 
> > Hello,
> > Starting with kernel 6.11.x series up to current 6.12-rc7
> > when I unlock my encrypted hdd (storage, not root) with cryptsetup:
> > 
> > cryptsetup luksOpen /dev/sda1 hdd5 -d keyfile_hdd5
> > 
> > it triggers an alignment inconsistency:
> > In dmesg log, I read:
> > ----
> > [  105.841278] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > [  105.841292] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > [  105.841601] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > [  105.841607] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > ----

Hi

Does this patch fix it?

Mikulas



From: Mikulas Patocka <mpatocka@redhat.com>

There was a bug report [1] where the user got a warning alignment
inconsistency. The user has optimal I/O 16776704 (0xFFFE00) and physical
block size 4096. Note that the optimal I/O size may be set by the DMA
engines or SCSI controllers and they have no knowledge about the disks
attached to them, so the situation with optimal I/O not aligned to
physical block size may happen.

This commit makes blk_validate_limits round down optimal I/O size to the
physical block size of the block device.

Closes: https://lore.kernel.org/dm-devel/1426ad71-79b4-4062-b2bf-84278be66a5d@redhat.com/T/ [1]
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Fixes: a23634644afc ("block: take io_opt and io_min into account for max_sectors")
Cc: stable@vger.kernel.org	# v6.11+

---
 block/blk-settings.c |    7 +++++++
 1 file changed, 7 insertions(+)

Index: linux-2.6/block/blk-settings.c
===================================================================
--- linux-2.6.orig/block/blk-settings.c	2024-11-18 15:24:18.000000000 +0100
+++ linux-2.6/block/blk-settings.c	2024-11-18 15:29:24.000000000 +0100
@@ -250,6 +250,13 @@ static int blk_validate_limits(struct qu
 		lim->io_min = lim->physical_block_size;
 
 	/*
+	 * The optimal I/O size may not be aligned to physical block size
+	 * (because it may be limited by dma engines which have no clue about
+	 * block size of the disks attached to them), so we round it down here.
+	 */
+	lim->io_opt = round_down(lim->io_opt, lim->physical_block_size);
+
+	/*
 	 * max_hw_sectors has a somewhat weird default for historical reason,
 	 * but driver really should set their own instead of relying on this
 	 * value.

Re: [PATCH] blk-settings: round down io_opt to physical_block_size

[Christoph Hellwig]

On Mon, Nov 18, 2024 at 03:52:50PM +0100, Mikulas Patocka wrote:
> 
> 
> On Mon, 18 Nov 2024, Mikulas Patocka wrote:
> 
> > On Sat, 16 Nov 2024, dap581 wrote:
> > 
> > > Hello,
> > > Starting with kernel 6.11.x series up to current 6.12-rc7
> > > when I unlock my encrypted hdd (storage, not root) with cryptsetup:
> > > 
> > > cryptsetup luksOpen /dev/sda1 hdd5 -d keyfile_hdd5
> > > 
> > > it triggers an alignment inconsistency:
> > > In dmesg log, I read:
> > > ----
> > > [  105.841278] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > > [  105.841292] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > > [  105.841601] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > > [  105.841607] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > > ----
> 
> Hi
> 
> Does this patch fix it?

FYI, this rounding down looks reasonable to me:

Reviewed-by: Christoph Hellwig <hch@lst.de>

Re: kernel 6.12-rc7 device-mapper inconsistency

[dap581]

>On Monday, November 18th, 2024 at 1:27 PM, Mikulas Patocka <mpatocka@redhat.com> wrote:
> Hi 
> Please apply this patch and report what it prints. So, that we can find
> out which limits cause the problem.
> Mikulas

Hello,
Sorry for the delay, with patch applied to 6.12.0 released today:
dmesg output after running luksOpen on hdd (sda)

[  109.421097] failure 4: fffe00, 1000
[  109.421112] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
[  109.421123] failure 4: fffe00, 1000
[  109.421126] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
[  109.421379] failure 4: fffe00, 1000
[  109.421384] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
[  109.421392] failure 4: fffe00, 1000
[  109.421395] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216

>Hi
>Does this patch fix it?
---
 block/blk-settings.c |    7 +++++++
 1 file changed, 7 insertions(+)

Index: linux-2.6/block/blk-settings.c
===================================================================
--- linux-2.6.orig/block/blk-settings.c	2024-11-18 15:24:18.000000000 +0100
+++ linux-2.6/block/blk-settings.c	2024-11-18 15:29:24.000000000 +0100
@@ -250,6 +250,13 @@ static int blk_validate_limits(struct qu
 		lim->io_min = lim->physical_block_size;
 
 	/*
+	 * The optimal I/O size may not be aligned to physical block size
+	 * (because it may be limited by dma engines which have no clue about
+	 * block size of the disks attached to them), so we round it down here.
+	 */
+	lim->io_opt = round_down(lim->io_opt, lim->physical_block_size);
+
+	/*
 	 * max_hw_sectors has a somewhat weird default for historical reason,
 	 * but driver really should set their own instead of relying on this
 	 * value.

----

Yes!
With the above patch applied (blk-settings: round down io_opt to physical_block_size)
No more "device-mapper inconsistency" when I unlock my hdd, I also tried the other hdd to make sure.
All good.

lsblk -t

NAME          ALIGNMENT MIN-IO   OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE    RA WSAME
sda                   0   4096 16773120    4096     512    1 bfq       256 32760    0B
└─sda1                0   4096 16773120    4096     512    1 bfq       256 32760    0B
  └─raid-hdd4         0   4096 16773120    4096    4096    1               32760    0B
sdb                   0   4096 16773120    4096     512    1 bfq       256 32760    0B
└─sdb1                0   4096 16773120    4096     512    1 bfq       256 32760    0B
  └─raid-hdd3         0   4096 16773120    4096    4096    1               32760    0B
sdc                   0   4096 16773120    4096     512    1 bfq       256 32760    0B
└─sdc1                0   4096 16773120    4096     512    1 bfq       256 32760    0B
  └─raid-hdd2         0   4096 16773120    4096    4096    1               32760    0B
sdd                   0   4096 16773120    4096     512    1 bfq       256 32760    0B
└─sdd1                0   4096 16773120    4096     512    1 bfq       256 32760    0B
  └─raid-hdd1         0   4096 16773120    4096    4096    1               32760    0B

Thanks!

Re: [PATCH] blk-settings: round down io_opt to physical_block_size

[Jens Axboe]

On Mon, 18 Nov 2024 15:52:50 +0100, Mikulas Patocka wrote:
> On Mon, 18 Nov 2024, Mikulas Patocka wrote:
> 
> > On Sat, 16 Nov 2024, dap581 wrote:
> >
> > > Hello,
> > > Starting with kernel 6.11.x series up to current 6.12-rc7
> > > when I unlock my encrypted hdd (storage, not root) with cryptsetup:
> > >
> > > cryptsetup luksOpen /dev/sda1 hdd5 -d keyfile_hdd5
> > >
> > > it triggers an alignment inconsistency:
> > > In dmesg log, I read:
> > > ----
> > > [  105.841278] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > > [  105.841292] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > > [  105.841601] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > > [  105.841607] device-mapper: table: 254:1: adding target device sda1 caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=16777216
> > > ----
> 
> [...]

Applied, thanks!

[1/1] blk-settings: round down io_opt to physical_block_size
      commit: df199821cd4bf0c77c4fc4ca2fa978cedf5c8bcf

Best regards,
-- 
Jens Axboe