Re: [PATCH v2] block: Increase BLK_DEF_MAX_SECTORS_CAP

[Damien Le Moal <dlemoal@kernel.org>]

On 4/1/26 19:32, Mira Limbeck wrote:
> Sorry if I wasn't clear enough, we did test the following mainline
> kernels without any downstream patches (git tags):
> v6.16 (unaffected)
> v6.17 (affected)
> v7.0-rc5 (affected)
> 
> Afterwards we started to bisect between mainline 6.16
> (038d61fd642278bab63ee8ef722c50d10ab01e8f) and mainline 6.17
> (e5f0a698b34ed76002dc5cff3804a61c80233a7a) without any downstream
> patches, which led us to this commit as the first bad one:
> 9b8b84879d4adc506b0d3944e20b28d9f3f6994b

Note: the proper way to reference a patch is to use 12-digits commit ID and
patch title:

9b8b84879d4a ("block: Increase BLK_DEF_MAX_SECTORS_CAP")

as that make it easier to know what one is talking about without having to go
look what patch that ID references.

> Building our downstream kernel 6.17 with this commit reverted, fixed it.

Nope, this is likely not fixing anything but rather hiding the issue. With this
patch reverted, the default max_sectors_kb will be 1280, so all requests will be
chunked to that size at most, and your devices will not see large commands.
However, simply doing something like:

echo 4096 > /sys/block/<dev>/queue/max_sectors_kb

will put your system in a state that is equivalent to the patch being applied
and you will likely see the issue again. Try.

> To make sure that's also the case for the current mainline kernel, we've
> tried 7.0-rc6 today (v7.0-rc6, 7aaa8047eafd0bd628065b15757d9b48c5f9c07d,
> affected), and again with this commit reverted (unaffected).

Same comment as above.

> Here the logs from 7.0-rc6:
> 
> Apr 01 11:41:19 pve-test-hba kernel: sd 9:2:2:0: [sdc] tag#3962 page boundary curr_buff: 0x00000000f4d7cfce
> Apr 01 11:41:19 pve-test-hba kernel: BUG: unable to handle page fault for address: ff3a241243d70000
> Apr 01 11:41:19 pve-test-hba kernel: #PF: supervisor write access in kernel mode
> Apr 01 11:41:19 pve-test-hba kernel: #PF: error_code(0x0002) - not-present page
> Apr 01 11:41:19 pve-test-hba kernel: PGD 100010067 P4D 10066d067 PUD 10066e067 PMD 11f0fa067 PTE 0
> Apr 01 11:41:19 pve-test-hba kernel: Oops: Oops: 0002 [#3] SMP NOPTI
> Apr 01 11:41:19 pve-test-hba kernel: CPU: 15 UID: 0 PID: 6695 Comm: vgs Tainted: G      D W   E       7.0.0-rc6 #19 PREEMPT(full)
> Apr 01 11:41:19 pve-test-hba kernel: Tainted: [D]=DIE, [W]=WARN, [E]=UNSIGNED_MODULE
> Apr 01 11:41:19 pve-test-hba kernel: Hardware name: <snip>
> Apr 01 11:41:19 pve-test-hba kernel: RIP: 0010:_base_build_sg_scmd_ieee+0x478/0x590 [mpt3sas]

There may be an issue with the mpt3sas driver with large commands.

However, I am using that driver all day long and doing lots of testing with
gigantic read/write commands all the time. I have never seen any issues.
The difference is that I am using the SAS-SATA FW for the Broadcom HBA, so no
NVMe support, and my target devices are SAS or SATA HDDs, not SSDs.

Something may be wrong with the NVMe support in that HBA, or, your SSDs do not
like large commands and cause issues. That is easy to test: try connecting your
SSDs directly to PCI and test them by issuing large read/write commands with fio
(you will need to use iomem=mmaphuge option to use hugepages for the IO buffers
to ensure that you do not get the IOs chunked into small commands due to memory
fragmentation).

At least from my point of view and my tests, that commit is perfectly fine. As
mentioned above, it is only changing the default value, and that's something
that can be done manually even without this patch. So this is definitely not the
root cause and it is simply exposing a problem that already existed.
We have seen that for several devices in the ata subsystem and had to quirk many
drives that are choking on large commands.

We just need to figure out where the problem is (HBA and/or SSD) and can then
look into how to avoid that problem.

-- 
Damien Le Moal
Western Digital Research