Re: NVMe CLI Invalid PRP Entry Status Failures

[Christoph Hellwig <hch@infradead.org>]

On Sat, May 24, 2025 at 08:26:17AM -0600, Keith Busch wrote:
> > There is no issue if nvme-cli sends a transfer length of up to 128K.
> > Anything more than 128K is failed as ENINVAL. I guess this is coming
> > from the limitation of BIO_MAX_VECS as 256. Since this was working on
> > older kernels, did anything change in this regard?
> 
> Well, it's passing a virtually contiguous address, so if we assume your
> page size is 4k, 256 vectors would allow up to 1MB without a problem.
> 
> But the NVMe pci driver has its own limit of 128 vectors, so 512k is the
> largest you can safely go before you need to increase the size of pages
> via hugetlbfs.
> 
> But you say you're doing something smaller than 512k, and your device's
> MDTS is bigger than that, so what else is limiting here your transfer
> size here? Do you have some udev rule that is reducing the size of your
> max sectors value?
> 
> Check the value of /sys/block/nvmeXnY/queue/max_sectors_kb
> 
> See if it matches /sys/block/nvmeXnY/queue/max_hw_sectors_kb

bio_split_rw_at doesn't look at max_sectors unless that is passed
in as the argument.

It would be good to just throw in debug printks to see which splitting
decision in bio_split_rw_at triggers, including checking the exact
condition in bvec_split_segs.