nvme-pci: Micron 2200 (1344:5410) needs NVME_QUIRK_DMAPOOL_ALIGN_512

nvme-pci: Micron 2200 (1344:5410) needs NVME_QUIRK_DMAPOOL_ALIGN_512

[Omar Alzand]

[-- Attachment #1: Type: text/plain, Size: 4245 bytes --]

Hello,

My Micron 2200V returns a bad last page on reads of exactly 33 pages
(132 KiB). Setting NVME_QUIRK_DMAPOOL_ALIGN_512 through the nvme.quirks
parameter stopped it in my testing. I think 1344:5410 needs the same
quirk as the Micron 4100AT, added by Bean Huo's patch "nvme-pci: add
NVME_QUIRK_DMAPOOL_ALIGN_512 for Micron 4100AT".

Hardware and software:

  Laptop:    Dell Latitude 5510, BIOS 1.43.0, i7-10610U
  Drive:     Micron_2200V_MTFDHBA512TCK, firmware P1MA0V4
  PCI ID:    1344:5410 (lspci names it "2200S NVMe SSD [Cassandra]")
  Kernel:    7.2.9-1-MANJARO
  IOMMU:     DMA domain type Translated
             ("Intel-IOMMU force enabled due to platform opt in")
  Queue:     max_sectors_kb 512, max_segments 129, scheduler none

Symptom:

Btrfs logged intermittent "csum failed" read errors, while a read-only
scrub found nothing and retrying the read succeeded. The journal shows
these errors on 6.18.x as well, from April 2026 onward. The quirks
parameter isn't available in my 6.18.55 build, so I could only test the
fix on 7.2.9.

Raw device test:

This test bypasses the filesystem. It builds a reference copy of each
location with single-page O_DIRECT reads. Then 8 threads read N pages
with O_DIRECT from /dev/nvme0n1 into a buffer pre-filled with 0xAA,
while 8 more threads issue mixed-size reads in the background. Each
size ran for 12 seconds.

  pages    KiB    reads     bad pages
      4     16    89389     0
      8     32    80764     0
     16     64    59585     0
     31    124    34906     0
     32    128    23515     0
     33    132    17715     1001, all at page 33
     34    136    13735     0
     35    140    12047     0
     40    160     9864     0
     64    256     6535     0
     65    260     5972     0
    128    512     3427     0

The failure rate at 33 pages is 5.65%, near 1 in 16. That would fit
one of the sixteen 256-byte blocks in a dmapool page being affected,
but I haven't confirmed this.

A 33-page read needs a PRP list of 32 entries, which exactly fills a
256-byte small-pool descriptor. That would make page 33 the final entry
in the list.

In the two bad pages I sampled, 3584 of the 4096 bytes still held the
0xAA marker, so that data never reached the buffer. Where it went, I
don't know.

Differences from the 4100AT description:

That patch says the IOMMU faults the command. I saw no DMAR faults and
no nvme errors in the kernel log, and the reads returned success. So
this may not be identical, and I can't rule out the undelivered data
having gone somewhere else.

Other checks:

 - Setting max_sectors_kb to 128 gave 0 bad pages in 86008 reads
   requested at 33 pages, because those reads get split.
 - Writes looked clean. 2015 write requests of 33 pages to a nodatacow
   file under O_DIRECT, each read back page by page, had 0 bad pages.
 - SMART reports 0 media errors and 100% available spare.
 - fwupd offers no newer firmware for this drive.

With the quirk:

I booted with nvme.quirks=1344:5410:dmapool_align_512. The controller's
quirks file then listed dmapool_align_512, and max_sectors_kb was back
to 512.

  pages    reads     bad pages
     32    47785     0
     33   133280     0
     34    32537     0

At the earlier rate, about 7500 bad pages would be expected at 33
pages. This run used a separate test program with longer runs for 33
pages, but the same read and background load pattern. Reading all of
/usr and /opt with a cold cache also gave 0 Btrfs checksum errors, and
the kernel log showed no faults.

Suggested change, after the existing 1344:5407 entry in nvme_id_table.
I tested the quirk through the module parameter, not this table entry:

{ PCI_DEVICE(0x1344, 0x5410), /* Micron 2200 */
.driver_data = NVME_QUIRK_DMAPOOL_ALIGN_512, },

I have only tested one drive, a 2200V. Other models may share this PCI
ID, and I don't know whether a firmware-specific condition is involved.
Bean, could you check that against what Micron knows about this
erratum?

The test program is attached (read-only, run as root). I can run other
tests or try a patch if that helps.

I used Claude, an AI assistant, to help design the tests and write this
report. The numbers come from runs on my laptop.

Thanks,
Omar Alzand
omaribrahimfadel@gmail.com

[-- Attachment #2: nvme-33page-test.py.txt --]
[-- Type: text/plain, Size: 1832 bytes --]

#!/usr/bin/python3 -I
"""Raw read test for the Micron 2200 33-page PRP-list fault. Read-only. Run as root.
Reads 32, 33 and 34 page requests under mixed load and compares each page with a reference."""
import os,mmap,random,threading,time,collections
PG=4096; DEV="/dev/nvme0n1"; random.seed(7); N=300; MAXP=34
ps=int(open("/sys/class/block/nvme0n1p4/start").read())
print("quirks:",open("/sys/class/nvme/nvme0/quirks").read().strip(),"| max_sectors_kb:",open("/sys/block/nvme0n1/queue/max_sectors_kb").read().strip())
fd=os.open(DEV,os.O_RDONLY|os.O_DIRECT); locs=[ps*512+random.randrange(0,25<<18)*PG for _ in range(N)]; one=mmap.mmap(-1,PG); refs=[]
for l in locs:
    r=bytearray()
    for i in range(MAXP): os.preadv(fd,[one],l+i*PG); r+=bytes(one[:PG])
    refs.append(bytes(r))
def worker(p,stop,st,s):
    g=random.Random(s); n=p*PG; b=mmap.mmap(-1,n); f=os.open(DEV,os.O_RDONLY|os.O_DIRECT); m=b"\xAA"*n
    while not stop.is_set():
        i=g.randrange(N); b.seek(0); b.write(m); os.preadv(f,[b],locs[i]); st["reads"]+=1
        if bytes(b[:n])!=refs[i][:n]:
            for k in range(p):
                if b[k*PG:(k+1)*PG]!=refs[i][k*PG:(k+1)*PG]: st[k+1]+=1
def noise(stop,s):
    g=random.Random(s); f=os.open(DEV,os.O_RDONLY|os.O_DIRECT); bs=[mmap.mmap(-1,p*PG) for p in (3,5,7,9,16,20,64)]
    while not stop.is_set(): os.preadv(f,[g.choice(bs)],locs[g.randrange(N)])
for p,sec in ((32,10),(33,30),(34,10)):
    stop=threading.Event(); st=collections.Counter()
    ts=[threading.Thread(target=worker,args=(p,stop,st,s)) for s in range(8)]+[threading.Thread(target=noise,args=(stop,100+s)) for s in range(8)]
    [t.start() for t in ts]; time.sleep(sec); stop.set(); [t.join() for t in ts]
    print(f"{p} pages: reads {st['reads']}, bad pages by position { {k:v for k,v in st.items() if k!='reads'} }",flush=True)

Re: nvme-pci: Micron 2200 (1344:5410) needs NVME_QUIRK_DMAPOOL_ALIGN_512

[Christoph Hellwig]

Can you send an actual patch for this, please?