Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

[Bagas Sanjaya]

Hi,

I notice a regression report on Bugzilla [1]. Quoting from it:

> I bought a new 4 TB Lexar NM790 and I was using kernel 6.3.13 at the time. It wasn't recognized, with these messages in dmesg:
> 
> [ 358.950147] nvme nvme0: pci function 0000:06:00.0
> [ 358.958327] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
> 
> My other NVMe appears correctly in the nvme list though.
> 
> 
> So I tried using other kernels I had installed at the time: 6.3.7, 6.4.10, 6.5.0rc6, 6.5.0, 6.5.1 and none of these recognized the disk.
> I installed the 6.1.50 lts kernel from arch repositories (I can compile my own too if this would be an issue) and then the device was correctly recognized:
> 
> [    4.654613] nvme 0000:06:00.0: platform quirk: setting simple suspend
> [    4.654632] nvme nvme0: pci function 0000:06:00.0
> [    4.667290] nvme nvme0: allocated 40 MiB host memory buffer.
> [    4.709473] nvme nvme0: 16/0/0 default/read/poll queues
> 
> And then it appears alongside the other nvme:
> [15:58] [6836] [patola@risadinha patola]% sudo nvme list
> Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev  
> --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
> /dev/nvme1n1          /dev/ng1n1            2K36292CEKD9         XPG GAMMIX S11 Pro                       0x1          1.39  TB /   2.05  TB    512   B +  0 B   42B4S9NA
> /dev/nvme0n1          /dev/ng0n1            NF9755R000057P2202   Lexar SSD NM790 4TB                      0x1          4.10  TB /   4.10  TB    512   B +  0 B   12237   
> 
> And I was able to read and write from it, pvcreate and so on, so it's working. But I can't use a higher kernel version so apparently this is a regression.
> 
> There are other people with the same NVMe model (although different capacities) reporting the same issue on this reddit thread: https://www.reddit.com/r/archlinux/comments/15xbxeo/nvme_device_not_ready_aborting_initialisation/
> 
> I am not sure but I think this issue might've been introducted after this patch: https://bugzilla.kernel.org/show_bug.cgi?id=215742

See Bugzilla for the full thread and proposed quirk fix.

Anyway, I'm adding this regression to be tracked by regzbot:

#regzbot introduced: v6.1.50..v6.3.13 https://bugzilla.kernel.org/show_bug.cgi?id=217863
#regzbot link: https://www.reddit.com/r/archlinux/comments/15xbxeo/nvme_device_not_ready_aborting_initialisation/

Thanks.

[1]: https://bugzilla.kernel.org/show_bug.cgi?id=217863

-- 
An old man doll... just what I always wanted! - Clara

Re: Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

[Linux regression tracking (Thorsten Leemhuis)]

On 04.09.23 13:07, Bagas Sanjaya wrote:
> 
> I notice a regression report on Bugzilla [1]. Quoting from it:
>
>> I bought a new 4 TB Lexar NM790 and I was using kernel 6.3.13 at the time. It wasn't recognized, with these messages in dmesg:
>>
>> [ 358.950147] nvme nvme0: pci function 0000:06:00.0
>> [ 358.958327] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
>>
>> My other NVMe appears correctly in the nvme list though.
>>
>>
>> So I tried using other kernels I had installed at the time: 6.3.7, 6.4.10, 6.5.0rc6, 6.5.0, 6.5.1 and none of these recognized the disk.
>> I installed the 6.1.50 lts kernel from arch repositories (I can compile my own too if this would be an issue) and then the device was correctly recognized:
>>
>> [    4.654613] nvme 0000:06:00.0: platform quirk: setting simple suspend
>> [    4.654632] nvme nvme0: pci function 0000:06:00.0
>> [    4.667290] nvme nvme0: allocated 40 MiB host memory buffer.
>> [    4.709473] nvme nvme0: 16/0/0 default/read/poll queues

FWIW, the quoted mail missed one crucial detail:
"""
 Claudio Sampaio 2023-09-02 19:04:29 UTC

Adding the two lines

│ 3457   { PCI_DEVICE(0x1d97, 0x1602), /* Lexar NM790 */
│ 3458   │ .driver_data = NVME_QUIRK_BOGUS_NID, },

in file drivers/nvme/host/pci.c made my NVMe work correctly. Compiled a
new 6.5.1 kernel and everything works.
"""

@NVME maintainers: is there anything more you need from Claudio at this
point?

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Re: Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

[Keith Busch]

On Tue, Sep 05, 2023 at 01:37:36PM +0200, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 04.09.23 13:07, Bagas Sanjaya wrote:
> > 
> > I notice a regression report on Bugzilla [1]. Quoting from it:
> >
> >> I bought a new 4 TB Lexar NM790 and I was using kernel 6.3.13 at the time. It wasn't recognized, with these messages in dmesg:
> >>
> >> [ 358.950147] nvme nvme0: pci function 0000:06:00.0
> >> [ 358.958327] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
> >>
> >> My other NVMe appears correctly in the nvme list though.
> >>
> >>
> >> So I tried using other kernels I had installed at the time: 6.3.7, 6.4.10, 6.5.0rc6, 6.5.0, 6.5.1 and none of these recognized the disk.
> >> I installed the 6.1.50 lts kernel from arch repositories (I can compile my own too if this would be an issue) and then the device was correctly recognized:
> >>
> >> [    4.654613] nvme 0000:06:00.0: platform quirk: setting simple suspend
> >> [    4.654632] nvme nvme0: pci function 0000:06:00.0
> >> [    4.667290] nvme nvme0: allocated 40 MiB host memory buffer.
> >> [    4.709473] nvme nvme0: 16/0/0 default/read/poll queues
> 
> FWIW, the quoted mail missed one crucial detail:
> """
>  Claudio Sampaio 2023-09-02 19:04:29 UTC
> 
> Adding the two lines
> 
> │ 3457   { PCI_DEVICE(0x1d97, 0x1602), /* Lexar NM790 */
> │ 3458   │ .driver_data = NVME_QUIRK_BOGUS_NID, },
> 
> in file drivers/nvme/host/pci.c made my NVMe work correctly. Compiled a
> new 6.5.1 kernel and everything works.
> """
> 
> @NVME maintainers: is there anything more you need from Claudio at this
> point?

Yes: it doesn't really make any sense. The report says the device
stopped showing up with message:

  nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0

That (a) happens long before the mentioned quirk is considered by the
driver, and (b) the "quirk" behavior is now the default in 6.5 and
several of the listed stable kernels anyway.

It more likely sounds like the device is flaky and either never becomes
ready due to some unspecified internal firmware condition, or
inaccurately reports how long it actually needs to become ready in
worst-case-scenario.

Re: Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

[Linux regression tracking (Thorsten Leemhuis)]

On 05.09.23 16:35, Keith Busch wrote:
> On Tue, Sep 05, 2023 at 01:37:36PM +0200, Linux regression tracking (Thorsten Leemhuis) wrote:
>> On 04.09.23 13:07, Bagas Sanjaya wrote:
>>>
>>> I notice a regression report on Bugzilla [1]. Quoting from it:
>>>
>>>> I bought a new 4 TB Lexar NM790 and I was using kernel 6.3.13 at the time. It wasn't recognized, with these messages in dmesg:
>>>>
>>>> [ 358.950147] nvme nvme0: pci function 0000:06:00.0
>>>> [ 358.958327] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
>>>>
>>>> My other NVMe appears correctly in the nvme list though.
>>>>
>>>>
>>>> So I tried using other kernels I had installed at the time: 6.3.7, 6.4.10, 6.5.0rc6, 6.5.0, 6.5.1 and none of these recognized the disk.
>>>> I installed the 6.1.50 lts kernel from arch repositories (I can compile my own too if this would be an issue) and then the device was correctly recognized:
>>>>
>>>> [    4.654613] nvme 0000:06:00.0: platform quirk: setting simple suspend
>>>> [    4.654632] nvme nvme0: pci function 0000:06:00.0
>>>> [    4.667290] nvme nvme0: allocated 40 MiB host memory buffer.
>>>> [    4.709473] nvme nvme0: 16/0/0 default/read/poll queues
>>
>> FWIW, the quoted mail missed one crucial detail:
>> """
>>  Claudio Sampaio 2023-09-02 19:04:29 UTC
>>
>> Adding the two lines
>>
>> │ 3457   { PCI_DEVICE(0x1d97, 0x1602), /* Lexar NM790 */
>> │ 3458   │ .driver_data = NVME_QUIRK_BOGUS_NID, },
>>
>> in file drivers/nvme/host/pci.c made my NVMe work correctly. Compiled a
>> new 6.5.1 kernel and everything works.
>> """
>>
>> @NVME maintainers: is there anything more you need from Claudio at this
>> point?
> 
> Yes: it doesn't really make any sense. The report says the device
> stopped showing up with message:
> 
>   nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
> 
> That (a) happens long before the mentioned quirk is considered by the
> driver, and (b) the "quirk" behavior is now the default in 6.5 and
> several of the listed stable kernels anyway.
> 
> It more likely sounds like the device is flaky and either never becomes
> ready due to some unspecified internal firmware condition, or
> inaccurately reports how long it actually needs to become ready in
> worst-case-scenario.

Thx, I kinda suspected something like that, but I kept my mouth shut, as
I feared comments from the cheap seats might be more harmful then helpful.

But what can Claudio do to find the root cause? Check hardware
(especially the connectors), update firmware, ...? And if that doesn't
lead to anything, bisect the issue?

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Re: Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

[Keith Busch]

On Tue, Sep 05, 2023 at 04:49:11PM +0200, Linux regression tracking (Thorsten Leemhuis) wrote:
> But what can Claudio do to find the root cause? Check hardware
> (especially the connectors), update firmware, ...? And if that doesn't
> lead to anything, bisect the issue?

Try the current 6.5.1 as-is where the failure was previously seen and
verify if this observation is indeed 100% reproducible with the "device
not ready" kernel message. Full system power cycle between tests, too.

If this is truly a regression, my only guess is some platform power
setting that a newer kernel changed. I am currently suspicious of that
right now since 6.5.1 was reported to fail but succeed with a "quirk"
that doesn't accomplish anything. I'm more leaning toward my "device is
not reliable" theory.

Re: Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

[Keith Busch]

On Tue, Sep 05, 2023 at 05:50:06PM +0200, Cláudio Sampaio wrote:
> Hi Thorsten and Keith,
> 
> Thanks for the details. I'm still unsure if responding by email is better
> or adding to the ticket, but here it goes: I have tried for days both with
> complete power off of the machine and cycle-booting all kernels in
> succession and without exception, 6.1.x LTS and the patched 6.5.1 kernel
> always recognize and operate the NVME, whilst the other kernels also fail
> with the same error message. As this is my "production" desktop, though,
> during the week it's more difficult to me to perform tests with it, but I
> will try to do it in a more methodic way and also with 6.5.1 vanilla.
> 
> As for the reason the Lexar doesn't catch the quirk default, I can't say I
> catch the complex logic of the driver activation, but I found out how to
> "fix" for my case because there are three other Lexar models in the pci.c
> file: NM610, NM620 and NM760 (this one with an additional quirk marked on
> it on the code, NVME_QUIRK_IGNORE_DEV_SUBNQN) -- so I guess whatever
> justifies the exception for them also justifies for my model, NM790. Might
> even be the case that I would need NVME_QUIRK_IGNORE_DEV_SUBNQN (not sure
> what it does) like in the NM760 case, but it activates correctly without it.

The existing Lexar quirks for the identifier existed before the default
kernel behavior changed with respect to how identifiers are considered.

But the report says the device failed to enumerate with a "device not
ready" error message. That error message happens *before* identifiers
are checked, so the quirk should be a no-op with respect to that error
message. And the driver abandons the device after printing that message,
so no futher action should be taken no matter what quirk you've set.

In order for this quirk to have any effect at all, the error you should
have seen should look like a "duplicate IDs" message.

Re: Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

[Cláudio Sampaio]

Hi Thorsten and Keith,

Thanks for the details. I'm still unsure if responding by email is
better or adding to the ticket, but here it goes: I have tried for
days both with complete power off of the machine and cycle-booting all
kernels in succession and without exception, 6.1.x LTS and the patched
6.5.1 kernel always recognize and operate the NVME, whilst the other
kernels also fail with the same error message. As this is my
"production" desktop, though, during the week it's more difficult to
me to perform tests with it, but I will try to do it in a more
methodic way and also with 6.5.1 vanilla.

As for the reason the Lexar doesn't catch the quirk default, I can't
say I catch the complex logic of the driver activation, but I found
out how to "fix" for my case because there are three other Lexar
models in the pci.c file: NM610, NM620 and NM760 (this one with an
additional quirk marked on it on the code,
NVME_QUIRK_IGNORE_DEV_SUBNQN) -- so I guess whatever justifies the
exception for them also justifies for my model, NM790. Might even be
the case that I would need NVME_QUIRK_IGNORE_DEV_SUBNQN (not sure what
it does) like in the NM760 case, but it activates correctly without
it.

-- 
Cláudio "Patola" Sampaio
MakerLinux Labs - Campinas, SP
Resume Gmail - YOUTUBE!
Facebook - Facebook da MakerLinux - Lattes

Re: Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

[Felix Yan]

[-- Attachment #1.1: Type: text/plain, Size: 949 bytes --]

On 9/4/23 14:07, Bagas Sanjaya wrote:
> Hi,
> 
> I notice a regression report on Bugzilla [1]. Quoting from it:
> 
>> I bought a new 4 TB Lexar NM790 and I was using kernel 6.3.13 at the time. It wasn't recognized, with these messages in dmesg:
>>
>> [ 358.950147] nvme nvme0: pci function 0000:06:00.0
>> [ 358.958327] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0

Hi,

This looks very much the same as the other MAXIO MAP1602 issue mentioned 
in: 
https://lore.kernel.org/lkml/7cd693dd-a6d7-4aab-aef0-76a8366ceee6@archlinux.org/ 
and Lexar NM790 is indeed also using the same controller.

And as I have mentioned there, 6.1 LTS kernels work without a problem 
because there are some differences at calculating the resulting timeout 
value. Latest kernels including the 6.5.x branch makes the ending result 
zero and breaks all 4 TiB SSDs with this controller as far as I know.

-- 
Regards,
Felix Yan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Fwd: Lexar NM790 SSDs are not recognized anymore after 6.1.50 LTS

[Cláudio Sampaio]

Hi, Keith... Just to give you guys a response concerning this, I'm
sorry for the late reply -- too much work. But yes, you are correct,
due to having tried patching the kernel in different days and too much
stuff going on at the same time, I applied this two-line patch to the
same source where I have applied the other patch that multiplies the
timeout by 2 and occurs at an earlier time on activation. I thought I
had an unpatched kernel at the time and ended up compiling it this
way. Sorry for the mistake, but I also saw that now there's a better
patch for the issue.

On Tue, Sep 5, 2023 at 4:35 PM Keith Busch <kbusch@kernel.org> wrote:
>
> On Tue, Sep 05, 2023 at 01:37:36PM +0200, Linux regression tracking (Thorsten Leemhuis) wrote:
> > On 04.09.23 13:07, Bagas Sanjaya wrote:
> > >
> > > I notice a regression report on Bugzilla [1]. Quoting from it:
> > >
> > >> I bought a new 4 TB Lexar NM790 and I was using kernel 6.3.13 at the time. It wasn't recognized, with these messages in dmesg:
> > >>
> > >> [ 358.950147] nvme nvme0: pci function 0000:06:00.0
> > >> [ 358.958327] nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
> > >>
> > >> My other NVMe appears correctly in the nvme list though.
> > >>
> > >>
> > >> So I tried using other kernels I had installed at the time: 6.3.7, 6.4.10, 6.5.0rc6, 6.5.0, 6.5.1 and none of these recognized the disk.
> > >> I installed the 6.1.50 lts kernel from arch repositories (I can compile my own too if this would be an issue) and then the device was correctly recognized:
> > >>
> > >> [    4.654613] nvme 0000:06:00.0: platform quirk: setting simple suspend
> > >> [    4.654632] nvme nvme0: pci function 0000:06:00.0
> > >> [    4.667290] nvme nvme0: allocated 40 MiB host memory buffer.
> > >> [    4.709473] nvme nvme0: 16/0/0 default/read/poll queues
> >
> > FWIW, the quoted mail missed one crucial detail:
> > """
> >  Claudio Sampaio 2023-09-02 19:04:29 UTC
> >
> > Adding the two lines
> >
> > │ 3457   { PCI_DEVICE(0x1d97, 0x1602), /* Lexar NM790 */
> > │ 3458   │ .driver_data = NVME_QUIRK_BOGUS_NID, },
> >
> > in file drivers/nvme/host/pci.c made my NVMe work correctly. Compiled a
> > new 6.5.1 kernel and everything works.
> > """
> >
> > @NVME maintainers: is there anything more you need from Claudio at this
> > point?
>
> Yes: it doesn't really make any sense. The report says the device
> stopped showing up with message:
>
>   nvme nvme0: Device not ready; aborting initialisation, CSTS=0x0
>
> That (a) happens long before the mentioned quirk is considered by the
> driver, and (b) the "quirk" behavior is now the default in 6.5 and
> several of the listed stable kernels anyway.
>
> It more likely sounds like the device is flaky and either never becomes
> ready due to some unspecified internal firmware condition, or
> inaccurately reports how long it actually needs to become ready in
> worst-case-scenario.



-- 
Cláudio "Patola" Sampaio
MakerLinux Labs - Campinas, SP
Resume Gmail - YOUTUBE!
Facebook - Facebook da MakerLinux - Lattes