mainline boot fail nvme/block?

mainline boot fail nvme/block?

[Genes Lists]

[-- Attachment #1: Type: text/plain, Size: 982 bytes --]

Mainline fails to boot - 6.17.1 works fine.
Same kernel on an older laptop without any nvme works just fine.

It seems to get stuck enumerating disks within the initramfs created by
dracut.

Subject to bisect verification, it may have started after commit

  e1b1d03ceec343362524318c076b110066ffe305

  Merge tag 'for-6.18/block-20250929' of 
  git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux

Machine is dell xps 9320 laptop (firmware 2.23.0) with nvme
partitioned:

    # lsblk -f
    NAME        FSTYPE      FSVER LABEL FSAVAIL FSUSE% MOUNTPOINTS    
    sr0
    nvme0n1
    ├─nvme0n1p1 vfat        FAT32 ESP   2.6G    12% /boot
    ├─nvme0n1p2 ext4        1.0   root  77.7G    42% / 
    └─nvme0n1p3 crypto_LUKS 2                          
      └─home    btrfs             home  1.3T    26% /opt
                                                    /home             



Will try do bisect over the weekend.


-- 
Gene

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: mainline boot fail nvme/block?

[Jens Axboe]

On 10/10/25 8:29 AM, Genes Lists wrote:
> Mainline fails to boot - 6.17.1 works fine.
> Same kernel on an older laptop without any nvme works just fine.
> 
> It seems to get stuck enumerating disks within the initramfs created by
> dracut.
> 
> Subject to bisect verification, it may have started after commit
> 
>   e1b1d03ceec343362524318c076b110066ffe305
> 
>   Merge tag 'for-6.18/block-20250929' of 
>   git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
> 
> Machine is dell xps 9320 laptop (firmware 2.23.0) with nvme
> partitioned:
> 
>     # lsblk -f
>     NAME        FSTYPE      FSVER LABEL FSAVAIL FSUSE% MOUNTPOINTS    
>     sr0
>     nvme0n1
>     ├─nvme0n1p1 vfat        FAT32 ESP   2.6G    12% /boot
>     ├─nvme0n1p2 ext4        1.0   root  77.7G    42% / 
>     └─nvme0n1p3 crypto_LUKS 2                          
>       └─home    btrfs             home  1.3T    26% /opt
>                                                     /home             
> 
> 
> 
> Will try do bisect over the weekend.

That'd be great, because there's really not much to glean from this bug
report.

-- 
Jens Axboe

Re: mainline boot fail nvme/block? [BISECTED]

[Genes Lists]

[-- Attachment #1: Type: text/plain, Size: 6271 bytes --]

On Fri, 2025-10-10 at 08:54 -0600, Jens Axboe wrote:
> On 10/10/25 8:29 AM, Genes Lists wrote:
> > Mainline fails to boot - 6.17.1 works fine.
> > Same kernel on an older laptop without any nvme works just fine.
> > 
> > It seems to get stuck enumerating disks within the initramfs
> > created by
> > dracut.
> > 
> > ,,,
> > 
> > Machine is dell xps 9320 laptop (firmware 2.23.0) with nvme
> > partitioned:
> > 
> >     # lsblk -f
> >     NAME        FSTYPE      FSVER LABEL FSAVAIL FSUSE%
> > MOUNTPOINTS    
> >     sr0
> >     nvme0n1
> >     ├─nvme0n1p1 vfat        FAT32 ESP   2.6G    12% /boot
> >     ├─nvme0n1p2 ext4        1.0   root  77.7G    42% / 
> >     └─nvme0n1p3 crypto_LUKS 2                          
> >       └─home    btrfs             home  1.3T    26% /opt
> >                                                    
> > /home             
> > 
> > 
> > 
> > Will try do bisect over the weekend.
> 
> That'd be great, because there's really not much to glean from this
> bug
> report.

Bisect landed here. (cc linux-pci@vger.kernel.org)
Hopefully it is helpful, even though I don't see MSI in lspci output
(which is provided below).

gene


54f45a30c0d0153d2be091ba2d683ab6db6d1d5b is the first bad commit
commit 54f45a30c0d0153d2be091ba2d683ab6db6d1d5b (HEAD)
Author: Inochi Amaoto <inochiama@gmail.com>
Date:   Thu Aug 14 07:28:32 2025 +0800

    PCI/MSI: Add startup/shutdown for per device domains

    As the RISC-V PLIC cannot apply affinity settings without invoking
    irq_enable(), it will make the interrupt unavailble when used as an
    underlying interrupt chip for the MSI controller.

    Implement the irq_startup() and irq_shutdown() callbacks for the
PCI MSI
    and MSI-X templates.

    For chips that specify MSI_FLAG_PCI_MSI_STARTUP_PARENT, the parent
startup
    and shutdown functions are invoked. That allows the interrupt on
the parent
    chip to be enabled if the interrupt has not been enabled during
    allocation. This is necessary for MSI controllers which use PLIC as
    underlying parent interrupt chip.

    Suggested-by: Thomas Gleixner <tglx@linutronix.de>
    Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
    Tested-by: Chen Wang <unicorn_wang@outlook.com> # Pioneerbox
    Reviewed-by: Chen Wang <unicorn_wang@outlook.com>
    Acked-by: Bjorn Helgaas <bhelgaas@google.com>
    Link: https://lore.kernel.org/all/20250813232835.43458-3-
inochiama@gmail.com

 drivers/pci/msi/irqdomain.c | 52
++++++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/msi.h         |  2 ++
 2 files changed, 54 insertions(+)


----------------------------------------- lspci output ----------------
In case helpful here's lspci output:

0000:00:00.0 Host bridge: Intel Corporation Raptor Lake-P/U 4p+8e cores
Host Bridge/DRAM Controller
0000:00:02.0 VGA compatible controller: Intel Corporation Raptor Lake-P
[Iris Xe Graphics] (rev 04)
0000:00:04.0 Signal processing controller: Intel Corporation Raptor
Lake Dynamic Platform and Thermal Framework Processor Participant
0000:00:05.0 Multimedia controller: Intel Corporation Raptor Lake IPU
0000:00:06.0 System peripheral: Intel Corporation RST VMD Managed
Controller
0000:00:07.0 PCI bridge: Intel Corporation Raptor Lake-P Thunderbolt 4
PCI Express Root Port #0
0000:00:07.2 PCI bridge: Intel Corporation Raptor Lake-P Thunderbolt 4
PCI Express Root Port #2
0000:00:08.0 System peripheral: Intel Corporation GNA Scoring
Accelerator module
0000:00:0a.0 Signal processing controller: Intel Corporation Raptor
Lake Crashlog and Telemetry (rev 01)
0000:00:0d.0 USB controller: Intel Corporation Raptor Lake-P
Thunderbolt 4 USB Controller
0000:00:0d.2 USB controller: Intel Corporation Raptor Lake-P
Thunderbolt 4 NHI #0
0000:00:0d.3 USB controller: Intel Corporation Raptor Lake-P
Thunderbolt 4 NHI #1
0000:00:0e.0 RAID bus controller: Intel Corporation Volume Management
Device NVMe RAID Controller Intel Corporation
0000:00:12.0 Serial controller: Intel Corporation Alder Lake-P
Integrated Sensor Hub (rev 01)
0000:00:14.0 USB controller: Intel Corporation Alder Lake PCH USB 3.2
xHCI Host Controller (rev 01)
0000:00:14.2 RAM memory: Intel Corporation Alder Lake PCH Shared SRAM
(rev 01)
0000:00:14.3 Network controller: Intel Corporation Raptor Lake PCH CNVi
WiFi (rev 01)
0000:00:15.0 Serial bus controller: Intel Corporation Alder Lake PCH
Serial IO I2C Controller #0 (rev 01)
0000:00:15.1 Serial bus controller: Intel Corporation Alder Lake PCH
Serial IO I2C Controller #1 (rev 01)
0000:00:16.0 Communication controller: Intel Corporation Alder Lake PCH
HECI Controller (rev 01)
0000:00:1e.0 Communication controller: Intel Corporation Alder Lake PCH
UART #0 (rev 01)
0000:00:1e.3 Serial bus controller: Intel Corporation Alder Lake SPI
Controller (rev 01)
0000:00:1f.0 ISA bridge: Intel Corporation Raptor Lake LPC/eSPI
Controller (rev 01)
0000:00:1f.3 Multimedia audio controller: Intel Corporation Raptor
Lake-P/U/H cAVS (rev 01)
0000:00:1f.4 SMBus: Intel Corporation Alder Lake PCH-P SMBus Host
Controller (rev 01)
0000:00:1f.5 Serial bus controller: Intel Corporation Alder Lake-P PCH
SPI Controller (rev 01)
0000:01:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
Ridge 2020] (rev 02)
0000:02:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
Ridge 2020] (rev 02)
0000:02:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
Ridge 2020] (rev 02)
0000:02:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
Ridge 2020] (rev 02)
0000:02:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
Ridge 2020] (rev 02)
0000:02:04.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
Ridge 2020] (rev 02)
10000:e0:06.0 PCI bridge: Intel Corporation Raptor Lake PCIe 4.0
Graphics Port
10000:e1:00.0 Non-Volatile memory controller: SK hynix Platinum
P41/PC801 NVMe Solid State Drive


-- 
Gene

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: mainline boot fail nvme/block? [BISECTED]

[Inochi Amaoto]

On Fri, Oct 10, 2025 at 07:49:34PM -0400, Genes Lists wrote:
> On Fri, 2025-10-10 at 08:54 -0600, Jens Axboe wrote:
> > On 10/10/25 8:29 AM, Genes Lists wrote:
> > > Mainline fails to boot - 6.17.1 works fine.
> > > Same kernel on an older laptop without any nvme works just fine.
> > > 
> > > It seems to get stuck enumerating disks within the initramfs
> > > created by
> > > dracut.
> > > 
> > > ,,,
> > > 
> > > Machine is dell xps 9320 laptop (firmware 2.23.0) with nvme
> > > partitioned:
> > > 
> > >     # lsblk -f
> > >     NAME        FSTYPE      FSVER LABEL FSAVAIL FSUSE%
> > > MOUNTPOINTS    
> > >     sr0
> > >     nvme0n1
> > >     ├─nvme0n1p1 vfat        FAT32 ESP   2.6G    12% /boot
> > >     ├─nvme0n1p2 ext4        1.0   root  77.7G    42% / 
> > >     └─nvme0n1p3 crypto_LUKS 2                          
> > >       └─home    btrfs             home  1.3T    26% /opt
> > >                                                    
> > > /home             
> > > 
> > > 
> > > 
> > > Will try do bisect over the weekend.
> > 
> > That'd be great, because there's really not much to glean from this
> > bug
> > report.
> 
> Bisect landed here. (cc linux-pci@vger.kernel.org)
> Hopefully it is helpful, even though I don't see MSI in lspci output
> (which is provided below).
> 
> gene
> 
> 
> 54f45a30c0d0153d2be091ba2d683ab6db6d1d5b is the first bad commit
> commit 54f45a30c0d0153d2be091ba2d683ab6db6d1d5b (HEAD)
> Author: Inochi Amaoto <inochiama@gmail.com>
> Date:   Thu Aug 14 07:28:32 2025 +0800
> 
>     PCI/MSI: Add startup/shutdown for per device domains
> 
>     As the RISC-V PLIC cannot apply affinity settings without invoking
>     irq_enable(), it will make the interrupt unavailble when used as an
>     underlying interrupt chip for the MSI controller.
> 
>     Implement the irq_startup() and irq_shutdown() callbacks for the
> PCI MSI
>     and MSI-X templates.
> 
>     For chips that specify MSI_FLAG_PCI_MSI_STARTUP_PARENT, the parent
> startup
>     and shutdown functions are invoked. That allows the interrupt on
> the parent
>     chip to be enabled if the interrupt has not been enabled during
>     allocation. This is necessary for MSI controllers which use PLIC as
>     underlying parent interrupt chip.
> 
>     Suggested-by: Thomas Gleixner <tglx@linutronix.de>
>     Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
>     Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
>     Tested-by: Chen Wang <unicorn_wang@outlook.com> # Pioneerbox
>     Reviewed-by: Chen Wang <unicorn_wang@outlook.com>
>     Acked-by: Bjorn Helgaas <bhelgaas@google.com>
>     Link: https://lore.kernel.org/all/20250813232835.43458-3-
> inochiama@gmail.com
> 
>  drivers/pci/msi/irqdomain.c | 52
> ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  include/linux/msi.h         |  2 ++
>  2 files changed, 54 insertions(+)
> 
> 
> ----------------------------------------- lspci output ----------------
> In case helpful here's lspci output:
> 
> 0000:00:00.0 Host bridge: Intel Corporation Raptor Lake-P/U 4p+8e cores
> Host Bridge/DRAM Controller
> 0000:00:02.0 VGA compatible controller: Intel Corporation Raptor Lake-P
> [Iris Xe Graphics] (rev 04)
> 0000:00:04.0 Signal processing controller: Intel Corporation Raptor
> Lake Dynamic Platform and Thermal Framework Processor Participant
> 0000:00:05.0 Multimedia controller: Intel Corporation Raptor Lake IPU
> 0000:00:06.0 System peripheral: Intel Corporation RST VMD Managed
> Controller
> 0000:00:07.0 PCI bridge: Intel Corporation Raptor Lake-P Thunderbolt 4
> PCI Express Root Port #0
> 0000:00:07.2 PCI bridge: Intel Corporation Raptor Lake-P Thunderbolt 4
> PCI Express Root Port #2
> 0000:00:08.0 System peripheral: Intel Corporation GNA Scoring
> Accelerator module
> 0000:00:0a.0 Signal processing controller: Intel Corporation Raptor
> Lake Crashlog and Telemetry (rev 01)
> 0000:00:0d.0 USB controller: Intel Corporation Raptor Lake-P
> Thunderbolt 4 USB Controller
> 0000:00:0d.2 USB controller: Intel Corporation Raptor Lake-P
> Thunderbolt 4 NHI #0
> 0000:00:0d.3 USB controller: Intel Corporation Raptor Lake-P
> Thunderbolt 4 NHI #1
> 0000:00:0e.0 RAID bus controller: Intel Corporation Volume Management
> Device NVMe RAID Controller Intel Corporation
> 0000:00:12.0 Serial controller: Intel Corporation Alder Lake-P
> Integrated Sensor Hub (rev 01)
> 0000:00:14.0 USB controller: Intel Corporation Alder Lake PCH USB 3.2
> xHCI Host Controller (rev 01)
> 0000:00:14.2 RAM memory: Intel Corporation Alder Lake PCH Shared SRAM
> (rev 01)
> 0000:00:14.3 Network controller: Intel Corporation Raptor Lake PCH CNVi
> WiFi (rev 01)
> 0000:00:15.0 Serial bus controller: Intel Corporation Alder Lake PCH
> Serial IO I2C Controller #0 (rev 01)
> 0000:00:15.1 Serial bus controller: Intel Corporation Alder Lake PCH
> Serial IO I2C Controller #1 (rev 01)
> 0000:00:16.0 Communication controller: Intel Corporation Alder Lake PCH
> HECI Controller (rev 01)
> 0000:00:1e.0 Communication controller: Intel Corporation Alder Lake PCH
> UART #0 (rev 01)
> 0000:00:1e.3 Serial bus controller: Intel Corporation Alder Lake SPI
> Controller (rev 01)
> 0000:00:1f.0 ISA bridge: Intel Corporation Raptor Lake LPC/eSPI
> Controller (rev 01)
> 0000:00:1f.3 Multimedia audio controller: Intel Corporation Raptor
> Lake-P/U/H cAVS (rev 01)
> 0000:00:1f.4 SMBus: Intel Corporation Alder Lake PCH-P SMBus Host
> Controller (rev 01)
> 0000:00:1f.5 Serial bus controller: Intel Corporation Alder Lake-P PCH
> SPI Controller (rev 01)
> 0000:01:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
> Ridge 2020] (rev 02)
> 0000:02:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
> Ridge 2020] (rev 02)
> 0000:02:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
> Ridge 2020] (rev 02)
> 0000:02:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
> Ridge 2020] (rev 02)
> 0000:02:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
> Ridge 2020] (rev 02)
> 0000:02:04.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen
> Ridge 2020] (rev 02)
> 10000:e0:06.0 PCI bridge: Intel Corporation Raptor Lake PCIe 4.0
> Graphics Port
> 10000:e1:00.0 Non-Volatile memory controller: SK hynix Platinum
> P41/PC801 NVMe Solid State Drive
> 
> 
> -- 
> Gene

I think this is caused by VMD device, which I have a temporary solution
here [1]. Since I have no idea about how VMD works, I hope if anyone
can help to convert this as an formal fix.

[1] https://lore.kernel.org/all/qs2vydzm6xngul77xuwjli7h757gzfhmb4siiklzogihz5oplw@gsvgn75lib6t/

Regards,
Inochi

Re: mainline boot fail nvme/block? [BISECTED]

[Genes Lists]

[-- Attachment #1: Type: text/plain, Size: 3211 bytes --]

On Mon, 2025-10-13 at 16:46 +0800, Inochi Amaoto wrote:
> On Fri, Oct 10, 2025 at 07:49:34PM -0400, Genes Lists wrote:
> > On Fri, 2025-10-10 at 08:54 -0600, Jens Axboe wrote:
> > > On 10/10/25 8:29 AM, Genes Lists wrote:
> > > > Mainline fails to boot - 6.17.1 works fine.
> > > > Same kernel on an older laptop without any nvme works just
> > > > fine.
> > > > 
> > > > It seems to get stuck enumerating disks within the initramfs
> > > > created by
> > > > dracut.
> > > > 
> > > > ,
...

> > Bisect landed here. (cc linux-pci@vger.kernel.org)
> > Hopefully it is helpful, even though I don't see MSI in lspci
> > output
> > (which is provided below).
> > 
> > gene
> > 
> > 
> > 54f45a30c0d0153d2be091ba2d683ab6db6d1d5b is the first bad commit
> > commit 54f45a30c0d0153d2be091ba2d683ab6db6d1d5b (HEAD)
> > Author: Inochi Amaoto <inochiama@gmail.com>
> > Date:   Thu Aug 14 07:28:32 2025 +0800
> > 
> >     PCI/MSI: Add startup/shutdown for per device domains
> > 
> >     As the RISC-V PLIC cannot apply affinity settings without
> > invoking
> >     irq_enable(), it will make the interrupt unavailble when used
> > as an
> >     underlying interrupt chip for the MSI controller.
> > 
> >     Implement the irq_startup() and irq_shutdown() callbacks for
> > the
> > PCI MSI
> >     and MSI-X templates.
> > 
> >     For chips that specify MSI_FLAG_PCI_MSI_STARTUP_PARENT, the
> > parent
> > startup
> >     and shutdown functions are invoked. That allows the interrupt
> > on
> > the parent
> >     chip to be enabled if the interrupt has not been enabled during
> >     allocation. This is necessary for MSI controllers which use
> > PLIC as
> >     underlying parent interrupt chip.
> > 
> >     Suggested-by: Thomas Gleixner <tglx@linutronix.de>
> >     Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
> >     Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> >     Tested-by: Chen Wang <unicorn_wang@outlook.com> # Pioneerbox
> >     Reviewed-by: Chen Wang <unicorn_wang@outlook.com>
> >     Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> >     Link: https://lore.kernel.org/all/20250813232835.43458-3-
> > inochiama@gmail.com
> > 
> >  drivers/pci/msi/irqdomain.c | 52
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  include/linux/msi.h         |  2 ++
> >  2 files changed, 54 insertions(+)
> > 
> > 
...

> 
> 
> I think this is caused by VMD device, which I have a temporary
> solution
> here [1]. Since I have no idea about how VMD works, I hope if anyone
> can help to convert this as an formal fix.
> 
> [1]
> https://lore.kernel.org/all/qs2vydzm6xngul77xuwjli7h757gzfhmb4siiklzo
> gihz5oplw@gsvgn75lib6t/
> 
> Regards,
> Inochi

Thank you Inochi

I tried this patch over 6.18-rc1.

 It get's further than without the patch but around the time I get
prompted for passphrase for the luks partition
(root is not encrypted) it crashes. 

I have uploaded 2 images I took of the screen when this happens and
uploaded them to here:

    https://0x0.st/KSNz.jpg
    https://0x0.st/KSNi.jpg



-- 
Gene

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: mainline boot fail nvme/block? [BISECTED]

[Inochi Amaoto]

On Mon, Oct 13, 2025 at 07:45:05AM -0400, Genes Lists wrote:
> On Mon, 2025-10-13 at 16:46 +0800, Inochi Amaoto wrote:
> > On Fri, Oct 10, 2025 at 07:49:34PM -0400, Genes Lists wrote:
> > > On Fri, 2025-10-10 at 08:54 -0600, Jens Axboe wrote:
> > > > On 10/10/25 8:29 AM, Genes Lists wrote:
> > > > > Mainline fails to boot - 6.17.1 works fine.
> > > > > Same kernel on an older laptop without any nvme works just
> > > > > fine.
> > > > > 
> > > > > It seems to get stuck enumerating disks within the initramfs
> > > > > created by
> > > > > dracut.
> > > > > 
> > > > > ,
> ...
> 
> > > Bisect landed here. (cc linux-pci@vger.kernel.org)
> > > Hopefully it is helpful, even though I don't see MSI in lspci
> > > output
> > > (which is provided below).
> > > 
> > > gene
> > > 
> > > 
> > > 54f45a30c0d0153d2be091ba2d683ab6db6d1d5b is the first bad commit
> > > commit 54f45a30c0d0153d2be091ba2d683ab6db6d1d5b (HEAD)
> > > Author: Inochi Amaoto <inochiama@gmail.com>
> > > Date:   Thu Aug 14 07:28:32 2025 +0800
> > > 
> > >     PCI/MSI: Add startup/shutdown for per device domains
> > > 
> > >     As the RISC-V PLIC cannot apply affinity settings without
> > > invoking
> > >     irq_enable(), it will make the interrupt unavailble when used
> > > as an
> > >     underlying interrupt chip for the MSI controller.
> > > 
> > >     Implement the irq_startup() and irq_shutdown() callbacks for
> > > the
> > > PCI MSI
> > >     and MSI-X templates.
> > > 
> > >     For chips that specify MSI_FLAG_PCI_MSI_STARTUP_PARENT, the
> > > parent
> > > startup
> > >     and shutdown functions are invoked. That allows the interrupt
> > > on
> > > the parent
> > >     chip to be enabled if the interrupt has not been enabled during
> > >     allocation. This is necessary for MSI controllers which use
> > > PLIC as
> > >     underlying parent interrupt chip.
> > > 
> > >     Suggested-by: Thomas Gleixner <tglx@linutronix.de>
> > >     Signed-off-by: Inochi Amaoto <inochiama@gmail.com>
> > >     Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> > >     Tested-by: Chen Wang <unicorn_wang@outlook.com> # Pioneerbox
> > >     Reviewed-by: Chen Wang <unicorn_wang@outlook.com>
> > >     Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> > >     Link: https://lore.kernel.org/all/20250813232835.43458-3-
> > > inochiama@gmail.com
> > > 
> > >  drivers/pci/msi/irqdomain.c | 52
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >  include/linux/msi.h         |  2 ++
> > >  2 files changed, 54 insertions(+)
> > > 
> > > 
> ...
> 
> > 
> > 
> > I think this is caused by VMD device, which I have a temporary
> > solution
> > here [1]. Since I have no idea about how VMD works, I hope if anyone
> > can help to convert this as an formal fix.
> > 
> > [1]
> > https://lore.kernel.org/all/qs2vydzm6xngul77xuwjli7h757gzfhmb4siiklzo
> > gihz5oplw@gsvgn75lib6t/
> > 
> > Regards,
> > Inochi
> 
> Thank you Inochi
> 
> I tried this patch over 6.18-rc1.
> 
>  It get's further than without the patch but around the time I get
> prompted for passphrase for the luks partition
> (root is not encrypted) it crashes. 
> 
> I have uploaded 2 images I took of the screen when this happens and
> uploaded them to here:
> 
>     https://0x0.st/KSNz.jpg
>     https://0x0.st/KSNi.jpg
> 

This picture is only a WARNING from perf_get_x86_pmu_capability,
and no other information. So I am not sure whether it is caused
by this change. But from the original report I have, it solves
the problem at that time.

By the way, can you test the following change?
https://lore.kernel.org/all/2hyxqqdootjw5yepbimacuuapfsf26c5mmu5w2jsdmamxvsjdq@gnibocldkuz5/

If it is OK, I will send a patch for it.

Regards,
Inochi

Re: mainline boot fail nvme/block? [BISECTED]

[Genes Lists]

[-- Attachment #1: Type: text/plain, Size: 8544 bytes --]

On Tue, 2025-10-14 at 08:54 +0800, Inochi Amaoto wrote:
> On Mon, Oct 13, 2025 at 07:45:05AM -0400, Genes Lists wrote:
> > 
...

> > Thank you Inochi
> > 
> > I tried this patch over 6.18-rc1.
> > 
> >  It get's further than without the patch but around the time I get
> > prompted for passphrase for the luks partition
> > (root is not encrypted) it crashes. 
> > 
> > I have uploaded 2 images I took of the screen when this happens and
> > uploaded them to here:
> > 
> >     https://0x0.st/KSNz.jpg
> >     https://0x0.st/KSNi.jpg
> > 
> 
> This picture is only a WARNING from perf_get_x86_pmu_capability,
> and no other information. So I am not sure whether it is caused
> by this change. But from the original report I have, it solves
> the problem at that time.
> 
> By the way, can you test the following change?
> https://lore.kernel.org/all/2hyxqqdootjw5yepbimacuuapfsf26c5mmu5w2jsd
> mamxvsjdq@gnibocldkuz5/
> 
> If it is OK, I will send a patch for it.
> 
> Regards,
> Inochi


With this patch it boots with the same/similar warning as before, which
I will include below since it's text instead of image.

Tested-by: Gene C <gene@sapience.com>

Thank you

gene

Warning from 6.18-rc1 with above patch:


[  +0.003929] ------------[ cut here ]------------
[  +0.000004] WARNING: CPU: 7 PID: 584 at arch/x86/events/core.c:3089
perf_get_x86_pmu_capability+0x11/0xb0
[  +0.000010] Modules linked in: snd_hda_codec sr_mod(+) iwlmvm(+)
kvm_intel(+) dm_crypt cdrom encrypted_keys snd_>
[  +0.000060]  industrialio mei_me processor_thermal_wt_req i2c_smbus
spi_intel_pci intel_ipu6 soundcore processor>
[  +0.000058]  ghash_clmulni_intel aesni_intel video intel_ish_ipc
drm_display_helper intel_lpss_pci thunderbolt i>
[  +0.000025] CPU: 7 UID: 0 PID: 584 Comm: (udev-worker) Not tainted
6.18.0-rc1-test-1-00002-ge9cc50c96bb9 #2 PREE>
[  +0.000005] Hardware name: Dell Inc. XPS 9320/0CR6NC, BIOS 2.23.0
07/03/2025
[  +0.000002] RIP: 0010:perf_get_x86_pmu_capability+0x11/0xb0
[  +0.000004] Code: eb 9c e8 22 38 f8 00 66 90 90 90 90 90 90 90 90 90
90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 4>
[  +0.000002] RSP: 0018:ffffd1e041edba58 EFLAGS: 00010202
[  +0.000003] RAX: 0000000000000000 RBX: ffffffffc206f000 RCX:
00000000c0000080
[  +0.000003] RDX: ffffffffc1e396e0 RSI: ffffffffc1e39408 RDI:
ffffffffc1e396e0
[  +0.000001] RBP: 0000000000000001 R08: 0000000000000000 R09:
ffffffffb0e763fb
[  +0.000002] R10: ffff8d0fdad72460 R11: ffff8d0fc0042600 R12:
0000000000000000
[  +0.000001] R13: ffffffffc17e4ca0 R14: 000071be141fd2f2 R15:
0000000000000000
[  +0.000002] FS:  000071be140c6880(0000) GS:ffff8d177bf79000(0000)
knlGS:0000000000000000
[  +0.000002] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.000002] CR2: 00007d5cd598b808 CR3: 0000000109583004 CR4:
0000000000f70ef0
[  +0.000002] PKRU: 55555554
[  +0.000001] Call Trace:
[  +0.000003]  <TASK>
[  +0.000002]  kvm_init_pmu_capability+0x27/0x130 [kvm
83ffe9a0591f43a0ce126662332dfe4cf2561fa4]
[  +0.000119]  kvm_x86_vendor_init+0x1de/0x19d0 [kvm
83ffe9a0591f43a0ce126662332dfe4cf2561fa4]
[  +0.000085]  ? __pfx_vt_init+0x10/0x10 [kvm_intel
5fa84b05f575edf3826c8f8519ca550622307061]
[  +0.000023]  vmx_init+0xf6/0x170 [kvm_intel
5fa84b05f575edf3826c8f8519ca550622307061]
[  +0.000015]  vt_init+0xf/0x360 [kvm_intel
5fa84b05f575edf3826c8f8519ca550622307061]
[  +0.000012]  do_one_initcall+0x5b/0x300
[  +0.000009]  do_init_module+0x62/0x250
[  +0.000005]  ? init_module_from_file+0x8a/0xe0
[  +0.000004]  init_module_from_file+0x8a/0xe0
[  +0.000006]  idempotent_init_module+0x114/0x310
[  +0.000005]  __x64_sys_finit_module+0x6d/0xd0
[  +0.000004]  ? syscall_trace_enter+0x8d/0x1d0
[  +0.000003]  do_syscall_64+0x81/0x7f0
[  +0.000005]  ? __wait_for_common+0x162/0x190
[  +0.000005]  ? __pfx_schedule_timeout+0x10/0x10
[  +0.000004]  ? __rseq_handle_notify_resume+0xa6/0x490
[  +0.000005]  ? idempotent_init_module+0x1df/0x310
[  +0.000005]  ? switch_fpu_return+0x4e/0xd0
[  +0.000003]  ? do_syscall_64+0x226/0x7f0
[  +0.000003]  ? do_syscall_64+0x226/0x7f0
[  +0.000003]  ? do_user_addr_fault+0x21a/0x690
[  +0.000006]  ? exc_page_fault+0x7e/0x1a0
[  +0.000013] Bluetooth: hci0: Firmware SHA1: 0x937bca4a
[  +0.003913] Bluetooth: hci0: Fseq status: Success (0x00)
[  +0.000011] Bluetooth: hci0: Fseq executed: 00.00.02.41
[  +0.000004] Bluetooth: hci0: Fseq BT Top: 00.00.02.41
[  +8.297442] Key type trusted registered
[  +0.014388] Key type encrypted registered
[  +0.017993] sr 0:0:0:0: Power-on or device reset occurred
[  +0.005943] sr 0:0:0:0: [sr0] scsi3-mmc drive: 24x/24x writer dvd-ram
cd/rw xa/form2 cdda tray
[  +0.000005] cdrom: Uniform CD-ROM driver Revision: 3.20
[  +0.003929] ------------[ cut here ]------------
[  +0.000004] WARNING: CPU: 7 PID: 584 at arch/x86/events/core.c:3089
perf_get_x86_pmu_capability+0x11/0xb0
[  +0.000010] Modules linked in: snd_hda_codec sr_mod(+) iwlmvm(+)
kvm_intel(+) dm_crypt cdrom encrypted_keys snd_>
[  +0.000060]  industrialio mei_me processor_thermal_wt_req i2c_smbus
spi_intel_pci intel_ipu6 soundcore processor>
[  +0.000058]  ghash_clmulni_intel aesni_intel video intel_ish_ipc
drm_display_helper intel_lpss_pci thunderbolt i>
[  +0.000025] CPU: 7 UID: 0 PID: 584 Comm: (udev-worker) Not tainted
6.18.0-rc1-test-1-00002-ge9cc50c96bb9 #2 PREE>
[  +0.000005] Hardware name: Dell Inc. XPS 9320/0CR6NC, BIOS 2.23.0
07/03/2025
[  +0.000002] RIP: 0010:perf_get_x86_pmu_capability+0x11/0xb0
[  +0.000004] Code: eb 9c e8 22 38 f8 00 66 90 90 90 90 90 90 90 90 90
90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 4>
[  +0.000002] RSP: 0018:ffffd1e041edba58 EFLAGS: 00010202
[  +0.000003] RAX: 0000000000000000 RBX: ffffffffc206f000 RCX:
00000000c0000080
[  +0.000003] RDX: ffffffffc1e396e0 RSI: ffffffffc1e39408 RDI:
ffffffffc1e396e0
[  +0.000001] RBP: 0000000000000001 R08: 0000000000000000 R09:
ffffffffb0e763fb
[  +0.000002] R10: ffff8d0fdad72460 R11: ffff8d0fc0042600 R12:
0000000000000000
[  +0.000001] R13: ffffffffc17e4ca0 R14: 000071be141fd2f2 R15:
0000000000000000
[  +0.000002] FS:  000071be140c6880(0000) GS:ffff8d177bf79000(0000)
knlGS:0000000000000000
[  +0.000002] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  +0.000002] CR2: 00007d5cd598b808 CR3: 0000000109583004 CR4:
0000000000f70ef0
[  +0.000002] PKRU: 55555554
[  +0.000001] Call Trace:
[  +0.000003]  <TASK>
[  +0.000002]  kvm_init_pmu_capability+0x27/0x130 [kvm
83ffe9a0591f43a0ce126662332dfe4cf2561fa4]
[  +0.000119]  kvm_x86_vendor_init+0x1de/0x19d0 [kvm
83ffe9a0591f43a0ce126662332dfe4cf2561fa4]
[  +0.000085]  ? __pfx_vt_init+0x10/0x10 [kvm_intel
5fa84b05f575edf3826c8f8519ca550622307061]
[  +0.000023]  vmx_init+0xf6/0x170 [kvm_intel
5fa84b05f575edf3826c8f8519ca550622307061]
[  +0.000015]  vt_init+0xf/0x360 [kvm_intel
5fa84b05f575edf3826c8f8519ca550622307061]
[  +0.000012]  do_one_initcall+0x5b/0x300
[  +0.000009]  do_init_module+0x62/0x250
[  +0.000005]  ? init_module_from_file+0x8a/0xe0
[  +0.000004]  init_module_from_file+0x8a/0xe0
[  +0.000006]  idempotent_init_module+0x114/0x310
[  +0.000005]  __x64_sys_finit_module+0x6d/0xd0
[  +0.000004]  ? syscall_trace_enter+0x8d/0x1d0
[  +0.000003]  do_syscall_64+0x81/0x7f0
[  +0.000005]  ? __wait_for_common+0x162/0x190
[  +0.000005]  ? __pfx_schedule_timeout+0x10/0x10
[  +0.000004]  ? __rseq_handle_notify_resume+0xa6/0x490
[  +0.000005]  ? idempotent_init_module+0x1df/0x310
[  +0.000005]  ? switch_fpu_return+0x4e/0xd0
[  +0.000003]  ? do_syscall_64+0x226/0x7f0
[  +0.000003]  ? do_syscall_64+0x226/0x7f0
[  +0.000003]  ? do_user_addr_fault+0x21a/0x690
[  +0.000006]  ? exc_page_fault+0x7e/0x1a0
[  +0.000004]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  +0.000003] RIP: 0033:0x71be1391876d
[  +0.000045] Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa
48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c>
[  +0.000002] RSP: 002b:00007ffdfcc7da58 EFLAGS: 00000246 ORIG_RAX:
0000000000000139
[  +0.000004] RAX: ffffffffffffffda RBX: 000064e3aeb57f80 RCX:
000071be1391876d
[  +0.000001] RDX: 0000000000000004 RSI: 000071be141fd2f2 RDI:
0000000000000032
[  +0.000002] RBP: 00007ffdfcc7daf0 R08: 0000000000000000 R09:
000064e3aeb528f0
[  +0.000001] R10: 0000000000000000 R11: 0000000000000246 R12:
000071be141fd2f2
[  +0.000001] R13: 0000000000020000 R14: 000064e3aeb507f0 R15:
000064e3aeb57f80
[  +0.000003]  </TASK>
[  +0.000001] ---[ end trace 0000000000000000 ]---



-- 
Gene

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: mainline boot fail nvme/block? [BISECTED]

[Bjorn Helgaas]

On Fri, Oct 10, 2025 at 07:49:34PM -0400, Genes Lists wrote:
> On Fri, 2025-10-10 at 08:54 -0600, Jens Axboe wrote:
> > On 10/10/25 8:29 AM, Genes Lists wrote:
> > > Mainline fails to boot - 6.17.1 works fine.
> > > Same kernel on an older laptop without any nvme works just fine.
> ...

> Bisect landed here. (cc linux-pci@vger.kernel.org)
> 
> 54f45a30c0d0153d2be091ba2d683ab6db6d1d5b is the first bad commit
> commit 54f45a30c0d0153d2be091ba2d683ab6db6d1d5b (HEAD)
> Author: Inochi Amaoto <inochiama@gmail.com>
> Date:   Thu Aug 14 07:28:32 2025 +0800
> 
>     PCI/MSI: Add startup/shutdown for per device domains

#regzbot report: https://lore.kernel.org/r/4b392af8847cc19720ffcd53865f60ab3edc56b3.camel@sapience.com
#regzbot introduced: 54f45a30c0d0
#regzbot fix: e433110eb5bf

54f45a30c0d0 ("PCI/MSI: Add startup/shutdown for per device domains")
e433110eb5bf ("PCI: vmd: Override irq_startup()/irq_shutdown() in vmd_init_dev_msi_info()")