Re: nvme boot problems after merge of mm-stable tree into linux-next

Re: nvme boot problems after merge of mm-stable tree into linux-next

[Andrew Morton]

(cc linux-mm)

On Sun, 12 Feb 2023 08:27:32 -0800 Guenter Roeck <linux@roeck-us.net> wrote:

> Hi,
> 
> I am having problems booting from nvme in linux-next. Bisect points to 06e5887cdcc11
> ("Merge branch 'mm-stable' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm").
> The problem was first seen on February 8.
> 
> Some of the log messages as well as the bisect log are attached. Note that the problem
> is not seen in all architectures. The list below may not be complete since several
> architectures/platforms fail to boot for other reasons.
> 
> Guenter
> 
> ---
> 
> # bad: [6ba8a227fd19d19779005fb66ad7562608e1df83] Add linux-next specific files for 20230210
> # good: [4ec5183ec48656cec489c49f989c508b68b518e3] Linux 6.2-rc7
> git bisect start 'HEAD' 'v6.2-rc7'
> # good: [94613f0efc69ed41f9229ef5c294db3ec37145da] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git
> git bisect good 94613f0efc69ed41f9229ef5c294db3ec37145da
> # good: [19e62c715fe70dae4582c2874ed3e66715d09af6] Merge branch 'rcu/next' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
> git bisect good 19e62c715fe70dae4582c2874ed3e66715d09af6
> # good: [5d8b7ecef7f4a681b6e5538db59ff26c389c0ab6] Merge branch 'for-next' of https://gitlab.com/peda-linux/mux.git
> git bisect good 5d8b7ecef7f4a681b6e5538db59ff26c389c0ab6
> # good: [c349bf6ec83903b20fe570c5609b9a864a64e09c] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/krisman/unicode.git
> git bisect good c349bf6ec83903b20fe570c5609b9a864a64e09c
> # good: [5a06a9f17454df38f35672be522ff5eb9b4277d2] selftest: add testing unsharing and counting ksm zero page
> git bisect good 5a06a9f17454df38f35672be522ff5eb9b4277d2
> # good: [f5d115a7b06e5661ed5218ffa9a2644c4ff1c135] Merge branch 'mm-nonmm-unstable' into mm-everything
> git bisect good f5d115a7b06e5661ed5218ffa9a2644c4ff1c135
> # good: [9a863a6a51394bff480c959b713874c090a8f5c6] mm/gup: make locked never NULL in the internal GUP functions
> git bisect good 9a863a6a51394bff480c959b713874c090a8f5c6
> # good: [80fe99787924df7e6048204be7403a2d0b48918c] Merge branch 'for-next/kspp' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git
> git bisect good 80fe99787924df7e6048204be7403a2d0b48918c
> # good: [bed9e516f1183faa0e484479701cc669efd9049a] Merge branch 'vfio-no-iommu' into iommufd.git for-next
> git bisect good bed9e516f1183faa0e484479701cc669efd9049a
> # bad: [06e5887cdcc111d64caa632c39e00a81340f0262] Merge branch 'mm-stable' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
> git bisect bad 06e5887cdcc111d64caa632c39e00a81340f0262
> # good: [223ec6ab265ead0b319bc2f15d0d1be05078a74b] mm/memremap.c: fix outdated comment in devm_memremap_pages
> git bisect good 223ec6ab265ead0b319bc2f15d0d1be05078a74b
> # good: [b942a520d9e43bc31f0808d2f2267a1ddba75518] bcache: Replace zero-length arrays with DECLARE_FLEX_ARRAY() helper
> git bisect good b942a520d9e43bc31f0808d2f2267a1ddba75518
> # good: [3592abd2f44dad9d7232b09e85e8dd1b7fdf91e2] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git
> git bisect good 3592abd2f44dad9d7232b09e85e8dd1b7fdf91e2
> # first bad commit: [06e5887cdcc111d64caa632c39e00a81340f0262] Merge branch 'mm-stable' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
> 
> ---
> alpha:
> 
> Unable to handle kernel paging request at virtual address 0000000000000008
> swapper(0): Oops 1
> pc = [<fffffc0000439100>]  ra = [<fffffc00004390e0>]  ps = 0007    Tainted: G                 N
> pc is at dma_pool_free+0x60/0xf0
> ra is at dma_pool_free+0x40/0xf0
> v0 = 0000000000000007  t0 = 0000000000000000  t1 = fffffc00012037c0
> t2 = 0000000000000001  t3 = fffffc0002ac1100  t4 = fffffc00000400a0
> t5 = fffffc00000400a0  t6 = 0000000000000000  t7 = fffffc00011b0000
> s0 = fffffc0001e6ae40  s1 = 0000000000000000  s2 = 0000000000000007
> s3 = 0000000000000000  s4 = fffffc0001e6ae50  s5 = 0000000000000005
> s6 = fffffc0001eb7932
> a0 = fffffc0001e6ae50  a1 = 0000000000000000  a2 = 0000000000000000
> a3 = 0000000000000002  a4 = 0000000000000000  a5 = 000000000000005a
> t8 = 0000000000000000  t9 = fffffc0001b67248  t10= 0000000000000000
> t11= 0000000000000140  pv = fffffc000037b400  at = 0000000000000001
> gp = fffffc000123cf78  sp = fffffc00011b3b80
> Disabling lock debugging due to kernel taint
> Trace:
> [<fffffc0000a0e1dc>] nvme_unmap_data+0xac/0x220
> [<fffffc0000a0e350>] nvme_pci_complete_batch+0x0/0xe0
> [<fffffc0000a0fda8>] nvme_poll_cq+0x4b8/0x510
> [<fffffc0000a0e350>] nvme_pci_complete_batch+0x0/0xe0
> [<fffffc0000a119f4>] nvme_irq+0x34/0xd0
> [<fffffc00003b076c>] tick_handle_periodic+0x8c/0xc0
> [<fffffc00003857a0>] __handle_irq_event_percpu+0x80/0x1c0
> [<fffffc00003859e8>] handle_irq_event+0x68/0x110
> [<fffffc000038af24>] handle_level_irq+0xe4/0x220
> [<fffffc0000316020>] handle_irq+0x90/0x100
> [<fffffc0000325204>] clipper_srm_device_interrupt+0x24/0x40
> [<fffffc00003161c0>] do_entInt+0x130/0x1e0
> [<fffffc0000311510>] ret_from_sys_call+0x0/0x10
> [<fffffc00003736b8>] do_idle+0x88/0x1e0
> [<fffffc0000df7460>] cpu_idle_poll.isra.0+0x0/0x70
> [<fffffc0000df74a0>] cpu_idle_poll.isra.0+0x40/0x70
> [<fffffc0000373bd8>] cpu_startup_entry+0x28/0x40
> [<fffffc0000df7640>] rest_init+0x100/0x104
> [<fffffc000031001c>] _stext+0x1c/0x20
> [<fffffc0000310000>] _stext+0x0/0x20
> 
> ---
> mips (all variants):
> 
> CPU 0 Unable to handle kernel paging request at virtual address 00000004, epc == 802d9eb8, ra == 802d9ea4
> Oops[#1]:
> CPU: 0 PID: 0 Comm: swapper/0 Tainted: G                 N 6.2.0-rc7-next-20230210 #1
> $ 0   : 00000000 00000001 00000000 00000000
> $ 4   : 8260e888 00000000 810ee780 00000001
> $ 8   : 00000000 813dba24 05345ded 0000020e
> $12   : ffffffff 00000001 8198835c 00000000
> $16   : 8260e880 00000000 00000000 01244800
> $20   : 8260e888 820afec8 808b6c24 a2668290
> $24   : 00000002 810f0000
> $28   : 810c4000 820afe18 81030000 802d9ea4
> Hi    : 000656d0
> Lo    : 4b877998
> epc   : 802d9eb8 dma_pool_free+0x48/0xa0
> ra    : 802d9ea4 dma_pool_free+0x34/0xa0
> Status: 1000a402	KERNEL EXL
> Cause : 0080000c (ExcCode 03)
> BadVA : 00000004
> PrId  : 00019300 (MIPS 24Kc)
> Modules linked in:
> Process swapper/0 (pid: 0, threadinfo=d5273c62, task=cb66b3ba, tls=00000000)
> Stack : ffffffff fffffffe ffffffff ffffffff 821e0dc0 01244800 820b1000 811b61c0
>          810eaa0c 808b6af0 811b61c0 8275c000 820afec8 808b6c24 00000000 801c92f8
>          82604700 821e0dc0 00000001 811b61c0 810eaa0c 808b7b34 810d3b3c 8123447c
>          00000002 8123447c 00000000 80d6660c 81030000 810d3b2c 810f0000 00000000
>          810d3ac0 811b01e0 811b0200 0000000b 00000001 82650c80 00000000 808b86c0
>          ...
> Call Trace:
> [<802d9eb8>] dma_pool_free+0x48/0xa0
> [<808b6af0>] nvme_unmap_data+0x84/0x1b8
> [<808b7b34>] nvme_poll_cq+0x444/0x558
> [<808b86c0>] nvme_irq+0x34/0x84
> [<801b8020>] __handle_irq_event_percpu+0x280/0x434
> [<801b8284>] handle_irq_event+0x50/0xbc
> [<801bdf18>] handle_level_irq+0xf8/0x218
> [<801b6eec>] generic_handle_domain_irq+0x2c/0x44
> [<801b6eec>] generic_handle_domain_irq+0x2c/0x44
> [<80d734d8>] do_domain_IRQ+0x20/0x2c
> [<80795b88>] plat_irq_dispatch+0x64/0xcc
> [<801039a0>] handle_int+0x160/0x16c
> [<80d679b0>] r4k_wait+0xc/0x2c
> [<80d67e38>] default_idle_call+0x98/0x384
> [<80189670>] do_idle+0xfc/0x1cc
> [<80189b60>] cpu_startup_entry+0x2c/0x34
> [<80d68310>] rest_init+0x118/0x1c0
> [<811bc788>] arch_post_acpi_subsys_init+0x0/0x8
> 
> 
> ---
> sparc64:
> 
>      5.070356] Unable to handle kernel NULL pointer dereference
> [    5.071034] tsk->{mm,active_mm}->context = 0000000000000001
> [    5.071519] tsk->{mm,active_mm}->pgd = fffff800048b0000
> [    5.071812]               \|/ ____ \|/
> [    5.071812]               "@'/ .. \`@"
> [    5.071812]               /_| \__/ |_\
> [    5.071812]                  \__U_/
> [    5.072549] init(1): Oops [#1]
> [    5.073008] CPU: 0 PID: 1 Comm: init Tainted: G                 N 6.2.0-rc7-next-20230210 #1
> [    5.073599] TSTATE: 0000004480e01605 TPC: 000000000061bdb0 TNPC: 000000000061bdb4 Y: 00000000    Tainted: G                 N
> [    5.074154] TPC: <dma_pool_free+0x30/0xa0>
> [    5.074778] g0: 0265a89d01ced61d g1: 0000000000000000 g2: 0000000000000000 g3: 000000000000000a
> [    5.075311] g4: fffff800041669a0 g5: fffff8001dca0000 g6: fffff80004168000 g7: 0000000080b583eb
> [    5.075821] o0: 000000000000000e o1: 000000000000000e o2: 0000000000000006 o3: 0000000000000002
> [    5.076265] o4: 0000000000000000 o5: 000000000000000e sp: fffff8001fdff191 ret_pc: 000000000061bd90
> [    5.076716] RPC: <dma_pool_free+0x10/0xa0>
> [    5.076978] l0: 0000000001b61a58 l1: 000000000000000e l2: 000000000208c278 l3: 0000000000000002
> [    5.077421] l4: 81b8e1e5a4e0c637 l5: fffff80004167420 l6: 000000000000401e l7: 0000000000000001
> [    5.077862] i0: fffff800045bae90 i1: 000000000000000e i2: 0000000000000000 i3: 0000000000000002
> [    5.078299] i4: 0000000000000000 i5: fffff800045bae80 i6: fffff8001fdff241 i7: 0000000000b53e54
> [    5.078746] I7: <nvme_unmap_data+0x74/0x140>
> [    5.079031] Call Trace:
> [    5.079384] [<0000000000b53e54>] nvme_unmap_data+0x74/0x140
> [    5.079752] [<0000000000b54338>] nvme_handle_cqe+0x338/0x3e0
> [    5.080053] [<0000000000b5460c>] nvme_irq+0x8c/0x180
> [    5.080316] [<00000000004e7564>] __handle_irq_event_percpu+0x204/0x3a0
> [    5.080647] [<00000000004e7774>] handle_irq_event+0x34/0xc0
> [    5.080934] [<00000000004ec750>] handle_fasteoi_irq+0x90/0x220
> [    5.081231] [<00000000004e6128>] generic_handle_irq+0x28/0x40
> [    5.081526] [<0000000000f4d12c>] handler_irq+0xac/0x100
> [    5.081848] [<00000000004274b0>] sys_call_table+0x760/0x970
> [    5.082191] [<000000000090196c>] __blk_mq_run_hw_queue+0x16c/0x2a0
> [    5.082646] [<000000000090c0dc>] blk_mq_sched_insert_requests+0xdc/0x5c0
> [    5.082993] [<00000000009053c0>] blk_mq_flush_plug_list+0xe0/0x700
> [    5.083311] [<00000000008f6280>] __blk_flush_plug+0xc0/0x100
> [    5.083602] [<00000000008f66c4>] blk_finish_plug+0x24/0x40
> [    5.083886] [<00000000005b48c4>] read_pages+0x164/0x280
> [    5.084159] [<00000000005b4af8>] page_cache_ra_unbounded+0x118/0x1a0

Re: nvme boot problems after merge of mm-stable tree into linux-next

[Bagas Sanjaya]

On 2/14/23 00:47, Andrew Morton wrote:
>> I am having problems booting from nvme in linux-next. Bisect points to 06e5887cdcc11
>> ("Merge branch 'mm-stable' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm").
>> The problem was first seen on February 8.
>>

Hi,

Can you please directly test mm-stable tree instead and bisect from there?

-- 
An old man doll... just what I always wanted! - Clara

Re: nvme boot problems after merge of mm-stable tree into linux-next

[Guenter Roeck]

On 2/13/23 18:23, Bagas Sanjaya wrote:
> On 2/14/23 00:47, Andrew Morton wrote:
>>> I am having problems booting from nvme in linux-next. Bisect points to 06e5887cdcc11
>>> ("Merge branch 'mm-stable' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm").
>>> The problem was first seen on February 8.
>>>
> 
> Hi,
> 
> Can you please directly test mm-stable tree instead and bisect from there?
> 

I'd love to, but as it turns out today's next release (next-20230213)
bisects to a different merge. Still mm, but this time it is the merge
of mm-everything, and both the merge of mm-stable as well as mm/mm-stable
itself are fine. See below for the updated bisect results. mm/mm-everything
itself (mm-hotfixes-stable-2023-02-13-13-50-593-g48de1df158c9) is fine as well.
The merge of mm-nonmm-stable into linux-next (v6.2-rc7-12363-g214c8cfc3507)
is also fine. However, the merge of mm-everything into linux-next) is broken.

Overall, that suggests that there may have been a change somewhere
else in the kernel which is incompatible with a change in one of
the mm branches. It is interesting that the problem in next-20230210
was seen after the merge of mm-stable, and that it is now seen after
the merge of mm-everything.

Guenter

---
# bad: [09e41676e35ab06e4bce8870ea3bf1f191c3cb90] Add linux-next specific files for 20230213
# good: [4ec5183ec48656cec489c49f989c508b68b518e3] Linux 6.2-rc7
git bisect start 'HEAD' 'v6.2-rc7'
# good: [8b065aee8dfbecc978324b204fc897168c9adcd0] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git
git bisect good 8b065aee8dfbecc978324b204fc897168c9adcd0
# good: [72655d7bf4966cc46ac85ef74b26eb74e251ae4a] Merge branch 'rcu/next' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
git bisect good 72655d7bf4966cc46ac85ef74b26eb74e251ae4a
# good: [11b208f696c4111c8bd66863b8306d7857266e4c] Merge branch 'staging-next' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git
git bisect good 11b208f696c4111c8bd66863b8306d7857266e4c
# good: [d69b9b369439219613ca6df66745217f1f56a533] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/rppt/memblock.git
git bisect good d69b9b369439219613ca6df66745217f1f56a533
# good: [37598f5a9d8b63b91cce0cb6bac5f6374ed1bb80] mlock: convert mlock to vma iterator
git bisect good 37598f5a9d8b63b91cce0cb6bac5f6374ed1bb80
# good: [1c2080951a87c3f146c9f9f837ec3c877d4a3c87] Merge branch 'mm-nonmm-unstable' into mm-everything
git bisect good 1c2080951a87c3f146c9f9f837ec3c877d4a3c87
# good: [d1b7f15191d2c203b48eff421f65effce20faa4b] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/krisman/unicode.git
git bisect good d1b7f15191d2c203b48eff421f65effce20faa4b
# good: [7223173e7652230265ba89c5a12eacab1d416797] Merge branch 'bitmap-for-next' of https://github.com/norov/linux.git
git bisect good 7223173e7652230265ba89c5a12eacab1d416797
# good: [70d207cb345c41262a24973e1a40f5015c2c309f] Merge branch 'for-next/kspp' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git
git bisect good 70d207cb345c41262a24973e1a40f5015c2c309f
# good: [84798f2849942bb5e8817417adfdfa6241df2835] iommufd: Add three missing structures in ucmd_buffer
git bisect good 84798f2849942bb5e8817417adfdfa6241df2835
# good: [e5773cfccc0b3c35cef89129f45087d43c136a2d] Merge branch 'for-next/kspp' of git://git.kernel.org/pub/scm/linux/kernel/git/gustavoars/linux.git
git bisect good e5773cfccc0b3c35cef89129f45087d43c136a2d
# good: [77ce632bfb804e4d71bd7aa106cc5b10f164cb14] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git
git bisect good 77ce632bfb804e4d71bd7aa106cc5b10f164cb14
# good: [214c8cfc350729dcce2f2e6d228b996ff568626c] Merge branch 'mm-nonmm-stable' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
git bisect good 214c8cfc350729dcce2f2e6d228b996ff568626c
# bad: [a037e6164093ebb92745768a59e1e2ff8c731d57] Merge branch 'mm-everything' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
git bisect bad a037e6164093ebb92745768a59e1e2ff8c731d57
# first bad commit: [a037e6164093ebb92745768a59e1e2ff8c731d57] Merge branch 'mm-everything' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Re: nvme boot problems after merge of mm-stable tree into linux-next

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

Hi Guenter,

On Mon, 13 Feb 2023 20:01:33 -0800 Guenter Roeck <linux@roeck-us.net> wrote:
>
> Overall, that suggests that there may have been a change somewhere
> else in the kernel which is incompatible with a change in one of
> the mm branches. It is interesting that the problem in next-20230210
> was seen after the merge of mm-stable, and that it is now seen after
> the merge of mm-everything.

That could be because I moved some merge resolution fix patches from
the merge of mm-everything to the merge of mm-stable (they were all for
conflicts involving commits that are now in mm-stable).  So one of my
merge resolutions could be bad.  Or there could be some interaction of
something in mm-everything (that is not in mm-stable or
mm-nonmm-stable) that is interacting with some other change in the rest
of -next.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: nvme boot problems after merge of mm-stable tree into linux-next

[Guenter Roeck]

On 2/13/23 22:20, Stephen Rothwell wrote:
> Hi Guenter,
> 
> On Mon, 13 Feb 2023 20:01:33 -0800 Guenter Roeck <linux@roeck-us.net> wrote:
>>
>> Overall, that suggests that there may have been a change somewhere
>> else in the kernel which is incompatible with a change in one of
>> the mm branches. It is interesting that the problem in next-20230210
>> was seen after the merge of mm-stable, and that it is now seen after
>> the merge of mm-everything.
> 
> That could be because I moved some merge resolution fix patches from
> the merge of mm-everything to the merge of mm-stable (they were all for
> conflicts involving commits that are now in mm-stable).  So one of my
> merge resolutions could be bad.  Or there could be some interaction of
> something in mm-everything (that is not in mm-stable or
> mm-nonmm-stable) that is interacting with some other change in the rest
> of -next.
> 

In next-20230210, the crash happened after the merge of mm-stable.
In next-20230213, the crash happens after the merge of mm-everything.
That means both the merge of mm-stable and mm-nonmm-stable are now fine.
So I would suspect that something in the merge of mm-everything
interacts with the rest of -next.

Guenter

Re: nvme boot problems after merge of mm-stable tree into linux-next

[Bagas Sanjaya]

On 2/14/23 13:49, Guenter Roeck wrote:
> In next-20230210, the crash happened after the merge of mm-stable.
> In next-20230213, the crash happens after the merge of mm-everything.
> That means both the merge of mm-stable and mm-nonmm-stable are now fine.
> So I would suspect that something in the merge of mm-everything
> interacts with the rest of -next.
> 

OK, thanks!

-- 
An old man doll... just what I always wanted! - Clara

Re: nvme boot problems after merge of mm-stable tree into linux-next

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

Hi all,

On Tue, 14 Feb 2023 14:29:29 +0700 Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>
> On 2/14/23 13:49, Guenter Roeck wrote:
> > In next-20230210, the crash happened after the merge of mm-stable.
> > In next-20230213, the crash happens after the merge of mm-everything.
> > That means both the merge of mm-stable and mm-nonmm-stable are now fine.
> > So I would suspect that something in the merge of mm-everything
> > interacts with the rest of -next.
> >   
> 
> OK, thanks!

Did this get resolved?

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: nvme boot problems after merge of mm-stable tree into linux-next

[Guenter Roeck]

On 2/26/23 14:14, Stephen Rothwell wrote:
> Hi all,
> 
> On Tue, 14 Feb 2023 14:29:29 +0700 Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>>
>> On 2/14/23 13:49, Guenter Roeck wrote:
>>> In next-20230210, the crash happened after the merge of mm-stable.
>>> In next-20230213, the crash happens after the merge of mm-everything.
>>> That means both the merge of mm-stable and mm-nonmm-stable are now fine.
>>> So I would suspect that something in the merge of mm-everything
>>> interacts with the rest of -next.
>>>    
>>
>> OK, thanks!
> 
> Did this get resolved?
> 

I think so. Either the offending patch was reverted, or it has been fixed.
I still see various boot problems in -next (and most of them are now
seen in mainline as well), but none of them are nvme related.

Guenter

Re: nvme boot problems after merge of mm-stable tree into linux-next

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

Hi Guenter,

On Sun, 26 Feb 2023 14:53:29 -0800 Guenter Roeck <linux@roeck-us.net> wrote:
>
> On 2/26/23 14:14, Stephen Rothwell wrote:
> > 
> > Did this get resolved?
> 
> I think so. Either the offending patch was reverted, or it has been fixed.
> I still see various boot problems in -next (and most of them are now
> seen in mainline as well), but none of them are nvme related.

Thanks.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]