* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
[not found] ` <DM6PR04MB538673AA2E912100093BBA7CE2242@DM6PR04MB5386.namprd04.prod.outlook.com>
@ 2024-03-11 16:52 ` Rosenow, James
2024-03-11 17:26 ` Philippe Gerum
0 siblings, 1 reply; 10+ messages in thread
From: Rosenow, James @ 2024-03-11 16:52 UTC (permalink / raw)
To: xenomai@lists.linux.dev
We have a COM express carrier with an on-board SSD. The SSD has Ubuntu 22.04 installed on it, on top of that, I've installed the latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel from the linux-dovetail repository . Given that our board supports COM Express, we can easily swap between various processors and have been doing so looking for the best price/performance ratio and of course, low latency.
I have two modules that I am currently testing, an i3-11100HE and an i7-9850HL. Since the SSD is on the base board, I'm running the exact same version of Ubuntu, kernel and root file system regardless of the installed processor board.
The gen9 processor boots and runs the Xenomai kernel without any trouble at all. I'm having SMI issues but that is another subject. When I put the gen11 board on the carrier, it hangs at 'loading initial ramdisk .'. It is noteworthy that if I interrupt grub and select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen 9 boot and run fine. I've verified the Xenomai kernel runs fine on generation 7 and 6 CPU's as well.
I've searched the web for possible reasons why Linux would hang at that point; for Linux in general, it is a common thing and I've tried many, many of the solutions that others have found to work, all to no avail. I do not know how to get additional information on what is hanging as there is really no context from which to proceed.
I'm looking for suggestions on next steps to root cause this hang.
Thanks and regards,
Jim
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
2024-03-11 16:52 ` Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work Rosenow, James
@ 2024-03-11 17:26 ` Philippe Gerum
2024-03-13 16:11 ` Rosenow, James
0 siblings, 1 reply; 10+ messages in thread
From: Philippe Gerum @ 2024-03-11 17:26 UTC (permalink / raw)
To: Rosenow, James; +Cc: xenomai@lists.linux.dev
"Rosenow, James" <jim.rosenow@mts.com> writes:
> We have a COM express carrier with an on-board SSD. The SSD has
> Ubuntu 22.04 installed on it, on top of that, I've installed the
> latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel
> from the linux-dovetail repository . Given that our board supports
> COM Express, we can easily swap between various processors and have
> been doing so looking for the best price/performance ratio and of
> course, low latency.
>
> I have two modules that I am currently testing, an i3-11100HE and an i7-9850HL. Since the SSD is on the base board, I'm running the exact same version of Ubuntu, kernel and root file system regardless of the installed processor board.
>
> The gen9 processor boots and runs the Xenomai kernel without any
> trouble at all. I'm having SMI issues but that is another subject.
> When I put the gen11 board on the carrier, it hangs at 'loading
> initial ramdisk .'. It is noteworthy that if I interrupt grub and
> select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen 9
> boot and run fine. I've verified the Xenomai kernel runs fine on
> generation 7 and 6 CPU's as well.
>
> I've searched the web for possible reasons why Linux would hang at that point; for Linux in general, it is a common thing and I've tried many, many of the solutions that others have found to work, all to no avail. I do not know how to get additional information on what is hanging as there is really no context from which to proceed.
>
> I'm looking for suggestions on next steps to root cause this hang.
>
> Thanks and regards,
> Jim
Could you boot the machine with the stock kernel that works, then paste
the output of /proc/interrupts? This may help finding out which IRQ
controller might be involved in the issue.
Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE only
(i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still causes
the issue. That might help pointing the finger at either a generic IRQ
pipelining issue, or something related to the tick device instead.
--
Philippe.
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
2024-03-11 17:26 ` Philippe Gerum
@ 2024-03-13 16:11 ` Rosenow, James
2024-03-13 18:32 ` Rosenow, James
2024-03-20 21:25 ` Philippe Gerum
0 siblings, 2 replies; 10+ messages in thread
From: Rosenow, James @ 2024-03-13 16:11 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai@lists.linux.dev
> "Rosenow, James" <jim.rosenow@mts.com> writes:
>
> > We have a COM express carrier with an on-board SSD. The SSD has
> > Ubuntu 22.04 installed on it, on top of that, I've installed the
> > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel
> > from the linux-dovetail repository . Given that our board supports
> > COM Express, we can easily swap between various processors and have
> > been doing so looking for the best price/performance ratio and of
> > course, low latency.
> >
> > I have two modules that I am currently testing, an i3-11100HE and an i7-
> 9850HL. Since the SSD is on the base board, I'm running the exact same
> version of Ubuntu, kernel and root file system regardless of the installed
> processor board.
> >
> > The gen9 processor boots and runs the Xenomai kernel without any
> > trouble at all. I'm having SMI issues but that is another subject.
> > When I put the gen11 board on the carrier, it hangs at 'loading
> > initial ramdisk .'. It is noteworthy that if I interrupt grub and
> > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen 9
> > boot and run fine. I've verified the Xenomai kernel runs fine on
> > generation 7 and 6 CPU's as well.
> >
> > I've searched the web for possible reasons why Linux would hang at that
> point; for Linux in general, it is a common thing and I've tried many,
> many of the solutions that others have found to work, all to no avail. I
> do not know how to get additional information on what is hanging as there
> is really no context from which to proceed.
> >
> > I'm looking for suggestions on next steps to root cause this hang.
> >
> > Thanks and regards,
> > Jim
>
> Could you boot the machine with the stock kernel that works, then paste
> the output of /proc/interrupts? This may help finding out which IRQ
> controller might be involved in the issue.
>
Hello Philippe,
Here is the output of /proc/interrupts from the stock 6.5.0 kernel.
$ cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 15 0 0 0 IO-APIC 2-edge timer
3: 0 0 0 2 IO-APIC 3-edge
4: 0 0 0 2 IO-APIC 4-edge
8: 0 0 0 0 IO-APIC 8-edge rtc0
9: 611 154 0 0 IO-APIC 9-fasteoi acpi
14: 0 0 0 0 IO-APIC 14-fasteoi INT34C6:00
16: 699 0 0 0 IO-APIC 16-fasteoi i801_smbus, intel_ish_ipc
18: 0 0 0 0 IO-APIC 18-fasteoi i2c_designware.0
27: 0 0 0 0 IO-APIC 27-fasteoi idma64.1, i2c_designware.1
29: 0 0 0 0 IO-APIC 29-fasteoi idma64.2, i2c_designware.2
30: 0 0 0 0 IO-APIC 30-fasteoi idma64.3, i2c_designware.3
31: 0 0 0 0 IO-APIC 31-fasteoi idma64.4, i2c_designware.4
32: 0 0 0 0 IO-APIC 32-fasteoi idma64.5, i2c_designware.5
126: 0 0 0 0 PCI-MSI-0000:00:0d.0 0-edge xhci_hcd
127: 0 0 0 21855 PCI-MSI-0000:00:17.0 0-edge ahci[0000:00:17.0]
128: 0 0 1 0 PCI-MSIX-0000:04:00.0 0-edge enp4s0
129: 371 0 0 35 PCI-MSIX-0000:04:00.0 1-edge enp4s0-TxRx-0
130: 36 0 0 310 PCI-MSIX-0000:04:00.0 2-edge enp4s0-TxRx-1
131: 0 36 334 0 PCI-MSIX-0000:04:00.0 3-edge enp4s0-TxRx-2
132: 0 0 285 0 PCI-MSIX-0000:04:00.0 4-edge enp4s0-TxRx-3
133: 1681 74 0 0 PCI-MSI-0000:00:14.0 0-edge xhci_hcd
134: 0 0 0 0 accel_3d-dev0 accel_3d_consumer0
136: 0 0 0 0 als-dev1 als_consumer1
142: 0 0 0 0 magn_3d-dev3 magn_3d_consumer3
150: 49 0 0 0 PCI-MSI-0000:00:16.0 0-edge mei_me
157: 5552 319 0 0 PCI-MSI-0000:00:02.0 0-edge i915
NMI: 1 0 3 3 Non-maskable interrupts
LOC: 25207 348 30717 29537 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 1 0 3 3 Performance monitoring interrupts
IWI: 2162 11 119 207 IRQ work interrupts
RTR: 2 0 0 0 APIC ICR read retries
RES: 2902 326 3213 2405 Rescheduling interrupts
CAL: 46790 5794 43018 41150 Function call interrupts
TLB: 273 0 258 149 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 1 2 2 2 Machine check polls
ERR: 2
MIS: 0
PIN: 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 Posted-interrupt wakeup event
> Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE only
> (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still causes
> the issue. That might help pointing the finger at either a generic IRQ
> pipelining issue, or something related to the tick device instead.
>
I will try this today and respond as soon as I get the results.
Regards,
Jim
> --
> Philippe.
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
2024-03-13 16:11 ` Rosenow, James
@ 2024-03-13 18:32 ` Rosenow, James
2024-03-19 15:19 ` Rosenow, James
2024-03-20 21:36 ` Philippe Gerum
2024-03-20 21:25 ` Philippe Gerum
1 sibling, 2 replies; 10+ messages in thread
From: Rosenow, James @ 2024-03-13 18:32 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai@lists.linux.dev
> > "Rosenow, James" <jim.rosenow@mts.com> writes:
> >
> > > We have a COM express carrier with an on-board SSD. The SSD has
> > > Ubuntu 22.04 installed on it, on top of that, I've installed the
> > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel
> > > from the linux-dovetail repository . Given that our board supports
> > > COM Express, we can easily swap between various processors and have
> > > been doing so looking for the best price/performance ratio and of
> > > course, low latency.
> > >
> > > I have two modules that I am currently testing, an i3-11100HE and an
> > > i7-
> > 9850HL. Since the SSD is on the base board, I'm running the exact
> > same version of Ubuntu, kernel and root file system regardless of the
> > installed processor board.
> > >
> > > The gen9 processor boots and runs the Xenomai kernel without any
> > > trouble at all. I'm having SMI issues but that is another subject.
> > > When I put the gen11 board on the carrier, it hangs at 'loading
> > > initial ramdisk .'. It is noteworthy that if I interrupt grub and
> > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen
> > > 9 boot and run fine. I've verified the Xenomai kernel runs fine on
> > > generation 7 and 6 CPU's as well.
> > >
> > > I've searched the web for possible reasons why Linux would hang at
> > > that
> > point; for Linux in general, it is a common thing and I've tried many,
> > many of the solutions that others have found to work, all to no avail.
> > I do not know how to get additional information on what is hanging as
> > there is really no context from which to proceed.
> > >
> > > I'm looking for suggestions on next steps to root cause this hang.
> > >
> > > Thanks and regards,
> > > Jim
> >
> > Could you boot the machine with the stock kernel that works, then
> > paste the output of /proc/interrupts? This may help finding out which
> > IRQ controller might be involved in the issue.
> >
>
> Hello Philippe,
> Here is the output of /proc/interrupts from the stock 6.5.0 kernel.
>
> $ cat /proc/interrupts
> CPU0 CPU1 CPU2 CPU3
> 0: 15 0 0 0 IO-APIC 2-edge
> timer
> 3: 0 0 0 2 IO-APIC 3-edge
> 4: 0 0 0 2 IO-APIC 4-edge
> 8: 0 0 0 0 IO-APIC 8-edge
> rtc0
> 9: 611 154 0 0 IO-APIC 9-fasteoi
> acpi
> 14: 0 0 0 0 IO-APIC 14-fasteoi
> INT34C6:00
> 16: 699 0 0 0 IO-APIC 16-fasteoi
> i801_smbus, intel_ish_ipc
> 18: 0 0 0 0 IO-APIC 18-fasteoi
> i2c_designware.0
> 27: 0 0 0 0 IO-APIC 27-fasteoi
> idma64.1, i2c_designware.1
> 29: 0 0 0 0 IO-APIC 29-fasteoi
> idma64.2, i2c_designware.2
> 30: 0 0 0 0 IO-APIC 30-fasteoi
> idma64.3, i2c_designware.3
> 31: 0 0 0 0 IO-APIC 31-fasteoi
> idma64.4, i2c_designware.4
> 32: 0 0 0 0 IO-APIC 32-fasteoi
> idma64.5, i2c_designware.5
> 126: 0 0 0 0 PCI-MSI-0000:00:0d.0
> 0-edge xhci_hcd
> 127: 0 0 0 21855 PCI-MSI-0000:00:17.0
> 0-edge ahci[0000:00:17.0]
> 128: 0 0 1 0 PCI-MSIX-0000:04:00.0
> 0-edge enp4s0
> 129: 371 0 0 35 PCI-MSIX-0000:04:00.0
> 1-edge enp4s0-TxRx-0
> 130: 36 0 0 310 PCI-MSIX-0000:04:00.0
> 2-edge enp4s0-TxRx-1
> 131: 0 36 334 0 PCI-MSIX-0000:04:00.0
> 3-edge enp4s0-TxRx-2
> 132: 0 0 285 0 PCI-MSIX-0000:04:00.0
> 4-edge enp4s0-TxRx-3
> 133: 1681 74 0 0 PCI-MSI-0000:00:14.0
> 0-edge xhci_hcd
> 134: 0 0 0 0 accel_3d-dev0
> accel_3d_consumer0
> 136: 0 0 0 0 als-dev1
> als_consumer1
> 142: 0 0 0 0 magn_3d-dev3
> magn_3d_consumer3
> 150: 49 0 0 0 PCI-MSI-0000:00:16.0
> 0-edge mei_me
> 157: 5552 319 0 0 PCI-MSI-0000:00:02.0
> 0-edge i915
> NMI: 1 0 3 3 Non-maskable
> interrupts
> LOC: 25207 348 30717 29537 Local timer interrupts
> SPU: 0 0 0 0 Spurious interrupts
> PMI: 1 0 3 3 Performance monitoring
> interrupts
> IWI: 2162 11 119 207 IRQ work interrupts
> RTR: 2 0 0 0 APIC ICR read retries
> RES: 2902 326 3213 2405 Rescheduling
> interrupts
> CAL: 46790 5794 43018 41150 Function call
> interrupts
> TLB: 273 0 258 149 TLB shootdowns
> TRM: 0 0 0 0 Thermal event
> interrupts
> THR: 0 0 0 0 Threshold APIC
> interrupts
> DFR: 0 0 0 0 Deferred Error APIC
> interrupts
> MCE: 0 0 0 0 Machine check
> exceptions
> MCP: 1 2 2 2 Machine check polls
> ERR: 2
> MIS: 0
> PIN: 0 0 0 0 Posted-interrupt
> notification event
> NPI: 0 0 0 0 Nested posted-
> interrupt event
> PIW: 0 0 0 0 Posted-interrupt
> wakeup event
>
> > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE only
> > (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still
> > causes the issue. That might help pointing the finger at either a
> > generic IRQ pipelining issue, or something related to the tick device
> instead.
> >
>
> I will try this today and respond as soon as I get the results.
> Regards,
> Jim
>
I can confirm that the system hangs with "CONFIG_IRQ_PIPELINE=y" and "CONFIG_DOVETAIL is not set".
This is identical to the original failure.
-Jim
> > --
> > Philippe.
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
2024-03-13 18:32 ` Rosenow, James
@ 2024-03-19 15:19 ` Rosenow, James
2024-03-19 15:40 ` Philippe Gerum
2024-03-20 21:36 ` Philippe Gerum
1 sibling, 1 reply; 10+ messages in thread
From: Rosenow, James @ 2024-03-19 15:19 UTC (permalink / raw)
To: Philippe Gerum; +Cc: xenomai@lists.linux.dev
Anyone have any advice on how to root cause this hang.
The requested /proc/interrupts output is below and the system does hang with only CONFIG_IRQ_PIPELINE enabled.
Thanks in advance.
Jim
> > > "Rosenow, James" <jim.rosenow@mts.com> writes:
> > >
> > > > We have a COM express carrier with an on-board SSD. The SSD has
> > > > Ubuntu 22.04 installed on it, on top of that, I've installed the
> > > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase'
> > > > kernel from the linux-dovetail repository . Given that our board
> > > > supports COM Express, we can easily swap between various
> > > > processors and have been doing so looking for the best
> > > > price/performance ratio and of course, low latency.
> > > >
> > > > I have two modules that I am currently testing, an i3-11100HE and
> > > > an
> > > > i7-
> > > 9850HL. Since the SSD is on the base board, I'm running the exact
> > > same version of Ubuntu, kernel and root file system regardless of
> > > the installed processor board.
> > > >
> > > > The gen9 processor boots and runs the Xenomai kernel without any
> > > > trouble at all. I'm having SMI issues but that is another subject.
> > > > When I put the gen11 board on the carrier, it hangs at 'loading
> > > > initial ramdisk .'. It is noteworthy that if I interrupt grub and
> > > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and
> > > > gen
> > > > 9 boot and run fine. I've verified the Xenomai kernel runs fine
> > > > on generation 7 and 6 CPU's as well.
> > > >
> > > > I've searched the web for possible reasons why Linux would hang at
> > > > that
> > > point; for Linux in general, it is a common thing and I've tried
> > > many, many of the solutions that others have found to work, all to no
> avail.
> > > I do not know how to get additional information on what is hanging
> > > as there is really no context from which to proceed.
> > > >
> > > > I'm looking for suggestions on next steps to root cause this hang.
> > > >
> > > > Thanks and regards,
> > > > Jim
> > >
> > > Could you boot the machine with the stock kernel that works, then
> > > paste the output of /proc/interrupts? This may help finding out
> > > which IRQ controller might be involved in the issue.
> > >
> >
> > Hello Philippe,
> > Here is the output of /proc/interrupts from the stock 6.5.0 kernel.
> >
> > $ cat /proc/interrupts
> > CPU0 CPU1 CPU2 CPU3
> > 0: 15 0 0 0 IO-APIC 2-edge
> > timer
> > 3: 0 0 0 2 IO-APIC 3-edge
> > 4: 0 0 0 2 IO-APIC 4-edge
> > 8: 0 0 0 0 IO-APIC 8-edge
> > rtc0
> > 9: 611 154 0 0 IO-APIC 9-fasteoi
> > acpi
> > 14: 0 0 0 0 IO-APIC 14-fasteoi
> > INT34C6:00
> > 16: 699 0 0 0 IO-APIC 16-fasteoi
> > i801_smbus, intel_ish_ipc
> > 18: 0 0 0 0 IO-APIC 18-fasteoi
> > i2c_designware.0
> > 27: 0 0 0 0 IO-APIC 27-fasteoi
> > idma64.1, i2c_designware.1
> > 29: 0 0 0 0 IO-APIC 29-fasteoi
> > idma64.2, i2c_designware.2
> > 30: 0 0 0 0 IO-APIC 30-fasteoi
> > idma64.3, i2c_designware.3
> > 31: 0 0 0 0 IO-APIC 31-fasteoi
> > idma64.4, i2c_designware.4
> > 32: 0 0 0 0 IO-APIC 32-fasteoi
> > idma64.5, i2c_designware.5
> > 126: 0 0 0 0 PCI-MSI-0000:00:0d.0
> > 0-edge xhci_hcd
> > 127: 0 0 0 21855 PCI-MSI-0000:00:17.0
> > 0-edge ahci[0000:00:17.0]
> > 128: 0 0 1 0 PCI-MSIX-0000:04:00.0
> > 0-edge enp4s0
> > 129: 371 0 0 35 PCI-MSIX-0000:04:00.0
> > 1-edge enp4s0-TxRx-0
> > 130: 36 0 0 310 PCI-MSIX-0000:04:00.0
> > 2-edge enp4s0-TxRx-1
> > 131: 0 36 334 0 PCI-MSIX-0000:04:00.0
> > 3-edge enp4s0-TxRx-2
> > 132: 0 0 285 0 PCI-MSIX-0000:04:00.0
> > 4-edge enp4s0-TxRx-3
> > 133: 1681 74 0 0 PCI-MSI-0000:00:14.0
> > 0-edge xhci_hcd
> > 134: 0 0 0 0 accel_3d-dev0
> > accel_3d_consumer0
> > 136: 0 0 0 0 als-dev1
> > als_consumer1
> > 142: 0 0 0 0 magn_3d-dev3
> > magn_3d_consumer3
> > 150: 49 0 0 0 PCI-MSI-0000:00:16.0
> > 0-edge mei_me
> > 157: 5552 319 0 0 PCI-MSI-0000:00:02.0
> > 0-edge i915
> > NMI: 1 0 3 3 Non-maskable
> > interrupts
> > LOC: 25207 348 30717 29537 Local timer
> interrupts
> > SPU: 0 0 0 0 Spurious interrupts
> > PMI: 1 0 3 3 Performance
> monitoring
> > interrupts
> > IWI: 2162 11 119 207 IRQ work interrupts
> > RTR: 2 0 0 0 APIC ICR read
> retries
> > RES: 2902 326 3213 2405 Rescheduling
> > interrupts
> > CAL: 46790 5794 43018 41150 Function call
> > interrupts
> > TLB: 273 0 258 149 TLB shootdowns
> > TRM: 0 0 0 0 Thermal event
> > interrupts
> > THR: 0 0 0 0 Threshold APIC
> > interrupts
> > DFR: 0 0 0 0 Deferred Error APIC
> > interrupts
> > MCE: 0 0 0 0 Machine check
> > exceptions
> > MCP: 1 2 2 2 Machine check polls
> > ERR: 2
> > MIS: 0
> > PIN: 0 0 0 0 Posted-interrupt
> > notification event
> > NPI: 0 0 0 0 Nested posted-
> > interrupt event
> > PIW: 0 0 0 0 Posted-interrupt
> > wakeup event
> >
> > > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE
> > > only (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top)
> > > still causes the issue. That might help pointing the finger at
> > > either a generic IRQ pipelining issue, or something related to the
> > > tick device
> > instead.
> > >
> >
> > I will try this today and respond as soon as I get the results.
> > Regards,
> > Jim
> >
>
> I can confirm that the system hangs with "CONFIG_IRQ_PIPELINE=y" and
> "CONFIG_DOVETAIL is not set".
> This is identical to the original failure.
>
> -Jim
>
> > > --
> > > Philippe.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
2024-03-19 15:19 ` Rosenow, James
@ 2024-03-19 15:40 ` Philippe Gerum
2024-03-20 7:44 ` Chen, Hongzhan
0 siblings, 1 reply; 10+ messages in thread
From: Philippe Gerum @ 2024-03-19 15:40 UTC (permalink / raw)
To: Rosenow, James; +Cc: xenomai@lists.linux.dev
"Rosenow, James" <jim.rosenow@mts.com> writes:
> Anyone have any advice on how to root cause this hang.
>
> The requested /proc/interrupts output is below and the system does hang with only CONFIG_IRQ_PIPELINE enabled.
>
I'll have a look asap this week.
--
Philippe.
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
2024-03-19 15:40 ` Philippe Gerum
@ 2024-03-20 7:44 ` Chen, Hongzhan
0 siblings, 0 replies; 10+ messages in thread
From: Chen, Hongzhan @ 2024-03-20 7:44 UTC (permalink / raw)
To: Philippe Gerum, Rosenow, James; +Cc: xenomai@lists.linux.dev
>-----Original Message-----
>From: Philippe Gerum <rpm@xenomai.org>
>Sent: Tuesday, March 19, 2024 11:41 PM
>To: Rosenow, James <jim.rosenow@mts.com>
>Cc: xenomai@lists.linux.dev
>Subject: Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9,
>7, and 6 work.
>
>
>"Rosenow, James" <jim.rosenow@mts.com> writes:
>
>> Anyone have any advice on how to root cause this hang.
>>
>> The requested /proc/interrupts output is below and the system does hang
>with only CONFIG_IRQ_PIPELINE enabled.
>>
>
>I'll have a look asap this week.
I also reproduced such issue and am looking into it on Gen 11 CPU.
Regards
Hongzhan Chen
>
>--
>Philippe.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
2024-03-13 16:11 ` Rosenow, James
2024-03-13 18:32 ` Rosenow, James
@ 2024-03-20 21:25 ` Philippe Gerum
1 sibling, 0 replies; 10+ messages in thread
From: Philippe Gerum @ 2024-03-20 21:25 UTC (permalink / raw)
To: Rosenow, James; +Cc: xenomai@lists.linux.dev
"Rosenow, James" <jim.rosenow@mts.com> writes:
>> "Rosenow, James" <jim.rosenow@mts.com> writes:
>>
>> > We have a COM express carrier with an on-board SSD. The SSD has
>> > Ubuntu 22.04 installed on it, on top of that, I've installed the
>> > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel
>> > from the linux-dovetail repository . Given that our board supports
>> > COM Express, we can easily swap between various processors and have
>> > been doing so looking for the best price/performance ratio and of
>> > course, low latency.
>> >
>> > I have two modules that I am currently testing, an i3-11100HE and an i7-
>> 9850HL. Since the SSD is on the base board, I'm running the exact same
>> version of Ubuntu, kernel and root file system regardless of the installed
>> processor board.
>> >
>> > The gen9 processor boots and runs the Xenomai kernel without any
>> > trouble at all. I'm having SMI issues but that is another subject.
>> > When I put the gen11 board on the carrier, it hangs at 'loading
>> > initial ramdisk .'. It is noteworthy that if I interrupt grub and
>> > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen 9
>> > boot and run fine. I've verified the Xenomai kernel runs fine on
>> > generation 7 and 6 CPU's as well.
>> >
>> > I've searched the web for possible reasons why Linux would hang at that
>> point; for Linux in general, it is a common thing and I've tried many,
>> many of the solutions that others have found to work, all to no avail. I
>> do not know how to get additional information on what is hanging as there
>> is really no context from which to proceed.
>> >
>> > I'm looking for suggestions on next steps to root cause this hang.
>> >
>> > Thanks and regards,
>> > Jim
>>
>> Could you boot the machine with the stock kernel that works, then paste
>> the output of /proc/interrupts? This may help finding out which IRQ
>> controller might be involved in the issue.
>>
>
> Hello Philippe,
> Here is the output of /proc/interrupts from the stock 6.5.0 kernel.
>
> $ cat /proc/interrupts
> CPU0 CPU1 CPU2 CPU3
> 134: 0 0 0 0 accel_3d-dev0 accel_3d_consumer0
> 136: 0 0 0 0 als-dev1 als_consumer1
> 142: 0 0 0 0 magn_3d-dev3 magn_3d_consumer3
Could you disable these features before retrying?
CONFIG_HID_SENSOR_MAGNETOMETER_3D
CONFIG_HID_SENSOR_ACCEL_3D
CONFIG_HID_SENSOR_ALS? (maybe.. there is a truckload of light sensors).
If the issue persists, disabling all IIO drivers could be a sensible
option to confirm that there may be an issue with some irqchip those
drivers depend on when it comes to dealing with Dovetail.
--
Philippe.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
2024-03-13 18:32 ` Rosenow, James
2024-03-19 15:19 ` Rosenow, James
@ 2024-03-20 21:36 ` Philippe Gerum
2024-03-21 5:56 ` Chen, Hongzhan
1 sibling, 1 reply; 10+ messages in thread
From: Philippe Gerum @ 2024-03-20 21:36 UTC (permalink / raw)
To: Rosenow, James; +Cc: xenomai@lists.linux.dev
"Rosenow, James" <jim.rosenow@mts.com> writes:
>> > "Rosenow, James" <jim.rosenow@mts.com> writes:
>> >
>> > > We have a COM express carrier with an on-board SSD. The SSD has
>> > > Ubuntu 22.04 installed on it, on top of that, I've installed the
>> > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel
>> > > from the linux-dovetail repository . Given that our board supports
>> > > COM Express, we can easily swap between various processors and have
>> > > been doing so looking for the best price/performance ratio and of
>> > > course, low latency.
>> > >
>> > > I have two modules that I am currently testing, an i3-11100HE and an
>> > > i7-
>> > 9850HL. Since the SSD is on the base board, I'm running the exact
>> > same version of Ubuntu, kernel and root file system regardless of the
>> > installed processor board.
>> > >
>> > > The gen9 processor boots and runs the Xenomai kernel without any
>> > > trouble at all. I'm having SMI issues but that is another subject.
>> > > When I put the gen11 board on the carrier, it hangs at 'loading
>> > > initial ramdisk .'. It is noteworthy that if I interrupt grub and
>> > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen
>> > > 9 boot and run fine. I've verified the Xenomai kernel runs fine on
>> > > generation 7 and 6 CPU's as well.
>> > >
>> > > I've searched the web for possible reasons why Linux would hang at
>> > > that
>> > point; for Linux in general, it is a common thing and I've tried many,
>> > many of the solutions that others have found to work, all to no avail.
>> > I do not know how to get additional information on what is hanging as
>> > there is really no context from which to proceed.
>> > >
>> > > I'm looking for suggestions on next steps to root cause this hang.
>> > >
>> > > Thanks and regards,
>> > > Jim
>> >
>> > Could you boot the machine with the stock kernel that works, then
>> > paste the output of /proc/interrupts? This may help finding out which
>> > IRQ controller might be involved in the issue.
>> >
>>
>> Hello Philippe,
>> Here is the output of /proc/interrupts from the stock 6.5.0 kernel.
>>
>> $ cat /proc/interrupts
>> CPU0 CPU1 CPU2 CPU3
>> 0: 15 0 0 0 IO-APIC 2-edge
>> timer
>> 3: 0 0 0 2 IO-APIC 3-edge
>> 4: 0 0 0 2 IO-APIC 4-edge
>> 8: 0 0 0 0 IO-APIC 8-edge
>> rtc0
>> 9: 611 154 0 0 IO-APIC 9-fasteoi
>> acpi
>> 14: 0 0 0 0 IO-APIC 14-fasteoi
>> INT34C6:00
>> 16: 699 0 0 0 IO-APIC 16-fasteoi
>> i801_smbus, intel_ish_ipc
>> 18: 0 0 0 0 IO-APIC 18-fasteoi
>> i2c_designware.0
>> 27: 0 0 0 0 IO-APIC 27-fasteoi
>> idma64.1, i2c_designware.1
>> 29: 0 0 0 0 IO-APIC 29-fasteoi
>> idma64.2, i2c_designware.2
>> 30: 0 0 0 0 IO-APIC 30-fasteoi
>> idma64.3, i2c_designware.3
>> 31: 0 0 0 0 IO-APIC 31-fasteoi
>> idma64.4, i2c_designware.4
>> 32: 0 0 0 0 IO-APIC 32-fasteoi
>> idma64.5, i2c_designware.5
>> 126: 0 0 0 0 PCI-MSI-0000:00:0d.0
>> 0-edge xhci_hcd
>> 127: 0 0 0 21855 PCI-MSI-0000:00:17.0
>> 0-edge ahci[0000:00:17.0]
>> 128: 0 0 1 0 PCI-MSIX-0000:04:00.0
>> 0-edge enp4s0
>> 129: 371 0 0 35 PCI-MSIX-0000:04:00.0
>> 1-edge enp4s0-TxRx-0
>> 130: 36 0 0 310 PCI-MSIX-0000:04:00.0
>> 2-edge enp4s0-TxRx-1
>> 131: 0 36 334 0 PCI-MSIX-0000:04:00.0
>> 3-edge enp4s0-TxRx-2
>> 132: 0 0 285 0 PCI-MSIX-0000:04:00.0
>> 4-edge enp4s0-TxRx-3
>> 133: 1681 74 0 0 PCI-MSI-0000:00:14.0
>> 0-edge xhci_hcd
>> 134: 0 0 0 0 accel_3d-dev0
>> accel_3d_consumer0
>> 136: 0 0 0 0 als-dev1
>> als_consumer1
>> 142: 0 0 0 0 magn_3d-dev3
>> magn_3d_consumer3
>> 150: 49 0 0 0 PCI-MSI-0000:00:16.0
>> 0-edge mei_me
>> 157: 5552 319 0 0 PCI-MSI-0000:00:02.0
>> 0-edge i915
>> NMI: 1 0 3 3 Non-maskable
>> interrupts
>> LOC: 25207 348 30717 29537 Local timer interrupts
>> SPU: 0 0 0 0 Spurious interrupts
>> PMI: 1 0 3 3 Performance monitoring
>> interrupts
>> IWI: 2162 11 119 207 IRQ work interrupts
>> RTR: 2 0 0 0 APIC ICR read retries
>> RES: 2902 326 3213 2405 Rescheduling
>> interrupts
>> CAL: 46790 5794 43018 41150 Function call
>> interrupts
>> TLB: 273 0 258 149 TLB shootdowns
>> TRM: 0 0 0 0 Thermal event
>> interrupts
>> THR: 0 0 0 0 Threshold APIC
>> interrupts
>> DFR: 0 0 0 0 Deferred Error APIC
>> interrupts
>> MCE: 0 0 0 0 Machine check
>> exceptions
>> MCP: 1 2 2 2 Machine check polls
>> ERR: 2
>> MIS: 0
>> PIN: 0 0 0 0 Posted-interrupt
>> notification event
>> NPI: 0 0 0 0 Nested posted-
>> interrupt event
>> PIW: 0 0 0 0 Posted-interrupt
>> wakeup event
>>
>> > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE only
>> > (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still
>> > causes the issue. That might help pointing the finger at either a
>> > generic IRQ pipelining issue, or something related to the tick device
>> instead.
>> >
>>
>> I will try this today and respond as soon as I get the results.
>> Regards,
>> Jim
>>
>
> I can confirm that the system hangs with "CONFIG_IRQ_PIPELINE=y" and "CONFIG_DOVETAIL is not set".
> This is identical to the original failure.
>
Ok, so this points the finger at irq pipelining. Something may be
missing in some irqchip driver to support this.
--
Philippe.
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.
2024-03-20 21:36 ` Philippe Gerum
@ 2024-03-21 5:56 ` Chen, Hongzhan
0 siblings, 0 replies; 10+ messages in thread
From: Chen, Hongzhan @ 2024-03-21 5:56 UTC (permalink / raw)
To: Philippe Gerum, Rosenow, James; +Cc: xenomai@lists.linux.dev
Hi James
Could you share your complete config after enable CONFIG_IRQ_PIPELINE? I will try to reproduce
on my side. I met similar thing before because of kernel configuration issue.
Regards
Hongzhan Chen
>-----Original Message-----
>From: Philippe Gerum <rpm@xenomai.org>
>Sent: Thursday, March 21, 2024 5:36 AM
>To: Rosenow, James <jim.rosenow@mts.com>
>Cc: xenomai@lists.linux.dev
>Subject: Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9,
>7, and 6 work.
>
>
>"Rosenow, James" <jim.rosenow@mts.com> writes:
>
>>> > "Rosenow, James" <jim.rosenow@mts.com> writes:
>>> >
>>> > > We have a COM express carrier with an on-board SSD. The SSD has
>>> > > Ubuntu 22.04 installed on it, on top of that, I've installed the
>>> > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel
>>> > > from the linux-dovetail repository . Given that our board supports
>>> > > COM Express, we can easily swap between various processors and have
>>> > > been doing so looking for the best price/performance ratio and of
>>> > > course, low latency.
>>> > >
>>> > > I have two modules that I am currently testing, an i3-11100HE and an
>>> > > i7-
>>> > 9850HL. Since the SSD is on the base board, I'm running the exact
>>> > same version of Ubuntu, kernel and root file system regardless of the
>>> > installed processor board.
>>> > >
>>> > > The gen9 processor boots and runs the Xenomai kernel without any
>>> > > trouble at all. I'm having SMI issues but that is another subject.
>>> > > When I put the gen11 board on the carrier, it hangs at 'loading
>>> > > initial ramdisk .'. It is noteworthy that if I interrupt grub and
>>> > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen
>>> > > 9 boot and run fine. I've verified the Xenomai kernel runs fine on
>>> > > generation 7 and 6 CPU's as well.
>>> > >
>>> > > I've searched the web for possible reasons why Linux would hang at
>>> > > that
>>> > point; for Linux in general, it is a common thing and I've tried many,
>>> > many of the solutions that others have found to work, all to no avail.
>>> > I do not know how to get additional information on what is hanging as
>>> > there is really no context from which to proceed.
>>> > >
>>> > > I'm looking for suggestions on next steps to root cause this hang.
>>> > >
>>> > > Thanks and regards,
>>> > > Jim
>>> >
>>> > Could you boot the machine with the stock kernel that works, then
>>> > paste the output of /proc/interrupts? This may help finding out which
>>> > IRQ controller might be involved in the issue.
>>> >
>>>
>>> Hello Philippe,
>>> Here is the output of /proc/interrupts from the stock 6.5.0 kernel.
>>>
>>> $ cat /proc/interrupts
>>> CPU0 CPU1 CPU2 CPU3
>>> 0: 15 0 0 0 IO-APIC 2-edge
>>> timer
>>> 3: 0 0 0 2 IO-APIC 3-edge
>>> 4: 0 0 0 2 IO-APIC 4-edge
>>> 8: 0 0 0 0 IO-APIC 8-edge
>>> rtc0
>>> 9: 611 154 0 0 IO-APIC 9-fasteoi
>>> acpi
>>> 14: 0 0 0 0 IO-APIC 14-fasteoi
>>> INT34C6:00
>>> 16: 699 0 0 0 IO-APIC 16-fasteoi
>>> i801_smbus, intel_ish_ipc
>>> 18: 0 0 0 0 IO-APIC 18-fasteoi
>>> i2c_designware.0
>>> 27: 0 0 0 0 IO-APIC 27-fasteoi
>>> idma64.1, i2c_designware.1
>>> 29: 0 0 0 0 IO-APIC 29-fasteoi
>>> idma64.2, i2c_designware.2
>>> 30: 0 0 0 0 IO-APIC 30-fasteoi
>>> idma64.3, i2c_designware.3
>>> 31: 0 0 0 0 IO-APIC 31-fasteoi
>>> idma64.4, i2c_designware.4
>>> 32: 0 0 0 0 IO-APIC 32-fasteoi
>>> idma64.5, i2c_designware.5
>>> 126: 0 0 0 0 PCI-MSI-0000:00:0d.0
>>> 0-edge xhci_hcd
>>> 127: 0 0 0 21855 PCI-MSI-0000:00:17.0
>>> 0-edge ahci[0000:00:17.0]
>>> 128: 0 0 1 0 PCI-MSIX-0000:04:00.0
>>> 0-edge enp4s0
>>> 129: 371 0 0 35 PCI-MSIX-0000:04:00.0
>>> 1-edge enp4s0-TxRx-0
>>> 130: 36 0 0 310 PCI-MSIX-0000:04:00.0
>>> 2-edge enp4s0-TxRx-1
>>> 131: 0 36 334 0 PCI-MSIX-0000:04:00.0
>>> 3-edge enp4s0-TxRx-2
>>> 132: 0 0 285 0 PCI-MSIX-0000:04:00.0
>>> 4-edge enp4s0-TxRx-3
>>> 133: 1681 74 0 0 PCI-MSI-0000:00:14.0
>>> 0-edge xhci_hcd
>>> 134: 0 0 0 0 accel_3d-dev0
>>> accel_3d_consumer0
>>> 136: 0 0 0 0 als-dev1
>>> als_consumer1
>>> 142: 0 0 0 0 magn_3d-dev3
>>> magn_3d_consumer3
>>> 150: 49 0 0 0 PCI-MSI-0000:00:16.0
>>> 0-edge mei_me
>>> 157: 5552 319 0 0 PCI-MSI-0000:00:02.0
>>> 0-edge i915
>>> NMI: 1 0 3 3 Non-maskable
>>> interrupts
>>> LOC: 25207 348 30717 29537 Local timer interrupts
>>> SPU: 0 0 0 0 Spurious interrupts
>>> PMI: 1 0 3 3 Performance monitoring
>>> interrupts
>>> IWI: 2162 11 119 207 IRQ work interrupts
>>> RTR: 2 0 0 0 APIC ICR read retries
>>> RES: 2902 326 3213 2405 Rescheduling
>>> interrupts
>>> CAL: 46790 5794 43018 41150 Function call
>>> interrupts
>>> TLB: 273 0 258 149 TLB shootdowns
>>> TRM: 0 0 0 0 Thermal event
>>> interrupts
>>> THR: 0 0 0 0 Threshold APIC
>>> interrupts
>>> DFR: 0 0 0 0 Deferred Error APIC
>>> interrupts
>>> MCE: 0 0 0 0 Machine check
>>> exceptions
>>> MCP: 1 2 2 2 Machine check polls
>>> ERR: 2
>>> MIS: 0
>>> PIN: 0 0 0 0 Posted-interrupt
>>> notification event
>>> NPI: 0 0 0 0 Nested posted-
>>> interrupt event
>>> PIW: 0 0 0 0 Posted-interrupt
>>> wakeup event
>>>
>>> > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE
>only
>>> > (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still
>>> > causes the issue. That might help pointing the finger at either a
>>> > generic IRQ pipelining issue, or something related to the tick device
>>> instead.
>>> >
>>>
>>> I will try this today and respond as soon as I get the results.
>>> Regards,
>>> Jim
>>>
>>
>> I can confirm that the system hangs with "CONFIG_IRQ_PIPELINE=y" and
>"CONFIG_DOVETAIL is not set".
>> This is identical to the original failure.
>>
>
>Ok, so this points the finger at irq pipelining. Something may be
>missing in some irqchip driver to support this.
>
>--
>Philippe.
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-03-21 5:57 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <DM6PR04MB538655375C57CF07EDDE5089E2242@DM6PR04MB5386.namprd04.prod.outlook.com>
[not found] ` <DM6PR04MB538673AA2E912100093BBA7CE2242@DM6PR04MB5386.namprd04.prod.outlook.com>
2024-03-11 16:52 ` Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work Rosenow, James
2024-03-11 17:26 ` Philippe Gerum
2024-03-13 16:11 ` Rosenow, James
2024-03-13 18:32 ` Rosenow, James
2024-03-19 15:19 ` Rosenow, James
2024-03-19 15:40 ` Philippe Gerum
2024-03-20 7:44 ` Chen, Hongzhan
2024-03-20 21:36 ` Philippe Gerum
2024-03-21 5:56 ` Chen, Hongzhan
2024-03-20 21:25 ` Philippe Gerum
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.