* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. [not found] ` <DM6PR04MB538673AA2E912100093BBA7CE2242@DM6PR04MB5386.namprd04.prod.outlook.com> @ 2024-03-11 16:52 ` Rosenow, James 2024-03-11 17:26 ` Philippe Gerum 0 siblings, 1 reply; 10+ messages in thread From: Rosenow, James @ 2024-03-11 16:52 UTC (permalink / raw) To: xenomai@lists.linux.dev We have a COM express carrier with an on-board SSD. The SSD has Ubuntu 22.04 installed on it, on top of that, I've installed the latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel from the linux-dovetail repository . Given that our board supports COM Express, we can easily swap between various processors and have been doing so looking for the best price/performance ratio and of course, low latency. I have two modules that I am currently testing, an i3-11100HE and an i7-9850HL. Since the SSD is on the base board, I'm running the exact same version of Ubuntu, kernel and root file system regardless of the installed processor board. The gen9 processor boots and runs the Xenomai kernel without any trouble at all. I'm having SMI issues but that is another subject. When I put the gen11 board on the carrier, it hangs at 'loading initial ramdisk .'. It is noteworthy that if I interrupt grub and select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen 9 boot and run fine. I've verified the Xenomai kernel runs fine on generation 7 and 6 CPU's as well. I've searched the web for possible reasons why Linux would hang at that point; for Linux in general, it is a common thing and I've tried many, many of the solutions that others have found to work, all to no avail. I do not know how to get additional information on what is hanging as there is really no context from which to proceed. I'm looking for suggestions on next steps to root cause this hang. Thanks and regards, Jim ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. 2024-03-11 16:52 ` Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work Rosenow, James @ 2024-03-11 17:26 ` Philippe Gerum 2024-03-13 16:11 ` Rosenow, James 0 siblings, 1 reply; 10+ messages in thread From: Philippe Gerum @ 2024-03-11 17:26 UTC (permalink / raw) To: Rosenow, James; +Cc: xenomai@lists.linux.dev "Rosenow, James" <jim.rosenow@mts.com> writes: > We have a COM express carrier with an on-board SSD. The SSD has > Ubuntu 22.04 installed on it, on top of that, I've installed the > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel > from the linux-dovetail repository . Given that our board supports > COM Express, we can easily swap between various processors and have > been doing so looking for the best price/performance ratio and of > course, low latency. > > I have two modules that I am currently testing, an i3-11100HE and an i7-9850HL. Since the SSD is on the base board, I'm running the exact same version of Ubuntu, kernel and root file system regardless of the installed processor board. > > The gen9 processor boots and runs the Xenomai kernel without any > trouble at all. I'm having SMI issues but that is another subject. > When I put the gen11 board on the carrier, it hangs at 'loading > initial ramdisk .'. It is noteworthy that if I interrupt grub and > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen 9 > boot and run fine. I've verified the Xenomai kernel runs fine on > generation 7 and 6 CPU's as well. > > I've searched the web for possible reasons why Linux would hang at that point; for Linux in general, it is a common thing and I've tried many, many of the solutions that others have found to work, all to no avail. I do not know how to get additional information on what is hanging as there is really no context from which to proceed. > > I'm looking for suggestions on next steps to root cause this hang. > > Thanks and regards, > Jim Could you boot the machine with the stock kernel that works, then paste the output of /proc/interrupts? This may help finding out which IRQ controller might be involved in the issue. Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE only (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still causes the issue. That might help pointing the finger at either a generic IRQ pipelining issue, or something related to the tick device instead. -- Philippe. ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. 2024-03-11 17:26 ` Philippe Gerum @ 2024-03-13 16:11 ` Rosenow, James 2024-03-13 18:32 ` Rosenow, James 2024-03-20 21:25 ` Philippe Gerum 0 siblings, 2 replies; 10+ messages in thread From: Rosenow, James @ 2024-03-13 16:11 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai@lists.linux.dev > "Rosenow, James" <jim.rosenow@mts.com> writes: > > > We have a COM express carrier with an on-board SSD. The SSD has > > Ubuntu 22.04 installed on it, on top of that, I've installed the > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel > > from the linux-dovetail repository . Given that our board supports > > COM Express, we can easily swap between various processors and have > > been doing so looking for the best price/performance ratio and of > > course, low latency. > > > > I have two modules that I am currently testing, an i3-11100HE and an i7- > 9850HL. Since the SSD is on the base board, I'm running the exact same > version of Ubuntu, kernel and root file system regardless of the installed > processor board. > > > > The gen9 processor boots and runs the Xenomai kernel without any > > trouble at all. I'm having SMI issues but that is another subject. > > When I put the gen11 board on the carrier, it hangs at 'loading > > initial ramdisk .'. It is noteworthy that if I interrupt grub and > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen 9 > > boot and run fine. I've verified the Xenomai kernel runs fine on > > generation 7 and 6 CPU's as well. > > > > I've searched the web for possible reasons why Linux would hang at that > point; for Linux in general, it is a common thing and I've tried many, > many of the solutions that others have found to work, all to no avail. I > do not know how to get additional information on what is hanging as there > is really no context from which to proceed. > > > > I'm looking for suggestions on next steps to root cause this hang. > > > > Thanks and regards, > > Jim > > Could you boot the machine with the stock kernel that works, then paste > the output of /proc/interrupts? This may help finding out which IRQ > controller might be involved in the issue. > Hello Philippe, Here is the output of /proc/interrupts from the stock 6.5.0 kernel. $ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 15 0 0 0 IO-APIC 2-edge timer 3: 0 0 0 2 IO-APIC 3-edge 4: 0 0 0 2 IO-APIC 4-edge 8: 0 0 0 0 IO-APIC 8-edge rtc0 9: 611 154 0 0 IO-APIC 9-fasteoi acpi 14: 0 0 0 0 IO-APIC 14-fasteoi INT34C6:00 16: 699 0 0 0 IO-APIC 16-fasteoi i801_smbus, intel_ish_ipc 18: 0 0 0 0 IO-APIC 18-fasteoi i2c_designware.0 27: 0 0 0 0 IO-APIC 27-fasteoi idma64.1, i2c_designware.1 29: 0 0 0 0 IO-APIC 29-fasteoi idma64.2, i2c_designware.2 30: 0 0 0 0 IO-APIC 30-fasteoi idma64.3, i2c_designware.3 31: 0 0 0 0 IO-APIC 31-fasteoi idma64.4, i2c_designware.4 32: 0 0 0 0 IO-APIC 32-fasteoi idma64.5, i2c_designware.5 126: 0 0 0 0 PCI-MSI-0000:00:0d.0 0-edge xhci_hcd 127: 0 0 0 21855 PCI-MSI-0000:00:17.0 0-edge ahci[0000:00:17.0] 128: 0 0 1 0 PCI-MSIX-0000:04:00.0 0-edge enp4s0 129: 371 0 0 35 PCI-MSIX-0000:04:00.0 1-edge enp4s0-TxRx-0 130: 36 0 0 310 PCI-MSIX-0000:04:00.0 2-edge enp4s0-TxRx-1 131: 0 36 334 0 PCI-MSIX-0000:04:00.0 3-edge enp4s0-TxRx-2 132: 0 0 285 0 PCI-MSIX-0000:04:00.0 4-edge enp4s0-TxRx-3 133: 1681 74 0 0 PCI-MSI-0000:00:14.0 0-edge xhci_hcd 134: 0 0 0 0 accel_3d-dev0 accel_3d_consumer0 136: 0 0 0 0 als-dev1 als_consumer1 142: 0 0 0 0 magn_3d-dev3 magn_3d_consumer3 150: 49 0 0 0 PCI-MSI-0000:00:16.0 0-edge mei_me 157: 5552 319 0 0 PCI-MSI-0000:00:02.0 0-edge i915 NMI: 1 0 3 3 Non-maskable interrupts LOC: 25207 348 30717 29537 Local timer interrupts SPU: 0 0 0 0 Spurious interrupts PMI: 1 0 3 3 Performance monitoring interrupts IWI: 2162 11 119 207 IRQ work interrupts RTR: 2 0 0 0 APIC ICR read retries RES: 2902 326 3213 2405 Rescheduling interrupts CAL: 46790 5794 43018 41150 Function call interrupts TLB: 273 0 258 149 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts DFR: 0 0 0 0 Deferred Error APIC interrupts MCE: 0 0 0 0 Machine check exceptions MCP: 1 2 2 2 Machine check polls ERR: 2 MIS: 0 PIN: 0 0 0 0 Posted-interrupt notification event NPI: 0 0 0 0 Nested posted-interrupt event PIW: 0 0 0 0 Posted-interrupt wakeup event > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE only > (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still causes > the issue. That might help pointing the finger at either a generic IRQ > pipelining issue, or something related to the tick device instead. > I will try this today and respond as soon as I get the results. Regards, Jim > -- > Philippe. ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. 2024-03-13 16:11 ` Rosenow, James @ 2024-03-13 18:32 ` Rosenow, James 2024-03-19 15:19 ` Rosenow, James 2024-03-20 21:36 ` Philippe Gerum 2024-03-20 21:25 ` Philippe Gerum 1 sibling, 2 replies; 10+ messages in thread From: Rosenow, James @ 2024-03-13 18:32 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai@lists.linux.dev > > "Rosenow, James" <jim.rosenow@mts.com> writes: > > > > > We have a COM express carrier with an on-board SSD. The SSD has > > > Ubuntu 22.04 installed on it, on top of that, I've installed the > > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel > > > from the linux-dovetail repository . Given that our board supports > > > COM Express, we can easily swap between various processors and have > > > been doing so looking for the best price/performance ratio and of > > > course, low latency. > > > > > > I have two modules that I am currently testing, an i3-11100HE and an > > > i7- > > 9850HL. Since the SSD is on the base board, I'm running the exact > > same version of Ubuntu, kernel and root file system regardless of the > > installed processor board. > > > > > > The gen9 processor boots and runs the Xenomai kernel without any > > > trouble at all. I'm having SMI issues but that is another subject. > > > When I put the gen11 board on the carrier, it hangs at 'loading > > > initial ramdisk .'. It is noteworthy that if I interrupt grub and > > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen > > > 9 boot and run fine. I've verified the Xenomai kernel runs fine on > > > generation 7 and 6 CPU's as well. > > > > > > I've searched the web for possible reasons why Linux would hang at > > > that > > point; for Linux in general, it is a common thing and I've tried many, > > many of the solutions that others have found to work, all to no avail. > > I do not know how to get additional information on what is hanging as > > there is really no context from which to proceed. > > > > > > I'm looking for suggestions on next steps to root cause this hang. > > > > > > Thanks and regards, > > > Jim > > > > Could you boot the machine with the stock kernel that works, then > > paste the output of /proc/interrupts? This may help finding out which > > IRQ controller might be involved in the issue. > > > > Hello Philippe, > Here is the output of /proc/interrupts from the stock 6.5.0 kernel. > > $ cat /proc/interrupts > CPU0 CPU1 CPU2 CPU3 > 0: 15 0 0 0 IO-APIC 2-edge > timer > 3: 0 0 0 2 IO-APIC 3-edge > 4: 0 0 0 2 IO-APIC 4-edge > 8: 0 0 0 0 IO-APIC 8-edge > rtc0 > 9: 611 154 0 0 IO-APIC 9-fasteoi > acpi > 14: 0 0 0 0 IO-APIC 14-fasteoi > INT34C6:00 > 16: 699 0 0 0 IO-APIC 16-fasteoi > i801_smbus, intel_ish_ipc > 18: 0 0 0 0 IO-APIC 18-fasteoi > i2c_designware.0 > 27: 0 0 0 0 IO-APIC 27-fasteoi > idma64.1, i2c_designware.1 > 29: 0 0 0 0 IO-APIC 29-fasteoi > idma64.2, i2c_designware.2 > 30: 0 0 0 0 IO-APIC 30-fasteoi > idma64.3, i2c_designware.3 > 31: 0 0 0 0 IO-APIC 31-fasteoi > idma64.4, i2c_designware.4 > 32: 0 0 0 0 IO-APIC 32-fasteoi > idma64.5, i2c_designware.5 > 126: 0 0 0 0 PCI-MSI-0000:00:0d.0 > 0-edge xhci_hcd > 127: 0 0 0 21855 PCI-MSI-0000:00:17.0 > 0-edge ahci[0000:00:17.0] > 128: 0 0 1 0 PCI-MSIX-0000:04:00.0 > 0-edge enp4s0 > 129: 371 0 0 35 PCI-MSIX-0000:04:00.0 > 1-edge enp4s0-TxRx-0 > 130: 36 0 0 310 PCI-MSIX-0000:04:00.0 > 2-edge enp4s0-TxRx-1 > 131: 0 36 334 0 PCI-MSIX-0000:04:00.0 > 3-edge enp4s0-TxRx-2 > 132: 0 0 285 0 PCI-MSIX-0000:04:00.0 > 4-edge enp4s0-TxRx-3 > 133: 1681 74 0 0 PCI-MSI-0000:00:14.0 > 0-edge xhci_hcd > 134: 0 0 0 0 accel_3d-dev0 > accel_3d_consumer0 > 136: 0 0 0 0 als-dev1 > als_consumer1 > 142: 0 0 0 0 magn_3d-dev3 > magn_3d_consumer3 > 150: 49 0 0 0 PCI-MSI-0000:00:16.0 > 0-edge mei_me > 157: 5552 319 0 0 PCI-MSI-0000:00:02.0 > 0-edge i915 > NMI: 1 0 3 3 Non-maskable > interrupts > LOC: 25207 348 30717 29537 Local timer interrupts > SPU: 0 0 0 0 Spurious interrupts > PMI: 1 0 3 3 Performance monitoring > interrupts > IWI: 2162 11 119 207 IRQ work interrupts > RTR: 2 0 0 0 APIC ICR read retries > RES: 2902 326 3213 2405 Rescheduling > interrupts > CAL: 46790 5794 43018 41150 Function call > interrupts > TLB: 273 0 258 149 TLB shootdowns > TRM: 0 0 0 0 Thermal event > interrupts > THR: 0 0 0 0 Threshold APIC > interrupts > DFR: 0 0 0 0 Deferred Error APIC > interrupts > MCE: 0 0 0 0 Machine check > exceptions > MCP: 1 2 2 2 Machine check polls > ERR: 2 > MIS: 0 > PIN: 0 0 0 0 Posted-interrupt > notification event > NPI: 0 0 0 0 Nested posted- > interrupt event > PIW: 0 0 0 0 Posted-interrupt > wakeup event > > > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE only > > (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still > > causes the issue. That might help pointing the finger at either a > > generic IRQ pipelining issue, or something related to the tick device > instead. > > > > I will try this today and respond as soon as I get the results. > Regards, > Jim > I can confirm that the system hangs with "CONFIG_IRQ_PIPELINE=y" and "CONFIG_DOVETAIL is not set". This is identical to the original failure. -Jim > > -- > > Philippe. ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. 2024-03-13 18:32 ` Rosenow, James @ 2024-03-19 15:19 ` Rosenow, James 2024-03-19 15:40 ` Philippe Gerum 2024-03-20 21:36 ` Philippe Gerum 1 sibling, 1 reply; 10+ messages in thread From: Rosenow, James @ 2024-03-19 15:19 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai@lists.linux.dev Anyone have any advice on how to root cause this hang. The requested /proc/interrupts output is below and the system does hang with only CONFIG_IRQ_PIPELINE enabled. Thanks in advance. Jim > > > "Rosenow, James" <jim.rosenow@mts.com> writes: > > > > > > > We have a COM express carrier with an on-board SSD. The SSD has > > > > Ubuntu 22.04 installed on it, on top of that, I've installed the > > > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' > > > > kernel from the linux-dovetail repository . Given that our board > > > > supports COM Express, we can easily swap between various > > > > processors and have been doing so looking for the best > > > > price/performance ratio and of course, low latency. > > > > > > > > I have two modules that I am currently testing, an i3-11100HE and > > > > an > > > > i7- > > > 9850HL. Since the SSD is on the base board, I'm running the exact > > > same version of Ubuntu, kernel and root file system regardless of > > > the installed processor board. > > > > > > > > The gen9 processor boots and runs the Xenomai kernel without any > > > > trouble at all. I'm having SMI issues but that is another subject. > > > > When I put the gen11 board on the carrier, it hangs at 'loading > > > > initial ramdisk .'. It is noteworthy that if I interrupt grub and > > > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and > > > > gen > > > > 9 boot and run fine. I've verified the Xenomai kernel runs fine > > > > on generation 7 and 6 CPU's as well. > > > > > > > > I've searched the web for possible reasons why Linux would hang at > > > > that > > > point; for Linux in general, it is a common thing and I've tried > > > many, many of the solutions that others have found to work, all to no > avail. > > > I do not know how to get additional information on what is hanging > > > as there is really no context from which to proceed. > > > > > > > > I'm looking for suggestions on next steps to root cause this hang. > > > > > > > > Thanks and regards, > > > > Jim > > > > > > Could you boot the machine with the stock kernel that works, then > > > paste the output of /proc/interrupts? This may help finding out > > > which IRQ controller might be involved in the issue. > > > > > > > Hello Philippe, > > Here is the output of /proc/interrupts from the stock 6.5.0 kernel. > > > > $ cat /proc/interrupts > > CPU0 CPU1 CPU2 CPU3 > > 0: 15 0 0 0 IO-APIC 2-edge > > timer > > 3: 0 0 0 2 IO-APIC 3-edge > > 4: 0 0 0 2 IO-APIC 4-edge > > 8: 0 0 0 0 IO-APIC 8-edge > > rtc0 > > 9: 611 154 0 0 IO-APIC 9-fasteoi > > acpi > > 14: 0 0 0 0 IO-APIC 14-fasteoi > > INT34C6:00 > > 16: 699 0 0 0 IO-APIC 16-fasteoi > > i801_smbus, intel_ish_ipc > > 18: 0 0 0 0 IO-APIC 18-fasteoi > > i2c_designware.0 > > 27: 0 0 0 0 IO-APIC 27-fasteoi > > idma64.1, i2c_designware.1 > > 29: 0 0 0 0 IO-APIC 29-fasteoi > > idma64.2, i2c_designware.2 > > 30: 0 0 0 0 IO-APIC 30-fasteoi > > idma64.3, i2c_designware.3 > > 31: 0 0 0 0 IO-APIC 31-fasteoi > > idma64.4, i2c_designware.4 > > 32: 0 0 0 0 IO-APIC 32-fasteoi > > idma64.5, i2c_designware.5 > > 126: 0 0 0 0 PCI-MSI-0000:00:0d.0 > > 0-edge xhci_hcd > > 127: 0 0 0 21855 PCI-MSI-0000:00:17.0 > > 0-edge ahci[0000:00:17.0] > > 128: 0 0 1 0 PCI-MSIX-0000:04:00.0 > > 0-edge enp4s0 > > 129: 371 0 0 35 PCI-MSIX-0000:04:00.0 > > 1-edge enp4s0-TxRx-0 > > 130: 36 0 0 310 PCI-MSIX-0000:04:00.0 > > 2-edge enp4s0-TxRx-1 > > 131: 0 36 334 0 PCI-MSIX-0000:04:00.0 > > 3-edge enp4s0-TxRx-2 > > 132: 0 0 285 0 PCI-MSIX-0000:04:00.0 > > 4-edge enp4s0-TxRx-3 > > 133: 1681 74 0 0 PCI-MSI-0000:00:14.0 > > 0-edge xhci_hcd > > 134: 0 0 0 0 accel_3d-dev0 > > accel_3d_consumer0 > > 136: 0 0 0 0 als-dev1 > > als_consumer1 > > 142: 0 0 0 0 magn_3d-dev3 > > magn_3d_consumer3 > > 150: 49 0 0 0 PCI-MSI-0000:00:16.0 > > 0-edge mei_me > > 157: 5552 319 0 0 PCI-MSI-0000:00:02.0 > > 0-edge i915 > > NMI: 1 0 3 3 Non-maskable > > interrupts > > LOC: 25207 348 30717 29537 Local timer > interrupts > > SPU: 0 0 0 0 Spurious interrupts > > PMI: 1 0 3 3 Performance > monitoring > > interrupts > > IWI: 2162 11 119 207 IRQ work interrupts > > RTR: 2 0 0 0 APIC ICR read > retries > > RES: 2902 326 3213 2405 Rescheduling > > interrupts > > CAL: 46790 5794 43018 41150 Function call > > interrupts > > TLB: 273 0 258 149 TLB shootdowns > > TRM: 0 0 0 0 Thermal event > > interrupts > > THR: 0 0 0 0 Threshold APIC > > interrupts > > DFR: 0 0 0 0 Deferred Error APIC > > interrupts > > MCE: 0 0 0 0 Machine check > > exceptions > > MCP: 1 2 2 2 Machine check polls > > ERR: 2 > > MIS: 0 > > PIN: 0 0 0 0 Posted-interrupt > > notification event > > NPI: 0 0 0 0 Nested posted- > > interrupt event > > PIW: 0 0 0 0 Posted-interrupt > > wakeup event > > > > > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE > > > only (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) > > > still causes the issue. That might help pointing the finger at > > > either a generic IRQ pipelining issue, or something related to the > > > tick device > > instead. > > > > > > > I will try this today and respond as soon as I get the results. > > Regards, > > Jim > > > > I can confirm that the system hangs with "CONFIG_IRQ_PIPELINE=y" and > "CONFIG_DOVETAIL is not set". > This is identical to the original failure. > > -Jim > > > > -- > > > Philippe. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. 2024-03-19 15:19 ` Rosenow, James @ 2024-03-19 15:40 ` Philippe Gerum 2024-03-20 7:44 ` Chen, Hongzhan 0 siblings, 1 reply; 10+ messages in thread From: Philippe Gerum @ 2024-03-19 15:40 UTC (permalink / raw) To: Rosenow, James; +Cc: xenomai@lists.linux.dev "Rosenow, James" <jim.rosenow@mts.com> writes: > Anyone have any advice on how to root cause this hang. > > The requested /proc/interrupts output is below and the system does hang with only CONFIG_IRQ_PIPELINE enabled. > I'll have a look asap this week. -- Philippe. ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. 2024-03-19 15:40 ` Philippe Gerum @ 2024-03-20 7:44 ` Chen, Hongzhan 0 siblings, 0 replies; 10+ messages in thread From: Chen, Hongzhan @ 2024-03-20 7:44 UTC (permalink / raw) To: Philippe Gerum, Rosenow, James; +Cc: xenomai@lists.linux.dev >-----Original Message----- >From: Philippe Gerum <rpm@xenomai.org> >Sent: Tuesday, March 19, 2024 11:41 PM >To: Rosenow, James <jim.rosenow@mts.com> >Cc: xenomai@lists.linux.dev >Subject: Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, >7, and 6 work. > > >"Rosenow, James" <jim.rosenow@mts.com> writes: > >> Anyone have any advice on how to root cause this hang. >> >> The requested /proc/interrupts output is below and the system does hang >with only CONFIG_IRQ_PIPELINE enabled. >> > >I'll have a look asap this week. I also reproduced such issue and am looking into it on Gen 11 CPU. Regards Hongzhan Chen > >-- >Philippe. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. 2024-03-13 18:32 ` Rosenow, James 2024-03-19 15:19 ` Rosenow, James @ 2024-03-20 21:36 ` Philippe Gerum 2024-03-21 5:56 ` Chen, Hongzhan 1 sibling, 1 reply; 10+ messages in thread From: Philippe Gerum @ 2024-03-20 21:36 UTC (permalink / raw) To: Rosenow, James; +Cc: xenomai@lists.linux.dev "Rosenow, James" <jim.rosenow@mts.com> writes: >> > "Rosenow, James" <jim.rosenow@mts.com> writes: >> > >> > > We have a COM express carrier with an on-board SSD. The SSD has >> > > Ubuntu 22.04 installed on it, on top of that, I've installed the >> > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel >> > > from the linux-dovetail repository . Given that our board supports >> > > COM Express, we can easily swap between various processors and have >> > > been doing so looking for the best price/performance ratio and of >> > > course, low latency. >> > > >> > > I have two modules that I am currently testing, an i3-11100HE and an >> > > i7- >> > 9850HL. Since the SSD is on the base board, I'm running the exact >> > same version of Ubuntu, kernel and root file system regardless of the >> > installed processor board. >> > > >> > > The gen9 processor boots and runs the Xenomai kernel without any >> > > trouble at all. I'm having SMI issues but that is another subject. >> > > When I put the gen11 board on the carrier, it hangs at 'loading >> > > initial ramdisk .'. It is noteworthy that if I interrupt grub and >> > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen >> > > 9 boot and run fine. I've verified the Xenomai kernel runs fine on >> > > generation 7 and 6 CPU's as well. >> > > >> > > I've searched the web for possible reasons why Linux would hang at >> > > that >> > point; for Linux in general, it is a common thing and I've tried many, >> > many of the solutions that others have found to work, all to no avail. >> > I do not know how to get additional information on what is hanging as >> > there is really no context from which to proceed. >> > > >> > > I'm looking for suggestions on next steps to root cause this hang. >> > > >> > > Thanks and regards, >> > > Jim >> > >> > Could you boot the machine with the stock kernel that works, then >> > paste the output of /proc/interrupts? This may help finding out which >> > IRQ controller might be involved in the issue. >> > >> >> Hello Philippe, >> Here is the output of /proc/interrupts from the stock 6.5.0 kernel. >> >> $ cat /proc/interrupts >> CPU0 CPU1 CPU2 CPU3 >> 0: 15 0 0 0 IO-APIC 2-edge >> timer >> 3: 0 0 0 2 IO-APIC 3-edge >> 4: 0 0 0 2 IO-APIC 4-edge >> 8: 0 0 0 0 IO-APIC 8-edge >> rtc0 >> 9: 611 154 0 0 IO-APIC 9-fasteoi >> acpi >> 14: 0 0 0 0 IO-APIC 14-fasteoi >> INT34C6:00 >> 16: 699 0 0 0 IO-APIC 16-fasteoi >> i801_smbus, intel_ish_ipc >> 18: 0 0 0 0 IO-APIC 18-fasteoi >> i2c_designware.0 >> 27: 0 0 0 0 IO-APIC 27-fasteoi >> idma64.1, i2c_designware.1 >> 29: 0 0 0 0 IO-APIC 29-fasteoi >> idma64.2, i2c_designware.2 >> 30: 0 0 0 0 IO-APIC 30-fasteoi >> idma64.3, i2c_designware.3 >> 31: 0 0 0 0 IO-APIC 31-fasteoi >> idma64.4, i2c_designware.4 >> 32: 0 0 0 0 IO-APIC 32-fasteoi >> idma64.5, i2c_designware.5 >> 126: 0 0 0 0 PCI-MSI-0000:00:0d.0 >> 0-edge xhci_hcd >> 127: 0 0 0 21855 PCI-MSI-0000:00:17.0 >> 0-edge ahci[0000:00:17.0] >> 128: 0 0 1 0 PCI-MSIX-0000:04:00.0 >> 0-edge enp4s0 >> 129: 371 0 0 35 PCI-MSIX-0000:04:00.0 >> 1-edge enp4s0-TxRx-0 >> 130: 36 0 0 310 PCI-MSIX-0000:04:00.0 >> 2-edge enp4s0-TxRx-1 >> 131: 0 36 334 0 PCI-MSIX-0000:04:00.0 >> 3-edge enp4s0-TxRx-2 >> 132: 0 0 285 0 PCI-MSIX-0000:04:00.0 >> 4-edge enp4s0-TxRx-3 >> 133: 1681 74 0 0 PCI-MSI-0000:00:14.0 >> 0-edge xhci_hcd >> 134: 0 0 0 0 accel_3d-dev0 >> accel_3d_consumer0 >> 136: 0 0 0 0 als-dev1 >> als_consumer1 >> 142: 0 0 0 0 magn_3d-dev3 >> magn_3d_consumer3 >> 150: 49 0 0 0 PCI-MSI-0000:00:16.0 >> 0-edge mei_me >> 157: 5552 319 0 0 PCI-MSI-0000:00:02.0 >> 0-edge i915 >> NMI: 1 0 3 3 Non-maskable >> interrupts >> LOC: 25207 348 30717 29537 Local timer interrupts >> SPU: 0 0 0 0 Spurious interrupts >> PMI: 1 0 3 3 Performance monitoring >> interrupts >> IWI: 2162 11 119 207 IRQ work interrupts >> RTR: 2 0 0 0 APIC ICR read retries >> RES: 2902 326 3213 2405 Rescheduling >> interrupts >> CAL: 46790 5794 43018 41150 Function call >> interrupts >> TLB: 273 0 258 149 TLB shootdowns >> TRM: 0 0 0 0 Thermal event >> interrupts >> THR: 0 0 0 0 Threshold APIC >> interrupts >> DFR: 0 0 0 0 Deferred Error APIC >> interrupts >> MCE: 0 0 0 0 Machine check >> exceptions >> MCP: 1 2 2 2 Machine check polls >> ERR: 2 >> MIS: 0 >> PIN: 0 0 0 0 Posted-interrupt >> notification event >> NPI: 0 0 0 0 Nested posted- >> interrupt event >> PIW: 0 0 0 0 Posted-interrupt >> wakeup event >> >> > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE only >> > (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still >> > causes the issue. That might help pointing the finger at either a >> > generic IRQ pipelining issue, or something related to the tick device >> instead. >> > >> >> I will try this today and respond as soon as I get the results. >> Regards, >> Jim >> > > I can confirm that the system hangs with "CONFIG_IRQ_PIPELINE=y" and "CONFIG_DOVETAIL is not set". > This is identical to the original failure. > Ok, so this points the finger at irq pipelining. Something may be missing in some irqchip driver to support this. -- Philippe. ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. 2024-03-20 21:36 ` Philippe Gerum @ 2024-03-21 5:56 ` Chen, Hongzhan 0 siblings, 0 replies; 10+ messages in thread From: Chen, Hongzhan @ 2024-03-21 5:56 UTC (permalink / raw) To: Philippe Gerum, Rosenow, James; +Cc: xenomai@lists.linux.dev Hi James Could you share your complete config after enable CONFIG_IRQ_PIPELINE? I will try to reproduce on my side. I met similar thing before because of kernel configuration issue. Regards Hongzhan Chen >-----Original Message----- >From: Philippe Gerum <rpm@xenomai.org> >Sent: Thursday, March 21, 2024 5:36 AM >To: Rosenow, James <jim.rosenow@mts.com> >Cc: xenomai@lists.linux.dev >Subject: Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, >7, and 6 work. > > >"Rosenow, James" <jim.rosenow@mts.com> writes: > >>> > "Rosenow, James" <jim.rosenow@mts.com> writes: >>> > >>> > > We have a COM express carrier with an on-board SSD. The SSD has >>> > > Ubuntu 22.04 installed on it, on top of that, I've installed the >>> > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel >>> > > from the linux-dovetail repository . Given that our board supports >>> > > COM Express, we can easily swap between various processors and have >>> > > been doing so looking for the best price/performance ratio and of >>> > > course, low latency. >>> > > >>> > > I have two modules that I am currently testing, an i3-11100HE and an >>> > > i7- >>> > 9850HL. Since the SSD is on the base board, I'm running the exact >>> > same version of Ubuntu, kernel and root file system regardless of the >>> > installed processor board. >>> > > >>> > > The gen9 processor boots and runs the Xenomai kernel without any >>> > > trouble at all. I'm having SMI issues but that is another subject. >>> > > When I put the gen11 board on the carrier, it hangs at 'loading >>> > > initial ramdisk .'. It is noteworthy that if I interrupt grub and >>> > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen >>> > > 9 boot and run fine. I've verified the Xenomai kernel runs fine on >>> > > generation 7 and 6 CPU's as well. >>> > > >>> > > I've searched the web for possible reasons why Linux would hang at >>> > > that >>> > point; for Linux in general, it is a common thing and I've tried many, >>> > many of the solutions that others have found to work, all to no avail. >>> > I do not know how to get additional information on what is hanging as >>> > there is really no context from which to proceed. >>> > > >>> > > I'm looking for suggestions on next steps to root cause this hang. >>> > > >>> > > Thanks and regards, >>> > > Jim >>> > >>> > Could you boot the machine with the stock kernel that works, then >>> > paste the output of /proc/interrupts? This may help finding out which >>> > IRQ controller might be involved in the issue. >>> > >>> >>> Hello Philippe, >>> Here is the output of /proc/interrupts from the stock 6.5.0 kernel. >>> >>> $ cat /proc/interrupts >>> CPU0 CPU1 CPU2 CPU3 >>> 0: 15 0 0 0 IO-APIC 2-edge >>> timer >>> 3: 0 0 0 2 IO-APIC 3-edge >>> 4: 0 0 0 2 IO-APIC 4-edge >>> 8: 0 0 0 0 IO-APIC 8-edge >>> rtc0 >>> 9: 611 154 0 0 IO-APIC 9-fasteoi >>> acpi >>> 14: 0 0 0 0 IO-APIC 14-fasteoi >>> INT34C6:00 >>> 16: 699 0 0 0 IO-APIC 16-fasteoi >>> i801_smbus, intel_ish_ipc >>> 18: 0 0 0 0 IO-APIC 18-fasteoi >>> i2c_designware.0 >>> 27: 0 0 0 0 IO-APIC 27-fasteoi >>> idma64.1, i2c_designware.1 >>> 29: 0 0 0 0 IO-APIC 29-fasteoi >>> idma64.2, i2c_designware.2 >>> 30: 0 0 0 0 IO-APIC 30-fasteoi >>> idma64.3, i2c_designware.3 >>> 31: 0 0 0 0 IO-APIC 31-fasteoi >>> idma64.4, i2c_designware.4 >>> 32: 0 0 0 0 IO-APIC 32-fasteoi >>> idma64.5, i2c_designware.5 >>> 126: 0 0 0 0 PCI-MSI-0000:00:0d.0 >>> 0-edge xhci_hcd >>> 127: 0 0 0 21855 PCI-MSI-0000:00:17.0 >>> 0-edge ahci[0000:00:17.0] >>> 128: 0 0 1 0 PCI-MSIX-0000:04:00.0 >>> 0-edge enp4s0 >>> 129: 371 0 0 35 PCI-MSIX-0000:04:00.0 >>> 1-edge enp4s0-TxRx-0 >>> 130: 36 0 0 310 PCI-MSIX-0000:04:00.0 >>> 2-edge enp4s0-TxRx-1 >>> 131: 0 36 334 0 PCI-MSIX-0000:04:00.0 >>> 3-edge enp4s0-TxRx-2 >>> 132: 0 0 285 0 PCI-MSIX-0000:04:00.0 >>> 4-edge enp4s0-TxRx-3 >>> 133: 1681 74 0 0 PCI-MSI-0000:00:14.0 >>> 0-edge xhci_hcd >>> 134: 0 0 0 0 accel_3d-dev0 >>> accel_3d_consumer0 >>> 136: 0 0 0 0 als-dev1 >>> als_consumer1 >>> 142: 0 0 0 0 magn_3d-dev3 >>> magn_3d_consumer3 >>> 150: 49 0 0 0 PCI-MSI-0000:00:16.0 >>> 0-edge mei_me >>> 157: 5552 319 0 0 PCI-MSI-0000:00:02.0 >>> 0-edge i915 >>> NMI: 1 0 3 3 Non-maskable >>> interrupts >>> LOC: 25207 348 30717 29537 Local timer interrupts >>> SPU: 0 0 0 0 Spurious interrupts >>> PMI: 1 0 3 3 Performance monitoring >>> interrupts >>> IWI: 2162 11 119 207 IRQ work interrupts >>> RTR: 2 0 0 0 APIC ICR read retries >>> RES: 2902 326 3213 2405 Rescheduling >>> interrupts >>> CAL: 46790 5794 43018 41150 Function call >>> interrupts >>> TLB: 273 0 258 149 TLB shootdowns >>> TRM: 0 0 0 0 Thermal event >>> interrupts >>> THR: 0 0 0 0 Threshold APIC >>> interrupts >>> DFR: 0 0 0 0 Deferred Error APIC >>> interrupts >>> MCE: 0 0 0 0 Machine check >>> exceptions >>> MCP: 1 2 2 2 Machine check polls >>> ERR: 2 >>> MIS: 0 >>> PIN: 0 0 0 0 Posted-interrupt >>> notification event >>> NPI: 0 0 0 0 Nested posted- >>> interrupt event >>> PIW: 0 0 0 0 Posted-interrupt >>> wakeup event >>> >>> > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE >only >>> > (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still >>> > causes the issue. That might help pointing the finger at either a >>> > generic IRQ pipelining issue, or something related to the tick device >>> instead. >>> > >>> >>> I will try this today and respond as soon as I get the results. >>> Regards, >>> Jim >>> >> >> I can confirm that the system hangs with "CONFIG_IRQ_PIPELINE=y" and >"CONFIG_DOVETAIL is not set". >> This is identical to the original failure. >> > >Ok, so this points the finger at irq pipelining. Something may be >missing in some irqchip driver to support this. > >-- >Philippe. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work. 2024-03-13 16:11 ` Rosenow, James 2024-03-13 18:32 ` Rosenow, James @ 2024-03-20 21:25 ` Philippe Gerum 1 sibling, 0 replies; 10+ messages in thread From: Philippe Gerum @ 2024-03-20 21:25 UTC (permalink / raw) To: Rosenow, James; +Cc: xenomai@lists.linux.dev "Rosenow, James" <jim.rosenow@mts.com> writes: >> "Rosenow, James" <jim.rosenow@mts.com> writes: >> >> > We have a COM express carrier with an on-board SSD. The SSD has >> > Ubuntu 22.04 installed on it, on top of that, I've installed the >> > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel >> > from the linux-dovetail repository . Given that our board supports >> > COM Express, we can easily swap between various processors and have >> > been doing so looking for the best price/performance ratio and of >> > course, low latency. >> > >> > I have two modules that I am currently testing, an i3-11100HE and an i7- >> 9850HL. Since the SSD is on the base board, I'm running the exact same >> version of Ubuntu, kernel and root file system regardless of the installed >> processor board. >> > >> > The gen9 processor boots and runs the Xenomai kernel without any >> > trouble at all. I'm having SMI issues but that is another subject. >> > When I put the gen11 board on the carrier, it hangs at 'loading >> > initial ramdisk .'. It is noteworthy that if I interrupt grub and >> > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen 9 >> > boot and run fine. I've verified the Xenomai kernel runs fine on >> > generation 7 and 6 CPU's as well. >> > >> > I've searched the web for possible reasons why Linux would hang at that >> point; for Linux in general, it is a common thing and I've tried many, >> many of the solutions that others have found to work, all to no avail. I >> do not know how to get additional information on what is hanging as there >> is really no context from which to proceed. >> > >> > I'm looking for suggestions on next steps to root cause this hang. >> > >> > Thanks and regards, >> > Jim >> >> Could you boot the machine with the stock kernel that works, then paste >> the output of /proc/interrupts? This may help finding out which IRQ >> controller might be involved in the issue. >> > > Hello Philippe, > Here is the output of /proc/interrupts from the stock 6.5.0 kernel. > > $ cat /proc/interrupts > CPU0 CPU1 CPU2 CPU3 > 134: 0 0 0 0 accel_3d-dev0 accel_3d_consumer0 > 136: 0 0 0 0 als-dev1 als_consumer1 > 142: 0 0 0 0 magn_3d-dev3 magn_3d_consumer3 Could you disable these features before retrying? CONFIG_HID_SENSOR_MAGNETOMETER_3D CONFIG_HID_SENSOR_ACCEL_3D CONFIG_HID_SENSOR_ALS? (maybe.. there is a truckload of light sensors). If the issue persists, disabling all IIO drivers could be a sensible option to confirm that there may be an issue with some irqchip those drivers depend on when it comes to dealing with Dovetail. -- Philippe. ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-03-21 5:57 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <DM6PR04MB538655375C57CF07EDDE5089E2242@DM6PR04MB5386.namprd04.prod.outlook.com>
[not found] ` <DM6PR04MB538673AA2E912100093BBA7CE2242@DM6PR04MB5386.namprd04.prod.outlook.com>
2024-03-11 16:52 ` Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work Rosenow, James
2024-03-11 17:26 ` Philippe Gerum
2024-03-13 16:11 ` Rosenow, James
2024-03-13 18:32 ` Rosenow, James
2024-03-19 15:19 ` Rosenow, James
2024-03-19 15:40 ` Philippe Gerum
2024-03-20 7:44 ` Chen, Hongzhan
2024-03-20 21:36 ` Philippe Gerum
2024-03-21 5:56 ` Chen, Hongzhan
2024-03-20 21:25 ` Philippe Gerum
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.