Re: Hang at "loading initial ramdisk ..." on Generation 11 CPU, Gen 9, 7, and 6 work.

[Philippe Gerum <rpm@xenomai.org>]

"Rosenow, James" <jim.rosenow@mts.com> writes:

>> > "Rosenow, James" <jim.rosenow@mts.com> writes:
>> >
>> > > We have a COM express carrier with an on-board SSD.  The SSD has
>> > > Ubuntu 22.04 installed on it, on top of that, I've installed the
>> > > latest Xenomai (master) alongside the 'v6.5-dovetail-rebase' kernel
>> > > from the linux-dovetail repository .  Given that our board supports
>> > > COM Express, we can easily swap between various processors and have
>> > > been doing so looking for the best price/performance ratio and of
>> > > course, low latency.
>> > >
>> > > I have two modules that I am currently testing, an i3-11100HE and an
>> > > i7-
>> > 9850HL.  Since the SSD is on the base board, I'm running the exact
>> > same version of Ubuntu, kernel and root file system regardless of the
>> > installed processor board.
>> > >
>> > > The gen9 processor boots and runs the Xenomai kernel without any
>> > > trouble at all.  I'm having SMI issues but that is another subject.
>> > > When I put the gen11 board on the carrier, it hangs at 'loading
>> > > initial ramdisk .'.  It is noteworthy that if I interrupt grub and
>> > > select the stock 6.5.x Ubuntu kernel on either, both gen 11 and gen
>> > > 9 boot and run fine.  I've verified the Xenomai kernel runs fine on
>> > > generation 7 and 6 CPU's as well.
>> > >
>> > > I've searched the web for possible reasons why Linux would hang at
>> > > that
>> > point; for Linux in general, it is a common thing and I've tried many,
>> > many of the solutions that others have found to work, all to no avail.
>> > I do not know how to get additional information on what is hanging as
>> > there is really no context from which to proceed.
>> > >
>> > > I'm looking for suggestions on next steps to root cause this hang.
>> > >
>> > > Thanks and regards,
>> > > Jim
>> >
>> > Could you boot the machine with the stock kernel that works, then
>> > paste the output of /proc/interrupts? This may help finding out which
>> > IRQ controller might be involved in the issue.
>> >
>> 
>> Hello Philippe,
>> Here is the output of /proc/interrupts from the stock 6.5.0 kernel.
>> 
>> $ cat /proc/interrupts
>>             CPU0       CPU1       CPU2       CPU3
>>    0:         15          0          0          0   IO-APIC    2-edge
>> timer
>>    3:          0          0          0          2   IO-APIC    3-edge
>>    4:          0          0          0          2   IO-APIC    4-edge
>>    8:          0          0          0          0   IO-APIC    8-edge
>> rtc0
>>    9:        611        154          0          0   IO-APIC    9-fasteoi
>> acpi
>>   14:          0          0          0          0   IO-APIC   14-fasteoi
>> INT34C6:00
>>   16:        699          0          0          0   IO-APIC   16-fasteoi
>> i801_smbus, intel_ish_ipc
>>   18:          0          0          0          0   IO-APIC   18-fasteoi
>> i2c_designware.0
>>   27:          0          0          0          0   IO-APIC   27-fasteoi
>> idma64.1, i2c_designware.1
>>   29:          0          0          0          0   IO-APIC   29-fasteoi
>> idma64.2, i2c_designware.2
>>   30:          0          0          0          0   IO-APIC   30-fasteoi
>> idma64.3, i2c_designware.3
>>   31:          0          0          0          0   IO-APIC   31-fasteoi
>> idma64.4, i2c_designware.4
>>   32:          0          0          0          0   IO-APIC   32-fasteoi
>> idma64.5, i2c_designware.5
>>  126:          0          0          0          0  PCI-MSI-0000:00:0d.0
>> 0-edge      xhci_hcd
>>  127:          0          0          0      21855  PCI-MSI-0000:00:17.0
>> 0-edge      ahci[0000:00:17.0]
>>  128:          0          0          1          0  PCI-MSIX-0000:04:00.0
>> 0-edge      enp4s0
>>  129:        371          0          0         35  PCI-MSIX-0000:04:00.0
>> 1-edge      enp4s0-TxRx-0
>>  130:         36          0          0        310  PCI-MSIX-0000:04:00.0
>> 2-edge      enp4s0-TxRx-1
>>  131:          0         36        334          0  PCI-MSIX-0000:04:00.0
>> 3-edge      enp4s0-TxRx-2
>>  132:          0          0        285          0  PCI-MSIX-0000:04:00.0
>> 4-edge      enp4s0-TxRx-3
>>  133:       1681         74          0          0  PCI-MSI-0000:00:14.0
>> 0-edge      xhci_hcd
>>  134:          0          0          0          0  accel_3d-dev0
>> accel_3d_consumer0
>>  136:          0          0          0          0  als-dev1
>> als_consumer1
>>  142:          0          0          0          0  magn_3d-dev3
>> magn_3d_consumer3
>>  150:         49          0          0          0  PCI-MSI-0000:00:16.0
>> 0-edge      mei_me
>>  157:       5552        319          0          0  PCI-MSI-0000:00:02.0
>> 0-edge      i915
>>  NMI:          1          0          3          3   Non-maskable
>> interrupts
>>  LOC:      25207        348      30717      29537   Local timer interrupts
>>  SPU:          0          0          0          0   Spurious interrupts
>>  PMI:          1          0          3          3   Performance monitoring
>> interrupts
>>  IWI:       2162         11        119        207   IRQ work interrupts
>>  RTR:          2          0          0          0   APIC ICR read retries
>>  RES:       2902        326       3213       2405   Rescheduling
>> interrupts
>>  CAL:      46790       5794      43018      41150   Function call
>> interrupts
>>  TLB:        273          0        258        149   TLB shootdowns
>>  TRM:          0          0          0          0   Thermal event
>> interrupts
>>  THR:          0          0          0          0   Threshold APIC
>> interrupts
>>  DFR:          0          0          0          0   Deferred Error APIC
>> interrupts
>>  MCE:          0          0          0          0   Machine check
>> exceptions
>>  MCP:          1          2          2          2   Machine check polls
>>  ERR:          2
>>  MIS:          0
>>  PIN:          0          0          0          0   Posted-interrupt
>> notification event
>>  NPI:          0          0          0          0   Nested posted-
>> interrupt event
>>  PIW:          0          0          0          0   Posted-interrupt
>> wakeup event
>> 
>> > Also, you may want to check whether enabling CONFIG_IRQ_PIPELINE only
>> > (i.e. w/o CONFIG_DOVETAIL and therefore no Xenomai on top) still
>> > causes the issue. That might help pointing the finger at either a
>> > generic IRQ pipelining issue, or something related to the tick device
>> instead.
>> >
>> 
>> I will try this today and respond as soon as I get the results.
>> Regards,
>> Jim
>> 
>
> I can confirm that the system hangs with "CONFIG_IRQ_PIPELINE=y" and "CONFIG_DOVETAIL is not set".
> This is identical to the original failure.
>

Ok, so this points the finger at irq pipelining. Something may be
missing in some irqchip driver to support this.

-- 
Philippe.