kvm-78 - kernel panic after using system_reset except when using -no-kvm-irqchip

kvm-78 - kernel panic after using system_reset except when using -no-kvm-irqchip

[Charles Duffy]

This happens every other use of system_reset -- ie. resetting the system 
  using system_reset while in this panic'd state results in correct 
functionality. Following the suggestion and booting the guest with 
noapic results in other IRQ-related errors.

The guest kernel is CentOS 5 2.6.18-53.el5 on x86_64. The host is 
running 2.6.27.5.

Kernel command line: ro root=/dev/VolGroup01/LogVol00
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
Memory: 746368k/76936k available (2434k kernel code, 21180k reserved, 
1235k data, 192k init)
Calibrating delay using timer specific routine.. 4793.50 BogoMIPS 
(lpj=2396750)
Security Framework v1.0.0 initialized
SELinux:  Initializing.
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 256
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 2048K
SMP alternatives: switching to UP code
ACPI: Core revision 20060707
irq 25, desc: ffffffff803afc80, depth: 1, count: 0, unhandled: 0
->handle_irq():  ffffffff800b54e3, handle_bad_irq+0x0/0x1f6
->chip(): ffffffff802ea700, 0xffffffff802ea700
->action(): 0000000000000000
   IRQ_DISABLED set
unexpected IRQ trap at vector 19
..MP-BIOS bug: 8254 timer not connected to IO-APIC
timer doesn't work through the IO-APIC - disabling NMI Watchdog!
Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the 
'noapic' kernel parameter

Re: kvm-78 - kernel panic after using system_reset except when using -no-kvm-irqchip

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2353 bytes --]

Charles Duffy wrote:
> This happens every other use of system_reset -- ie. resetting the system
>  using system_reset while in this panic'd state results in correct
> functionality. Following the suggestion and booting the guest with
> noapic results in other IRQ-related errors.
> 
> The guest kernel is CentOS 5 2.6.18-53.el5 on x86_64. The host is
> running 2.6.27.5.
> 
> Kernel command line: ro root=/dev/VolGroup01/LogVol00
> Initializing CPU#0
> PID hash table entries: 4096 (order: 12, 32768 bytes)
> Console: colour VGA+ 80x25
> Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
> Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
> Checking aperture...
> Memory: 746368k/76936k available (2434k kernel code, 21180k reserved,
> 1235k data, 192k init)
> Calibrating delay using timer specific routine.. 4793.50 BogoMIPS
> (lpj=2396750)
> Security Framework v1.0.0 initialized
> SELinux:  Initializing.
> selinux_register_security:  Registering secondary module capability
> Capability LSM initialized as secondary
> Mount-cache hash table entries: 256
> CPU: L1 I cache: 32K, L1 D cache: 32K
> CPU: L2 cache: 2048K
> SMP alternatives: switching to UP code
> ACPI: Core revision 20060707
> irq 25, desc: ffffffff803afc80, depth: 1, count: 0, unhandled: 0
> ->handle_irq():  ffffffff800b54e3, handle_bad_irq+0x0/0x1f6
> ->chip(): ffffffff802ea700, 0xffffffff802ea700
> ->action(): 0000000000000000
>   IRQ_DISABLED set
> unexpected IRQ trap at vector 19
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> timer doesn't work through the IO-APIC - disabling NMI Watchdog!
> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the
> 'noapic' kernel parameter

Still too early for me, so I didn't get yet if you can trigger this
guest panic reliably or only sporadically (like I can). In the former
case I would be very interested in the how!

So far my theory on this is that the guest happens to loose too many PIC
timer ticks during the test window where it checks the PIC IRQ routing
(it does so with 10 ticks, out of which at least 4 must be delivered).
The theory also says that this is hard to avoid, maybe just less likely
with Gleb's timer drift compensation patches.

Jan

PS: Booting with noapic leaves many real boxes useless as well.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: kvm-78 - kernel panic after using system_reset except when using -no-kvm-irqchip

[Charles Duffy]

Jan Kiszka wrote:
> Charles Duffy wrote:
>> irq 25, desc: ffffffff803afc80, depth: 1, count: 0, unhandled: 0
>> ->handle_irq():  ffffffff800b54e3, handle_bad_irq+0x0/0x1f6
>> ->chip(): ffffffff802ea700, 0xffffffff802ea700
>> ->action(): 0000000000000000
>>   IRQ_DISABLED set
>> unexpected IRQ trap at vector 19
>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
>> timer doesn't work through the IO-APIC - disabling NMI Watchdog!
>> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the
>> 'noapic' kernel parameter
> 
> Still too early for me, so I didn't get yet if you can trigger this
> guest panic reliably or only sporadically (like I can). In the former
> case I would be very interested in the how!

Reproduced 100% of the time when using system_reset at the qemu monitor 
console coming from a non-panic'd state. (Thus, it happens only every 
other time doing a series of boots and system_reset monitor commands).

> So far my theory on this is that the guest happens to loose too many PIC
> timer ticks during the test window where it checks the PIC IRQ routing
> (it does so with 10 ticks, out of which at least 4 must be delivered).
> The theory also says that this is hard to avoid, maybe just less likely
> with Gleb's timer drift compensation patches.
> 
> Jan
> 
> PS: Booting with noapic leaves many real boxes useless as well.
> 

Re: kvm-78 - kernel panic after using system_reset except when using -no-kvm-irqchip

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1265 bytes --]

Charles Duffy wrote:
> Jan Kiszka wrote:
>> Charles Duffy wrote:
>>> irq 25, desc: ffffffff803afc80, depth: 1, count: 0, unhandled: 0
>>> ->handle_irq():  ffffffff800b54e3, handle_bad_irq+0x0/0x1f6
>>> ->chip(): ffffffff802ea700, 0xffffffff802ea700
>>> ->action(): 0000000000000000
>>>   IRQ_DISABLED set
>>> unexpected IRQ trap at vector 19
>>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
>>> timer doesn't work through the IO-APIC - disabling NMI Watchdog!
>>> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the
>>> 'noapic' kernel parameter
>>
>> Still too early for me, so I didn't get yet if you can trigger this
>> guest panic reliably or only sporadically (like I can). In the former
>> case I would be very interested in the how!
> 
> Reproduced 100% of the time when using system_reset at the qemu monitor
> console coming from a non-panic'd state. (Thus, it happens only every
> other time doing a series of boots and system_reset monitor commands).

Mmmh, /that/ sounds like some (re-)initialization issue of the
virtualized hardware. OK, once that kernel boots again on latest kvm.git
(currently broken due to a different issue), I will try to reproduce it.

Thanks,
Jan

PS: Please keep CCs.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: kvm-78 - kernel panic after using system_reset except when using -no-kvm-irqchip

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

Jan Kiszka wrote:
> Charles Duffy wrote:
>> Jan Kiszka wrote:
>>> Charles Duffy wrote:
>>>> irq 25, desc: ffffffff803afc80, depth: 1, count: 0, unhandled: 0
>>>> ->handle_irq():  ffffffff800b54e3, handle_bad_irq+0x0/0x1f6
>>>> ->chip(): ffffffff802ea700, 0xffffffff802ea700
>>>> ->action(): 0000000000000000
>>>>   IRQ_DISABLED set
>>>> unexpected IRQ trap at vector 19
>>>> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
>>>> timer doesn't work through the IO-APIC - disabling NMI Watchdog!
>>>> Kernel panic - not syncing: IO-APIC + timer doesn't work! Try using the
>>>> 'noapic' kernel parameter
>>> Still too early for me, so I didn't get yet if you can trigger this
>>> guest panic reliably or only sporadically (like I can). In the former
>>> case I would be very interested in the how!
>> Reproduced 100% of the time when using system_reset at the qemu monitor
>> console coming from a non-panic'd state. (Thus, it happens only every
>> other time doing a series of boots and system_reset monitor commands).
> 
> Mmmh, /that/ sounds like some (re-)initialization issue of the
> virtualized hardware. OK, once that kernel boots again on latest kvm.git
> (currently broken due to a different issue), I will try to reproduce it.

OK, I can reproduce now. It's an in-kernel irqchip, -no-kvm-irqchip
doesn't suffer. Will see if I can find the reason.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]