Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Avi Kivity]

On 06/20/2011 06:38 PM, Daniel P. Berrange wrote:
> On Mon, Jun 20, 2011 at 06:31:23PM +0300, Avi Kivity wrote:
> >  On 06/20/2011 04:38 PM, Daniel Gollub wrote:
> >  >Introduce panic hypercall to enable the crashing guest to notify the
> >  >host. This enables the host to run some actions as soon a guest
> >  >crashed (kernel panic).
> >  >
> >  >This patch series introduces the panic hypercall at the host end.
> >  >As well as the hypercall for KVM paravirtuliazed Linux guests, by
> >  >registering the hypercall to the panic_notifier_list.
> >  >
> >  >The basic idea is to create KVM crashdump automatically as soon the
> >  >guest paniced and power-cycle the VM (e.g. libvirt<on_crash />).
> >
> >  This would be more easily done via a "panic device" (I/O port or
> >  memory-mapped address) that the guest hits.  It would be intercepted
> >  by qemu without any new code in kvm.\
> >
> >  However, I'm not sure I see the gain.  Most enterprisey guests
> >  already contain in-guest crash dumpers which provide more
> >  information than a qemu memory dump could, since they know exact
> >  load addresses etc. and are integrated with crash analysis tools.
> >  What do you have in mind?
>
> Well libvirt can capture a "core" file by doing 'virsh dump $GUESTNAME'.
> This actually uses the QEMU monitor migration command to capture the
> entire of QEMU memory. The 'crash' command line tool actually knows
> how to analyse this data format as it would a normal kernel crashdump.

Interesting.

> I think having a way for a guest OS to notify the host that is has
> crashed would be useful. libvirt could automatically do a crash
> dump of the QEMU memory, or at least pause the guest CPUs and notify
> the management app of the crash, which can then decide what todo.
> You can also use tools like 'virt-dmesg' which uses libvirt to peek
> into guest memory to extract the most recent kernel dmesg logs (even
> if the guest OS itself is crashed&  didn't manage to send them out
> via netconsole or something else).

I agree.  But let's do this via a device, this way kvm need not be changed.

Do ILO cards / IPMI support something like this?  We could follow their 
lead in that case.

> This series does need to introduce a QMP event notification upon
> crash, so that the crash notification can be propagated to mgmt
> layers above QEMU.

Yes.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Jan Kiszka]

On 2011-06-20 17:45, Avi Kivity wrote:
>> This series does need to introduce a QMP event notification upon
>> crash, so that the crash notification can be propagated to mgmt
>> layers above QEMU.
> 
> Yes.

I think the best way to deal with that is to stop the VM on guest panic.
There is already WIP to signal stop reasons via QMP. Maybe we need to
differentiate between hypervisor and guest triggered panics
(VMSTOP_GUEST_PANIC?), but the rest should come for free.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Daniel Gollub]

[-- Attachment #1: Type: Text/Plain, Size: 3954 bytes --]

On Monday, June 20, 2011 05:45:36 pm Avi Kivity wrote:
> > >  However, I'm not sure I see the gain.  Most enterprisey guests
> > >  already contain in-guest crash dumpers which provide more
> > >  information than a qemu memory dump could, since they know exact
> > >  load addresses etc. and are integrated with crash analysis tools.
> > >  What do you have in mind?

Right kexec/kdump works perfectly already inside the guest. But:

 - in the "field" a lot of people still manage to setup VM guest without
   kexec/kdump properly setup (even though most enterprisey distribution try
   hard to setup this up out-of-the-box .. still people manage to not have
   kexec/kdump loaded once they run into a crash).

 - you don't have to reserve disk space for a crashdump for each guest
   e.g. if you run 4 guests with 60 GB of memory each you would loose
   somehow 4*60 GB space ... just for the (rare) case that each of those
   guest could write a crashdump, uncompressed ...

 - legacy distribution - no or buggy kexec

 - maybe writing a crashdump+reboot with QEMU/libvirt is faster then
   with in-guest kexec/kdump? (haven't tested yet)

 - single place on the VM-host to collect coredumps


> > 
> > Well libvirt can capture a "core" file by doing 'virsh dump $GUESTNAME'.
> > This actually uses the QEMU monitor migration command to capture the
> > entire of QEMU memory. The 'crash' command line tool actually knows
> > how to analyse this data format as it would a normal kernel crashdump.
> 
> Interesting.

Right. I'm using the kvmdump support of the crash utility now and then ... it 
could be more often. But unfortunately the people who run KVM in a productive 
environment with some strict service-level-agreement often just reboot, due to 
time pressure, or run out of disk space in the guest, or just forgot that they 
got told to do always "virsh dump" on a freeze or crash.


> 
> > I think having a way for a guest OS to notify the host that is has
> > crashed would be useful. libvirt could automatically do a crash
> > dump of the QEMU memory, or at least pause the guest CPUs and notify
> > the management app of the crash, which can then decide what todo.
> > You can also use tools like 'virt-dmesg' which uses libvirt to peek
> > into guest memory to extract the most recent kernel dmesg logs (even
> > if the guest OS itself is crashed&  didn't manage to send them out
> > via netconsole or something else).
> 
> I agree.  But let's do this via a device, this way kvm need not be changed.

Is a device reliable enough if the guest kernel crashes?
Do you mean something like a hardware watchdog?

> 
> Do ILO cards / IPMI support something like this?  We could follow their 
> lead in that case.

The only two things which came to my mind are:

 * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires
   in-guest kexec/kdump
 * Hardware-Watchdog (also available in qemu/libvirt)


lguest and xen have something similar. They also have an hypercall which get 
called by a function registered in the panic_notifier_list. Not quite sure if 
you want to follow their lead.

Something I forgot to mention: This panic hypercall could also sit within an 
external kernel module ... to support (legacy) distribution.

> 
> > This series does need to introduce a QMP event notification upon
> > crash, so that the crash notification can be propagated to mgmt
> > layers above QEMU.
> 
> Yes.

Already done. I posted the QEMU relevant changes as a separated series to the 
KVM list ... since the initial implementation is KVM specific (KVM hypercall)

Best Regards,
Daniel

-- 
Daniel Gollub
Linux Consultant & Developer
Tel.: +49-160 47 73 970 
Mail: gollub@b1-systems.de

B1 Systems GmbH
Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de
GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Avi Kivity]

On 06/20/2011 07:26 PM, Daniel Gollub wrote:
> >
> >  I agree.  But let's do this via a device, this way kvm need not be changed.
>
> Is a device reliable enough if the guest kernel crashes?
> Do you mean something like a hardware watchdog?

I'm proposing a 1:1 equivalent.  Instead of issuing a hypercall that 
tells the host about the panic, write to an I/O port that tells the host 
about the panic.

> >
> >  Do ILO cards / IPMI support something like this?  We could follow their
> >  lead in that case.
>
> The only two things which came to my mind are:
>
>   * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires
>     in-guest kexec/kdump
>   * Hardware-Watchdog (also available in qemu/libvirt)

A watchdog has the advantage that is also detects lockups.

In fact you could implement the panic device via the existing 
watchdogs.  Simply program the timer for the minimum interval and 
*don't* service the interrupt.  This would work for non-virt setups as 
well as another way to issue a reset.

> lguest and xen have something similar. They also have an hypercall which get
> called by a function registered in the panic_notifier_list. Not quite sure if
> you want to follow their lead.

We could do the same, except s/hypercall/writel/.

> Something I forgot to mention: This panic hypercall could also sit within an
> external kernel module ... to support (legacy) distribution.

Yes.

> >
> >  >  This series does need to introduce a QMP event notification upon
> >  >  crash, so that the crash notification can be propagated to mgmt
> >  >  layers above QEMU.
> >
> >  Yes.
>
> Already done. I posted the QEMU relevant changes as a separated series to the
> KVM list ... since the initial implementation is KVM specific (KVM hypercall)

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Jan Kiszka]

On 2011-06-20 18:34, Avi Kivity wrote:
>> >
>> >  Do ILO cards / IPMI support something like this?  We could follow
>> their
>> >  lead in that case.
>>
>> The only two things which came to my mind are:
>>
>>   * NMI (aka. ipmitool diag) - already available in qemu/kvm - but
>> requires
>>     in-guest kexec/kdump
>>   * Hardware-Watchdog (also available in qemu/libvirt)
> 
> A watchdog has the advantage that is also detects lockups.
> 
> In fact you could implement the panic device via the existing
> watchdogs.  Simply program the timer for the minimum interval and
> *don't* service the interrupt.  This would work for non-virt setups as
> well as another way to issue a reset.

If you manage to bring down the other guest CPUs fast enough. Otherwise,
they may corrupt your crashdump before the host had a chance to collect
all pieces. Synchronous signaling to the hypervisor is a bit safer.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Avi Kivity]

On 06/20/2011 08:13 PM, Jan Kiszka wrote:
> >  A watchdog has the advantage that is also detects lockups.
> >
> >  In fact you could implement the panic device via the existing
> >  watchdogs.  Simply program the timer for the minimum interval and
> >  *don't* service the interrupt.  This would work for non-virt setups as
> >  well as another way to issue a reset.
>
> If you manage to bring down the other guest CPUs fast enough. Otherwise,
> they may corrupt your crashdump before the host had a chance to collect
> all pieces. Synchronous signaling to the hypervisor is a bit safer.

You could NMI-IPI them.  But I agree a synchronous signal is better 
(note it's not race-free itself).

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Gleb Natapov]

On Mon, Jun 20, 2011 at 08:23:20PM +0300, Avi Kivity wrote:
> On 06/20/2011 08:13 PM, Jan Kiszka wrote:
> >>  A watchdog has the advantage that is also detects lockups.
> >>
> >>  In fact you could implement the panic device via the existing
> >>  watchdogs.  Simply program the timer for the minimum interval and
> >>  *don't* service the interrupt.  This would work for non-virt setups as
> >>  well as another way to issue a reset.
> >
> >If you manage to bring down the other guest CPUs fast enough. Otherwise,
> >they may corrupt your crashdump before the host had a chance to collect
> >all pieces. Synchronous signaling to the hypervisor is a bit safer.
> 
> You could NMI-IPI them.  But I agree a synchronous signal is better
> (note it's not race-free itself).
> 
But kexec/kdump has exactly same race, so this is at least not worse that
alternative.

--
			Gleb.

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Gleb Natapov]

On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote:
> >The only two things which came to my mind are:
> >
> >  * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires
> >    in-guest kexec/kdump
> >  * Hardware-Watchdog (also available in qemu/libvirt)
> 
> A watchdog has the advantage that is also detects lockups.
> 
And has disadvantage that all time base heuristics are bite us in the
end.

--
			Gleb.

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Avi Kivity]

On 06/21/2011 09:02 AM, Gleb Natapov wrote:
> On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote:
> >  >The only two things which came to my mind are:
> >  >
> >  >   * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires
> >  >     in-guest kexec/kdump
> >  >   * Hardware-Watchdog (also available in qemu/libvirt)
> >
> >  A watchdog has the advantage that is also detects lockups.
> >
> And has disadvantage that all time base heuristics are bite us in the
> end.

The perf-based watchdog counts clocks-not-halted, not time, so it is 
safe from time issues.  We could make the hardware watchdog cheat in the 
same way.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Gleb Natapov]

On Tue, Jun 21, 2011 at 11:09:21AM +0300, Avi Kivity wrote:
> On 06/21/2011 09:02 AM, Gleb Natapov wrote:
> >On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote:
> >>  >The only two things which came to my mind are:
> >>  >
> >>  >   * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires
> >>  >     in-guest kexec/kdump
> >>  >   * Hardware-Watchdog (also available in qemu/libvirt)
> >>
> >>  A watchdog has the advantage that is also detects lockups.
> >>
> >And has disadvantage that all time base heuristics are bite us in the
> >end.
> 
> The perf-based watchdog counts clocks-not-halted, not time, so it is
> safe from time issues.
So it counts only instruction that guest actually executed? That's
perfect then. How much overhead it has in a guest though?

>                         We could make the hardware watchdog cheat in
> the same way.
> 
Something like steal time, but for watchdog. But this become complicated fast.
Watchdog emulation will have to move into kernel for starter.

--
			Gleb.

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Avi Kivity]

On 06/21/2011 11:41 AM, Gleb Natapov wrote:
> On Tue, Jun 21, 2011 at 11:09:21AM +0300, Avi Kivity wrote:
> >  On 06/21/2011 09:02 AM, Gleb Natapov wrote:
> >  >On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote:
> >  >>   >The only two things which came to my mind are:
> >  >>   >
> >  >>   >    * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires
> >  >>   >      in-guest kexec/kdump
> >  >>   >    * Hardware-Watchdog (also available in qemu/libvirt)
> >  >>
> >  >>   A watchdog has the advantage that is also detects lockups.
> >  >>
> >  >And has disadvantage that all time base heuristics are bite us in the
> >  >end.
> >
> >  The perf-based watchdog counts clocks-not-halted, not time, so it is
> >  safe from time issues.
> So it counts only instruction that guest actually executed? That's
> perfect then. How much overhead it has in a guest though?

Should be pretty low, especially if the guest is idle.  There were some 
negative reports about the pmu From David Ahern, so it needs to be verified.


> >                          We could make the hardware watchdog cheat in
> >  the same way.
> >
> Something like steal time, but for watchdog. But this become complicated fast.
> Watchdog emulation will have to move into kernel for starter.

Why?  You can use a performance counter from userspace.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Gleb Natapov]

On Tue, Jun 21, 2011 at 11:56:26AM +0300, Avi Kivity wrote:
> On 06/21/2011 11:41 AM, Gleb Natapov wrote:
> >On Tue, Jun 21, 2011 at 11:09:21AM +0300, Avi Kivity wrote:
> >>  On 06/21/2011 09:02 AM, Gleb Natapov wrote:
> >>  >On Mon, Jun 20, 2011 at 07:34:57PM +0300, Avi Kivity wrote:
> >>  >>   >The only two things which came to my mind are:
> >>  >>   >
> >>  >>   >    * NMI (aka. ipmitool diag) - already available in qemu/kvm - but requires
> >>  >>   >      in-guest kexec/kdump
> >>  >>   >    * Hardware-Watchdog (also available in qemu/libvirt)
> >>  >>
> >>  >>   A watchdog has the advantage that is also detects lockups.
> >>  >>
> >>  >And has disadvantage that all time base heuristics are bite us in the
> >>  >end.
> >>
> >>  The perf-based watchdog counts clocks-not-halted, not time, so it is
> >>  safe from time issues.
> >So it counts only instruction that guest actually executed? That's
> >perfect then. How much overhead it has in a guest though?
> 
> Should be pretty low, especially if the guest is idle.  There were
> some negative reports about the pmu From David Ahern, so it needs to
> be verified.
But it will be running not only when the guest is idle.

> 
> 
> >>                          We could make the hardware watchdog cheat in
> >>  the same way.
> >>
> >Something like steal time, but for watchdog. But this become complicated fast.
> >Watchdog emulation will have to move into kernel for starter.
> 
> Why?  You can use a performance counter from userspace.
> 
Heh, haven't thought about such way of implementing watchdog device.
But the same question again: what impact on performance constantly running
guest under perf is? Doesn't running guest under perf involve a lot of
NMIs (and hence vmexists)?

--
			Gleb.

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[Avi Kivity]

On 06/21/2011 12:03 PM, Gleb Natapov wrote:> >>
> >  >Something like steal time, but for watchdog. But this become complicated fast.
> >  >Watchdog emulation will have to move into kernel for starter.
> >
> >  Why?  You can use a performance counter from userspace.
> >
> Heh, haven't thought about such way of implementing watchdog device.
> But the same question again: what impact on performance constantly running
> guest under perf is?

Same answer.

> Doesn't running guest under perf involve a lot of
> NMIs (and hence vmexists)?

The NMI rate is proportional to how you program the counter.  If you 
program it to exit every 2 billion cycles, that's what you'll get.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [PATCH 0/2] Introduce panic hypercall

[shawn che]

HI,

I am using kvm-kmod with version 2.6.36, but I encountered a fatal problem.
In linux run on KVM,I using request(irq,*,*,*) function to request a
interrupt line. However, When I using a irq bigger than 15, it will
return error.
Now I have simulated a PCI device and decide to using IOAPIC to send
interrupts.So I decide assign 20,21 line for my device . but in guest
os linux I could not request these number for my device.
I know PIC  just have 16 interrupt lines.  I am not sure if the KVM
really support APIC, and how to request a irq line more than 15?

Waiting for your help on line.
Thanks..

Shawn.