Re: Thoughts on VM fence infrastructure

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Felipe Franciosi (felipe@nutanix.com) wrote:
> 
> 
> > On Sep 30, 2019, at 6:59 PM, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
> > 
> > * Felipe Franciosi (felipe@nutanix.com) wrote:
> >> 
> >> 
> >>> On Sep 30, 2019, at 6:11 PM, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
> >>> 
> >>> * Felipe Franciosi (felipe@nutanix.com) wrote:
> >>>> 
> >>>> 
> >>>>> On Sep 30, 2019, at 5:03 PM, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
> >>>>> 
> >>>>> * Felipe Franciosi (felipe@nutanix.com) wrote:
> >>>>>> Hi David,
> >>>>>> 
> >>>>>>> On Sep 30, 2019, at 3:29 PM, Dr. David Alan Gilbert <dgilbert@redhat.com> wrote:
> >>>>>>> 
> >>>>>>> * Felipe Franciosi (felipe@nutanix.com) wrote:
> >>>>>>>> Heyall,
> >>>>>>>> 
> >>>>>>>> We have a use case where a host should self-fence (and all VMs should
> >>>>>>>> die) if it doesn't hear back from a heartbeat within a certain time
> >>>>>>>> period. Lots of ideas were floated around where libvirt could take
> >>>>>>>> care of killing VMs or a separate service could do it. The concern
> >>>>>>>> with those is that various failures could lead to _those_ services
> >>>>>>>> being unavailable and the fencing wouldn't be enforced as it should.
> >>>>>>>> 
> >>>>>>>> Ultimately, it feels like Qemu should be responsible for this
> >>>>>>>> heartbeat and exit (or execute a custom callback) on timeout.
> >>>>>>> 
> >>>>>>> It doesn't feel doing it inside qemu would be any safer;  something
> >>>>>>> outside QEMU can forcibly emit a kill -9 and qemu *will* stop.
> >>>>>> 
> >>>>>> The argument above is that we would have to rely on this external
> >>>>>> service being functional. Consider the case where the host is
> >>>>>> dysfunctional, with this service perhaps crashed and a corrupt
> >>>>>> filesystem preventing it from restarting. The VMs would never die.
> >>>>> 
> >>>>> Yeh that could fail.
> >>>>> 
> >>>>>> It feels like a Qemu timer-driven heartbeat check and calls abort() /
> >>>>>> exit() would be more reliable. Thoughts?
> >>>>> 
> >>>>> OK, yes; perhaps using a timer_create and telling it to send a fatal
> >>>>> signal is pretty solid; it would take the kernel to do that once it's
> >>>>> set.
> >>>> 
> >>>> I'm confused about why the kernel needs to be involved. If this is a
> >>>> timer off the Qemu main loop, it can just check on the heartbeat
> >>>> condition (which should be customisable) and call abort() if that's
> >>>> not satisfied. If you agree on that I'd like to talk about how that
> >>>> check could be made customisable.
> >>> 
> >>> There are times when the main loop can get blocked even though the CPU
> >>> threads can be running and can in some configurations perform IO
> >>> even without the main loop (I think!).
> >> 
> >> Ah, that's a very good point. Indeed, you can perform IO in those
> >> cases specially when using vhost devices.
> >> 
> >>> By setting a timer in the kernel that sends a signal to qemu, the kernel
> >>> will send that signal however broken qemu is.
> >> 
> >> Got you now. That's probably better. Do you reckon a signal is
> >> preferable over SIGEV_THREAD?
> > 
> > Not sure; probably the safest is getting the kernel to SIGKILL it - but
> > that's a complete nightmare to debug - your process just goes *pop*
> > with no apparent reason why.
> > I've not used SIGEV_THREAD - it looks promising though.
> 
> I'm worried that SIGEV_THREAD could be a bit heavyweight (if it fires
> up a new thread each time). On the other hand, as you said, SIGKILL
> makes it harder to debug.
> 
> Also, asking the kernel to defer the SIGKILL (ie. updating the timer)
> needs to come from Qemu itself (eg. a timer in the main loop,
> something we already ruled unsuitable, or a qmp command which
> constitutes an external dependency that we also ruled undesirable).

OK, there's two reasons I think this isn't that bad/is good:
   a) It's an external dependency - but if it fails the result is the
      system fails, rather than the system keeps on running; so I think
      that's the balance you were after; it's the opposite from
      the external watchdog.

   b) You need some external system anyway to tell QEMU when it's
      OK - what's your definitino of a failed system?

> What if, when self-fencing is enabled, Qemu kicks off a new thread
> from the start which does nothing but periodically wake up, verify the
> heartbeat condition and log()+abort() if required? (Then we wouldn't
> need the kernel timer.)

I'd make that thread bump the kernel timer along.

> > 
> >> I'm still wondering how to make this customisable so that different
> >> types of heartbeat could be implemented (preferably without creating
> >> external dependencies per discussion above). Thoughts welcome.
> > 
> > Yes, you need something to enable it, and some safe way to retrigger
> > the timer.  A qmp command marked as 'oob' might be the right way -
> > another qm command can't block it.
> 
> This qmp approach is slightly different than the external dependency
> that itself kills Qemu; if it doesn't run, then Qemu dies because the
> kernel timer is not updated. But this is also a heavyweight approach.
> We are talking about a service that needs to frequently connect to all
> running VMs on a host to reset the timer.
> 
> But it does allow for the customisable heartbeat: the logic behind
> what triggers the command is completely flexible.
> 
> Thinking about this idea of a separate Qemu thread, one thing that
> came to mind is this:
> 
> qemu -fence heartbeat=/path/to/file,deadline=60[,recheck=5]
> 
> Qemu could fire up a thread that stat()s <file> (every <recheck>
> seconds or on a default interval) and log()+abort() the whole process
> if the last modification time of the file is older than <deadline>. If
> <file> goes away (ie. stat() gives ENOENT), then it either fences
> immediately or ignores it, not sure which is more sensible.
> 
> Thoughts?

As above; I'm OK with using a file with that; but I'd make that thread
bump the kernel timer along; if that thread gets stuck somehow the
kernel still nukes your process.

Dave

> F.
> 
> > 
> > Dave
> > 
> > 
> >> F.
> >> 
> >>> 
> >>>> 
> >>>>> 
> >>>>> IMHO the safer way is to kick the host off the network by reprogramming
> >>>>> switches; so even if the qemu is actually alive it can't get anywhere.
> >>>>> 
> >>>>> Dave
> >>>> 
> >>>> Naturally some off-host STONITH is preferable, but that's not always
> >>>> available. A self-fencing mechanism right at the heart of the emulator
> >>>> can do the job without external hardware dependencies.
> >>> 
> >>> Dave
> >>> 
> >>>> Cheers,
> >>>> Felipe
> >>>> 
> >>>>> 
> >>>>> 
> >>>>>> Felipe
> >>>>>> 
> >>>>>>> 
> >>>>>>>> Does something already exist for this purpose which could be used?
> >>>>>>>> Would a generic Qemu-fencing infrastructure be something of interest?
> >>>>>>> Dave
> >>>>>>> 
> >>>>>>> 
> >>>>>>>> Cheers,
> >>>>>>>> F.
> >>>>>>>> 
> >>>>>>> --
> >>>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>>>>> 
> >>>>> --
> >>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>>> 
> >>> --
> >>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >> 
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK