Re: Current LibXL Status

[Ian Campbell <ian.campbell@citrix.com>]

create !
title it libxl exit() on ENOMEM incompatible with gc'd languages
thanks

On Thu, 2015-11-19 at 10:55 +0000, Andrew Cooper wrote:
> On 19/11/15 09:20, Ian Campbell wrote:
> > On Wed, 2015-11-18 at 18:32 +0000, Martin Osterloh wrote:
> > 
> > > I wanted to inquire about the current state of LibXL and in
> > > particular 
> > > if there are any issues with using it in long-running processes.
> > It is currently being used by libvirtd which I think has shaken out
> > most of
> > the issues with that environment.
> > 
> > There are certain to be other bugs, but nothing show-stopping.
> 
> There really is a show-stopper, which I have stated before.

Ah yes, nobody ever made a proposal to fix this so it slipped off my radar.

I've recorded it in the BTS this time.

> Languages such as OCaml use -ENOMEM as a hint to run the garbage
> collector some more.  I expect Haskell is the same.
> 
> It is not appropriate for libxl (or any library for that matter) to use
> exit() as its method of resolving out-of-memory conditions.

Note that the decision to take this approach was widely consulted at the
time, including with Ocaml folks, it wasn't just done on a whim. That's not
to say we cannot reconsider and find a different, better, approach which
works for such languages.

I think retrofitting all the necessary error paths to libxl to report
memory allocation failures back up to the caller is going to be untenable
and probably a project which would never actually be complete.

But I notice that there are only ~10 calls to libxl__alloc_failed in libxl.

I think it would be possible to turn each of those into a retry loop which
calls an application provided hook function on each iteration.

If no hook is provided by the application then the current behaviour would
remain.

Language bindings for Ocaml/haskel/etc would set the hook and use it to
call into their gc. You'd probably want to limit the number of retry
attempts and provide a way for the app to say "no, we really are out of
memory" if it wants.

Would such an approach work for Ocaml and haskell?

In particular I'd be concerned about things like the ocaml interpreter lock
(caml_enter_blocking_section etc) and calls to libxl not consistently
holding/dropping it potentially leading to deadlocks.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel