[RFC PATCH 00/20] libxl child process handling Ian Campbell <Ian.Campbell@citrix.com>, Stefano Stabellini <stefano.stabellini@eu.citrix.com>

[RFC PATCH 00/20] libxl child process handling Ian Campbell <Ian.Campbell@citrix.com>, Stefano Stabellini <stefano.stabellini@eu.citrix.com>

[Ian Jackson]

Once again, I have not executed the code in this series!
I have however updated it, added several yak-shaving patches,
and a new patch to make libxl_run_bootloader an ao function.

These three are intended to be bugfixes to sort out
the deadlock problem that Roger Pau Monne reported.
 01/20 libxl: ao: allow immediate completion
 02/20 libxl: fix hang due to libxl__initiate_device_remove
 03/20 libxl: Fix eventloop_iteration over-locking
Roger, can you confirm whether they work for you ?

These are other bugfixes:
 04/20 libxl: Fix leak of ctx->lock
 06/20 tools: Correct PTHREAD options in config/StdGNU.mk
 07/20 libxl: Use PTHREAD_CFLAGS, LDFLAGS, LIBS
 08/20 tools: Use PTHREAD_CFLAGS, _LDFLAGS, _LIBS
 19/20 libxl: Clean up setdefault in do_domain_create

These are clarifications and improvements related to memory allocation:
 09/20 libxl: Crash (more sensibly) on malloc failure
 10/20 libxl: Make libxl__zalloc et al tolerate a NULL gc

These are preparatory work:
 05/20 libxl: abolish libxl_ctx_postfork
 11/20 libxl: Introduce some convenience macros
 14/20 libxl: include <ctype.h> and introduce CTYPE helper macro
 15/20 libxl: include <_libxl_paths.h> in libxl_internal.h
 17/20 libxl: Provide libxl_string_list_length

These are new event-related infrastructure:
 12/20 libxl: Protect fds with CLOEXEC even with forking threads
 13/20 libxl: libxl_event.c:beforepoll_internal, REQUIRE_FDS
 16/20 libxl: event API: new facilities for waiting for subprocesses
 18/20 libxl: Introduce libxl__sendmsg_fds and libxl__recvmsg_fds

And here, finally, is a user!
 20/20 libxl: ao: Convert libxl_run_bootloader

In the pipeline, I have a half-done reorganisation of libxl_create.c's
do_domain_create to make it asynchronous.  When that's completed I'll
be able to test this stack.

Ian.

[PATCH 01/20] libxl: ao: allow immediate completion

[Ian Jackson]

Make it possible to complete an ao during its initating function.

Previously this was not generally possible because initiators did not
have an egc.  But there is no reason why an ao initiator should not
have an egc, so make the standard macros provide one.

Change the internal documentation comments accordingly.  (This change,
which means that an initiator function may call a completion callback
directly, is already consistent with the documented external API.)

We also invent of a new state flag "constructing" which indicates
whether we are between ao__create and ao__inprogress.  This is a
slightly optimisation which allows ao_complete to not bother poking
the wakeup pipe, since the logic in ao__inprogress will not run the
event loop if the ao is complete on entry.

Also fix the wording in the libxl_internal.h comment forbidding use of
ao_how-taking functions from within libxl.  (There are sadly currently
some such functions.)

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
---
 tools/libxl/libxl_event.c    |    7 ++++++-
 tools/libxl/libxl_internal.h |   14 ++++++++++----
 2 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 271949a..c89add8 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -1225,7 +1225,9 @@ void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc)
 
     if (ao->poller) {
         assert(ao->in_initiator);
-        libxl__poller_wakeup(egc, ao->poller);
+        if (!ao->constructing)
+            /* don't bother with this if we're not in the event loop */
+            libxl__poller_wakeup(egc, ao->poller);
     } else if (ao->how.callback) {
         LIBXL_TAILQ_INSERT_TAIL(&egc->aos_for_callback, ao, entry_for_callback);
     } else {
@@ -1251,6 +1253,7 @@ libxl__ao *libxl__ao_create(libxl_ctx *ctx, uint32_t domid,
     if (!ao) goto out;
 
     ao->magic = LIBXL__AO_MAGIC;
+    ao->constructing = 1;
     ao->in_initiator = 1;
     ao->poller = 0;
     ao->domid = domid;
@@ -1275,7 +1278,9 @@ int libxl__ao_inprogress(libxl__ao *ao)
     int rc;
 
     assert(ao->magic == LIBXL__AO_MAGIC);
+    assert(ao->constructing);
     assert(ao->in_initiator);
+    ao->constructing = 0;
 
     if (ao->poller) {
         /* Caller wants it done synchronously. */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index e0a1070..b1e0588 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -347,7 +347,7 @@ struct libxl__egc {
 
 struct libxl__ao {
     uint32_t magic;
-    unsigned in_initiator:1, complete:1, notified:1;
+    unsigned constructing:1, in_initiator:1, complete:1, notified:1;
     int rc;
     libxl__gc gc;
     libxl_asyncop_how how;
@@ -1209,7 +1209,11 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
  * operation ("ao") machinery.  The function should take a parameter
  * const libxl_asyncop_how *ao_how and must start with a call to
  * AO_INITIATOR_ENTRY.  These functions MAY NOT be called from
- * outside libxl, because they can cause reentrancy callbacks.
+ * inside libxl, because they can cause reentrancy callbacks.
+ *
+ * For the same reason functions taking an ao_how may make themselves
+ * an egc with EGC_INIT (and they will generally want to, to be able
+ * to immediately complete an ao during its setup).
  *
  * Lifecycle of an ao:
  *
@@ -1240,8 +1244,7 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
  *   directly or indirectly, should call libxl__ao_complete (with the
  *   ctx locked, as it will generally already be in any event callback
  *   function).  This must happen exactly once for each ao (and not if
- *   the ao has been destroyed, obviously), and it may not happen
- *   until libxl__ao_inprogress has been called on the ao.
+ *   the ao has been destroyed, obviously).
  *
  * - Note that during callback functions, two gcs are available:
  *    - The one in egc, whose lifetime is only this callback
@@ -1255,12 +1258,14 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
     libxl__ctx_lock(ctx);                                       \
     libxl__ao *ao = libxl__ao_create(ctx, domid, ao_how);       \
     if (!ao) { libxl__ctx_unlock(ctx); return ERROR_NOMEM; }    \
+    libxl__egc egc[1]; LIBXL_INIT_EGC(egc[0],ctx);              \
     AO_GC;
 
 #define AO_INPROGRESS ({                                        \
         libxl_ctx *ao__ctx = libxl__gc_owner(&ao->gc);          \
         int ao__rc = libxl__ao_inprogress(ao);                  \
         libxl__ctx_unlock(ao__ctx); /* gc is now invalid */     \
+        EGC_FREE;                                               \
         (ao__rc);                                               \
    })
 
@@ -1269,6 +1274,7 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
         assert(rc);                                             \
         libxl__ao_abort(ao);                                    \
         libxl__ctx_unlock(ao__ctx); /* gc is now invalid */     \
+        EGC_FREE;                                               \
         (rc);                                                   \
     })
 
-- 
1.7.2.5

Re: [PATCH 01/20] libxl: ao: allow immediate completion

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> Make it possible to complete an ao during its initating function.
> 
> Previously this was not generally possible because initiators did not
> have an egc.  But there is no reason why an ao initiator should not
> have an egc, so make the standard macros provide one.
> 
> Change the internal documentation comments accordingly.  (This change,
> which means that an initiator function may call a completion callback
> directly, is already consistent with the documented external API.)
> 
> We also invent of a new state flag "constructing" which indicates
> whether we are between ao__create and ao__inprogress.  This is a
> slightly optimisation which allows ao_complete to not bother poking
> the wakeup pipe, since the logic in ao__inprogress will not run the
> event loop if the ao is complete on entry.
> 
> Also fix the wording in the libxl_internal.h comment forbidding use of
> ao_how-taking functions from within libxl.  (There are sadly currently
> some such functions.)
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Roger Pau Monne <roger.pau@entel.upc.edu>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl_event.c    |    7 ++++++-
>  tools/libxl/libxl_internal.h |   14 ++++++++++----
>  2 files changed, 16 insertions(+), 5 deletions(-)
> 
> diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
> index 271949a..c89add8 100644
> --- a/tools/libxl/libxl_event.c
> +++ b/tools/libxl/libxl_event.c
> @@ -1225,7 +1225,9 @@ void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc)
>  
>      if (ao->poller) {
>          assert(ao->in_initiator);
> -        libxl__poller_wakeup(egc, ao->poller);
> +        if (!ao->constructing)
> +            /* don't bother with this if we're not in the event loop */
> +            libxl__poller_wakeup(egc, ao->poller);
>      } else if (ao->how.callback) {
>          LIBXL_TAILQ_INSERT_TAIL(&egc->aos_for_callback, ao, entry_for_callback);
>      } else {
> @@ -1251,6 +1253,7 @@ libxl__ao *libxl__ao_create(libxl_ctx *ctx, uint32_t domid,
>      if (!ao) goto out;
>  
>      ao->magic = LIBXL__AO_MAGIC;
> +    ao->constructing = 1;
>      ao->in_initiator = 1;
>      ao->poller = 0;
>      ao->domid = domid;
> @@ -1275,7 +1278,9 @@ int libxl__ao_inprogress(libxl__ao *ao)
>      int rc;
>  
>      assert(ao->magic == LIBXL__AO_MAGIC);
> +    assert(ao->constructing);
>      assert(ao->in_initiator);
> +    ao->constructing = 0;
>  
>      if (ao->poller) {
>          /* Caller wants it done synchronously. */
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index e0a1070..b1e0588 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -347,7 +347,7 @@ struct libxl__egc {
>  
>  struct libxl__ao {
>      uint32_t magic;
> -    unsigned in_initiator:1, complete:1, notified:1;
> +    unsigned constructing:1, in_initiator:1, complete:1, notified:1;
>      int rc;
>      libxl__gc gc;
>      libxl_asyncop_how how;
> @@ -1209,7 +1209,11 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
>   * operation ("ao") machinery.  The function should take a parameter
>   * const libxl_asyncop_how *ao_how and must start with a call to
>   * AO_INITIATOR_ENTRY.  These functions MAY NOT be called from
> - * outside libxl, because they can cause reentrancy callbacks.
> + * inside libxl, because they can cause reentrancy callbacks.
> + *
> + * For the same reason functions taking an ao_how may make themselves
> + * an egc with EGC_INIT (and they will generally want to, to be able
> + * to immediately complete an ao during its setup).
>   *
>   * Lifecycle of an ao:
>   *
> @@ -1240,8 +1244,7 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
>   *   directly or indirectly, should call libxl__ao_complete (with the
>   *   ctx locked, as it will generally already be in any event callback
>   *   function).  This must happen exactly once for each ao (and not if
> - *   the ao has been destroyed, obviously), and it may not happen
> - *   until libxl__ao_inprogress has been called on the ao.
> + *   the ao has been destroyed, obviously).
>   *
>   * - Note that during callback functions, two gcs are available:
>   *    - The one in egc, whose lifetime is only this callback
> @@ -1255,12 +1258,14 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
>      libxl__ctx_lock(ctx);                                       \
>      libxl__ao *ao = libxl__ao_create(ctx, domid, ao_how);       \
>      if (!ao) { libxl__ctx_unlock(ctx); return ERROR_NOMEM; }    \
> +    libxl__egc egc[1]; LIBXL_INIT_EGC(egc[0],ctx);              \
>      AO_GC;
>  
>  #define AO_INPROGRESS ({                                        \
>          libxl_ctx *ao__ctx = libxl__gc_owner(&ao->gc);          \
>          int ao__rc = libxl__ao_inprogress(ao);                  \
>          libxl__ctx_unlock(ao__ctx); /* gc is now invalid */     \
> +        EGC_FREE;                                               \
>          (ao__rc);                                               \
>     })
>  
> @@ -1269,6 +1274,7 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
>          assert(rc);                                             \
>          libxl__ao_abort(ao);                                    \
>          libxl__ctx_unlock(ao__ctx); /* gc is now invalid */     \
> +        EGC_FREE;                                               \
>          (rc);                                                   \
>      })
>  

[PATCH 02/20] libxl: fix hang due to libxl__initiate_device_remove

[Ian Jackson]

libxl__initiate_device_remove might discover that the operation was
complete, immediately (typically, if the device is already removed).

Previously, in this situation, it would return 0 to the caller but
never call libxl__ao_complete.  Fix this.  This necessitates passing
the egc in from the functions which are the ao initiators.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
---
 tools/libxl/libxl.c          |    8 ++++----
 tools/libxl/libxl_device.c   |   18 ++++++++++++------
 tools/libxl/libxl_internal.h |    3 ++-
 3 files changed, 18 insertions(+), 11 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 5344366..c54f86a 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -1406,7 +1406,7 @@ int libxl_device_disk_remove(libxl_ctx *ctx, uint32_t domid,
     rc = libxl__device_from_disk(gc, domid, disk, &device);
     if (rc != 0) goto out;
 
-    rc = libxl__initiate_device_remove(ao, &device);
+    rc = libxl__initiate_device_remove(egc, ao, &device);
     if (rc) goto out;
 
     return AO_INPROGRESS;
@@ -1873,7 +1873,7 @@ int libxl_device_nic_remove(libxl_ctx *ctx, uint32_t domid,
     rc = libxl__device_from_nic(gc, domid, nic, &device);
     if (rc != 0) goto out;
 
-    rc = libxl__initiate_device_remove(ao, &device);
+    rc = libxl__initiate_device_remove(egc, ao, &device);
     if (rc) goto out;
 
     return AO_INPROGRESS;
@@ -2220,7 +2220,7 @@ int libxl_device_vkb_remove(libxl_ctx *ctx, uint32_t domid,
     rc = libxl__device_from_vkb(gc, domid, vkb, &device);
     if (rc != 0) goto out;
 
-    rc = libxl__initiate_device_remove(ao, &device);
+    rc = libxl__initiate_device_remove(egc, ao, &device);
     if (rc) goto out;
 
     return AO_INPROGRESS;
@@ -2353,7 +2353,7 @@ int libxl_device_vfb_remove(libxl_ctx *ctx, uint32_t domid,
     rc = libxl__device_from_vfb(gc, domid, vfb, &device);
     if (rc != 0) goto out;
 
-    rc = libxl__initiate_device_remove(ao, &device);
+    rc = libxl__initiate_device_remove(egc, ao, &device);
     if (rc) goto out;
 
     return AO_INPROGRESS;
diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
index c2880e0..c7e057d 100644
--- a/tools/libxl/libxl_device.c
+++ b/tools/libxl/libxl_device.c
@@ -376,7 +376,8 @@ static void device_remove_callback(libxl__egc *egc, libxl__ev_devstate *ds,
     device_remove_cleanup(gc, aorm);
 }
 
-int libxl__initiate_device_remove(libxl__ao *ao, libxl__device *dev)
+int libxl__initiate_device_remove(libxl__egc *egc, libxl__ao *ao,
+                                  libxl__device *dev)
 {
     AO_GC;
     libxl_ctx *ctx = libxl__gc_owner(gc);
@@ -388,11 +389,11 @@ int libxl__initiate_device_remove(libxl__ao *ao, libxl__device *dev)
     libxl__ao_device_remove *aorm = 0;
 
     if (!state)
-        goto out;
+        goto out_ok;
     if (atoi(state) != 4) {
         libxl__device_destroy_tapdisk(gc, be_path);
         xs_rm(ctx->xsh, XBT_NULL, be_path);
-        goto out;
+        goto out_ok;
     }
 
 retry_transaction:
@@ -404,7 +405,7 @@ retry_transaction:
             goto retry_transaction;
         else {
             rc = ERROR_FAIL;
-            goto out;
+            goto out_fail;
         }
     }
 
@@ -417,13 +418,18 @@ retry_transaction:
     rc = libxl__ev_devstate_wait(gc, &aorm->ds, device_remove_callback,
                                  state_path, XenbusStateClosed,
                                  LIBXL_DESTROY_TIMEOUT * 1000);
-    if (rc) goto out;
+    if (rc) goto out_fail;
 
     return 0;
 
- out:
+ out_fail:
+    assert(rc);
     device_remove_cleanup(gc, aorm);
     return rc;
+
+ out_ok:
+    libxl__ao_complete(egc, ao, 0);
+    return 0;
 }
 
 int libxl__device_destroy(libxl__gc *gc, libxl__device *dev)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index b1e0588..5167b71 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -692,7 +692,8 @@ _hidden int libxl__wait_for_backend(libxl__gc *gc, char *be_path, char *state);
  * this is done, the ao will be completed.  An error
  * return from libxl__initiate_device_remove means that the ao
  * will _not_ be completed and the caller must do so. */
-_hidden int libxl__initiate_device_remove(libxl__ao*, libxl__device *dev);
+_hidden int libxl__initiate_device_remove(libxl__egc*, libxl__ao*,
+                                          libxl__device *dev);
 
 /*
  * libxl__ev_devstate - waits a given time for a device to
-- 
1.7.2.5

Re: [PATCH 02/20] libxl: fix hang due to libxl__initiate_device_remove

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> libxl__initiate_device_remove might discover that the operation was
> complete, immediately (typically, if the device is already removed).
> 
> Previously, in this situation, it would return 0 to the caller but
> never call libxl__ao_complete.  Fix this.  This necessitates passing
> the egc in from the functions which are the ao initiators.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Roger Pau Monne <roger.pau@entel.upc.edu>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[PATCH 03/20] libxl: Fix eventloop_iteration over-locking

[Ian Jackson]

eventloop_iteration's head comment says that it must be called with
the ctx locked exactly once, and this is indeed true, and it's done
correctly at both the call sites.

However, it takes out the lock an additional time itself.  This is
wrong because it prevents the unlocks around poll from being
effective.  This would mean that a multithreaded event-loop using
program might suffer from undesired blocking, as one thread trying to
enter libxl might end up stalled by another thread waiting for a slow
event.  So remove those two lock calls.

Also add a couple of comments documenting the locking behaviour of
libxl__ao_inprogress and libxl__egc_cleanup.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_event.c    |    4 ----
 tools/libxl/libxl_internal.h |    4 ++--
 2 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index c89add8..5ac6334 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -1058,8 +1058,6 @@ static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
     int rc;
     struct timeval now;
     
-    CTX_LOCK;
-
     rc = libxl__gettimeofday(gc, &now);
     if (rc) goto out;
 
@@ -1102,8 +1100,6 @@ static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
     afterpoll_internal(egc, poller,
                        poller->fd_polls_allocd, poller->fd_polls, now);
 
-    CTX_UNLOCK;
-
     rc = 0;
  out:
     return rc;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 5167b71..04f13f6 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1191,7 +1191,7 @@ libxl__device_model_version_running(libxl__gc *gc, uint32_t domid);
 _hidden void libxl__egc_cleanup(libxl__egc *egc);
   /* Frees memory allocated within this egc's gc, and and report all
    * occurred events via callback, if applicable.  May reenter the
-   * application; see restrictions above. */
+   * application; see restrictions above.  The ctx must be UNLOCKED. */
 
 /* convenience macros: */
 
@@ -1287,7 +1287,7 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
  * libxl__ao_inprogress MUST be called with the ctx locked exactly once. */
 _hidden libxl__ao *libxl__ao_create(libxl_ctx*, uint32_t domid,
                                     const libxl_asyncop_how*);
-_hidden int libxl__ao_inprogress(libxl__ao *ao);
+_hidden int libxl__ao_inprogress(libxl__ao *ao); /* temporarily unlocks */
 _hidden void libxl__ao_abort(libxl__ao *ao);
 _hidden void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc);
 
-- 
1.7.2.5

Re: [PATCH 03/20] libxl: Fix eventloop_iteration over-locking

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> eventloop_iteration's head comment says that it must be called with
> the ctx locked exactly once, and this is indeed true, and it's done
> correctly at both the call sites.
> 
> However, it takes out the lock an additional time itself.  This is
> wrong because it prevents the unlocks around poll from being
> effective.  This would mean that a multithreaded event-loop using
> program might suffer from undesired blocking, as one thread trying to
> enter libxl might end up stalled by another thread waiting for a slow
> event.  So remove those two lock calls.
> 
> Also add a couple of comments documenting the locking behaviour of
> libxl__ao_inprogress and libxl__egc_cleanup.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>

A comment in eventloop_iteration along the lines "no need to lock, we
must be called with lock held once" might be nice to prevent future
re-occurences of the same error.

> ---
>  tools/libxl/libxl_event.c    |    4 ----
>  tools/libxl/libxl_internal.h |    4 ++--
>  2 files changed, 2 insertions(+), 6 deletions(-)
> 
> diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
> index c89add8..5ac6334 100644
> --- a/tools/libxl/libxl_event.c
> +++ b/tools/libxl/libxl_event.c
> @@ -1058,8 +1058,6 @@ static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
>      int rc;
>      struct timeval now;
>      
> -    CTX_LOCK;
> -
>      rc = libxl__gettimeofday(gc, &now);
>      if (rc) goto out;
>  
> @@ -1102,8 +1100,6 @@ static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
>      afterpoll_internal(egc, poller,
>                         poller->fd_polls_allocd, poller->fd_polls, now);
>  
> -    CTX_UNLOCK;
> -
>      rc = 0;
>   out:
>      return rc;
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 5167b71..04f13f6 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -1191,7 +1191,7 @@ libxl__device_model_version_running(libxl__gc *gc, uint32_t domid);
>  _hidden void libxl__egc_cleanup(libxl__egc *egc);
>    /* Frees memory allocated within this egc's gc, and and report all
>     * occurred events via callback, if applicable.  May reenter the
> -   * application; see restrictions above. */
> +   * application; see restrictions above.  The ctx must be UNLOCKED. */
>  
>  /* convenience macros: */
>  
> @@ -1287,7 +1287,7 @@ _hidden void libxl__egc_cleanup(libxl__egc *egc);
>   * libxl__ao_inprogress MUST be called with the ctx locked exactly once. */
>  _hidden libxl__ao *libxl__ao_create(libxl_ctx*, uint32_t domid,
>                                      const libxl_asyncop_how*);
> -_hidden int libxl__ao_inprogress(libxl__ao *ao);
> +_hidden int libxl__ao_inprogress(libxl__ao *ao); /* temporarily unlocks */
>  _hidden void libxl__ao_abort(libxl__ao *ao);
>  _hidden void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc);
>  

Re: [PATCH 03/20] libxl: Fix eventloop_iteration over-locking

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 03/20] libxl: Fix eventloop_iteration over-locking"):
> A comment in eventloop_iteration along the lines "no need to lock, we
> must be called with lock held once" might be nice to prevent future
> re-occurences of the same error.

You don't see it in this patch, but there's something along these
lines already.  Let me add some more context:

> > @@ -1058,8 +1058,6 @@ static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
     static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
         /* The CTX must be locked EXACTLY ONCE so that this function
          * can unlock it when it polls.
          */
         EGC_GC;
> >      int rc;
> >      struct timeval now;
> >      
> > -    CTX_LOCK;
> > -

Ian.

Re: [PATCH 03/20] libxl: Fix eventloop_iteration over-locking

[Ian Campbell]

On Mon, 2012-03-19 at 17:48 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 03/20] libxl: Fix eventloop_iteration over-locking"):
> > A comment in eventloop_iteration along the lines "no need to lock, we
> > must be called with lock held once" might be nice to prevent future
> > re-occurences of the same error.
> 
> You don't see it in this patch, but there's something along these
> lines already.  Let me add some more context:
> 
> > > @@ -1058,8 +1058,6 @@ static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
>      static int eventloop_iteration(libxl__egc *egc, libxl__poller *poller) {
>          /* The CTX must be locked EXACTLY ONCE so that this function
>           * can unlock it when it polls.
>           */

Brilliant, thanks!

>          EGC_GC;
> > >      int rc;
> > >      struct timeval now;
> > >      
> > > -    CTX_LOCK;
> > > -
> 
> Ian.

[PATCH 04/20] libxl: Fix leak of ctx->lock

[Ian Jackson]

A mutex created with pthread_mutex_init, like ctx->lock, may need to
be destroyed with pthread_mutex_destroy.

Also, previously, if libxl__init_recursive_mutex failed, the nascent
ctx would be leaked.  Add some comments which will hopefully make
these kind of mistakes less likely in future.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl.c |   17 +++++++++++++----
 1 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index c54f86a..5827bd4 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -39,10 +39,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     memset(ctx, 0, sizeof(libxl_ctx));
     ctx->lg = lg;
 
-    if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
-        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Failed to initialize mutex");
-        return ERROR_FAIL;
-    }
+    /* First initialise pointers (cannot fail) */
 
     LIBXL_TAILQ_INIT(&ctx->occurred);
 
@@ -61,6 +58,16 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     LIBXL_TAILQ_INIT(&ctx->death_list);
     libxl__ev_xswatch_init(&ctx->death_watch);
 
+    /* The mutex is special because we can't idempotently destroy it */
+
+    if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
+        LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Failed to initialize mutex");
+        free(ctx);
+        ctx = 0;
+    }
+
+    /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
+
     rc = libxl__poller_init(ctx, &ctx->poller_app);
     if (rc) goto out;
 
@@ -150,6 +157,8 @@ int libxl_ctx_free(libxl_ctx *ctx)
 
     discard_events(&ctx->occurred);
 
+    pthread_mutex_destroy(&ctx->lock);
+
     GC_FREE;
     free(ctx);
     return 0;
-- 
1.7.2.5

[PATCH 05/20] libxl: abolish libxl_ctx_postfork

[Ian Jackson]

libxl's task has become too complicated (particularly in the presence
of both forking and multithreading) to support reuse of the same
libxl_ctx after fork.

So abolish libxl_ctx_fork.  xl instead simply initialises a new
libxl_ctx.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tools/libxl/libxl.h       |    1 -
 tools/libxl/libxl_utils.c |    7 -------
 tools/libxl/xl.c          |    8 ++++++++
 tools/libxl/xl.h          |    1 +
 tools/libxl/xl_cmdimpl.c  |    8 ++------
 5 files changed, 11 insertions(+), 14 deletions(-)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 6b69030..dce087c 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -378,7 +378,6 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
                     unsigned flags /* none currently defined */,
                     xentoollog_logger *lg);
 int libxl_ctx_free(libxl_ctx *ctx /* 0 is OK */);
-int libxl_ctx_postfork(libxl_ctx *ctx);
 
 /* domain related functions */
 typedef int (*libxl_console_ready)(libxl_ctx *ctx, uint32_t domid, void *priv);
diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index d6cd78d..0cbd85e 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -365,13 +365,6 @@ READ_WRITE_EXACTLY(read, 1, /* */)
 READ_WRITE_EXACTLY(write, 0, const)
 
 
-int libxl_ctx_postfork(libxl_ctx *ctx) {
-    if (ctx->xsh) xs_daemon_destroy_postfork(ctx->xsh);
-    ctx->xsh = xs_daemon_open();
-    if (!ctx->xsh) return ERROR_FAIL;
-    return 0;
-}
-
 pid_t libxl_fork(libxl_ctx *ctx)
 {
     pid_t pid;
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index df9b1e7..9fd67b4 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -95,6 +95,14 @@ static void parse_global_config(const char *configfile,
     xlu_cfg_destroy(config);
 }
 
+void postfork(void)
+{
+    if (libxl_ctx_alloc(&ctx, LIBXL_VERSION, 0, (xentoollog_logger*)logger)) {
+        fprintf(stderr, "cannot reinit xl context after fork\n");
+        exit(-1);
+    }
+}
+
 int main(int argc, char **argv)
 {
     int opt = 0;
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index 702b208..394a128 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -104,6 +104,7 @@ struct cmd_spec *cmdtable_lookup(const char *s);
 
 extern libxl_ctx *ctx;
 extern xentoollog_logger_stdiostream *logger;
+void postfork(void);
 
 /* global options */
 extern int autoballoon;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 1d59b89..18ddf31 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1408,7 +1408,7 @@ static int autoconnect_console(libxl_ctx *ctx, uint32_t domid, void *priv)
     } else if (*pid > 0)
         return 0;
 
-    libxl_ctx_postfork(ctx);
+    postfork();
 
     sleep(1);
     libxl_primary_console_exec(ctx, domid);
@@ -1695,11 +1695,7 @@ start:
             goto out;
         }
 
-        rc = libxl_ctx_postfork(ctx);
-        if (rc) {
-            LOG("failed to reinitialise context after fork");
-            exit(-1);
-        }
+        postfork();
 
         if (asprintf(&name, "xl-%s", d_config.c_info.name) < 0) {
             LOG("Failed to allocate memory in asprintf");
-- 
1.7.2.5

[PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk

[Ian Jackson]

It is not correct to say -lpthread.  The correct option is -pthread,
which may have sundry other effects on code generation etc.  It needs
to be passed both to compilation and linking.

Fix the configure test to test -pthread, and plumb the resulting flag
through to PTHREAD_{CFLAGS,LDFLAGS} in Tools.mk; also substitute
PTHREAD_LIBS (although this will currently always be empty).
Remove PTHREAD_LIBS setting from StdGNU.mk.

Fix the one user (libxc) to use PTHREAD_{CFLAGS,LDFLAGS} too.

There are still some other users in tree which pass -pthread or
-lpthread by adding it as a literal to their own compiler options.
These will be fixed in a later patch.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
---
 config/StdGNU.mk     |    1 -
 config/Tools.mk.in   |    4 ++
 tools/configure      |   95 +++++++++++++++++++++++++++++++++++---------------
 tools/configure.ac   |    5 ++-
 tools/libxc/Makefile |    4 ++-
 tools/m4/pthread.m4  |   41 +++++++++++++++++++++
 tools/m4/savevar.m4  |    6 +++
 7 files changed, 124 insertions(+), 32 deletions(-)
 create mode 100644 tools/m4/pthread.m4
 create mode 100644 tools/m4/savevar.m4

diff --git a/config/StdGNU.mk b/config/StdGNU.mk
index e2c9335..e2f2e1e 100644
--- a/config/StdGNU.mk
+++ b/config/StdGNU.mk
@@ -67,7 +67,6 @@ XEN_CONFIG_DIR = $(CONFIG_DIR)/xen
 XEN_SCRIPT_DIR = $(XEN_CONFIG_DIR)/scripts
 
 SOCKET_LIBS =
-PTHREAD_LIBS = -lpthread
 UTIL_LIBS = -lutil
 DLOPEN_LIBS = -ldl
 
diff --git a/config/Tools.mk.in b/config/Tools.mk.in
index 057b055..8247e4a 100644
--- a/config/Tools.mk.in
+++ b/config/Tools.mk.in
@@ -23,6 +23,10 @@ PREPEND_LIB         := @PREPEND_LIB@
 APPEND_INCLUDES     := @APPEND_INCLUDES@
 APPEND_LIB          := @APPEND_LIB@
 
+PTHREAD_CFLAGS      := @PTHREAD_CFLAGS@
+PTHREAD_LDFLAGS     := @PTHREAD_LDFLAGS@
+PTHREAD_LIBS        := @PTHREAD_LIBS@
+
 # Download GIT repositories via HTTP or GIT's own protocol?
 # GIT's protocol is faster and more robust, when it works at all (firewalls
 # may block it). We make it the default, but if your GIT repository downloads
diff --git a/tools/configure b/tools/configure
index c686130..7ee4e3e 100755
--- a/tools/configure
+++ b/tools/configure
@@ -602,6 +602,9 @@ POW_LIB
 LIBOBJS
 ALLOCA
 libiconv
+PTHREAD_LIBS
+PTHREAD_LDFLAGS
+PTHREAD_CFLAGS
 libgcrypt
 libext2fs
 system_aio
@@ -3844,6 +3847,9 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
 
 
 
+
+
+
 # pkg.m4 - Macros to locate and utilise pkg-config.            -*- Autoconf -*-
 # serial 1 (pkg-config-0.24)
 #
@@ -3908,6 +3914,16 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
 
 
 
+
+
+
+
+
+
+
+
+
+
 # Enable/disable options
 # Check whether --enable-githttp was given.
 if test "${enable_githttp+set}" = set; then :
@@ -7010,47 +7026,70 @@ else
 fi
 
 
-{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lpthread" >&5
-$as_echo_n "checking for pthread_create in -lpthread... " >&6; }
-if test "${ac_cv_lib_pthread_pthread_create+set}" = set; then :
+
+    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread flag" >&5
+$as_echo_n "checking for pthread flag... " >&6; }
+if test "${ax_cv_pthread_flags+set}" = set; then :
   $as_echo_n "(cached) " >&6
 else
-  ac_check_lib_save_LIBS=$LIBS
-LIBS="-lpthread  $LIBS"
-cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+
+        ax_cv_pthread_flags=-pthread
+
+    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LIBS=""
+
+
+    saved_CFLAGS="$CFLAGS"
+
+    saved_LDFLAGS="$LDFLAGS"
+
+    saved_LIBS="$LIBS"
+
+
+    CFLAGS="$CFLAGS $PTHREAD_CFLAGS"
+
+    LDFLAGS="$LDFLAGS $PTHREAD_LDFLAGS"
+
+    LIBS="$LIBS $PTHREAD_LIBS"
+
+        cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 
-/* Override any GCC internal prototype to avoid an error.
-   Use char because int might match the return type of a GCC
-   builtin and then its argument prototype would still apply.  */
-#ifdef __cplusplus
-extern "C"
-#endif
-char pthread_create ();
-int
-main ()
-{
-return pthread_create ();
-  ;
-  return 0;
+#include <pthread.h>
+int main(void) {
+  pthread_atfork(0,0,0);
+  pthread_create(0,0,0,0);
 }
+
 _ACEOF
 if ac_fn_c_try_link "$LINENO"; then :
-  ac_cv_lib_pthread_pthread_create=yes
+
 else
-  ac_cv_lib_pthread_pthread_create=no
+  ax_cv_pthread_flags=failed
 fi
 rm -f core conftest.err conftest.$ac_objext \
     conftest$ac_exeext conftest.$ac_ext
-LIBS=$ac_check_lib_save_LIBS
-fi
-{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthread_pthread_create" >&5
-$as_echo "$ac_cv_lib_pthread_pthread_create" >&6; }
-if test "x$ac_cv_lib_pthread_pthread_create" = x""yes; then :
 
-else
-  as_fn_error $? "Could not find libpthread" "$LINENO" 5
+    CFLAGS="$saved_CFLAGS"
+
+    LDFLAGS="$saved_LDFLAGS"
+
+    LIBS="$saved_LIBS"
+
+
 fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ax_cv_pthread_flags" >&5
+$as_echo "$ax_cv_pthread_flags" >&6; }
+    if test "x$ax_cv_pthread_flags" = xfailed; then
+        as_fn_error $? "-pthread does not work" "$LINENO" 5
+    fi
+
+    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LIBS=""
+
+
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_gettime in -lrt" >&5
 $as_echo_n "checking for clock_gettime in -lrt... " >&6; }
diff --git a/tools/configure.ac b/tools/configure.ac
index 3d53a95..3da0c82 100644
--- a/tools/configure.ac
+++ b/tools/configure.ac
@@ -23,6 +23,7 @@ AC_USE_SYSTEM_EXTENSIONS
 AC_CANONICAL_HOST
 
 # M4 Macro includes
+m4_include([m4/savevar.m4])
 m4_include([m4/enable_feature.m4])
 m4_include([m4/disable_feature.m4])
 m4_include([m4/path_or_fail.m4])
@@ -34,6 +35,7 @@ m4_include([m4/set_cflags_ldflags.m4])
 m4_include([m4/uuid.m4])
 m4_include([m4/pkg.m4])
 m4_include([m4/curses.m4])
+m4_include([m4/pthread.m4])
 
 # Enable/disable options
 AX_ARG_ENABLE_AND_EXPORT([githttp], [Download GIT repositories via HTTP])
@@ -123,8 +125,7 @@ AC_CHECK_LIB([ext2fs], [ext2fs_open2], [libext2fs="y"], [libext2fs="n"])
 AC_SUBST(libext2fs)
 AC_CHECK_LIB([gcrypt], [gcry_md_hash_buffer], [libgcrypt="y"], [libgcrypt="n"])
 AC_SUBST(libgcrypt)
-AC_CHECK_LIB([pthread], [pthread_create], [] ,
-    [AC_MSG_ERROR([Could not find libpthread])])
+AX_CHECK_PTHREAD
 AC_CHECK_LIB([rt], [clock_gettime])
 AC_CHECK_LIB([yajl], [yajl_alloc], [],
     [AC_MSG_ERROR([Could not find yajl])])
diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
index 55eb755..a1ba134 100644
--- a/tools/libxc/Makefile
+++ b/tools/libxc/Makefile
@@ -73,6 +73,8 @@ CFLAGS   += -I. $(CFLAGS_xeninclude)
 # Needed for posix_fadvise64() in xc_linux.c
 CFLAGS-$(CONFIG_Linux) += -D_GNU_SOURCE
 
+CFLAGS	+= $(PTHREAD_CFLAGS)
+
 # Define this to make it possible to run valgrind on code linked with these
 # libraries.
 #CFLAGS   += -DVALGRIND -O0 -ggdb3
@@ -157,7 +159,7 @@ libxenctrl.so.$(MAJOR): libxenctrl.so.$(MAJOR).$(MINOR)
 	ln -sf $< $@
 
 libxenctrl.so.$(MAJOR).$(MINOR): $(CTRL_PIC_OBJS)
-	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
+	$(CC) $(LDFLAGS) $(PTHREAD_LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
 
 # libxenguest
 
diff --git a/tools/m4/pthread.m4 b/tools/m4/pthread.m4
new file mode 100644
index 0000000..57ea85c
--- /dev/null
+++ b/tools/m4/pthread.m4
@@ -0,0 +1,41 @@
+# We define, separately, PTHREAD_CFLAGS, _LDFLAGS and _LIBS
+# even though currently we don't set them very separately.
+# This means that the makefiles will not need to change in
+# the future if we make the test more sophisticated.
+
+AC_DEFUN([AX_PTHREAD_CV2VARS],[
+    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
+    PTHREAD_LIBS=""
+])
+
+# We invoke AX_PTHREAD_VARS with the name of another macro
+# which is then expanded once for each variable.
+AC_DEFUN([AX_PTHREAD_VARS],[$1(CFLAGS) $1(LDFLAGS) $1(LIBS)])
+
+AC_DEFUN([AX_PTHREAD_VAR_APPLY],[
+    $1="$$1 $PTHREAD_$1"
+])
+AC_DEFUN([AX_PTHREAD_VAR_SUBST],[AC_SUBST(PTHREAD_$1)])
+
+AC_DEFUN([AX_CHECK_PTHREAD],[
+    AC_CACHE_CHECK([for pthread flag], [ax_cv_pthread_flags], [
+        ax_cv_pthread_flags=-pthread
+        AX_PTHREAD_CV2VARS
+        AX_PTHREAD_VARS([AX_SAVEVAR_SAVE])
+        AX_PTHREAD_VARS([AX_PTHREAD_VAR_APPLY])
+        AC_LINK_IFELSE([
+#include <pthread.h>
+int main(void) {
+  pthread_atfork(0,0,0);
+  pthread_create(0,0,0,0);
+}
+],[],[ax_cv_pthread_flags=failed])
+        AX_PTHREAD_VARS([AX_SAVEVAR_RESTORE])
+    ])
+    if test "x$ax_cv_pthread_flags" = xfailed; then
+        AC_MSG_ERROR([-pthread does not work])
+    fi
+    AX_PTHREAD_CV2VARS
+    AX_PTHREAD_VARS([AX_PTHREAD_VAR_SUBST])
+])
diff --git a/tools/m4/savevar.m4 b/tools/m4/savevar.m4
new file mode 100644
index 0000000..2156bee
--- /dev/null
+++ b/tools/m4/savevar.m4
@@ -0,0 +1,6 @@
+AC_DEFUN([AX_SAVEVAR_SAVE],[
+    saved_$1="$$1"
+])
+AC_DEFUN([AX_SAVEVAR_RESTORE],[
+    $1="$saved_$1"
+])
-- 
1.7.2.5

Re: [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk

[Roger Pau Monné]

2012/3/16 Ian Jackson <ian.jackson@eu.citrix.com>:
> It is not correct to say -lpthread.  The correct option is -pthread,
> which may have sundry other effects on code generation etc.  It needs
> to be passed both to compilation and linking.
>
> Fix the configure test to test -pthread, and plumb the resulting flag
> through to PTHREAD_{CFLAGS,LDFLAGS} in Tools.mk; also substitute
> PTHREAD_LIBS (although this will currently always be empty).
> Remove PTHREAD_LIBS setting from StdGNU.mk.
>
> Fix the one user (libxc) to use PTHREAD_{CFLAGS,LDFLAGS} too.
>
> There are still some other users in tree which pass -pthread or
> -lpthread by adding it as a literal to their own compiler options.
> These will be fixed in a later patch.
>
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
Acked-by: Roger Pau Monne <roger.pau@entel.upc.edu>

> ---
>  config/StdGNU.mk     |    1 -
>  config/Tools.mk.in   |    4 ++
>  tools/configure      |   95 +++++++++++++++++++++++++++++++++++---------------
>  tools/configure.ac   |    5 ++-
>  tools/libxc/Makefile |    4 ++-
>  tools/m4/pthread.m4  |   41 +++++++++++++++++++++
>  tools/m4/savevar.m4  |    6 +++
>  7 files changed, 124 insertions(+), 32 deletions(-)
>  create mode 100644 tools/m4/pthread.m4
>  create mode 100644 tools/m4/savevar.m4
>

[...]

> diff --git a/tools/m4/pthread.m4 b/tools/m4/pthread.m4
> new file mode 100644
> index 0000000..57ea85c
> --- /dev/null
> +++ b/tools/m4/pthread.m4
> @@ -0,0 +1,41 @@
> +# We define, separately, PTHREAD_CFLAGS, _LDFLAGS and _LIBS
> +# even though currently we don't set them very separately.
> +# This means that the makefiles will not need to change in
> +# the future if we make the test more sophisticated.
> +
> +AC_DEFUN([AX_PTHREAD_CV2VARS],[
> +    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
> +    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
> +    PTHREAD_LIBS=""
> +])
> +
> +# We invoke AX_PTHREAD_VARS with the name of another macro
> +# which is then expanded once for each variable.
> +AC_DEFUN([AX_PTHREAD_VARS],[$1(CFLAGS) $1(LDFLAGS) $1(LIBS)])
> +
> +AC_DEFUN([AX_PTHREAD_VAR_APPLY],[
> +    $1="$$1 $PTHREAD_$1"
> +])
> +AC_DEFUN([AX_PTHREAD_VAR_SUBST],[AC_SUBST(PTHREAD_$1)])
> +
> +AC_DEFUN([AX_CHECK_PTHREAD],[
> +    AC_CACHE_CHECK([for pthread flag], [ax_cv_pthread_flags], [
> +        ax_cv_pthread_flags=-pthread
> +        AX_PTHREAD_CV2VARS
> +        AX_PTHREAD_VARS([AX_SAVEVAR_SAVE])
> +        AX_PTHREAD_VARS([AX_PTHREAD_VAR_APPLY])
> +        AC_LINK_IFELSE([
> +#include <pthread.h>
> +int main(void) {
> +  pthread_atfork(0,0,0);
> +  pthread_create(0,0,0,0);
> +}

You can also use something like this here, which is more autoconf style:

AC_LANG_PROGRAM([#include <pthread.h>],[
                pthread_atfork(0,0,0);
                pthread_create(0,0,0,0);])

But as is looks correct to me, I'm not a big fan of autoconf macros.

> +],[],[ax_cv_pthread_flags=failed])
> +        AX_PTHREAD_VARS([AX_SAVEVAR_RESTORE])
> +    ])
> +    if test "x$ax_cv_pthread_flags" = xfailed; then
> +        AC_MSG_ERROR([-pthread does not work])
> +    fi
> +    AX_PTHREAD_CV2VARS
> +    AX_PTHREAD_VARS([AX_PTHREAD_VAR_SUBST])
> +])
> diff --git a/tools/m4/savevar.m4 b/tools/m4/savevar.m4
> new file mode 100644
> index 0000000..2156bee
> --- /dev/null
> +++ b/tools/m4/savevar.m4
> @@ -0,0 +1,6 @@
> +AC_DEFUN([AX_SAVEVAR_SAVE],[
> +    saved_$1="$$1"
> +])
> +AC_DEFUN([AX_SAVEVAR_RESTORE],[
> +    $1="$saved_$1"
> +])

This little macros will prove useful on other places, such as
python_devel.m4, that actually uses ac_previous_* to temporary store
variables.

> --
> 1.7.2.5
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk

[Ian Jackson]

Roger Pau Monné writes ("Re: [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk"):
> 2012/3/16 Ian Jackson <ian.jackson@eu.citrix.com>:
> > It is not correct to say -lpthread.  The correct option is -pthread,
> > which may have sundry other effects on code generation etc.  It needs
> > to be passed both to compilation and linking.
...
> Acked-by: Roger Pau Monne <roger.pau@entel.upc.edu>

Thanks.

> > +        AC_LINK_IFELSE([
> > +#include <pthread.h>
> > +int main(void) {
> > +  pthread_atfork(0,0,0);
> > +  pthread_create(0,0,0,0);
> > +}
> 
> You can also use something like this here, which is more autoconf style:
> 
> AC_LANG_PROGRAM([#include <pthread.h>],[
>                 pthread_atfork(0,0,0);
>                 pthread_create(0,0,0,0);])
> 
> But as is looks correct to me, I'm not a big fan of autoconf macros.

Thanks for the comment.  I think I agree with your sentiment so I'll
leave this as it is.

> > +],[],[ax_cv_pthread_flags=failed])
> > +AC_DEFUN([AX_SAVEVAR_SAVE],[
> > +    saved_$1="$$1"
> > +])
> > +AC_DEFUN([AX_SAVEVAR_RESTORE],[
> > +    $1="$saved_$1"
> > +])
> 
> This little macros will prove useful on other places, such as
> python_devel.m4, that actually uses ac_previous_* to temporary store
> variables.

Right, that was the idea.

Ian.

Re: [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> It is not correct to say -lpthread.  The correct option is -pthread,
> which may have sundry other effects on code generation etc.  It needs
> to be passed both to compilation and linking.

I guess this is a gcc-ism rather than a Linux-ism and so applies to the
BSDs too (at least so long as they use gcc and not clang)?

> Fix the configure test to test -pthread, and plumb the resulting flag
> through to PTHREAD_{CFLAGS,LDFLAGS} in Tools.mk; also substitute
> PTHREAD_LIBS (although this will currently always be empty).
> Remove PTHREAD_LIBS setting from StdGNU.mk.
> 
> Fix the one user (libxc) to use PTHREAD_{CFLAGS,LDFLAGS} too.
> 
> There are still some other users in tree which pass -pthread or
> -lpthread by adding it as a literal to their own compiler options.
> These will be fixed in a later patch.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
> ---
>  config/StdGNU.mk     |    1 -
>  config/Tools.mk.in   |    4 ++
>  tools/configure      |   95 +++++++++++++++++++++++++++++++++++---------------
>  tools/configure.ac   |    5 ++-
>  tools/libxc/Makefile |    4 ++-
>  tools/m4/pthread.m4  |   41 +++++++++++++++++++++
>  tools/m4/savevar.m4  |    6 +++
>  7 files changed, 124 insertions(+), 32 deletions(-)
>  create mode 100644 tools/m4/pthread.m4
>  create mode 100644 tools/m4/savevar.m4
> 
> diff --git a/config/StdGNU.mk b/config/StdGNU.mk
> index e2c9335..e2f2e1e 100644
> --- a/config/StdGNU.mk
> +++ b/config/StdGNU.mk
> @@ -67,7 +67,6 @@ XEN_CONFIG_DIR = $(CONFIG_DIR)/xen
>  XEN_SCRIPT_DIR = $(XEN_CONFIG_DIR)/scripts
>  
>  SOCKET_LIBS =
> -PTHREAD_LIBS = -lpthread
>  UTIL_LIBS = -lutil
>  DLOPEN_LIBS = -ldl
>  
> diff --git a/config/Tools.mk.in b/config/Tools.mk.in
> index 057b055..8247e4a 100644
> --- a/config/Tools.mk.in
> +++ b/config/Tools.mk.in
> @@ -23,6 +23,10 @@ PREPEND_LIB         := @PREPEND_LIB@
>  APPEND_INCLUDES     := @APPEND_INCLUDES@
>  APPEND_LIB          := @APPEND_LIB@
>  
> +PTHREAD_CFLAGS      := @PTHREAD_CFLAGS@
> +PTHREAD_LDFLAGS     := @PTHREAD_LDFLAGS@
> +PTHREAD_LIBS        := @PTHREAD_LIBS@
> +
>  # Download GIT repositories via HTTP or GIT's own protocol?
>  # GIT's protocol is faster and more robust, when it works at all (firewalls
>  # may block it). We make it the default, but if your GIT repository downloads
> diff --git a/tools/configure b/tools/configure
> index c686130..7ee4e3e 100755
> --- a/tools/configure
> +++ b/tools/configure
> @@ -602,6 +602,9 @@ POW_LIB
>  LIBOBJS
>  ALLOCA
>  libiconv
> +PTHREAD_LIBS
> +PTHREAD_LDFLAGS
> +PTHREAD_CFLAGS
>  libgcrypt
>  libext2fs
>  system_aio
> @@ -3844,6 +3847,9 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
>  
> 
> 
> +
> +
> +
>  # pkg.m4 - Macros to locate and utilise pkg-config.            -*- Autoconf -*-
>  # serial 1 (pkg-config-0.24)
>  #
> @@ -3908,6 +3914,16 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
>  
> 
> 
> +
> +
> +
> +
> +
> +
> +
> +
> +
> +
>  # Enable/disable options
>  # Check whether --enable-githttp was given.
>  if test "${enable_githttp+set}" = set; then :
> @@ -7010,47 +7026,70 @@ else
>  fi
>  
> 
> -{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lpthread" >&5
> -$as_echo_n "checking for pthread_create in -lpthread... " >&6; }
> -if test "${ac_cv_lib_pthread_pthread_create+set}" = set; then :
> +
> +    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread flag" >&5
> +$as_echo_n "checking for pthread flag... " >&6; }
> +if test "${ax_cv_pthread_flags+set}" = set; then :
>    $as_echo_n "(cached) " >&6
>  else
> -  ac_check_lib_save_LIBS=$LIBS
> -LIBS="-lpthread  $LIBS"
> -cat confdefs.h - <<_ACEOF >conftest.$ac_ext
> +
> +        ax_cv_pthread_flags=-pthread
> +
> +    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
> +    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
> +    PTHREAD_LIBS=""
> +
> +
> +    saved_CFLAGS="$CFLAGS"
> +
> +    saved_LDFLAGS="$LDFLAGS"
> +
> +    saved_LIBS="$LIBS"
> +
> +
> +    CFLAGS="$CFLAGS $PTHREAD_CFLAGS"
> +
> +    LDFLAGS="$LDFLAGS $PTHREAD_LDFLAGS"
> +
> +    LIBS="$LIBS $PTHREAD_LIBS"
> +
> +        cat confdefs.h - <<_ACEOF >conftest.$ac_ext
>  /* end confdefs.h.  */
>  
> -/* Override any GCC internal prototype to avoid an error.
> -   Use char because int might match the return type of a GCC
> -   builtin and then its argument prototype would still apply.  */
> -#ifdef __cplusplus
> -extern "C"
> -#endif
> -char pthread_create ();
> -int
> -main ()
> -{
> -return pthread_create ();
> -  ;
> -  return 0;
> +#include <pthread.h>
> +int main(void) {
> +  pthread_atfork(0,0,0);
> +  pthread_create(0,0,0,0);
>  }
> +
>  _ACEOF
>  if ac_fn_c_try_link "$LINENO"; then :
> -  ac_cv_lib_pthread_pthread_create=yes
> +
>  else
> -  ac_cv_lib_pthread_pthread_create=no
> +  ax_cv_pthread_flags=failed
>  fi
>  rm -f core conftest.err conftest.$ac_objext \
>      conftest$ac_exeext conftest.$ac_ext
> -LIBS=$ac_check_lib_save_LIBS
> -fi
> -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthread_pthread_create" >&5
> -$as_echo "$ac_cv_lib_pthread_pthread_create" >&6; }
> -if test "x$ac_cv_lib_pthread_pthread_create" = x""yes; then :
>  
> -else
> -  as_fn_error $? "Could not find libpthread" "$LINENO" 5
> +    CFLAGS="$saved_CFLAGS"
> +
> +    LDFLAGS="$saved_LDFLAGS"
> +
> +    LIBS="$saved_LIBS"
> +
> +
>  fi
> +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ax_cv_pthread_flags" >&5
> +$as_echo "$ax_cv_pthread_flags" >&6; }
> +    if test "x$ax_cv_pthread_flags" = xfailed; then
> +        as_fn_error $? "-pthread does not work" "$LINENO" 5
> +    fi
> +
> +    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
> +    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
> +    PTHREAD_LIBS=""
> +
> +
>  
>  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_gettime in -lrt" >&5
>  $as_echo_n "checking for clock_gettime in -lrt... " >&6; }
> diff --git a/tools/configure.ac b/tools/configure.ac
> index 3d53a95..3da0c82 100644
> --- a/tools/configure.ac
> +++ b/tools/configure.ac
> @@ -23,6 +23,7 @@ AC_USE_SYSTEM_EXTENSIONS
>  AC_CANONICAL_HOST
>  
>  # M4 Macro includes
> +m4_include([m4/savevar.m4])
>  m4_include([m4/enable_feature.m4])
>  m4_include([m4/disable_feature.m4])
>  m4_include([m4/path_or_fail.m4])
> @@ -34,6 +35,7 @@ m4_include([m4/set_cflags_ldflags.m4])
>  m4_include([m4/uuid.m4])
>  m4_include([m4/pkg.m4])
>  m4_include([m4/curses.m4])
> +m4_include([m4/pthread.m4])
>  
>  # Enable/disable options
>  AX_ARG_ENABLE_AND_EXPORT([githttp], [Download GIT repositories via HTTP])
> @@ -123,8 +125,7 @@ AC_CHECK_LIB([ext2fs], [ext2fs_open2], [libext2fs="y"], [libext2fs="n"])
>  AC_SUBST(libext2fs)
>  AC_CHECK_LIB([gcrypt], [gcry_md_hash_buffer], [libgcrypt="y"], [libgcrypt="n"])
>  AC_SUBST(libgcrypt)
> -AC_CHECK_LIB([pthread], [pthread_create], [] ,
> -    [AC_MSG_ERROR([Could not find libpthread])])
> +AX_CHECK_PTHREAD
>  AC_CHECK_LIB([rt], [clock_gettime])
>  AC_CHECK_LIB([yajl], [yajl_alloc], [],
>      [AC_MSG_ERROR([Could not find yajl])])
> diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
> index 55eb755..a1ba134 100644
> --- a/tools/libxc/Makefile
> +++ b/tools/libxc/Makefile
> @@ -73,6 +73,8 @@ CFLAGS   += -I. $(CFLAGS_xeninclude)
>  # Needed for posix_fadvise64() in xc_linux.c
>  CFLAGS-$(CONFIG_Linux) += -D_GNU_SOURCE
>  
> +CFLAGS	+= $(PTHREAD_CFLAGS)
> +
>  # Define this to make it possible to run valgrind on code linked with these
>  # libraries.
>  #CFLAGS   += -DVALGRIND -O0 -ggdb3
> @@ -157,7 +159,7 @@ libxenctrl.so.$(MAJOR): libxenctrl.so.$(MAJOR).$(MINOR)
>  	ln -sf $< $@
>  
>  libxenctrl.so.$(MAJOR).$(MINOR): $(CTRL_PIC_OBJS)
> -	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
> +	$(CC) $(LDFLAGS) $(PTHREAD_LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
>  
>  # libxenguest
>  
> diff --git a/tools/m4/pthread.m4 b/tools/m4/pthread.m4
> new file mode 100644
> index 0000000..57ea85c
> --- /dev/null
> +++ b/tools/m4/pthread.m4
> @@ -0,0 +1,41 @@
> +# We define, separately, PTHREAD_CFLAGS, _LDFLAGS and _LIBS
> +# even though currently we don't set them very separately.
> +# This means that the makefiles will not need to change in
> +# the future if we make the test more sophisticated.
> +
> +AC_DEFUN([AX_PTHREAD_CV2VARS],[
> +    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
> +    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
> +    PTHREAD_LIBS=""
> +])
> +
> +# We invoke AX_PTHREAD_VARS with the name of another macro
> +# which is then expanded once for each variable.
> +AC_DEFUN([AX_PTHREAD_VARS],[$1(CFLAGS) $1(LDFLAGS) $1(LIBS)])
> +
> +AC_DEFUN([AX_PTHREAD_VAR_APPLY],[
> +    $1="$$1 $PTHREAD_$1"
> +])
> +AC_DEFUN([AX_PTHREAD_VAR_SUBST],[AC_SUBST(PTHREAD_$1)])
> +
> +AC_DEFUN([AX_CHECK_PTHREAD],[
> +    AC_CACHE_CHECK([for pthread flag], [ax_cv_pthread_flags], [
> +        ax_cv_pthread_flags=-pthread
> +        AX_PTHREAD_CV2VARS
> +        AX_PTHREAD_VARS([AX_SAVEVAR_SAVE])
> +        AX_PTHREAD_VARS([AX_PTHREAD_VAR_APPLY])
> +        AC_LINK_IFELSE([
> +#include <pthread.h>
> +int main(void) {
> +  pthread_atfork(0,0,0);
> +  pthread_create(0,0,0,0);
> +}
> +],[],[ax_cv_pthread_flags=failed])
> +        AX_PTHREAD_VARS([AX_SAVEVAR_RESTORE])
> +    ])
> +    if test "x$ax_cv_pthread_flags" = xfailed; then
> +        AC_MSG_ERROR([-pthread does not work])
> +    fi
> +    AX_PTHREAD_CV2VARS
> +    AX_PTHREAD_VARS([AX_PTHREAD_VAR_SUBST])
> +])
> diff --git a/tools/m4/savevar.m4 b/tools/m4/savevar.m4
> new file mode 100644
> index 0000000..2156bee
> --- /dev/null
> +++ b/tools/m4/savevar.m4
> @@ -0,0 +1,6 @@
> +AC_DEFUN([AX_SAVEVAR_SAVE],[
> +    saved_$1="$$1"
> +])
> +AC_DEFUN([AX_SAVEVAR_RESTORE],[
> +    $1="$saved_$1"
> +])

Re: [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk"):
> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > It is not correct to say -lpthread.  The correct option is -pthread,
> > which may have sundry other effects on code generation etc.  It needs
> > to be passed both to compilation and linking.
> 
> I guess this is a gcc-ism rather than a Linux-ism and so applies to the
> BSDs too (at least so long as they use gcc and not clang)?

I think so, yes.  Some proprietary unices even have a different
compiler (cc_r) but none of them currently have any dom0 support
afaik.

Ian.

Re: [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk

[Tim Deegan]

At 16:05 +0000 on 20 Mar (1332259529), Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk"):
> > On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > > It is not correct to say -lpthread.  The correct option is -pthread,
> > > which may have sundry other effects on code generation etc.  It needs
> > > to be passed both to compilation and linking.
> > 
> > I guess this is a gcc-ism rather than a Linux-ism and so applies to the
> > BSDs too (at least so long as they use gcc and not clang)?

Clang also accepts -pthread. 

Tim.

Re: [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk

[Roger Pau Monné]

Sorry, I've mean to send this to the list, but my phone doesn't like
that, so I'm sending the mail again (you will receive this one two
times Ian Campbell).

On Mar 20, 2012 9:44 AM, "Ian Campbell" <Ian.Campbell@citrix.com> wrote:
>
> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > It is not correct to say -lpthread.  The correct option is -pthread,
> > which may have sundry other effects on code generation etc.  It needs
> > to be passed both to compilation and linking.
>
> I guess this is a gcc-ism rather than a Linux-ism and so applies to the
> BSDs too (at least so long as they use gcc and not clang)?

From what I've read (not tested) BSD is fine with -pthread, but
Solaris seems to only accept -lpthread (again from the information
I've found online).

>
> > Fix the configure test to test -pthread, and plumb the resulting flag
> > through to PTHREAD_{CFLAGS,LDFLAGS} in Tools.mk; also substitute
> > PTHREAD_LIBS (although this will currently always be empty).
> > Remove PTHREAD_LIBS setting from StdGNU.mk.
> >
> > Fix the one user (libxc) to use PTHREAD_{CFLAGS,LDFLAGS} too.
> >
> > There are still some other users in tree which pass -pthread or
> > -lpthread by adding it as a literal to their own compiler options.
> > These will be fixed in a later patch.
> >
> > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> > Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
> > ---
> >  config/StdGNU.mk     |    1 -
> >  config/Tools.mk.in   |    4 ++
> >  tools/configure      |   95 +++++++++++++++++++++++++++++++++++---------------
> >  tools/configure.ac   |    5 ++-
> >  tools/libxc/Makefile |    4 ++-
> >  tools/m4/pthread.m4  |   41 +++++++++++++++++++++
> >  tools/m4/savevar.m4  |    6 +++
> >  7 files changed, 124 insertions(+), 32 deletions(-)
> >  create mode 100644 tools/m4/pthread.m4
> >  create mode 100644 tools/m4/savevar.m4
> >
> > diff --git a/config/StdGNU.mk b/config/StdGNU.mk
> > index e2c9335..e2f2e1e 100644
> > --- a/config/StdGNU.mk
> > +++ b/config/StdGNU.mk
> > @@ -67,7 +67,6 @@ XEN_CONFIG_DIR = $(CONFIG_DIR)/xen
> >  XEN_SCRIPT_DIR = $(XEN_CONFIG_DIR)/scripts
> >
> >  SOCKET_LIBS =
> > -PTHREAD_LIBS = -lpthread
> >  UTIL_LIBS = -lutil
> >  DLOPEN_LIBS = -ldl
> >
> > diff --git a/config/Tools.mk.in b/config/Tools.mk.in
> > index 057b055..8247e4a 100644
> > --- a/config/Tools.mk.in
> > +++ b/config/Tools.mk.in
> > @@ -23,6 +23,10 @@ PREPEND_LIB         := @PREPEND_LIB@
> >  APPEND_INCLUDES     := @APPEND_INCLUDES@
> >  APPEND_LIB          := @APPEND_LIB@
> >
> > +PTHREAD_CFLAGS      := @PTHREAD_CFLAGS@
> > +PTHREAD_LDFLAGS     := @PTHREAD_LDFLAGS@
> > +PTHREAD_LIBS        := @PTHREAD_LIBS@
> > +
> >  # Download GIT repositories via HTTP or GIT's own protocol?
> >  # GIT's protocol is faster and more robust, when it works at all (firewalls
> >  # may block it). We make it the default, but if your GIT repository downloads
> > diff --git a/tools/configure b/tools/configure
> > index c686130..7ee4e3e 100755
> > --- a/tools/configure
> > +++ b/tools/configure
> > @@ -602,6 +602,9 @@ POW_LIB
> >  LIBOBJS
> >  ALLOCA
> >  libiconv
> > +PTHREAD_LIBS
> > +PTHREAD_LDFLAGS
> > +PTHREAD_CFLAGS
> >  libgcrypt
> >  libext2fs
> >  system_aio
> > @@ -3844,6 +3847,9 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
> >
> >
> >
> > +
> > +
> > +
> >  # pkg.m4 - Macros to locate and utilise pkg-config.            -*- Autoconf -*-
> >  # serial 1 (pkg-config-0.24)
> >  #
> > @@ -3908,6 +3914,16 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
> >
> >
> >
> > +
> > +
> > +
> > +
> > +
> > +
> > +
> > +
> > +
> > +
> >  # Enable/disable options
> >  # Check whether --enable-githttp was given.
> >  if test "${enable_githttp+set}" = set; then :
> > @@ -7010,47 +7026,70 @@ else
> >  fi
> >
> >
> > -{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread_create in -lpthread" >&5
> > -$as_echo_n "checking for pthread_create in -lpthread... " >&6; }
> > -if test "${ac_cv_lib_pthread_pthread_create+set}" = set; then :
> > +
> > +    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for pthread flag" >&5
> > +$as_echo_n "checking for pthread flag... " >&6; }
> > +if test "${ax_cv_pthread_flags+set}" = set; then :
> >    $as_echo_n "(cached) " >&6
> >  else
> > -  ac_check_lib_save_LIBS=$LIBS
> > -LIBS="-lpthread  $LIBS"
> > -cat confdefs.h - <<_ACEOF >conftest.$ac_ext
> > +
> > +        ax_cv_pthread_flags=-pthread
> > +
> > +    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
> > +    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
> > +    PTHREAD_LIBS=""
> > +
> > +
> > +    saved_CFLAGS="$CFLAGS"
> > +
> > +    saved_LDFLAGS="$LDFLAGS"
> > +
> > +    saved_LIBS="$LIBS"
> > +
> > +
> > +    CFLAGS="$CFLAGS $PTHREAD_CFLAGS"
> > +
> > +    LDFLAGS="$LDFLAGS $PTHREAD_LDFLAGS"
> > +
> > +    LIBS="$LIBS $PTHREAD_LIBS"
> > +
> > +        cat confdefs.h - <<_ACEOF >conftest.$ac_ext
> >  /* end confdefs.h.  */
> >
> > -/* Override any GCC internal prototype to avoid an error.
> > -   Use char because int might match the return type of a GCC
> > -   builtin and then its argument prototype would still apply.  */
> > -#ifdef __cplusplus
> > -extern "C"
> > -#endif
> > -char pthread_create ();
> > -int
> > -main ()
> > -{
> > -return pthread_create ();
> > -  ;
> > -  return 0;
> > +#include <pthread.h>
> > +int main(void) {
> > +  pthread_atfork(0,0,0);
> > +  pthread_create(0,0,0,0);
> >  }
> > +
> >  _ACEOF
> >  if ac_fn_c_try_link "$LINENO"; then :
> > -  ac_cv_lib_pthread_pthread_create=yes
> > +
> >  else
> > -  ac_cv_lib_pthread_pthread_create=no
> > +  ax_cv_pthread_flags=failed
> >  fi
> >  rm -f core conftest.err conftest.$ac_objext \
> >      conftest$ac_exeext conftest.$ac_ext
> > -LIBS=$ac_check_lib_save_LIBS
> > -fi
> > -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_pthread_pthread_create" >&5
> > -$as_echo "$ac_cv_lib_pthread_pthread_create" >&6; }
> > -if test "x$ac_cv_lib_pthread_pthread_create" = x""yes; then :
> >
> > -else
> > -  as_fn_error $? "Could not find libpthread" "$LINENO" 5
> > +    CFLAGS="$saved_CFLAGS"
> > +
> > +    LDFLAGS="$saved_LDFLAGS"
> > +
> > +    LIBS="$saved_LIBS"
> > +
> > +
> >  fi
> > +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ax_cv_pthread_flags" >&5
> > +$as_echo "$ax_cv_pthread_flags" >&6; }
> > +    if test "x$ax_cv_pthread_flags" = xfailed; then
> > +        as_fn_error $? "-pthread does not work" "$LINENO" 5
> > +    fi
> > +
> > +    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
> > +    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
> > +    PTHREAD_LIBS=""
> > +
> > +
> >
> >  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_gettime in -lrt" >&5
> >  $as_echo_n "checking for clock_gettime in -lrt... " >&6; }
> > diff --git a/tools/configure.ac b/tools/configure.ac
> > index 3d53a95..3da0c82 100644
> > --- a/tools/configure.ac
> > +++ b/tools/configure.ac
> > @@ -23,6 +23,7 @@ AC_USE_SYSTEM_EXTENSIONS
> >  AC_CANONICAL_HOST
> >
> >  # M4 Macro includes
> > +m4_include([m4/savevar.m4])
> >  m4_include([m4/enable_feature.m4])
> >  m4_include([m4/disable_feature.m4])
> >  m4_include([m4/path_or_fail.m4])
> > @@ -34,6 +35,7 @@ m4_include([m4/set_cflags_ldflags.m4])
> >  m4_include([m4/uuid.m4])
> >  m4_include([m4/pkg.m4])
> >  m4_include([m4/curses.m4])
> > +m4_include([m4/pthread.m4])
> >
> >  # Enable/disable options
> >  AX_ARG_ENABLE_AND_EXPORT([githttp], [Download GIT repositories via HTTP])
> > @@ -123,8 +125,7 @@ AC_CHECK_LIB([ext2fs], [ext2fs_open2], [libext2fs="y"], [libext2fs="n"])
> >  AC_SUBST(libext2fs)
> >  AC_CHECK_LIB([gcrypt], [gcry_md_hash_buffer], [libgcrypt="y"], [libgcrypt="n"])
> >  AC_SUBST(libgcrypt)
> > -AC_CHECK_LIB([pthread], [pthread_create], [] ,
> > -    [AC_MSG_ERROR([Could not find libpthread])])
> > +AX_CHECK_PTHREAD
> >  AC_CHECK_LIB([rt], [clock_gettime])
> >  AC_CHECK_LIB([yajl], [yajl_alloc], [],
> >      [AC_MSG_ERROR([Could not find yajl])])
> > diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
> > index 55eb755..a1ba134 100644
> > --- a/tools/libxc/Makefile
> > +++ b/tools/libxc/Makefile
> > @@ -73,6 +73,8 @@ CFLAGS   += -I. $(CFLAGS_xeninclude)
> >  # Needed for posix_fadvise64() in xc_linux.c
> >  CFLAGS-$(CONFIG_Linux) += -D_GNU_SOURCE
> >
> > +CFLAGS       += $(PTHREAD_CFLAGS)
> > +
> >  # Define this to make it possible to run valgrind on code linked with these
> >  # libraries.
> >  #CFLAGS   += -DVALGRIND -O0 -ggdb3
> > @@ -157,7 +159,7 @@ libxenctrl.so.$(MAJOR): libxenctrl.so.$(MAJOR).$(MINOR)
> >       ln -sf $< $@
> >
> >  libxenctrl.so.$(MAJOR).$(MINOR): $(CTRL_PIC_OBJS)
> > -     $(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
> > +     $(CC) $(LDFLAGS) $(PTHREAD_LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libxenctrl.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(DLOPEN_LIBS) $(PTHREAD_LIBS) $(APPEND_LDFLAGS)
> >
> >  # libxenguest
> >
> > diff --git a/tools/m4/pthread.m4 b/tools/m4/pthread.m4
> > new file mode 100644
> > index 0000000..57ea85c
> > --- /dev/null
> > +++ b/tools/m4/pthread.m4
> > @@ -0,0 +1,41 @@
> > +# We define, separately, PTHREAD_CFLAGS, _LDFLAGS and _LIBS
> > +# even though currently we don't set them very separately.
> > +# This means that the makefiles will not need to change in
> > +# the future if we make the test more sophisticated.
> > +
> > +AC_DEFUN([AX_PTHREAD_CV2VARS],[
> > +    PTHREAD_CFLAGS="$ax_cv_pthread_flags"
> > +    PTHREAD_LDFLAGS="$ax_cv_pthread_flags"
> > +    PTHREAD_LIBS=""
> > +])
> > +
> > +# We invoke AX_PTHREAD_VARS with the name of another macro
> > +# which is then expanded once for each variable.
> > +AC_DEFUN([AX_PTHREAD_VARS],[$1(CFLAGS) $1(LDFLAGS) $1(LIBS)])
> > +
> > +AC_DEFUN([AX_PTHREAD_VAR_APPLY],[
> > +    $1="$$1 $PTHREAD_$1"
> > +])
> > +AC_DEFUN([AX_PTHREAD_VAR_SUBST],[AC_SUBST(PTHREAD_$1)])
> > +
> > +AC_DEFUN([AX_CHECK_PTHREAD],[
> > +    AC_CACHE_CHECK([for pthread flag], [ax_cv_pthread_flags], [
> > +        ax_cv_pthread_flags=-pthread
> > +        AX_PTHREAD_CV2VARS
> > +        AX_PTHREAD_VARS([AX_SAVEVAR_SAVE])
> > +        AX_PTHREAD_VARS([AX_PTHREAD_VAR_APPLY])
> > +        AC_LINK_IFELSE([
> > +#include <pthread.h>
> > +int main(void) {
> > +  pthread_atfork(0,0,0);
> > +  pthread_create(0,0,0,0);
> > +}
> > +],[],[ax_cv_pthread_flags=failed])
> > +        AX_PTHREAD_VARS([AX_SAVEVAR_RESTORE])
> > +    ])
> > +    if test "x$ax_cv_pthread_flags" = xfailed; then
> > +        AC_MSG_ERROR([-pthread does not work])
> > +    fi
> > +    AX_PTHREAD_CV2VARS
> > +    AX_PTHREAD_VARS([AX_PTHREAD_VAR_SUBST])
> > +])
> > diff --git a/tools/m4/savevar.m4 b/tools/m4/savevar.m4
> > new file mode 100644
> > index 0000000..2156bee
> > --- /dev/null
> > +++ b/tools/m4/savevar.m4
> > @@ -0,0 +1,6 @@
> > +AC_DEFUN([AX_SAVEVAR_SAVE],[
> > +    saved_$1="$$1"
> > +])
> > +AC_DEFUN([AX_SAVEVAR_RESTORE],[
> > +    $1="$saved_$1"
> > +])
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH 06/20] tools: Correct PTHREAD options in config/StdGNU.mk

[Ian Campbell]

On Tue, 2012-03-20 at 16:47 +0000, Roger Pau Monné wrote:
> Sorry, I've mean to send this to the list, but my phone doesn't like
> that, so I'm sending the mail again (you will receive this one two
> times Ian Campbell).

No problem.

> On Mar 20, 2012 9:44 AM, "Ian Campbell" <Ian.Campbell@citrix.com> wrote:
> >
> > On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > > It is not correct to say -lpthread.  The correct option is -pthread,
> > > which may have sundry other effects on code generation etc.  It needs
> > > to be passed both to compilation and linking.
> >
> > I guess this is a gcc-ism rather than a Linux-ism and so applies to the
> > BSDs too (at least so long as they use gcc and not clang)?
> 
> From what I've read (not tested) BSD is fine with -pthread, but
> Solaris seems to only accept -lpthread (again from the information
> I've found online).

I think we should end the pretence that Solaris is a supported dom0
platform any more. I'm reasonably certain it won't even build.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH 07/20] libxl: Use PTHREAD_CFLAGS, LDFLAGS, LIBS

[Ian Jackson]

This is going to be needed for pthread_atfork.  It is a mystery why it
hasn't been needed before.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/Makefile |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index e44fcfa..80c36ad 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -22,6 +22,10 @@ endif
 LIBXL_LIBS =
 LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(UTIL_LIBS) $(LIBUUID_LIBS)
 
+CFLAGS += $(PTHREAD_CFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
+LIBXL_LIBS += $(PTHREAD_LIBS)
+
 LIBXLU_LIBS =
 
 LIBXL_OBJS-y = osdeps.o libxl_paths.o libxl_bootloader.o flexarray.o
-- 
1.7.2.5

Re: [PATCH 07/20] libxl: Use PTHREAD_CFLAGS, LDFLAGS, LIBS

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> This is going to be needed for pthread_atfork.  It is a mystery why it
> hasn't been needed before.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/Makefile |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
> 
> diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
> index e44fcfa..80c36ad 100644
> --- a/tools/libxl/Makefile
> +++ b/tools/libxl/Makefile
> @@ -22,6 +22,10 @@ endif
>  LIBXL_LIBS =
>  LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(UTIL_LIBS) $(LIBUUID_LIBS)
>  
> +CFLAGS += $(PTHREAD_CFLAGS)
> +LDFLAGS += $(PTHREAD_LDFLAGS)
> +LIBXL_LIBS += $(PTHREAD_LIBS)
> +
>  LIBXLU_LIBS =
>  
>  LIBXL_OBJS-y = osdeps.o libxl_paths.o libxl_bootloader.o flexarray.o

[PATCH 08/20] tools: Use PTHREAD_CFLAGS, _LDFLAGS, _LIBS

[Ian Jackson]

Replace all literal occurrences of -lpthread and -pthread in Makefiles
by references to PTHREAD_CFLAGS, PTHREAD_LDFLAGS and PTHREAD_LIBS.
These are the new variables set by configure, and currently expand to
-pthread on the compilation and link lines as is required.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/blktap/drivers/Makefile       |    7 +++++--
 tools/libfsimage/common/Makefile    |    5 ++++-
 tools/vtpm_manager/manager/Makefile |    4 +++-
 tools/xenpaging/Makefile            |    5 +++--
 4 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/tools/blktap/drivers/Makefile b/tools/blktap/drivers/Makefile
index d2bd02f..7535011 100644
--- a/tools/blktap/drivers/Makefile
+++ b/tools/blktap/drivers/Makefile
@@ -35,8 +35,11 @@ else
 AIOLIBS     := -laio
 endif
 
-LDLIBS_blktapctrl := $(MEMSHRLIBS) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) -L../lib -lblktap -lrt -lm -lpthread
-LDLIBS_img := $(AIOLIBS) $(CRYPT_LIB) -lpthread -lz
+CFLAGS += $(PTHREAD_CFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
+
+LDLIBS_blktapctrl := $(MEMSHRLIBS) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) -L../lib -lblktap -lrt -lm $(PTHREAD_LIBS)
+LDLIBS_img := $(AIOLIBS) $(CRYPT_LIB) $(PTHREAD_LIBS) -lz
 
 BLK-OBJS-y  := block-aio.o
 BLK-OBJS-y  += block-sync.o
diff --git a/tools/libfsimage/common/Makefile b/tools/libfsimage/common/Makefile
index 11621e7..3684913 100644
--- a/tools/libfsimage/common/Makefile
+++ b/tools/libfsimage/common/Makefile
@@ -8,6 +8,9 @@ LDFLAGS-$(CONFIG_SunOS) = -Wl,-M -Wl,mapfile-SunOS
 LDFLAGS-$(CONFIG_Linux) = -Wl,mapfile-GNU
 LDFLAGS = $(LDFLAGS-y)
 
+CFLAGS += $(PTHREAD_CFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
+
 LIB_SRCS-y = fsimage.c fsimage_plugin.c fsimage_grub.c
 
 PIC_OBJS := $(patsubst %.c,%.opic,$(LIB_SRCS-y))
@@ -37,7 +40,7 @@ libfsimage.so.$(MAJOR): libfsimage.so.$(MAJOR).$(MINOR)
 	ln -sf $< $@
 
 libfsimage.so.$(MAJOR).$(MINOR): $(PIC_OBJS)
-	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libfsimage.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ -lpthread
+	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libfsimage.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(PTHREAD_LIBS)
 
 -include $(DEPS)
 
diff --git a/tools/vtpm_manager/manager/Makefile b/tools/vtpm_manager/manager/Makefile
index 149f01d..7564af1 100644
--- a/tools/vtpm_manager/manager/Makefile
+++ b/tools/vtpm_manager/manager/Makefile
@@ -33,4 +33,6 @@ $(BIN): $(OBJS)
 
 # libraries
 LIBS += ../tcs/libTCS.a ../util/libTCGUtils.a ../crypto/libtcpaCrypto.a
-LIBS += -lcrypto -lpthread -lm
+LIBS += -lcrypto $(PTHREAD_LIBS) -lm
+CFLAGS += $(PTHREAD_CFLAGS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
diff --git a/tools/xenpaging/Makefile b/tools/xenpaging/Makefile
index 08230a6..548d9dd 100644
--- a/tools/xenpaging/Makefile
+++ b/tools/xenpaging/Makefile
@@ -1,8 +1,9 @@
 XEN_ROOT=$(CURDIR)/../..
 include $(XEN_ROOT)/tools/Rules.mk
 
-CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenstore)
-LDLIBS += $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) -pthread
+CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenstore) $(PTHREAD_CFLAGS)
+LDLIBS += $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(PTHREAD_LIBS)
+LDFLAGS += $(PTHREAD_LDFLAGS)
 
 POLICY    = default
 
-- 
1.7.2.5

Re: [PATCH 08/20] tools: Use PTHREAD_CFLAGS, _LDFLAGS, _LIBS

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> Replace all literal occurrences of -lpthread and -pthread in Makefiles
> by references to PTHREAD_CFLAGS, PTHREAD_LDFLAGS and PTHREAD_LIBS.
> These are the new variables set by configure, and currently expand to
> -pthread on the compilation and link lines as is required.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/blktap/drivers/Makefile       |    7 +++++--
>  tools/libfsimage/common/Makefile    |    5 ++++-
>  tools/vtpm_manager/manager/Makefile |    4 +++-
>  tools/xenpaging/Makefile            |    5 +++--
>  4 files changed, 15 insertions(+), 6 deletions(-)
> 
> diff --git a/tools/blktap/drivers/Makefile b/tools/blktap/drivers/Makefile
> index d2bd02f..7535011 100644
> --- a/tools/blktap/drivers/Makefile
> +++ b/tools/blktap/drivers/Makefile
> @@ -35,8 +35,11 @@ else
>  AIOLIBS     := -laio
>  endif
>  
> -LDLIBS_blktapctrl := $(MEMSHRLIBS) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) -L../lib -lblktap -lrt -lm -lpthread
> -LDLIBS_img := $(AIOLIBS) $(CRYPT_LIB) -lpthread -lz
> +CFLAGS += $(PTHREAD_CFLAGS)
> +LDFLAGS += $(PTHREAD_LDFLAGS)
> +
> +LDLIBS_blktapctrl := $(MEMSHRLIBS) $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) -L../lib -lblktap -lrt -lm $(PTHREAD_LIBS)
> +LDLIBS_img := $(AIOLIBS) $(CRYPT_LIB) $(PTHREAD_LIBS) -lz
>  
>  BLK-OBJS-y  := block-aio.o
>  BLK-OBJS-y  += block-sync.o
> diff --git a/tools/libfsimage/common/Makefile b/tools/libfsimage/common/Makefile
> index 11621e7..3684913 100644
> --- a/tools/libfsimage/common/Makefile
> +++ b/tools/libfsimage/common/Makefile
> @@ -8,6 +8,9 @@ LDFLAGS-$(CONFIG_SunOS) = -Wl,-M -Wl,mapfile-SunOS
>  LDFLAGS-$(CONFIG_Linux) = -Wl,mapfile-GNU
>  LDFLAGS = $(LDFLAGS-y)
>  
> +CFLAGS += $(PTHREAD_CFLAGS)
> +LDFLAGS += $(PTHREAD_LDFLAGS)
> +
>  LIB_SRCS-y = fsimage.c fsimage_plugin.c fsimage_grub.c
>  
>  PIC_OBJS := $(patsubst %.c,%.opic,$(LIB_SRCS-y))
> @@ -37,7 +40,7 @@ libfsimage.so.$(MAJOR): libfsimage.so.$(MAJOR).$(MINOR)
>  	ln -sf $< $@
>  
>  libfsimage.so.$(MAJOR).$(MINOR): $(PIC_OBJS)
> -	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libfsimage.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ -lpthread
> +	$(CC) $(LDFLAGS) -Wl,$(SONAME_LDFLAG) -Wl,libfsimage.so.$(MAJOR) $(SHLIB_LDFLAGS) -o $@ $^ $(PTHREAD_LIBS)
>  
>  -include $(DEPS)
>  
> diff --git a/tools/vtpm_manager/manager/Makefile b/tools/vtpm_manager/manager/Makefile
> index 149f01d..7564af1 100644
> --- a/tools/vtpm_manager/manager/Makefile
> +++ b/tools/vtpm_manager/manager/Makefile
> @@ -33,4 +33,6 @@ $(BIN): $(OBJS)
>  
>  # libraries
>  LIBS += ../tcs/libTCS.a ../util/libTCGUtils.a ../crypto/libtcpaCrypto.a
> -LIBS += -lcrypto -lpthread -lm
> +LIBS += -lcrypto $(PTHREAD_LIBS) -lm
> +CFLAGS += $(PTHREAD_CFLAGS)
> +LDFLAGS += $(PTHREAD_LDFLAGS)
> diff --git a/tools/xenpaging/Makefile b/tools/xenpaging/Makefile
> index 08230a6..548d9dd 100644
> --- a/tools/xenpaging/Makefile
> +++ b/tools/xenpaging/Makefile
> @@ -1,8 +1,9 @@
>  XEN_ROOT=$(CURDIR)/../..
>  include $(XEN_ROOT)/tools/Rules.mk
>  
> -CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenstore)
> -LDLIBS += $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) -pthread
> +CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenstore) $(PTHREAD_CFLAGS)
> +LDLIBS += $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(PTHREAD_LIBS)
> +LDFLAGS += $(PTHREAD_LDFLAGS)
>  
>  POLICY    = default
>  

[PATCH 09/20] libxl: Crash (more sensibly) on malloc failure

[Ian Jackson]

Formally change the libxl memory allocation failure policy to "crash".

Previously we had a very uneven approach; much code assumed that
libxl__sprintf (for example) would never return NULL, but some code
was written more carefully.

We think it is unlikely that we will be able to make the library
actually robust against allocation failure (since that would be an
awful lot of never-tested error paths) and few calling environments
will be able to cope anyway.  So, instead, adopt the alternative
approach: provide allocation functions which never return null, but
will crash the whole process instead.

Consequently,
 - New noreturn function libxl__alloc_failed which may be used for
   printing a vaguely-useful error message, rather than simply
   dereferencing a null pointer.
 - libxl__ptr_add now returns void as it crashes on failure.
 - libxl__zalloc, _calloc, _strdup, _strndup, crash on failure using
   libxl__alloc_failed.  So all the code that uses these can no longer
   dereference null on malloc failure.

While we're at it, make libxl__ptr_add use realloc rather than
emulating it with calloc and free, and make it grow the array
exponentially rather than linearly.

Things left to do:
 - Provide versions of malloc, realloc and free which do the
   checking but which do not enroll the pointer in the gc.
 - Remove a lot of now-spurious error handling.
 - Remove the ERROR_NOMEM error code.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl.h          |    4 ++
 tools/libxl/libxl_internal.c |   79 ++++++++++++++++++------------------------
 tools/libxl/libxl_internal.h |    5 ++-
 3 files changed, 42 insertions(+), 46 deletions(-)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index dce087c..985806b 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -123,6 +123,10 @@
  * No temporary objects allocated from the pool may be explicitly freed.
  * Therefore public functions which initialize a libxl__gc MUST call
  * libxl__free_all() before returning.
+ *
+ * Memory allocation failures are not handled gracefully.  If malloc
+ * (or realloc) fails, libxl will cause the entire process to print
+ * a message to stderr and exit with status 255.
  */
 /*
  * libxl types
diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c
index 12c32dc..dfa2153 100644
--- a/tools/libxl/libxl_internal.c
+++ b/tools/libxl/libxl_internal.c
@@ -17,40 +17,40 @@
 
 #include "libxl_internal.h"
 
-int libxl__error_set(libxl__gc *gc, int code)
-{
-    return 0;
+void libxl__alloc_failed(const char *func, size_t nmemb, size_t size) {
+    fprintf(stderr,
+            "libxl: FATAL ERROR: memory allocation failure (%s, %lu x %lu)\n",
+            func, (unsigned long)nmemb, (unsigned long)size);
+    fflush(stderr);
+    _exit(-1);
 }
 
-int libxl__ptr_add(libxl__gc *gc, void *ptr)
+void libxl__ptr_add(libxl__gc *gc, void *ptr)
 {
     int i;
-    void **re;
 
     if (!ptr)
-        return 0;
+        return;
 
     /* fast case: we have space in the array for storing the pointer */
     for (i = 0; i < gc->alloc_maxsize; i++) {
         if (!gc->alloc_ptrs[i]) {
             gc->alloc_ptrs[i] = ptr;
-            return 0;
+            return;
         }
     }
-    /* realloc alloc_ptrs manually with calloc/free/replace */
-    re = calloc(gc->alloc_maxsize + 25, sizeof(void *));
-    if (!re)
-        return -1;
-    for (i = 0; i < gc->alloc_maxsize; i++)
-        re[i] = gc->alloc_ptrs[i];
-    /* assign the next pointer */
-    re[i] = ptr;
+    int new_maxsize = gc->alloc_maxsize * 2 + 25;
+    assert(new_maxsize < INT_MAX / sizeof(void*) / 2);
+    gc->alloc_ptrs = realloc(gc->alloc_ptrs, new_maxsize * sizeof(void *));
+    if (!gc->alloc_ptrs)
+        libxl__alloc_failed(__func__, sizeof(void*), new_maxsize);
 
-    /* replace the old alloc_ptr */
-    free(gc->alloc_ptrs);
-    gc->alloc_ptrs = re;
-    gc->alloc_maxsize += 25;
-    return 0;
+    gc->alloc_ptrs[gc->alloc_maxsize++] = ptr;
+
+    while (gc->alloc_maxsize < new_maxsize)
+        gc->alloc_ptrs[gc->alloc_maxsize++] = 0;
+
+    return;
 }
 
 void libxl__free_all(libxl__gc *gc)
@@ -71,10 +71,7 @@ void libxl__free_all(libxl__gc *gc)
 void *libxl__zalloc(libxl__gc *gc, int bytes)
 {
     void *ptr = calloc(bytes, 1);
-    if (!ptr) {
-        libxl__error_set(gc, ENOMEM);
-        return NULL;
-    }
+    if (!ptr) libxl__alloc_failed(__func__, bytes, 1);
 
     libxl__ptr_add(gc, ptr);
     return ptr;
@@ -83,10 +80,7 @@ void *libxl__zalloc(libxl__gc *gc, int bytes)
 void *libxl__calloc(libxl__gc *gc, size_t nmemb, size_t size)
 {
     void *ptr = calloc(nmemb, size);
-    if (!ptr) {
-        libxl__error_set(gc, ENOMEM);
-        return NULL;
-    }
+    if (!ptr) libxl__alloc_failed(__func__, nmemb, size);
 
     libxl__ptr_add(gc, ptr);
     return ptr;
@@ -97,9 +91,8 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size)
     void *new_ptr = realloc(ptr, new_size);
     int i = 0;
 
-    if (new_ptr == NULL && new_size != 0) {
-        return NULL;
-    }
+    if (new_ptr == NULL && new_size != 0)
+        libxl__alloc_failed(__func__, new_size, 1);
 
     if (ptr == NULL) {
         libxl__ptr_add(gc, new_ptr);
@@ -112,7 +105,6 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size)
         }
     }
 
-
     return new_ptr;
 }
 
@@ -126,16 +118,13 @@ char *libxl__sprintf(libxl__gc *gc, const char *fmt, ...)
     ret = vsnprintf(NULL, 0, fmt, ap);
     va_end(ap);
 
-    if (ret < 0) {
-        return NULL;
-    }
+    assert(ret >= 0);
 
     s = libxl__zalloc(gc, ret + 1);
-    if (s) {
-        va_start(ap, fmt);
-        ret = vsnprintf(s, ret + 1, fmt, ap);
-        va_end(ap);
-    }
+    va_start(ap, fmt);
+    ret = vsnprintf(s, ret + 1, fmt, ap);
+    va_end(ap);
+
     return s;
 }
 
@@ -143,8 +132,9 @@ char *libxl__strdup(libxl__gc *gc, const char *c)
 {
     char *s = strdup(c);
 
-    if (s)
-        libxl__ptr_add(gc, s);
+    if (!s) libxl__alloc_failed(__func__, strlen(c), 1);
+
+    libxl__ptr_add(gc, s);
 
     return s;
 }
@@ -153,8 +143,7 @@ char *libxl__strndup(libxl__gc *gc, const char *c, size_t n)
 {
     char *s = strndup(c, n);
 
-    if (s)
-        libxl__ptr_add(gc, s);
+    if (!s) libxl__alloc_failed(__func__, n, 1);
 
     return s;
 }
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 04f13f6..e59a19b 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -116,6 +116,9 @@ typedef struct libxl__gc libxl__gc;
 typedef struct libxl__egc libxl__egc;
 typedef struct libxl__ao libxl__ao;
 
+void libxl__alloc_failed(const char *func, size_t nmemb, size_t size)
+    __attribute__((noreturn));
+
 typedef struct libxl__ev_fd libxl__ev_fd;
 typedef void libxl__ev_fd_callback(libxl__egc *egc, libxl__ev_fd *ev,
                                    int fd, short events, short revents);
@@ -380,7 +383,7 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc)
  * collection on exit from the outermost libxl callframe.
  */
 /* register @ptr in @gc for free on exit from outermost libxl callframe. */
-_hidden int libxl__ptr_add(libxl__gc *gc, void *ptr);
+_hidden void libxl__ptr_add(libxl__gc *gc, void *ptr);
 /* if this is the outermost libxl callframe then free all pointers in @gc */
 _hidden void libxl__free_all(libxl__gc *gc);
 /* allocate and zero @bytes. (similar to a gc'd malloc(3)+memzero()) */
-- 
1.7.2.5

Re: [PATCH 09/20] libxl: Crash (more sensibly) on malloc failure

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> Formally change the libxl memory allocation failure policy to "crash".
> 
> Previously we had a very uneven approach; much code assumed that
> libxl__sprintf (for example) would never return NULL, but some code
> was written more carefully.
> 
> We think it is unlikely that we will be able to make the library
> actually robust against allocation failure (since that would be an
> awful lot of never-tested error paths) and few calling environments
> will be able to cope anyway.  So, instead, adopt the alternative
> approach: provide allocation functions which never return null, but
> will crash the whole process instead.
> 
> Consequently,
>  - New noreturn function libxl__alloc_failed which may be used for
>    printing a vaguely-useful error message, rather than simply
>    dereferencing a null pointer.

I guess the log infrastructure does (or could do) memory allocation and
so can't be used here in the normal way.

Perhaps libxl could format a (short) message into a static emergency
buffer and use a (new, optional) emergency variant of the logger
callback which does not accept a format string, which should maximise
the chances of writing to the log? Even if we could just get the message
"Out of memory, aborting" into the logs that would be useful.

Alternatively perhaps an application could be allowed to provide a
function which is called first in libxl__alloc_failed, the application
ought should be in a better position to fprintf to the appropriate fd to
log the issue somewhere useful.

My concern is that many users of libxl will not necessarily be capturing
stderr and so the appearance will be of random unexplained exiting.

>  - libxl__ptr_add now returns void as it crashes on failure.
>  - libxl__zalloc, _calloc, _strdup, _strndup, crash on failure using
>    libxl__alloc_failed.  So all the code that uses these can no longer
>    dereference null on malloc failure.

I took a look for a gcc __attribute__ which means "cannot return NULL"
but sadly there doesn't seem to be one, this would have allowed gcc to
warn us about (now) pointless error handling. I don't know if gcc is
smart enough to catch this if we make each of these a macro which did
	if (!res) abort()
(or something similar) after calling the inner-function which does the
work, I don't think it's worth the effort anyway.

> diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c
> index 12c32dc..dfa2153 100644
> --- a/tools/libxl/libxl_internal.c
> +++ b/tools/libxl/libxl_internal.c
> @@ -17,40 +17,40 @@
>  
>  #include "libxl_internal.h"
>  
> -int libxl__error_set(libxl__gc *gc, int code)
> -{
> -    return 0;
> +void libxl__alloc_failed(const char *func, size_t nmemb, size_t size) {
> +    fprintf(stderr,
> +            "libxl: FATAL ERROR: memory allocation failure (%s, %lu x %lu)\n",
> +            func, (unsigned long)nmemb, (unsigned long)size);
> +    fflush(stderr);
> +    _exit(-1);
>  }
>  
> -int libxl__ptr_add(libxl__gc *gc, void *ptr)
> +void libxl__ptr_add(libxl__gc *gc, void *ptr)
>  {
>      int i;
> -    void **re;
>  
>      if (!ptr)
> -        return 0;
> +        return;
>  
>      /* fast case: we have space in the array for storing the pointer */
>      for (i = 0; i < gc->alloc_maxsize; i++) {
>          if (!gc->alloc_ptrs[i]) {
>              gc->alloc_ptrs[i] = ptr;
> -            return 0;
> +            return;
>          }
>      }
> -    /* realloc alloc_ptrs manually with calloc/free/replace */
> -    re = calloc(gc->alloc_maxsize + 25, sizeof(void *));
> -    if (!re)
> -        return -1;
> -    for (i = 0; i < gc->alloc_maxsize; i++)
> -        re[i] = gc->alloc_ptrs[i];
> -    /* assign the next pointer */
> -    re[i] = ptr;
> +    int new_maxsize = gc->alloc_maxsize * 2 + 25;
> +    assert(new_maxsize < INT_MAX / sizeof(void*) / 2);

This is not exactly the same as ENOMEM but there is an argument for
treating it similarly rather than asserting? I suppose the counter
argument is that the cap on max size is so huge that we "assert" it can
never be reached in normal use?

[...]

> @@ -126,16 +118,13 @@ char *libxl__sprintf(libxl__gc *gc, const char *fmt, ...)
>      ret = vsnprintf(NULL, 0, fmt, ap);
>      va_end(ap);
>  
> -    if (ret < 0) {
> -        return NULL;
> -    }
> +    assert(ret >= 0);

If ret is -ve then it can be ENOMEM in which case we should call the
emergency function?

Oh, it can't be ENOMEM because you called with NULL. I guess all the
other possible returns (EILSEQ, EINVAL, EOVERFLOW) are abort-worthy.

>  
>      s = libxl__zalloc(gc, ret + 1);
> -    if (s) {
> -        va_start(ap, fmt);
> -        ret = vsnprintf(s, ret + 1, fmt, ap);
> -        va_end(ap);
> -    }
> +    va_start(ap, fmt);
> +    ret = vsnprintf(s, ret + 1, fmt, ap);

No check on ret here? In this case I think it can be ENOMEM.

> +    va_end(ap);
> +
>      return s;
>  }
>  

Re: [PATCH 09/20] libxl: Crash (more sensibly) on malloc failure

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 09/20] libxl: Crash (more sensibly) on malloc failure"):
> I guess the log infrastructure does (or could do) memory allocation and
> so can't be used here in the normal way.

Yes.

> Perhaps libxl could format a (short) message into a static emergency
> buffer and use a (new, optional) emergency variant of the logger
> callback which does not accept a format string, which should maximise
> the chances of writing to the log? Even if we could just get the message
> "Out of memory, aborting" into the logs that would be useful.

In fact libxl__logv already has this special case built-in.

> My concern is that many users of libxl will not necessarily be capturing
> stderr and so the appearance will be of random unexplained exiting.

Yes.  Good point.  I will fix this, and have it call libxl__log as
well as printing to stderr.

> >  - libxl__ptr_add now returns void as it crashes on failure.
> >  - libxl__zalloc, _calloc, _strdup, _strndup, crash on failure using
> >    libxl__alloc_failed.  So all the code that uses these can no longer
> >    dereference null on malloc failure.
> 
> I took a look for a gcc __attribute__ which means "cannot return NULL"
> but sadly there doesn't seem to be one, this would have allowed gcc to
> warn us about (now) pointless error handling.

Do we really need a warning about this ?  It's harmless.

> I don't know if gcc is
> smart enough to catch this if we make each of these a macro which did
> 	if (!res) abort()
> (or something similar) after calling the inner-function which does the
> work, I don't think it's worth the effort anyway.

Quite.

Ian.

Re: [PATCH 09/20] libxl: Crash (more sensibly) on malloc failure

[Ian Campbell]

On Tue, 2012-03-20 at 16:22 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 09/20] libxl: Crash (more sensibly) on malloc failure"):
> > I guess the log infrastructure does (or could do) memory allocation and
> > so can't be used here in the normal way.
> 
> Yes.
> 
> > Perhaps libxl could format a (short) message into a static emergency
> > buffer and use a (new, optional) emergency variant of the logger
> > callback which does not accept a format string, which should maximise
> > the chances of writing to the log? Even if we could just get the message
> > "Out of memory, aborting" into the logs that would be useful.
> 
> In fact libxl__logv already has this special case built-in.
> 
> > My concern is that many users of libxl will not necessarily be capturing
> > stderr and so the appearance will be of random unexplained exiting.
> 
> Yes.  Good point.  I will fix this, and have it call libxl__log as
> well as printing to stderr.

Great, thanks.

> > >  - libxl__ptr_add now returns void as it crashes on failure.
> > >  - libxl__zalloc, _calloc, _strdup, _strndup, crash on failure using
> > >    libxl__alloc_failed.  So all the code that uses these can no longer
> > >    dereference null on malloc failure.
> > 
> > I took a look for a gcc __attribute__ which means "cannot return NULL"
> > but sadly there doesn't seem to be one, this would have allowed gcc to
> > warn us about (now) pointless error handling.
> 
> Do we really need a warning about this ?  It's harmless.

More as a guide to help us to easily find and nuke all the unnecessary
error handling, rather than because it's something to worry about.

> > I don't know if gcc is
> > smart enough to catch this if we make each of these a macro which did
> > 	if (!res) abort()
> > (or something similar) after calling the inner-function which does the
> > work, I don't think it's worth the effort anyway.
> 
> Quite.
> 
> Ian.

[PATCH 10/20] libxl: Make libxl__zalloc et al tolerate a NULL gc

[Ian Jackson]

Arrange that if we pass NULL as a gc, we simply don't register the
pointer.  This instantly gives us non-gc'ing but error-checking
versions of malloc, realloc, vasprintf, etc.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_internal.c |    5 ++++-
 tools/libxl/libxl_internal.h |   21 +++++++++++++--------
 2 files changed, 17 insertions(+), 9 deletions(-)

diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c
index dfa2153..de7c3a8 100644
--- a/tools/libxl/libxl_internal.c
+++ b/tools/libxl/libxl_internal.c
@@ -29,6 +29,9 @@ void libxl__ptr_add(libxl__gc *gc, void *ptr)
 {
     int i;
 
+    if (!gc)
+        return;
+
     if (!ptr)
         return;
 
@@ -96,7 +99,7 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size)
 
     if (ptr == NULL) {
         libxl__ptr_add(gc, new_ptr);
-    } else if (new_ptr != ptr) {
+    } else if (new_ptr != ptr && gc != NULL) {
         for (i = 0; i < gc->alloc_maxsize; i++) {
             if (gc->alloc_ptrs[i] == ptr) {
                 gc->alloc_ptrs[i] = new_ptr;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index e59a19b..86f25be 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -381,30 +381,35 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc)
  *
  * All pointers returned by these functions are registered for garbage
  * collection on exit from the outermost libxl callframe.
+ *
+ * However, where the argument is stated to be "gc_opt", NULL may be
+ * passed instead, in which case no garbage collection will occur; the
+ * pointer must later be freed with free().  This is for memory
+ * allocations of types (b) and (c).
  */
 /* register @ptr in @gc for free on exit from outermost libxl callframe. */
-_hidden void libxl__ptr_add(libxl__gc *gc, void *ptr);
+_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr);
 /* if this is the outermost libxl callframe then free all pointers in @gc */
 _hidden void libxl__free_all(libxl__gc *gc);
 /* allocate and zero @bytes. (similar to a gc'd malloc(3)+memzero()) */
-_hidden void *libxl__zalloc(libxl__gc *gc, int bytes);
+_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes);
 /* allocate and zero memory for an array of @nmemb members of @size each.
  * (similar to a gc'd calloc(3)). */
-_hidden void *libxl__calloc(libxl__gc *gc, size_t nmemb, size_t size);
+_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size);
 /* change the size of the memory block pointed to by @ptr to @new_size bytes.
  * unlike other allocation functions here any additional space between the
  * oldsize and @new_size is not initialised (similar to a gc'd realloc(3)). */
-_hidden void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size);
+_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size);
 /* print @fmt into an allocated string large enoughto contain the result.
  * (similar to gc'd asprintf(3)). */
-_hidden char *libxl__sprintf(libxl__gc *gc, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3);
+_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3);
 /* duplicate the string @c (similar to a gc'd strdup(3)). */
-_hidden char *libxl__strdup(libxl__gc *gc, const char *c);
+_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c);
 /* duplicate at most @n bytes of string @c (similar to a gc'd strndup(3)). */
-_hidden char *libxl__strndup(libxl__gc *gc, const char *c, size_t n);
+_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n);
 /* strip the last path component from @s and return as a newly allocated
  * string. (similar to a gc'd dirname(3)). */
-_hidden char *libxl__dirname(libxl__gc *gc, const char *s);
+_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s);
 
 _hidden char **libxl__xs_kvs_of_flexarray(libxl__gc *gc, flexarray_t *array, int length);
 
-- 
1.7.2.5

Re: [PATCH 10/20] libxl: Make libxl__zalloc et al tolerate a NULL gc

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> Arrange that if we pass NULL as a gc, we simply don't register the
> pointer.  This instantly gives us non-gc'ing but error-checking
> versions of malloc, realloc, vasprintf, etc.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

We could consider __attribute__((nonnull)) for the other helper
functions in that section of libxl_internal.h.

> ---
>  tools/libxl/libxl_internal.c |    5 ++++-
>  tools/libxl/libxl_internal.h |   21 +++++++++++++--------
>  2 files changed, 17 insertions(+), 9 deletions(-)
> 
> diff --git a/tools/libxl/libxl_internal.c b/tools/libxl/libxl_internal.c
> index dfa2153..de7c3a8 100644
> --- a/tools/libxl/libxl_internal.c
> +++ b/tools/libxl/libxl_internal.c
> @@ -29,6 +29,9 @@ void libxl__ptr_add(libxl__gc *gc, void *ptr)
>  {
>      int i;
>  
> +    if (!gc)
> +        return;
> +
>      if (!ptr)
>          return;
>  
> @@ -96,7 +99,7 @@ void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size)
>  
>      if (ptr == NULL) {
>          libxl__ptr_add(gc, new_ptr);
> -    } else if (new_ptr != ptr) {
> +    } else if (new_ptr != ptr && gc != NULL) {
>          for (i = 0; i < gc->alloc_maxsize; i++) {
>              if (gc->alloc_ptrs[i] == ptr) {
>                  gc->alloc_ptrs[i] = new_ptr;
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index e59a19b..86f25be 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -381,30 +381,35 @@ static inline libxl_ctx *libxl__gc_owner(libxl__gc *gc)
>   *
>   * All pointers returned by these functions are registered for garbage
>   * collection on exit from the outermost libxl callframe.
> + *
> + * However, where the argument is stated to be "gc_opt", NULL may be
> + * passed instead, in which case no garbage collection will occur; the
> + * pointer must later be freed with free().  This is for memory
> + * allocations of types (b) and (c).
>   */
>  /* register @ptr in @gc for free on exit from outermost libxl callframe. */
> -_hidden void libxl__ptr_add(libxl__gc *gc, void *ptr);
> +_hidden void libxl__ptr_add(libxl__gc *gc_opt, void *ptr);
>  /* if this is the outermost libxl callframe then free all pointers in @gc */
>  _hidden void libxl__free_all(libxl__gc *gc);
>  /* allocate and zero @bytes. (similar to a gc'd malloc(3)+memzero()) */
> -_hidden void *libxl__zalloc(libxl__gc *gc, int bytes);
> +_hidden void *libxl__zalloc(libxl__gc *gc_opt, int bytes);
>  /* allocate and zero memory for an array of @nmemb members of @size each.
>   * (similar to a gc'd calloc(3)). */
> -_hidden void *libxl__calloc(libxl__gc *gc, size_t nmemb, size_t size);
> +_hidden void *libxl__calloc(libxl__gc *gc_opt, size_t nmemb, size_t size);
>  /* change the size of the memory block pointed to by @ptr to @new_size bytes.
>   * unlike other allocation functions here any additional space between the
>   * oldsize and @new_size is not initialised (similar to a gc'd realloc(3)). */
> -_hidden void *libxl__realloc(libxl__gc *gc, void *ptr, size_t new_size);
> +_hidden void *libxl__realloc(libxl__gc *gc_opt, void *ptr, size_t new_size);
>  /* print @fmt into an allocated string large enoughto contain the result.
>   * (similar to gc'd asprintf(3)). */
> -_hidden char *libxl__sprintf(libxl__gc *gc, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3);
> +_hidden char *libxl__sprintf(libxl__gc *gc_opt, const char *fmt, ...) PRINTF_ATTRIBUTE(2, 3);
>  /* duplicate the string @c (similar to a gc'd strdup(3)). */
> -_hidden char *libxl__strdup(libxl__gc *gc, const char *c);
> +_hidden char *libxl__strdup(libxl__gc *gc_opt, const char *c);
>  /* duplicate at most @n bytes of string @c (similar to a gc'd strndup(3)). */
> -_hidden char *libxl__strndup(libxl__gc *gc, const char *c, size_t n);
> +_hidden char *libxl__strndup(libxl__gc *gc_opt, const char *c, size_t n);
>  /* strip the last path component from @s and return as a newly allocated
>   * string. (similar to a gc'd dirname(3)). */
> -_hidden char *libxl__dirname(libxl__gc *gc, const char *s);
> +_hidden char *libxl__dirname(libxl__gc *gc_opt, const char *s);
>  
>  _hidden char **libxl__xs_kvs_of_flexarray(libxl__gc *gc, flexarray_t *array, int length);
>  

Re: [PATCH 10/20] libxl: Make libxl__zalloc et al tolerate a NULL gc

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 10/20] libxl: Make libxl__zalloc et al tolerate a NULL gc"):
> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > Arrange that if we pass NULL as a gc, we simply don't register the
> > pointer.  This instantly gives us non-gc'ing but error-checking
> > versions of malloc, realloc, vasprintf, etc.
...
> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Thanks.

> We could consider __attribute__((nonnull)) for the other helper
> functions in that section of libxl_internal.h.

The only function in that section which isn't declared as taking
gc_opt (ie which can't be passed NULL) is libxl__free_all, and it
seems unlikely anyone would mistakenly pass 0 to it (esp. since it has
few callers).

Ian.

Re: [PATCH 10/20] libxl: Make libxl__zalloc et al tolerate a NULL gc

[Ian Campbell]

On Tue, 2012-03-20 at 16:26 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 10/20] libxl: Make libxl__zalloc et al tolerate a NULL gc"):
> > On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > > Arrange that if we pass NULL as a gc, we simply don't register the
> > > pointer.  This instantly gives us non-gc'ing but error-checking
> > > versions of malloc, realloc, vasprintf, etc.
> ...
> > Acked-by: Ian Campbell <ian.campbell@citrix.com>
> 
> Thanks.
> 
> > We could consider __attribute__((nonnull)) for the other helper
> > functions in that section of libxl_internal.h.
> 
> The only function in that section which isn't declared as taking
> gc_opt (ie which can't be passed NULL) is libxl__free_all, and it
> seems unlikely anyone would mistakenly pass 0 to it (esp. since it has
> few callers).

Oh, right, I was looking at libxl__xs_kvs_of_flexarray etc but that's
actually the next section I guess.

Ian.

[PATCH 11/20] libxl: Introduce some convenience macros

[Ian Jackson]

We introduce:
   <type> *GCNEW(<type> *var);
   <type> *GCNEW_ARRAY(<type> *var, ssize_t nmemb);
   <type> *GCREALLOC_ARRAY(<type> *var, size_t nmemb);
   char *GCSPRINTF(const char *fmt, ...);
   void LOG(<xtl_level_suffix>, const char *fmt, ...);
   void LOGE(<xtl_level_suffix>, const char *fmt, ...);
   void LOGEV(<xtl_level_suffix>, int errnoval, const char *fmt, ...);
all of which expect, in the calling context,
   libxl__gc *gc;

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_internal.h |   72 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 72 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 86f25be..465d564 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1346,6 +1346,78 @@ _hidden void libxl__ao__destroy(libxl_ctx*, libxl__ao *ao);
 #define GC_FREE       libxl__free_all(gc)
 #define CTX           libxl__gc_owner(gc)
 
+/* Allocation macros all of which use the gc. */
+
+#define ARRAY_SIZE_OK(ptr, nmemb) ((nmemb) < INT_MAX / (sizeof(*(ptr)) * 2))
+
+/*
+ * Expression statement  <type> *GCNEW(<type> *var);
+ * Uses                  libxl__gc *gc;
+ *
+ * Allocates a new object of type <type> from the gc and zeroes it
+ * with memset.  Sets var to point to the new object or zero (setting
+ * errno).  Returns the new value of var.
+ */
+#define GCNEW(var)                                      \
+    (((var) = libxl__zalloc((gc),sizeof(*(var)))))
+
+/*
+ * Expression statement  <type> *GCNEW_ARRAY(<type> *var, ssize_t nmemb);
+ * Uses                  libxl__gc *gc;
+ *
+ * Like GCNEW but allocates an array of nmemb elements, as if from
+ * calloc.  Does check for integer overflow due to large nmemb.  If
+ * nmemb is 0 may succeed by returning 0.
+ */
+#define GCNEW_ARRAY(var, nmemb)                                 \
+    ((var) = libxl__calloc((gc), (nmemb), sizeof(*(var))))
+    
+/*
+ * Expression statement  <type> *GCREALLOC_ARRAY(<type> *var, size_t nmemb);
+ * Uses                  libxl__gc *gc;
+ *
+ * Reallocates the array var to be of size nmemb elements.  Updates
+ * var and returns the new value of var.  Does check for integer
+ * overflow due to large nmemb.
+ *
+ * Do not pass nmemb==0.  old may be 0 on entry.
+ */
+#define GCREALLOC_ARRAY(var, nmemb)                                     \
+    (assert(nmemb > 0),                                                 \
+     assert(ARRAY_SIZE_OK((var), (nmemb))),                             \
+     (var) = libxl__realloc((gc), (var), (nmemb)*sizeof(*(var)))))
+
+
+/*
+ * Expression            char *GCSPRINTF(const char *fmt, ...);
+ * Uses                  libxl__gc *gc;
+ *
+ * Trivial convenience wrapper for libxl__sprintf.
+ */
+#define GCSPRINTF(fmt, ...) (libxl__sprintf((gc), (fmt), __VA_ARGS__))
+
+
+/*
+ * Expression statements
+ *    void LOG(<xtl_level_suffix>, const char *fmt, ...);
+ *    void LOGE(<xtl_level_suffix>, const char *fmt, ...);
+ *    void LOGEV(<xtl_level_suffix>, int errnoval, const char *fmt, ...);
+ * Use
+ *    libxl__gc *gc;
+ *
+ * Trivial convenience wrappers for LIBXL__LOG, LIBXL__LOG_ERRNO and
+ * LIBXL__LOG_ERRNOVAL, respectively (and thus for libxl__log).
+ *
+ * XTL_<xtl_level_suffix> should exist and be an xentoollog.h log level
+ * So <xtl_level_suffix> should be one of
+ *   DEBUG VERBOSE DETAIL PROGRESS INFO NOTICE WARN ERROR ERROR CRITICAL
+ * Of these, most of libxl uses
+ *   DEBUG INFO WARN ERROR
+ */
+#define LOG(l,f, ...)     LIBXL__LOG(CTX,XTL_##l,(f),##__VA_ARGS__)
+#define LOGE(l,f, ...)    LIBXL__LOG_ERRNO(CTX,XTL_##l,(f),##__VA_ARGS__)
+#define LOGEV(l,e,f, ...) LIBXL__LOG_ERRNOVAL(CTX,XTL_##l,(e),(f),##__VA_ARGS__)
+
 
 /* Locking functions.  See comment for "lock" member of libxl__ctx. */
 
-- 
1.7.2.5

Re: [PATCH 11/20] libxl: Introduce some convenience macros

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> We introduce:
>    <type> *GCNEW(<type> *var);
>    <type> *GCNEW_ARRAY(<type> *var, ssize_t nmemb);
>    <type> *GCREALLOC_ARRAY(<type> *var, size_t nmemb);
>    char *GCSPRINTF(const char *fmt, ...);
>    void LOG(<xtl_level_suffix>, const char *fmt, ...);
>    void LOGE(<xtl_level_suffix>, const char *fmt, ...);
>    void LOGEV(<xtl_level_suffix>, int errnoval, const char *fmt, ...);
> all of which expect, in the calling context,
>    libxl__gc *gc;

All of these get used later in the series?

> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl_internal.h |   72 ++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 72 insertions(+), 0 deletions(-)
> 
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 86f25be..465d564 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -1346,6 +1346,78 @@ _hidden void libxl__ao__destroy(libxl_ctx*, libxl__ao *ao);
>  #define GC_FREE       libxl__free_all(gc)
>  #define CTX           libxl__gc_owner(gc)
>  
> +/* Allocation macros all of which use the gc. */
> +
> +#define ARRAY_SIZE_OK(ptr, nmemb) ((nmemb) < INT_MAX / (sizeof(*(ptr)) * 2))
> +
> +/*
> + * Expression statement  <type> *GCNEW(<type> *var);
> + * Uses                  libxl__gc *gc;
> + *
> + * Allocates a new object of type <type> from the gc and zeroes it
> + * with memset.  Sets var to point to the new object or zero (setting
> + * errno).  Returns the new value of var.
> + */
> +#define GCNEW(var)                                      \
> +    (((var) = libxl__zalloc((gc),sizeof(*(var)))))
> +
> +/*
> + * Expression statement  <type> *GCNEW_ARRAY(<type> *var, ssize_t nmemb);
> + * Uses                  libxl__gc *gc;
> + *
> + * Like GCNEW but allocates an array of nmemb elements, as if from
> + * calloc.  Does check for integer overflow due to large nmemb.  If
> + * nmemb is 0 may succeed by returning 0.
> + */
> +#define GCNEW_ARRAY(var, nmemb)                                 \
> +    ((var) = libxl__calloc((gc), (nmemb), sizeof(*(var))))
> +    
> +/*
> + * Expression statement  <type> *GCREALLOC_ARRAY(<type> *var, size_t nmemb);
> + * Uses                  libxl__gc *gc;
> + *
> + * Reallocates the array var to be of size nmemb elements.  Updates
> + * var and returns the new value of var.  Does check for integer
> + * overflow due to large nmemb.
> + *
> + * Do not pass nmemb==0.  old may be 0 on entry.
> + */
> +#define GCREALLOC_ARRAY(var, nmemb)                                     \
> +    (assert(nmemb > 0),                                                 \
> +     assert(ARRAY_SIZE_OK((var), (nmemb))),                             \
> +     (var) = libxl__realloc((gc), (var), (nmemb)*sizeof(*(var)))))
> +
> +
> +/*
> + * Expression            char *GCSPRINTF(const char *fmt, ...);
> + * Uses                  libxl__gc *gc;
> + *
> + * Trivial convenience wrapper for libxl__sprintf.
> + */
> +#define GCSPRINTF(fmt, ...) (libxl__sprintf((gc), (fmt), __VA_ARGS__))
> +
> +
> +/*
> + * Expression statements
> + *    void LOG(<xtl_level_suffix>, const char *fmt, ...);
> + *    void LOGE(<xtl_level_suffix>, const char *fmt, ...);
> + *    void LOGEV(<xtl_level_suffix>, int errnoval, const char *fmt, ...);
> + * Use
> + *    libxl__gc *gc;
> + *
> + * Trivial convenience wrappers for LIBXL__LOG, LIBXL__LOG_ERRNO and
> + * LIBXL__LOG_ERRNOVAL, respectively (and thus for libxl__log).
> + *
> + * XTL_<xtl_level_suffix> should exist and be an xentoollog.h log level
> + * So <xtl_level_suffix> should be one of
> + *   DEBUG VERBOSE DETAIL PROGRESS INFO NOTICE WARN ERROR ERROR CRITICAL
> + * Of these, most of libxl uses
> + *   DEBUG INFO WARN ERROR
> + */
> +#define LOG(l,f, ...)     LIBXL__LOG(CTX,XTL_##l,(f),##__VA_ARGS__)
> +#define LOGE(l,f, ...)    LIBXL__LOG_ERRNO(CTX,XTL_##l,(f),##__VA_ARGS__)
> +#define LOGEV(l,e,f, ...) LIBXL__LOG_ERRNOVAL(CTX,XTL_##l,(e),(f),##__VA_ARGS__)
> +
>  
>  /* Locking functions.  See comment for "lock" member of libxl__ctx. */
>  

Re: [PATCH 11/20] libxl: Introduce some convenience macros

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 11/20] libxl: Introduce some convenience macros"):
> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > We introduce:
> >    <type> *GCNEW(<type> *var);
> >    <type> *GCNEW_ARRAY(<type> *var, ssize_t nmemb);
> >    <type> *GCREALLOC_ARRAY(<type> *var, size_t nmemb);
> >    char *GCSPRINTF(const char *fmt, ...);
> >    void LOG(<xtl_level_suffix>, const char *fmt, ...);
> >    void LOGE(<xtl_level_suffix>, const char *fmt, ...);
> >    void LOGEV(<xtl_level_suffix>, int errnoval, const char *fmt, ...);
> > all of which expect, in the calling context,
> >    libxl__gc *gc;
> 
> All of these get used later in the series?

Many of them do.  I'm not sure I use all of the LOG* family but
they're orthogonally necessary.

I supplied GCREALLOC_ARRAY because it seemed to fit in with the family
which had malloc and an array malloc.  It doesn't have a caller yet
even in my tip; there's a potential call site in libxl_qmp.c but if
that's not enough to justify it I can take it out for now.

Ian.

Re: [PATCH 11/20] libxl: Introduce some convenience macros

[Ian Campbell]

On Tue, 2012-03-20 at 16:39 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 11/20] libxl: Introduce some convenience macros"):
> > On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > > We introduce:
> > >    <type> *GCNEW(<type> *var);
> > >    <type> *GCNEW_ARRAY(<type> *var, ssize_t nmemb);
> > >    <type> *GCREALLOC_ARRAY(<type> *var, size_t nmemb);
> > >    char *GCSPRINTF(const char *fmt, ...);
> > >    void LOG(<xtl_level_suffix>, const char *fmt, ...);
> > >    void LOGE(<xtl_level_suffix>, const char *fmt, ...);
> > >    void LOGEV(<xtl_level_suffix>, int errnoval, const char *fmt, ...);
> > > all of which expect, in the calling context,
> > >    libxl__gc *gc;
> > 
> > All of these get used later in the series?
> 
> Many of them do.  I'm not sure I use all of the LOG* family but
> they're orthogonally necessary.

Makes sense.

> I supplied GCREALLOC_ARRAY because it seemed to fit in with the family
> which had malloc and an array malloc.  It doesn't have a caller yet
> even in my tip; there's a potential call site in libxl_qmp.c but if
> that's not enough to justify it I can take it out for now.

Might as well keep it now you've written it.

Thanks,
Ian.

[PATCH 12/20] libxl: Protect fds with CLOEXEC even with forking threads

[Ian Jackson]

We introduce a new "carefd" concept, which relates to fds that we care
about not being inherited by long-lived children.

As yet we do not use this anywhere in libxl.  Until all locations in
libxl which make such fds are converted, libxl__postfork may not work
entirely properly.  If these locations do not use O_CLOEXEC (or use
calls for which there is no O_CLOEXEC) then multithreaded programs may
not work properly.

This introduces a new API call libxl_postfork_child_noexec which must
be called by applications which make long-running non-execing
children.  Add the appropriate call to xl's postfork function.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/Makefile         |    2 +-
 tools/libxl/libxl.c          |    3 +
 tools/libxl/libxl_event.h    |   12 ++++
 tools/libxl/libxl_fork.c     |  146 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   63 ++++++++++++++++++
 tools/libxl/xl.c             |    3 +
 6 files changed, 228 insertions(+), 1 deletions(-)
 create mode 100644 tools/libxl/libxl_fork.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 80c36ad..b30d11f 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -53,7 +53,7 @@ LIBXL_LIBS += -lyajl
 LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
 			libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
 			libxl_internal.o libxl_utils.o libxl_uuid.o libxl_json.o \
-			libxl_qmp.o libxl_event.o $(LIBXL_OBJS-y)
+			libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
 LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
 
 $(LIBXL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest) $(CFLAGS_libxenstore) $(CFLAGS_libblktapctl) -include $(XEN_ROOT)/tools/config.h
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 5827bd4..469f66a 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -68,6 +68,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
 
     /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
 
+    rc = libxl__atfork_init(ctx);
+    if (rc) goto out;
+
     rc = libxl__poller_init(ctx, &ctx->poller_app);
     if (rc) goto out;
 
diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
index ea553f6..41aebc0 100644
--- a/tools/libxl/libxl_event.h
+++ b/tools/libxl/libxl_event.h
@@ -371,6 +371,18 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
  */
 void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
 
+
+/*
+ * An application which initialises a libxl_ctx in a parent process
+ * and then forks a child which does not quickly exec, must
+ * instead libxl__postfork_child_noexec in the child.  One call
+ * on any existing (or specially made) ctx is sufficient; after
+ * this all previously existing libxl_ctx's are invalidated and
+ * must not be used - or even freed.
+ */
+void libxl_postfork_child_noexec(libxl_ctx *ctx);
+
+
 #endif
 
 /*
diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
new file mode 100644
index 0000000..4aaa0b5
--- /dev/null
+++ b/tools/libxl/libxl_fork.c
@@ -0,0 +1,146 @@
+/*
+ * Copyright (C) 2012      Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ */
+/*
+ * Internal child process machinery for use by other parts of libxl
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/*
+ * carefd arrangements
+ *
+ * carefd_begin and _unlock take out the no_forking lock, which we
+ * also take and release in our pthread_atfork handlers.  So when this
+ * lock is held the whole process cannot fork.  We therefore protect
+ * our fds from leaking into children made by other threads.
+ *
+ * We maintain a list of all the carefds, so that if the application
+ * wants to fork a long-running but non-execing child, we can close
+ * them all.
+ *
+ * So the record function sets CLOEXEC for the benefit of execing
+ * children, and makes a note of the fd for the benefit of non-execing
+ * ones.
+ */
+
+struct libxl__carefd {
+    LIBXL_LIST_ENTRY(libxl__carefd) entry;
+    int fd;
+};
+
+static pthread_mutex_t no_forking = PTHREAD_MUTEX_INITIALIZER;
+static int atfork_registered;
+static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
+    LIBXL_LIST_HEAD_INITIALIZER(carefds);
+
+static void atfork_lock(void)
+{
+    int r = pthread_mutex_lock(&no_forking);
+    assert(!r);
+}
+
+static void atfork_unlock(void)
+{
+    int r = pthread_mutex_unlock(&no_forking);
+    assert(!r);
+}
+
+int libxl__atfork_init(libxl_ctx *ctx)
+{
+    int r, rc;
+    
+    atfork_lock();
+    if (atfork_registered) { rc = 0; goto out; }
+
+    r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
+    if (r) {
+        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "pthread_atfork failed");
+        rc = ERROR_NOMEM;
+        goto out;
+    }
+
+    atfork_registered = 1;
+    rc = 0;
+ out:
+    atfork_unlock();
+    return rc;
+}
+
+void libxl__carefd_begin(void) { atfork_lock(); }
+void libxl__carefd_unlock(void) { atfork_unlock(); }
+
+libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd)
+{
+    libxl__carefd *cf = 0;
+
+    libxl_fd_set_cloexec(ctx, fd, 1);
+    cf = libxl__zalloc(NULL, sizeof(*cf));
+    cf->fd = fd;
+    LIBXL_LIST_INSERT_HEAD(&carefds, cf, entry);
+    return cf;
+}
+
+libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd)
+{
+    libxl__carefd *cf = 0;
+
+    cf = libxl__carefd_record(ctx, fd);
+    libxl__carefd_unlock();
+    return cf;
+}
+
+void libxl_postfork_child_noexec(libxl_ctx *ctx)
+{
+    libxl__carefd *cf, *cf_tmp;
+    int r;
+
+    atfork_lock();
+    LIBXL_LIST_FOREACH_SAFE(cf, &carefds, entry, cf_tmp) {
+        if (cf->fd >= 0) {
+            r = close(cf->fd);
+            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_WARNING, "failed to close fd=%d"
+                             " (perhaps of another libxl ctx)", cf->fd);
+        }
+        free(cf);
+    }
+    LIBXL_LIST_INIT(&carefds);
+    atfork_unlock();
+}
+
+int libxl__carefd_close(libxl__carefd *cf)
+{
+    if (!cf) return 0;
+    int r = cf->fd < 0 ? 0 : close(cf->fd);
+    int esave = errno;
+    LIBXL_LIST_REMOVE(cf, entry);
+    free(cf);
+    errno = esave;
+    return r;
+}
+
+int libxl__carefd_fd(const libxl__carefd *cf)
+{
+    if (!cf) return -1;
+    return cf->fd;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 465d564..8091ca2 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -606,6 +606,9 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
 void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
 
 
+int libxl__atfork_init(libxl_ctx *ctx);
+
+
 /* from xl_dom */
 _hidden libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid);
 _hidden int libxl__domain_shutdown_reason(libxl__gc *gc, uint32_t domid);
@@ -1302,6 +1305,66 @@ _hidden void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc);
 /* For use by ao machinery ONLY */
 _hidden void libxl__ao__destroy(libxl_ctx*, libxl__ao *ao);
 
+
+/*
+ * File descriptors and CLOEXEC
+ */
+
+/*
+ * For libxl functions which create file descriptors, at least one
+ * of the following must be true:
+ *  (a) libxl does not care if copies of this open-file are inherited
+ *      by random children and might remain open indefinitely
+ *  (b) libxl must take extra care for the fd (the actual descriptor,
+ *      not the open-file) as below.  We call this a "carefd".
+ *
+ * The rules for opening a carefd are:
+ *  (i)   Before bringing any carefds into existence,
+ *        libxl code must call libxl__carefd_begin.
+ *  (ii)  Then for each carefd brought into existence,
+ *        libxl code must call libxl__carefd_record
+ *        and remember the libxl__carefd_record*.
+ *  (iii) Then it must call libxl__carefd_unlock.
+ *  (iv)  When in a child process the fd is to be passed across
+ *        exec by libxl, the libxl code must unset FD_CLOEXEC
+ *        on the fd eg by using libxl_fd_set_cloexec.
+ *  (v)   Later, when the fd is to be closed in the same process,
+ *        libxl code must not call close.  Instead, it must call
+ *        libxl__carefd_close.
+ * Steps (ii) and (iii) can be combined by calling the convenience
+ * function libxl__carefd_opened.
+ */
+/* libxl__carefd_begin and _unlock (or _opened) must be called always
+ * in pairs.  They may be called with the CTX lock held.  In between
+ * _begin and _unlock, the following are prohibited:
+ *   - anything which might block
+ *   - any callbacks to the application
+ *   - nested calls to libxl__carefd_begin
+ *   - fork (libxl__fork)
+ * In general nothing should be done before _unlock that could be done
+ * afterwards.
+ */
+typedef struct libxl__carefd libxl__carefd;
+
+void libxl__carefd_begin(void);
+void libxl__carefd_unlock(void);
+
+/* fd may be -1, in which case this returns a dummy libxl__fd_record
+ * on which it _carefd_close is a no-op.  Cannot fail. */
+libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd);
+
+/* Combines _record and _unlock in a single call.  If fd==-1,
+ * still does the unlock, but returns 0.  Cannot fail. */
+libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd);
+
+/* Works just like close(2).  You may pass NULL, in which case it's
+ * a successful no-op. */
+int libxl__carefd_close(libxl__carefd*);
+
+/* You may pass NULL in which case the answer is -1. */
+int libxl__carefd_fd(const libxl__carefd*);
+
+
 /*
  * Convenience macros.
  */
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index 9fd67b4..d806077 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -97,6 +97,9 @@ static void parse_global_config(const char *configfile,
 
 void postfork(void)
 {
+    libxl_postfork_child_noexec(ctx); /* in case we don't exit/exec */
+    ctx = 0;
+
     if (libxl_ctx_alloc(&ctx, LIBXL_VERSION, 0, (xentoollog_logger*)logger)) {
         fprintf(stderr, "cannot reinit xl context after fork\n");
         exit(-1);
-- 
1.7.2.5

Re: [PATCH 12/20] libxl: Protect fds with CLOEXEC even with forking threads

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> We introduce a new "carefd" concept, which relates to fds that we care
> about not being inherited by long-lived children.
> 
> As yet we do not use this anywhere in libxl.  Until all locations in
> libxl which make such fds are converted, libxl__postfork may not work
> entirely properly.  If these locations do not use O_CLOEXEC (or use
> calls for which there is no O_CLOEXEC) then multithreaded programs may
> not work properly.
> 
> This introduces a new API call libxl_postfork_child_noexec which must
> be called by applications which make long-running non-execing
> children.  Add the appropriate call to xl's postfork function.

One of the xl callers of postfork does quickly exec. I presume it is
harmless to call the libxl noexec function anyway?

> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> ---
>  tools/libxl/Makefile         |    2 +-
>  tools/libxl/libxl.c          |    3 +
>  tools/libxl/libxl_event.h    |   12 ++++
>  tools/libxl/libxl_fork.c     |  146 ++++++++++++++++++++++++++++++++++++++++++
>  tools/libxl/libxl_internal.h |   63 ++++++++++++++++++
>  tools/libxl/xl.c             |    3 +
>  6 files changed, 228 insertions(+), 1 deletions(-)
>  create mode 100644 tools/libxl/libxl_fork.c
> 
> diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
> index 80c36ad..b30d11f 100644
> --- a/tools/libxl/Makefile
> +++ b/tools/libxl/Makefile
> @@ -53,7 +53,7 @@ LIBXL_LIBS += -lyajl
>  LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
>                         libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
>                         libxl_internal.o libxl_utils.o libxl_uuid.o libxl_json.o \
> -                       libxl_qmp.o libxl_event.o $(LIBXL_OBJS-y)
> +                       libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
>  LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
> 
>  $(LIBXL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest) $(CFLAGS_libxenstore) $(CFLAGS_libblktapctl) -include $(XEN_ROOT)/tools/config.h
> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> index 5827bd4..469f66a 100644
> --- a/tools/libxl/libxl.c
> +++ b/tools/libxl/libxl.c
> @@ -68,6 +68,9 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
> 
>      /* Now ctx is safe for ctx_free; failures simply set rc and "goto out" */
> 
> +    rc = libxl__atfork_init(ctx);
> +    if (rc) goto out;
> +
>      rc = libxl__poller_init(ctx, &ctx->poller_app);
>      if (rc) goto out;
> 
> diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
> index ea553f6..41aebc0 100644
> --- a/tools/libxl/libxl_event.h
> +++ b/tools/libxl/libxl_event.h
> @@ -371,6 +371,18 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
>   */
>  void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
> 
> +
> +/*
> + * An application which initialises a libxl_ctx in a parent process
> + * and then forks a child which does not quickly exec, must
> + * instead libxl__postfork_child_noexec in the child.  One call

One too many underscores after libxl here.

> + * on any existing (or specially made) ctx is sufficient; after
> + * this all previously existing libxl_ctx's are invalidated and
> + * must not be used - or even freed.
> + */
> +void libxl_postfork_child_noexec(libxl_ctx *ctx);
> +
> +
>  #endif
> 
>  /*
> diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
> new file mode 100644
> index 0000000..4aaa0b5
> --- /dev/null
> +++ b/tools/libxl/libxl_fork.c
> @@ -0,0 +1,146 @@
> +/*
> + * Copyright (C) 2012      Citrix Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU Lesser General Public License as published
> + * by the Free Software Foundation; version 2.1 only. with the special
> + * exception on linking described in file LICENSE.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU Lesser General Public License for more details.
> + */
> +/*
> + * Internal child process machinery for use by other parts of libxl
> + */
> +
> +#include "libxl_osdeps.h" /* must come before any other headers */
> +
> +#include "libxl_internal.h"
> +
> +/*
> + * carefd arrangements
> + *
> + * carefd_begin and _unlock take out the no_forking lock, which we
> + * also take and release in our pthread_atfork handlers.  So when this
> + * lock is held the whole process cannot fork.  We therefore protect
> + * our fds from leaking into children made by other threads.
> + *
> + * We maintain a list of all the carefds, so that if the application
> + * wants to fork a long-running but non-execing child, we can close
> + * them all.
> + *
> + * So the record function sets CLOEXEC for the benefit of execing
> + * children, and makes a note of the fd for the benefit of non-execing
> + * ones.
> + */
> +
> +struct libxl__carefd {
> +    LIBXL_LIST_ENTRY(libxl__carefd) entry;
> +    int fd;
> +};
> +
> +static pthread_mutex_t no_forking = PTHREAD_MUTEX_INITIALIZER;
> +static int atfork_registered;
> +static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
> +    LIBXL_LIST_HEAD_INITIALIZER(carefds);
> +
> +static void atfork_lock(void)
> +{
> +    int r = pthread_mutex_lock(&no_forking);
> +    assert(!r);
> +}
> +
> +static void atfork_unlock(void)
> +{
> +    int r = pthread_mutex_unlock(&no_forking);
> +    assert(!r);
> +}
> +
> +int libxl__atfork_init(libxl_ctx *ctx)
> +{
> +    int r, rc;
> +
> +    atfork_lock();
> +    if (atfork_registered) { rc = 0; goto out; }
> +
> +    r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
> +    if (r) {
> +        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "pthread_atfork failed");
> +        rc = ERROR_NOMEM;

I thought we were calling the magic log+exit function on these now ;-)

Surprising that ENOMEM is the only error that can be returned here
(you'd think all manner of things could go wrong when threads are
involved). No matter.

> +        goto out;
> +    }
> +
> +    atfork_registered = 1;
> +    rc = 0;
> + out:
> +    atfork_unlock();
> +    return rc;
> +}
> +
> +void libxl__carefd_begin(void) { atfork_lock(); }
> +void libxl__carefd_unlock(void) { atfork_unlock(); }
> +
> +libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd)
> +{
> +    libxl__carefd *cf = 0;
> +
> +    libxl_fd_set_cloexec(ctx, fd, 1);
> +    cf = libxl__zalloc(NULL, sizeof(*cf));
> +    cf->fd = fd;
> +    LIBXL_LIST_INSERT_HEAD(&carefds, cf, entry);
> +    return cf;
> +}
> +
> +libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd)
> +{
> +    libxl__carefd *cf = 0;
> +
> +    cf = libxl__carefd_record(ctx, fd);
> +    libxl__carefd_unlock();
> +    return cf;
> +}
> +
> +void libxl_postfork_child_noexec(libxl_ctx *ctx)
> +{
> +    libxl__carefd *cf, *cf_tmp;
> +    int r;
> +
> +    atfork_lock();
> +    LIBXL_LIST_FOREACH_SAFE(cf, &carefds, entry, cf_tmp) {
> +        if (cf->fd >= 0) {

Where can an fd < 0 come from? Shouldn't we reject those at
libxl__carefd_record? Or does it make error handling easier in the
callers somehow?

> +            r = close(cf->fd);
> +            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_WARNING, "failed to close fd=%d"
> +                             " (perhaps of another libxl ctx)", cf->fd);

Shouldn't there be an if (r < 0) before this logging?

> +        }
> +        free(cf);
> +    }
> +    LIBXL_LIST_INIT(&carefds);
> +    atfork_unlock();
> +}
> +
> +int libxl__carefd_close(libxl__carefd *cf)
> +{
> +    if (!cf) return 0;
> +    int r = cf->fd < 0 ? 0 : close(cf->fd);
> +    int esave = errno;
> +    LIBXL_LIST_REMOVE(cf, entry);

Are all the LIBXL_LIST_foo thread safe?

This function isn't called with the fork lock held.

> +    free(cf);
> +    errno = esave;
> +    return r;
> +}
> +
> +int libxl__carefd_fd(const libxl__carefd *cf)
> +{
> +    if (!cf) return -1;
> +    return cf->fd;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 465d564..8091ca2 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -606,6 +606,9 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
>  void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
> 
> 
> +int libxl__atfork_init(libxl_ctx *ctx);
> +
> +
>  /* from xl_dom */
>  _hidden libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid);
>  _hidden int libxl__domain_shutdown_reason(libxl__gc *gc, uint32_t domid);
> @@ -1302,6 +1305,66 @@ _hidden void libxl__ao_complete(libxl__egc *egc, libxl__ao *ao, int rc);
>  /* For use by ao machinery ONLY */
>  _hidden void libxl__ao__destroy(libxl_ctx*, libxl__ao *ao);
> 
> +
> +/*
> + * File descriptors and CLOEXEC
> + */
> +
> +/*
> + * For libxl functions which create file descriptors, at least one
> + * of the following must be true:
> + *  (a) libxl does not care if copies of this open-file are inherited
> + *      by random children and might remain open indefinitely
> + *  (b) libxl must take extra care for the fd (the actual descriptor,
> + *      not the open-file) as below.  We call this a "carefd".
> + *
> + * The rules for opening a carefd are:
> + *  (i)   Before bringing any carefds into existence,
> + *        libxl code must call libxl__carefd_begin.
> + *  (ii)  Then for each carefd brought into existence,
> + *        libxl code must call libxl__carefd_record
> + *        and remember the libxl__carefd_record*.
> + *  (iii) Then it must call libxl__carefd_unlock.
> + *  (iv)  When in a child process the fd is to be passed across
> + *        exec by libxl, the libxl code must unset FD_CLOEXEC
> + *        on the fd eg by using libxl_fd_set_cloexec.
> + *  (v)   Later, when the fd is to be closed in the same process,
> + *        libxl code must not call close.  Instead, it must call
> + *        libxl__carefd_close.
> + * Steps (ii) and (iii) can be combined by calling the convenience
> + * function libxl__carefd_opened.
> + */
> +/* libxl__carefd_begin and _unlock (or _opened) must be called always
> + * in pairs.  They may be called with the CTX lock held.  In between
> + * _begin and _unlock, the following are prohibited:
> + *   - anything which might block
> + *   - any callbacks to the application
> + *   - nested calls to libxl__carefd_begin
> + *   - fork (libxl__fork)
> + * In general nothing should be done before _unlock that could be done
> + * afterwards.
> + */
> +typedef struct libxl__carefd libxl__carefd;
> +
> +void libxl__carefd_begin(void);
> +void libxl__carefd_unlock(void);
> +
> +/* fd may be -1, in which case this returns a dummy libxl__fd_record
> + * on which it _carefd_close is a no-op.  Cannot fail. */

answers my earlier question about fd==-1.

> +libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd);
> +
> +/* Combines _record and _unlock in a single call.  If fd==-1,
> + * still does the unlock, but returns 0.  Cannot fail. */
> +libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd);
> +
> +/* Works just like close(2).  You may pass NULL, in which case it's
> + * a successful no-op. */
> +int libxl__carefd_close(libxl__carefd*);
> +
> +/* You may pass NULL in which case the answer is -1. */
> +int libxl__carefd_fd(const libxl__carefd*);
> +
> +
>  /*
>   * Convenience macros.
>   */
> diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
> index 9fd67b4..d806077 100644
> --- a/tools/libxl/xl.c
> +++ b/tools/libxl/xl.c
> @@ -97,6 +97,9 @@ static void parse_global_config(const char *configfile,
> 
>  void postfork(void)
>  {
> +    libxl_postfork_child_noexec(ctx); /* in case we don't exit/exec */
> +    ctx = 0;
> +
>      if (libxl_ctx_alloc(&ctx, LIBXL_VERSION, 0, (xentoollog_logger*)logger)) {
>          fprintf(stderr, "cannot reinit xl context after fork\n");
>          exit(-1);
> --
> 1.7.2.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH 12/20] libxl: Protect fds with CLOEXEC even with forking threads

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 12/20] libxl: Protect fds with CLOEXEC even with forking threads"):
> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > This introduces a new API call libxl_postfork_child_noexec which must
> > be called by applications which make long-running non-execing
> > children.  Add the appropriate call to xl's postfork function.
> 
> One of the xl callers of postfork does quickly exec. I presume it is
> harmless to call the libxl noexec function anyway?

Yes.  I have clarified the doc comment.

> > + * An application which initialises a libxl_ctx in a parent process
> > + * and then forks a child which does not quickly exec, must
> > + * instead libxl__postfork_child_noexec in the child.  One call
> 
> One too many underscores after libxl here.

Fixed.

> > +    r = pthread_atfork(atfork_lock, atfork_unlock, atfork_unlock);
> > +    if (r) {
> > +        LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "pthread_atfork failed");
> > +        rc = ERROR_NOMEM;
> 
> I thought we were calling the magic log+exit function on these now ;-)

Good point.  I have fixed this.  I'll also have to update the alloc
failure handler not to always want to print the allocation size.

> > +            r = close(cf->fd);
> > +            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_WARNING, "failed to close fd=%d"
> > +                             " (perhaps of another libxl ctx)", cf->fd);
> 
> Shouldn't there be an if (r < 0) before this logging?

Yes.  I did say I hadn't executed it :-).

> > +int libxl__carefd_close(libxl__carefd *cf)
> > +{
> > +    if (!cf) return 0;
> > +    int r = cf->fd < 0 ? 0 : close(cf->fd);
> > +    int esave = errno;
> > +    LIBXL_LIST_REMOVE(cf, entry);
> 
> Are all the LIBXL_LIST_foo thread safe?

No.

> This function isn't called with the fork lock held.

Good point.

Thanks,
Ian.

[PATCH 13/20] libxl: libxl_event.c:beforepoll_internal, REQUIRE_FDS

[Ian Jackson]

Introduce definition and use of a new function-local macro REQUIRE_FDS
to avoid repeatedly spelling out which fds we are interested in.

We are going to introduce a new fd for the SIGCHLD self-pipe.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_event.c |   82 ++++++++++++++++++++++++++++++--------------
 1 files changed, 56 insertions(+), 26 deletions(-)

diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 5ac6334..5405299 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -593,6 +593,45 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
     int rc;
 
     /*
+     * We need to look at the fds we want twice: firstly, to count
+     * them so we can make the rindex array big enough, and secondly
+     * to actually fill the arrays in.
+     *
+     * To ensure correctness and avoid repeating the logic for
+     * deciding which fds are relevant, we define a macro
+     *    REQUIRE_FDS( BODY )
+     * which calls
+     *    do{
+     *        int req_fd;
+     *        int req_events;
+     *        BODY;
+     *    }while(0)
+     * for each fd with a nonzero events.  This is invoked twice.
+     *
+     * The definition of REQUIRE_FDS is simplified with the helper
+     * macro
+     *    void REQUIRE_FD(int req_fd, int req_events, BODY);
+     */
+
+#define REQUIRE_FDS(BODY) do{                                          \
+                                                                       \
+        LIBXL_LIST_FOREACH(efd, &CTX->efds, entry)                     \
+            REQUIRE_FD(efd->fd, efd->events, BODY);                    \
+                                                                       \
+        REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, BODY);              \
+                                                                       \
+    }while(0)
+
+#define REQUIRE_FD(req_fd_, req_events_, BODY) do{      \
+        int req_events = (req_events_);                 \
+        int req_fd = (req_fd_);                         \
+        if (req_events) {                               \
+            BODY;                                       \
+        }                                               \
+    }while(0)
+
+
+    /*
      * In order to be able to efficiently find the libxl__ev_fd
      * for a struct poll during _afterpoll, we maintain a shadow
      * data structure in CTX->fd_beforepolled: each slot in
@@ -609,13 +648,13 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
          * not to mess with fd_rindex.
          */
 
-        int maxfd = poller->wakeup_pipe[0] + 1;
-        LIBXL_LIST_FOREACH(efd, &CTX->efds, entry) {
-            if (!efd->events)
-                continue;
-            if (efd->fd >= maxfd)
-                maxfd = efd->fd + 1;
-        }
+        int maxfd = 0;
+
+        REQUIRE_FDS({
+            if (req_fd >= maxfd)
+                maxfd = req_fd + 1;
+        });
+
         /* make sure our array is as big as *nfds_io */
         if (poller->fd_rindex_allocd < maxfd) {
             assert(maxfd < INT_MAX / sizeof(int) / 2);
@@ -630,25 +669,16 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
 
     int used = 0;
 
-#define REQUIRE_FD(req_fd, req_events, efd) do{                 \
-        if ((req_events)) {                                     \
-            if (used < *nfds_io) {                              \
-                fds[used].fd = (req_fd);                        \
-                fds[used].events = (req_events);                \
-                fds[used].revents = 0;                          \
-                assert((req_fd) < poller->fd_rindex_allocd);    \
-                poller->fd_rindex[(req_fd)] = used;             \
-            }                                                   \
-            used++;                                             \
-        }                                                       \
-    }while(0)
-
-    LIBXL_LIST_FOREACH(efd, &CTX->efds, entry)
-        REQUIRE_FD(efd->fd, efd->events, efd);
-
-    REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, 0);
-
-#undef REQUIRE_FD
+    REQUIRE_FDS({
+        if (used < *nfds_io) {
+            fds[used].fd = req_fd;
+            fds[used].events = req_events;
+            fds[used].revents = 0;
+            assert(req_fd < poller->fd_rindex_allocd);
+            poller->fd_rindex[req_fd] = used;
+        }
+        used++;
+    });
 
     rc = used <= *nfds_io ? 0 : ERROR_BUFFERFULL;
 
-- 
1.7.2.5

Re: [PATCH 13/20] libxl: libxl_event.c:beforepoll_internal, REQUIRE_FDS

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> Introduce definition and use of a new function-local macro REQUIRE_FDS
> to avoid repeatedly spelling out which fds we are interested in.
> 
> We are going to introduce a new fd for the SIGCHLD self-pipe.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl_event.c |   82 ++++++++++++++++++++++++++++++--------------
>  1 files changed, 56 insertions(+), 26 deletions(-)
> 
> diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
> index 5ac6334..5405299 100644
> --- a/tools/libxl/libxl_event.c
> +++ b/tools/libxl/libxl_event.c
> @@ -593,6 +593,45 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
>      int rc;
>  
>      /*
> +     * We need to look at the fds we want twice: firstly, to count
> +     * them so we can make the rindex array big enough, and secondly
> +     * to actually fill the arrays in.
> +     *
> +     * To ensure correctness and avoid repeating the logic for
> +     * deciding which fds are relevant, we define a macro
> +     *    REQUIRE_FDS( BODY )
> +     * which calls
> +     *    do{
> +     *        int req_fd;
> +     *        int req_events;
> +     *        BODY;
> +     *    }while(0)
> +     * for each fd with a nonzero events.  This is invoked twice.
> +     *
> +     * The definition of REQUIRE_FDS is simplified with the helper
> +     * macro
> +     *    void REQUIRE_FD(int req_fd, int req_events, BODY);
> +     */
> +
> +#define REQUIRE_FDS(BODY) do{                                          \
> +                                                                       \
> +        LIBXL_LIST_FOREACH(efd, &CTX->efds, entry)                     \
> +            REQUIRE_FD(efd->fd, efd->events, BODY);                    \
> +                                                                       \
> +        REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, BODY);              \
> +                                                                       \
> +    }while(0)
> +
> +#define REQUIRE_FD(req_fd_, req_events_, BODY) do{      \
> +        int req_events = (req_events_);                 \
> +        int req_fd = (req_fd_);                         \
> +        if (req_events) {                               \
> +            BODY;                                       \
> +        }                                               \
> +    }while(0)
> +
> +
> +    /*
>       * In order to be able to efficiently find the libxl__ev_fd
>       * for a struct poll during _afterpoll, we maintain a shadow
>       * data structure in CTX->fd_beforepolled: each slot in
> @@ -609,13 +648,13 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
>           * not to mess with fd_rindex.
>           */
>  
> -        int maxfd = poller->wakeup_pipe[0] + 1;
> -        LIBXL_LIST_FOREACH(efd, &CTX->efds, entry) {
> -            if (!efd->events)
> -                continue;
> -            if (efd->fd >= maxfd)
> -                maxfd = efd->fd + 1;
> -        }
> +        int maxfd = 0;
> +
> +        REQUIRE_FDS({
> +            if (req_fd >= maxfd)
> +                maxfd = req_fd + 1;
> +        });
> +
>          /* make sure our array is as big as *nfds_io */
>          if (poller->fd_rindex_allocd < maxfd) {
>              assert(maxfd < INT_MAX / sizeof(int) / 2);
> @@ -630,25 +669,16 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
>  
>      int used = 0;
>  
> -#define REQUIRE_FD(req_fd, req_events, efd) do{                 \
> -        if ((req_events)) {                                     \
> -            if (used < *nfds_io) {                              \
> -                fds[used].fd = (req_fd);                        \
> -                fds[used].events = (req_events);                \
> -                fds[used].revents = 0;                          \
> -                assert((req_fd) < poller->fd_rindex_allocd);    \
> -                poller->fd_rindex[(req_fd)] = used;             \
> -            }                                                   \
> -            used++;                                             \
> -        }                                                       \
> -    }while(0)
> -
> -    LIBXL_LIST_FOREACH(efd, &CTX->efds, entry)
> -        REQUIRE_FD(efd->fd, efd->events, efd);
> -
> -    REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, 0);
> -
> -#undef REQUIRE_FD
> +    REQUIRE_FDS({
> +        if (used < *nfds_io) {
> +            fds[used].fd = req_fd;
> +            fds[used].events = req_events;
> +            fds[used].revents = 0;
> +            assert(req_fd < poller->fd_rindex_allocd);
> +            poller->fd_rindex[req_fd] = used;
> +        }
> +        used++;
> +    });
>  
>      rc = used <= *nfds_io ? 0 : ERROR_BUFFERFULL;
>  

[PATCH 14/20] libxl: include <ctype.h> and introduce CTYPE helper macro

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_internal.h |   20 ++++++++++++++++++++
 1 files changed, 20 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 8091ca2..69b6f74 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -33,6 +33,7 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <ctype.h>
 
 #include <sys/mman.h>
 #include <sys/poll.h>
@@ -1529,6 +1530,25 @@ static inline void libxl__ctx_unlock(libxl_ctx *ctx) {
     } while(0)
 
 
+/*
+ * int CTYPE(ISFOO, char c);
+ * int CTYPE(toupper, char c);
+ * int CTYPE(tolower, char c);
+ *
+ * This is necessary because passing a simple char to a ctype.h
+ * is forbidden.  ctype.h macros take ints derived from _unsigned_ chars.
+ *
+ * If you have a char which might be EOF then you should already have
+ * it in an int representing an unsigned char, and you can use the
+ * <ctype.h> macros directly.  This generally happens only with values
+ * from fgetc et al.
+ *
+ * For any value known to be a character (eg, anything that came from
+ * a char[]), use CTYPE.
+ */
+#define CTYPE(isfoo,c) (isfoo((unsigned char)(c)))
+
+
 #endif
 
 /*
-- 
1.7.2.5

Re: [PATCH 14/20] libxl: include <ctype.h> and introduce CTYPE helper macro

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Shouldn't there be a handful of cleanups off the back of this?

Huh, grep suggests only
tools/libxl/xl_cmdimpl.c:        if (!isdigit((uint8_t)p[i])) {
which is not useful in the context of libxl_internal.h

I have a feeling the others I was thinking of are in blktap or something

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl_internal.h |   20 ++++++++++++++++++++
>  1 files changed, 20 insertions(+), 0 deletions(-)
> 
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 8091ca2..69b6f74 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -33,6 +33,7 @@
>  #include <stdlib.h>
>  #include <string.h>
>  #include <unistd.h>
> +#include <ctype.h>
>  
>  #include <sys/mman.h>
>  #include <sys/poll.h>
> @@ -1529,6 +1530,25 @@ static inline void libxl__ctx_unlock(libxl_ctx *ctx) {
>      } while(0)
>  
> 
> +/*
> + * int CTYPE(ISFOO, char c);
> + * int CTYPE(toupper, char c);
> + * int CTYPE(tolower, char c);
> + *
> + * This is necessary because passing a simple char to a ctype.h
> + * is forbidden.  ctype.h macros take ints derived from _unsigned_ chars.
> + *
> + * If you have a char which might be EOF then you should already have
> + * it in an int representing an unsigned char, and you can use the
> + * <ctype.h> macros directly.  This generally happens only with values
> + * from fgetc et al.
> + *
> + * For any value known to be a character (eg, anything that came from
> + * a char[]), use CTYPE.
> + */
> +#define CTYPE(isfoo,c) (isfoo((unsigned char)(c)))
> +
> +
>  #endif
>  
>  /*

Re: [PATCH 14/20] libxl: include <ctype.h> and introduce CTYPE helper macro

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 14/20] libxl: include <ctype.h> and introduce CTYPE helper macro"):
> Huh, grep suggests only
> tools/libxl/xl_cmdimpl.c:        if (!isdigit((uint8_t)p[i])) {
> which is not useful in the context of libxl_internal.h

Right.  And we don't really want to export some kind of LIBXL__CTYPE;
it's supposed to be convenient, not longwinded.

> I have a feeling the others I was thinking of are in blktap or something

Very likely.

Thanks,
Ian.

[PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h

[Ian Jackson]

Ie, we permit general code in libxl direct access to the manifest
constants such as XEN_RUN_DIR.  This simplifies their use in (eg)
format strings.

This might be controversial because it will make it difficult to make
any of these runtime-configurable later without changing lots of use
sites.  But I don't think it's likely we'll want to do that.

For the moment, leave existing call sites of all the functions in
libxl_paths.c unchanged.  The simplified use arrangements can be used
in new code and when we update call sites for other reasons.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_internal.h |    1 +
 tools/libxl/libxl_paths.c    |    1 -
 2 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 69b6f74..983de3e 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -51,6 +51,7 @@
 #include <xen/io/xenbus.h>
 
 #include "libxl.h"
+#include "_libxl_paths.h"
 
 #if __GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1)
 #define _hidden __attribute__((visibility("hidden")))
diff --git a/tools/libxl/libxl_paths.c b/tools/libxl/libxl_paths.c
index a95d29f..388b344 100644
--- a/tools/libxl/libxl_paths.c
+++ b/tools/libxl/libxl_paths.c
@@ -14,7 +14,6 @@
 
 #include "libxl_osdeps.h" /* must come before any other headers */
 #include "libxl_internal.h"
-#include "_libxl_paths.h"
 
 const char *libxl_sbindir_path(void)
 {
-- 
1.7.2.5

Re: [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> Ie, we permit general code in libxl direct access to the manifest
> constants such as XEN_RUN_DIR.  This simplifies their use in (eg)
> format strings.
> 
> This might be controversial because it will make it difficult to make
> any of these runtime-configurable later without changing lots of use
> sites.  But I don't think it's likely we'll want to do that.

I don't have any particular feeling one way or the other about this

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> For the moment, leave existing call sites of all the functions in
> libxl_paths.c unchanged.  The simplified use arrangements can be used
> in new code and when we update call sites for other reasons.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>


> ---
>  tools/libxl/libxl_internal.h |    1 +
>  tools/libxl/libxl_paths.c    |    1 -
>  2 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 69b6f74..983de3e 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -51,6 +51,7 @@
>  #include <xen/io/xenbus.h>
>  
>  #include "libxl.h"
> +#include "_libxl_paths.h"
>  
>  #if __GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1)
>  #define _hidden __attribute__((visibility("hidden")))
> diff --git a/tools/libxl/libxl_paths.c b/tools/libxl/libxl_paths.c
> index a95d29f..388b344 100644
> --- a/tools/libxl/libxl_paths.c
> +++ b/tools/libxl/libxl_paths.c
> @@ -14,7 +14,6 @@
>  
>  #include "libxl_osdeps.h" /* must come before any other headers */
>  #include "libxl_internal.h"
> -#include "_libxl_paths.h"
>  
>  const char *libxl_sbindir_path(void)
>  {

Re: [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h

[Roger Pau Monné]

2012/3/20 Ian Campbell <Ian.Campbell@citrix.com>:
> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
>> Ie, we permit general code in libxl direct access to the manifest
>> constants such as XEN_RUN_DIR.  This simplifies their use in (eg)
>> format strings.
>>
>> This might be controversial because it will make it difficult to make
>> any of these runtime-configurable later without changing lots of use
>> sites.  But I don't think it's likely we'll want to do that.
>
> I don't have any particular feeling one way or the other about this
>
> Acked-by: Ian Campbell <ian.campbell@citrix.com>

_libxl_paths.h should be generated before trying to use
libxl_internal.h, if not the build fails.

8<-------------------------------------------------------------

libxl: add _libxl_paths.h as a prerequisite to libxl_internal.h

Signed-off-by: Roger Pau Monne <roger.pau@entel.upc.edu>
---
 tools/libxl/Makefile |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index b30d11f..c0f83a9 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -106,7 +106,7 @@ libxl_paths.c: _libxl_paths.h

 libxl.h: _libxl_types.h
 libxl_json.h: _libxl_types_json.h
-libxl_internal.h: _libxl_types_internal.h
+libxl_internal.h: _libxl_types_internal.h _libxl_paths.h
 libxl_internal_json.h: _libxl_types_internal_json.h

 $(LIBXL_OBJS) $(LIBXLU_OBJS) $(XL_OBJS): libxl.h
-- 
1.7.2.5

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h

[Roger Pau Monné]

2012/3/20 Roger Pau Monné <roger.pau@entel.upc.edu>:
> 2012/3/20 Ian Campbell <Ian.Campbell@citrix.com>:
>> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
>>> Ie, we permit general code in libxl direct access to the manifest
>>> constants such as XEN_RUN_DIR.  This simplifies their use in (eg)
>>> format strings.
>>>
>>> This might be controversial because it will make it difficult to make
>>> any of these runtime-configurable later without changing lots of use
>>> sites.  But I don't think it's likely we'll want to do that.
>>
>> I don't have any particular feeling one way or the other about this
>>
>> Acked-by: Ian Campbell <ian.campbell@citrix.com>
>
> _libxl_paths.h should be generated before trying to use
> libxl_internal.h, if not the build fails.

This also removes the libxl_paths.c dependency, which is no longer necessary:

8<-------------------------------------------------------------

libxl: add _libxl_paths.h as a prerequisite to libxl_internal.h

Also remove the libxl_paths.c dependency, which is no longer necessary.

Signed-off-by: Roger Pau Monne <roger.pau@entel.upc.edu>
---
 tools/libxl/Makefile |    4 +---
 1 files changed, 1 insertions(+), 3 deletions(-)

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index b30d11f..a88827a 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -102,11 +102,9 @@ _libxl_list.h:
$(XEN_INCLUDE)/xen-external/bsd-sys-queue-h-seddery $(XEN_INCLUDE
 	perl $^ --prefix=libxl >$@.new
 	$(call move-if-changed,$@.new,$@)

-libxl_paths.c: _libxl_paths.h
-
 libxl.h: _libxl_types.h
 libxl_json.h: _libxl_types_json.h
-libxl_internal.h: _libxl_types_internal.h
+libxl_internal.h: _libxl_types_internal.h _libxl_paths.h
 libxl_internal_json.h: _libxl_types_internal_json.h

 $(LIBXL_OBJS) $(LIBXLU_OBJS) $(XL_OBJS): libxl.h
-- 
1.7.2.5

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h

[Ian Jackson]

Roger Pau Monné writes ("Re: [Xen-devel] [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h"):
> _libxl_paths.h should be generated before trying to use
> libxl_internal.h, if not the build fails.
...
> libxl: add _libxl_paths.h as a prerequisite to libxl_internal.h

Thanks.  This patch seems like it should be folded into mine ?

Ian.

Re: [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h

[Roger Pau Monné]

2012/3/21 Ian Jackson <Ian.Jackson@eu.citrix.com>:
> Roger Pau Monné writes ("Re: [Xen-devel] [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h"):
>> _libxl_paths.h should be generated before trying to use
>> libxl_internal.h, if not the build fails.
> ...
>> libxl: add _libxl_paths.h as a prerequisite to libxl_internal.h
>
> Thanks.  This patch seems like it should be folded into mine ?

Sure!

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h

[Ian Jackson]

Roger Pau Monné writes ("Re: [Xen-devel] [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h"):
> 2012/3/21 Ian Jackson <Ian.Jackson@eu.citrix.com>:
> > Roger Pau Monné writes ("Re: [Xen-devel] [PATCH 15/20] libxl: include <_libxl_paths.h> in libxl_internal.h"):
> >> _libxl_paths.h should be generated before trying to use
> >> libxl_internal.h, if not the build fails.
> > ...
> >> libxl: add _libxl_paths.h as a prerequisite to libxl_internal.h
> >
> > Thanks.  This patch seems like it should be folded into mine ?
> 
> Sure!

I've done this, with the 2nd version.

Thanks,
Ian.

[PATCH 16/20] libxl: event API: new facilities for waiting for subprocesses

[Ian Jackson]

The current arrangements in libxl for spawning subprocesses have two
key problems: (i) they make unwarranted (and largely undocumented)
assumptions about the caller's use of subprocesses, (ii) they aren't
integrated into the event system and can't be made asynchronous etc.

So replace them with a new set of facilities.

Primarily, from the point of view of code inside libxl, this is
libxl__ev_child_fork which is both (a) a version of fork() and (b) an
event source which generates a callback when the child dies.

>From the point of view of the application, we fully document our use
of SIGCHLD.  The application can tell us whether it wants to own
SIGCHLD or not; if it does, it has to tell us about deaths of our
children.

Currently there are no callers in libxl which use these facilities.
All code in libxl which forks needs to be converted and libxl_fork
needse to be be abolished.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
---
 tools/libxl/libxl.c          |   17 +++-
 tools/libxl/libxl_event.c    |   52 +++++++--
 tools/libxl/libxl_event.h    |  142 ++++++++++++++++++++++++-
 tools/libxl/libxl_fork.c     |  243 ++++++++++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_internal.h |   61 ++++++++++-
 5 files changed, 495 insertions(+), 20 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 469f66a..ee184ca 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -39,7 +39,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     memset(ctx, 0, sizeof(libxl_ctx));
     ctx->lg = lg;
 
-    /* First initialise pointers (cannot fail) */
+    /* First initialise pointers etc. (cannot fail) */
 
     LIBXL_TAILQ_INIT(&ctx->occurred);
 
@@ -58,6 +58,11 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
     LIBXL_TAILQ_INIT(&ctx->death_list);
     libxl__ev_xswatch_init(&ctx->death_watch);
 
+    ctx->childproc_hooks = &libxl__childproc_default_hooks;
+    ctx->childproc_user = 0;
+        
+    ctx->sigchld_selfpipe[0] = -1;
+
     /* The mutex is special because we can't idempotently destroy it */
 
     if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
@@ -160,6 +165,16 @@ int libxl_ctx_free(libxl_ctx *ctx)
 
     discard_events(&ctx->occurred);
 
+    /* If we have outstanding children, then the application inherits
+     * them; we wish the application good luck with understanding
+     * this if and when it reaps them. */
+    libxl__sigchld_removehandler(ctx);
+
+    if (ctx->sigchld_selfpipe[0] >= 0) {
+        close(ctx->sigchld_selfpipe[0]);
+        close(ctx->sigchld_selfpipe[1]);
+    }
+
     pthread_mutex_destroy(&ctx->lock);
 
     GC_FREE;
diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
index 5405299..dcd5802 100644
--- a/tools/libxl/libxl_event.c
+++ b/tools/libxl/libxl_event.c
@@ -620,6 +620,10 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
                                                                        \
         REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, BODY);              \
                                                                        \
+        int selfpipe = libxl__fork_selfpipe_active(CTX);               \
+        if (selfpipe >= 0)                                             \
+            REQUIRE_FD(selfpipe, POLLIN, BODY);                        \
+                                                                       \
     }while(0)
 
 #define REQUIRE_FD(req_fd_, req_events_, BODY) do{      \
@@ -749,10 +753,11 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
                                int nfds, const struct pollfd *fds,
                                struct timeval now)
 {
+    /* May make callbacks into the appliction for child processes.
+     * ctx must be locked exactly once */
     EGC_GC;
     libxl__ev_fd *efd;
 
-
     LIBXL_LIST_FOREACH(efd, &CTX->efds, entry) {
         if (!efd->events)
             continue;
@@ -763,11 +768,16 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
     }
 
     if (afterpoll_check_fd(poller,fds,nfds, poller->wakeup_pipe[0],POLLIN)) {
-        char buf[256];
-        int r = read(poller->wakeup_pipe[0], buf, sizeof(buf));
-        if (r < 0)
-            if (errno != EINTR && errno != EWOULDBLOCK)
-                LIBXL__EVENT_DISASTER(egc, "read wakeup", errno, 0);
+        int e = libxl__self_pipe_eatall(poller->wakeup_pipe[0]);
+        if (e) LIBXL__EVENT_DISASTER(egc, "read wakeup", e, 0);
+    }
+
+    int selfpipe = libxl__fork_selfpipe_active(CTX);
+    if (selfpipe >= 0 &&
+        afterpoll_check_fd(poller,fds,nfds, selfpipe, POLLIN)) {
+        int e = libxl__self_pipe_eatall(selfpipe);
+        if (e) LIBXL__EVENT_DISASTER(egc, "read sigchld pipe", e, 0);
+        libxl__fork_selfpipe_woken(egc);
     }
 
     for (;;) {
@@ -1063,16 +1073,36 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p)
 
 void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p)
 {
+    int e = libxl__self_pipe_wakeup(p->wakeup_pipe[1]);
+    if (e) LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", e, 0);
+}
+
+int libxl__self_pipe_wakeup(int fd)
+{
     static const char buf[1] = "";
 
     for (;;) {
-        int r = write(p->wakeup_pipe[1], buf, 1);
-        if (r==1) return;
+        int r = write(fd, buf, 1);
+        if (r==1) return 0;
         assert(r==-1);
         if (errno == EINTR) continue;
-        if (errno == EWOULDBLOCK) return;
-        LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", errno, 0);
-        return;
+        if (errno == EWOULDBLOCK) return 0;
+        assert(errno);
+        return errno;
+    }
+}
+
+int libxl__self_pipe_eatall(int fd)
+{
+    char buf[256];
+    for (;;) {
+        int r = read(fd, buf, sizeof(buf));
+        if (r >= 0) return 0;
+        assert(r == -1);
+        if (errno == EINTR) continue;
+        if (errno == EWOULDBLOCK) return 0;
+        assert(errno);
+        return errno;
     }
 }
 
diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
index 41aebc0..26c33eb 100644
--- a/tools/libxl/libxl_event.h
+++ b/tools/libxl/libxl_event.h
@@ -163,11 +163,6 @@ void libxl_event_register_callbacks(libxl_ctx *ctx,
  * After libxl_ctx_free, all corresponding evgen handles become
  * invalid and must no longer be passed to evdisable.
  *
- * Events enabled with evenable prior to a fork and libxl_ctx_postfork
- * are no longer generated after the fork/postfork; however the evgen
- * structures are still valid and must be passed to evdisable if the
- * memory they use should not be leaked.
- *
  * Applications should ensure that they eventually retrieve every
  * event using libxl_event_check or libxl_event_wait, since events
  * which occur but are not retreived by the application will be queued
@@ -372,6 +367,142 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
 void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
 
 
+/*======================================================================*/
+
+/*
+ * Subprocess handling.
+ *
+ * Unfortunately the POSIX interface makes this very awkward.
+ *
+ * There are two possible arrangements for collecting statuses from
+ * wait/waitpid.
+ *
+ * For naive programs:
+ *
+ *     libxl will keep a SIGCHLD handler installed whenever it has an
+ *     active (unreaped) child.  It will reap all children with
+ *     wait(); any children it does not recognise will be passed to
+ *     the application via an optional callback (and will result in
+ *     logged warnings if no callback is provided or the callback
+ *     denies responsibility for the child).
+ *
+ *     libxl may have children whenever:
+ *
+ *       - libxl is performing an operation which can be made
+ *         asynchronous; ie one taking a libxl_asyncop_how, even
+ *         if NULL is passed indicating that the operation is
+ *         synchronous; or
+ *
+ *       - events of any kind are being generated, as requested
+ *         by libxl_evenable_....
+ *
+ *     A multithreaded application which is naive in this sense may
+ *     block SIGCHLD on some of its threads, but there must be at
+ *     least one thread that has SIGCHLD unblocked.  libxl will not
+ *     modify the blocking flag for SIGCHLD (except that it may create
+ *     internal service threads with all signals blocked).
+ *
+ *     A naive program must only have at any one time only
+ *     one libxl context which might have children.
+ *
+ * For programs which run their own children alongside libxl's:
+ *
+ *     The application must install a SIGCHLD handler and reap (at
+ *     least) all of libxl's children and pass their exit status
+ *     to libxl by calling libxl_childproc_exited.
+ *
+ *     An application which does this must call
+ *     libxl_childproc_setmode.
+ * 
+ * An application which fails to call setmode, or which passes 0 for
+ * hooks, while it uses any libxl operation which might
+ * create or use child processes (see above):
+ *   - Must not have any child processes running.
+ *   - Must not install a SIGCHLD handler.
+ *   - Must not reap any children.
+ */
+
+
+typedef enum {
+    /* libxl owns SIGCHLD whenever it has a child. */
+    libxl_sigchld_owner_libxl,
+
+    /* Application promises to call libxl_childproc_exited but NOT
+     * from within a signal handler.  libxl will not itself arrange to
+     * (un)block or catch SIGCHLD. */
+    libxl_sigchld_owner_mainloop,
+
+    /* libxl owns SIGCHLD all the time, and the application is
+     * relying on libxl's event loop for reaping its own children. */
+    libxl_sigchld_owner_libxl_always,
+} libxl_sigchld_owner;
+
+typedef struct {
+    libxl_sigchld_owner chldowner;
+
+    /* All of these are optional: */
+
+    /* Called by libxl instead of fork.  Should behave exactly like
+     * fork, including setting errno etc.  May NOT reenter into libxl.
+     * Application may use this to discover pids of libxl's children,
+     * for example.
+     */
+    pid_t (*fork_replacement)(void *user);
+
+    /* With libxl_sigchld_owner_libxl, called by libxl when it has
+     * reaped a pid.  (Not permitted with _owner_mainloop.)
+     *
+     * Should return 0 if the child was recognised by the application
+     * (or if the application does not keep those kind of records),
+     * ERROR_NOT_READY if the application knows that the child is not
+     * the application's; if it returns another error code it is a
+     * disaster as described for libxl_event_register_callbacks.
+     * (libxl will report unexpected children to its error log.)
+     *
+     * If not supplied, the application is assumed not to start
+     * any children of its own.
+     *
+     * This function is NOT called from within the signal handler.
+     * Rather it will be called from inside a libxl's event handling
+     * code and thus only when libxl is running, for example from
+     * within libxl_event_wait.  (libxl uses the self-pipe trick
+     * to implement this.)
+     *
+     * childproc_exited_callback may call back into libxl, but it
+     * is best to avoid making long-running libxl calls as that might
+     * stall the calling event loop while the nested operation
+     * completes.
+     */
+    int (*reaped_callback)(pid_t, int status, void *user);
+} libxl_childproc_hooks;
+
+/* hooks may be 0 in which is equivalent to &{ libxl_sigchld_owner_libxl, 0, 0 }
+ *
+ * May not be called when libxl might have any child processes, or the
+ * behaviour is undefined.  So it is best to call this at
+ * initialisation.
+ */
+void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
+                             void *user);
+
+/*
+ * This function is for an application which owns SIGCHLD and which
+ * therefore reaps all of the process's children.
+ *
+ * May be called only by an application which has called setmode with
+ * chldowner == libxl_sigchld_owner_mainloop.  If pid was a process started
+ * by this instance of libxl, returns 0 after doing whatever
+ * processing is appropriate.  Otherwise silently returns
+ * ERROR_NOT_READY.  No other error returns are possible.
+ *
+ * May NOT be called from within a signal handler which might
+ * interrupt any libxl operation.  The application will almost
+ * certainly need to use the self-pipe trick (or a working pselect or
+ * ppoll) to implement this.
+ */
+int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status);
+
+
 /*
  * An application which initialises a libxl_ctx in a parent process
  * and then forks a child which does not quickly exec, must
@@ -383,6 +514,7 @@ void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
 void libxl_postfork_child_noexec(libxl_ctx *ctx);
 
 
+
 #endif
 
 /*
diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
index 4aaa0b5..ee62b61 100644
--- a/tools/libxl/libxl_fork.c
+++ b/tools/libxl/libxl_fork.c
@@ -46,6 +46,12 @@ static int atfork_registered;
 static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
     LIBXL_LIST_HEAD_INITIALIZER(carefds);
 
+/* non-null iff installed, protected by no_forking */
+static libxl_ctx *sigchld_owner;
+static struct sigaction sigchld_saved_action;
+
+static void sigchld_removehandler_core(void);
+
 static void atfork_lock(void)
 {
     int r = pthread_mutex_lock(&no_forking);
@@ -108,6 +114,7 @@ void libxl_postfork_child_noexec(libxl_ctx *ctx)
     int r;
 
     atfork_lock();
+
     LIBXL_LIST_FOREACH_SAFE(cf, &carefds, entry, cf_tmp) {
         if (cf->fd >= 0) {
             r = close(cf->fd);
@@ -117,6 +124,10 @@ void libxl_postfork_child_noexec(libxl_ctx *ctx)
         free(cf);
     }
     LIBXL_LIST_INIT(&carefds);
+
+    if (sigchld_owner)
+        sigchld_removehandler_core();
+
     atfork_unlock();
 }
 
@@ -138,6 +149,238 @@ int libxl__carefd_fd(const libxl__carefd *cf)
 }
 
 /*
+ * Actual child process handling
+ */
+
+static void sigchld_handler(int signo)
+{
+    int e = libxl__self_pipe_wakeup(sigchld_owner->sigchld_selfpipe[1]);
+    assert(!e); /* errors are probably EBADF, very bad */
+}
+
+static void sigchld_removehandler_core(void)
+{
+    struct sigaction was;
+    int r;
+    
+    r = sigaction(SIGCHLD, &sigchld_saved_action, &was);
+    assert(!r);
+    assert(!(was.sa_flags & SA_SIGINFO));
+    assert(was.sa_handler == sigchld_handler);
+    sigchld_owner = 0;
+}
+
+void libxl__sigchld_removehandler(libxl_ctx *ctx) /* non-reentrant */
+{
+    atfork_lock();
+    if (sigchld_owner == ctx)
+        sigchld_removehandler_core();
+    atfork_unlock();
+}
+
+int libxl__sigchld_installhandler(libxl_ctx *ctx) /* non-reentrant */
+{
+    int r, rc;
+
+    if (ctx->sigchld_selfpipe[0] < 0) {
+        r = pipe(ctx->sigchld_selfpipe);
+        if (r) {
+            ctx->sigchld_selfpipe[0] = -1;
+            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+                             "failed to create sigchld pipe");
+            rc = ERROR_FAIL;
+            goto out;
+        }
+    }
+
+    atfork_lock();
+    if (sigchld_owner != ctx) {
+        struct sigaction ours;
+
+        assert(!sigchld_owner);
+        sigchld_owner = ctx;
+
+        memset(&ours,0,sizeof(ours));
+        ours.sa_handler = sigchld_handler;
+        sigemptyset(&ours.sa_mask);
+        ours.sa_flags = SA_NOCLDSTOP | SA_RESTART;
+        r = sigaction(SIGCHLD, &ours, &sigchld_saved_action);
+        assert(!r);
+
+        assert(((void)"application must negotiate with libxl about SIGCHLD",
+                !(sigchld_saved_action.sa_flags & SA_SIGINFO) &&
+                (sigchld_saved_action.sa_handler == SIG_DFL ||
+                 sigchld_saved_action.sa_handler == SIG_IGN)));
+    }
+    atfork_unlock();
+
+    rc = 0;
+ out:
+    return rc;
+}
+
+static int chldmode_ours(libxl_ctx *ctx)
+{
+    return ctx->childproc_hooks->chldowner == libxl_sigchld_owner_libxl;
+}
+
+int libxl__fork_selfpipe_active(libxl_ctx *ctx)
+{
+    /* Returns the fd to read, or -1 */
+    if (!chldmode_ours(ctx))
+        return -1;
+
+    if (LIBXL_LIST_EMPTY(&ctx->children))
+        return -1;
+
+    return ctx->sigchld_selfpipe[0];
+}
+
+static void perhaps_removehandler(libxl_ctx *ctx)
+{
+    if (LIBXL_LIST_EMPTY(&ctx->children) &&
+        ctx->childproc_hooks->chldowner != libxl_sigchld_owner_libxl_always)
+        libxl__sigchld_removehandler(ctx);
+}
+
+static int childproc_reaped(libxl__egc *egc, pid_t pid, int status)
+{
+    EGC_GC;
+    libxl__ev_child *ch;
+
+    LIBXL_LIST_FOREACH(ch, &CTX->children, entry)
+        if (ch->pid == pid)
+            goto found;
+
+    /* not found */
+    return ERROR_NOT_READY;
+
+ found:
+    LIBXL_LIST_REMOVE(ch, entry);
+    ch->pid = -1;
+    ch->callback(egc, ch, pid, status);
+
+    perhaps_removehandler(CTX);
+
+    return 0;
+}
+
+int libxl_childproc_reaped(libxl_ctx *ctx, pid_t pid, int status)
+{
+    EGC_INIT(ctx);
+    CTX_LOCK;
+    int rc = childproc_reaped(egc, pid, status);
+    CTX_UNLOCK;
+    EGC_FREE;
+    return rc;
+}
+
+void libxl__fork_selfpipe_woken(libxl__egc *egc) {
+    /* May make callbacks into the appliction for child processes.
+     * ctx must be locked EXACTLY ONCE */
+    EGC_GC;
+
+    while (chldmode_ours(CTX) /* in case the app changes the mode */) {
+        int status;
+        pid_t pid = waitpid(-1, &status, WNOHANG);
+
+        if (pid == 0) return;
+
+        if (pid == -1) {
+            if (errno == ECHILD) return;
+            if (errno == EINTR) continue;
+            LIBXL__EVENT_DISASTER(egc, "waitpid() failed", errno, 0);
+            return;
+        }
+
+        int rc = childproc_reaped(egc, pid, status);
+
+        if (rc) {
+            if (CTX->childproc_hooks->reaped_callback) {
+                CTX_UNLOCK;
+                CTX->childproc_hooks->reaped_callback
+                    (pid, status, CTX->childproc_user);
+                CTX_LOCK;
+            } else {
+                libxl_report_child_exitstatus(CTX, XTL_WARN,
+                                "unknown child", (long)pid, status);
+            }
+        }
+    }
+}
+
+pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *ch,
+                           libxl__ev_child_callback *death)
+{
+    CTX_LOCK;
+    int rc;
+
+    if (chldmode_ours(CTX)) {
+        rc = libxl__sigchld_installhandler(CTX);
+        if (rc) goto out;
+    }
+
+    pid_t pid =
+        CTX->childproc_hooks->fork_replacement
+        ? CTX->childproc_hooks->fork_replacement(CTX->childproc_user)
+        : fork();
+    if (pid == -1) {
+        LOGE(ERROR, "fork failed");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    if (!pid) {
+        /* woohoo! */
+        return 0; /* Yes, CTX is left locked in the child. */
+    }
+
+    ch->pid = pid;
+    ch->callback = death;
+    LIBXL_LIST_INSERT_HEAD(&CTX->children, ch, entry);
+    rc = pid;
+
+ out:
+    perhaps_removehandler(CTX);
+    CTX_UNLOCK;
+    return rc;
+}
+
+void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
+                             void *user)
+{
+    GC_INIT(ctx);
+    CTX_LOCK;
+
+    assert(LIBXL_LIST_EMPTY(&CTX->children));
+
+    if (!hooks)
+        hooks = &libxl__childproc_default_hooks;
+
+    ctx->childproc_hooks = hooks;
+    ctx->childproc_user = user;
+
+    switch (ctx->childproc_hooks->chldowner) {
+    case libxl_sigchld_owner_mainloop:
+    case libxl_sigchld_owner_libxl:
+        libxl__sigchld_removehandler(ctx);
+        break;
+    case libxl_sigchld_owner_libxl_always:
+        libxl__sigchld_installhandler(ctx);
+        break;
+    default:
+        abort();
+    }
+
+    CTX_UNLOCK;
+    GC_FREE;
+}
+
+const libxl_childproc_hooks libxl__childproc_default_hooks = {
+    libxl_sigchld_owner_libxl, 0, 0
+};
+
+/*
  * Local variables:
  * mode: C
  * c-basic-offset: 4
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 983de3e..0a344cf 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -184,6 +184,19 @@ typedef struct libxl__ev_watch_slot {
 libxl__ev_xswatch *libxl__watch_slot_contents(libxl__gc *gc, int slotnum);
 
 
+typedef struct libxl__ev_child libxl__ev_child;
+typedef void libxl__ev_child_callback(libxl__egc *egc, libxl__ev_child*,
+                                      pid_t pid, int status);
+struct libxl__ev_child {
+    /* caller should include this in their own struct */
+    /* read-only for caller: */
+    pid_t pid; /* -1 means unused ("unregistered", ie Idle) */
+    libxl__ev_child_callback *callback;
+    /* remainder is private for libxl__ev_... */
+    LIBXL_LIST_ENTRY(struct libxl__ev_child) entry;
+};
+
+
 /*
  * evgen structures, which are the state we use for generating
  * events for the caller.
@@ -263,6 +276,8 @@ struct libxl__ctx {
     const libxl_event_hooks *event_hooks;
     void *event_hooks_user;
 
+    LIBXL_LIST_ENTRY(struct libxl__ctx);
+
     pthread_mutex_t lock; /* protects data structures hanging off the ctx */
       /* Always use libxl__ctx_lock and _unlock (or the convenience
        * macors CTX_LOCK and CTX_UNLOCK) to manipulate this.
@@ -303,10 +318,14 @@ struct libxl__ctx {
     
     LIBXL_LIST_HEAD(, libxl_evgen_disk_eject) disk_eject_evgens;
 
-    /* for callers who reap children willy-nilly; caller must only
-     * set this after libxl_init and before any other call - or
-     * may leave them untouched */
+    const libxl_childproc_hooks *childproc_hooks;
+    void *childproc_user;
+    int sigchld_selfpipe[2]; /* [0]==-1 means handler not installed */
+    LIBXL_LIST_HEAD(, libxl__ev_child) children;
+
+    /* This is obsolete and must be removed: */
     int (*waitpid_instead)(pid_t pid, int *status, int flags);
+
     libxl_version_info version_info;
 };
 
@@ -554,6 +573,33 @@ static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw)
 
 
 /*
+ * For making subprocesses.  This is the only permitted mechanism for
+ * code in libxl to do so.
+ *
+ * In the parent, returns the pid, filling in childw_out.
+ * In the child, returns 0.
+ * If it fails, returns a libxl error (all of which are -ve).
+ *
+ * The child should go on to exec (or exit) soon, and should not make
+ * any further libxl event calls in the meantime.
+ *
+ * The parent may signal the child but it must not reap it.  That will
+ * be done by the event machinery.  Either death or cldstop may be
+ * NULL in which case that kind of event is ignored.
+ *
+ * It is not possible to "deregister" the child death event source.
+ * It will generate exactly one event callback; until then the childw
+ * is Active and may not be reused.
+ */
+_hidden pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *childw_out,
+                                 libxl__ev_child_callback *death);
+static inline void libxl__ev_child_init(libxl__ev_child *childw_out)
+                { childw_out->pid = -1; }
+static inline int libxl__ev_child_inuse(libxl__ev_child *childw_out)
+                { return childw_out->pid >= 0; }
+
+
+/*
  * Other event-handling support provided by the libxl event core to
  * the rest of libxl.
  */
@@ -607,6 +653,15 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
  * ctx must be locked. */
 void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
 
+/* Internal to fork and child reaping machinery */
+extern const libxl_childproc_hooks libxl__childproc_default_hooks;
+int libxl__sigchld_installhandler(libxl_ctx *ctx); /* non-reentrant;logs errs */
+void libxl__sigchld_removehandler(libxl_ctx *ctx); /* non-reentrant */
+int libxl__fork_selfpipe_active(libxl_ctx *ctx); /* returns read fd or -1 */
+void libxl__fork_selfpipe_woken(libxl__egc *egc);
+int libxl__self_pipe_wakeup(int fd); /* returns 0 or -1 setting errno */
+int libxl__self_pipe_eatall(int fd); /* returns 0 or -1 setting errno */
+
 
 int libxl__atfork_init(libxl_ctx *ctx);
 
-- 
1.7.2.5

Re: [PATCH 16/20] libxl: event API: new facilities for waiting for subprocesses

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> The current arrangements in libxl for spawning subprocesses have two
> key problems: (i) they make unwarranted (and largely undocumented)
> assumptions about the caller's use of subprocesses, (ii) they aren't
> integrated into the event system and can't be made asynchronous etc.
> 
> So replace them with a new set of facilities.
> 
> Primarily, from the point of view of code inside libxl, this is
> libxl__ev_child_fork which is both (a) a version of fork() and (b) an
> event source which generates a callback when the child dies.
> 
> From the point of view of the application, we fully document our use
> of SIGCHLD.  The application can tell us whether it wants to own
> SIGCHLD or not; if it does, it has to tell us about deaths of our
> children.
> 
> Currently there are no callers in libxl which use these facilities.
> All code in libxl which forks needs to be converted and libxl_fork
> needse to be be abolished.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Roger Pau Monne <roger.pau@entel.upc.edu>
> ---
>  tools/libxl/libxl.c          |   17 +++-
>  tools/libxl/libxl_event.c    |   52 +++++++--
>  tools/libxl/libxl_event.h    |  142 ++++++++++++++++++++++++-
>  tools/libxl/libxl_fork.c     |  243 ++++++++++++++++++++++++++++++++++++++++++
>  tools/libxl/libxl_internal.h |   61 ++++++++++-
>  5 files changed, 495 insertions(+), 20 deletions(-)
> 
> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> index 469f66a..ee184ca 100644
> --- a/tools/libxl/libxl.c
> +++ b/tools/libxl/libxl.c
> @@ -39,7 +39,7 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
>      memset(ctx, 0, sizeof(libxl_ctx));
>      ctx->lg = lg;
> 
> -    /* First initialise pointers (cannot fail) */
> +    /* First initialise pointers etc. (cannot fail) */
> 
>      LIBXL_TAILQ_INIT(&ctx->occurred);
> 
> @@ -58,6 +58,11 @@ int libxl_ctx_alloc(libxl_ctx **pctx, int version,
>      LIBXL_TAILQ_INIT(&ctx->death_list);
>      libxl__ev_xswatch_init(&ctx->death_watch);
> 
> +    ctx->childproc_hooks = &libxl__childproc_default_hooks;
> +    ctx->childproc_user = 0;
> +
> +    ctx->sigchld_selfpipe[0] = -1;
> +
>      /* The mutex is special because we can't idempotently destroy it */
> 
>      if (libxl__init_recursive_mutex(ctx, &ctx->lock) < 0) {
> @@ -160,6 +165,16 @@ int libxl_ctx_free(libxl_ctx *ctx)
> 
>      discard_events(&ctx->occurred);
> 
> +    /* If we have outstanding children, then the application inherits
> +     * them; we wish the application good luck with understanding
> +     * this if and when it reaps them. */

:-)

> +    libxl__sigchld_removehandler(ctx);
> +
> +    if (ctx->sigchld_selfpipe[0] >= 0) {
> +        close(ctx->sigchld_selfpipe[0]);
> +        close(ctx->sigchld_selfpipe[1]);
> +    }
> +
>      pthread_mutex_destroy(&ctx->lock);
> 
>      GC_FREE;
> diff --git a/tools/libxl/libxl_event.c b/tools/libxl/libxl_event.c
> index 5405299..dcd5802 100644
> --- a/tools/libxl/libxl_event.c
> +++ b/tools/libxl/libxl_event.c
> @@ -620,6 +620,10 @@ static int beforepoll_internal(libxl__gc *gc, libxl__poller *poller,
>                                                                         \
>          REQUIRE_FD(poller->wakeup_pipe[0], POLLIN, BODY);              \
>                                                                         \
> +        int selfpipe = libxl__fork_selfpipe_active(CTX);               \
> +        if (selfpipe >= 0)                                             \
> +            REQUIRE_FD(selfpipe, POLLIN, BODY);                        \
> +                                                                       \
>      }while(0)
> 
>  #define REQUIRE_FD(req_fd_, req_events_, BODY) do{      \
> @@ -749,10 +753,11 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
>                                 int nfds, const struct pollfd *fds,
>                                 struct timeval now)
>  {
> +    /* May make callbacks into the appliction for child processes.

Typo:                                application

> +     * ctx must be locked exactly once */
>      EGC_GC;
>      libxl__ev_fd *efd;
> 
> -
>      LIBXL_LIST_FOREACH(efd, &CTX->efds, entry) {
>          if (!efd->events)
>              continue;
> @@ -763,11 +768,16 @@ static void afterpoll_internal(libxl__egc *egc, libxl__poller *poller,
>      }
> 
>      if (afterpoll_check_fd(poller,fds,nfds, poller->wakeup_pipe[0],POLLIN)) {
> -        char buf[256];
> -        int r = read(poller->wakeup_pipe[0], buf, sizeof(buf));
> -        if (r < 0)
> -            if (errno != EINTR && errno != EWOULDBLOCK)
> -                LIBXL__EVENT_DISASTER(egc, "read wakeup", errno, 0);
> +        int e = libxl__self_pipe_eatall(poller->wakeup_pipe[0]);
> +        if (e) LIBXL__EVENT_DISASTER(egc, "read wakeup", e, 0);
> +    }
> +
> +    int selfpipe = libxl__fork_selfpipe_active(CTX);
> +    if (selfpipe >= 0 &&
> +        afterpoll_check_fd(poller,fds,nfds, selfpipe, POLLIN)) {
> +        int e = libxl__self_pipe_eatall(selfpipe);
> +        if (e) LIBXL__EVENT_DISASTER(egc, "read sigchld pipe", e, 0);
> +        libxl__fork_selfpipe_woken(egc);
>      }
> 
>      for (;;) {
> @@ -1063,16 +1073,36 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p)
> 
>  void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p)
>  {
> +    int e = libxl__self_pipe_wakeup(p->wakeup_pipe[1]);
> +    if (e) LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", e, 0);
> +}
> +
> +int libxl__self_pipe_wakeup(int fd)
> +{
>      static const char buf[1] = "";
> 
>      for (;;) {
> -        int r = write(p->wakeup_pipe[1], buf, 1);
> -        if (r==1) return;
> +        int r = write(fd, buf, 1);
> +        if (r==1) return 0;
>          assert(r==-1);
>          if (errno == EINTR) continue;
> -        if (errno == EWOULDBLOCK) return;
> -        LIBXL__EVENT_DISASTER(egc, "cannot poke watch pipe", errno, 0);
> -        return;
> +        if (errno == EWOULDBLOCK) return 0;
> +        assert(errno);
> +        return errno;
> +    }
> +}
> +
> +int libxl__self_pipe_eatall(int fd)
> +{
> +    char buf[256];
> +    for (;;) {
> +        int r = read(fd, buf, sizeof(buf));
> +        if (r >= 0) return 0;

Is there some 256 byte limit on the number of bytes pending in the pipe
somewhere? Wouldn't r==256 require us to (potentially) go round the loop
again?

> +        assert(r == -1);
> +        if (errno == EINTR) continue;
> +        if (errno == EWOULDBLOCK) return 0;
> +        assert(errno);
> +        return errno;
>      }
>  }
> 
> diff --git a/tools/libxl/libxl_event.h b/tools/libxl/libxl_event.h
> index 41aebc0..26c33eb 100644
> --- a/tools/libxl/libxl_event.h
> +++ b/tools/libxl/libxl_event.h
> @@ -163,11 +163,6 @@ void libxl_event_register_callbacks(libxl_ctx *ctx,
>   * After libxl_ctx_free, all corresponding evgen handles become
>   * invalid and must no longer be passed to evdisable.
>   *
> - * Events enabled with evenable prior to a fork and libxl_ctx_postfork
> - * are no longer generated after the fork/postfork; however the evgen
> - * structures are still valid and must be passed to evdisable if the
> - * memory they use should not be leaked.
> - *
>   * Applications should ensure that they eventually retrieve every
>   * event using libxl_event_check or libxl_event_wait, since events
>   * which occur but are not retreived by the application will be queued
> @@ -372,6 +367,142 @@ void libxl_osevent_occurred_fd(libxl_ctx *ctx, void *for_libxl,
>  void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
> 
> 
> +/*======================================================================*/
> +
> +/*
> + * Subprocess handling.
> + *
> + * Unfortunately the POSIX interface makes this very awkward.

To say the least...

> + *
> + * There are two possible arrangements for collecting statuses from
> + * wait/waitpid.
> + *
> + * For naive programs:
> + *
> + *     libxl will keep a SIGCHLD handler installed whenever it has an
> + *     active (unreaped) child.  It will reap all children with
> + *     wait(); any children it does not recognise will be passed to
> + *     the application via an optional callback (and will result in
> + *     logged warnings if no callback is provided or the callback
> + *     denies responsibility for the child).
> + *
> + *     libxl may have children whenever:
> + *
> + *       - libxl is performing an operation which can be made
> + *         asynchronous; ie one taking a libxl_asyncop_how, even
> + *         if NULL is passed indicating that the operation is
> + *         synchronous; or
> + *
> + *       - events of any kind are being generated, as requested
> + *         by libxl_evenable_....
> + *
> + *     A multithreaded application which is naive in this sense may
> + *     block SIGCHLD on some of its threads, but there must be at
> + *     least one thread that has SIGCHLD unblocked.  libxl will not
> + *     modify the blocking flag for SIGCHLD (except that it may create
> + *     internal service threads with all signals blocked).
> + *
> + *     A naive program must only have at any one time only
> + *     one libxl context which might have children.

> + * For programs which run their own children alongside libxl's:
> + *
> + *     The application must install a SIGCHLD handler and reap (at
> + *     least) all of libxl's children and pass their exit status
> + *     to libxl by calling libxl_childproc_exited.
> + *
> + *     An application which does this must call
> + *     libxl_childproc_setmode.
> + *
> + * An application which fails to call setmode, or which passes 0 for
> + * hooks, while it uses any libxl operation which might
> + * create or use child processes (see above):
> + *   - Must not have any child processes running.
> + *   - Must not install a SIGCHLD handler.
> + *   - Must not reap any children.
> + */
> +
> +
> +typedef enum {
> +    /* libxl owns SIGCHLD whenever it has a child. */
> +    libxl_sigchld_owner_libxl,
> +
> +    /* Application promises to call libxl_childproc_exited but NOT
> +     * from within a signal handler.  libxl will not itself arrange to
> +     * (un)block or catch SIGCHLD. */
> +    libxl_sigchld_owner_mainloop,
> +
> +    /* libxl owns SIGCHLD all the time, and the application is
> +     * relying on libxl's event loop for reaping its own children. */
> +    libxl_sigchld_owner_libxl_always,

The comment above doesn't seem to cover this third case?

> +} libxl_sigchld_owner;
> +
> +typedef struct {
> +    libxl_sigchld_owner chldowner;
> +
> +    /* All of these are optional: */
> +
> +    /* Called by libxl instead of fork.  Should behave exactly like
> +     * fork, including setting errno etc.  May NOT reenter into libxl.
> +     * Application may use this to discover pids of libxl's children,
> +     * for example.
> +     */
> +    pid_t (*fork_replacement)(void *user);
> +
> +    /* With libxl_sigchld_owner_libxl, called by libxl when it has
> +     * reaped a pid.  (Not permitted with _owner_mainloop.)
> +     *
> +     * Should return 0 if the child was recognised by the application
> +     * (or if the application does not keep those kind of records),
> +     * ERROR_NOT_READY if the application knows that the child is not

ERROR_NOT_READY seems like an odd name. ERROR_UNKNOWN_CHILD perhaps?

> +     * the application's; if it returns another error code it is a
> +     * disaster as described for libxl_event_register_callbacks.
> +     * (libxl will report unexpected children to its error log.)
> +     *
> +     * If not supplied, the application is assumed not to start
> +     * any children of its own.
> +     *
> +     * This function is NOT called from within the signal handler.
> +     * Rather it will be called from inside a libxl's event handling
> +     * code and thus only when libxl is running, for example from
> +     * within libxl_event_wait.  (libxl uses the self-pipe trick
> +     * to implement this.)
> +     *
> +     * childproc_exited_callback may call back into libxl, but it
> +     * is best to avoid making long-running libxl calls as that might
> +     * stall the calling event loop while the nested operation
> +     * completes.
> +     */
> +    int (*reaped_callback)(pid_t, int status, void *user);
> +} libxl_childproc_hooks;
> +
> +/* hooks may be 0 in which is equivalent to &{ libxl_sigchld_owner_libxl, 0, 0 }
> + *
> + * May not be called when libxl might have any child processes, or the
> + * behaviour is undefined.  So it is best to call this at
> + * initialisation.
> + */
> +void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
> +                             void *user);
> +
> +/*
> + * This function is for an application which owns SIGCHLD and which
> + * therefore reaps all of the process's children.
> + *
> + * May be called only by an application which has called setmode with
> + * chldowner == libxl_sigchld_owner_mainloop.  If pid was a process started
> + * by this instance of libxl, returns 0 after doing whatever
> + * processing is appropriate.  Otherwise silently returns
> + * ERROR_NOT_READY.  No other error returns are possible.
> + *
> + * May NOT be called from within a signal handler which might
> + * interrupt any libxl operation.  The application will almost
> + * certainly need to use the self-pipe trick (or a working pselect or
> + * ppoll) to implement this.
> + */
> +int libxl_childproc_reaped(libxl_ctx *ctx, pid_t, int status);
> +
> +
>  /*
>   * An application which initialises a libxl_ctx in a parent process
>   * and then forks a child which does not quickly exec, must
> @@ -383,6 +514,7 @@ void libxl_osevent_occurred_timeout(libxl_ctx *ctx, void *for_libxl);
>  void libxl_postfork_child_noexec(libxl_ctx *ctx);
> 
> 
> +

The blank lines appear to be breeding ;-)

>  #endif
> 
>  /*
> diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
> index 4aaa0b5..ee62b61 100644
> --- a/tools/libxl/libxl_fork.c
> +++ b/tools/libxl/libxl_fork.c
> @@ -46,6 +46,12 @@ static int atfork_registered;
>  static LIBXL_LIST_HEAD(, libxl__carefd) carefds =
>      LIBXL_LIST_HEAD_INITIALIZER(carefds);
> 
> +/* non-null iff installed, protected by no_forking */
> +static libxl_ctx *sigchld_owner;
> +static struct sigaction sigchld_saved_action;
> +
> +static void sigchld_removehandler_core(void);
> +
>  static void atfork_lock(void)
>  {
[...]
> @@ -117,6 +124,10 @@ void libxl_postfork_child_noexec(libxl_ctx *ctx)
>          free(cf);
>      }
>      LIBXL_LIST_INIT(&carefds);
> +
> +    if (sigchld_owner)
> +        sigchld_removehandler_core();
> +
>      atfork_unlock();
>  }
> 
> @@ -138,6 +149,238 @@ int libxl__carefd_fd(const libxl__carefd *cf)
[...]
> +
> +int libxl__sigchld_installhandler(libxl_ctx *ctx) /* non-reentrant */
> +{
> +    int r, rc;
> +
> +    if (ctx->sigchld_selfpipe[0] < 0) {
> +        r = pipe(ctx->sigchld_selfpipe);
> +        if (r) {
> +            ctx->sigchld_selfpipe[0] = -1;
> +            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
> +                             "failed to create sigchld pipe");
> +            rc = ERROR_FAIL;
> +            goto out;
> +        }
> +    }
> +
> +    atfork_lock();
> +    if (sigchld_owner != ctx) {

A non-NULL sigchld_owner which is != ctx is a pretty serious error here,
isn't it?

> +        struct sigaction ours;
> +
> +        assert(!sigchld_owner);
> +        sigchld_owner = ctx;
> +
> +        memset(&ours,0,sizeof(ours));
> +        ours.sa_handler = sigchld_handler;
> +        sigemptyset(&ours.sa_mask);
> +        ours.sa_flags = SA_NOCLDSTOP | SA_RESTART;
> +        r = sigaction(SIGCHLD, &ours, &sigchld_saved_action);
> +        assert(!r);
> +
> +        assert(((void)"application must negotiate with libxl about SIGCHLD",
> +                !(sigchld_saved_action.sa_flags & SA_SIGINFO) &&
> +                (sigchld_saved_action.sa_handler == SIG_DFL ||
> +                 sigchld_saved_action.sa_handler == SIG_IGN)));
> +    }
> +    atfork_unlock();
> +
> +    rc = 0;
> + out:
> +    return rc;
> +}
[...]
> +void libxl__fork_selfpipe_woken(libxl__egc *egc) {
> +    /* May make callbacks into the appliction for child processes.

                                      application

> +     * ctx must be locked EXACTLY ONCE */
> +    EGC_GC;
> +
> +    while (chldmode_ours(CTX) /* in case the app changes the mode */) {

setmode takes the CTX lock and ctx must be locked here too, so can this
happen? Ah, we unlock inside the loop, ok then.

Changing the mode in the callback seems, er, ambitious given you can't
call it when libxl has any children and we recommend you do it once at
start of day. Can we not just outlaw it?

> +        int status;
> +        pid_t pid = waitpid(-1, &status, WNOHANG);
> +
> +        if (pid == 0) return;
> +
> +        if (pid == -1) {
> +            if (errno == ECHILD) return;
> +            if (errno == EINTR) continue;
> +            LIBXL__EVENT_DISASTER(egc, "waitpid() failed", errno, 0);
> +            return;
> +        }
> +
> +        int rc = childproc_reaped(egc, pid, status);
> +
> +        if (rc) {
> +            if (CTX->childproc_hooks->reaped_callback) {
> +                CTX_UNLOCK;
> +                CTX->childproc_hooks->reaped_callback
> +                    (pid, status, CTX->childproc_user);
> +                CTX_LOCK;
> +            } else {
> +                libxl_report_child_exitstatus(CTX, XTL_WARN,
> +                                "unknown child", (long)pid, status);
> +            }
> +        }
> +    }
> +}
> +
> +pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *ch,
> +                           libxl__ev_child_callback *death)
> +{
> +    CTX_LOCK;
> +    int rc;
> +
> +    if (chldmode_ours(CTX)) {
> +        rc = libxl__sigchld_installhandler(CTX);
> +        if (rc) goto out;
> +    }
> +
> +    pid_t pid =
> +        CTX->childproc_hooks->fork_replacement
> +        ? CTX->childproc_hooks->fork_replacement(CTX->childproc_user)
> +        : fork();
> +    if (pid == -1) {
> +        LOGE(ERROR, "fork failed");
> +        rc = ERROR_FAIL;
> +        goto out;
> +    }
> +
> +    if (!pid) {
> +        /* woohoo! */

\o/

> +        return 0; /* Yes, CTX is left locked in the child. */
> +    }
> +
> +    ch->pid = pid;
> +    ch->callback = death;
> +    LIBXL_LIST_INSERT_HEAD(&CTX->children, ch, entry);
> +    rc = pid;
> +
> + out:
> +    perhaps_removehandler(CTX);
> +    CTX_UNLOCK;
> +    return rc;
> +}
> +
> +void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
> +                             void *user)
> +{
> +    GC_INIT(ctx);
> +    CTX_LOCK;
> +
> +    assert(LIBXL_LIST_EMPTY(&CTX->children));
> +
> +    if (!hooks)
> +        hooks = &libxl__childproc_default_hooks;
           user = NULL;

(probably doesn't matter much)

> +
> +    ctx->childproc_hooks = hooks;
> +    ctx->childproc_user = user;
> +
> +    switch (ctx->childproc_hooks->chldowner) {
> +    case libxl_sigchld_owner_mainloop:
> +    case libxl_sigchld_owner_libxl:
> +        libxl__sigchld_removehandler(ctx);
> +        break;
> +    case libxl_sigchld_owner_libxl_always:
> +        libxl__sigchld_installhandler(ctx);
> +        break;
> +    default:
> +        abort();
> +    }
> +
> +    CTX_UNLOCK;
> +    GC_FREE;
> +}
> +
> +const libxl_childproc_hooks libxl__childproc_default_hooks = {
> +    libxl_sigchld_owner_libxl, 0, 0
> +};
> +
> +/*
>   * Local variables:
>   * mode: C
>   * c-basic-offset: 4
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 983de3e..0a344cf 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -184,6 +184,19 @@ typedef struct libxl__ev_watch_slot {
>  libxl__ev_xswatch *libxl__watch_slot_contents(libxl__gc *gc, int slotnum);
> 
> 
> +typedef struct libxl__ev_child libxl__ev_child;
> +typedef void libxl__ev_child_callback(libxl__egc *egc, libxl__ev_child*,
> +                                      pid_t pid, int status);
> +struct libxl__ev_child {
> +    /* caller should include this in their own struct */
> +    /* read-only for caller: */
> +    pid_t pid; /* -1 means unused ("unregistered", ie Idle) */
> +    libxl__ev_child_callback *callback;
> +    /* remainder is private for libxl__ev_... */
> +    LIBXL_LIST_ENTRY(struct libxl__ev_child) entry;
> +};
> +
> +
>  /*
>   * evgen structures, which are the state we use for generating
>   * events for the caller.
> @@ -263,6 +276,8 @@ struct libxl__ctx {
>      const libxl_event_hooks *event_hooks;
>      void *event_hooks_user;
> 
> +    LIBXL_LIST_ENTRY(struct libxl__ctx);

Shouldn't there be a field name here? Or is it deliberately anon? (what
if we get a second list)

> +
>      pthread_mutex_t lock; /* protects data structures hanging off the ctx */
>        /* Always use libxl__ctx_lock and _unlock (or the convenience
>         * macors CTX_LOCK and CTX_UNLOCK) to manipulate this.
> @@ -303,10 +318,14 @@ struct libxl__ctx {
> 
>      LIBXL_LIST_HEAD(, libxl_evgen_disk_eject) disk_eject_evgens;
> 
> -    /* for callers who reap children willy-nilly; caller must only
> -     * set this after libxl_init and before any other call - or
> -     * may leave them untouched */
> +    const libxl_childproc_hooks *childproc_hooks;
> +    void *childproc_user;
> +    int sigchld_selfpipe[2]; /* [0]==-1 means handler not installed */
> +    LIBXL_LIST_HEAD(, libxl__ev_child) children;
> +
> +    /* This is obsolete and must be removed: */
>      int (*waitpid_instead)(pid_t pid, int *status, int flags);
> +
>      libxl_version_info version_info;
>  };
> 
> @@ -554,6 +573,33 @@ static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw)
> 
> 
>  /*
> + * For making subprocesses.  This is the only permitted mechanism for
> + * code in libxl to do so.
> + *
> + * In the parent, returns the pid, filling in childw_out.
> + * In the child, returns 0.
> + * If it fails, returns a libxl error (all of which are -ve).
> + *
> + * The child should go on to exec (or exit) soon, and should not make
> + * any further libxl event calls in the meantime.
> + *
> + * The parent may signal the child but it must not reap it.  That will
> + * be done by the event machinery.  Either death or cldstop may be

"cldstop" did you rename that childw_out and miss this comment or is it
something else?

> + * NULL in which case that kind of event is ignored.
> + *
> + * It is not possible to "deregister" the child death event source.
> + * It will generate exactly one event callback; until then the childw
> + * is Active and may not be reused.
> + */
> +_hidden pid_t libxl__ev_child_fork(libxl__gc *gc, libxl__ev_child *childw_out,
> +                                 libxl__ev_child_callback *death);
> +static inline void libxl__ev_child_init(libxl__ev_child *childw_out)
> +                { childw_out->pid = -1; }
> +static inline int libxl__ev_child_inuse(libxl__ev_child *childw_out)
> +                { return childw_out->pid >= 0; }
> +
> +
> +/*
>   * Other event-handling support provided by the libxl event core to
>   * the rest of libxl.
>   */
> @@ -607,6 +653,15 @@ void libxl__poller_put(libxl_ctx *ctx, libxl__poller *p);
>   * ctx must be locked. */
>  void libxl__poller_wakeup(libxl__egc *egc, libxl__poller *p);
> 
> +/* Internal to fork and child reaping machinery */
> +extern const libxl_childproc_hooks libxl__childproc_default_hooks;
> +int libxl__sigchld_installhandler(libxl_ctx *ctx); /* non-reentrant;logs errs */
> +void libxl__sigchld_removehandler(libxl_ctx *ctx); /* non-reentrant */
> +int libxl__fork_selfpipe_active(libxl_ctx *ctx); /* returns read fd or -1 */
> +void libxl__fork_selfpipe_woken(libxl__egc *egc);
> +int libxl__self_pipe_wakeup(int fd); /* returns 0 or -1 setting errno */
> +int libxl__self_pipe_eatall(int fd); /* returns 0 or -1 setting errno */
> +
> 
>  int libxl__atfork_init(libxl_ctx *ctx);
> 
> --
> 1.7.2.5
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH 16/20] libxl: event API: new facilities for waiting for subprocesses

[Ian Jackson]

Thanks for the thorough review!

Ian Campbell writes ("Re: [Xen-devel] [PATCH 16/20] libxl: event API: new facilities for waiting for subprocesses"):
> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > +    /* May make callbacks into the appliction for child processes.
> 
> Typo:                                application

Fixed.

> > +int libxl__self_pipe_eatall(int fd)
> > +{
> > +    char buf[256];
> > +    for (;;) {
> > +        int r = read(fd, buf, sizeof(buf));
> > +        if (r >= 0) return 0;
> 
> Is there some 256 byte limit on the number of bytes pending in the pipe
> somewhere? Wouldn't r==256 require us to (potentially) go round the loop
> again?

If this loop has to go round again, it means we have had 256
wakeup events of some kind.  I don't think this is really an
efficiency concern and it makes the code simpler not to have to worry
about it.

> > +typedef enum {
> > +    /* libxl owns SIGCHLD whenever it has a child. */
> > +    libxl_sigchld_owner_libxl,
> > +
> > +    /* Application promises to call libxl_childproc_exited but NOT
> > +     * from within a signal handler.  libxl will not itself arrange to
> > +     * (un)block or catch SIGCHLD. */
> > +    libxl_sigchld_owner_mainloop,
> > +
> > +    /* libxl owns SIGCHLD all the time, and the application is
> > +     * relying on libxl's event loop for reaping its own children. */
> > +    libxl_sigchld_owner_libxl_always,
> 
> The comment above doesn't seem to cover this third case?

Good point.  I have improved this.

> > +    /* With libxl_sigchld_owner_libxl, called by libxl when it has
> > +     * reaped a pid.  (Not permitted with _owner_mainloop.)
> > +     *
> > +     * Should return 0 if the child was recognised by the application
> > +     * (or if the application does not keep those kind of records),
> > +     * ERROR_NOT_READY if the application knows that the child is not
> 
> ERROR_NOT_READY seems like an odd name. ERROR_UNKNOWN_CHILD perhaps?

I was reusing an existing error code.  Do you think it should have its
own ?

> >  void libxl_postfork_child_noexec(libxl_ctx *ctx);
> >
> >
> > +
> 
> The blank lines appear to be breeding ;-)

OK I think that is one too many :-).

> > +int libxl__sigchld_installhandler(libxl_ctx *ctx) /* non-reentrant */
> > +{
> > +    int r, rc;
> > +
> > +    if (ctx->sigchld_selfpipe[0] < 0) {
> > +        r = pipe(ctx->sigchld_selfpipe);
> > +        if (r) {
> > +            ctx->sigchld_selfpipe[0] = -1;
> > +            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
> > +                             "failed to create sigchld pipe");
> > +            rc = ERROR_FAIL;
> > +            goto out;
> > +        }
> > +    }
> > +
> > +    atfork_lock();
> > +    if (sigchld_owner != ctx) {
> 
> A non-NULL sigchld_owner which is != ctx is a pretty serious error here,
> isn't it?

Yes.

> > +        struct sigaction ours;
> > +
> > +        assert(!sigchld_owner);

Hence this assertion.

> [...]
> > +void libxl__fork_selfpipe_woken(libxl__egc *egc) {
> > +    /* May make callbacks into the appliction for child processes.
> 
>                                       application

I'm rather afflicted (afflicated?) with these.

> > +     * ctx must be locked EXACTLY ONCE */
> > +    EGC_GC;
> > +
> > +    while (chldmode_ours(CTX) /* in case the app changes the mode */) {
> 
> setmode takes the CTX lock and ctx must be locked here too, so can this
> happen? Ah, we unlock inside the loop, ok then.
> 
> Changing the mode in the callback seems, er, ambitious given you can't
> call it when libxl has any children and we recommend you do it once at
> start of day. Can we not just outlaw it?

The callback might well be what makes the application decide it needs
to change the mode.  It would need an if() anyway since you might
respond to the fd readability after the mode has been changed.

> > +void libxl_childproc_setmode(libxl_ctx *ctx, const libxl_childproc_hooks *hooks,
> > +                             void *user)
> > +{
> > +    GC_INIT(ctx);
> > +    CTX_LOCK;
> > +
> > +    assert(LIBXL_LIST_EMPTY(&CTX->children));
> > +
> > +    if (!hooks)
> > +        hooks = &libxl__childproc_default_hooks;
>            user = NULL;
> 
> (probably doesn't matter much)

The default "hooks" naturally don't use user; in fact they are both
null function pointers.

> > @@ -263,6 +276,8 @@ struct libxl__ctx {
> >      const libxl_event_hooks *event_hooks;
> >      void *event_hooks_user;
> >
> > +    LIBXL_LIST_ENTRY(struct libxl__ctx);
> 
> Shouldn't there be a field name here? Or is it deliberately anon? (what
> if we get a second list)

This is a leftover hunk.  It can't actually be used because the
LIBXL_LIST_* macros require an actual member name.  I will remove it.

> > @@ -554,6 +573,33 @@ static inline int libxl__ev_xswatch_isregistered(const libxl__ev_xswatch *xw)
> >
> >
> >  /*
> > + * For making subprocesses.  This is the only permitted mechanism for
> > + * code in libxl to do so.
> > + *
> > + * In the parent, returns the pid, filling in childw_out.
> > + * In the child, returns 0.
> > + * If it fails, returns a libxl error (all of which are -ve).
> > + *
> > + * The child should go on to exec (or exit) soon, and should not make
> > + * any further libxl event calls in the meantime.
> > + *
> > + * The parent may signal the child but it must not reap it.  That will
> > + * be done by the event machinery.  Either death or cldstop may be
> 
> "cldstop" did you rename that childw_out and miss this comment or is it
> something else?

I was going to provide a mechanism analogous to waitpid's WUNTRACED
but thought better of it.  I have fixed the comment.

Thanks,
Ian.

Re: [PATCH 16/20] libxl: event API: new facilities for waiting for subprocesses

[Ian Campbell]

On Tue, 2012-03-20 at 17:24 +0000, Ian Jackson wrote:
> Thanks for the thorough review!
> 
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 16/20] libxl: event API: new facilities for waiting for subprocesses"):
> > On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > > +    /* May make callbacks into the appliction for child processes.
> > 
> > Typo:                                application
> 
> Fixed.
> 
> > > +int libxl__self_pipe_eatall(int fd)
> > > +{
> > > +    char buf[256];
> > > +    for (;;) {
> > > +        int r = read(fd, buf, sizeof(buf));
> > > +        if (r >= 0) return 0;
> > 
> > Is there some 256 byte limit on the number of bytes pending in the pipe
> > somewhere? Wouldn't r==256 require us to (potentially) go round the loop
> > again?
> 
> If this loop has to go round again, it means we have had 256
> wakeup events of some kind.  I don't think this is really an
> efficiency concern and it makes the code simpler not to have to worry
> about it.

My concern is that we would exit the loop _without_ going round again
because if r == 256 then also r>= 0 and we return i.e. we don't actually
"eatall" the buffer, or does that not matter?

> > > +    /* With libxl_sigchld_owner_libxl, called by libxl when it has
> > > +     * reaped a pid.  (Not permitted with _owner_mainloop.)
> > > +     *
> > > +     * Should return 0 if the child was recognised by the application
> > > +     * (or if the application does not keep those kind of records),
> > > +     * ERROR_NOT_READY if the application knows that the child is not
> > 
> > ERROR_NOT_READY seems like an odd name. ERROR_UNKNOWN_CHILD perhaps?
> 
> I was reusing an existing error code.  Do you think it should have its
> own ?

You once told me we shouldn't be shy about adding error codes for
specific things..

> > > +int libxl__sigchld_installhandler(libxl_ctx *ctx) /* non-reentrant */
> > > +{
> > > +    int r, rc;
> > > +
> > > +    if (ctx->sigchld_selfpipe[0] < 0) {
> > > +        r = pipe(ctx->sigchld_selfpipe);
> > > +        if (r) {
> > > +            ctx->sigchld_selfpipe[0] = -1;
> > > +            LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
> > > +                             "failed to create sigchld pipe");
> > > +            rc = ERROR_FAIL;
> > > +            goto out;
> > > +        }
> > > +    }
> > > +
> > > +    atfork_lock();
> > > +    if (sigchld_owner != ctx) {
> > 
> > A non-NULL sigchld_owner which is != ctx is a pretty serious error here,
> > isn't it?
> 
> Yes.
> 
> > > +        struct sigaction ours;
> > > +
> > > +        assert(!sigchld_owner);
> 
> Hence this assertion.

Right.

> > > +     * ctx must be locked EXACTLY ONCE */
> > > +    EGC_GC;
> > > +
> > > +    while (chldmode_ours(CTX) /* in case the app changes the mode */) {
> > 
> > setmode takes the CTX lock and ctx must be locked here too, so can this
> > happen? Ah, we unlock inside the loop, ok then.
> > 
> > Changing the mode in the callback seems, er, ambitious given you can't
> > call it when libxl has any children and we recommend you do it once at
> > start of day. Can we not just outlaw it?
> 
> The callback might well be what makes the application decide it needs
> to change the mode.  It would need an if() anyway since you might
> respond to the fd readability after the mode has been changed.

OK
Ian.

Re: [PATCH 16/20] libxl: event API: new facilities for waiting for subprocesses

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 16/20] libxl: event API: new facilities for waiting for subprocesses"):
> On Tue, 2012-03-20 at 17:24 +0000, Ian Jackson wrote:
> > If this loop has to go round again, it means we have had 256
> > wakeup events of some kind.  I don't think this is really an
> > efficiency concern and it makes the code simpler not to have to worry
> > about it.
> 
> My concern is that we would exit the loop _without_ going round again
> because if r == 256 then also r>= 0 and we return i.e. we don't actually
> "eatall" the buffer, or does that not matter?

So the fd stays readable and we will go round the event loop to read
it again.  Evidently this is too confusing, though, so I'll just make
the function really eat all.

> > > ERROR_NOT_READY seems like an odd name. ERROR_UNKNOWN_CHILD perhaps?
> > 
> > I was reusing an existing error code.  Do you think it should have its
> > own ?
> 
> You once told me we shouldn't be shy about adding error codes for
> specific things..

OK, I'll add one.  Also I notice that I actually ignore the return
value from reaped_callback which is not right.  So I will fix that
too.

Thanks,
Ian.

[PATCH 17/20] libxl: Provide libxl_string_list_length

[Ian Jackson]

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl.c |    8 ++++++++
 tools/libxl/libxl.h |    1 +
 2 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index ee184ca..59992b6 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -195,6 +195,14 @@ void libxl_string_list_dispose(libxl_string_list *psl)
     free(sl);
 }
 
+int libxl_string_list_length(const libxl_string_list *psl)
+{
+    if (!psl) return 0;
+    int i = 0;
+    while (*psl++) i++;
+    return i;
+}
+
 void libxl_key_value_list_dispose(libxl_key_value_list *pkvl)
 {
     int i;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 985806b..50bae2f 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -195,6 +195,7 @@ typedef uint8_t libxl_mac[6];
 
 typedef char **libxl_string_list;
 void libxl_string_list_dispose(libxl_string_list *sl);
+int libxl_string_list_length(const libxl_string_list *sl);
 
 typedef char **libxl_key_value_list;
 void libxl_key_value_list_dispose(libxl_key_value_list *kvl);
-- 
1.7.2.5

Re: [PATCH 17/20] libxl: Provide libxl_string_list_length

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl.c |    8 ++++++++
>  tools/libxl/libxl.h |    1 +
>  2 files changed, 9 insertions(+), 0 deletions(-)
> 
> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> index ee184ca..59992b6 100644
> --- a/tools/libxl/libxl.c
> +++ b/tools/libxl/libxl.c
> @@ -195,6 +195,14 @@ void libxl_string_list_dispose(libxl_string_list *psl)
>      free(sl);
>  }
>  
> +int libxl_string_list_length(const libxl_string_list *psl)
> +{
> +    if (!psl) return 0;
> +    int i = 0;
> +    while (*psl++) i++;
> +    return i;
> +}
> +
>  void libxl_key_value_list_dispose(libxl_key_value_list *pkvl)
>  {
>      int i;
> diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
> index 985806b..50bae2f 100644
> --- a/tools/libxl/libxl.h
> +++ b/tools/libxl/libxl.h
> @@ -195,6 +195,7 @@ typedef uint8_t libxl_mac[6];
>  
>  typedef char **libxl_string_list;
>  void libxl_string_list_dispose(libxl_string_list *sl);
> +int libxl_string_list_length(const libxl_string_list *sl);
>  
>  typedef char **libxl_key_value_list;
>  void libxl_key_value_list_dispose(libxl_key_value_list *kvl);

[PATCH 18/20] libxl: Introduce libxl__sendmsg_fds and libxl__recvmsg_fds

[Ian Jackson]

We will want to reuse the fd-sending code, so break it out into its
own function, and provide the corresponding sending function.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl_internal.h |   11 ++++
 tools/libxl/libxl_qmp.c      |   31 ++-----------
 tools/libxl/libxl_utils.c    |  104 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 119 insertions(+), 27 deletions(-)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 0a344cf..d486af2 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1108,6 +1108,17 @@ _hidden void libxl__qmp_cleanup(libxl__gc *gc, uint32_t domid);
 _hidden int libxl__qmp_initializations(libxl__gc *gc, uint32_t domid,
                                        const libxl_domain_config *guest_config);
 
+/* on failure, logs */
+int libxl__sendmsg_fds(libxl__gc *gc, int carrier,
+                       const void *data, size_t datalen,
+                       int nfds, const int fds[], const char *what);
+
+/* Insists on receiving exactly nfds and datalen.  On failure, logs
+ * and leaves *fds untouched. */
+int libxl__recvmsg_fds(libxl__gc *gc, int carrier,
+                       void *databuf, size_t datalen,
+                       int nfds, int fds[], const char *what);
+
 /* from libxl_json */
 #include <yajl/yajl_gen.h>
 
diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index f5a3edc..83c22b3 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -544,38 +544,15 @@ static int qmp_send_fd(libxl__gc *gc, libxl__qmp_handler *qmp,
                        qmp_request_context *context,
                        int fd)
 {
-    struct msghdr msg = { 0 };
-    struct cmsghdr *cmsg;
-    char control[CMSG_SPACE(sizeof (fd))];
-    struct iovec iov;
     char *buf = NULL;
+    int rc;
 
     buf = qmp_send_prepare(gc, qmp, "getfd", args, callback, opaque, context);
 
-    /* Response data */
-    iov.iov_base = buf;
-    iov.iov_len  = strlen(buf);
-
-    /* compose the message */
-    msg.msg_iov = &iov;
-    msg.msg_iovlen = 1;
-    msg.msg_control = control;
-    msg.msg_controllen = sizeof (control);
-
-    /* attach open fd */
-    cmsg = CMSG_FIRSTHDR(&msg);
-    cmsg->cmsg_level = SOL_SOCKET;
-    cmsg->cmsg_type = SCM_RIGHTS;
-    cmsg->cmsg_len = CMSG_LEN(sizeof (fd));
-    *(int *)CMSG_DATA(cmsg) = fd;
-
-    msg.msg_controllen = cmsg->cmsg_len;
+    rc = libxl__sendmsg_fds(gc, qmp->qmp_fd, buf, strlen(buf), 1, &fd,
+                            "QMP message to QEMU");
+    if (rc) return rc;
 
-    if (sendmsg(qmp->qmp_fd, &msg, 0) < 0) {
-        LIBXL__LOG_ERRNO(qmp->ctx, LIBXL__LOG_ERROR,
-                         "Failed to send a QMP message to QEMU.");
-        return ERROR_FAIL;
-    }
     if (libxl_write_exactly(qmp->ctx, qmp->qmp_fd, "\r\n", 2,
                             "CRLF", "QMP socket")) {
         return ERROR_FAIL;
diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index 0cbd85e..d23ae05 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -500,6 +500,110 @@ void libxl_cputopology_list_free(libxl_cputopology *list, int nr)
     free(list);
 }
 
+int libxl__sendmsg_fds(libxl__gc *gc, int carrier,
+                       const void *data, size_t datalen,
+                       int nfds, const int fds[], const char *what) {
+    struct msghdr msg = { 0 };
+    struct cmsghdr *cmsg;
+    size_t spaceneeded = nfds * sizeof(fds[0]);
+    char control[CMSG_SPACE(spaceneeded)];
+    struct iovec iov;
+    int r;
+
+    iov.iov_base = (void*)data;
+    iov.iov_len  = datalen;
+
+    /* compose the message */
+    msg.msg_iov = &iov;
+    msg.msg_iovlen = 1;
+    msg.msg_control = control;
+    msg.msg_controllen = sizeof(control);
+
+    /* attach open fd */
+    cmsg = CMSG_FIRSTHDR(&msg);
+    cmsg->cmsg_level = SOL_SOCKET;
+    cmsg->cmsg_type = SCM_RIGHTS;
+    cmsg->cmsg_len = CMSG_LEN(spaceneeded);
+    memcpy(CMSG_DATA(cmsg), fds, spaceneeded);
+
+    msg.msg_controllen = cmsg->cmsg_len;
+
+    r = sendmsg(carrier, &msg, 0);
+    if (r < 0) {
+        LOGE(ERROR, "failed to send fd-carrying message (%s)", what);
+        return ERROR_FAIL;
+    }
+
+    return 0;
+}
+
+int libxl__recvmsg_fds(libxl__gc *gc, int carrier,
+                       void *databuf, size_t datalen,
+                       int nfds, int fds[], const char *what)
+{
+    struct msghdr msg = { 0 };
+    struct cmsghdr *cmsg;
+    size_t spaceneeded = nfds * sizeof(fds[0]);
+    char control[CMSG_SPACE(spaceneeded)];
+    struct iovec iov;
+    int r;
+
+    iov.iov_base = databuf;
+    iov.iov_len  = datalen;
+
+    msg.msg_iov = &iov;
+    msg.msg_iovlen = 1;
+    msg.msg_control = control;
+    msg.msg_controllen = sizeof(control);
+
+    for (;;) {
+        r = recvmsg(carrier, &msg, 0);
+        if (r < 0) {
+            if (errno == EINTR) continue;
+            if (errno == EWOULDBLOCK) return -1;
+            LOGE(ERROR,"recvmsg failed (%s)", what);
+            return ERROR_FAIL;
+        }
+        if (r == 0) {
+            LOG(ERROR,"recvmsg got EOF (%s)", what);
+            return ERROR_FAIL;
+        }
+        cmsg = CMSG_FIRSTHDR(&msg);
+        if (cmsg->cmsg_len <= CMSG_LEN(0)) {
+            LOG(ERROR,"recvmsg got no control msg"
+                " when expecting fds (%s)", what);
+            return ERROR_FAIL;
+        }
+        if (cmsg->cmsg_level != SOL_SOCKET || cmsg->cmsg_type != SCM_RIGHTS) {
+            LOG(ERROR, "recvmsg got unexpected"
+                " cmsg_level %d (!=%d) or _type %d (!=%d) (%s)",
+                cmsg->cmsg_level, SOL_SOCKET,
+                cmsg->cmsg_type, SCM_RIGHTS,
+                what);
+            return ERROR_FAIL;
+        }
+        if (cmsg->cmsg_len != CMSG_LEN(spaceneeded) ||
+            msg.msg_controllen != cmsg->cmsg_len) {
+            LOG(ERROR, "recvmsg got unexpected"
+                " number of fds or extra control data"
+                " (%ld bytes' worth, expected %ld) (%s)",
+                (long)CMSG_LEN(spaceneeded), (long)cmsg->cmsg_len,
+                what);
+            int i, fd;
+            unsigned char *p;
+            for (i=0, p=CMSG_DATA(cmsg);
+                 CMSG_SPACE(i * sizeof(fds[0]));
+                 i++, i+=sizeof(fd)) {
+                memcpy(&fd, p, sizeof(fd));
+                close(fd);
+            }
+            return ERROR_FAIL;
+        }
+        memcpy(fds, CMSG_DATA(cmsg), spaceneeded);
+        return 0;
+    }
+}         
+
 void libxl_dominfo_list_free(libxl_dominfo *list, int nr)
 {
     int i;
-- 
1.7.2.5

Re: [PATCH 18/20] libxl: Introduce libxl__sendmsg_fds and libxl__recvmsg_fds

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> We will want to reuse the fd-sending code, so break it out into its
> own function, and provide the corresponding sending function.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

> ---
>  tools/libxl/libxl_internal.h |   11 ++++
>  tools/libxl/libxl_qmp.c      |   31 ++-----------
>  tools/libxl/libxl_utils.c    |  104 ++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 119 insertions(+), 27 deletions(-)
> 
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 0a344cf..d486af2 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -1108,6 +1108,17 @@ _hidden void libxl__qmp_cleanup(libxl__gc *gc, uint32_t domid);
>  _hidden int libxl__qmp_initializations(libxl__gc *gc, uint32_t domid,
>                                         const libxl_domain_config *guest_config);
>  
> +/* on failure, logs */
> +int libxl__sendmsg_fds(libxl__gc *gc, int carrier,
> +                       const void *data, size_t datalen,
> +                       int nfds, const int fds[], const char *what);
> +
> +/* Insists on receiving exactly nfds and datalen.  On failure, logs
> + * and leaves *fds untouched. */
> +int libxl__recvmsg_fds(libxl__gc *gc, int carrier,
> +                       void *databuf, size_t datalen,
> +                       int nfds, int fds[], const char *what);
> +
>  /* from libxl_json */
>  #include <yajl/yajl_gen.h>
>  
> diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
> index f5a3edc..83c22b3 100644
> --- a/tools/libxl/libxl_qmp.c
> +++ b/tools/libxl/libxl_qmp.c
> @@ -544,38 +544,15 @@ static int qmp_send_fd(libxl__gc *gc, libxl__qmp_handler *qmp,
>                         qmp_request_context *context,
>                         int fd)
>  {
> -    struct msghdr msg = { 0 };
> -    struct cmsghdr *cmsg;
> -    char control[CMSG_SPACE(sizeof (fd))];
> -    struct iovec iov;
>      char *buf = NULL;
> +    int rc;
>  
>      buf = qmp_send_prepare(gc, qmp, "getfd", args, callback, opaque, context);
>  
> -    /* Response data */
> -    iov.iov_base = buf;
> -    iov.iov_len  = strlen(buf);
> -
> -    /* compose the message */
> -    msg.msg_iov = &iov;
> -    msg.msg_iovlen = 1;
> -    msg.msg_control = control;
> -    msg.msg_controllen = sizeof (control);
> -
> -    /* attach open fd */
> -    cmsg = CMSG_FIRSTHDR(&msg);
> -    cmsg->cmsg_level = SOL_SOCKET;
> -    cmsg->cmsg_type = SCM_RIGHTS;
> -    cmsg->cmsg_len = CMSG_LEN(sizeof (fd));
> -    *(int *)CMSG_DATA(cmsg) = fd;
> -
> -    msg.msg_controllen = cmsg->cmsg_len;
> +    rc = libxl__sendmsg_fds(gc, qmp->qmp_fd, buf, strlen(buf), 1, &fd,
> +                            "QMP message to QEMU");
> +    if (rc) return rc;
>  
> -    if (sendmsg(qmp->qmp_fd, &msg, 0) < 0) {
> -        LIBXL__LOG_ERRNO(qmp->ctx, LIBXL__LOG_ERROR,
> -                         "Failed to send a QMP message to QEMU.");
> -        return ERROR_FAIL;
> -    }
>      if (libxl_write_exactly(qmp->ctx, qmp->qmp_fd, "\r\n", 2,
>                              "CRLF", "QMP socket")) {
>          return ERROR_FAIL;
> diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
> index 0cbd85e..d23ae05 100644
> --- a/tools/libxl/libxl_utils.c
> +++ b/tools/libxl/libxl_utils.c
> @@ -500,6 +500,110 @@ void libxl_cputopology_list_free(libxl_cputopology *list, int nr)
>      free(list);
>  }
>  
> +int libxl__sendmsg_fds(libxl__gc *gc, int carrier,
> +                       const void *data, size_t datalen,
> +                       int nfds, const int fds[], const char *what) {
> +    struct msghdr msg = { 0 };
> +    struct cmsghdr *cmsg;
> +    size_t spaceneeded = nfds * sizeof(fds[0]);
> +    char control[CMSG_SPACE(spaceneeded)];
> +    struct iovec iov;
> +    int r;
> +
> +    iov.iov_base = (void*)data;
> +    iov.iov_len  = datalen;
> +
> +    /* compose the message */
> +    msg.msg_iov = &iov;
> +    msg.msg_iovlen = 1;
> +    msg.msg_control = control;
> +    msg.msg_controllen = sizeof(control);
> +
> +    /* attach open fd */
> +    cmsg = CMSG_FIRSTHDR(&msg);
> +    cmsg->cmsg_level = SOL_SOCKET;
> +    cmsg->cmsg_type = SCM_RIGHTS;
> +    cmsg->cmsg_len = CMSG_LEN(spaceneeded);
> +    memcpy(CMSG_DATA(cmsg), fds, spaceneeded);
> +
> +    msg.msg_controllen = cmsg->cmsg_len;
> +
> +    r = sendmsg(carrier, &msg, 0);
> +    if (r < 0) {
> +        LOGE(ERROR, "failed to send fd-carrying message (%s)", what);
> +        return ERROR_FAIL;
> +    }
> +
> +    return 0;
> +}
> +
> +int libxl__recvmsg_fds(libxl__gc *gc, int carrier,
> +                       void *databuf, size_t datalen,
> +                       int nfds, int fds[], const char *what)
> +{
> +    struct msghdr msg = { 0 };
> +    struct cmsghdr *cmsg;
> +    size_t spaceneeded = nfds * sizeof(fds[0]);
> +    char control[CMSG_SPACE(spaceneeded)];
> +    struct iovec iov;
> +    int r;
> +
> +    iov.iov_base = databuf;
> +    iov.iov_len  = datalen;
> +
> +    msg.msg_iov = &iov;
> +    msg.msg_iovlen = 1;
> +    msg.msg_control = control;
> +    msg.msg_controllen = sizeof(control);
> +
> +    for (;;) {
> +        r = recvmsg(carrier, &msg, 0);
> +        if (r < 0) {
> +            if (errno == EINTR) continue;
> +            if (errno == EWOULDBLOCK) return -1;
> +            LOGE(ERROR,"recvmsg failed (%s)", what);
> +            return ERROR_FAIL;
> +        }
> +        if (r == 0) {
> +            LOG(ERROR,"recvmsg got EOF (%s)", what);
> +            return ERROR_FAIL;
> +        }
> +        cmsg = CMSG_FIRSTHDR(&msg);
> +        if (cmsg->cmsg_len <= CMSG_LEN(0)) {
> +            LOG(ERROR,"recvmsg got no control msg"
> +                " when expecting fds (%s)", what);
> +            return ERROR_FAIL;
> +        }
> +        if (cmsg->cmsg_level != SOL_SOCKET || cmsg->cmsg_type != SCM_RIGHTS) {
> +            LOG(ERROR, "recvmsg got unexpected"
> +                " cmsg_level %d (!=%d) or _type %d (!=%d) (%s)",
> +                cmsg->cmsg_level, SOL_SOCKET,
> +                cmsg->cmsg_type, SCM_RIGHTS,
> +                what);
> +            return ERROR_FAIL;
> +        }
> +        if (cmsg->cmsg_len != CMSG_LEN(spaceneeded) ||
> +            msg.msg_controllen != cmsg->cmsg_len) {
> +            LOG(ERROR, "recvmsg got unexpected"
> +                " number of fds or extra control data"
> +                " (%ld bytes' worth, expected %ld) (%s)",
> +                (long)CMSG_LEN(spaceneeded), (long)cmsg->cmsg_len,
> +                what);
> +            int i, fd;
> +            unsigned char *p;
> +            for (i=0, p=CMSG_DATA(cmsg);
> +                 CMSG_SPACE(i * sizeof(fds[0]));
> +                 i++, i+=sizeof(fd)) {
> +                memcpy(&fd, p, sizeof(fd));
> +                close(fd);
> +            }
> +            return ERROR_FAIL;
> +        }
> +        memcpy(fds, CMSG_DATA(cmsg), spaceneeded);
> +        return 0;
> +    }
> +}         
> +
>  void libxl_dominfo_list_free(libxl_dominfo *list, int nr)
>  {
>      int i;

[PATCH 19/20] libxl: Clean up setdefault in do_domain_create

[Ian Jackson]

do_domain_create called libxl__domain_create_info_setdefault twice.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Ian Campbell <Ian.Campbell@citrix.com>
---
 tools/libxl/libxl_create.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)

diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 8417661..30dbc32 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -539,9 +539,6 @@ static int do_domain_create(libxl__gc *gc, libxl_domain_config *d_config,
     ret = libxl__domain_create_info_setdefault(gc, &d_config->c_info);
     if (ret) goto error_out;
 
-    ret = libxl__domain_create_info_setdefault(gc, &d_config->c_info);
-    if (ret) goto error_out;
-
     ret = libxl__domain_make(gc, &d_config->c_info, &domid);
     if (ret) {
         LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "cannot make domain: %d", ret);
-- 
1.7.2.5

Re: [PATCH 19/20] libxl: Clean up setdefault in do_domain_create

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> do_domain_create called libxl__domain_create_info_setdefault twice.

Thanks, probably some sort of rebase snafu while I was working on that
series.

> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <Ian.Campbell@citrix.com>

> ---
>  tools/libxl/libxl_create.c |    3 ---
>  1 files changed, 0 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
> index 8417661..30dbc32 100644
> --- a/tools/libxl/libxl_create.c
> +++ b/tools/libxl/libxl_create.c
> @@ -539,9 +539,6 @@ static int do_domain_create(libxl__gc *gc, libxl_domain_config *d_config,
>      ret = libxl__domain_create_info_setdefault(gc, &d_config->c_info);
>      if (ret) goto error_out;
>  
> -    ret = libxl__domain_create_info_setdefault(gc, &d_config->c_info);
> -    if (ret) goto error_out;
> -
>      ret = libxl__domain_make(gc, &d_config->c_info, &domid);
>      if (ret) {
>          LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "cannot make domain: %d", ret);

[PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Ian Jackson]

Convert libxl_run_bootloader to an ao_how-taking function.

It's implemented in terms of libxl__bootloader_run, which can be used
internally.  The resulting code is pretty much a rewrite.

Significant changes include:

- We provide and use a new asynchronous internal facility for pty
  creation, which runs openpty in a child for SIGCHLD etc. safety
  (see the comment at the top of libxl__openptys).

- We direct the bootloader's results to a file, not a pipe.  This
  makes it simpler to deal with as we don't have to read it
  concurrently along with everything else.

- We now issue a warning if we find unexpected statements in the
  bootloader results.

- The arrangements for buffering of bootloader input and output
  are completely changed.  Now we have a fixed limit of 64k
  on output, and 4k on input, and discard the oldest data when
  this overflows (which it shouldn't).  There is no timeout
  any more.

- This is implemented using a new asynchronous internal data-
  copying facility.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 tools/libxl/libxl.c            |    4 +
 tools/libxl/libxl.h            |    3 +-
 tools/libxl/libxl_bootloader.c |  957 ++++++++++++++++++++++++++--------------
 tools/libxl/libxl_create.c     |    8 +-
 tools/libxl/libxl_internal.h   |   97 ++++
 5 files changed, 733 insertions(+), 336 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 59992b6..9ad02b7 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -1748,6 +1748,10 @@ int libxl_device_disk_local_detach(libxl_ctx *ctx, libxl_device_disk *disk)
      * For other device types assume that the blktap2 process is
      * needed by the soon to be started domain and do nothing.
      */
+    /*
+     * FIXME
+     * This appears to leak the disk in failure paths
+     */
 
     return 0;
 }
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 50bae2f..c99b62d 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -415,7 +415,8 @@ int libxl_get_max_cpus(libxl_ctx *ctx);
 int libxl_run_bootloader(libxl_ctx *ctx,
                          libxl_domain_build_info *info,
                          libxl_device_disk *disk,
-                         uint32_t domid);
+                         uint32_t domid,
+                         libxl_asyncop_how *ao_how);
 
   /* 0 means ERROR_ENOMEM, which we have logged */
 
diff --git a/tools/libxl/libxl_bootloader.c b/tools/libxl/libxl_bootloader.c
index 2774062..725fa09 100644
--- a/tools/libxl/libxl_bootloader.c
+++ b/tools/libxl/libxl_bootloader.c
@@ -15,70 +15,219 @@
 #include "libxl_osdeps.h" /* must come before any other headers */
 
 #include <termios.h>
+#include <utmp.h>
 
 #include "libxl_internal.h"
 
-#define XENCONSOLED_BUF_SIZE 16
-#define BOOTLOADER_BUF_SIZE 4096
-#define BOOTLOADER_TIMEOUT 1
+#define BOOTLOADER_BUF_OUT 65536
+#define BOOTLOADER_BUF_IN   4096
 
-static char **make_bootloader_args(libxl__gc *gc,
-                                   libxl_domain_build_info *info,
-                                   uint32_t domid,
-                                   const char *fifo, char *disk)
+static void bootloader_gotptys(libxl__egc *egc, libxl__openpty_state *op);
+static void bootloader_keystrokes_copyfail(libxl__egc *egc,
+       libxl__datacopier_state *dc, int onwrite, int errnoval);
+static void bootloader_display_copyfail(libxl__egc *egc,
+       libxl__datacopier_state *dc, int onwrite, int errnoval);
+static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
+                                pid_t pid, int status);
+
+/*----- openpty -----*/
+
+/*
+ * opens count (>0) ptys like count calls to openpty, and then
+ * calls back.  On entry, all op[].master and op[].slave must be
+ * 0.  On callback, either rc==0 and master and slave are non-0,
+ * or rc is a libxl error and they are both 0.  If libxl__openpty
+ * returns non-0 no callback will happen and everything is left
+ * cleaned up.
+ */
+
+/* implementation */
+    
+static void openpty_cleanup(libxl__openpty_state *op)
+{
+    int i;
+
+    for (i=0; i<op->count; i++) {
+        libxl__openpty_result *res = &op->results[i];
+        libxl__carefd_close(res->master);  res->master = 0;
+        libxl__carefd_close(res->slave);   res->slave = 0;
+    }
+}
+
+static void openpty_exited(libxl__egc *egc, libxl__ev_child *child,
+                           pid_t pid, int status) {
+    libxl__openpty_state *op = CONTAINER_OF(child, *op, child);
+    EGC_GC;
+
+    if (status) {
+        /* Perhaps the child gave us the fds and then exited nonzero.
+         * Well that would be odd but we don't really care. */
+        libxl_report_child_exitstatus(CTX, op->rc ? LIBXL__LOG_ERROR
+                                                  : LIBXL__LOG_WARNING,
+                                      "openpty child", pid, status);
+    }
+    if (op->rc)
+        openpty_cleanup(op);
+    op->callback(egc, op);
+}
+
+int libxl__openptys(libxl__gc *gc, libxl__openpty_state *op,
+                    const struct termios *termp,
+                    const struct winsize *winp) {
+    /*
+     * This is completely crazy.  openpty calls grantpt which the spec
+     * says may fork, and may not be called with a SIGCHLD handler.
+     * Now our application may have a SIGCHLD handler so that's bad.
+     * We could perhaps block it but we'd need to block it on all
+     * threads.  This is just Too Hard.
+     *
+     * So instead, we run openpty in a child process.  That child
+     * process then of course has only our own thread and our own
+     * signal handlers.  We pass the fds back.
+     *
+     * Since our only current caller actually wants two ptys, we
+     * support calling openpty multiple times for a single fork.
+     */
+    int count = op->count;
+    int r, i, rc, sockets[2], ptyfds[count][2];
+    libxl__carefd *for_child = 0;
+    pid_t pid = -1;
+
+    for (i=0; i<count; i++) {
+        ptyfds[i][0] = ptyfds[i][1] = -1;
+        libxl__openpty_result *res = &op->results[i];
+        assert(!res->master);
+        assert(!res->slave);
+    }
+    sockets[0] = sockets[1] = -1; /* 0 is for us, 1 for our child */
+
+    libxl__carefd_begin();
+    r = socketpair(AF_UNIX, SOCK_STREAM, 0, sockets);
+    if (r) { sockets[0] = sockets[1] = -1; }
+    for_child = libxl__carefd_opened(CTX, sockets[1]);
+    if (r) { LOGE(ERROR,"socketpair failed"); rc = ERROR_FAIL; goto out; }
+
+    pid = libxl__ev_child_fork(gc, &op->child, openpty_exited);
+    if (pid == -1) {
+        rc = ERROR_FAIL;
+        goto out;
+    }
+
+    if (!pid) {
+        /* child */
+        close(sockets[0]);
+        signal(SIGCHLD, SIG_DFL);
+
+        for (i=0; i<count; i++) {
+            r = openpty(&ptyfds[i][0], &ptyfds[i][1], NULL, termp, winp);
+            if (r) { LOGE(ERROR,"openpty failed"); _exit(-1); }
+        }
+        rc = libxl__sendmsg_fds(gc, sockets[1], "",1,
+                                2*count, &ptyfds[0][0], "ptys");
+        if (rc) { LOGE(ERROR,"sendmsg to parent failed"); _exit(-1); }
+        _exit(0);
+    }
+
+    libxl__carefd_close(for_child);
+    for_child = 0;
+
+    /* this should be fast so do it synchronously */
+
+    libxl__carefd_begin();
+    char buf[1];
+    rc = libxl__recvmsg_fds(gc, sockets[1], buf,1,
+                            2*count, &ptyfds[0][0], "ptys");
+    if (!rc) {
+        for (i=0; i<count; i++) {
+            libxl__openpty_result *res = &op->results[i];
+            res->master = libxl__carefd_record(CTX, ptyfds[i][0]);
+            res->slave =  libxl__carefd_record(CTX, ptyfds[i][1]);
+        }
+    }
+    /* now the pty fds are in the carefds, if they were ever open */
+    libxl__carefd_unlock();
+    if (rc)
+        goto out;
+
+    rc = 0;
+
+ out:
+    if (sockets[0] >= 0) close(sockets[0]);
+    if (sockets[1] >= 0) close(sockets[1]);
+    libxl__carefd_close(for_child);
+    if (libxl__ev_child_inuse(&op->child)) {
+        op->rc = rc;
+        /* we will get a callback when the child dies */
+        return 0;
+    }
+
+    assert(rc);
+    openpty_cleanup(op);
+    return rc;
+}
+
+/*----- bootloader arguments -----*/
+
+static void bootloader_arg(libxl__bootloader_state *bl, const char *arg) {
+    assert(bl->nargs < bl->argsspace);
+    bl->args[bl->nargs++] = arg;
+}
+
+static void make_bootloader_args(libxl__gc *gc, libxl__bootloader_state *bl)
 {
-    flexarray_t *args;
-    int nr = 0;
+    const libxl_domain_build_info *info = bl->info;
+
+    bl->argsspace = 7 + libxl_string_list_length(&info->u.pv.bootloader_args);
+
+    GCNEW_ARRAY(bl->args, bl->argsspace);
 
-    args = flexarray_make(1, 1);
-    if (!args)
-        return NULL;
+#define ARG(arg) bootloader_arg(bl, (arg))
 
-    flexarray_set(args, nr++, (char *)info->u.pv.bootloader);
+    ARG(info->u.pv.bootloader);
 
     if (info->u.pv.kernel.path)
-        flexarray_set(args, nr++, libxl__sprintf(gc, "--kernel=%s",
-                                                 info->u.pv.kernel.path));
+        ARG(libxl__sprintf(gc, "--kernel=%s", info->u.pv.kernel.path));
     if (info->u.pv.ramdisk.path)
-        flexarray_set(args, nr++, libxl__sprintf(gc, "--ramdisk=%s", info->u.pv.ramdisk.path));
+        ARG(libxl__sprintf(gc, "--ramdisk=%s", info->u.pv.ramdisk.path));
     if (info->u.pv.cmdline && *info->u.pv.cmdline != '\0')
-        flexarray_set(args, nr++, libxl__sprintf(gc, "--args=%s", info->u.pv.cmdline));
+        ARG(libxl__sprintf(gc, "--args=%s", info->u.pv.cmdline));
 
-    flexarray_set(args, nr++, libxl__sprintf(gc, "--output=%s", fifo));
-    flexarray_set(args, nr++, "--output-format=simple0");
-    flexarray_set(args, nr++, libxl__sprintf(gc, "--output-directory=%s", "/var/run/libxl/"));
+    ARG(libxl__sprintf(gc, "--output=%s", bl->outputpath));
+    ARG("--output-format=simple0");
+    ARG(libxl__sprintf(gc, "--output-directory=%s", "/var/run/xen/"));
 
     if (info->u.pv.bootloader_args) {
         char **p = info->u.pv.bootloader_args;
         while (*p) {
-            flexarray_set(args, nr++, *p);
+            ARG(*p);
             p++;
         }
     }
 
-    flexarray_set(args, nr++, disk);
+    ARG(bl->diskpath);
 
     /* Sentinal for execv */
-    flexarray_set(args, nr++, NULL);
+    ARG(NULL);
 
-    return (char **) flexarray_contents(args); /* Frees args */
+#undef ARG
 }
 
-static int open_xenconsoled_pty(int *master, int *slave, char *slave_path, size_t slave_path_len)
+/*----- synchronous subroutines -----*/
+
+static int setup_xenconsoled_pty(libxl__egc *egc, libxl__bootloader_state *bl,
+                                 char *slave_path, size_t slave_path_len)
 {
+    EGC_GC;
     struct termios termattr;
-    int ret;
-
-    ret = openpty(master, slave, NULL, NULL, NULL);
-    if (ret < 0)
-        return -1;
-
-    ret = ttyname_r(*slave, slave_path, slave_path_len);
-    if (ret == -1) {
-        close(*master);
-        close(*slave);
-        *master = *slave = -1;
-        return -1;
+    int r, rc;
+    int slave = libxl__carefd_fd(bl->ptys[1].slave);
+    int master = libxl__carefd_fd(bl->ptys[1].master);
+
+    r = ttyname_r(slave, slave_path, slave_path_len);
+    if (r == -1) {
+        LOGE(ERROR,"ttyname_r failed");
+        rc = ERROR_FAIL;
+        goto out;
     }
 
     /*
@@ -92,309 +241,360 @@ static int open_xenconsoled_pty(int *master, int *slave, char *slave_path, size_
      * for it.
      */
 #if !defined(__sun__) && !defined(__NetBSD__)
-    tcgetattr(*master, &termattr);
+    tcgetattr(master, &termattr);
     cfmakeraw(&termattr);
-    tcsetattr(*master, TCSANOW, &termattr);
+    tcsetattr(master, TCSANOW, &termattr);
 
-    close(*slave);
-    *slave = -1;
+    close(slave);
+    slave = -1;
 #else
-    tcgetattr(*slave, &termattr);
+    tcgetattr(slave, &termattr);
     cfmakeraw(&termattr);
-    tcsetattr(*slave, TCSANOW, &termattr);
+    tcsetattr(slave, TCSANOW, &termattr);
 #endif
 
-    fcntl(*master, F_SETFL, O_NDELAY);
-    fcntl(*master, F_SETFD, FD_CLOEXEC);
-
     return 0;
-}
 
-static pid_t fork_exec_bootloader(int *master, const char *arg0, char **args)
-{
-    struct termios termattr;
-    pid_t pid = forkpty(master, NULL, NULL, NULL);
-    if (pid == -1)
-        return -1;
-    else if (pid == 0) {
-        setenv("TERM", "vt100", 1);
-        libxl__exec(-1, -1, -1, arg0, args);
-        return -1;
-    }
+ out:
+    return rc;
+}
 
-    /*
-     * On Solaris, the master pty side does not have terminal semantics,
-     * so don't try to set any attributes, as it will fail.
-     */
-#if !defined(__sun__)
-    tcgetattr(*master, &termattr);
-    cfmakeraw(&termattr);
-    tcsetattr(*master, TCSANOW, &termattr);
-#endif
+static const char *bootloader_result_command(const char *buf,
+                         const char *prefix, size_t prefixlen) {
+    if (strncmp(buf, prefix, prefixlen))
+        return 0;
 
-    fcntl(*master, F_SETFL, O_NDELAY);
+    const char *rhs = buf + prefixlen;
+    if (!CTYPE(isspace,*rhs))
+        return 0;
 
-    return pid;
+    while (CTYPE(isspace,*rhs))
+        rhs++;
+    return rhs;
 }
 
-/*
- * filedescriptors:
- *   fifo_fd        - bootstring output from the bootloader
- *   xenconsoled_fd - input/output from/to xenconsole
- *   bootloader_fd  - input/output from/to pty that controls the bootloader
- * The filedescriptors are NDELAY, so it's ok to try to read
- * bigger chunks than may be available, to keep e.g. curses
- * screen redraws in the bootloader efficient. xenconsoled_fd is the side that
- * gets xenconsole input, which will be keystrokes, so a small number
- * is sufficient. bootloader_fd is pygrub output, which will be curses screen
- * updates, so a larger number (1024) is appropriate there.
- *
- * For writeable descriptors, only include them in the set for select
- * if there is actual data to write, otherwise this would loop too fast,
- * eating up CPU time.
- */
-static char * bootloader_interact(libxl__gc *gc, int xenconsoled_fd, int bootloader_fd, int fifo_fd)
+static int parse_bootloader_result(libxl__egc *egc,
+                                   libxl__bootloader_state *bl)
 {
-    int ret;
-
-    size_t nr_out = 0, size_out = 0;
-    char *output = NULL;
-    struct timeval wait;
-
-    /* input from xenconsole. read on xenconsoled_fd write to bootloader_fd */
-    int xenconsoled_prod = 0, xenconsoled_cons = 0;
-    char xenconsoled_buf[XENCONSOLED_BUF_SIZE];
-    /* output from bootloader. read on bootloader_fd write to xenconsoled_fd */
-    int bootloader_prod = 0, bootloader_cons = 0;
-    char bootloader_buf[BOOTLOADER_BUF_SIZE];
-
-    while(1) {
-        fd_set wsel, rsel;
-        int nfds;
-
-        /* Set timeout to 1s before starting to discard data */
-        wait.tv_sec = BOOTLOADER_TIMEOUT;
-        wait.tv_usec = 0;
-
-        /* Move buffers around to drop already consumed data */
-        if (xenconsoled_cons > 0) {
-            xenconsoled_prod -= xenconsoled_cons;
-            memmove(xenconsoled_buf, &xenconsoled_buf[xenconsoled_cons],
-                    xenconsoled_prod);
-            xenconsoled_cons = 0;
+    EGC_GC;
+    char buf[PATH_MAX];
+    FILE *f = 0;
+    int rc = ERROR_FAIL;
+    libxl_domain_build_info *info = bl->info;
+
+    f = fopen(bl->outputpath, "r");
+    if (!f) {
+        LOGE(ERROR,"open bootloader output file %s", bl->outputpath);
+        goto out;
+    }
+
+    for (;;) {
+        if (!fgets(buf, sizeof(buf), f)) {
+            if (feof(f)) break;
+            assert(ferror(f));
+            LOGE(ERROR,"read bootloader output file %s", bl->outputpath);
+            goto out;
         }
-        if (bootloader_cons > 0) {
-            bootloader_prod -= bootloader_cons;
-            memmove(bootloader_buf, &bootloader_buf[bootloader_cons],
-                    bootloader_prod);
-            bootloader_cons = 0;
+        int l = strlen(buf);
+        assert(l > 0);
+        if (buf[l-1] != '\n') {
+            LOG(WARN,"bootloader output contained long line `%.50s...'", buf);
+            int c;
+            while ((c=fgetc(f)) != EOF && c !='\n')
+                continue;
+            continue;
         }
 
-        FD_ZERO(&rsel);
-        FD_SET(fifo_fd, &rsel);
-        nfds = fifo_fd + 1;
-        if (xenconsoled_prod < XENCONSOLED_BUF_SIZE) {
-            /* The buffer is not full, try to read more data */
-            FD_SET(xenconsoled_fd, &rsel);
-            nfds = xenconsoled_fd + 1 > nfds ? xenconsoled_fd + 1 : nfds;
-        } 
-        if (bootloader_prod < BOOTLOADER_BUF_SIZE) {
-            /* The buffer is not full, try to read more data */
-            FD_SET(bootloader_fd, &rsel);
-            nfds = bootloader_fd + 1 > nfds ? bootloader_fd + 1 : nfds;
-        }
+        const char *rhs;
+#define COMMAND(s) ((rhs = bootloader_result_command(buf, s, sizeof(s)-1)))
 
-        FD_ZERO(&wsel);
-        if (bootloader_prod > 0) {
-            /* The buffer has data to consume */
-            FD_SET(xenconsoled_fd, &wsel);
-            nfds = xenconsoled_fd + 1 > nfds ? xenconsoled_fd + 1 : nfds;
-        }
-        if (xenconsoled_prod > 0) {
-            /* The buffer has data to consume */
-            FD_SET(bootloader_fd, &wsel);
-            nfds = bootloader_fd + 1 > nfds ? bootloader_fd + 1 : nfds;
+        if (COMMAND("kernel")) {
+            free(info->u.pv.kernel.path);
+            info->u.pv.kernel.path = libxl__strdup(NULL, rhs);
+            libxl__file_reference_map(&info->u.pv.kernel);
+            unlink(info->u.pv.kernel.path);
+        } else if (COMMAND("ramdisk")) {
+            free(info->u.pv.ramdisk.path);
+            info->u.pv.ramdisk.path = libxl__strdup(NULL, rhs);
+            libxl__file_reference_map(&info->u.pv.ramdisk);
+            unlink(info->u.pv.ramdisk.path);
+        } else if (COMMAND("args")) {
+            free(info->u.pv.cmdline);
+            info->u.pv.cmdline = libxl__strdup(NULL, rhs);
+        } else {
+            LOG(WARN, "unexpected output from bootloader: `%s'", buf);
         }
+    }
+    rc = 0;
 
-        if (xenconsoled_prod == XENCONSOLED_BUF_SIZE ||
-            bootloader_prod == BOOTLOADER_BUF_SIZE)
-            ret = select(nfds, &rsel, &wsel, NULL, &wait);
-        else
-            ret = select(nfds, &rsel, &wsel, NULL, NULL);
-        if (ret < 0) {
-            if (errno == EINTR)
-                continue;
-            goto out_err;
-        }
+ out:
+    if (f) fclose(f);
+    return rc;
+}
+
+
+/*----- data copier -----*/
+
+void libxl__datacopier_init(libxl__datacopier_state *dc)
+{
+    libxl__ev_fd_init(&dc->toread);
+    libxl__ev_fd_init(&dc->towrite);
+    LIBXL_TAILQ_INIT(&dc->bufs);
+}
+
+void libxl__datacopier_kill(libxl__gc *gc, libxl__datacopier_state *dc)
+{
+    libxl__datacopier_buf *buf, *tbuf;
+
+    libxl__ev_fd_deregister(gc, &dc->toread);
+    libxl__ev_fd_deregister(gc, &dc->towrite);
+    LIBXL_TAILQ_FOREACH_SAFE(buf, &dc->bufs, entry, tbuf)
+        free(buf);
+    LIBXL_TAILQ_INIT(&dc->bufs);
+}
 
-        /* Input from xenconsole, read xenconsoled_fd, write bootloader_fd */
-        if (ret == 0 && xenconsoled_prod == XENCONSOLED_BUF_SIZE) {
-            /* Drop the buffer */
-            xenconsoled_prod = 0;
-            xenconsoled_cons = 0;
-        } else if (FD_ISSET(xenconsoled_fd, &rsel)) {
-            ret = read(xenconsoled_fd, &xenconsoled_buf[xenconsoled_prod], XENCONSOLED_BUF_SIZE - xenconsoled_prod);
-            if (ret < 0 && errno != EIO && errno != EAGAIN)
-                goto out_err;
-            if (ret > 0)
-                xenconsoled_prod += ret;
+static void datacopier_callback(libxl__egc *egc, libxl__datacopier_state *dc,
+                                int onwrite, int errnoval)
+{
+    EGC_GC;
+    libxl__datacopier_kill(gc, dc);
+    dc->callback(egc, dc, onwrite, errnoval);
+}
+
+static void datacopier_writable(libxl__egc *egc, libxl__ev_fd *ev,
+                                int fd, short events, short revents);
+
+static void datacopier_check_state(libxl__egc *egc, libxl__datacopier_state *dc)
+{
+    EGC_GC;
+    int rc;
+    
+    if (dc->used) {
+        if (!libxl__ev_fd_isregistered(&dc->towrite)) {
+            rc = libxl__ev_fd_register(gc, &dc->towrite, datacopier_writable,
+                                       dc->writefd, POLLOUT);
+            if (!rc) {
+                LOG(ERROR, "unable to establish write event"
+                    " during copy (%s)", dc->what);
+                datacopier_callback(egc, dc, -1, 0);
+                return;
+            }
         }
-        if (FD_ISSET(bootloader_fd, &wsel)) {
-            ret = write(bootloader_fd, &xenconsoled_buf[xenconsoled_cons], xenconsoled_prod - xenconsoled_cons);
-            if (ret < 0 && errno != EIO && errno != EAGAIN)
-                goto out_err;
-            if (ret > 0)
-                xenconsoled_cons += ret;
+    } else if (!libxl__ev_fd_isregistered(&dc->toread)) {
+        /* we have had eof */
+        libxl__datacopier_kill(gc, dc);
+        dc->callback(egc, dc, 0, 0);
+        return;
+    } else {
+        /* nothing buffered, but still reading */
+        libxl__ev_fd_deregister(gc, &dc->towrite);
+    }
+}
+
+static void datacopier_readable(libxl__egc *egc, libxl__ev_fd *ev,
+                                int fd, short events, short revents) {
+    libxl__datacopier_state *dc = CONTAINER_OF(ev, *dc, toread);
+    EGC_GC;
+
+    if (revents & ~POLLIN) {
+        LOG(ERROR, "unexpected poll event 0x%x (should be POLLIN)"
+            " during copy (%s)", revents, dc->what);
+        datacopier_callback(egc, dc, -1, 0);
+        return;
+    }
+    assert(revents & POLLIN);
+    for (;;) {
+        while (dc->used >= dc->maxsz) {
+            libxl__datacopier_buf *rm = LIBXL_TAILQ_FIRST(&dc->bufs);
+            dc->used -= rm->used;
+            assert(dc->used >= 0);
+            LIBXL_TAILQ_REMOVE(&dc->bufs, rm, entry);
+            free(rm);
         }
 
-        /* Input from bootloader, read bootloader_fd, write xenconsoled_fd */
-        if (ret == 0 && bootloader_prod == BOOTLOADER_BUF_SIZE) {
-            /* Drop the buffer */
-            bootloader_prod = 0;
-            bootloader_cons = 0;
-        } else if (FD_ISSET(bootloader_fd, &rsel)) {
-            ret = read(bootloader_fd, &bootloader_buf[bootloader_prod], BOOTLOADER_BUF_SIZE - bootloader_prod);
-            if (ret < 0 && errno != EIO && errno != EAGAIN)
-                goto out_err;
-            if (ret > 0)
-                bootloader_prod += ret;
+        libxl__datacopier_buf *buf =
+            LIBXL_TAILQ_LAST(&dc->bufs, libxl__datacopier_bufs);
+        if (buf->used >= sizeof(buf->buf)) {
+            buf = malloc(sizeof(*buf));
+            if (!buf) libxl__alloc_failed(__func__, 1, sizeof(*buf));
+            buf->used = 0;
+            LIBXL_TAILQ_INSERT_TAIL(&dc->bufs, buf, entry);
+        }
+        int r = read(ev->fd,
+                     buf->buf + buf->used,
+                     sizeof(buf->buf) - buf->used);
+        if (r < 0) {
+            if (errno == EINTR) continue;
+            if (errno == EWOULDBLOCK) break;
+            LOGE(ERROR, "error reading during copy (%s)", dc->what);
+            datacopier_callback(egc, dc, 0, errno);
+            return;
         }
-        if (FD_ISSET(xenconsoled_fd, &wsel)) {
-            ret = write(xenconsoled_fd, &bootloader_buf[bootloader_cons], bootloader_prod - bootloader_cons);
-            if (ret < 0 && errno != EIO && errno != EAGAIN)
-                goto out_err;
-            if (ret > 0)
-                bootloader_cons += ret;
+        if (r == 0) {
+            libxl__ev_fd_deregister(gc, &dc->toread);
+            break;
         }
+        buf->used += r;
+        dc->used += r;
+        assert(buf->used <= sizeof(buf->buf));
+    }
+    datacopier_check_state(egc, dc);
+}
 
-        if (FD_ISSET(fifo_fd, &rsel)) {
-            if (size_out - nr_out < 256) {
-                char *temp;
-                size_t new_size = size_out == 0 ? 32 : size_out * 2;
-
-                temp = realloc(output, new_size);
-                if (temp == NULL)
-                    goto out_err;
-                output = temp;
-                memset(output + size_out, 0, new_size - size_out);
-                size_out = new_size;
-            }
+static void datacopier_writable(libxl__egc *egc, libxl__ev_fd *ev,
+                                int fd, short events, short revents) {
+    EGC_GC;
+    libxl__datacopier_state *dc = CONTAINER_OF(ev, *dc, towrite);
 
-            ret = read(fifo_fd, output + nr_out, size_out - nr_out);
-            if (ret > 0)
-                  nr_out += ret;
-            if (ret == 0)
-                break;
+    if (revents & ~POLLOUT) {
+        LOG(ERROR, "unexpected poll event 0x%x (should be POLLOUT)"
+            " during copy (%s)", revents, dc->what);
+        datacopier_callback(egc, dc, -1, 0);
+        return;
+    }
+    assert(revents & POLLOUT);
+    for (;;) {
+        libxl__datacopier_buf *buf = LIBXL_TAILQ_FIRST(&dc->bufs);
+        if (!buf)
+            break;
+        int r = write(ev->fd, buf->buf, buf->used);
+        if (r < 0) {
+            if (errno == EINTR) continue;
+            if (errno == EWOULDBLOCK) break;
+            LOGE(ERROR, "error writing during copy (%s)", dc->what);
+            datacopier_callback(egc, dc, 1, errno);
+            return;
         }
+        assert(r > 0);
+        assert(r <= buf->used);
+        buf->used -= r;
+        dc->used -= r;
+        assert(dc->used >= 0);
+        memmove(buf->buf, buf->buf+r, buf->used);
     }
+    datacopier_check_state(egc, dc);
+}
+
+int libxl__datacopier_start(libxl__gc *gc, libxl__datacopier_state *dc)
+{
+    int rc;
+
+    libxl__datacopier_init(dc);
+
+    rc = libxl__ev_fd_register(gc, &dc->toread, datacopier_readable,
+                               dc->readfd, POLLIN);
+    if (rc) goto out;
 
-    libxl__ptr_add(gc, output);
-    return output;
+    rc = libxl__ev_fd_register(gc, &dc->towrite, datacopier_writable,
+                               dc->writefd, POLLOUT);
+    if (rc) goto out;
+
+    return 0;
 
-out_err:
-    free(output);
-    return NULL;
+ out:
+    libxl__datacopier_kill(gc, dc);
+    return rc;
 }
 
-static void parse_bootloader_result(libxl__gc *gc,
-                                    libxl_domain_build_info *info,
-                                    const char *o)
+/*----- init and cleanup -----*/
+
+void libxl__bootloader_init(libxl__bootloader_state *bl)
 {
-    while (*o != '\0') {
-        if (strncmp("kernel ", o, strlen("kernel ")) == 0) {
-            free(info->u.pv.kernel.path);
-            info->u.pv.kernel.path = strdup(o + strlen("kernel "));
-            libxl__file_reference_map(&info->u.pv.kernel);
-            unlink(info->u.pv.kernel.path);
-        } else if (strncmp("ramdisk ", o, strlen("ramdisk ")) == 0) {
-            free(info->u.pv.ramdisk.path);
-            info->u.pv.ramdisk.path = strdup(o + strlen("ramdisk "));
-            libxl__file_reference_map(&info->u.pv.ramdisk);
-            unlink(info->u.pv.ramdisk.path);
-        } else if (strncmp("args ", o, strlen("args ")) == 0) {
-            free(info->u.pv.cmdline);
-            info->u.pv.cmdline = strdup(o + strlen("args "));
-        }
+    bl->diskpath = NULL;
+    bl->ptys[0].master = bl->ptys[0].slave = 0;
+    bl->ptys[1].master = bl->ptys[1].slave = 0;
+    libxl__ev_child_init(&bl->child);
+    libxl__datacopier_init(&bl->keystrokes);
+    libxl__datacopier_init(&bl->display);
+}
 
-        o = o + strlen(o) + 1;
+static void bootloader_cleanup(libxl__egc *egc, libxl__bootloader_state *bl)
+{
+    libxl__ao *ao = bl->ao;
+    AO_GC;
+    int i;
+
+    if (bl->diskpath) {
+        libxl_device_disk_local_detach(CTX, bl->disk);
+        free(bl->diskpath);
+        bl->diskpath = 0;
+    }
+    libxl__datacopier_kill(gc, &bl->keystrokes);
+    libxl__datacopier_kill(gc, &bl->display);
+    for (i=0; i<2; i++) {
+        libxl__carefd_close(bl->ptys[0].master);
+        libxl__carefd_close(bl->ptys[0].slave);
     }
 }
 
-int libxl_run_bootloader(libxl_ctx *ctx,
-                         libxl_domain_build_info *info,
-                         libxl_device_disk *disk,
-                         uint32_t domid)
+static void bootloader_callback(libxl__egc *egc, libxl__bootloader_state *bl,
+                                int rc)
 {
-    GC_INIT(ctx);
-    int ret, rc = 0;
-    char *fifo = NULL;
-    char *diskpath = NULL;
-    char **args = NULL;
-
-    char tempdir_template[] = "/var/run/libxl/bl.XXXXXX";
-    char *tempdir;
-
-    char *dom_console_xs_path;
-    char dom_console_slave_tty_path[PATH_MAX];
-
-    int xenconsoled_fd = -1, xenconsoled_slave = -1;
-    int bootloader_fd = -1, fifo_fd = -1;
+    bootloader_cleanup(egc, bl);
+    bl->callback(egc, bl, rc);
+}
 
-    int blrc;
-    pid_t pid;
-    char *blout;
+/*----- main flow of control -----*/
 
-    struct stat st_buf;
+void libxl__bootloader_run(libxl__egc *egc, libxl__bootloader_state *bl)
+{
+    libxl__ao *ao = bl->ao;
+    AO_GC;
+    libxl_domain_build_info *info = bl->info;
+    uint32_t domid = bl->domid;
+    int rc, r;
 
-    rc = libxl__domain_build_info_setdefault(gc, info);
-    if (rc) goto out;
+    libxl__bootloader_init(bl);
 
-    if (info->type != LIBXL_DOMAIN_TYPE_PV || !info->u.pv.bootloader)
+    if (info->type != LIBXL_DOMAIN_TYPE_PV || !info->u.pv.bootloader) {
+        bl->callback(egc, bl, 0);
+        rc = 0;
         goto out;
+    }
 
-    rc = libxl__domain_build_info_setdefault(gc, info);
-    if (rc) goto out;
-
-    rc = ERROR_INVAL;
-    if (!disk)
-        goto out;
+    bl->outputdir = GCSPRINTF(XEN_RUN_DIR "/bootloader.%"PRIu32".d", domid);
+    bl->outputpath = GCSPRINTF(XEN_RUN_DIR "/bootloader.%"PRIu32".out", domid);
 
-    rc = ERROR_FAIL;
-    ret = mkdir("/var/run/libxl/", S_IRWXU);
-    if (ret < 0 && errno != EEXIST)
+    for (;;) {
+        r = mkdir(bl->outputdir, 0600);
+        if (!r) break;
+        if (errno == EINTR) continue;
+        if (errno == EEXIST) break;
+        LOGE(ERROR, "failed to create bootloader dir %s", bl->outputdir);
+        rc = ERROR_FAIL;
         goto out;
+    }
 
-    ret = stat("/var/run/libxl/", &st_buf);
-    if (ret < 0)
-        goto out;
+    make_bootloader_args(gc, bl);
 
-    if (!S_ISDIR(st_buf.st_mode))
+    bl->diskpath = libxl_device_disk_local_attach(CTX, bl->disk);
+    if (!bl->diskpath) {
+        rc = ERROR_FAIL;
         goto out;
+    }
 
-    tempdir = mkdtemp(tempdir_template);
-    if (tempdir == NULL)
-        goto out;
+    bl->openpty.callback = bootloader_gotptys;
+    bl->openpty.count = 2;
+    bl->openpty.results = bl->ptys;
+    rc = libxl__openptys(gc, &bl->openpty, 0,0);
+    if (rc) goto out;
 
-    ret = asprintf(&fifo, "%s/fifo", tempdir);
-    if (ret < 0) {
-        fifo = NULL;
-        goto out_close;
-    }
+    return;
 
-    ret = mkfifo(fifo, 0600);
-    if (ret < 0) {
-        goto out_close;
-    }
+ out:
+    assert(rc);
+    bootloader_callback(egc, bl, rc);
+}
 
-    diskpath = libxl_device_disk_local_attach(ctx, disk);
-    if (!diskpath) {
-        goto out_close;
-    }
+static void bootloader_gotptys(libxl__egc *egc, libxl__openpty_state *op)
+{
+    libxl__bootloader_state *bl = CONTAINER_OF(op, *bl, openpty);
+    EGC_GC;
+    int rc, r;
 
-    args = make_bootloader_args(gc, info, domid, fifo, diskpath);
-    if (args == NULL) {
-        rc = ERROR_NOMEM;
-        goto out_close;
+    if (bl->openpty.rc) {
+        rc = bl->openpty.rc;
+        goto out;
     }
 
     /*
@@ -407,76 +607,167 @@ int libxl_run_bootloader(libxl_ctx *ctx,
      * where we copy characters between the two master fds, as well as
      * listening on the bootloader's fifo for the results.
      */
-    ret = open_xenconsoled_pty(&xenconsoled_fd, &xenconsoled_slave,
+
+    char *dom_console_xs_path;
+    char dom_console_slave_tty_path[PATH_MAX];
+    rc = setup_xenconsoled_pty(egc, bl,
                                &dom_console_slave_tty_path[0],
                                sizeof(dom_console_slave_tty_path));
-    if (ret < 0) {
-        goto out_close;
-    }
+    if (rc) goto out;
+
+    char *dompath = libxl__xs_get_dompath(gc, bl->domid);
+    if (!dompath) { rc = ERROR_FAIL; goto out; }
 
-    dom_console_xs_path = libxl__sprintf(gc, "%s/console/tty", libxl__xs_get_dompath(gc, domid));
-    libxl__xs_write(gc, XBT_NULL, dom_console_xs_path, "%s", dom_console_slave_tty_path);
+    dom_console_xs_path = GCSPRINTF("%s/console/tty", dompath);
 
-    pid = fork_exec_bootloader(&bootloader_fd, info->u.pv.bootloader, args);
-    if (pid < 0) {
-        goto out_close;
+    rc = libxl__xs_write(gc, XBT_NULL, dom_console_xs_path, "%s",
+                         dom_console_slave_tty_path);
+    if (!rc) {
+        LOGE(ERROR,"xs write console path %s := %s failed",
+             dom_console_xs_path, dom_console_slave_tty_path);
+        rc = ERROR_FAIL;
+        goto out;
     }
 
-    while (1) {
-        if (waitpid(pid, &blrc, WNOHANG) == pid)
-            goto out_close;
+    int bootloader_master = libxl__carefd_fd(bl->ptys[0].master);
+    int xenconsole_master = libxl__carefd_fd(bl->ptys[1].master);
 
-        fifo_fd = open(fifo, O_RDONLY);
-        if (fifo_fd > -1)
-            break;
+    libxl_fd_set_nonblock(CTX, bootloader_master, 1);
+    libxl_fd_set_nonblock(CTX, xenconsole_master, 1);
 
-        if (errno == EINTR)
-            continue;
+    bl->keystrokes.readfd = xenconsole_master;
+    bl->keystrokes.writefd = bootloader_master;
+    bl->keystrokes.maxsz = BOOTLOADER_BUF_OUT;
+    bl->keystrokes.what = GCSPRINTF("bootloader input for %"PRIu32, bl->domid);
+    bl->keystrokes.callback = bootloader_keystrokes_copyfail;
+    rc = libxl__datacopier_start(gc, &bl->keystrokes);
+    if (rc) goto out;
 
-        goto out_close;
-    }
+    bl->display.readfd = bootloader_master;
+    bl->display.writefd = xenconsole_master;
+    bl->display.maxsz = BOOTLOADER_BUF_IN;
+    bl->display.what = GCSPRINTF("bootloader output for %"PRIu32, bl->domid);
+    bl->display.callback = bootloader_display_copyfail;
+    rc = libxl__datacopier_start(gc, &bl->display);
+    if (rc) goto out;
 
-    fcntl(fifo_fd, F_SETFL, O_NDELAY);
+    struct termios termattr;
 
-    blout = bootloader_interact(gc, xenconsoled_fd, bootloader_fd, fifo_fd);
-    if (blout == NULL) {
-        goto out_close;
+    pid_t pid = libxl__ev_child_fork(gc, &bl->child, bootloader_finished);
+    if (pid == -1) {
+        rc = ERROR_FAIL;
+        goto out;
     }
 
-    pid = waitpid(pid, &blrc, 0);
-    if (pid == -1 || (pid > 0 && WIFEXITED(blrc) && WEXITSTATUS(blrc) != 0)) {
-        goto out_close;
+    if (!pid) {
+        /* child */
+        r = login_tty(libxl__carefd_fd(bl->ptys[0].slave));
+        if (r) { LOGE(ERROR, "login_tty failed"); exit(-1); }
+        setenv("TERM", "vt100", 1);
+        libxl__exec(-1, -1, -1, bl->args[0], (char**)bl->args);
+        exit(-1);
     }
 
-    parse_bootloader_result(gc, info, blout);
+    /* parent */
+    libxl__carefd_close(bl->ptys[0].slave);
+    bl->ptys[0].slave = 0;
 
-    rc = 0;
-out_close:
-    if (diskpath) {
-        libxl_device_disk_local_detach(ctx, disk);
-        free(diskpath);
+    /*
+     * On Solaris, the master pty side does not have terminal semantics,
+     * so don't try to set any attributes, as it will fail.
+     */
+#if !defined(__sun__)
+    tcgetattr(bootloader_master, &termattr);
+    cfmakeraw(&termattr);
+    tcsetattr(bootloader_master, TCSANOW, &termattr);
+#endif
+
+    return;
+
+ out:
+    bootloader_callback(egc, bl, rc);
+}
+
+/* perhaps one of these will be called, but perhaps not */
+static void bootloader_copyfail(libxl__egc *egc, const char *which,
+       libxl__bootloader_state *bl, int onwrite, int errnoval)
+{
+    EGC_GC;
+
+    if (!onwrite && !errnoval)
+        LOG(ERROR, "unexpected eof copying %s", which);
+    libxl__datacopier_kill(gc, &bl->keystrokes);
+    libxl__datacopier_kill(gc, &bl->display);
+    bl->rc = ERROR_FAIL;
+}
+static void bootloader_keystrokes_copyfail(libxl__egc *egc,
+       libxl__datacopier_state *dc, int onwrite, int errnoval)
+{
+    libxl__bootloader_state *bl = CONTAINER_OF(dc, *bl, keystrokes);
+    bootloader_copyfail(egc, "bootloader input", bl, onwrite, errnoval);
+}
+static void bootloader_display_copyfail(libxl__egc *egc,
+       libxl__datacopier_state *dc, int onwrite, int errnoval)
+{
+    libxl__bootloader_state *bl = CONTAINER_OF(dc, *bl, display);
+    bootloader_copyfail(egc, "bootloader output", bl, onwrite, errnoval);
+}
+
+static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
+                                pid_t pid, int status) {
+    libxl__bootloader_state *bl = CONTAINER_OF(child, *bl, child);
+    EGC_GC;
+    int rc;
+
+    libxl__datacopier_kill(gc, &bl->keystrokes);
+    libxl__datacopier_kill(gc, &bl->display);
+
+    if (status) {
+        libxl_report_child_exitstatus(CTX, XTL_ERROR, "bootloader",
+                                      pid, status);
+        rc = ERROR_FAIL;
+        goto out;
     }
-    if (fifo_fd > -1)
-        close(fifo_fd);
-    if (bootloader_fd > -1)
-        close(bootloader_fd);
-    if (xenconsoled_fd > -1)
-        close(xenconsoled_fd);
-    if (xenconsoled_slave > -1)
-        close(xenconsoled_slave);
-
-    if (fifo) {
-        unlink(fifo);
-        free(fifo);
+
+    if (bl->rc) {
+        /* datacopier went wrong */
+        rc = bl->rc;
+        goto out;
     }
 
-    rmdir(tempdir);
+    rc = parse_bootloader_result(egc, bl);
+    if (rc) goto out;
 
-    free(args);
+    rc = 0;
 
-out:
-    GC_FREE;
-    return rc;
+ out:
+    bootloader_callback(egc, bl, rc);
+}
+
+/*----- entrypoint for external callers -----*/
+
+static void run_bootloader_done(libxl__egc *egc,
+                                libxl__bootloader_state *st, int rc)
+{
+    libxl__ao_complete(egc, st->ao, rc);
+}
+
+int libxl_run_bootloader(libxl_ctx *ctx,
+                         libxl_domain_build_info *info,
+                         libxl_device_disk *disk,
+                         uint32_t domid,
+                         libxl_asyncop_how *ao_how)
+{
+    AO_CREATE(ctx,domid,ao_how);
+    libxl__bootloader_state *bl;
+
+    GCNEW(bl);
+    bl->ao = ao;
+    bl->callback = run_bootloader_done;
+    bl->info = info;
+    bl->disk = disk;
+    libxl__bootloader_run(egc, bl);
+    return AO_INPROGRESS;
 }
 
 /*
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 30dbc32..da461b6 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -559,8 +559,12 @@ static int do_domain_create(libxl__gc *gc, libxl_domain_config *d_config,
         if (ret) goto error_out;
     }
 
-    if ( restore_fd < 0 ) {
-        ret = libxl_run_bootloader(ctx, &d_config->b_info, d_config->num_disks > 0 ? &d_config->disks[0] : NULL, domid);
+    libxl_device_disk *bootdisk =
+        d_config->num_disks > 0 ? &d_config->disks[0] : NULL;
+
+    if (restore_fd < 0 && bootdisk) {
+        ret = libxl_run_bootloader(ctx, &d_config->b_info, bootdisk, domid,
+                                   0 /* fixme-ao */);
         if (ret) {
             LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
                        "failed to run bootloader: %d", ret);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index d486af2..5b73db8 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -42,6 +42,7 @@
 #include <sys/time.h>
 #include <sys/types.h>
 #include <sys/wait.h>
+#include <sys/socket.h>
 
 #include <xs.h>
 #include <xenctrl.h>
@@ -1433,6 +1434,102 @@ int libxl__carefd_close(libxl__carefd*);
 int libxl__carefd_fd(const libxl__carefd*);
 
 
+/*----- openpty -----*/
+
+typedef struct libxl__openpty_state libxl__openpty_state;
+typedef struct libxl__openpty_result libxl__openpty_result;
+typedef void libxl__openpty_callback(libxl__egc *egc, libxl__openpty_state *op);
+
+struct libxl__openpty_state {
+    /* caller must fill these in, and they must all remain valid */
+    libxl__openpty_callback *callback;
+    int count;
+    libxl__openpty_result *results; /* actual size is count, out parameter */
+    /* public, result, caller may only read in callback */
+    int rc;
+    /* private for implementation */
+    libxl__ev_child child;
+};
+
+struct libxl__openpty_result {
+    libxl__carefd *master, *slave;
+};
+
+int libxl__openptys(libxl__gc *gc, libxl__openpty_state *op,
+                    const struct termios *termp,
+                    const struct winsize *winp);
+
+
+/*----- datacopier: copies data from one fd to another -----*/
+
+typedef struct libxl__datacopier_state libxl__datacopier_state;
+typedef struct libxl__datacopier_buf libxl__datacopier_buf;
+
+/* onwrite==1 means failure happend when writing, logged, errno is valid
+ * onwrite==0 means failure happend when reading
+ *     errno==0 means we got eof and all data was written
+ *     errno!=0 means we had a read error, logged
+ * onwrite==-1 means some other internal failure, errnoval not valid, logged
+ * in all cases copier is killed before calling this callback */
+typedef void libxl__datacopier_callback(libxl__egc *egc,
+     libxl__datacopier_state *dc, int onwrite, int errnoval);
+
+struct libxl__datacopier_buf {
+    /* private to datacopier */
+    LIBXL_TAILQ_ENTRY(libxl__datacopier_buf) entry;
+    int used;
+    char buf[1000];
+};
+
+struct libxl__datacopier_state {
+    /* caller must fill these in, and they must all remain valid */
+    int readfd, writefd;
+    ssize_t maxsz;
+    const char *what; /* for error msgs */
+    libxl__datacopier_callback *callback;
+    /* remaining fields are private to datacopier */
+    libxl__ev_fd toread, towrite;
+    ssize_t used;
+    LIBXL_TAILQ_HEAD(libxl__datacopier_bufs, libxl__datacopier_buf) bufs;
+};
+
+_hidden void libxl__datacopier_init(libxl__datacopier_state *dc);
+_hidden void libxl__datacopier_kill(libxl__gc *gc, libxl__datacopier_state *dc);
+_hidden int libxl__datacopier_start(libxl__gc *gc, libxl__datacopier_state *dc);
+
+
+/*----- bootloader -----*/
+
+typedef struct libxl__bootloader_state libxl__bootloader_state;
+typedef void libxl__run_bootloader_callback(libxl__egc*,
+                                libxl__bootloader_state*, int rc);
+
+struct libxl__bootloader_state {
+    /* caller must fill these in, and they must all remain valid */
+    libxl__ao *ao;
+    libxl__run_bootloader_callback *callback;
+    libxl_domain_build_info *info; /* u.pv.{kernel,ramdisk,cmdline} updated */
+    libxl_device_disk *disk;
+    uint32_t domid;
+    /* private to libxl__run_bootloader */
+    char *outputpath, *outputdir;
+    char *diskpath; /* not from gc, represents actually attached disk */
+    libxl__openpty_state openpty;
+    libxl__openpty_result ptys[2];  /* [0] is for bootloader */
+    libxl__ev_child child;
+    int nargs, argsspace;
+    const char **args;
+    libxl__datacopier_state keystrokes, display;
+    int rc;
+};
+
+_hidden void libxl__bootloader_init(libxl__bootloader_state *bl);
+
+/* Will definitely call st->callback, perhaps reentrantly.
+ * If callback is passed rc==0, will have updated st->info appropriately */
+_hidden void libxl__bootloader_run(libxl__egc*, libxl__bootloader_state *st);
+
+
 /*
  * Convenience macros.
  */
-- 
1.7.2.5

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Ian Campbell]

On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> Convert libxl_run_bootloader to an ao_how-taking function.
> 
> It's implemented in terms of libxl__bootloader_run, which can be used
> internally.  The resulting code is pretty much a rewrite.

It's always annoying when diff fails to notice this and treats any blank
line as a common point. I've stripped most of the - lines out to make
something easier to review.

> 
> Significant changes include:
> 
> - We provide and use a new asynchronous internal facility for pty
>   creation, which runs openpty in a child for SIGCHLD etc. safety
>   (see the comment at the top of libxl__openptys).
> 
> - We direct the bootloader's results to a file, not a pipe.  This
>   makes it simpler to deal with as we don't have to read it
>   concurrently along with everything else.
> 
> - We now issue a warning if we find unexpected statements in the
>   bootloader results.
> 
> - The arrangements for buffering of bootloader input and output
>   are completely changed.  Now we have a fixed limit of 64k
>   on output, and 4k on input, and discard the oldest data when
>   this overflows (which it shouldn't).  There is no timeout
>   any more.
> 
> - This is implemented using a new asynchronous internal data-
>   copying facility.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
> ---
>  tools/libxl/libxl.c            |    4 +
>  tools/libxl/libxl.h            |    3 +-
>  tools/libxl/libxl_bootloader.c |  957 ++++++++++++++++++++++++++--------------
>  tools/libxl/libxl_create.c     |    8 +-
>  tools/libxl/libxl_internal.h   |   97 ++++
>  5 files changed, 733 insertions(+), 336 deletions(-)

> diff --git a/tools/libxl/libxl_bootloader.c b/tools/libxl/libxl_bootloader.c
> index 2774062..725fa09 100644
> --- a/tools/libxl/libxl_bootloader.c
> +++ b/tools/libxl/libxl_bootloader.c
> @@ -15,70 +15,219 @@
>  #include "libxl_osdeps.h" /* must come before any other headers */
> 
>  #include <termios.h>
> +#include <utmp.h>
> 
>  #include "libxl_internal.h"
> 
> -#define XENCONSOLED_BUF_SIZE 16
> -#define BOOTLOADER_BUF_SIZE 4096
> -#define BOOTLOADER_TIMEOUT 1
> +#define BOOTLOADER_BUF_OUT 65536
> +#define BOOTLOADER_BUF_IN   4096
> 
> -static char **make_bootloader_args(libxl__gc *gc,
> -                                   libxl_domain_build_info *info,
> -                                   uint32_t domid,
> -                                   const char *fifo, char *disk)
> +static void bootloader_gotptys(libxl__egc *egc, libxl__openpty_state *op);
> +static void bootloader_keystrokes_copyfail(libxl__egc *egc,
> +       libxl__datacopier_state *dc, int onwrite, int errnoval);
> +static void bootloader_display_copyfail(libxl__egc *egc,
> +       libxl__datacopier_state *dc, int onwrite, int errnoval);
> +static void bootloader_finished(libxl__egc *egc, libxl__ev_child *child,
> +                                pid_t pid, int status);
> +
> +/*----- openpty -----*/
> +
> +/*
> + * opens count (>0) ptys like count calls to openpty, and then
> + * calls back.  On entry, all op[].master and op[].slave must be
> + * 0.  On callback, either rc==0 and master and slave are non-0,
> + * or rc is a libxl error and they are both 0.  If libxl__openpty
> + * returns non-0 no callback will happen and everything is left
> + * cleaned up.
> + */
> +
> +/* implementation */
> +
> +static void openpty_cleanup(libxl__openpty_state *op)
> +{
> +    int i;
> +
> +    for (i=0; i<op->count; i++) {
> +        libxl__openpty_result *res = &op->results[i];
> +        libxl__carefd_close(res->master);  res->master = 0;
> +        libxl__carefd_close(res->slave);   res->slave = 0;
> +    }
> +}
> +
> +static void openpty_exited(libxl__egc *egc, libxl__ev_child *child,
> +                           pid_t pid, int status) {
> +    libxl__openpty_state *op = CONTAINER_OF(child, *op, child);
> +    EGC_GC;
> +
> +    if (status) {
> +        /* Perhaps the child gave us the fds and then exited nonzero.
> +         * Well that would be odd but we don't really care. */
> +        libxl_report_child_exitstatus(CTX, op->rc ? LIBXL__LOG_ERROR
> +                                                  : LIBXL__LOG_WARNING,
> +                                      "openpty child", pid, status);
> +    }
> +    if (op->rc)
> +        openpty_cleanup(op);
> +    op->callback(egc, op);
> +}
> +
> +int libxl__openptys(libxl__gc *gc, libxl__openpty_state *op,
> +                    const struct termios *termp,
> +                    const struct winsize *winp) {

"{" on next line please, (here and elsewhere)

libxl__openptys might be deserving of its own home outside
libxl_bootloader.c?

> +    /*
> +     * This is completely crazy.  openpty calls grantpt which the spec
> +     * says may fork, and may not be called with a SIGCHLD handler.
> +     * Now our application may have a SIGCHLD handler so that's bad.
> +     * We could perhaps block it but we'd need to block it on all
> +     * threads.  This is just Too Hard.
> +     *
> +     * So instead, we run openpty in a child process.  That child
> +     * process then of course has only our own thread and our own
> +     * signal handlers.  We pass the fds back.
> +     *
> +     * Since our only current caller actually wants two ptys, we
> +     * support calling openpty multiple times for a single fork.
> +     */
> +    int count = op->count;
> +    int r, i, rc, sockets[2], ptyfds[count][2];
> +    libxl__carefd *for_child = 0;
> +    pid_t pid = -1;
> +
> +    for (i=0; i<count; i++) {
> +        ptyfds[i][0] = ptyfds[i][1] = -1;
> +        libxl__openpty_result *res = &op->results[i];
> +        assert(!res->master);
> +        assert(!res->slave);
> +    }
> +    sockets[0] = sockets[1] = -1; /* 0 is for us, 1 for our child */
> +
> +    libxl__carefd_begin();
> +    r = socketpair(AF_UNIX, SOCK_STREAM, 0, sockets);
> +    if (r) { sockets[0] = sockets[1] = -1; }
> +    for_child = libxl__carefd_opened(CTX, sockets[1]);
> +    if (r) { LOGE(ERROR,"socketpair failed"); rc = ERROR_FAIL; goto out; }
> +
> +    pid = libxl__ev_child_fork(gc, &op->child, openpty_exited);
> +    if (pid == -1) {
> +        rc = ERROR_FAIL;
> +        goto out;
> +    }
> +
> +    if (!pid) {
> +        /* child */
> +        close(sockets[0]);
> +        signal(SIGCHLD, SIG_DFL);
> +
> +        for (i=0; i<count; i++) {
> +            r = openpty(&ptyfds[i][0], &ptyfds[i][1], NULL, termp, winp);
> +            if (r) { LOGE(ERROR,"openpty failed"); _exit(-1); }
> +        }
> +        rc = libxl__sendmsg_fds(gc, sockets[1], "",1,
> +                                2*count, &ptyfds[0][0], "ptys");
> +        if (rc) { LOGE(ERROR,"sendmsg to parent failed"); _exit(-1); }
> +        _exit(0);
> +    }
> +
> +    libxl__carefd_close(for_child);

this is sockets[1], which we then read below (via recvmsg_fds)? I
suspect one or the other is wrong (I think the read).

> +    for_child = 0;
> +
> +    /* this should be fast so do it synchronously */
> +
> +    libxl__carefd_begin();
> +    char buf[1];
> +    rc = libxl__recvmsg_fds(gc, sockets[1], buf,1,
> +                            2*count, &ptyfds[0][0], "ptys");
> +    if (!rc) {
> +        for (i=0; i<count; i++) {
> +            libxl__openpty_result *res = &op->results[i];
> +            res->master = libxl__carefd_record(CTX, ptyfds[i][0]);
> +            res->slave =  libxl__carefd_record(CTX, ptyfds[i][1]);
> +        }
> +    }
> +    /* now the pty fds are in the carefds, if they were ever open */
> +    libxl__carefd_unlock();
> +    if (rc)
> +        goto out;
> +
> +    rc = 0;
> +
> + out:
> +    if (sockets[0] >= 0) close(sockets[0]);
> +    if (sockets[1] >= 0) close(sockets[1]);
> +    libxl__carefd_close(for_child);

for_child == sockets[1], so now we've closed it twice.

> +    if (libxl__ev_child_inuse(&op->child)) {
> +        op->rc = rc;
> +        /* we will get a callback when the child dies */
> +        return 0;
> +    }
> +
> +    assert(rc);
> +    openpty_cleanup(op);
> +    return rc;
> +}
> +
> +/*----- bootloader arguments -----*/
> +
> +static void bootloader_arg(libxl__bootloader_state *bl, const char *arg) {
> +    assert(bl->nargs < bl->argsspace);
> +    bl->args[bl->nargs++] = arg;
> +}

I'm not a huge fan of the flexarray stuff but is reinventing them again
useful?

> +
> +static void make_bootloader_args(libxl__gc *gc, libxl__bootloader_state *bl)
>  {
> +    const libxl_domain_build_info *info = bl->info;
> +
> +    bl->argsspace = 7 + libxl_string_list_length(&info->u.pv.bootloader_args);
> +
> +    GCNEW_ARRAY(bl->args, bl->argsspace);
> 
> +#define ARG(arg) bootloader_arg(bl, (arg))
> 
> +    ARG(info->u.pv.bootloader);
> 
>      if (info->u.pv.kernel.path)
> -        flexarray_set(args, nr++, libxl__sprintf(gc, "--kernel=%s",
> -                                                 info->u.pv.kernel.path));
> +        ARG(libxl__sprintf(gc, "--kernel=%s", info->u.pv.kernel.path));
>      if (info->u.pv.ramdisk.path)
> -        flexarray_set(args, nr++, libxl__sprintf(gc, "--ramdisk=%s", info->u.pv.ramdisk.path));
> +        ARG(libxl__sprintf(gc, "--ramdisk=%s", info->u.pv.ramdisk.path));
>      if (info->u.pv.cmdline && *info->u.pv.cmdline != '\0')
> -        flexarray_set(args, nr++, libxl__sprintf(gc, "--args=%s", info->u.pv.cmdline));
> +        ARG(libxl__sprintf(gc, "--args=%s", info->u.pv.cmdline));
> 
> -    flexarray_set(args, nr++, libxl__sprintf(gc, "--output=%s", fifo));
> -    flexarray_set(args, nr++, "--output-format=simple0");
> -    flexarray_set(args, nr++, libxl__sprintf(gc, "--output-directory=%s", "/var/run/libxl/"));
> +    ARG(libxl__sprintf(gc, "--output=%s", bl->outputpath));
> +    ARG("--output-format=simple0");
> +    ARG(libxl__sprintf(gc, "--output-directory=%s", "/var/run/xen/"));

Output path changed? If we are going to do that then perhaps it should
include an additional element from mkdtemp()?

(comes back later.. why not use XEN_RUN_DIR? Also you seem to have
created bl->outputdir for this purpose anyway?)

>      if (info->u.pv.bootloader_args) {
>          char **p = info->u.pv.bootloader_args;
>          while (*p) {
> -            flexarray_set(args, nr++, *p);
> +            ARG(*p);
>              p++;
>          }
>      }
> 
> -    flexarray_set(args, nr++, disk);
> +    ARG(bl->diskpath);
> 
>      /* Sentinal for execv */

Pre-existing:
          Sentinel

> -    flexarray_set(args, nr++, NULL);
> +    ARG(NULL);
> 
> -    return (char **) flexarray_contents(args); /* Frees args */

Hopefully I'll find blargs getting free'd later on...

> +#undef ARG
>  }
> 
> +/*----- synchronous subroutines -----*/
> +
> +static int setup_xenconsoled_pty(libxl__egc *egc, libxl__bootloader_state *bl,
> +                                 char *slave_path, size_t slave_path_len)
>  {
> +    EGC_GC;
>      struct termios termattr;
> +    int r, rc;
> +    int slave = libxl__carefd_fd(bl->ptys[1].slave);
> +    int master = libxl__carefd_fd(bl->ptys[1].master);
> +
> +    r = ttyname_r(slave, slave_path, slave_path_len);
> +    if (r == -1) {
> +        LOGE(ERROR,"ttyname_r failed");
> +        rc = ERROR_FAIL;
> +        goto out;
>      }
> 
>      /*
> @@ -92,309 +241,360 @@ static int open_xenconsoled_pty(int *master, int *slave, char *slave_path, size_
>       * for it.
>       */
>  #if !defined(__sun__) && !defined(__NetBSD__)
> +    tcgetattr(master, &termattr);
>      cfmakeraw(&termattr);
> +    tcsetattr(master, TCSANOW, &termattr);
> 
> +    close(slave);
> +    slave = -1;
>  #else
> +    tcgetattr(slave, &termattr);
>      cfmakeraw(&termattr);
> +    tcsetattr(slave, TCSANOW, &termattr);
>  #endif

Could this stuff be usefully pushed into libxl__openpty?

> 
> -    fcntl(*master, F_SETFL, O_NDELAY);

Why not?

> -    fcntl(*master, F_SETFD, FD_CLOEXEC);

This one I understand (done by carefd stuff already)

>      return 0;
[...]
> + out:
> +    return rc;
> +}
> 
> -    /*
> -     * On Solaris, the master pty side does not have terminal semantics,
> -     * so don't try to set any attributes, as it will fail.
> -     */
> -#if !defined(__sun__)
> -    tcgetattr(*master, &termattr);
> -    cfmakeraw(&termattr);
> -    tcsetattr(*master, TCSANOW, &termattr);
> -#endif

I think we are right to give up any pretence of supporting Solaris now.
[....]
> +/*----- data copier -----*/
> +static void datacopier_check_state(libxl__egc *egc, libxl__datacopier_state *dc)
> +{
> +    EGC_GC;
> +    int rc;
> +
> +    if (dc->used) {
> +        if (!libxl__ev_fd_isregistered(&dc->towrite)) {
> +            rc = libxl__ev_fd_register(gc, &dc->towrite, datacopier_writable,
> +                                       dc->writefd, POLLOUT);
> +            if (!rc) {
> +                LOG(ERROR, "unable to establish write event"
> +                    " during copy (%s)", dc->what);
> +                datacopier_callback(egc, dc, -1, 0);
> +                return;
> +            }
>          }
[...]
> +    } else if (!libxl__ev_fd_isregistered(&dc->toread)) {
> +        /* we have had eof */
> +        libxl__datacopier_kill(gc, dc);
> +        dc->callback(egc, dc, 0, 0);
> +        return;
> +    } else {
> +        /* nothing buffered, but still reading */
> +        libxl__ev_fd_deregister(gc, &dc->towrite);

is it worth the effort to handle only registering for write events when
something is buffered? It would be simpler to just set it up at start of
day and leave it until done?

> +    }
> +}
> +
> +static void datacopier_readable(libxl__egc *egc, libxl__ev_fd *ev,
> +                                int fd, short events, short revents) {
> +    libxl__datacopier_state *dc = CONTAINER_OF(ev, *dc, toread);
> +    EGC_GC;
> +
> +    if (revents & ~POLLIN) {
> +        LOG(ERROR, "unexpected poll event 0x%x (should be POLLIN)"
> +            " during copy (%s)", revents, dc->what);
> +        datacopier_callback(egc, dc, -1, 0);
> +        return;
> +    }
> +    assert(revents & POLLIN);
> +    for (;;) {
> +        while (dc->used >= dc->maxsz) {

This is the overflow discard loop, correct? (took me a while to figure
that out, in the meantime I was v. confused ;-))

> +            libxl__datacopier_buf *rm = LIBXL_TAILQ_FIRST(&dc->bufs);
> +            dc->used -= rm->used;
> +            assert(dc->used >= 0);
> +            LIBXL_TAILQ_REMOVE(&dc->bufs, rm, entry);
> +            free(rm);
>          }
> 
[...]

> +/*----- init and cleanup -----*/
> +
> +void libxl__bootloader_init(libxl__bootloader_state *bl)
>  {
[...]
> +    bl->diskpath = NULL;
> +    bl->ptys[0].master = bl->ptys[0].slave = 0;
> +    bl->ptys[1].master = bl->ptys[1].slave = 0;

Not -1? A more typical "invalid" fd value.

> +    libxl__ev_child_init(&bl->child);
> +    libxl__datacopier_init(&bl->keystrokes);
> +    libxl__datacopier_init(&bl->display);
> +}
> 
> +static void bootloader_cleanup(libxl__egc *egc, libxl__bootloader_state *bl)
> +{
> +    libxl__ao *ao = bl->ao;
> +    AO_GC;
> +    int i;
> +
> +    if (bl->diskpath) {
> +        libxl_device_disk_local_detach(CTX, bl->disk);
> +        free(bl->diskpath);
> +        bl->diskpath = 0;
> +    }
> +    libxl__datacopier_kill(gc, &bl->keystrokes);
> +    libxl__datacopier_kill(gc, &bl->display);
> +    for (i=0; i<2; i++) {
> +        libxl__carefd_close(bl->ptys[0].master);
> +        libxl__carefd_close(bl->ptys[0].slave);

Don't you mean [i] rather than [0] both times here?

>      }
>  }
[...]
> +/*----- main flow of control -----*/
> 
> +void libxl__bootloader_run(libxl__egc *egc, libxl__bootloader_state *bl)
> +{
> +    libxl__ao *ao = bl->ao;
> +    AO_GC;
> +    libxl_domain_build_info *info = bl->info;
> +    uint32_t domid = bl->domid;
> +    int rc, r;
> 
> +    libxl__bootloader_init(bl);

> 
> -    if (info->type != LIBXL_DOMAIN_TYPE_PV || !info->u.pv.bootloader)
> +    if (info->type != LIBXL_DOMAIN_TYPE_PV || !info->u.pv.bootloader) {
> +        bl->callback(egc, bl, 0);
> +        rc = 0;
>          goto out;
> +    }
> 
[...]
> +    bl->outputdir = GCSPRINTF(XEN_RUN_DIR "/bootloader.%"PRIu32".d", domid);
> +    bl->outputpath = GCSPRINTF(XEN_RUN_DIR "/bootloader.%"PRIu32".out", domid);

You should use this instead of the hardcoded /var/run/xen way above?
> 
[...]
> +    for (;;) {
> +        r = mkdir(bl->outputdir, 0600);
> +        if (!r) break;
> +        if (errno == EINTR) continue;
> +        if (errno == EEXIST) break;
> +        LOGE(ERROR, "failed to create bootloader dir %s", bl->outputdir);
> +        rc = ERROR_FAIL;
>          goto out;
> +    }
> 
[...]
> +    make_bootloader_args(gc, bl);
> 
[...]
> +    bl->diskpath = libxl_device_disk_local_attach(CTX, bl->disk);
> +    if (!bl->diskpath) {
> +        rc = ERROR_FAIL;
>          goto out;
> +    }
> 
[...]
> +    bl->openpty.callback = bootloader_gotptys;
> +    bl->openpty.count = 2;
> +    bl->openpty.results = bl->ptys;
> +    rc = libxl__openptys(gc, &bl->openpty, 0,0);
> +    if (rc) goto out;
> 
[...]
> +    return;
> 
[...]
> + out:
> +    assert(rc);
> +    bootloader_callback(egc, bl, rc);
> +}
> 
[...]
> +static void bootloader_gotptys(libxl__egc *egc, libxl__openpty_state *op)
> +{
> +    libxl__bootloader_state *bl = CONTAINER_OF(op, *bl, openpty);
> +    EGC_GC;
> +    int rc, r;
> 
[...]
> +    if (bl->openpty.rc) {
> +        rc = bl->openpty.rc;
> +        goto out;
>      }
> 
>      /*
> @@ -407,76 +607,167 @@ int libxl_run_bootloader(libxl_ctx *ctx,
>       * where we copy characters between the two master fds, as well as
>       * listening on the bootloader's fifo for the results.
>       */
[...]
> +
> +    char *dom_console_xs_path;
> +    char dom_console_slave_tty_path[PATH_MAX];
> +    rc = setup_xenconsoled_pty(egc, bl,
>                                 &dom_console_slave_tty_path[0],
>                                 sizeof(dom_console_slave_tty_path));
[...]
> +    if (rc) goto out;
> +
> +    char *dompath = libxl__xs_get_dompath(gc, bl->domid);
> +    if (!dompath) { rc = ERROR_FAIL; goto out; }
> 
[...]
> +    dom_console_xs_path = GCSPRINTF("%s/console/tty", dompath);
> 
[...]
> +    rc = libxl__xs_write(gc, XBT_NULL, dom_console_xs_path, "%s",
> +                         dom_console_slave_tty_path);
> +    if (!rc) {
> +        LOGE(ERROR,"xs write console path %s := %s failed",
> +             dom_console_xs_path, dom_console_slave_tty_path);
> +        rc = ERROR_FAIL;
> +        goto out;
>      }
> 
[...]
> +    int bootloader_master = libxl__carefd_fd(bl->ptys[0].master);
> +    int xenconsole_master = libxl__carefd_fd(bl->ptys[1].master);
> 
[...]
> +    libxl_fd_set_nonblock(CTX, bootloader_master, 1);
> +    libxl_fd_set_nonblock(CTX, xenconsole_master, 1);
> 
[...]
> +    bl->keystrokes.readfd = xenconsole_master;
> +    bl->keystrokes.writefd = bootloader_master;
> +    bl->keystrokes.maxsz = BOOTLOADER_BUF_OUT;
> +    bl->keystrokes.what = GCSPRINTF("bootloader input for %"PRIu32, bl->domid);
> +    bl->keystrokes.callback = bootloader_keystrokes_copyfail;
> +    rc = libxl__datacopier_start(gc, &bl->keystrokes);
> +    if (rc) goto out;
> 
[...]
> +    bl->display.readfd = bootloader_master;
> +    bl->display.writefd = xenconsole_master;
> +    bl->display.maxsz = BOOTLOADER_BUF_IN;
> +    bl->display.what = GCSPRINTF("bootloader output for %"PRIu32, bl->domid);
> +    bl->display.callback = bootloader_display_copyfail;
> +    rc = libxl__datacopier_start(gc, &bl->display);
> +    if (rc) goto out;
> 
[...]
> +    struct termios termattr;
> 
[...]
> +    pid_t pid = libxl__ev_child_fork(gc, &bl->child, bootloader_finished);
> +    if (pid == -1) {
> +        rc = ERROR_FAIL;
> +        goto out;
>      }
> 
[...]
> +    if (!pid) {
> +        /* child */
> +        r = login_tty(libxl__carefd_fd(bl->ptys[0].slave));

login_tty is a new one to me. When I lookup up the manpage I happened to
notice:
       Link with -lutil.

> +        if (r) { LOGE(ERROR, "login_tty failed"); exit(-1); }
> +        setenv("TERM", "vt100", 1);
> +        libxl__exec(-1, -1, -1, bl->args[0], (char**)bl->args);
> +        exit(-1);
>      }
> 
[...]
> +    /* parent */
> +    libxl__carefd_close(bl->ptys[0].slave);
> +    bl->ptys[0].slave = 0;
> 
[...]
> +    /*
> +     * On Solaris, the master pty side does not have terminal semantics,
> +     * so don't try to set any attributes, as it will fail.
> +     */

/insert comment about supporting Solaris again here too...

> +#if !defined(__sun__)
> +    tcgetattr(bootloader_master, &termattr);
> +    cfmakeraw(&termattr);
> +    tcsetattr(bootloader_master, TCSANOW, &termattr);
> +#endif
> +
> +    return;
> +
> + out:
> +    bootloader_callback(egc, bl, rc);
> +}
> +
[...]
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index d486af2..5b73db8 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -42,6 +42,7 @@
>  #include <sys/time.h>
>  #include <sys/types.h>
>  #include <sys/wait.h>
> +#include <sys/socket.h>
> 
>  #include <xs.h>
>  #include <xenctrl.h>
> @@ -1433,6 +1434,102 @@ int libxl__carefd_close(libxl__carefd*);
>  int libxl__carefd_fd(const libxl__carefd*);
> 
[...]
> +/*----- datacopier: copies data from one fd to another -----*/
> +
> +typedef struct libxl__datacopier_state libxl__datacopier_state;
> +typedef struct libxl__datacopier_buf libxl__datacopier_buf;
> +
> +/* onwrite==1 means failure happend when writing, logged, errno is valid
> + * onwrite==0 means failure happend when reading

                              happened (twice)
[...]

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > +int libxl__openptys(libxl__gc *gc, libxl__openpty_state *op,
> > +                    const struct termios *termp,
> > +                    const struct winsize *winp) {
> 
> "{" on next line please, (here and elsewhere)

Oops.

> libxl__openptys might be deserving of its own home outside
> libxl_bootloader.c?

I considered this but it's not that long and there are currently no
other callers.

> > +        if (rc) { LOGE(ERROR,"sendmsg to parent failed"); _exit(-1); }
> > +        _exit(0);
> > +    }
> > +
> > +    libxl__carefd_close(for_child);
> 
> this is sockets[1], which we then read below (via recvmsg_fds)? I
> suspect one or the other is wrong (I think the read).

Yes, the read is wrong.

> > + out:
> > +    if (sockets[0] >= 0) close(sockets[0]);
> > +    if (sockets[1] >= 0) close(sockets[1]);
> > +    libxl__carefd_close(for_child);
> 
> for_child == sockets[1], so now we've closed it twice.

Well spotted.

> > +static void bootloader_arg(libxl__bootloader_state *bl, const char *arg) {
> > +    assert(bl->nargs < bl->argsspace);
> > +    bl->args[bl->nargs++] = arg;
> > +}
> 
> I'm not a huge fan of the flexarray stuff but is reinventing them again
> useful?

The flexarray stuff doesn't use the gc.  And we don't need a
variable-sized buffer.  So I don't think it buys us much here.

> (comes back later.. why not use XEN_RUN_DIR? Also you seem to have
> created bl->outputdir for this purpose anyway?)

The bug is that I wasn't using bl->outputdir.  It needs to have the
domid in it for obvious reasons.

> >      /* Sentinal for execv */
> 
> Pre-existing:
>           Sentinel

I'll fix that.

> > -    flexarray_set(args, nr++, NULL);
> > +    ARG(NULL);
> >
> > -    return (char **) flexarray_contents(args); /* Frees args */
> 
> Hopefully I'll find blargs getting free'd later on...

bl->args is from the gc.

More later...

Ian.

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
> On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > @@ -92,309 +241,360 @@ static int open_xenconsoled_pty(int *master, int *slave, char *slave_path, size_
> >       * for it.
> >       */
> >  #if !defined(__sun__) && !defined(__NetBSD__)
> > +    tcgetattr(master, &termattr);
> >      cfmakeraw(&termattr);
> > +    tcsetattr(master, TCSANOW, &termattr);
> >
> > +    close(slave);
> > +    slave = -1;
> >  #else
> > +    tcgetattr(slave, &termattr);
> >      cfmakeraw(&termattr);
> > +    tcsetattr(slave, TCSANOW, &termattr);
> >  #endif
> 
> Could this stuff be usefully pushed into libxl__openpty?

Hmm.  This is a bit of a can of worms I think.  When I wrote my patch
I left it well alone.  TBH I think the current code is arguably wrong.
I can see no reason why we would try to do termios operations on the
pty master at all.  I think these functions should be performed on the
slave on all platforms.

But I don't want to introduce a perhaps-not-wonderfully-portable
change like that in this series.

> > -    fcntl(*master, F_SETFL, O_NDELAY);
> 
> Why not?

This is now done with libxl_fd_set_nonblock immediately after the call
to carefd.

> > -    /*
> > -     * On Solaris, the master pty side does not have terminal semantics,
> > -     * so don't try to set any attributes, as it will fail.
> > -     */
> > -#if !defined(__sun__)
> > -    tcgetattr(*master, &termattr);
> > -    cfmakeraw(&termattr);
> > -    tcsetattr(*master, TCSANOW, &termattr);
> > -#endif
> 
> I think we are right to give up any pretence of supporting Solaris now.

This code has not actually been removed by my patch.  It's still there
at the end of bootloader_gotptys.

> > +    } else if (!libxl__ev_fd_isregistered(&dc->toread)) {
> > +        /* we have had eof */
> > +        libxl__datacopier_kill(gc, dc);
> > +        dc->callback(egc, dc, 0, 0);
> > +        return;
> > +    } else {
> > +        /* nothing buffered, but still reading */
> > +        libxl__ev_fd_deregister(gc, &dc->towrite);
> 
> is it worth the effort to handle only registering for write events when
> something is buffered? It would be simpler to just set it up at start of
> day and leave it until done?

If you register for write events that means the event loop will spin
continually offering you the chance to write something.  Or to put it
another way, the event callback function must either make the original
condition it was triggering on not true, or it must disable the event.

> > +    assert(revents & POLLIN);
> > +    for (;;) {
> > +        while (dc->used >= dc->maxsz) {
> 
> This is the overflow discard loop, correct? (took me a while to figure
> that out, in the meantime I was v. confused ;-))

Yes.  We discard a whole buffer at a time.

> > +void libxl__bootloader_init(libxl__bootloader_state *bl)
> >  {
> [...]
> > +    bl->diskpath = NULL;
> > +    bl->ptys[0].master = bl->ptys[0].slave = 0;
> > +    bl->ptys[1].master = bl->ptys[1].slave = 0;
> 
> Not -1? A more typical "invalid" fd value.

These are libxl__carefd*.

> > +    for (i=0; i<2; i++) {
> > +        libxl__carefd_close(bl->ptys[0].master);
> > +        libxl__carefd_close(bl->ptys[0].slave);
> 
> Don't you mean [i] rather than [0] both times here?

Yes, oops.

> [...]
> > +    if (!pid) {
> > +        /* child */
> > +        r = login_tty(libxl__carefd_fd(bl->ptys[0].slave));
> 
> login_tty is a new one to me. When I lookup up the manpage I happened to
> notice:
>        Link with -lutil.

Hmm.  The BSD manpage is more forthcoming and suggests we need
something different there.  (This isn't a particularly portable
function.)

I think we need some more autoconfery.

> > +/* onwrite==1 means failure happend when writing, logged, errno is valid
> > + * onwrite==0 means failure happend when reading
> 
>                               happened (twice)

Fixed.

Ian.

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Ian Campbell]

On Wed, 2012-03-21 at 12:17 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
> > On Fri, 2012-03-16 at 16:26 +0000, Ian Jackson wrote:
> > > @@ -92,309 +241,360 @@ static int open_xenconsoled_pty(int *master, int *slave, char *slave_path, size_
> > >       * for it.
> > >       */
> > >  #if !defined(__sun__) && !defined(__NetBSD__)
> > > +    tcgetattr(master, &termattr);
> > >      cfmakeraw(&termattr);
> > > +    tcsetattr(master, TCSANOW, &termattr);
> > >
> > > +    close(slave);
> > > +    slave = -1;
> > >  #else
> > > +    tcgetattr(slave, &termattr);
> > >      cfmakeraw(&termattr);
> > > +    tcsetattr(slave, TCSANOW, &termattr);
> > >  #endif
> > 
> > Could this stuff be usefully pushed into libxl__openpty?
> 
> Hmm.  This is a bit of a can of worms I think.  When I wrote my patch
> I left it well alone.  TBH I think the current code is arguably wrong.
> I can see no reason why we would try to do termios operations on the
> pty master at all.  I think these functions should be performed on the
> slave on all platforms.
> 
> But I don't want to introduce a perhaps-not-wonderfully-portable
> change like that in this series.

That's very reasonable...

FWIW most of this code was copied from the xend equivalent (actually a C
python binding) which I think was actually cobbled together from actual
users of the platforms in question, not that this makes it correct
necessarily.

> This is now done with libxl_fd_set_nonblock immediately after the call
> to carefd.
[...]
> This code has not actually been removed by my patch.  It's still there
> at the end of bootloader_gotptys.

Sorry, I spotted both of these later and neglected to come back and
remove these comments.

> > > +    } else if (!libxl__ev_fd_isregistered(&dc->toread)) {
> > > +        /* we have had eof */
> > > +        libxl__datacopier_kill(gc, dc);
> > > +        dc->callback(egc, dc, 0, 0);
> > > +        return;
> > > +    } else {
> > > +        /* nothing buffered, but still reading */
> > > +        libxl__ev_fd_deregister(gc, &dc->towrite);
> > 
> > is it worth the effort to handle only registering for write events when
> > something is buffered? It would be simpler to just set it up at start of
> > day and leave it until done?
> 
> If you register for write events that means the event loop will spin
> continually offering you the chance to write something.  Or to put it
> another way, the event callback function must either make the original
> condition it was triggering on not true, or it must disable the event.

Right, this is obvious when you put it that way. In my defence it had
been a long review session...

Ian.

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Ian Jackson]

Ian Jackson writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
> Ian Campbell writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
> > Could this stuff be usefully pushed into libxl__openpty?
> 
> Hmm.  This is a bit of a can of worms I think.  When I wrote my patch
> I left it well alone.  TBH I think the current code is arguably wrong.
> I can see no reason why we would try to do termios operations on the
> pty master at all.  I think these functions should be performed on the
> slave on all platforms.
> 
> But I don't want to introduce a perhaps-not-wonderfully-portable
> change like that in this series.

Just to be clear: I left all of this pty termios code alone in my
series; it just shuffles about a bit but is otherwise unchanged as far
as possible.  The only intentional difference is that I replaced
forkpty with our own fork + openpty + login_tty.

> > login_tty is a new one to me. When I lookup up the manpage I happened to
> > notice:
> >        Link with -lutil.
> 
> Hmm.  The BSD manpage is more forthcoming and suggests we need
> something different there.  (This isn't a particularly portable
> function.)
> 
> I think we need some more autoconfery.

Like the below.  Previously libxl was linking with -lutil by virtue of
using UTIL_LIBS which came from config/StdGNU.mk.

Ian.

From: Ian Jackson <ian.jackson@eu.citrix.com>
Subject: [PATCH] autoconf: New test for openpty et al.

We may need to #include <libutil.h>, and/or link with -lutil, to use
openpty, login_tty, and the like.  Provide INCLUDE_LIBUTIL_H
(preprocessor constant, not always defined) and PTYFUNCS_LIBS
(makefile variable).

We link libxl against PTYFUNCS_LIBS (which comes from autoconf) rather
than UTIL_LIBS, and #include <libutil.h> where appropriate.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

---
 config/Tools.mk.in             |    2 +
 tools/configure                |   57 ++++++++++++++++++++++++++++++++++++++++
 tools/configure.ac             |    2 +
 tools/libxl/Makefile           |    2 +-
 tools/libxl/libxl_bootloader.c |    4 +++
 tools/m4/ptyfuncs.m4           |   24 +++++++++++++++++
 6 files changed, 90 insertions(+), 1 deletions(-)

diff --git a/config/Tools.mk.in b/config/Tools.mk.in
index 8247e4a..ddcb10e 100644
--- a/config/Tools.mk.in
+++ b/config/Tools.mk.in
@@ -27,6 +27,8 @@ PTHREAD_CFLAGS      := @PTHREAD_CFLAGS@
 PTHREAD_LDFLAGS     := @PTHREAD_LDFLAGS@
 PTHREAD_LIBS        := @PTHREAD_LIBS@
 
+PTYFUNCS_LIBS       := @PTYFUNCS_LIBS@
+
 # Download GIT repositories via HTTP or GIT's own protocol?
 # GIT's protocol is faster and more robust, when it works at all (firewalls
 # may block it). We make it the default, but if your GIT repository downloads
diff --git a/tools/configure b/tools/configure
index 7ee4e3e..e35eaa0 100755
--- a/tools/configure
+++ b/tools/configure
@@ -602,6 +602,7 @@ POW_LIB
 LIBOBJS
 ALLOCA
 libiconv
+PTYFUNCS_LIBS
 PTHREAD_LIBS
 PTHREAD_LDFLAGS
 PTHREAD_CFLAGS
@@ -3913,9 +3914,17 @@ case $host_os in *\ *) host_os=`echo "$host_os" | sed 's/ /-/g'`;; esac
 
 
 
+# We define, separately, PTHREAD_CFLAGS, _LDFLAGS and _LIBS
+# even though currently we don't set them very separately.
+# This means that the makefiles will not need to change in
+# the future if we make the test more sophisticated.
 
 
 
+# We invoke AX_PTHREAD_VARS with the name of another macro
+# which is then expanded once for each variable.
+
+
 
 
 
@@ -7091,6 +7100,54 @@ $as_echo "$ax_cv_pthread_flags" >&6; }
 
 
 
+
+    ac_fn_c_check_header_mongrel "$LINENO" "libutil.h" "ac_cv_header_libutil_h" "$ac_includes_default"
+if test "x$ac_cv_header_libutil_h" = x""yes; then :
+
+
+$as_echo "#define INCLUDE_LIBUTIL_H <libutil.h>" >>confdefs.h
+
+
+fi
+
+
+    { $as_echo "$as_me:${as_lineno-$LINENO}: checking for openpty et al" >&5
+$as_echo_n "checking for openpty et al... " >&6; }
+if test "${ax_cv_ptyfuncs_libs+set}" = set; then :
+  $as_echo_n "(cached) " >&6
+else
+
+        ax_cv_ptyfuncs_libs=-lutil
+
+    saved_LIBS="$LIBS"
+
+        LIBS="$LIBS $ax_cv_ptyfuncs_libs"
+        cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+#ifdef INCLUDE_LIBUTIL_H
+#include INCLUDE_LIBUTIL_H
+#endif
+int main(void) {
+  openpty(0,0,0,0,0);
+  login_tty(0);
+}
+
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+
+    LIBS="$saved_LIBS"
+
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ax_cv_ptyfuncs_libs" >&5
+$as_echo "$ax_cv_ptyfuncs_libs" >&6; }
+    PTYFUNCS_LIBS="$ax_cv_ptyfuncs_libs"
+
+
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_gettime in -lrt" >&5
 $as_echo_n "checking for clock_gettime in -lrt... " >&6; }
 if test "${ac_cv_lib_rt_clock_gettime+set}" = set; then :
diff --git a/tools/configure.ac b/tools/configure.ac
index 3da0c82..936a192 100644
--- a/tools/configure.ac
+++ b/tools/configure.ac
@@ -36,6 +36,7 @@ m4_include([m4/uuid.m4])
 m4_include([m4/pkg.m4])
 m4_include([m4/curses.m4])
 m4_include([m4/pthread.m4])
+m4_include([m4/ptyfuncs.m4])
 
 # Enable/disable options
 AX_ARG_ENABLE_AND_EXPORT([githttp], [Download GIT repositories via HTTP])
@@ -126,6 +127,7 @@ AC_SUBST(libext2fs)
 AC_CHECK_LIB([gcrypt], [gcry_md_hash_buffer], [libgcrypt="y"], [libgcrypt="n"])
 AC_SUBST(libgcrypt)
 AX_CHECK_PTHREAD
+AX_CHECK_PTYFUNCS
 AC_CHECK_LIB([rt], [clock_gettime])
 AC_CHECK_LIB([yajl], [yajl_alloc], [],
     [AC_MSG_ERROR([Could not find yajl])])
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index a88827a..2c9aebf 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -20,7 +20,7 @@ LIBUUID_LIBS += -luuid
 endif
 
 LIBXL_LIBS =
-LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(UTIL_LIBS) $(LIBUUID_LIBS)
+LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS)
 
 CFLAGS += $(PTHREAD_CFLAGS)
 LDFLAGS += $(PTHREAD_LDFLAGS)
diff --git a/tools/libxl/libxl_bootloader.c b/tools/libxl/libxl_bootloader.c
index 2774062..b50944a 100644
--- a/tools/libxl/libxl_bootloader.c
+++ b/tools/libxl/libxl_bootloader.c
@@ -16,6 +16,10 @@
 
 #include <termios.h>
 
+#ifdef INCLUDE_LIBUTIL_H
+#include INCLUDE_LIBUTIL_H
+#endif
+
 #include "libxl_internal.h"
 
 #define XENCONSOLED_BUF_SIZE 16
diff --git a/tools/m4/ptyfuncs.m4 b/tools/m4/ptyfuncs.m4
new file mode 100644
index 0000000..2846a6d
--- /dev/null
+++ b/tools/m4/ptyfuncs.m4
@@ -0,0 +1,24 @@
+AC_DEFUN([AX_CHECK_PTYFUNCS], [
+    AC_CHECK_HEADER([libutil.h],[
+      AC_DEFINE([INCLUDE_LIBUTIL_H],[<libutil.h>],[libutil header file name])
+    ])
+    AC_CACHE_CHECK([for openpty et al], [ax_cv_ptyfuncs_libs], [
+        ax_cv_ptyfuncs_libs=-lutil
+        AX_SAVEVAR_SAVE(LIBS)
+        LIBS="$LIBS $ax_cv_ptyfuncs_libs"
+        AC_LINK_IFELSE([
+#ifdef INCLUDE_LIBUTIL_H
+#include INCLUDE_LIBUTIL_H
+#endif
+int main(void) {
+  openpty(0,0,0,0,0);
+  login_tty(0);
+}
+])
+        AX_SAVEVAR_RESTORE(LIBS)],
+    [],[
+        AC_MSG_FAILURE([Unable to find -lutil for openpty and login_tty])
+    ])
+    PTYFUNCS_LIBS="$ax_cv_ptyfuncs_libs"
+    AC_SUBST(PTYFUNCS_LIBS)
+])
-- 
tg: (154e097..) t/xen/tools/ptyfuncs (depends on: t/xen/xl.event.children)

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Roger Pau Monné]

2012/3/21 Ian Jackson <Ian.Jackson@eu.citrix.com>:
> Ian Jackson writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
>> Ian Campbell writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
>> > Could this stuff be usefully pushed into libxl__openpty?
>>
>> Hmm.  This is a bit of a can of worms I think.  When I wrote my patch
>> I left it well alone.  TBH I think the current code is arguably wrong.
>> I can see no reason why we would try to do termios operations on the
>> pty master at all.  I think these functions should be performed on the
>> slave on all platforms.
>>
>> But I don't want to introduce a perhaps-not-wonderfully-portable
>> change like that in this series.
>
> Just to be clear: I left all of this pty termios code alone in my
> series; it just shuffles about a bit but is otherwise unchanged as far
> as possible.  The only intentional difference is that I replaced
> forkpty with our own fork + openpty + login_tty.
>
>> > login_tty is a new one to me. When I lookup up the manpage I happened to
>> > notice:
>> >        Link with -lutil.
>>
>> Hmm.  The BSD manpage is more forthcoming and suggests we need
>> something different there.  (This isn't a particularly portable
>> function.)
>>
>> I think we need some more autoconfery.
>
> Like the below.  Previously libxl was linking with -lutil by virtue of
> using UTIL_LIBS which came from config/StdGNU.mk.
>
> Ian.
>
> From: Ian Jackson <ian.jackson@eu.citrix.com>
> Subject: [PATCH] autoconf: New test for openpty et al.
>
> We may need to #include <libutil.h>, and/or link with -lutil, to use
> openpty, login_tty, and the like.  Provide INCLUDE_LIBUTIL_H
> (preprocessor constant, not always defined) and PTYFUNCS_LIBS
> (makefile variable).
>
> We link libxl against PTYFUNCS_LIBS (which comes from autoconf) rather
> than UTIL_LIBS, and #include <libutil.h> where appropriate.
>
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
> ---
>  config/Tools.mk.in             |    2 +
>  tools/configure                |   57 ++++++++++++++++++++++++++++++++++++++++
>  tools/configure.ac             |    2 +
>  tools/libxl/Makefile           |    2 +-
>  tools/libxl/libxl_bootloader.c |    4 +++
>  tools/m4/ptyfuncs.m4           |   24 +++++++++++++++++
>  6 files changed, 90 insertions(+), 1 deletions(-)
[...]
> diff --git a/tools/m4/ptyfuncs.m4 b/tools/m4/ptyfuncs.m4
> new file mode 100644
> index 0000000..2846a6d
> --- /dev/null
> +++ b/tools/m4/ptyfuncs.m4
> @@ -0,0 +1,24 @@
> +AC_DEFUN([AX_CHECK_PTYFUNCS], [
> +    AC_CHECK_HEADER([libutil.h],[
> +      AC_DEFINE([INCLUDE_LIBUTIL_H],[<libutil.h>],[libutil header file name])
> +    ])
> +    AC_CACHE_CHECK([for openpty et al], [ax_cv_ptyfuncs_libs], [
> +        ax_cv_ptyfuncs_libs=-lutil
> +        AX_SAVEVAR_SAVE(LIBS)
> +        LIBS="$LIBS $ax_cv_ptyfuncs_libs"
> +        AC_LINK_IFELSE([
> +#ifdef INCLUDE_LIBUTIL_H
> +#include INCLUDE_LIBUTIL_H
> +#endif

Shouldn't you add something like this (or perhaps add the necessary
AC_CHECK_HEADER for this headers, so we can get rid of the OS
conditional include dependency):

#if defined(__NetBSD__) || defined(__OpenBSD__)
#include <util.h>
#elif defined(__linux__)
#include <pty.h>
#elif defined(__sun__)
#include <stropts.h>
#endif
#include <utmp.h>

> +int main(void) {
> +  openpty(0,0,0,0,0);
> +  login_tty(0);
> +}
> +])
> +        AX_SAVEVAR_RESTORE(LIBS)],
> +    [],[
> +        AC_MSG_FAILURE([Unable to find -lutil for openpty and login_tty])
> +    ])
> +    PTYFUNCS_LIBS="$ax_cv_ptyfuncs_libs"
> +    AC_SUBST(PTYFUNCS_LIBS)
> +])
> --
> tg: (154e097..) t/xen/tools/ptyfuncs (depends on: t/xen/xl.event.children)
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Ian Jackson]

Roger Pau Monné writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
> > +    AC_CHECK_HEADER([libutil.h],[
> > +      AC_DEFINE([INCLUDE_LIBUTIL_H],[<libutil.h>],[libutil header file name])
> > +    ])
...
> Shouldn't you add something like this (or perhaps add the necessary
> AC_CHECK_HEADER for this headers, so we can get rid of the OS
> conditional include dependency):
> 
> #if defined(__NetBSD__) || defined(__OpenBSD__)
> #include <util.h>
> #elif defined(__linux__)
> #include <pty.h>
> #elif defined(__sun__)
> #include <stropts.h>
> #endif
> #include <utmp.h>

No, I don't think so.  Currently the only two platforms we really
support are *BSD and Linux, and I think what I have (include
<libutil.h> if it exists, and nothing non-portable otherwise) is
correct.

Does this test not work on *BSD ?

Certainly we don't want these kind of os-specific #ifdefs.  If there
are multiple possible headers we should call AC_CHECK_HEADER for them.

Ian.

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Roger Pau Monné]

2012/3/22 Ian Jackson <Ian.Jackson@eu.citrix.com>:
> Roger Pau Monné writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
>> > +    AC_CHECK_HEADER([libutil.h],[
>> > +      AC_DEFINE([INCLUDE_LIBUTIL_H],[<libutil.h>],[libutil header file name])
>> > +    ])
> ...
>> Shouldn't you add something like this (or perhaps add the necessary
>> AC_CHECK_HEADER for this headers, so we can get rid of the OS
>> conditional include dependency):
>>
>> #if defined(__NetBSD__) || defined(__OpenBSD__)
>> #include <util.h>
>> #elif defined(__linux__)
>> #include <pty.h>
>> #elif defined(__sun__)
>> #include <stropts.h>
>> #endif
>> #include <utmp.h>
>
> No, I don't think so.  Currently the only two platforms we really
> support are *BSD and Linux, and I think what I have (include
> <libutil.h> if it exists, and nothing non-portable otherwise) is
> correct.
>
> Does this test not work on *BSD ?

I haven't tried yet, in fact I have not tried this patch, it's just
that the manpage states that on NetBSD at least you have to include
util.h, it doesn't mention libutil.h.

http://netbsd.gw.com/cgi-bin/man-cgi?forkpty++NetBSD-current

And the linux man page states that pty.h and utmp.h
(http://linux.die.net/man/3/forkpty) should be included to use this
functions under Linux, but I think you have that covered. Anyway,
those includes are inside libxl_osdeps.h, so if they are not necessary
we should get rid of them, or add the necessary autoconf stuff to
handle that without all the OS conditionals.

> Certainly we don't want these kind of os-specific #ifdefs.  If there
> are multiple possible headers we should call AC_CHECK_HEADER for them.
>
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Roger Pau Monné]

2012/3/16 Ian Jackson <ian.jackson@eu.citrix.com>:
> Convert libxl_run_bootloader to an ao_how-taking function.
>
> It's implemented in terms of libxl__bootloader_run, which can be used
> internally.  The resulting code is pretty much a rewrite.
>
> Significant changes include:
>
> - We provide and use a new asynchronous internal facility for pty
>  creation, which runs openpty in a child for SIGCHLD etc. safety
>  (see the comment at the top of libxl__openptys).
>
> - We direct the bootloader's results to a file, not a pipe.  This
>  makes it simpler to deal with as we don't have to read it
>  concurrently along with everything else.
>
> - We now issue a warning if we find unexpected statements in the
>  bootloader results.
>
> - The arrangements for buffering of bootloader input and output
>  are completely changed.  Now we have a fixed limit of 64k
>  on output, and 4k on input, and discard the oldest data when
>  this overflows (which it shouldn't).  There is no timeout
>  any more.
>
> - This is implemented using a new asynchronous internal data-
>  copying facility.
>
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

I'm getting the following error when trying to launch DomUs using this patch:

Parsing config file debian.cfg
libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:137:disk_try_backend: Disk vdev=xvda,
backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:144:disk_try_backend: Disk vdev=xvda,
backend tap unsuitable because blktap not available
libxl: debug: libxl_device.c:219:libxl__device_disk_set_backend: Disk
vdev=xvda, using backend qdisk
libxl: debug: libxl_device.c:183:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=qdisk
libxl: debug: libxl.c:1732:libxl_device_disk_local_attach: locally
attaching qdisk /root/vm/disk.img

libxl: error: libxl_utils.c:564:libxl__recvmsg_fds: recvmsg failed
(ptys): Bad file descriptor
libxl: error: libxl_exec.c:124:libxl_report_child_exitstatus: openpty
child [1598] died due to fatal signal Broken pipe
libxl: error: libxl_create.c:570:do_domain_create: failed to run bootloader: -3
xc: debug: hypercall buffer: total allocations:21 total releases:21
xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
xc: debug: hypercall buffer: cache current size:2
xc: debug: hypercall buffer: cache hits:18 misses:2 toobig:1

I haven't tackled down the issue yet, but bisection says it is caused
by this patch.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader

[Ian Jackson]

Roger Pau Monné writes ("Re: [Xen-devel] [PATCH 20/20] libxl: ao: Convert libxl_run_bootloader"):
> I'm getting the following error when trying to launch DomUs using this patch:

Thanks for the report.  You're very brave :-).  I hadn't executed it
myself.

> libxl: error: libxl_utils.c:564:libxl__recvmsg_fds: recvmsg failed
> (ptys): Bad file descriptor
> libxl: error: libxl_exec.c:124:libxl_report_child_exitstatus: openpty
> child [1598] died due to fatal signal Broken pipe
> libxl: error: libxl_create.c:570:do_domain_create: failed to run bootloader: -3
> xc: debug: hypercall buffer: total allocations:21 total releases:21
> xc: debug: hypercall buffer: current allocations:0 maximum allocations:2
> xc: debug: hypercall buffer: cache current size:2
> xc: debug: hypercall buffer: cache hits:18 misses:2 toobig:1
> 
> I haven't tackled down the issue yet, but bisection says it is caused
> by this patch.

I'm sure you have better things to do than debug up my untested
patches...

Ian.