Shutdown problems in xs.c

[Jeremy Fitzhardinge <jeremy@goop.org>]

I've been getting deadlocks in xl, particularly "xl destroy".  It turns
out the main thread is stuck in a pthread_join while holding all the
mutexes, while the xenstore reading thread is stuck in a
pthread_mutex_lock before it can get to a cancellation point and exit.

This looks like it is a very long-standing deadlock (the code in
question mostly dates back to 2005), but perhaps something has changed
that makes it more likely to happen.  I think the original intention of
the code was to hold all the mutexes while doing the cancel/join to
avoid cancelling while the reader is holding any mutexes.  This fails
when the reader loop is not holding any, but needs to take one before
getting to a cancellation point (pthread_mutex_lock is not itself a
cancellation point).

The following two patches address it by 1) making sure that the read
thread has sufficient pthread cleanup handlers to free any
allocated-but-unused memory and release the mutexes when cancelled, and
2) do the pthread cancel/join while not holding any mutexes.

Thanks,
    J