All of lore.kernel.org
 help / color / mirror / Atom feed
* A small issue with multiple xs_daemon_open
@ 2010-06-08 10:15 Rafal Wojtczuk
  2010-06-08 10:41 ` Ian Jackson
  0 siblings, 1 reply; 3+ messages in thread
From: Rafal Wojtczuk @ 2010-06-08 10:15 UTC (permalink / raw)
  To: xen-devel

Hello,
I am experiencing a problem when opening two simultaneous connections (in a
single process) to xenstored via xs_daemon_open(). I would think such
an operation is supported and should work (no static variables in
libxenstore are involved); is it a correct assumption ? If so, the issue 
should be fixed.

The environment is xen-3.4.3, 2.6.32.9-xxx pvops dom0. The order of
operations is
xs1=xs_daemon_open()
xs_watch(xs1,...)
xs_read_watch(xs1,...)
xs2=xs_daemon_open()
xs_read(xs2,...)
xs_daemon_close(xs2)
xs_read(xs1,...)
xs_daemon_close(xs1)

There seems to be some sort of a race when calling the second 
xs_daemon_close() - sometimes it hangs infinitely. Backtrace:
(gdb) info threads
  2 Thread 0x7fe64697c710 (LWP 4154)  0x00007fe646b8dc44 in __lll_lock_wait()
   from /lib64/libpthread.so.0
* 1 Thread 0x7fe6482d2700 (LWP 4153)  0x00007fe646b87bfd in pthread_join ()
   from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007fe646b87bfd in pthread_join () from /lib64/libpthread.so.0
#1  0x00007fe647776fdc in xs_daemon_close () from /usr/lib64/libxenstore.so.3.0
#2  0x000000000040660a in peer_client_init ()
#3  0x0000000000405808 in main ()
(gdb) thread 2 
[Switching to thread 2 (Thread 0x7fe64697c710 (LWP 4154))]#0
0x00007fe646b8dc44 in __lll_lock_wait ()
   from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007fe646b8dc44 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fe646b88f15 in _L_lock_1017 () from /lib64/libpthread.so.0
#2  0x00007fe646b88de7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fe647775db8 in ?? () from /usr/lib64/libxenstore.so.3.0
#4  0x00007fe647775e3c in ?? () from /usr/lib64/libxenstore.so.3.0
#5  0x00007fe646b86a3a in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe64729967d in clone () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()
(gdb) 

You can see the code for peer_client_init() at
http://qubes-os.org/gitweb/?p=rafal/gui.git;a=blob;f=gui-common/txrx-vchan.c;h=afc691e922edce61683e46203fd9597d029052d3;hb=93f0509b791f8fc433675c58c81f6ea9469d2c91#l168
it calls libvchan_client_init, code at
http://qubes-os.org/gitweb/?p=rafal/gui.git;a=blob;f=vchan/vchan/init.c;h=4a3da4f22907445160a6fe3db9ed357f89ebc301;hb=93f0509b791f8fc433675c58c81f6ea9469d2c91#l212
note that libvchan_client_init calls xs_daemon_open/close by itself, too.

Naturally it is posible to change peer_client_init() to not call 
xs_daemon_open twice, but the title issue itself remains.

Regards,
Rafal Wojtczuk
Principal Researcher
Invisible Things Lab, Qubes-os project

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: A small issue with multiple xs_daemon_open
  2010-06-08 10:15 A small issue with multiple xs_daemon_open Rafal Wojtczuk
@ 2010-06-08 10:41 ` Ian Jackson
  2010-06-08 11:01   ` Keir Fraser
  0 siblings, 1 reply; 3+ messages in thread
From: Ian Jackson @ 2010-06-08 10:41 UTC (permalink / raw)
  To: Rafal Wojtczuk; +Cc: xen-devel

Rafal Wojtczuk writes ("[Xen-devel] A small issue with multiple xs_daemon_open"):
> I am experiencing a problem when opening two simultaneous connections (in a
> single process) to xenstored via xs_daemon_open(). I would think such
> an operation is supported and should work (no static variables in
> libxenstore are involved); is it a correct assumption ? If so, the issue 
> should be fixed.

This is definitely supposed to work.  But as you say there are no
static variables in libxenstore (even in 3.4.3).  However, some
threading bugs were fixed recently in xen-unstable changesets 21353,
21354 and 21374.  Do these look like they might be the cause ?

Ian.

changeset:   21374:9d53864d7be6
user:        Keir Fraser <keir.fraser@citrix.com>
date:        Thu May 13 12:21:16 2010 +0100
files:       tools/xenstore/xs.c
description:
xenstore: Fix cleanup_pop() definition for some (buggy) pthread.h headers.

Signed-off-by: Keir Fraser <keir.fraser@citrix.com>


changeset:   21354:9de69d816b11
user:        Keir Fraser <keir.fraser@citrix.com>
date:        Wed May 12 08:49:13 2010 +0100
files:       tools/xenstore/xs.c
description:
xs: avoid pthread_join deadlock in xs_daemon_close

Doing a pthread_cancel and join on the reader thread while holding all
the request/reply/watch mutexes can deadlock if the thread needs to
take any of those mutexes to exit.  Kill off the reader thread before
taking any mutexes (which should be redundant if we're
single-threaded at that point).

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>


changeset:   21353:2dd3141b3e3e
user:        Keir Fraser <keir.fraser@citrix.com>
date:        Wed May 12 08:48:14 2010 +0100
files:       tools/xenstore/xs.c
description:
xs: make sure mutexes are cleaned up and memory freed if the read thread is cancelled

If the read thread is terminated with pthread cancel, it must make
sure all memory is freed and mutexes are unlocked.

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: A small issue with multiple xs_daemon_open
  2010-06-08 10:41 ` Ian Jackson
@ 2010-06-08 11:01   ` Keir Fraser
  0 siblings, 0 replies; 3+ messages in thread
From: Keir Fraser @ 2010-06-08 11:01 UTC (permalink / raw)
  To: Ian Jackson, Rafal Wojtczuk; +Cc: xen-devel@lists.xensource.com

On 08/06/2010 11:41, "Ian Jackson" <Ian.Jackson@eu.citrix.com> wrote:

> Rafal Wojtczuk writes ("[Xen-devel] A small issue with multiple
> xs_daemon_open"):
>> I am experiencing a problem when opening two simultaneous connections (in a
>> single process) to xenstored via xs_daemon_open(). I would think such
>> an operation is supported and should work (no static variables in
>> libxenstore are involved); is it a correct assumption ? If so, the issue
>> should be fixed.
> 
> This is definitely supposed to work.  But as you say there are no
> static variables in libxenstore (even in 3.4.3).  However, some
> threading bugs were fixed recently in xen-unstable changesets 21353,
> 21354 and 21374.  Do these look like they might be the cause ?

Yes, I backported these to 4.0.x but not to 3.4.x since noone had reported
problems. I've now backported the fixes as xen-3.4-testing:19982. They'll go
into Xen 3.4.4.

 -- Keir

> Ian.
> 
> changeset:   21374:9d53864d7be6
> user:        Keir Fraser <keir.fraser@citrix.com>
> date:        Thu May 13 12:21:16 2010 +0100
> files:       tools/xenstore/xs.c
> description:
> xenstore: Fix cleanup_pop() definition for some (buggy) pthread.h headers.
> 
> Signed-off-by: Keir Fraser <keir.fraser@citrix.com>
> 
> 
> changeset:   21354:9de69d816b11
> user:        Keir Fraser <keir.fraser@citrix.com>
> date:        Wed May 12 08:49:13 2010 +0100
> files:       tools/xenstore/xs.c
> description:
> xs: avoid pthread_join deadlock in xs_daemon_close
> 
> Doing a pthread_cancel and join on the reader thread while holding all
> the request/reply/watch mutexes can deadlock if the thread needs to
> take any of those mutexes to exit.  Kill off the reader thread before
> taking any mutexes (which should be redundant if we're
> single-threaded at that point).
> 
> Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> 
> 
> changeset:   21353:2dd3141b3e3e
> user:        Keir Fraser <keir.fraser@citrix.com>
> date:        Wed May 12 08:48:14 2010 +0100
> files:       tools/xenstore/xs.c
> description:
> xs: make sure mutexes are cleaned up and memory freed if the read thread is
> cancelled
> 
> If the read thread is terminated with pthread cancel, it must make
> sure all memory is freed and mutexes are unlocked.
> 
> Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2010-06-08 11:01 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-06-08 10:15 A small issue with multiple xs_daemon_open Rafal Wojtczuk
2010-06-08 10:41 ` Ian Jackson
2010-06-08 11:01   ` Keir Fraser

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.