pthread_mutex_lock() and Xenstored

pthread_mutex_lock() and Xenstored

[Peter Teoh]

[-- Attachment #1.1: Type: text/plain, Size: 514 bytes --]

pthread_mutex_lock() are not async-signal safe (ref:
http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf)
but I still see that it is used extensively in xenstored implementation (eg, xs.c).

Moreover, pthread_mutex_lock() suffered a higher performance penalty than other synchronization option.   For a one-time effort like domain creation this is ok, but Xenstore is used repeatedly to access data, and therefore performance could potentially be enhanced.

Does all these sound logical?


[-- Attachment #1.2: Type: text/html, Size: 1382 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pthread_mutex_lock() and Xenstored

[Vincent Hanquez]

On Fri, Aug 31, 2007 at 01:51:18PM +0800, Peter Teoh wrote:
> pthread_mutex_lock() are not async-signal safe (ref:
> http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf)
> but I still see that it is used extensively in xenstored
> implementation (eg, xs.c).

xs is the client part of the xenstore protocol. it's not used in
xenstored (the daemon).

> Moreover, pthread_mutex_lock() suffered a higher performance penalty
> than other synchronization option.   For a one-time effort like domain
> creation this is ok, but Xenstore is used repeatedly to access data,
> and therefore performance could potentially be enhanced.
>
> Does all these sound logical?

Not really. xenstored is single threaded and has a file-backed-storage
anyway, so most of the time you're going to wait for that.

I expect the pthread_mutex_lock to be insignificant compared to the time
waiting for xenstored to reply. Also the xs interface is not meant to be
use massively in parallel.

Cheers,
-- 
Vincent Hanquez

Re: pthread_mutex_lock() and Xenstored

[Daniel P. Berrange]

On Fri, Aug 31, 2007 at 01:51:18PM +0800, Peter Teoh wrote:
> pthread_mutex_lock() are not async-signal safe (ref:
> http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf)
> but I still see that it is used extensively in xenstored implementation
> (eg, xs.c).

So what ?  The pthread_mutex_lock calls aren't used inside any signal
handlers, so the question of whether its async-signal safe is irrelevant

> Moreover, pthread_mutex_lock() suffered a higher performance penalty 
> than other synchronization option.   For a one-time effort like domain
> creation this is ok, but Xenstore is used repeatedly to access data, 
> and therefore performance could potentially be enhanced.

That presentation giving figures comparing pthread_mutex_lock with other
primitives is useless. It doesn't say what OS / version it was tested on,
or what those figures are measuring. Linux NPTL (2.6.x) threading has radically
different (better) pthread_mutex performance characteristics that the older
LinuxThreads impl (2.4.x) for example, and that's ignoring any other non-Linux
OS' impl of pthreads. The performance characteristics of a contended mutex lock,
vs an uncontended lock are also completely different & not considered in those
figures. 

Finally there's no performance profile data to suggest that mutex locking
is even remotely relevant to xenstore performance. There is however data to
show that the huge amount of I/O xenstored does hurts performance.

By all means consider how to improve the performance of xenstored, but do
so based on *measured* facts about where the current problems are, rather
than bullet points in a presentation unrelated to Xen.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: pthread_mutex_lock() and Xenstored

[Peter Teoh]

Thank you for your feedback.   I am still learning these stuff.

On 9/13/07, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Fri, Aug 31, 2007 at 01:51:18PM +0800, Peter Teoh wrote:
> > pthread_mutex_lock() are not async-signal safe (ref:
> > http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf)
> > but I still see that it is used extensively in xenstored implementation
> > (eg, xs.c).
>
> So what ?  The pthread_mutex_lock calls aren't used inside any signal
> handlers, so the question of whether its async-signal safe is irrelevant
>
> > Moreover, pthread_mutex_lock() suffered a higher performance penalty
> > than other synchronization option.   For a one-time effort like domain
> > creation this is ok, but Xenstore is used repeatedly to access data,
> > and therefore performance could potentially be enhanced.
>
> That presentation giving figures comparing pthread_mutex_lock with other
> primitives is useless. It doesn't say what OS / version it was tested on,
> or what those figures are measuring. Linux NPTL (2.6.x) threading has radically
> different (better) pthread_mutex performance characteristics that the older
> LinuxThreads impl (2.4.x) for example, and that's ignoring any other non-Linux
> OS' impl of pthreads. The performance characteristics of a contended mutex lock,
> vs an uncontended lock are also completely different & not considered in those
> figures.
>

Contended locks?   I will looked further into these, thanks.

> Finally there's no performance profile data to suggest that mutex locking
> is even remotely relevant to xenstore performance. There is however data to
> show that the huge amount of I/O xenstored does hurts performance.
>

And so, is there any particular reasons for Xenstored's slow
performance?   Any area needs improvement?

I looked into Xenstored's implementation, and found that it is using
TDB (trivial database, which is also used by SAMBA) as its backbone.
And inside the tdb.c are a lot of locking statements.   What is TDB?
Then I read this article:

http://www.coda.cs.cmu.edu/maillists/codadev/codadev-2003/0202.html

(paying attention to what he said about Dan Bernstein's CDB) and to
refer to what you said - Xenstored is single threaded, so we have a
"single reader/single writer situation" here, and so is a database
structure like tdb really needed?

What I think is we need to redesign these stuff, based on:

    * identifying the characteristic the relative frequency of
read/write access.
    * identifying concurrent readers, and writer - its different model
    * does databases need to be shareable across heterogeneous systems
by a simple file copy across the network like SAMBA?
    * Can't use thread libraries as they conflict with LWP.
    * identifying the minimal amount of mapping operation needed, and
possibly its corresponding hashing structures.
    * and finally, to come up with the simplest design that meet all
the requirements.

Can Dan's CDB be a possible alternative candidate for our database
design?   Please enlighten me :-).

> By all means consider how to improve the performance of xenstored, but do
> so based on *measured* facts about where the current problems are, rather
> than bullet points in a presentation unrelated to Xen.
>

Noted, thanks :-).

Re: pthread_mutex_lock() and Xenstored

[Bastian Blank]

On Sat, Sep 15, 2007 at 02:01:12PM +0800, Peter Teoh wrote:
>     * does databases need to be shareable across heterogeneous systems
> by a simple file copy across the network like SAMBA?

Most db files are not interchangable. And the xenstore db is usually
useless on another system. You want to define your own transfer format
if you want to do this and not rely on the db to support that.

>     * Can't use thread libraries as they conflict with LWP.

"Library for WWW in Perl"? Even wikipedia does not list a better match.
pthread is LSB and SUSv3. And with a proper db you can use fork if
necessary.

>     * identifying the minimal amount of mapping operation needed, and
> possibly its corresponding hashing structures.
>     * and finally, to come up with the simplest design that meet all
> the requirements.
> 
> Can Dan's CDB be a possible alternative candidate for our database
> design?   Please enlighten me :-).

Isn't CDB also read-only, aka needs to be rewritten on each write?
xenstored with tdb does this.

Usually I would say, bdb is a good choice for this usage. Supports
transactions, multiple reader. It is used by many things with large
writing frequency.

Bastian

-- 
No one wants war.
		-- Kirk, "Errand of Mercy", stardate 3201.7

Re: pthread_mutex_lock() and Xenstored

[Peter Teoh]

On 9/15/07, Bastian Blank <bastian@waldi.eu.org> wrote:
> On Sat, Sep 15, 2007 at 02:01:12PM +0800, Peter Teoh wrote:
> >     * does databases need to be shareable across heterogeneous systems
> > by a simple file copy across the network like SAMBA?
>
> Most db files are not interchangable. And the xenstore db is usually
> useless on another system. You want to define your own transfer format
> if you want to do this and not rely on the db to support that.
>
> >     * Can't use thread libraries as they conflict with LWP.
>
> "Library for WWW in Perl"? Even wikipedia does not list a better match.
> pthread is LSB and SUSv3. And with a proper db you can use fork if
> necessary.
>
> >     * identifying the minimal amount of mapping operation needed, and
> > possibly its corresponding hashing structures.
> >     * and finally, to come up with the simplest design that meet all
> > the requirements.
> >
> > Can Dan's CDB be a possible alternative candidate for our database
> > design?   Please enlighten me :-).
>
> Isn't CDB also read-only, aka needs to be rewritten on each write?
> xenstored with tdb does this.
>

Have not the time to look into CDB yet.   More important is to
understand in depth what actually is the requirements of Xenstored -
its basic need for a transaction concept?   ACID properties?   FIFO
queue properties?   Can we consider other alternatives like lock-free
database?   For example:

http://en.wikipedia.org/wiki/Lock-free_and_wait-free_algorithms
http://www.grame.fr/pub/LockFree.pdf

And then there is a concurrency vs performance tradeoffs issue as well
(eg, row-level locking in database usually implies higher overheads in
processing).   So it will be good if lock-free algo are favored.

Another possibility is also batching of transactions, just like
batching of hypercall in Xen, so as to cut down the overheads of
starting and closing the transaction.

> Usually I would say, bdb is a good choice for this usage. Supports
> transactions, multiple reader. It is used by many things with large
> writing frequency.
>

Yes, I agree, bDB is used in MySQL etc.

On 9/15/07, Daniel P. Berrange <berrange@redhat.com> wrote:

> I repeated those tests against Xen 3.1 a few months back and the characteristics
> haven't changed all that much. Running further OProfile tests on a debug build
> would confirm this.
>

Any possibility of accessing these data?

> Since it is not possible to ever actually restart xenstored, the easy answer
> is to not use any database on disk at all. Simply keep the entire information
> set in RAM. None of the data is keeps per VM needs to persist once the VM
> shuts down - all the important state is maintained by XenD which does persist.
>

Persistency is traded off here.   I thought if traditional filesystem
can do without the database concept (eg, rollback concept when
non-integrity of data is detected), why not here?

Any comments?   Apologized if I am wrong in any area.

Re: pthread_mutex_lock() and Xenstored

[Daniel P. Berrange]

On Sat, Sep 15, 2007 at 02:01:12PM +0800, Peter Teoh wrote:
> Thank you for your feedback.   I am still learning these stuff.
> 
> On 9/13/07, Daniel P. Berrange <berrange@redhat.com> wrote:
> > On Fri, Aug 31, 2007 at 01:51:18PM +0800, Peter Teoh wrote:
> > > pthread_mutex_lock() are not async-signal safe (ref:
> > > http://www.gelato.org/pdf/Illinois/gelato_IL2004_libatomic_boehm.pdf)
> > > but I still see that it is used extensively in xenstored implementation
> > > (eg, xs.c).
> >
> > So what ?  The pthread_mutex_lock calls aren't used inside any signal
> > handlers, so the question of whether its async-signal safe is irrelevant
> >
> > > Moreover, pthread_mutex_lock() suffered a higher performance penalty
> > > than other synchronization option.   For a one-time effort like domain
> > > creation this is ok, but Xenstore is used repeatedly to access data,
> > > and therefore performance could potentially be enhanced.
> >
> > That presentation giving figures comparing pthread_mutex_lock with other
> > primitives is useless. It doesn't say what OS / version it was tested on,
> > or what those figures are measuring. Linux NPTL (2.6.x) threading has radically
> > different (better) pthread_mutex performance characteristics that the older
> > LinuxThreads impl (2.4.x) for example, and that's ignoring any other non-Linux
> > OS' impl of pthreads. The performance characteristics of a contended mutex lock,
> > vs an uncontended lock are also completely different & not considered in those
> > figures.
> >
> 
> Contended locks?   I will looked further into these, thanks.
> 
> > Finally there's no performance profile data to suggest that mutex locking
> > is even remotely relevant to xenstore performance. There is however data to
> > show that the huge amount of I/O xenstored does hurts performance.
> >
> 
> And so, is there any particular reasons for Xenstored's slow
> performance?   Any area needs improvement?

The primary problem is the amount of I/O it does per transaction, and then
indirectly, the huge number of transactions XenD makes. 

http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00487.html

I repeated those tests against Xen 3.1 a few months back and the characteristics
haven't changed all that much. Running further OProfile tests on a debug build
would confirm this.

> (paying attention to what he said about Dan Bernstein's CDB) and to
> refer to what you said - Xenstored is single threaded, so we have a
> "single reader/single writer situation" here, and so is a database
> structure like tdb really needed?
> 
> What I think is we need to redesign these stuff, based on:

Since it is not possible to ever actually restart xenstored, the easy answer
is to not use any database on disk at all. Simply keep the entire information
set in RAM. None of the data is keeps per VM needs to persist once the VM
shuts down - all the important state is maintained by XenD which does persist.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: pthread_mutex_lock() and Xenstored

[John Levon]

On Sat, Sep 15, 2007 at 02:50:03PM +0100, Daniel P. Berrange wrote:

> Since it is not possible to ever actually restart xenstored, the easy answer
> is to not use any database on disk at all. Simply keep the entire information
> set in RAM. None of the data is keeps per VM needs to persist once the VM
> shuts down - all the important state is maintained by XenD which does persist.

This has to be fixed at some point though. And part of fixing it will
involve persisting the watches (that's actually the easy bit...)

john