Re: xenstored crashes with SIGSEGV

[Ian Campbell <Ian.Campbell@citrix.com>]

On Fri, 2014-12-12 at 17:45 +0100, Philipp Hahn wrote:
> Hello Ian,
> 
> On 12.12.2014 17:32, Ian Campbell wrote:
> > On Fri, 2014-12-12 at 17:14 +0100, Philipp Hahn wrote:
> >> We did enable tracing and now have the xenstored-trace.log of one crash:
> >> It contains 1.6 billion lines and is 83 GiB.
> >> It just shows xenstored to crash on TRANSACTION_START.
> >>
> >> Is there some tool to feed that trace back into a newly launched xenstored?
> > 
> > Not that I know of I'm afraid.
> 
> Okay, then I have to continue with my own tool.

If you do end up developing a tool to replay a xenstore trace then I
think that'd be something great to have in tree!

> > Do you get a core dump when this happens? You might need to fiddle with
> > ulimits (some distros disable by default). IIRC there is also some /proc
> > nob which controls where core dumps go on the filesystem.
> 
> Not for that specific trace: We first enabled generating core files, but
> only then discovered that this is not enough.

How wasn't it enough? You mean you couldn't use gdb to extract a
backtrace from the core file? Or was something else wrong?

>  Then we enabled
> --trace-file, but on that host something reseted generating the core file.
> We hopefully fixed all hosts so on the next crash we hopefully will get
> both a core file and the trace.

Great.

> >> My hope would be that xenstored crashes again, because then we could use
> >> all those other tools like valgrind more easily.
> > 
> > That would be handy. My fear would be that this bug is likely to be a
> > race condition of some sort, and the granularity/accuracy of the
> > playback would possibly need to be quite high to trigger the issue.
> 
> cxenstored looks single threaded to me, or am I wrong?

Nope, you are right, my mistake.

Ian.