Re: [Xen-devel] bug in xenstored? No notification to subscription on @introduceDomain

[George Shuklin <george.shuklin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

On 12.12.2011 17:46, Ian Campbell wrote:
> Please don't top post and don't drop people/lists from the CC. I have
> reinstated xen-devel and refrained from trimming the quotes as heavily
> as I normally would.
>
> Counter to my own advice I have also dropped xen-hosts-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org
> because last time I got a bounce in Russian to the effect that the group
> does not exist (according to google translate).
>
> On Mon, 2011-12-12 at 12:10 +0000, George Shuklin wrote:
>> Thanks for reply.
>>
>> The problem is we tried at least two different libraries - xs (+python
>> xen.lowlevel.xs) and our own library (pyxs), created from scratches on
>> pure python - both shows exactly same behavior. We loosing same time
>> @introduce and @release, but only for new domains. Older domains (which
>> starts before error appear) during shutdown/migration sends  @release
>> normally.
>>
>> I done strace, nothig is sending by xenstored to application socket when
>> 'new' domains appears and disappears (I'm not sure 100% due not very
>> good strace skills).
>>
>> Application performs write/read operations to/from xenstore (and do many
>> subscriptions, but only after @introduce) and older subscription works fine.
>>
>> PS We got other strange bug with memory leak in xenstored (happens only
>> with big amount of transactions, and ONLY with socket) - but this case
>> is still under research, so I decide not to post this (but may be it
>> related somehow?).
> Are the two event correlated? i.e. is the oxenstored process huge when
> these failures occur? Inability to allocate memory could explain some of
> your symptoms although I'd expect it to be more fatal more quickly and
> obviously than what you describe or to have wider impact.
Nope, memory leak occur only if transaction happens with subscription, 
but 'no notification' problem continues after we stops to use 
transaction (this cure memory leak completely, so I think this is 
separate issue, but I don't sure).

I still can't catch condition for lack of notifications for @introduce, 
sorry (I got one more this morning in test pool).

>> Sorry for question - how I can gather debug information for oxenstored?
> What sort of debug information are you after?
>
> There are various logging options which you could turn up to 11
> in /etc/xensource/xenstored.conf but I do not have a complete list of
> what they are, similarly for command line options -- perhaps someone on
> xen-api@ could chime in? Otherwise looking in the source might be the
> best way to find out what they are, try xenstore.ml, parse_args.ml
> logging.ml would be good places to start. (if having done so you feel
> motivated to write a patch to add docs/man/oxenstored.1.pod we would be
> much obliged...)
>
Ok, thanks, I'll dig to sources to set up them all. We heavily using 
xenstore for dynamic memory regulation (about five operations for every 
domain per second).