bug in xenstored? No notification to subscription on @introduceDomain

[George Shuklin <george.shuklin@gmail.com>]

Good day.

I think I met some strange bug in xenstored.

I using XCP for long time and all that time we have some funny bug we 
was not able to debug enough due product environment and very low chance 
to appear, now we was able to catch it in testing environment and done 
some research.

We have python application running in dom0 and waiting domain 
appearance. This implemented this via subscription to @introduceDomain 
xenstore key. Under some conditions we stops to receive notification on 
subscription. If we ran application as second instance it will receive 
that notification, if we restart application it will  receive too.

I unable to pinpoint exact condition for this, but this
a) Happens occasionally but consistently (about once a month in farm of 
50 hosts at least at one host)
b) Not related to xenstored uptime
c) Not related to load on xen or dom0
d) Not related to amount of domains
e) Occur at least at XCP 0.5, 1.0 and 1.1 (I don't know how to get 
version from xenstored)

Last time I got that on two hosts in lab at same time (with single guest 
domain without any high load) and done some experiments - so I can say 
exactly I wrote above.

The pieces from python code we ran:

from xen.lowlevel.xs import xs
conn = xs.xs()
conn.watch("@introduceDomain", "+")
conn.watch("@releaseDomain", "-")
conn.read_watch()