PV console bug in 4.0 / unstable

PV console bug in 4.0 / unstable

[Jim Fehlig]

I'm chasing a console bug in Xen 4.0 that only appears with managed PV
domains - and even then only after an initial start-->stop sequence.

Legacy PV domain config files generally do not specify a console device
explicitly but instead get one via qemu with

vfb=['type=vnc,vncunused=1']

When starting a PV domain, xend will create a console device if one is
not specified in the domain's config.  Details of the device are added
to the domain's config and, if the domain is managed, written to xend's
internal "db".  All of this occurs after the console details are written
to xenstore, so life is good for qemu backend reading xenstore.  If the
managed domain is subsequently stopped and started again, the console
device, now included in the domain's device refs, is created before
console info is written to xenstore.  The console backend in qemu fails
to read xenstore, causing con_init() to fail.  Meanwhile the frontend in
domU is waiting for backend and booting does not progress.  In
pseudo-operations

create_domain
for each device in device_refs: # no console device in device_refs
  create_device
write_device_info_to_xenstore
update_consoles
  # Console device found in xenstore.  If it does not exist in
  # domain device refs, add it and call DevController.createDevice
..
shutdown_domain
create_domain
for each device in device_refs: # now contains console device
  create_device # device created before info written to xenstore
write_device_info_to_xenstore

As a workaround, I've patched con_init() in
tools/ioemu-dir/hw/xen_console.c to retry xenstore read.  I suspect xend
could be patched to write required console details to xenstore before
creating the console device.  But the current behavior in xend has not
changed since Xen 3.3, the last version I found that doesn't exhibit
this issue.  The console code in qemu did change significantly between
3.3 and 3.4 however.

Any comments regarding a fix in xend vs qemu?  Perhaps alternate
solutions to this race?

Also noteworthy is that the issue does not appear when using 'ol
xenconsoled, i.e. removing vfb entry in domain config.

Regards,
Jim