Re: Driver domains and hotplug scripts, redux

["Roger Pau Monné" <roger.pau@entel.upc.edu>]

2012/1/5 Ian Jackson <Ian.Jackson@eu.citrix.com>:
> Roger Pau Monné writes ("Re: [Xen-devel] RFC: Still TODO for 4.2?"):
>> I don't know much about driver domains, but from what I've read they
>> should be running something like NetBSD xenbackend and listen for
>> xenstore events. Most of the functions that I've written on my hotplug
>> series can be used to create a little daemon, that's not the problem,
>> the problem is what can we use to synchronize hotplug script calling
>> and libxl (what comes to mind is using a dedicated xenstore variable
>> for each device, but someone might have a better idea).

Sorry I didn't reply earlier, I was still on holidays (mostly working
with the autotools stuff).

> This envisages devicer setup/teardown scripts in driver domains
> running in a different way to those in the same domain as the
> toolstack.  Are we sure this is a good idea ?

No, I think it's best that even is the driver domain is Dom0 the same
procedure should be executed, and xenbackendd should be running in
every driver domain, Dom0 included, toolstack should never execute
hotplug scripts directly.

> I think it would be preferable to have only one interface to device
> scripts, which is used everywhere.  That interface would have to
> involve initiation by the toolstack, and collection of resulting
> success/failure/etc., via xenstore.

Are we only going to use xenstore to share information between both
domains (Dom0 <--> Driver domain)?

I'm going to comment the vif case, but I think both vif and block
devices should follow the same approach, and hotplug script execution
has to be something "standard" and should not rely on the type of the
device.

> The sequence of events for vifs with a kernel-level backend needs
> to go like this:
>  * toolstack tells backend domain to create vif, via xenstore

How does the toolstack tell a domain to create a device? Creating a
xenstore entry like:

/local/domain/<domid>/backend/vif/...

does trigger the creation of a vif interface in the <domid> domain?

>  * backend kernel creates a virtual network interface vifNN
>  * something in backend domain notices that this vifNN
>    has appeared and consequently

This should be handled by xenbackendd (I know it will not exactly be
xenbackendd, but let's call it that way to simplify things), since it
should be listening to /local/domain/<domid>/backend/* for changes and
react upon them.

>  * device setup script runs, enslaves vifNN to bridge, adds
>    it to routing tables, gives it an address, etc.

Handled by hotplug scripts.

>  * something in backend domain domain tells toolstack vif is ready

Hotplug scripts should change backend state (and write the appropriate
values) to notify everything when ok. Since xenbackendd is the one
that executes the scripts, it should examine the exit code of the
called hotplug script and write the exit status code and message if
hotplug script execution is not successful. This values can be
retrieved from the toolstack and notify the user if something failed.

>  [ device is used ]
>  * toolstack tells backend domain to destroy vif; perhaps entire
>    xenstore directory is forcibly removed??

If entire xenstore directory is forcibly removed, how does xenbackendd
know the parameters to pass to the hotplug script to shutdown the
device? Do we have to keep a copy of this somewhere else (xenstore or
create a xenbackendd private database)?

Here we have two cases, whether it is a shutdown or a destroy:

When doing a shutdown the toolstack should wait to get a notification
from the driver domain that hotplug execution was done (either
successfully or not) and then proceed with the removal of xenstore
directory.

DomU closes device --> driver domain notices --> execution of hotplug
scripts --> write result to xenstore --> toolstack reads results of
hotplug teardown.

When doing a destroy, the toolstack should manually set the frontend
state to closed, and thus force the execution of hotplug scripts in
the driver domain? I know this has been a cause of discussion in
previous patches, but I really don't see the problem with modifying
the frontend status if the domain is already dead, it's just a way to
force the unplug of the device and the execution of hotplug scripts.
Normally the DomU should set the frontend status to closed, but since
we killed it from the toolstack, it should be the toolstack itself the
one in charge of setting the status to closed.

toolstack kills domain --> toolstack sets frontend status to closed
--> driver domain kernel notices frontend change and closes backend
--> xenbackendd noticies change --> execution of hotplug scripts -->
write results to xenstore --> toolstack reads results of hotplug
teardown.

>  * backend kernel removes virtual network interface immediately
>    and all routes, bridge enslavements, etc., are undone
>  * something in backend notices the removal
>  * device teardown script may need to remove eg firewall rules
>  * when this is complete, the backend domain notifies the
>    toolstack (how??)

Should the toolstack wait for a notification from the driver domain? I
think it is important that the toolstack is always aware of what
happens in the driver domain, and it should wait for the execution of
the teardown hotplug scripts and catch it's results, to notify the
user if it is not successful.

> For block devices with a kernel-level backend:
>  * toolstack tells backend domain to create vbd
>    parameters include: vbd number, target??, script??
>  * something in backend domain notices this and consequently
>  * device setup script runs, creates a suitable actual
>    block device in backend domain
>  * backend kernel picks up actual block device details and
>    becomes available to guest
>  * something in backend domain tells the toolstack all is well
>  [ device is used ]
>  * toolstack tells backend domain to destroy vbd; perhaps entire
>    xenstore directory is forcibly removed??
>  * backend kernel removes its actual backend and closes the
>    block device, and somehow notifies userspace when this
>    is done so that
>  * device teardown script cleans up, including making actual
>    block device go away (if it was one which the setup script
>    created)
>  * when this is complete, the backend domain notifies the
>    toolstack (how??)
>
> For block devices with a user-level backend:
>  * toolstack tells backend domain to create vbd
>    parameters include: vbd number, target??, script??
>  * userland backend notices this, does its housekeeping
>    and setup, and tells the toolstack all is well
>  [ device is used ]
>  * toolstack tells backend domain to destroy vbd; perhaps entire
>    xenstore directory is forcibly removed??
>  * userland backend removes its actual backend and closes the
>    resources it was using, and
>  * notifies the toolstack (how??)

When it comes to block hotplug scripts, we have to let xenbackendd
decide which kind of backend to use, so we should agree on what to
write to xenstore that can cover all types of block backends (phy,
qdisk, blktap...), since the toolstack probably doesn't have access to
the requested medium.

> Much of this seems to be covered by, or coverable by, the existing
> xenstore protocol.  I think we just need to define in more detail
> exactly how it should all work, and on each platform how the
> "something"s work.
>
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel