* Re: Is there example code and/or documentation for developing a Xen back-end/front-end driver pair?
2006-06-26 22:16 ` Randy Thelen
@ 2006-06-26 22:31 ` Ewan Mellor
2006-06-26 22:45 ` Randy Thelen
0 siblings, 1 reply; 6+ messages in thread
From: Ewan Mellor @ 2006-06-26 22:31 UTC (permalink / raw)
To: Randy Thelen; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 2416 bytes --]
On Mon, Jun 26, 2006 at 03:16:08PM -0700, Randy Thelen wrote:
> Anthony Liguori wrote:
>
> >(although it's probably stable enough now to document)
>
> I'm getting that feeling.
>
> >In principle, once you have bring up working, making it work in
> >Xend isn't so bad. Do you have specific questions or are you just
> >looking for a where to begin?
>
> I see this "orange of a problem" and I'm trying to get my fingernail
> under the orange peel.
>
> It appears that in order to get the front-end to talk to the back-
> end, using the standard xenbus and interface type files I seen in
> blkback/ and blkfront/ driver source code directories, that I need to
> begin in Xend. So, I'm looking in Xend and I -think- I get most of
> the architecture relevant here. But, it's not clear to me, yet,
> exactly what Xend asks of the back end driver in order to get the
> system going.
>
> For example, when it's decided that a blk back end will be used,
> blkif.py is run and part of what appears to happen is that the dom0
> blk driver is kicked somehow (!?) and a Python Event is used (in
> DevController.py) with an xswatch() call to wait for the backed to
> startup. And, it seems that the trigger to observing the startup is
> complete is the back end putting some data into XenStore (into
> statusPath = backpath + '/' + HOTPLUG_STATUS_NODE).
Attached are some notes of mine. They are quite old, but I think it all is
pretty much relevant.
Basically, a backend watches /local/domain/0/backend/<deviceClass> and a
frontend watches /local/domain/<domID>/device/<deviceClass>. When the tools
write into those directories in the store, the drivers get a watch fired, and
check those directories to see the new device details. The new internal state
is created based on that.
If you are using the normal Xenbus code, then you can register your device
driver with the Xenbus layer, and much of this is done for you.
The drivers themselves use a simple state machine to handshake between front
and backends.
The backends use hotplug / udev events inside domain 0 to trigger scripts for
simple bringup/teardown stuff, like attaching new vifs to bridges or
whatever. The tools watch for a specific node to be written in the store to
indicate that this hotplug phase has succeeded (it's usually the one that does
the final sanity checking and often has to bail out).
HTH,
Ewan.
[-- Attachment #2: drivers.txt --]
[-- Type: text/plain, Size: 8622 bytes --]
Architecture for Split Drivers Within Xen
=========================================
The block, net and TPM Xen drivers are "split" drivers: they have a portion in
a privileged domain handling the physical device -- the backend -- and a
frontend in the unprivileged domain acting as a proxy upon the backend. The
backend and frontend communicate using shared event channels and ring buffers,
in an architecture known as the Xenbus.
For concreteness, this section shall discuss the block driver, with
differences between that and the net driver highlighted where necessary. The
block driver is referred to in various places by the shorthands "blk" or
"vbd". I shall also refer to the privileged domain as domain 0, and the
unprivileged domain as domain U.
The basic architecture can be considered to be a chain of four protagonists, a
pair of devices communicating across the Xenbus, and then a pair of devices
using that bus, handling the device-specific interaction with the kernel
device layer.
priv. unpriv.
domain Xenbus interdomain Xenbus domain
kernel -- blkback -- backend -------------- frontend -- blkfront -- kernel
device instance instance interconnect instance instance device
layer layer
In order to establish communication across this chain, a number of parameters
need to be passed from privileged to unprivileged domain, and vice versa.
Some of these parameters are specific to the blkback/blkfront pair, and some
are more general, applying to all split drivers. All of these parameters are
passed using the Xen Store.
Device Initialisation
---------------------
To trigger the creation of a device connection, Xend (or another tool) writes
frontend and backend details to the store. These new details are seen by the
Xenbus driver instances, and initialisation begins.
The details to be written are:
/local/domain/0/backend/vbd/U/<deviceID>/...
frontend /local/domain/U/device/vbd/<deviceID>
frontend-id U
state XenbusStateInitialising
... <device-specific details>
/local/domain/U/device/vbd/<deviceID>/...
backend /local/domain/0/backend/vbd/U/<deviceID>
backend-id 0
state XenbusStateInitialising
... <device-specific details>
The Xenbus backend instance has a watch on /local/domain/0/backend/vbd and
the frontend instance has a watch on /local/domain/U/device/vbd. When the
device details above are written, these two watches fire, and the Xenbus
instances begin negotiation.
The backend reads the frontend and frontend-id nodes, and then places a watch
on <frontend>/state.
The frontend reads the backend and backend-id nodes, and then places a watch
on <backend>/state.
These two watches are handled symmetrically inside
xenbus_probe:read_otherend_details and the details are made available in
xenbus_device.otherend, xenbus_device.otherend_id, and
xenbus_device.otherend_watch.
For the backend, the hotplug subsystem is triggered, in order to bring the
physical device online.
Initialisation proceeds by calling the blkfront/blkback probe functions, in
order that they may perform device-specific initialisation, and when this is
complete, each driver will switch to a different state. blkback creates a
watch on the store, waiting for the hotplug scripts to complete, and switches
to XenbusStateInitWait. blkfront creates the ring buffer and event channel
for sharing with the backend, advertises those details in the store, and
switches to XenbusStateInitialised.
When blkback has received the physical device details from the hotplug
scripts, then it creates the necessary connection to the kernel device layer.
When it has received the ring-buffer details from the frontend (indicated by
the frontend state change) then it maps that connection. When both these
things have happened (in either order) then it writes the physical device
details to the store, for use by the frontend, and then switches to the
Connected state.
When blkfront sees the switch to the Connected state, it can read those
physical device details, connect to the kernel device layer itself, and also
switch to the Connected state.
Event diagram:
Xenbus Xenbus
Hotplug Backend Frontend
------- ------- --------
Initialising Initialising
| |
|<---start----+ |
| | |
| InitWait |
| | write
| | ring/
write | channel
physdets-------->| details
| |
|<---------------------Initialised
| |
write |
physdets |
| |
Connected---------------------->|
| |
| Connected
| |
The netfront driver does not need to wait for details from its backend, and so
can skip immediately to the Connected state.
Device Closedown
----------------
Orderly closedown can be requested by the user, as a device hotplug request to
Xend or other tools, or by the drivers when they encounter an error.
An orderly closedown can be accomplished by changing the backend state to
Closing. This will trigger the frontend to tear down it's kernel connection,
flushing through any requests that it has in flight, and then to change to
state Closed. The backend will respond to the frontend's change to Closed by
deregistering itself and switching to state Closed also. Frontends may tear
down immediately on error, without requiring the backend state to change to
Closing first.
Xenbus Xenbus
Hotplug Backend Frontend
------- ------- --------
(Written by control |
tools, e.g. Xend, |
or by backend on |
error) |
| |
Closing-------------------->Closing
| |
| |
| flush
| |
| |
Closed<--------------------Closed
| |
|<--------unregister |
| device |
| | |
remove
store
directories
|
or
Xenbus Xenbus
Hotplug Backend Frontend
------- ------- --------
| (Written by frontend
| on error)
| |
Closing<--------------------Closing
| +----------------------->|
| |
| flush
| |
| |
Closed<--------------------Closed
| |
<---------unregister |
| device |
| | |
remove
store
directories
|
Migration
---------
Migration differs from closedown in that the connection from frontend driver
to the kernel device layer is not disturbed; only the Xenbus connection is
torn down. When a driver is disconnected for this reason, it receives a call
from the lower layers. On resumption, the new backend details are read and
new watches established.
Device reconfiguration
----------------------
If live reconfiguration is required between backend and frontend, this is
handled with device-specific watches on the store. Each driver stays in the
Connected state throughout this.
[-- Attachment #3: Type: text/plain, Size: 138 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread