Is there example code and/or documentation for developing a Xen back-end/front-end driver pair?

Is there example code and/or documentation for developing a Xen back-end/front-end driver pair?

[Randy Thelen]

Folks --

I'm walking through all the Xen Python code and driver code to  
determine how to build a custom back-end/front-end driver pair.  I'm  
not using any of the existing transport types (blkif, netif, etc.).   
So, I need to start from scratch supporting configuration file  
elements in Xen down to the drivers in the kernel.

If there are any good pointers as to how to build something new from  
the ground up within the framework of the existing Xen Daemon and be- 
fe pairing, I'd really like to read it.

-- Randy

Re: Is there example code and/or documentation for developing a Xen back-end/front-end driver pair?

[Anthony Liguori]

Randy Thelen wrote:
> Folks --
>
> I'm walking through all the Xen Python code and driver code to 
> determine how to build a custom back-end/front-end driver pair.  I'm 
> not using any of the existing transport types (blkif, netif, etc.).  
> So, I need to start from scratch supporting configuration file 
> elements in Xen down to the drivers in the kernel.
>
> If there are any good pointers as to how to build something new from 
> the ground up within the framework of the existing Xen Daemon and 
> be-fe pairing, I'd really like to read it.

No.  It has changed too many times (although it's probably stable enough 
now to document).

In principle, once you have bring up working, making it work in Xend 
isn't so bad.  Do you have specific questions or are you just looking 
for a where to begin?

Regards,

Anthony Liguori



> -- Randy
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Is there example code and/or documentation for developing a Xen back-end/front-end driver pair?

[Randy Thelen]

Anthony Liguori wrote:

> (although it's probably stable enough now to document)

I'm getting that feeling.

> In principle, once you have bring up working, making it work in  
> Xend isn't so bad.  Do you have specific questions or are you just  
> looking for a where to begin?

I see this "orange of a problem" and I'm trying to get my fingernail  
under the orange peel.

It appears that in order to get the front-end to talk to the back- 
end, using the standard xenbus and interface type files I seen in  
blkback/ and blkfront/ driver source code directories, that I need to  
begin in Xend.  So, I'm looking in Xend and I -think- I get most of  
the architecture relevant here.  But, it's not clear to me, yet,  
exactly what Xend asks of the back end driver in order to get the  
system going.

For example, when it's decided that a blk back end will be used,  
blkif.py is run and part of what appears to happen is that the dom0  
blk driver is kicked somehow (!?) and a Python Event is used (in  
DevController.py) with an xswatch() call to wait for the backed to  
startup.  And, it seems that the trigger to observing the startup is  
complete is the back end putting some data into XenStore (into  
statusPath = backpath + '/' + HOTPLUG_STATUS_NODE).

I'm trying to figure out how this aspect of Xend works,  
specifically.  :-)

Any pointers there?  Or, am I way out in the weeds?

-- Randy

Re: Is there example code and/or documentation for developing a Xen back-end/front-end driver pair?

[Ewan Mellor]

[-- Attachment #1: Type: text/plain, Size: 2416 bytes --]

On Mon, Jun 26, 2006 at 03:16:08PM -0700, Randy Thelen wrote:

> Anthony Liguori wrote:
> 
> >(although it's probably stable enough now to document)
> 
> I'm getting that feeling.
> 
> >In principle, once you have bring up working, making it work in  
> >Xend isn't so bad.  Do you have specific questions or are you just  
> >looking for a where to begin?
> 
> I see this "orange of a problem" and I'm trying to get my fingernail  
> under the orange peel.
> 
> It appears that in order to get the front-end to talk to the back- 
> end, using the standard xenbus and interface type files I seen in  
> blkback/ and blkfront/ driver source code directories, that I need to  
> begin in Xend.  So, I'm looking in Xend and I -think- I get most of  
> the architecture relevant here.  But, it's not clear to me, yet,  
> exactly what Xend asks of the back end driver in order to get the  
> system going.
> 
> For example, when it's decided that a blk back end will be used,  
> blkif.py is run and part of what appears to happen is that the dom0  
> blk driver is kicked somehow (!?) and a Python Event is used (in  
> DevController.py) with an xswatch() call to wait for the backed to  
> startup.  And, it seems that the trigger to observing the startup is  
> complete is the back end putting some data into XenStore (into  
> statusPath = backpath + '/' + HOTPLUG_STATUS_NODE).

Attached are some notes of mine.  They are quite old, but I think it all is
pretty much relevant.

Basically, a backend watches /local/domain/0/backend/<deviceClass> and a
frontend watches /local/domain/<domID>/device/<deviceClass>.  When the tools
write into those directories in the store, the drivers get a watch fired, and
check those directories to see the new device details.  The new internal state
is created based on that.

If you are using the normal Xenbus code, then you can register your device
driver with the Xenbus layer, and much of this is done for you.

The drivers themselves use a simple state machine to handshake between front
and backends.

The backends use hotplug / udev events inside domain 0 to trigger scripts for
simple bringup/teardown stuff, like attaching new vifs to bridges or
whatever.  The tools watch for a specific node to be written in the store to
indicate that this hotplug phase has succeeded (it's usually the one that does
the final sanity checking and often has to bail out).

HTH,

Ewan.

[-- Attachment #2: drivers.txt --]
[-- Type: text/plain, Size: 8622 bytes --]

Architecture for Split Drivers Within Xen
=========================================

The block, net and TPM Xen drivers are "split" drivers: they have a portion in
a privileged domain handling the physical device -- the backend -- and a
frontend in the unprivileged domain acting as a proxy upon the backend.  The
backend and frontend communicate using shared event channels and ring buffers,
in an architecture known as the Xenbus.

For concreteness, this section shall discuss the block driver, with
differences between that and the net driver highlighted where necessary.  The
block driver is referred to in various places by the shorthands "blk" or
"vbd".  I shall also refer to the privileged domain as domain 0, and the
unprivileged domain as domain U.

The basic architecture can be considered to be a chain of four protagonists, a
pair of devices communicating across the Xenbus, and then a pair of devices
using that bus, handling the device-specific interaction with the kernel
device layer.


priv.                                                                 unpriv.
domain                  Xenbus  interdomain   Xenbus                  domain
kernel -- blkback  --  backend -------------- frontend -- blkfront -- kernel
device    instance    instance  interconnect  instance    instance    device
layer                                                                 layer


In order to establish communication across this chain, a number of parameters
need to be passed from privileged to unprivileged domain, and vice versa.
Some of these parameters are specific to the blkback/blkfront pair, and some
are more general, applying to all split drivers.  All of these parameters are
passed using the Xen Store.


Device Initialisation
---------------------

To trigger the creation of a device connection, Xend (or another tool) writes
frontend and backend details to the store.  These new details are seen by the
Xenbus driver instances, and initialisation begins.

The details to be written are:

/local/domain/0/backend/vbd/U/<deviceID>/...
    frontend       /local/domain/U/device/vbd/<deviceID>
    frontend-id    U
    state          XenbusStateInitialising
    ...            <device-specific details>

/local/domain/U/device/vbd/<deviceID>/...
    backend        /local/domain/0/backend/vbd/U/<deviceID>
    backend-id     0
    state          XenbusStateInitialising
    ...            <device-specific details>


The Xenbus backend instance has a watch on /local/domain/0/backend/vbd and
the frontend instance has a watch on /local/domain/U/device/vbd.  When the
device details above are written, these two watches fire, and the Xenbus
instances begin negotiation.

The backend reads the frontend and frontend-id nodes, and then places a watch
on <frontend>/state.

The frontend reads the backend and backend-id nodes, and then places a watch
on <backend>/state.

These two watches are handled symmetrically inside
xenbus_probe:read_otherend_details and the details are made available in
xenbus_device.otherend, xenbus_device.otherend_id, and
xenbus_device.otherend_watch.

For the backend, the hotplug subsystem is triggered, in order to bring the
physical device online.


Initialisation proceeds by calling the blkfront/blkback probe functions, in
order that they may perform device-specific initialisation, and when this is
complete, each driver will switch to a different state.  blkback creates a
watch on the store, waiting for the hotplug scripts to complete, and switches
to XenbusStateInitWait.  blkfront creates the ring buffer and event channel 
for sharing with the backend, advertises those details in the store, and
switches to XenbusStateInitialised.

When blkback has received the physical device details from the hotplug
scripts, then it creates the necessary connection to the kernel device layer.
When it has received the ring-buffer details from the frontend (indicated by
the frontend state change) then it maps that connection.  When both these
things have happened (in either order) then it writes the physical device
details to the store, for use by the frontend, and then switches to the
Connected state.

When blkfront sees the switch to the Connected state, it can read those
physical device details, connect to the kernel device layer itself, and also
switch to the Connected state.


Event diagram:

              Xenbus                       Xenbus
Hotplug       Backend                     Frontend
-------       -------                     --------

            Initialising                Initialising
                 |                           |
   |<---start----+                           |
   |             |                           |
   |          InitWait                       |
   |             |                         write
   |             |                         ring/
 write           |                        channel
physdets-------->|                        details
                 |                           |
                 |<---------------------Initialised
                 |                           |
               write                         |
              physdets                       |
                 |                           |
             Connected---------------------->|
                 |                           |
                 |                       Connected
                 |                           |

The netfront driver does not need to wait for details from its backend, and so
can skip immediately to the Connected state.


Device Closedown
----------------

Orderly closedown can be requested by the user, as a device hotplug request to
Xend or other tools, or by the drivers when they encounter an error.

An orderly closedown can be accomplished by changing the backend state to
Closing.  This will trigger the frontend to tear down it's kernel connection,
flushing through any requests that it has in flight, and then to change to
state Closed.  The backend will respond to the frontend's change to Closed by 
deregistering itself and switching to state Closed also.  Frontends may tear
down immediately on error, without requiring the backend state to change to
Closing first.


              Xenbus                       Xenbus
Hotplug       Backend                     Frontend
-------       -------                     --------

           (Written by control               |
            tools, e.g. Xend,                |
            or by backend on                 |
            error)                           |
                 |                           |
              Closing-------------------->Closing
                 |                           |
                 |                           |
                 |                         flush
                 |                           |
                 |                           |
               Closed<--------------------Closed
                 |                           |
   |<--------unregister                      |
   |           device                        |
   |             |                           |
 remove
 store
 directories
   |


or

              Xenbus                       Xenbus
Hotplug       Backend                     Frontend
-------       -------                     --------

                 |                    (Written by frontend
                 |                         on error)
                 |                           |
              Closing<--------------------Closing
                 |  +----------------------->|
                 |                           |
                 |                         flush
                 |                           |
                 |                           |
               Closed<--------------------Closed
                 |                           |
   <---------unregister                      |
   |           device                        |
   |             |                           |
 remove
 store
 directories
   |


Migration
---------

Migration differs from closedown in that the connection from frontend driver
to the kernel device layer is not disturbed; only the Xenbus connection is
torn down.  When a driver is disconnected for this reason, it receives a call
from the lower layers.  On resumption, the new backend details are read and
new watches established.


Device reconfiguration
----------------------

If live reconfiguration is required between backend and frontend, this is
handled with device-specific watches on the store.  Each driver stays in the
Connected state throughout this.


[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Is there example code and/or documentation for developing a Xen back-end/front-end driver pair?

[Randy Thelen]

Ewan Mellor wrote:

> Attached are some notes of mine.  They are quite old, but I think  
> it all is
> pretty much relevant.
>
> Basically, a backend watches /local/domain/0/backend/<deviceClass>  
> and a
> frontend watches /local/domain/<domID>/device/<deviceClass>.  When  
> the tools
> write into those directories in the store, the drivers get a watch  
> fired, and
> check those directories to see the new device details.  The new  
> internal state
> is created based on that.
>
> If you are using the normal Xenbus code, then you can register your  
> device
> driver with the Xenbus layer, and much of this is done for you.
>
> The drivers themselves use a simple state machine to handshake  
> between front
> and backends.
>
> The backends use hotplug / udev events inside domain 0 to trigger  
> scripts for
> simple bringup/teardown stuff, like attaching new vifs to bridges or
> whatever.  The tools watch for a specific node to be written in the  
> store to
> indicate that this hotplug phase has succeeded (it's usually the  
> one that does
> the final sanity checking and often has to bail out).
>
> HTH,

IHI {It Helps Indeed!}  Thanks.  Could I encourage you to post those  
on the XenSource Wiki?

-- Randy

Re: Is there example code and/or documentation for developing a Xen back-end/front-end driver pair?

[Ewan Mellor]

On Mon, Jun 26, 2006 at 03:45:04PM -0700, Randy Thelen wrote:

> Ewan Mellor wrote:
> 
> >Attached are some notes of mine.  They are quite old, but I think  
> >it all is
> >pretty much relevant.
> >
> >Basically, a backend watches /local/domain/0/backend/<deviceClass>  
> >and a
> >frontend watches /local/domain/<domID>/device/<deviceClass>.  When  
> >the tools
> >write into those directories in the store, the drivers get a watch  
> >fired, and
> >check those directories to see the new device details.  The new  
> >internal state
> >is created based on that.
> >
> >If you are using the normal Xenbus code, then you can register your  
> >device
> >driver with the Xenbus layer, and much of this is done for you.
> >
> >The drivers themselves use a simple state machine to handshake  
> >between front
> >and backends.
> >
> >The backends use hotplug / udev events inside domain 0 to trigger  
> >scripts for
> >simple bringup/teardown stuff, like attaching new vifs to bridges or
> >whatever.  The tools watch for a specific node to be written in the  
> >store to
> >indicate that this hotplug phase has succeeded (it's usually the  
> >one that does
> >the final sanity checking and often has to bail out).
> >
> >HTH,
> 
> IHI {It Helps Indeed!}  Thanks.  Could I encourage you to post those  
> on the XenSource Wiki?

I thought someone had already (obviously not).  I've put them on a
XenSplitDrivers page.

Cheers,

Ewan.