Re: Re: Interdomain comms

[Harry Butterworth <harry@hebutterworth.freeserve.co.uk>]

On Tue, 2005-05-10 at 11:09 +0100, Andrew Warfield wrote:
> Just considering the disk and net interfaces, the
> current device channels each make particular decisions regarding (a)
> what to copy and what to map, (b) when to send notification to get
> efficient batching through the scheduler, and most recently (c) which
> grant mechanism to use to pass pages securely across domains.

(a) looks like a property of the local_buffer_reference type. You might
be able to think of a really good way of doing (b) automatically
independent of any specific client and (c) is either like (a) or it's a
strategy passed when registering a buffer to get a remote buffer
reference.  I'm guessing here though---Is there a list of all the
considerations that went into the design of the current behaviour? That
would help in mapping it to any new proposal.

> The
> buffer_reference struct that Harry mentioned looks quite interesting
> as a start though in terms of describing a variety of underlying
> transports.  Do you have a paper reference on that work, Harry?

I can't give you a suitable reference, but there's not much more to the
concepts than what I've already written up on this list and I can chip
in with anything I've forgotten.

Here are a few more random details:

Buffer providers can register to get a local_buffer_reference type for
the buffers they will provide. As a minimum, they also have to register
methods to copy data between a buffer of that type and the stack. This
allows a core generic mechanism to implement any copy operation.

Buffer providers can register specialised methods to copy directly
between two specific local buffer types. These methods override the
generic mechanism.

As well as a copy operation which leaves the buffers identical, you
might want a more efficient move operation which leaves the source
undefined (or perhaps all zeroes or unmapped or something).

Some local_buffer_reference types might have wider APIs. For example, an
application might want to mess with and be frugal with individual pages
in which case it might know that it allocates all its buffers from a
buffer provider that provides buffers of a specific type.  Knowing the
type, it can down-cast the local buffer references it owns to get access
to the wider API to do more detailed work.

Reads can often complete when the data is received without waiting for
separate status.  I worked this optimisation into my implementation by
getting notification from a registered buffer when it had been filled.

My disconnects were cluster-wide and were coupled to the lifetime of
remote_buffer_references in the cluster.  I didn't think this was
appropriate for Xen so my sketchy API had per-connection disconnects and
didn't mention any coupling with the remote_buffer_reference lifetime.
This is relevant when it comes to defining what kind of transport errors
might happen and requires further thought.

> for the moment though, I think it would be interesting to
> see how well the existing local host cases can be generalized.  ;)

Yep, sounds like a good way to start :-)

>  
> > And with the domain control channel there's an implicit assumption
> > that 'there can be only one'. This means for example, that domain A
> > using a device with backend in domain B can't connect directly to domain B,
> > but has to be 'introduced' by xend. It'd be better if it could connect
> > directly.
> 
> This is not a flaw with the current implementation -- it's completely
> intentional.  By forcing control through xend we ensure that there is
> a single point for control logic, and for managing state.  Why do you
> feel it would be better to provide arbitrary point-to-point comms in a
> VMM environment that is specifically trying to provide isolation
> between guests?

There are two different issues here:

1) Having one Xend might be correct at the moment. The _assumption in
the guests_ that there is only one Xend is technically a minor flaw. If,
for example, the guest got an idc_address for Xend, the guest would be
decoupled from the design choice of one or many Xends.

2) The Xend introductions.  The weird aspect about these is that they
are a triangular protocol with phases initiated by xend in two
directions, potentially at the same time. I'm more used to control flow
following resource dependencies, for example: server tells third-party
about resource availability; third-party assigns resources to clients;
clients connect directly to server. This is also triangular but if you
put the server at the top and bottom it becomes a stack ordered by
dependencies.  I think stacks are much less confusing than triangles and
have more easily defined interlocks and less risk of introducing timing
windows.  I think it ought to be possible to get the security right for
a stack solution as well as the triangle solution. Maybe a secuity
expert could provide some help.