Re: Inter-domain Communication using Virtual Sockets (high-level design)

[David Vrabel <david.vrabel@citrix.com>]

On 11/06/13 19:07, David Vrabel wrote:
> All,
> 
> This is a high-level design document for an inter-domain communication
> system under the virtual sockets API (AF_VSOCK) recently added to Linux.
> 
> Two low-level transports are discussed: a shared ring based one
> requiring no additional hypervisor support and v4v.
> 
> The PDF (including the diagrams) is available here:
> 
> http://xenbits.xen.org/people/dvrabel/inter-domain-comms-C.pdf

This design was mentioned in a Xen Dev. Summit presentation and I was
reminded of the prototype I wrote a while back.  I haven't yet had the
time to update the design document to reflect the outcome of the prototype.

The prototype is available in this git repo:

git://xenbits.xen.org/people/dvrabel/idc-prototype.git

The prototype is entirely in userspace. A daemon (one per domain) takes
the role of the kernel, providing a system call like interface to other
programs (via XML-RPC).

The calls provided (and their POSIX equivalents are):

    idc_connect() (socket + bind + connect)
    idc_disconnect() (close/shutdown)
    idc_listen() (socket + bind + listen)
    idc_accept() (accept)
    idc_unlisten() (close)
    idc_send() (send)
    idc_recv() (recv)

All connections between two domains are multiplex over the same link.
The setup of the link is done via Xenstore (see link_mgr.c for the
sequence of operations) and connections are then requested using a
CONNECT_req/CONNECT_rsp pair over this link. The data link itself uses
libxenvchan.

Data is encapulated in DATA_ind messages.

Connections are disconnected with a DISCONNECT_ind message. If a link as
no further connections using it, it is disconnected. Disconnecting a
link requires no co-operation from the other peer (the DISCONNECT_ind is
advisory and has no response), so links can disconnected at any time if
the remote end is misbehaving.

The prototype has some key shortcomings:

    It uses grant map/unmap and not grant copies. A replacement link
layer using only grant copies should just be a drop in replacement for
the existing use of libvchan.

    The way the XML-RPC library was uses means it serializes all RPCs.
Simultaneous send/recv/accept/etc. is not possible as these calls may
block. This is only a limitation of the RPC implementation.

To run the demo/test programs:

    Install libxenctrl, libxenstore, libxenvchan into each domain (DUT).

    Run ./idc-setup domid... with all DUTs. This will setup the xenstore
keys to allows these domains to connect to each other.

    Run ./link_mgr in each DUT.

    In one domain run ./test_accept 80 to listen for a connection on
port 80.

    In another domain run ./test_conn domid to connect.

David