Re: [PATCH v6 0/3] libxenvchan: interdomain communications library

[Daniel De Graaf <dgdegra@tycho.nsa.gov>]

On 11/24/2011 03:02 PM, Anil Madhavapeddy wrote:
> On Fri, Sep 30, 2011 at 10:40:11AM -0400, Daniel De Graaf wrote:
>>>>
>>>> 2011/9/23 Daniel De Graaf <dgdegra@tycho.nsa.gov>
>>>>
>>>>> Changes since v5:
>>>>>  - Unify gntdev osdep interface
>>>>>  - Eliminate notify_result and revert mapping if notify ioctl fails
>>>>>  - Rename functions and structures to libxenvchan
>>>>>  - Use application-specified xenstore path for ring/event data
>>>>>  - Enforce maximum ring size of 2^20 bytes on client
>>>>>  - Change to LGPL 2.1
>>>>>
>>>>> [PATCH 1/3] libxc: add xc_gnttab_map_grant_ref_notify
>>>>> [PATCH 2/3] libxc: add xc_gntshr_* functions
>>>>> [PATCH 3/3] libvchan: interdomain communications library
>>>>>
> <snip>
>> This library depends on gntdev for the client and gntalloc for the server;
>> these were merged in 2.6.39. The client can be used with 2.6.32.x kernels
>> however it is not possible to detect when a peer crashes or exits without
>> calling libxenvchan_close() on that kernel.
> 
> I'm trying this with your most recent dom0 patches all included, and
> xen-unstable. In order to get decent performance, I tried the vchan-node
> examples with a cranked up buffer size, which fails immediately on 64-bit
> VMs.  This below patch fixes the xc_gntshr_share_pages call (which sets
> notify_offset to -1 and without this bounds check will send a big offset
> to the unmap notify ioctl).
> 
> With this, communication does work with the big buffers, but it never
> frees them, and so I can get an ENOSPACE error by calling vchan-node a few
> times. What's the intended cleanup path for the grants in normal use of
> the vchan library? Something's holding onto them, but I don't have a
> serial console on my current laptop in order to drop into Xen's serial and
> find out more.
> 

As long as things aren't crashing, you should be able to use "xl debug-key g"
and "xl dmesg" together to view the grant table debug output.

The ENOSPC bug is due to a missing gnttab_free_grant_reference call, the patch
to fix should be sent as a reply. The libxc patch fixing notify_offset is
incomplete; a more complete version will also be sent as a reply.

It is possible to observe this ENOSPC error even with proper cleanup if the
client refuses to release the grants it has mapped. Xen does not allow domains
to force unmapping of pages they have granted, so the best we can do is try to
remove any hanging grants on later gntalloc calls.