Re: [RFC] netif: staging grants for requests

[Paul Durrant <Paul.Durrant@citrix.com>]

> -----Original Message-----
> From: Joao Martins [mailto:joao.m.martins@oracle.com]
> Sent: 06 January 2017 20:09
> To: Paul Durrant <Paul.Durrant@citrix.com>
> Cc: xen-devel@lists.xenproject.org; Andrew Cooper
> <Andrew.Cooper3@citrix.com>; Wei Liu <wei.liu2@citrix.com>; Stefano
> Stabellini <sstabellini@kernel.org>
> Subject: Re: [RFC] netif: staging grants for requests
> 
> On 01/06/2017 09:33 AM, Paul Durrant wrote:
> >> -----Original Message-----
> >> From: Joao Martins [mailto:joao.m.martins@oracle.com]
> >> Sent: 14 December 2016 18:11
> >> To: xen-devel@lists.xenproject.org
> >> Cc: David Vrabel <david.vrabel@citrix.com>; Andrew Cooper
> >> <Andrew.Cooper3@citrix.com>; Wei Liu <wei.liu2@citrix.com>; Paul
> Durrant
> >> <Paul.Durrant@citrix.com>; Stefano Stabellini <sstabellini@kernel.org>
> >> Subject: [RFC] netif: staging grants for requests
> >>
> >> Hey,
> >>
> >> Back in the Xen hackaton '16 networking session there were a couple of
> ideas
> >> brought up. One of them was about exploring permanently mapped
> grants
> >> between
> >> xen-netback/xen-netfront.
> >>
> >> I started experimenting and came up with sort of a design document (in
> >> pandoc)
> >> on what it would like to be proposed. This is meant as a seed for
> discussion
> >> and also requesting input to know if this is a good direction. Of course, I
> >> am willing to try alternatives that we come up beyond the contents of the
> >> spec, or any other suggested changes ;)
> >>
> >> Any comments or feedback is welcome!
> >>
> >
> > Hi,
> Hey!
> 
> >
> > Sorry for the delay... I've been OOTO for three weeks.
> Thanks for the comments!
> 
> > I like the general approach or pre-granting buffers for RX so that the
> backend
> > can simply memcpy and tell the frontend which buffer a packet appears in
> Cool,
> 
> > but IIUC you are proposing use of a single pre-granted area for TX also,
> which would
> > presumably require the frontend to always copy on the TX side? I wonder if
> we
> > might go for a slightly different scheme...
> I see.
> 
> >
> > The assumption is that the working set of TX buffers in the guest OS is fairly
> > small (which is probably true for a small number of heavily used sockets
> and an
> > OS that uses a slab allocator)...
> Hmm, [speaking about linux] maybe for the skb allocation cache. For the
> remaining packet pages maybe not for say a scather-gather list...? But I guess
> it would need to be validated whether this working set is indeed kept small
> as
> this seems like a very strong assumption to comply with its various
> possibilities in workloads. Plus wouldn't we leak info from these pages if it
> wasn't used on the device but rather elsewhere in the guest stack?

Yes, potentially there is an information leak but I am assuming that the backend is also trusted by the frontend, which is pretty will baked into the protocol anyway. Also, if the working set (which is going to be OS/stack dependent) turned out to be a bit too large then the frontend can always fall back to a copy into a locally allocated buffer, as in your proposal, anyway.

> 
> > The guest TX code maintains a hash table of buffer addresses to grant refs.
> When
> > a packet is sent the code looks to see if it has already granted the buffer
> and
> > re-uses the existing ref if so, otherwise it grants the buffer and adds the
> new
> > ref into the table.
> 
> > The backend also maintains a hash of grant refs to addresses and,
> whenever it
> > sees a new ref, it grant maps it and adds the address into the table.
> Otherwise
> > it does a hash lookup and thus has a buffer address it can immediately
> memcpy
> > from.
> >
> > If the frontend wants the backend to release a grant ref (e.g. because it's
> > starting to run out of grant table) then a control message can be used to
> ask
> > for it back, at which point the backend removes the ref from its cache and
> > unmaps it.
> Wouldn't this be somewhat similar to the persistent grants in xen block
> drivers?

Yes, it would, and I'd rather that protocol was also re-worked in this fashion.

> 
> > Using this scheme we allow a guest OS to still use either a zero-copy
> approach
> > if it wishes to do so, or a static pre-grant... or something between
> > (e.g. pre-grant for headers, zero copy for bulk data).
> >
> > Does that sound reasonable?
> Not sure yet but it looks nice if we can indeed achieve the zero copy part. But
> I have two concerns: say a backend could be forced to always remove refs as
> its
> cache is always full having frontend not being able to reuse these pages
> (subject to its own allocator behavior, in case assumption above wouldn't be
> satisfied) nullifying backend effort into maintaining its mapped grefs table.
> One other concern is whether those pages (assumed to be reused) might be
> leaking
> off guest data to the backend (when not used on netfront).

As I said, the protocol already requires the backend to be trusted by the frontend (since grants cannot be revoked, if for no other reason) so information leakage is not a particular concern. What I want to avoid is a protocol that denies any possibility of zero-copy, even in the best case, which is the way things currently are with persistent grants in blkif.

  Paul

> 
> Joao
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel