Re: VIRTIO - compatibility with different virtualization solutions

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Fri, Feb 21, 2014 at 10:05:06AM +0000, Wei Liu wrote:
> On Thu, Feb 20, 2014 at 06:50:59PM -0800, Anthony Liguori wrote:
> > On Wed, Feb 19, 2014 at 5:31 PM, Rusty Russell <rusty@au1.ibm.com> wrote:
> > > Anthony Liguori <anthony@codemonkey.ws> writes:
> > >> On Tue, Feb 18, 2014 at 4:26 PM, Rusty Russell <rusty@au1.ibm.com> wrote:
> > >>> Daniel Kiper <daniel.kiper@oracle.com> writes:
> > >>>> Hi,
> > >>>>
> > >>>> Below you could find a summary of work in regards to VIRTIO compatibility with
> > >>>> different virtualization solutions. It was done mainly from Xen point of view
> > >>>> but results are quite generic and can be applied to wide spectrum
> > >>>> of virtualization platforms.
> > >>>
> > >>> Hi Daniel,
> > >>>
> > >>>         Sorry for the delayed response, I was pondering...  CC changed
> > >>> to virtio-dev.
> > >>>
> > >>> From a standard POV: It's possible to abstract out the where we use
> > >>> 'physical address' for 'address handle'.  It's also possible to define
> > >>> this per-platform (ie. Xen-PV vs everyone else).  This is sane, since
> > >>> Xen-PV is a distinct platform from x86.
> > >>
> > >> I'll go even further and say that "address handle" doesn't make sense too.
> > >
> > > I was trying to come up with a unique term, I wasn't trying to define
> > > semantics :)
> > 
> > Understood, that wasn't really directed at you.
> > 
> > > There are three debates here now: (1) what should the standard say, and
> > 
> > The standard should say, "physical address"

This conversation is heading towards - implementation needs it - hence lets
make the design have it. Which I am OK with - but if we are going that
route we might as well call this thing 'my-pony-number' because I think
each hypervisor will have a different view of it.

Some of them might use a physical address with some flag bits on it.
Some might use just physical address.

And some might want an 32-bit value that has no correlation to to physical
nor virtual addresses.
> > 
> > > (2) how would Linux implement it,
> > 
> > Linux should use the PCI DMA API.

Aye.
> > 
> > > (3) should we use each platform's PCI
> > > IOMMU.
> > 
> > Just like any other PCI device :-)

Aye.
> > 
> > >> Just using grant table references is not enough to make virtio work
> > >> well under Xen.  You really need to use bounce buffers ala persistent
> > >> grants.
> > >
> > > Wait, if you're using bounce buffers, you didn't make it "work well"!
> > 
> > Preaching to the choir man...  but bounce buffering is proven to be
> > faster than doing grant mappings on every request.  xen-blk does
> > bounce buffering by default and I suspect netfront is heading that
> > direction soon.
> > 
> 
> FWIW Annie Li @ Oracle once implemented a persistent map prototype for
> netfront and the result was not satisfying.

Which could be due to the traffic pattern. There is a lot of back/forth
traffic on a single ring in network (TCP with ACK/SYN).

With block the issue was a bit different and we do more of streaming
workloads.

> 
> > It would be a lot easier to simply have a global pool of grant tables
> > that effectively becomes the DMA pool.  Then the DMA API can bounce
> > into that pool and those addresses can be placed on the ring.
> > 
> > It's a little different for Xen because now the backends have to deal
> > with physical addresses but the concept is still the same.

Rusty, this is a part below is Xen specific - so you are welcome to gloss over it.

I presume you would also need some machinary for the hypervisor to give
access to this 64MB (or whatever size) pool (and we could make grant pages have
2MB granularity - so we just 32 grants) to the backend.

But the backend would have to know the grant entries to at least do the proper
mapping and unmapping (if it choose to)? And for that it needs
the grant value to make the proper hypercall to map its memory
(backend) to the frontend memory.

Or are you saying - instead of using grant entries just use physical
addresses - and naturally the hypervisor would have to use that as well.
Since it is just a number, why not make it at least something and we
won't need to keep a 'grant->physical address' lookup machinery?


> > 
> 
> How would you apply this to Xen's security model? How can hypervisor
> effectively enforce access control? "Handle" and "physical address" are
> essentially not the same concept, otherwise you wouldn't have proposed
> this change. Not saying I'm against this change, just this description
> is too vague for me to understand the bigger picture.
> 
> But a downside for sure is that if we go with this change we then have
> to maintain two different paths in backend. However small the difference
> is it is still a burden.

Or just in the grant machinery. The backends just plucks this number
in data structures and that is all it cares about.

> 
> Wei.