OASIS Virtual I/O Device (VIRTIO) TC

OASIS Virtual I/O Device (VIRTIO) TC

[Daniel Kiper]

Hi,

I was appointed by Xen Advisory Board to OASIS Virtual I/O Device
(VIRTIO) TC as a memeber. I will oversee its work from Xen point
of view, however, deliverables will be as much as possible
"virtualization platform" agnostic.

According to [1]:

The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify
virtual devices, making them more extensible and more recognizable.

[...]

The TC intends to define formal specifications for virtual device buses
(including PCI) for a variety of devices, including network devices.
Specification development will be based upon the "Virtio PCI Card
Specification" [2] v0.9.5, seeking solutions that support portability,
simplicity, least-surprise for driver authors, extensibility, and
performance. The specification will also document existing
implementations and practice.

[...]

Committee started its work on July 30, 2013. Currently, we are
solving some organizational issues and slowly starting core work.

I am going to post changes proposal for balloon device
(memory hotplug support) and add new timer/clock device
which is not specified in "Virtio PCI Card Specification".

All committee work is made in public. Please check [1] for
more details. However, only members can participate
in teleconferences and vote.

If you have any questions, ideas, concerns, etc. please feel
free to send them to virtio-comment@lists.oasis-open.org [3]
(register to this list first) or directly to me.

1) http://www.oasis-open.org/committees/virtio/
2) http://ozlabs.org/~rusty/virtio-spec/virtio-0.9.5.pdf
3) http://www.oasis-open.org/committees/comments/index.php?wg_abbrev=virtio

Daniel

PS I will be on vacation from 14th of August to 22nd of August.
   I will reply for all emails after 22nd of August.

Re: OASIS Virtual I/O Device (VIRTIO) TC

[Wei Liu]

On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote:
> Hi,
> 
> I was appointed by Xen Advisory Board to OASIS Virtual I/O Device
> (VIRTIO) TC as a memeber. I will oversee its work from Xen point
> of view, however, deliverables will be as much as possible
> "virtualization platform" agnostic.
> 
> According to [1]:
> 
> The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify
> virtual devices, making them more extensible and more recognizable.
> 
> [...]
> 
> The TC intends to define formal specifications for virtual device buses
> (including PCI) for a variety of devices, including network devices.
> Specification development will be based upon the "Virtio PCI Card
> Specification" [2] v0.9.5, seeking solutions that support portability,
> simplicity, least-surprise for driver authors, extensibility, and
> performance. The specification will also document existing
> implementations and practice.
> 

How is Xen PV driver fit in this? Xen PV doesn't have virtual PCI bus.

Wei.

Re: OASIS Virtual I/O Device (VIRTIO) TC

[Konrad Rzeszutek Wilk]

On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote:
> On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote:
> > Hi,
> > 
> > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device
> > (VIRTIO) TC as a memeber. I will oversee its work from Xen point
> > of view, however, deliverables will be as much as possible
> > "virtualization platform" agnostic.
> > 
> > According to [1]:
> > 
> > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify
> > virtual devices, making them more extensible and more recognizable.
> > 
> > [...]
> > 
> > The TC intends to define formal specifications for virtual device buses
> > (including PCI) for a variety of devices, including network devices.
> > Specification development will be based upon the "Virtio PCI Card
> > Specification" [2] v0.9.5, seeking solutions that support portability,
> > simplicity, least-surprise for driver authors, extensibility, and
> > performance. The specification will also document existing
> > implementations and practice.
> > 
> 
> How is Xen PV driver fit in this? Xen PV doesn't have virtual PCI bus.

You should know that better than anybody else I think :-)

One could implement an PCI over XenBus I suppose. Or just write an
XenBus PCI driver that would do all the neccessary things to respond
to the proper commands.
> 
> Wei.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: OASIS Virtual I/O Device (VIRTIO) TC

[Wei Liu]

On Thu, Aug 15, 2013 at 10:16:41AM -0400, Konrad Rzeszutek Wilk wrote:
> On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote:
> > On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote:
> > > Hi,
> > > 
> > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device
> > > (VIRTIO) TC as a memeber. I will oversee its work from Xen point
> > > of view, however, deliverables will be as much as possible
> > > "virtualization platform" agnostic.
> > > 
> > > According to [1]:
> > > 
> > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify
> > > virtual devices, making them more extensible and more recognizable.
> > > 
> > > [...]
> > > 
> > > The TC intends to define formal specifications for virtual device buses
> > > (including PCI) for a variety of devices, including network devices.
> > > Specification development will be based upon the "Virtio PCI Card
> > > Specification" [2] v0.9.5, seeking solutions that support portability,
> > > simplicity, least-surprise for driver authors, extensibility, and
> > > performance. The specification will also document existing
> > > implementations and practice.
> > > 
> > 
> > How is Xen PV driver fit in this? Xen PV doesn't have virtual PCI bus.
> 
> 

My limited knowledge of Virtio is pretty dated now, that's why I raised
that question. ;-)

> One could implement an PCI over XenBus I suppose. Or just write an
> XenBus PCI driver that would do all the neccessary things to respond
> to the proper commands.
> > 

Back in the date my impression was that XenBus is asynchronuous while
virtual PCI is synchronuous (i.e. trap-process-return), they are not
quite compatible.

Furtuer more using XenBus is mainly used for configuration, while Virtio
over PCI supports both configuration and sending notifications etc.

I'm not sure XenBus PCI driver would be a good idea...

Wei.

> > Wei.
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel

Re: OASIS Virtual I/O Device (VIRTIO) TC

[Ian Campbell]

On Thu, 2013-08-15 at 15:24 +0100, Wei Liu wrote:
> On Thu, Aug 15, 2013 at 10:16:41AM -0400, Konrad Rzeszutek Wilk wrote:
> > On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote:
> > > On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote:
> > > > Hi,
> > > > 
> > > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device
> > > > (VIRTIO) TC as a memeber. I will oversee its work from Xen point
> > > > of view, however, deliverables will be as much as possible
> > > > "virtualization platform" agnostic.
> > > > 
> > > > According to [1]:
> > > > 
> > > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify
> > > > virtual devices, making them more extensible and more recognizable.
> > > > 
> > > > [...]
> > > > 
> > > > The TC intends to define formal specifications for virtual device buses
> > > > (including PCI) for a variety of devices, including network devices.
> > > > Specification development will be based upon the "Virtio PCI Card
> > > > Specification" [2] v0.9.5, seeking solutions that support portability,
> > > > simplicity, least-surprise for driver authors, extensibility, and
> > > > performance. The specification will also document existing
> > > > implementations and practice.
> > > > 
> > > 
> > > How is Xen PV driver fit in this? Xen PV doesn't have virtual PCI bus.
> > 
> > 
> 
> My limited knowledge of Virtio is pretty dated now, that's why I raised
> that question. ;-)
> 
> > One could implement an PCI over XenBus I suppose. Or just write an
> > XenBus PCI driver that would do all the neccessary things to respond
> > to the proper commands.
> > > 
> 
> Back in the date my impression was that XenBus is asynchronuous while
> virtual PCI is synchronuous (i.e. trap-process-return), they are not
> quite compatible.
> 
> Furtuer more using XenBus is mainly used for configuration, while Virtio
> over PCI supports both configuration and sending notifications etc.
> 
> I'm not sure XenBus PCI driver would be a good idea...

I thought this part of virtio was pluggable and that PCI was one option
(although for a long time the only one). On ARM you can also use
virt_mmio instead these days. A shared ring for cfg + evtchn model for
notification doesn't seem too far fetched to me.

A far bigger problem IMHO with virtio is that it basically discards the
Xen security model. I don't know how hard it will be to retrofit gnttab
based access control into the virtio protocols. A lot would be my
guess...

Ian.

Re: OASIS Virtual I/O Device (VIRTIO) TC

[Wei Liu]

On Thu, Aug 15, 2013 at 03:58:46PM +0100, Ian Campbell wrote:
> On Thu, 2013-08-15 at 15:24 +0100, Wei Liu wrote:
> > On Thu, Aug 15, 2013 at 10:16:41AM -0400, Konrad Rzeszutek Wilk wrote:
> > > On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote:
> > > > On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote:
> > > > > Hi,
> > > > > 
> > > > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device
> > > > > (VIRTIO) TC as a memeber. I will oversee its work from Xen point
> > > > > of view, however, deliverables will be as much as possible
> > > > > "virtualization platform" agnostic.
> > > > > 
> > > > > According to [1]:
> > > > > 
> > > > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify
> > > > > virtual devices, making them more extensible and more recognizable.
> > > > > 
> > > > > [...]
> > > > > 
> > > > > The TC intends to define formal specifications for virtual device buses
> > > > > (including PCI) for a variety of devices, including network devices.
> > > > > Specification development will be based upon the "Virtio PCI Card
> > > > > Specification" [2] v0.9.5, seeking solutions that support portability,
> > > > > simplicity, least-surprise for driver authors, extensibility, and
> > > > > performance. The specification will also document existing
> > > > > implementations and practice.
> > > > > 
> > > > 
> > > > How is Xen PV driver fit in this? Xen PV doesn't have virtual PCI bus.
> > > 
> > > 
> > 
> > My limited knowledge of Virtio is pretty dated now, that's why I raised
> > that question. ;-)
> > 
> > > One could implement an PCI over XenBus I suppose. Or just write an
> > > XenBus PCI driver that would do all the neccessary things to respond
> > > to the proper commands.
> > > > 
> > 
> > Back in the date my impression was that XenBus is asynchronuous while
> > virtual PCI is synchronuous (i.e. trap-process-return), they are not
> > quite compatible.
> > 
> > Furtuer more using XenBus is mainly used for configuration, while Virtio
> > over PCI supports both configuration and sending notifications etc.
> > 
> > I'm not sure XenBus PCI driver would be a good idea...
> 
> I thought this part of virtio was pluggable and that PCI was one option
> (although for a long time the only one). On ARM you can also use

Yes this is true.

However the statement (?) in Daniel's first email (based upon the
"Virtio PCI Card Specification") gave me the impression that they
planned to support virtual PCI only...

> virt_mmio instead these days. A shared ring for cfg + evtchn model for
> notification doesn't seem too far fetched to me.
> 
> A far bigger problem IMHO with virtio is that it basically discards the
> Xen security model. I don't know how hard it will be to retrofit gnttab
> based access control into the virtio protocols. A lot would be my
> guess...
> 

Indeed, seems there is lots of work to do...

Wei.

> Ian.
> 

Re: OASIS Virtual I/O Device (VIRTIO) TC

[Daniel Kiper]

Sorry for late reply but I was on vaction. Slowly recovering.

On Thu, Aug 15, 2013 at 04:09:04PM +0100, Wei Liu wrote:
> On Thu, Aug 15, 2013 at 03:58:46PM +0100, Ian Campbell wrote:
> > On Thu, 2013-08-15 at 15:24 +0100, Wei Liu wrote:
> > > On Thu, Aug 15, 2013 at 10:16:41AM -0400, Konrad Rzeszutek Wilk wrote:
> > > > On Thu, Aug 15, 2013 at 02:37:28PM +0100, Wei Liu wrote:
> > > > > On Tue, Aug 13, 2013 at 08:41:28PM +0200, Daniel Kiper wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I was appointed by Xen Advisory Board to OASIS Virtual I/O Device
> > > > > > (VIRTIO) TC as a memeber. I will oversee its work from Xen point
> > > > > > of view, however, deliverables will be as much as possible
> > > > > > "virtualization platform" agnostic.
> > > > > >
> > > > > > According to [1]:
> > > > > >
> > > > > > The goal of the OASIS Virtual I/O Device (VIRTIO) TC is to simplify
> > > > > > virtual devices, making them more extensible and more recognizable.
> > > > > >
> > > > > > [...]
> > > > > >
> > > > > > The TC intends to define formal specifications for virtual device buses
> > > > > > (including PCI) for a variety of devices, including network devices.
> > > > > > Specification development will be based upon the "Virtio PCI Card
> > > > > > Specification" [2] v0.9.5, seeking solutions that support portability,
> > > > > > simplicity, least-surprise for driver authors, extensibility, and
> > > > > > performance. The specification will also document existing
> > > > > > implementations and practice.
> > > > > >
> > > > >
> > > > > How is Xen PV driver fit in this? Xen PV doesn't have virtual PCI bus.
> > > >
> > > >
> > >
> > > My limited knowledge of Virtio is pretty dated now, that's why I raised
> > > that question. ;-)
> > >
> > > > One could implement an PCI over XenBus I suppose. Or just write an
> > > > XenBus PCI driver that would do all the neccessary things to respond
> > > > to the proper commands.
> > > > >
> > >
> > > Back in the date my impression was that XenBus is asynchronuous while
> > > virtual PCI is synchronuous (i.e. trap-process-return), they are not
> > > quite compatible.
> > >
> > > Furtuer more using XenBus is mainly used for configuration, while Virtio
> > > over PCI supports both configuration and sending notifications etc.
> > >
> > > I'm not sure XenBus PCI driver would be a good idea...
> >
> > I thought this part of virtio was pluggable and that PCI was one option
> > (although for a long time the only one). On ARM you can also use
>
> Yes this is true.
>
> However the statement (?) in Daniel's first email (based upon the
> "Virtio PCI Card Specification") gave me the impression that they
> planned to support virtual PCI only...

Right, VIRTIO TC started its work on the base of "Virtio PCI Card Specification".
However, it was stated immediately that PCI specification should be excluded
from main specification and should form a separate attachment. It means that
VIRTIO would not require PCI bus in the future. Additionally, current document
contains specification for MMIO which looks like good alternative for PCI.

> > virt_mmio instead these days. A shared ring for cfg + evtchn model for
> > notification doesn't seem too far fetched to me.
> >
> > A far bigger problem IMHO with virtio is that it basically discards the
> > Xen security model. I don't know how hard it will be to retrofit gnttab
> > based access control into the virtio protocols. A lot would be my
> > guess...

I am going to convince VIRTIO TC that new spec should be Xen friendly too.

> Indeed, seems there is lots of work to do...

By the way, Wei, Konrad told me that you worked on VRITIO implementation
for Xen and you have hands-on-experience with it. Could you tell me what
issues you met during your work on VRITIO support for Xen environment?

Daniel

Re: OASIS Virtual I/O Device (VIRTIO) TC

[Wei Liu]

On Tue, Aug 27, 2013 at 11:37:41AM +0200, Daniel Kiper wrote:
[...]
> > Yes this is true.
> >
> > However the statement (?) in Daniel's first email (based upon the
> > "Virtio PCI Card Specification") gave me the impression that they
> > planned to support virtual PCI only...
> 
> Right, VIRTIO TC started its work on the base of "Virtio PCI Card Specification".
> However, it was stated immediately that PCI specification should be excluded
> from main specification and should form a separate attachment. It means that
> VIRTIO would not require PCI bus in the future. Additionally, current document
> contains specification for MMIO which looks like good alternative for PCI.
> 
> > > virt_mmio instead these days. A shared ring for cfg + evtchn model for
> > > notification doesn't seem too far fetched to me.
> > >
> > > A far bigger problem IMHO with virtio is that it basically discards the
> > > Xen security model. I don't know how hard it will be to retrofit gnttab
> > > based access control into the virtio protocols. A lot would be my
> > > guess...
> 
> I am going to convince VIRTIO TC that new spec should be Xen friendly too.
> 
> > Indeed, seems there is lots of work to do...
> 
> By the way, Wei, Konrad told me that you worked on VRITIO implementation
> for Xen and you have hands-on-experience with it. Could you tell me what
> issues you met during your work on VRITIO support for Xen environment?
> 

I would not say I have very deep understanding of it because I only
worked on it part-time for two months. But I will try to best to
remember and elaborate. :-)

For HVM guests Virtio works just fine. Xen uses QEMU as device model
which means we have Virtio devices for free. In this case all Virtio
devices use virtual PCI bus provided by QEMU. Basically we get most of
the things for free.

(A recent email on netdev@ from an engineer in Huawei suggests that
virto-net + vhost-net works for him, which is really great. Ref: Is
fallback vhost_net to qemu for live migrate
available？<521C1DCF.5090202@huawei.com>)

For PV guests it requires more work. Virtio is designed to be portable.
The PCI and MMIO implementations your mentioned are in fact called
"transport". As PV guests don't have virtual PCI buses and MMIO don't
seem to work on Xen PV either (it's very possible that I'm wrong on this
MMIO thing as I'm not familiar with it), I had to implement a new
transport. TBH the biggest issue is more of an engineering problem --
virtual PCI is sync while Xen DomU and Dom0 run in prarallel.
The approach I chose back then was just a hack mimicking PCI
which made everything slow. It was not meant to be canonical or
upstreamable. If I were to do this now I would choose to apply the Xen
PV model to the transport.

To make the whole thing work we also need backend supporting Virtio. The
closest thing at hand was QEMU, but again, the transport had to be
re-implemented.

The hack is on http://wiki.xen.org/wiki/Virtio_On_Xen

Wei.

> Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: OASIS Virtual I/O Device (VIRTIO) TC

[Daniel Kiper]

On Tue, Aug 27, 2013 at 12:06:59PM +0100, Wei Liu wrote:
> On Tue, Aug 27, 2013 at 11:37:41AM +0200, Daniel Kiper wrote:
> [...]
> > > Yes this is true.
> > >
> > > However the statement (?) in Daniel's first email (based upon the
> > > "Virtio PCI Card Specification") gave me the impression that they
> > > planned to support virtual PCI only...
> >
> > Right, VIRTIO TC started its work on the base of "Virtio PCI Card Specification".
> > However, it was stated immediately that PCI specification should be excluded
> > from main specification and should form a separate attachment. It means that
> > VIRTIO would not require PCI bus in the future. Additionally, current document
> > contains specification for MMIO which looks like good alternative for PCI.
> >
> > > > virt_mmio instead these days. A shared ring for cfg + evtchn model for
> > > > notification doesn't seem too far fetched to me.
> > > >
> > > > A far bigger problem IMHO with virtio is that it basically discards the
> > > > Xen security model. I don't know how hard it will be to retrofit gnttab
> > > > based access control into the virtio protocols. A lot would be my
> > > > guess...
> >
> > I am going to convince VIRTIO TC that new spec should be Xen friendly too.
> >
> > > Indeed, seems there is lots of work to do...
> >
> > By the way, Wei, Konrad told me that you worked on VRITIO implementation
> > for Xen and you have hands-on-experience with it. Could you tell me what
> > issues you met during your work on VRITIO support for Xen environment?
> >
>
> I would not say I have very deep understanding of it because I only
> worked on it part-time for two months. But I will try to best to
> remember and elaborate. :-)
>
> For HVM guests Virtio works just fine. Xen uses QEMU as device model
> which means we have Virtio devices for free. In this case all Virtio
> devices use virtual PCI bus provided by QEMU. Basically we get most of
> the things for free.
>
> (A recent email on netdev@ from an engineer in Huawei suggests that
> virto-net + vhost-net works for him, which is really great. Ref: Is
> fallback vhost_net to qemu for live migrate
> available？<521C1DCF.5090202@huawei.com>)
>
> For PV guests it requires more work. Virtio is designed to be portable.
> The PCI and MMIO implementations your mentioned are in fact called
> "transport". As PV guests don't have virtual PCI buses and MMIO don't
> seem to work on Xen PV either (it's very possible that I'm wrong on this
> MMIO thing as I'm not familiar with it), I had to implement a new
> transport. TBH the biggest issue is more of an engineering problem --
> virtual PCI is sync while Xen DomU and Dom0 run in prarallel.
> The approach I chose back then was just a hack mimicking PCI
> which made everything slow. It was not meant to be canonical or
> upstreamable. If I were to do this now I would choose to apply the Xen
> PV model to the transport.
>
> To make the whole thing work we also need backend supporting Virtio. The
> closest thing at hand was QEMU, but again, the transport had to be
> re-implemented.
>
> The hack is on http://wiki.xen.org/wiki/Virtio_On_Xen

Thanks for lengthy explanation. It looks that a lot of work has been done.
Could you tell or point where I could find info how buffers/rings are
shared between backend and frontend? Is grant page mechanism used at all?
How buffers/rings addresses are passed between backend and frontend?

Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: OASIS Virtual I/O Device (VIRTIO) TC

[Wei Liu]

On Thu, Aug 29, 2013 at 01:24:03PM +0200, Daniel Kiper wrote:
> >
> > To make the whole thing work we also need backend supporting Virtio. The
> > closest thing at hand was QEMU, but again, the transport had to be
> > re-implemented.
> >
> > The hack is on http://wiki.xen.org/wiki/Virtio_On_Xen
> 
> Thanks for lengthy explanation. It looks that a lot of work has been done.
> Could you tell or point where I could find info how buffers/rings are
> shared between backend and frontend? Is grant page mechanism used at all?
> How buffers/rings addresses are passed between backend and frontend?
> 

Grant table was not used there. I used foreign mapping in QEMU. In that
hack, MFN is passed on the ring. Looking at the original virtio_ring.c
it should be trivial to replace physical addree with a grant ref, thus
allowing proper use of grant table mechanism.

Wei.

> Daniel