Xen support for PCI-SIG I/O Virtualization

Xen support for PCI-SIG I/O Virtualization

[Masroor Vettuparambil]

[-- Attachment #1.1: Type: text/plain, Size: 374 bytes --]

Hi,
 
what is the current support of Xen for PCI-SIG IOV SR/MR spec? 
I have a PCIe device which supports multiple virtual functions(VF), and
want to enable direct assignment of each of these VFs to different
domUs.
So is it possible to assign a PCIe virtual channel/function to domU
directly?
Can anybody explain me how could I do this.
 
thank you,
 
Masroor

[-- Attachment #1.2: Type: text/html, Size: 1453 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen support for PCI-SIG I/O Virtualization

[Greg Law]

Masroor Vettuparambil wrote:
> Hi,
>  
> what is the current support of Xen for PCI-SIG IOV SR/MR spec?
> I have a PCIe device which supports multiple virtual functions(VF), and 
> want to enable direct assignment of each of these VFs to different domUs.
> So is it possible to assign a PCIe virtual channel/function to domU 
> directly?

Have you seen the work we did recently to allow smart NIC's to be 
accessed directly from the guest?  Essentially your "accelerated" driver 
is a plugin to netfront (which knows how to send/receive packets with 
safe direct access hardware to the NIC), and a plugin for netback which 
deals with the privileged part of programming the NIC (e.g. creating 
virtual devices and handing them out to the guests as appropriate). 
Among other things, this architecture means you can support migrating to 
machines with no smart hardware, or a different smart NIC.

Let me know if you want to know more.

As far as programming the IOMMU goes, I've seen various patches flying 
around so I know work has been done here, but I don't know any more than 
that.

Cheers,

Greg
-- 
Greg Law                                              +44 1223 518 040

RE: Xen support for PCI-SIG I/O Virtualization

[Woller, Thomas]

> Let me know if you want to know more.
Curious if your patches have any multi-root logic.. Or if only SR.

Do you have specific smart NICs that you recommend that work with your
patches/Xen?

thanks,
Tom

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Greg Law
> Sent: Wednesday, January 16, 2008 11:56 AM
> To: Masroor Vettuparambil
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Xen support for PCI-SIG I/O Virtualization
> 
> Masroor Vettuparambil wrote:
> > Hi,
> >  
> > what is the current support of Xen for PCI-SIG IOV SR/MR spec?
> > I have a PCIe device which supports multiple virtual functions(VF), 
> > and want to enable direct assignment of each of these VFs 
> to different domUs.
> > So is it possible to assign a PCIe virtual channel/function to domU 
> > directly?
> 
> Have you seen the work we did recently to allow smart NIC's 
> to be accessed directly from the guest?  Essentially your 
> "accelerated" driver is a plugin to netfront (which knows how 
> to send/receive packets with safe direct access hardware to 
> the NIC), and a plugin for netback which deals with the 
> privileged part of programming the NIC (e.g. creating virtual 
> devices and handing them out to the guests as appropriate). 
> Among other things, this architecture means you can support 
> migrating to machines with no smart hardware, or a different 
> smart NIC.
> 
> Let me know if you want to know more.
> 
> As far as programming the IOMMU goes, I've seen various 
> patches flying around so I know work has been done here, but 
> I don't know any more than that.
> 
> Cheers,
> 
> Greg
> -- 
> Greg Law                                              +44 1223 518 040
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 

Re: Xen support for PCI-SIG I/O Virtualization

[Greg Law]

Woller, Thomas wrote:
>> Let me know if you want to know more.
> Curious if your patches have any multi-root logic.. Or if only SR.

Actually, our patches do nothing with IOV (SR or MR).  They are a 
framework which allows one to write plugin drivers for netfront/netback 
for accelerated network tx/rx.  The framework does not mandate how this 
is implemented.  We expect it to be used by smart devices which can be 
accessed directly from the guest, but even that is not strictly 
necessary to use the framework.  If the NIC does allow safe access 
directly from the guest, it might be that the silicon uses PCI IOV, or 
some other mechanism (e.g. our smart NIC uses a proprietary mechanism 
which pre-dates IOV).

> 
> Do you have specific smart NICs that you recommend that work with your
> patches/Xen?

Yes, ours :)

You can currently buy them as "SMC Tigertron" cards.  Our plugin drivers 
will be submitted to this list soon.


Cheers,

Greg

-- 
Greg Law         Solarflare Communications            +44 1223 518 040

RE: Xen support for PCI-SIG I/O Virtualization

[Masroor Vettuparambil]

hi Greg, 

thanks for the comment.
I have yet to go through ur work. Please provide a quick link.

I would like to know whether your proprietary mechanism supports virtual
functions just like in IOV.
If so, can you brief me on how this is accessed from the host and
assigned to guest domains? 
Infact I am more interested how this is happening in IOV method.


regards
Masroor  

-----Original Message-----
From: Greg Law [mailto:glaw@solarflare.com] 
Sent: Thursday, January 17, 2008 4:59 PM
To: Woller, Thomas
Cc: Masroor Vettuparambil; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Xen support for PCI-SIG I/O Virtualization

Woller, Thomas wrote:
>> Let me know if you want to know more.
> Curious if your patches have any multi-root logic.. Or if only SR.

Actually, our patches do nothing with IOV (SR or MR).  They are a
framework which allows one to write plugin drivers for netfront/netback
for accelerated network tx/rx.  The framework does not mandate how this
is implemented.  We expect it to be used by smart devices which can be
accessed directly from the guest, but even that is not strictly
necessary to use the framework.  If the NIC does allow safe access
directly from the guest, it might be that the silicon uses PCI IOV, or
some other mechanism (e.g. our smart NIC uses a proprietary mechanism
which pre-dates IOV).

> 
> Do you have specific smart NICs that you recommend that work with your

> patches/Xen?

Yes, ours :)

You can currently buy them as "SMC Tigertron" cards.  Our plugin drivers
will be submitted to this list soon.


Cheers,

Greg

-- 
Greg Law         Solarflare Communications            +44 1223 518 040

Re: Xen support for PCI-SIG I/O Virtualization

[Greg Law]

Hi Masroor,

Masroor Vettuparambil wrote:
> hi Greg, 
> 
> thanks for the comment.
> I have yet to go through ur work. Please provide a quick link.
> 
> I would like to know whether your proprietary mechanism supports virtual
> functions just like in IOV.
> If so, can you brief me on how this is accessed from the host and
> assigned to guest domains? 
> Infact I am more interested how this is happening in IOV method.

The framework is deliberately very generic.  Essentially it is just a 
way for netfront to have a plugin which can handle tx/rx of packets 
directly.  How this works (IOV, proprietary, whatever) is not the 
concern of the framework: you write netfront and netback plugins to 
implement it as required by your hardware.

The framework is not really intended to avoid you needing to write lots 
of code - it's not a "library".  Rather it's the minimum amount of 
infrastructure required so that VMs can be migrated between different 
hosts with different types of smart NICs (or no smart NICs).

No one has written an implementation of plugins that use IOV yet, but 
I'm pretty confident it will work without major changes.

As for the code, a good place to start is the definition of struct 
netfront_accel_hooks in linux/drivers/xen/netfront/netfront.h

The set of patches were posted to xen-devel by Kieran Mansley Fri, 18 
May 2007.  Sadly, I'm not sure what the change set number is, but you 
can get more info here: http://lwn.net/Articles/235124/

g

-- 
Greg Law          Solarflare Communications           +44 1223 518 040