[PATCH 0/4] Support accelerated network plugin modules

[PATCH 0/4] Support accelerated network plugin modules

[Kieran Mansley]

This series of patches adds support to netfront and netback to allow
accelerated drivers to plug in and support a fast path for network
traffic.  This is designed to allow use of virtualisable "smart" NICs
directly from domU, with help/control from dom0.  Experiments with
Solarflare hardware have shown a doubling in network performance when
using this approach on 10Gbps networks, compared to the traditional
network path through netfront/netback.  i.e. Performance from domU
approaches that normally seen in dom0.

The patches are as follows:

1/4:  Add xenbus_for_each_[back,front]end functions to iterate each bus
 - this is just a pair of simple helper functions to allow each item on
the backend/frontend bus to be iterated.

2/4:  Add accel option to vif xend config
 - this allows users to specify which vif interfaces should be
accelerated, and which plugin module should be used.

3/4:  Frontend net driver acceleration
 - this provides hooks in the netfront driver to allow a plugin module
to attach and be called at appropriate points.

4/4:  Backend net driver acceleration
 - this provides hooks in the netback driver to allow a plugin module to
attach and be called at appropriate points.  


Kieran

Re: [PATCH 0/4] Support accelerated network plugin modules

[Keir Fraser]

On 8/5/07 10:54, "Kieran Mansley" <kmansley@solarflare.com> wrote:

> 3/4:  Frontend net driver acceleration
>  - this provides hooks in the netfront driver to allow a plugin module
> to attach and be called at appropriate points.
> 
> 4/4:  Backend net driver acceleration
>  - this provides hooks in the netback driver to allow a plugin module to
> attach and be called at appropriate points.

These probably should be reviewed by someone like Herbert Xu for sanity. I
can immediately see that Linux coding style is not followed (lack of spaces
around if(){, and LOUD MACROS are two problems I can see).
netfront_stop_queue and friends look like pointless wrappers.

 -- Keir

Re: [PATCH 0/4] Support accelerated network plugin modules

[Herbert Xu]

Kieran Mansley <kmansley@solarflare.com> wrote:
> This series of patches adds support to netfront and netback to allow
> accelerated drivers to plug in and support a fast path for network
> traffic.  This is designed to allow use of virtualisable "smart" NICs
> directly from domU, with help/control from dom0.  Experiments with
> Solarflare hardware have shown a doubling in network performance when
> using this approach on 10Gbps networks, compared to the traditional
> network path through netfront/netback.  i.e. Performance from domU
> approaches that normally seen in dom0.

Could you post some examples of how these hooks are actually used by
your driver? Without them it's hard for me to assess whether these
hooks are appropriate.

Also, since netfront may be merged into upstream Linux soon, please
cc these patches to netdev@vger.kernel.org so folks there can also
comment.

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [PATCH 0/4] Support accelerated network plugin modules

[Muli Ben-Yehuda]

On Wed, May 09, 2007 at 07:37:17PM +1000, Herbert Xu wrote:
> Kieran Mansley <kmansley@solarflare.com> wrote:
> > This series of patches adds support to netfront and netback to allow
> > accelerated drivers to plug in and support a fast path for network
> > traffic.  This is designed to allow use of virtualisable "smart" NICs
> > directly from domU, with help/control from dom0.  Experiments with
> > Solarflare hardware have shown a doubling in network performance when
> > using this approach on 10Gbps networks, compared to the traditional
> > network path through netfront/netback.  i.e. Performance from domU
> > approaches that normally seen in dom0.
> 
> Could you post some examples of how these hooks are actually used by
> your driver? Without them it's hard for me to assess whether these
> hooks are appropriate.

Seconded.

> Also, since netfront may be merged into upstream Linux soon, please
> cc these patches to netdev@vger.kernel.org so folks there can also
> comment.

Also seconded.

Cheers,
Muli

Re: [PATCH 0/4] Support accelerated network plugin modules

[Kieran Mansley]

On Wed, 2007-05-09 at 14:26 +0300, Muli Ben-Yehuda wrote:
> On Wed, May 09, 2007 at 07:37:17PM +1000, Herbert Xu wrote:
> > Kieran Mansley <kmansley@solarflare.com> wrote:
> > > This series of patches adds support to netfront and netback to allow
> > > accelerated drivers to plug in and support a fast path for network
> > > traffic.  This is designed to allow use of virtualisable "smart" NICs
> > > directly from domU, with help/control from dom0.  Experiments with
> > > Solarflare hardware have shown a doubling in network performance when
> > > using this approach on 10Gbps networks, compared to the traditional
> > > network path through netfront/netback.  i.e. Performance from domU
> > > approaches that normally seen in dom0.
> > 
> > Could you post some examples of how these hooks are actually used by
> > your driver? Without them it's hard for me to assess whether these
> > hooks are appropriate.
> 
> Seconded.

No problem.  I'll knock up a pair of simple dummy drivers to use these
as an example, and I can also provide our full version if you prefer - I
hesitated from including that to avoid confusion about what was being
suggested as a patch at the moment.

> > Also, since netfront may be merged into upstream Linux soon, please
> > cc these patches to netdev@vger.kernel.org so folks there can also
> > comment.
> 
> Also seconded.

I'll repost the patches with everyone's suggested modifications together
with the example drivers, and CC them.

Kieran

Re: [PATCH 0/4] Support accelerated network plugin modules

[Jambunathan K]

I have a question regarding the feature set that is proposed here. The
intention is to further my own understanding and get some additional
insights.

Doesn't accelerated network plugins solve the same problem as say a
"native" PCI passthroughed driver. Put other way, what can an
accelerator do that a passthroughed driver cannot do?

Regards,
Jambunathan K.

Kieran Mansley wrote:
> This series of patches adds support to netfront and netback to allow
> accelerated drivers to plug in and support a fast path for network
> traffic.  This is designed to allow use of virtualisable "smart" NICs
> directly from domU, with help/control from dom0.  Experiments with
> Solarflare hardware have shown a doubling in network performance when
> using this approach on 10Gbps networks, compared to the traditional
> network path through netfront/netback.  i.e. Performance from domU
> approaches that normally seen in dom0.
> 
> The patches are as follows:
> 
> 1/4:  Add xenbus_for_each_[back,front]end functions to iterate each bus
>  - this is just a pair of simple helper functions to allow each item on
> the backend/frontend bus to be iterated.
> 
> 2/4:  Add accel option to vif xend config
>  - this allows users to specify which vif interfaces should be
> accelerated, and which plugin module should be used.
> 
> 3/4:  Frontend net driver acceleration
>  - this provides hooks in the netfront driver to allow a plugin module
> to attach and be called at appropriate points.
> 
> 4/4:  Backend net driver acceleration
>  - this provides hooks in the netback driver to allow a plugin module to
> attach and be called at appropriate points.  
> 
> 
> Kieran
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: [PATCH 0/4] Support accelerated network plugin modules

[Kieran Mansley]

On Thu, 2007-05-31 at 22:04 +0530, Jambunathan K wrote:
> I have a question regarding the feature set that is proposed here. The
> intention is to further my own understanding and get some additional
> insights.
> 
> Doesn't accelerated network plugins solve the same problem as say a
> "native" PCI passthroughed driver. Put other way, what can an
> accelerator do that a passthroughed driver cannot do?

Functionally they're similar, in that they both enable a guest to get
direct access to the NIC to deliver higher network performance, but off
the top of my head:

 - PCI passthrough would require hardware to support multiple PCI
devices.  The accelerated plugin approach we've put forward doesn't (but
would also work fine with devices that virtualise at the PCI layer), and
so would support a wider range of devices.

 - You're limited with PCI passthrough to the number of PCI devices your
NIC can support, which is (for the NICs I've seen) considerably less
than the number of VIs that other NICs can support.  This will limit the
number of guests you can provide accelerated network interfaces to.

 - I'm not sure what the story is for migration with PCI passthrough,
but with the accelerated plugin approach your network interfaces can
transparently (as far as the guest's network stack is concerned) move
from being accelerated to not-accelerated (and vice-versa) depending on
what hardware is available, and migrate from one type of accelerated
hardware to another.  

 - A PCI-passthrough-based network interface might struggle to route
packets to a virtual machine on the same host.  The accelerated network
plugin can just fall back to the netfront/netback normal network path to
get packets to these, or use facilities on the NIC to achieve the same
if available.

In summary, it's "passthrough" rather than "PCI passthrough", if that
makes sense.

Kieran

Re: [PATCH 0/4] Support accelerated network plugin modules

[Jambunathan K]

Kieran


>> I have a question regarding the feature set that is proposed here. The
>> intention is to further my own understanding and get some additional
>> insights.
>>
>> Doesn't accelerated network plugins solve the same problem as say a
>> "native" PCI passthroughed driver. Put other way, what can an
>> accelerator do that a passthroughed driver cannot do?
> 
> Functionally they're similar, in that they both enable a guest to get
> direct access to the NIC to deliver higher network performance, but off
> the top of my head:
> 
>  - PCI passthrough would require hardware to support multiple PCI
> devices.  The accelerated plugin approach we've put forward doesn't (but
> would also work fine with devices that virtualise at the PCI layer), and
> so would support a wider range of devices.
> 
>  - You're limited with PCI passthrough to the number of PCI devices your
> NIC can support, which is (for the NICs I've seen) considerably less
> than the number of VIs that other NICs can support.  This will limit the
> number of guests you can provide accelerated network interfaces to.
> 
>  - I'm not sure what the story is for migration with PCI passthrough,
> but with the accelerated plugin approach your network interfaces can
> transparently (as far as the guest's network stack is concerned) move
> from being accelerated to not-accelerated (and vice-versa) depending on
> what hardware is available, and migrate from one type of accelerated
> hardware to another.  
> 
>  - A PCI-passthrough-based network interface might struggle to route
> packets to a virtual machine on the same host.  The accelerated network
> plugin can just fall back to the netfront/netback normal network path to
> get packets to these, or use facilities on the NIC to achieve the same
> if available.
> 
> In summary, it's "passthrough" rather than "PCI passthrough", if that
> makes sense.
> 

My understanding very much tallies with the points you have outlined above.

The patches you have submitted is of interest to us. It is possible that
in the near future we would get around to integrating and exercising
your changes against our own driver.

Jambunathan K.