[RFC]vhost/vhost-net backend for PCI cards

[RFC]vhost/vhost-net backend for PCI cards

[Nikhil Rao]

We are implementing a driver for a PCIe card that runs Linux. This card
needs "virtual" network/disk/console devices, so we have reused the
virtio devices on on the card and provided a host backend that interacts
with the virtio devices through the card's driver. 

this approach is very much like what was proposed on this thread
http://permalink.gmane.org/gmane.linux.ports.sh.devel/10379

We will posting the driver soon, so perhaps I am jumping the gun with my
question below about replacing our backend with vhost.

It is possible for vhost (along with vhost-net in the case of
virtio-net) to serve as the backend. The copy between virtio buffers and
skbs happens in the tun/tap driver which means tun/tap may need to use a
HW DMA engine (the card has one) for copy across the bus to get close to
the full PCIe bandwidth.

tun/tap was probably never designed for this use case, but reusing vhost
does simplify our backend since it is now only involved in setup and
potentially has a performance/memory footprint advantage due to avoiding
context switches/intermediate buffer copy and this idea can be
generalized to other cards as well.

Comments/suggestions ?

Thanks,
Nikhil

Re: [RFC]vhost/vhost-net backend for PCI cards

[Nikhil Rao]

> On Wed, 2013-02-27 at 11:17 -0800, Ira W. Snyder wrote:
> Hi Nikhil,
> 
> I don't have any code to offer, but may be able to provide some
> suggestions. I work on a system which has a single (x86) host computer,
> and many PowerPC data processing boards, which are connected via PCI.
> This sounds similar to your hardware.
> 
> Our system was developed before vhost existed. I built a (fairly dumb)
> network driver that just transfers packets over PCI using the PowerPC
> DMA controller. It works, however I think a more generic virtio solution
> will work better. A virtio solution will also allow other types of
> devices besides a network interface.
> 
> I have done some studying of rproc/rpmsg/vhost/vringh, and may have some
> suggestions about those pieces of kernel functionality.
> 
> A HW DMA engine is absolutely needed to get good performance over the
> PCI bus. I don't have experience with PCIe.
> 
> You may want to investigate rproc/rpmsg to help do virtio device
> discovery.

> When dealing with virtio, it may be helpful to think of your PCIe card
> as the "host". 

We wanted to support a host-based disk, using virtio-blk on the card
seemed to be a good way to do this, given that the card runs Linux.

also from a performance perspective, which would be better ? [virtio-net
on the card/backend on the host cpu] v/s [virtio-net on the
hostcpu/backend on the card] ?  given that the host cpu is more powerful
than the card cpu.

> In virtio nomenclature, the "host" is in charge of copying data. 
> Your HW DMA engine needs to be controlled by the "host".

In our case, the host driver controls the HW DMA engine (a subset of the
card DMA engine channels are under host control). But you may be 
referring to the case where the host doesn't have access to the card's
DMA engine.

> 
> Your main computer (the computer the PCIe card plugs into) will be a
> virtio "guest" and will run the virtio-net/virtio-console/etc. drivers.
> 
> Several vendors have contacted me privately to ask for the code for my
> (dumb) network-over-PCI driver. A generic solution to this problem will
> definitely find a userbase.
> 
> I look forward to porting the code to run on my PowerPC PCI boards when
> it becomes available. I am able to help review code as well.
> 
> Good luck!
> Ira

Re: [RFC]vhost/vhost-net backend for PCI cards

[Ira W. Snyder]

On Wed, Feb 27, 2013 at 03:50:54AM -0800, Nikhil Rao wrote:
> 
> We are implementing a driver for a PCIe card that runs Linux. This card
> needs "virtual" network/disk/console devices, so we have reused the
> virtio devices on on the card and provided a host backend that interacts
> with the virtio devices through the card's driver. 
> 
> this approach is very much like what was proposed on this thread
> http://permalink.gmane.org/gmane.linux.ports.sh.devel/10379
> 
> We will posting the driver soon, so perhaps I am jumping the gun with my
> question below about replacing our backend with vhost.
> 
> It is possible for vhost (along with vhost-net in the case of
> virtio-net) to serve as the backend. The copy between virtio buffers and
> skbs happens in the tun/tap driver which means tun/tap may need to use a
> HW DMA engine (the card has one) for copy across the bus to get close to
> the full PCIe bandwidth.
> 
> tun/tap was probably never designed for this use case, but reusing vhost
> does simplify our backend since it is now only involved in setup and
> potentially has a performance/memory footprint advantage due to avoiding
> context switches/intermediate buffer copy and this idea can be
> generalized to other cards as well.
> 
> Comments/suggestions ?
> 
> Thanks,
> Nikhil
> 
> _______________________________________________
> Virtualization mailing list
> Virtualization@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Hi Nikhil,

I don't have any code to offer, but may be able to provide some
suggestions. I work on a system which has a single (x86) host computer,
and many PowerPC data processing boards, which are connected via PCI.
This sounds similar to your hardware.

Our system was developed before vhost existed. I built a (fairly dumb)
network driver that just transfers packets over PCI using the PowerPC
DMA controller. It works, however I think a more generic virtio solution
will work better. A virtio solution will also allow other types of
devices besides a network interface.

I have done some studying of rproc/rpmsg/vhost/vringh, and may have some
suggestions about those pieces of kernel functionality.

A HW DMA engine is absolutely needed to get good performance over the
PCI bus. I don't have experience with PCIe.

You may want to investigate rproc/rpmsg to help do virtio device
discovery.

When dealing with virtio, it may be helpful to think of your PCIe card
as the "host". In virtio nomenclature, the "host" is in charge of
copying data. Your HW DMA engine needs to be controlled by the "host".

Your main computer (the computer the PCIe card plugs into) will be a
virtio "guest" and will run the virtio-net/virtio-console/etc. drivers.

Several vendors have contacted me privately to ask for the code for my
(dumb) network-over-PCI driver. A generic solution to this problem will
definitely find a userbase.

I look forward to porting the code to run on my PowerPC PCI boards when
it becomes available. I am able to help review code as well.

Good luck!
Ira

Re: [RFC]vhost/vhost-net backend for PCI cards

[Ira W. Snyder]

On Wed, Feb 27, 2013 at 04:58:20AM -0800, Nikhil Rao wrote:
> On Wed, 2013-02-27 at 11:17 -0800, Ira W. Snyder wrote:
> 
> > Hi Nikhil,
> > 
> > I don't have any code to offer, but may be able to provide some
> > suggestions. I work on a system which has a single (x86) host computer,
> > and many PowerPC data processing boards, which are connected via PCI.
> > This sounds similar to your hardware.
> > 
> > Our system was developed before vhost existed. I built a (fairly dumb)
> > network driver that just transfers packets over PCI using the PowerPC
> > DMA controller. It works, however I think a more generic virtio solution
> > will work better. A virtio solution will also allow other types of
> > devices besides a network interface.
> > 
> > I have done some studying of rproc/rpmsg/vhost/vringh, and may have some
> > suggestions about those pieces of kernel functionality.
> > 
> > A HW DMA engine is absolutely needed to get good performance over the
> > PCI bus. I don't have experience with PCIe.
> > 
> > You may want to investigate rproc/rpmsg to help do virtio device
> > discovery.
> 
> > When dealing with virtio, it may be helpful to think of your PCIe card
> > as the "host". 
> 
> We wanted to support a host-based disk, using virtio-blk on the card
> seemed to be a good way to do this, given that the card runs Linux.
> 
> also from a performance perspective, which would be better ? [virtio-net
> on the card/backend on the host cpu] v/s [virtio-net on the
> hostcpu/backend on the card] ?  given that the host cpu is more powerful
> than the card cpu.
> 

I never considered using virtio-blk, so I don't have any input about it.

I don't know much about virtio performance either. The experts on this
list will have to send their input.

> > In virtio nomenclature, the "host" is in charge of copying data. 
> > Your HW DMA engine needs to be controlled by the "host".
> 
> In our case, the host driver controls the HW DMA engine (a subset of the
> card DMA engine channels are under host control). But you may be
> referring to the case where the host doesn't have access to the card's
> DMA engine.
> 

That's right. On our PowerPC cards, the DMA hardware can be controlled
by either the PowerPC processor, or the PCI host system, but not both.
We need the DMA for various operations on the PowerPC card itself, so
the PCI host system cannot be used to control the DMA hardware.

This is also true for other vendors who contacted me privately.

Your PCIe card seems to have better features than any similar systems
I've worked with.

I skimmed the code being used with rpmsg/rproc/virtio on the ARM DSPs on
the OMAP platform. They use the DSP as the "virtio host" (it copies
memory, performing the same function as vhost) and the OMAP as the
"virtio guest" (it runs virtio-net/etc.).

Among all virtio "guest" drivers (virtio-net/virtio-console/etc.), I
think virtio-blk behaves differently from the rest. All of the others
work better with DMA hardware when the PCI card is the "virtio host."

You might want to contact Ohad Ben-Cohen for his advice. He did a lot of
work on drivers/rpmsg and drivers/remoteproc. He works with the OMAP
hardware.

Ira

> > 
> > Your main computer (the computer the PCIe card plugs into) will be a
> > virtio "guest" and will run the virtio-net/virtio-console/etc. drivers.
> > 
> > Several vendors have contacted me privately to ask for the code for my
> > (dumb) network-over-PCI driver. A generic solution to this problem will
> > definitely find a userbase.
> > 
> > I look forward to porting the code to run on my PowerPC PCI boards when
> > it becomes available. I am able to help review code as well.
> > 
> > Good luck!
> > Ira
> 
>