[Qemu-devel] Is IOThread for virtio-net a good idea?

[Qemu-devel] Is IOThread for virtio-net a good idea?

[Anton Kuchin]

As far as I can see currently IOThread offloading is used only for block 
devices and all others are emulated by main thread.

I expect that network devices can also benefit from processing in 
separate thread but I couldn't find any recent work in this direction. 
I'm going to implement a PoC but I want to ask if you know any previous 
attempts and do you know why it can be a total waste of time. Are there 
fundamental obstacles that prevent network emulation handling in IOThread?

Re: [Qemu-devel] Is IOThread for virtio-net a good idea?

[Michael S. Tsirkin]

On Mon, Feb 11, 2019 at 04:40:44PM +0300, Anton Kuchin wrote:
> As far as I can see currently IOThread offloading is used only for block
> devices and all others are emulated by main thread.
> 
> I expect that network devices can also benefit from processing in separate
> thread but I couldn't find any recent work in this direction. I'm going to
> implement a PoC but I want to ask if you know any previous attempts and do
> you know why it can be a total waste of time. Are there fundamental
> obstacles that prevent network emulation handling in IOThread?

No but vhost-net is there. Unlike block where you gain lots of
functionality such as snapshots there seems to be little to
be gained by doing it in userspace.

-- 
MST

Re: [Qemu-devel] Is IOThread for virtio-net a good idea?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

On Mon, Feb 11, 2019 at 09:52:01AM -0500, Michael S. Tsirkin wrote:
> On Mon, Feb 11, 2019 at 04:40:44PM +0300, Anton Kuchin wrote:
> > As far as I can see currently IOThread offloading is used only for block
> > devices and all others are emulated by main thread.
> > 
> > I expect that network devices can also benefit from processing in separate
> > thread but I couldn't find any recent work in this direction. I'm going to
> > implement a PoC but I want to ask if you know any previous attempts and do
> > you know why it can be a total waste of time. Are there fundamental
> > obstacles that prevent network emulation handling in IOThread?
> 
> No but vhost-net is there. Unlike block where you gain lots of
> functionality such as snapshots there seems to be little to
> be gained by doing it in userspace.

Anthony Liguori tried virtio-net in a dedicated thread a long time ago
but it was never completed/merged.

I think the argument for vhost-net in the kernel is that userspace
doesn't offer the same APIs as the kernel.  For example, is zero copy
possible from userspace?

Anton: If you want to do virtio-net in userspace, QEMU has
vhost-user-net support so you can perform emulation in a separate
process.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Is IOThread for virtio-net a good idea?

[Michael S. Tsirkin]

On Tue, Feb 12, 2019 at 11:55:05AM +0800, Stefan Hajnoczi wrote:
> On Mon, Feb 11, 2019 at 09:52:01AM -0500, Michael S. Tsirkin wrote:
> > On Mon, Feb 11, 2019 at 04:40:44PM +0300, Anton Kuchin wrote:
> > > As far as I can see currently IOThread offloading is used only for block
> > > devices and all others are emulated by main thread.
> > > 
> > > I expect that network devices can also benefit from processing in separate
> > > thread but I couldn't find any recent work in this direction. I'm going to
> > > implement a PoC but I want to ask if you know any previous attempts and do
> > > you know why it can be a total waste of time. Are there fundamental
> > > obstacles that prevent network emulation handling in IOThread?
> > 
> > No but vhost-net is there. Unlike block where you gain lots of
> > functionality such as snapshots there seems to be little to
> > be gained by doing it in userspace.
> 
> Anthony Liguori tried virtio-net in a dedicated thread a long time ago
> but it was never completed/merged.
> 
> I think the argument for vhost-net in the kernel is that userspace
> doesn't offer the same APIs as the kernel.  For example, is zero copy
> possible from userspace?

Esp with recent spectre/meltdown mitigations, with vhost we are looking
for new ways to batch access checks and avoid overhead.

If you are going to pass the packets back to kernel, you
are better off with them never leaving the kernel.

> Anton: If you want to do virtio-net in userspace, QEMU has
> vhost-user-net support so you can perform emulation in a separate
> process.
> 
> Stefan

And the main point of that is to use a userspace
driver for a hardware nic so packets never end up in kernel.

Re: [Qemu-devel] Is IOThread for virtio-net a good idea?

[Jason Wang]

On 2019/2/12 上午11:55, Stefan Hajnoczi wrote:
> On Mon, Feb 11, 2019 at 09:52:01AM -0500, Michael S. Tsirkin wrote:
>> On Mon, Feb 11, 2019 at 04:40:44PM +0300, Anton Kuchin wrote:
>>> As far as I can see currently IOThread offloading is used only for block
>>> devices and all others are emulated by main thread.
>>>
>>> I expect that network devices can also benefit from processing in separate
>>> thread but I couldn't find any recent work in this direction. I'm going to
>>> implement a PoC but I want to ask if you know any previous attempts and do
>>> you know why it can be a total waste of time. Are there fundamental
>>> obstacles that prevent network emulation handling in IOThread?
>> No but vhost-net is there. Unlike block where you gain lots of
>> functionality such as snapshots there seems to be little to
>> be gained by doing it in userspace.
> Anthony Liguori tried virtio-net in a dedicated thread a long time ago
> but it was never completed/merged.
>
> I think the argument for vhost-net in the kernel is that userspace
> doesn't offer the same APIs as the kernel.  For example, is zero copy
> possible from userspace?


Actually, it will be simpler than doing it at kernel somehow. E.g 
technically, you can write a userspace NIC driver in qemu to achieve 
that or doing it through dpdk pmd.

Thanks


>
> Anton: If you want to do virtio-net in userspace, QEMU has
> vhost-user-net support so you can perform emulation in a separate
> process.
>
> Stefan

Re: [Qemu-devel] Is IOThread for virtio-net a good idea?

[Jason Wang]

On 2019/2/11 下午10:52, Michael S. Tsirkin wrote:
> On Mon, Feb 11, 2019 at 04:40:44PM +0300, Anton Kuchin wrote:
>> As far as I can see currently IOThread offloading is used only for block
>> devices and all others are emulated by main thread.
>>
>> I expect that network devices can also benefit from processing in separate
>> thread but I couldn't find any recent work in this direction. I'm going to
>> implement a PoC but I want to ask if you know any previous attempts and do
>> you know why it can be a total waste of time. Are there fundamental
>> obstacles that prevent network emulation handling in IOThread?
> No but vhost-net is there. Unlike block where you gain lots of
> functionality such as snapshots there seems to be little to
> be gained by doing it in userspace.


I think all the three (vhost-net, vhost-user, vhost-dataplane) have 
their own advantages. E.g compared to vhost-user, vhost dataplane in 
qemu has the following advantages:

- Qemu dataplane could be much safer, e.g there's no need to share all 
its memory to another process.

- Easy to develop, there's no need for the backend to know anything 
about virtio. This will reduce the pain of maintaining compatibility, 
version etc. New features was limited to the changes of qemu itself.

- Faster access of VM metadata, Device IOTLB and RSS/Steering could be 
function calls instead of complex vhost-user protocol. I'm pretty sure 
we will hit more similar issues in the future.

And I can consider some use cases:

- link dpdk to qemu

- implement fast userspace backend like AF_XDP, netmap, packet socket or 
other (or even achieve this through dpdk pmd)

Thanks

Re: [Qemu-devel] Is IOThread for virtio-net a good idea?

[Jason Wang]

On 2019/2/11 下午9:40, Anton Kuchin wrote:
> As far as I can see currently IOThread offloading is used only for 
> block devices and all others are emulated by main thread.
>
> I expect that network devices can also benefit from processing in 
> separate thread but I couldn't find any recent work in this direction. 
> I'm going to implement a PoC but I want to ask if you know any 
> previous attempts and do you know why it can be a total waste of time. 
> Are there fundamental obstacles that prevent network emulation 
> handling in IOThread?
>

I think there're no obstacles.

The only question is whether or not you need to support legacy 
networking backends. If the answer is yes, you need to convert them not 
to assume context of Main Loop. But I don't think it's worth to support 
them. We should focus on high speed solution like linking dpdk.

Another issue is the virtio implementation. Dpdk has virtio library 
which is pretty optimized, we should consider to use it. But last time I 
check the code, it was tightly coupled with AF_UNIX transport of 
vhost-user. We may want to decouple it out of dpdk. Of course we can do 
it our own as well. (Yes I know there's a vhost-user implementation, but 
it was not optimized for performance).

I do have some prototype that is based on vhost-scsi-dataplane with a 
TAP backend. I can send it to you if you wish.

Thanks

Re: [Qemu-devel] Is IOThread for virtio-net a good idea?

[Jason Wang]

On 2019/2/12 下午2:48, Jason Wang wrote:
>
> On 2019/2/11 下午9:40, Anton Kuchin wrote:
>> As far as I can see currently IOThread offloading is used only for 
>> block devices and all others are emulated by main thread.
>>
>> I expect that network devices can also benefit from processing in 
>> separate thread but I couldn't find any recent work in this 
>> direction. I'm going to implement a PoC but I want to ask if you know 
>> any previous attempts and do you know why it can be a total waste of 
>> time. Are there fundamental obstacles that prevent network emulation 
>> handling in IOThread?
>>
>
> I think there're no obstacles.
>
> The only question is whether or not you need to support legacy 
> networking backends. If the answer is yes, you need to convert them 
> not to assume context of Main Loop. But I don't think it's worth to 
> support them. We should focus on high speed solution like linking dpdk.
>
> Another issue is the virtio implementation. Dpdk has virtio library 
> which is pretty optimized, we should consider to use it. But last time 
> I check the code, it was tightly coupled with AF_UNIX transport of 
> vhost-user. We may want to decouple it out of dpdk. Of course we can 
> do it our own as well. (Yes I know there's a vhost-user 
> implementation, but it was not optimized for performance).
>
> I do have some prototype that is based on vhost-scsi-dataplane with a 
> TAP backend. I can send it to you if you wish.
>

Note, what I did is a vhost implementation inside IOThread. It doesn't 
use qemu memory core (which is slow for e.g 10Mpps) but vhost memory table.

Thanks


> Thanks
>
>

Re: [Qemu-devel] Is IOThread for virtio-net a good idea?

[Michael S. Tsirkin]

On Tue, Feb 12, 2019 at 03:00:55PM +0800, Jason Wang wrote:
> 
> On 2019/2/12 下午2:48, Jason Wang wrote:
> > 
> > On 2019/2/11 下午9:40, Anton Kuchin wrote:
> > > As far as I can see currently IOThread offloading is used only for
> > > block devices and all others are emulated by main thread.
> > > 
> > > I expect that network devices can also benefit from processing in
> > > separate thread but I couldn't find any recent work in this
> > > direction. I'm going to implement a PoC but I want to ask if you
> > > know any previous attempts and do you know why it can be a total
> > > waste of time. Are there fundamental obstacles that prevent network
> > > emulation handling in IOThread?
> > > 
> > 
> > I think there're no obstacles.
> > 
> > The only question is whether or not you need to support legacy
> > networking backends. If the answer is yes, you need to convert them not
> > to assume context of Main Loop. But I don't think it's worth to support
> > them. We should focus on high speed solution like linking dpdk.
> > 
> > Another issue is the virtio implementation. Dpdk has virtio library
> > which is pretty optimized, we should consider to use it. But last time I
> > check the code, it was tightly coupled with AF_UNIX transport of
> > vhost-user. We may want to decouple it out of dpdk. Of course we can do
> > it our own as well. (Yes I know there's a vhost-user implementation, but
> > it was not optimized for performance).
> > 
> > I do have some prototype that is based on vhost-scsi-dataplane with a
> > TAP backend. I can send it to you if you wish.
> > 
> 
> Note, what I did is a vhost implementation inside IOThread. It doesn't use
> qemu memory core (which is slow for e.g 10Mpps) but vhost memory table.
> 
> Thanks

If you are going to all these lengths duplicating dpdk I would at least
make it out of process to qemu to improve security.
As an excercise, maybe start by generalizing pxe test to support vhost
user bridge. That would already be a win.


As for IOTLB, I advicated VTD support in vhost for a while.  VTD page
table parsing isn't a lot of code at all if you put invalidations in
userspace.  We ended up with the message-passing instead for portability
but we can go back for sure.



> 
> > Thanks
> > 
> > 

Re: [Qemu-devel] Is IOThread for virtio-net a good idea?

[Jason Wang]

On 2019/2/13 上午2:40, Michael S. Tsirkin wrote:
> On Tue, Feb 12, 2019 at 03:00:55PM +0800, Jason Wang wrote:
>> On 2019/2/12 下午2:48, Jason Wang wrote:
>>> On 2019/2/11 下午9:40, Anton Kuchin wrote:
>>>> As far as I can see currently IOThread offloading is used only for
>>>> block devices and all others are emulated by main thread.
>>>>
>>>> I expect that network devices can also benefit from processing in
>>>> separate thread but I couldn't find any recent work in this
>>>> direction. I'm going to implement a PoC but I want to ask if you
>>>> know any previous attempts and do you know why it can be a total
>>>> waste of time. Are there fundamental obstacles that prevent network
>>>> emulation handling in IOThread?
>>>>
>>> I think there're no obstacles.
>>>
>>> The only question is whether or not you need to support legacy
>>> networking backends. If the answer is yes, you need to convert them not
>>> to assume context of Main Loop. But I don't think it's worth to support
>>> them. We should focus on high speed solution like linking dpdk.
>>>
>>> Another issue is the virtio implementation. Dpdk has virtio library
>>> which is pretty optimized, we should consider to use it. But last time I
>>> check the code, it was tightly coupled with AF_UNIX transport of
>>> vhost-user. We may want to decouple it out of dpdk. Of course we can do
>>> it our own as well. (Yes I know there's a vhost-user implementation, but
>>> it was not optimized for performance).
>>>
>>> I do have some prototype that is based on vhost-scsi-dataplane with a
>>> TAP backend. I can send it to you if you wish.
>>>
>> Note, what I did is a vhost implementation inside IOThread. It doesn't use
>> qemu memory core (which is slow for e.g 10Mpps) but vhost memory table.
>>
>> Thanks
> If you are going to all these lengths duplicating dpdk I would at least
> make it out of process to qemu to improve security.


Well I think I want to do the inverse for easing the access to VM 
metadata e.g IOMMU or RSS.


> As an excercise, maybe start by generalizing pxe test to support vhost
> user bridge. That would already be a win.
>
>
> As for IOTLB, I advicated VTD support in vhost for a while.  VTD page
> table parsing isn't a lot of code at all if you put invalidations in
> userspace.  We ended up with the message-passing instead for portability
> but we can go back for sure.


I think implementing a vendor specific things in vhost is probably not 
good. And as we've discussed, this will end up duplicating the QI 
interface through vhost protocol which doesn't help for performance if 
you move the process out of qemu.

Thanks


>
>
>
>>> Thanks
>>>
>>>