Re: vhost compliant virtio based networking interface in container

[Tetsuya Mukawa <mukawa@igel.co.jp>]

On 2015/09/07 14:54, Xie, Huawei wrote:
> On 8/26/2015 5:23 PM, Tetsuya Mukawa wrote:
>> On 2015/08/25 18:56, Xie, Huawei wrote:
>>> On 8/25/2015 10:59 AM, Tetsuya Mukawa wrote:
>>>> Hi Xie and Yanping,
>>>>
>>>>
>>>> May I ask you some questions?
>>>> It seems we are also developing an almost same one.
>>> Good to know that we are tackling the same problem and have the similar
>>> idea.
>>> What is your status now? We had the POC running, and compliant with
>>> dpdkvhost.
>>> Interrupt like notification isn't supported.
>> We implemented vhost PMD first, so we just start implementing it.
>>
>>>> On 2015/08/20 19:14, Xie, Huawei wrote:
>>>>> Added dev@dpdk.org
>>>>>
>>>>> On 8/20/2015 6:04 PM, Xie, Huawei wrote:
>>>>>> Yanping:
>>>>>> I read your mail, seems what we did are quite similar. Here i wrote a
>>>>>> quick mail to describe our design. Let me know if it is the same thing.
>>>>>>
>>>>>> Problem Statement:
>>>>>> We don't have a high performance networking interface in container for
>>>>>> NFV. Current veth pair based interface couldn't be easily accelerated.
>>>>>>
>>>>>> The key components involved:
>>>>>>     1.    DPDK based virtio PMD driver in container.
>>>>>>     2.    device simulation framework in container.
>>>>>>     3.    dpdk(or kernel) vhost running in host.
>>>>>>
>>>>>> How virtio is created?
>>>>>> A:  There is no "real" virtio-pci device in container environment.
>>>>>> 1). Host maintains pools of memories, and shares memory to container.
>>>>>> This could be accomplished through host share a huge page file to container.
>>>>>> 2). Containers creates virtio rings based on the shared memory.
>>>>>> 3). Container creates mbuf memory pools on the shared memory.
>>>>>> 4) Container send the memory and vring information to vhost through
>>>>>> vhost message. This could be done either through ioctl call or vhost
>>>>>> user message.
>>>>>>
>>>>>> How vhost message is sent?
>>>>>> A: There are two alternative ways to do this.
>>>>>> 1) The customized virtio PMD is responsible for all the vring creation,
>>>>>> and vhost message sending.
>>>> Above is our approach so far.
>>>> It seems Yanping also takes this kind of approach.
>>>> We are using vhost-user functionality instead of using the vhost-net
>>>> kernel module.
>>>> Probably this is the difference between Yanping and us.
>>> In my current implementation, the device simulation layer talks to "user
>>> space" vhost through cuse interface. It could also be done through vhost
>>> user socket. This isn't the key point.
>>> Here vhost-user is kind of confusing, maybe user space vhost is more
>>> accurate, either cuse or unix domain socket. :).
>>>
>>> As for yanping, they are now connecting to vhost-net kernel module, but
>>> they are also trying to connect to "user space" vhost.  Correct me if wrong.
>>> Yes, there is some difference between these two. Vhost-net kernel module
>>> could directly access other process's memory, while using
>>> vhost-user(cuse/user), we need do the memory mapping.
>>>> BTW, we are going to submit a vhost PMD for DPDK-2.2.
>>>> This PMD is implemented on librte_vhost.
>>>> It allows DPDK application to handle a vhost-user(cuse) backend as a
>>>> normal NIC port.
>>>> This PMD should work with both Xie and Yanping approach.
>>>> (In the case of Yanping approach, we may need vhost-cuse)
>>>>
>>>>>> 2) We could do this through a lightweight device simulation framework.
>>>>>>     The device simulation creates simple PCI bus. On the PCI bus,
>>>>>> virtio-net PCI devices are created. The device simulations provides
>>>>>> IOAPI for MMIO/IO access.
>>>> Does it mean you implemented a kernel module?
>>>> If so, do you still need vhost-cuse functionality to handle vhost
>>>> messages n userspace?
>>> The device simulation is  a library running in user space in container. 
>>> It is linked with DPDK app. It creates pseudo buses and virtio-net PCI
>>> devices.
>>> The virtio-container-PMD configures the virtio-net pseudo devices
>>> through IOAPI provided by the device simulation rather than IO
>>> instructions as in KVM.
>>> Why we use device simulation?
>>> We could create other virtio devices in container, and provide an common
>>> way to talk to vhost-xx module.
>> Thanks for explanation.
>> At first reading, I thought the difference between approach1 and
>> approach2 is whether we need to implement a new kernel module, or not.
>> But I understand how you implemented.
>>
>> Please let me explain our design more.
>> We might use a kind of similar approach to handle a pseudo virtio-net
>> device in DPDK.
>> (Anyway, we haven't finished implementing yet, this overview might have
>> some technical problems)
>>
>> Step1. Separate virtio-net and vhost-user socket related code from QEMU,
>> then implement it as a separated program.
>> The program also has below features.
>>  - Create a directory that contains almost same files like
>> /sys/bus/pci/device/<pci address>/*
>>    (To scan these file located on outside sysfs, we need to fix EAL)
>>  - This dummy device is driven by dummy-virtio-net-driver. This name is
>> specified by '<pci addr>/driver' file.
>>  - Create a shared file that represents pci configuration space, then
>> mmap it, also specify the path in '<pci addr>/resource_path'
>>
>> The program will be GPL, but it will be like a bridge on the shared
>> memory between virtio-net PMD and DPDK vhost backend.
>> Actually, It will work under virtio-net PMD, but we don't need to link it.
>> So I guess we don't have GPL license issue.
>>
>> Step2. Fix pci scan code of EAL to scan dummy devices.
>>  - To scan above files, extend pci_scan() of EAL.
>>
>> Step3. Add a new kdrv type to EAL.
>>  - To handle the 'dummy-virtio-net-driver', add a new kdrv type to EAL.
>>
>> Step4. Implement pci_dummy_virtio_net_map/unmap().
>>  - It will have almost same functionality like pci_uio_map(), but for
>> dummy virtio-net device.
>>  - The dummy device will be mmaped using a path specified in '<pci
>> addr>/resource_path'.
>>
>> Step5. Add a new compile option for virtio-net device to replace IO
>> functions.
>>  - The IO functions of virtio-net PMD will be replaced by read() and
>> write() to access to the shared memory.
>>  - Add notification mechanism to IO functions. This will be used when
>> write() to the shared memory is done.
>>  (Not sure exactly, but probably we need it)
>>
>> Does it make sense?
>> I guess Step1&2 is different from your approach, but the rest might be
>> similar.
>>
>> Actually, we just need sysfs entries for a virtio-net dummy device, but
>> so far, I don't have a fine way to register them from user space without
>> loading a kernel module.
> Tetsuya:
> I don't quite get the details. Who will create those sysfs entries? A
> kernel module right?

Hi Xie,

I don't create sysfs entries. Just create a directory that contains
files looks like sysfs entries.
And initialize EAL with not only sysfs but also the above directory.

In quoted last sentence, I wanted to say we just needed files looks like
sysfs entries.
But I don't know a good way to create files under sysfs without loading
kernel module.
This is because I try to create the additional directory.

> The virtio-net is configured through read/write to sharing
> memory(between host and guest), right?

Yes, I agree.

> Where is shared vring created and shared memory created, on shared huge
> page between host and guest?

The vritqueues(vrings) are on guest hugepage.

Let me explain.
Guest container should have read/write access to a part of hugepage
directory on host.
(For example, /mnt/huge/conainer1/ is shared between host and guest.)
Also host and guest needs to communicate through a unix domain socket.
(For example, host and guest can communicate with using
"/tmp/container1/sock")

If we can do like above, a virtio-net PMD on guest can creates
virtqueues(vrings) on it's hugepage, and writes these information to a
pseudo virtio-net device that is a process created in guest container.
Then the pseudo virtio-net device sends it to vhost-user backend(host
DPDK application) through a unix domain socket.

So with my plan, there are 3 processes.
DPDK applications on host and guest, also a process that works like
virtio-net device.

> Who will talk to dpdkvhost?

If we need to talk to a cuse device or the vhost-net kernel module, an
above pseudo virtio-net device could talk to.
(But, so far, my target is only vhost-user.)

>> This is because I need to change pci_scan() also.
>>
>> It seems you have implemented a virtio-net pseudo device as BSD license.
>> If so, this kind of PMD would be nice to use it.
> Currently it is based on native linux kvm tool.

Great, I hadn't noticed this option.

>> In the case that it takes much time to implement some lost
>> functionalities like interrupt mode, using QEMU code might be an one of
>> options.
> For interrupt mode, i plan to use eventfd for sleep/wake, have not tried
> yet.
>> Anyway, we just need a fine virtual NIC between containers and host.
>> So we don't hold to our approach and implementation.
> Do you have comments to my implementation?
> We could publish the version without the device framework first for
> reference.

No I don't have. Could you please share it?
I am looking forward to seeing it.

Tetsuya