Re: [PATCH 2/3] net/virtio_user: fix wrong sequence of messages

[Yuanhan Liu <yuanhan.liu@linux.intel.com>]

On Fri, Sep 09, 2016 at 11:59:18AM +0800, Tan, Jianfeng wrote:
>                 It's actually a good catch. After a light thought, I think in DPDK vhost, we
>                 may need to create those virtqueues once unix socket gets connected, just
>                 like in vhost-net, virtqueues are created on char file open. Right?
> 
>             There is a difference: for vhost-net and tap mode, IIRC, it knows how
>             many queues before doing setup.
> 
>         No, from linux/drivers/vhost/net.c:vhost_net_open(), we can see that
>         virtqueues are allocated according to VHOST_NET_VQ_MAX.
> 
>     Well, if you took a closer look, you will find VHOST_NET_VQ_MAX is
>     defined to 2. That means it allocates a queue-pair only.
> 
>     And FYI, the MQ support for vhost-net is actually done in the tap
>     driver, but not in vhost-net driver. That results to the MQ
>     implementation is a bit different between vhost-net and vhost-user.
> 
>     Put simply, in vhost-net, one queue-pair has a backend fd associated
>     with it. Vhost requests for different queue-pair are sent by different
>     fd. That also means the vhost-net doesn't even need be aware of the
>     MQ stuff.
> 
>     However, for vhost-user implementation, all queue-pairs share one
>     socket fd. All requests all also sent over the single socket fd,
>     thus the backend (DPDK vhost) has to figure out how many queue
>     pairs are actually enabled: and we detect it by reading the
>     vring index of SET_VRING_CALL message; it's not quite right though.
> 
> 
> Aha, right, nice analysis.

Just some rough memory in my mind ;-)
> 
> 
> 
>         How about reconsidering previous suggestion to allocate vq once connection
>         is established?
> 
>     That's too much, because DPDK claims to support up to 0x8000
>     queue-pairs. Don't even to say that the vhost_virtqueue struct
>     was way too big before: it even holds an array of buf_vec with
>     size 256.
> 
> 
> Another mistake of my memory, I was remember it wrongly as only 8 VQs are
> supported.
> One thing not related, provided that VHOST_MAX_QUEUE_PAIRS equals to 0x8000,
> struct vhost_virtqueue  *virtqueue[VHOST_MAX_QUEUE_PAIRS * 2] spends 4MB for
> each virtio device, which could be a refined.

Yes, we could allocate a small array first, and then re-allocate if
necessary.

> 
> 
> 
>         Never mind, above fix on the vhost side will not take effect on existing
>         vpp-vhost implementations.
> 
>     Actually, I was talking about the DPDK vhost implementation :)
> 
> 
> This patch is talking about vpp's native vhost implementation, not dpdk-vhost,
> and not the way vpp uses dpdk-vhost.

Yes, I know. What I meant is there was a "workaround" in DPDK vhost
implementation, and since you bring this issue on the table again,
it's a chance to think about how can we fix it.

A rough idea come to my mind is we could check all the per-vring message
at the begining of vhost_user_msg_handler() and allocate related vq when
necessary (when it's the first vring message we got).

Yeah, I know it's a bit ugly, but it at least gets rid of that "not-that-true"
assumption.

> 
>         Still not working. VPP needs SET_VRING_CALL to create vq firstly.
> 
>     Didn't get it. In the proposal, SET_FEATURES is sent before every other
>     messages, thus it should not cause the issue you described in this patch.
> 
> 
> OK. Let me try to explain. We take three vhost implementations into
> consideration: dpdk-2.2-vhost, dpdk-master-vhost, vpp-native-vhost.
> 
> If set_feature before set_vring_call, dpdk-2.2-vhost will fail: inside
> set_feature handler, assigning header length to VQs which will be created in
> set_vring_call handler.

Oh, right. That was an in-correct implementation.

> So we need to keep set_vring_call firstly.
> Then set_feature needs to be sent
> before any other msgs, this is what vpp-native-vhost requires. In all, the
> sequence is like this:
> 1. set_vring_call,
> 2. set_feature,
> 3. other msgs
> 
> 
>     Besides, haven't we already sent SET_VRING_CALL before other messages
>     (well, execept the SET_FEATURES and SET_MEM_TABLE message)?
> 
> 
> Yes, set_vring_call is already in the first place, but we need to plugin
> set_feature between set_vring_call and other msgs. Previously, set_vring_call
> and other msgs are together.

Okay. Another thing I noticed is that virtio-user lacks some feature
negotiations, like GET_FEATURES and GET_PROTOCOL_FEATURES. I think you
might need add them back somewhen?

	--yliu