[Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Michael S. Tsirkin]

Reply-To: 

Thinking about the vhost-user protocol, VHOST_SET_MEM_TABLE
is used to update the memory mappings.

So shouldn't we want for response?
Otherwise e.g. guest can start using the memory
that vhost-user can't access.

Similarly, with an IOMMU vhost-user might access memory it shouldn't.

VHOST_SET_VRING_CALL is used for MSI-X masking.
Again, after vector is masted by switching the call fd,
backend shouldn't assert the old one.

Thoughts?

Re: [Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Linhaifeng]

I think maybe is not need for the backend to wait for response.

There is another way.vhost-user send "VHOST_GET_MEM_TABLE" to qemu then qemu send VHOST_SET_MEM_TABLE to update the regions of vhost-user.same as other command.
If qemu could response the request of the vhost-user.the vhost-user could update date at anytime.

I think it's very useful for Commercialization.

On 2014/9/17 16:38, Michael S. Tsirkin wrote:
> Reply-To: 
> 
> Thinking about the vhost-user protocol, VHOST_SET_MEM_TABLE
> is used to update the memory mappings.
> 
> So shouldn't we want for response?
> Otherwise e.g. guest can start using the memory
> that vhost-user can't access.
> 
> Similarly, with an IOMMU vhost-user might access memory it shouldn't.
> 
> VHOST_SET_VRING_CALL is used for MSI-X masking.
> Again, after vector is masted by switching the call fd,
> backend shouldn't assert the old one.
> 
> Thoughts?
> 
> 

Re: [Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Michael S. Tsirkin]

On Wed, Sep 17, 2014 at 05:39:04PM +0800, Linhaifeng wrote:
> I think maybe is not need for the backend to wait for response.
> 
> There is another way.vhost-user send "VHOST_GET_MEM_TABLE" to qemu then qemu send VHOST_SET_MEM_TABLE to update the regions of vhost-user.same as other command.
> If qemu could response the request of the vhost-user.the vhost-user could update date at anytime.

The updates are initiated by QEMU, as a result of IOMMU,
memory hotplug or some other configuration change.

> I think it's very useful for Commercialization.
> 
> On 2014/9/17 16:38, Michael S. Tsirkin wrote:
> > Reply-To: 
> > 
> > Thinking about the vhost-user protocol, VHOST_SET_MEM_TABLE
> > is used to update the memory mappings.
> > 
> > So shouldn't we want for response?
> > Otherwise e.g. guest can start using the memory
> > that vhost-user can't access.
> > 
> > Similarly, with an IOMMU vhost-user might access memory it shouldn't.
> > 
> > VHOST_SET_VRING_CALL is used for MSI-X masking.
> > Again, after vector is masted by switching the call fd,
> > backend shouldn't assert the old one.
> > 
> > Thoughts?
> > 
> > 

Re: [Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Linhaifeng]

On 2014/9/17 17:56, Michael S. Tsirkin wrote:
> On Wed, Sep 17, 2014 at 05:39:04PM +0800, Linhaifeng wrote:
>> I think maybe is not need for the backend to wait for response.
>>
>> There is another way.vhost-user send "VHOST_GET_MEM_TABLE" to qemu then qemu send VHOST_SET_MEM_TABLE to update the regions of vhost-user.same as other command.
>> If qemu could response the request of the vhost-user.the vhost-user could update date at anytime.
> 
> The updates are initiated by QEMU, as a result of IOMMU,
> memory hotplug or some other configuration change.

How to deal with the vhost-user restart?
when vhost-user restart it will lost the infomation which QEMU send.

In the kernel mode vhost will restart with QEMU but in the user mode vhost will not.

> 
>> I think it's very useful for Commercialization.
>>
>> On 2014/9/17 16:38, Michael S. Tsirkin wrote:
>>> Reply-To: 
>>>
>>> Thinking about the vhost-user protocol, VHOST_SET_MEM_TABLE
>>> is used to update the memory mappings.
>>>
>>> So shouldn't we want for response?
>>> Otherwise e.g. guest can start using the memory
>>> that vhost-user can't access.
>>>
>>> Similarly, with an IOMMU vhost-user might access memory it shouldn't.
>>>
>>> VHOST_SET_VRING_CALL is used for MSI-X masking.
>>> Again, after vector is masted by switching the call fd,
>>> backend shouldn't assert the old one.
>>>
>>> Thoughts?
>>>
>>>
> 
> .
> 

Re: [Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Linhaifeng]

On 2014/9/17 17:56, Michael S. Tsirkin wrote:
> On Wed, Sep 17, 2014 at 05:39:04PM +0800, Linhaifeng wrote:
>> I think maybe is not need for the backend to wait for response.
>>
>> There is another way.vhost-user send "VHOST_GET_MEM_TABLE" to qemu then qemu send VHOST_SET_MEM_TABLE to update the regions of vhost-user.same as other command.
>> If qemu could response the request of the vhost-user.the vhost-user could update date at anytime.
> 
> The updates are initiated by QEMU, as a result of IOMMU,
> memory hotplug or some other configuration change.

yes.when memory unplug/plug should tell vhost-user.

> 
>> I think it's very useful for Commercialization.
>>
>> On 2014/9/17 16:38, Michael S. Tsirkin wrote:
>>> Reply-To: 
>>>
>>> Thinking about the vhost-user protocol, VHOST_SET_MEM_TABLE
>>> is used to update the memory mappings.
>>>
>>> So shouldn't we want for response?
>>> Otherwise e.g. guest can start using the memory
>>> that vhost-user can't access.
>>>
>>> Similarly, with an IOMMU vhost-user might access memory it shouldn't.
>>>
>>> VHOST_SET_VRING_CALL is used for MSI-X masking.
>>> Again, after vector is masted by switching the call fd,
>>> backend shouldn't assert the old one.
>>>
>>> Thoughts?
>>>
>>>
> 
> .
> 

Re: [Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Michael S. Tsirkin]

On Thu, Sep 18, 2014 at 08:45:37AM +0800, Linhaifeng wrote:
> 
> 
> On 2014/9/17 17:56, Michael S. Tsirkin wrote:
> > On Wed, Sep 17, 2014 at 05:39:04PM +0800, Linhaifeng wrote:
> >> I think maybe is not need for the backend to wait for response.
> >>
> >> There is another way.vhost-user send "VHOST_GET_MEM_TABLE" to qemu then qemu send VHOST_SET_MEM_TABLE to update the regions of vhost-user.same as other command.
> >> If qemu could response the request of the vhost-user.the vhost-user could update date at anytime.
> > 
> > The updates are initiated by QEMU, as a result of IOMMU,
> > memory hotplug or some other configuration change.
> 
> How to deal with the vhost-user restart?
> when vhost-user restart it will lost the infomation which QEMU send.
> 
> In the kernel mode vhost will restart with QEMU but in the user mode vhost will not.

vhost-user must restart with qemu only.

The nature of virtio protocol is such that there's not enough in-memory
state for host to gracefully recover from losing VQ state.

We could add a new feature to allow recovery by reporting
failure to guest, and disabling processing new requests.
Guest could respond by recovering / discarding submitted
requests, re-enabling the device (likely by executing a reset)
and re-submitting requests.


This would need a new feature bit though, and would have to
be acknowledged by guest. As such this would have to be
virtio 1.0 feature, virtio 0.x is frozen.

> > 
> >> I think it's very useful for Commercialization.
> >>
> >> On 2014/9/17 16:38, Michael S. Tsirkin wrote:
> >>> Reply-To: 
> >>>
> >>> Thinking about the vhost-user protocol, VHOST_SET_MEM_TABLE
> >>> is used to update the memory mappings.
> >>>
> >>> So shouldn't we want for response?
> >>> Otherwise e.g. guest can start using the memory
> >>> that vhost-user can't access.
> >>>
> >>> Similarly, with an IOMMU vhost-user might access memory it shouldn't.
> >>>
> >>> VHOST_SET_VRING_CALL is used for MSI-X masking.
> >>> Again, after vector is masted by switching the call fd,
> >>> backend shouldn't assert the old one.
> >>>
> >>> Thoughts?
> >>>
> >>>
> > 
> > .
> > 

Re: [Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Michael S. Tsirkin]

On Thu, Sep 18, 2014 at 10:39:00AM +0800, Linhaifeng wrote:
> 
> 
> On 2014/9/17 17:56, Michael S. Tsirkin wrote:
> > On Wed, Sep 17, 2014 at 05:39:04PM +0800, Linhaifeng wrote:
> >> I think maybe is not need for the backend to wait for response.
> >>
> >> There is another way.vhost-user send "VHOST_GET_MEM_TABLE" to qemu then qemu send VHOST_SET_MEM_TABLE to update the regions of vhost-user.same as other command.
> >> If qemu could response the request of the vhost-user.the vhost-user could update date at anytime.
> > 
> > The updates are initiated by QEMU, as a result of IOMMU,
> > memory hotplug or some other configuration change.
> 
> yes.when memory unplug/plug should tell vhost-user.

Right. So VHOST_SET_MEM_TABLE needs a response then, at least
for cases when it's freeing memory.

When memory is added, we can avoid waiting for the response
if vhost-user can detect a fault and flush the incoming
command pipe looking for VHOST_SET_MEM_TABLE.


> > 
> >> I think it's very useful for Commercialization.
> >>
> >> On 2014/9/17 16:38, Michael S. Tsirkin wrote:
> >>> Reply-To: 
> >>>
> >>> Thinking about the vhost-user protocol, VHOST_SET_MEM_TABLE
> >>> is used to update the memory mappings.
> >>>
> >>> So shouldn't we want for response?
> >>> Otherwise e.g. guest can start using the memory
> >>> that vhost-user can't access.
> >>>
> >>> Similarly, with an IOMMU vhost-user might access memory it shouldn't.
> >>>
> >>> VHOST_SET_VRING_CALL is used for MSI-X masking.
> >>> Again, after vector is masted by switching the call fd,
> >>> backend shouldn't assert the old one.
> >>>
> >>> Thoughts?
> >>>
> >>>
> > 
> > .
> > 

Re: [Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Linhaifeng]

On 2014/9/18 13:16, Michael S. Tsirkin wrote:
> On Thu, Sep 18, 2014 at 08:45:37AM +0800, Linhaifeng wrote:
>>
>>
>> On 2014/9/17 17:56, Michael S. Tsirkin wrote:
>>> On Wed, Sep 17, 2014 at 05:39:04PM +0800, Linhaifeng wrote:
>>>> I think maybe is not need for the backend to wait for response.
>>>>
>>>> There is another way.vhost-user send "VHOST_GET_MEM_TABLE" to qemu then qemu send VHOST_SET_MEM_TABLE to update the regions of vhost-user.same as other command.
>>>> If qemu could response the request of the vhost-user.the vhost-user could update date at anytime.
>>>
>>> The updates are initiated by QEMU, as a result of IOMMU,
>>> memory hotplug or some other configuration change.
>>
>> How to deal with the vhost-user restart?
>> when vhost-user restart it will lost the infomation which QEMU send.
>>
>> In the kernel mode vhost will restart with QEMU but in the user mode vhost will not.
> 
> vhost-user must restart with qemu only.
> 


Sometimes qemu not allowed to restart. e.g. The customer want to update the vhost-user to a newer version but don't want to restart the VM.


> The nature of virtio protocol is such that there's not enough in-memory
> state for host to gracefully recover from losing VQ state.
> 
> We could add a new feature to allow recovery by reporting
> failure to guest, and disabling processing new requests.
> Guest could respond by recovering / discarding submitted
> requests, re-enabling the device (likely by executing a reset)
> and re-submitting requests.
> 
> 
> This would need a new feature bit though, and would have to
> be acknowledged by guest. As such this would have to be
> virtio 1.0 feature, virtio 0.x is frozen.
> 
>>>
>>>> I think it's very useful for Commercialization.
>>>>
>>>> On 2014/9/17 16:38, Michael S. Tsirkin wrote:
>>>>> Reply-To: 
>>>>>
>>>>> Thinking about the vhost-user protocol, VHOST_SET_MEM_TABLE
>>>>> is used to update the memory mappings.
>>>>>
>>>>> So shouldn't we want for response?
>>>>> Otherwise e.g. guest can start using the memory
>>>>> that vhost-user can't access.
>>>>>
>>>>> Similarly, with an IOMMU vhost-user might access memory it shouldn't.
>>>>>
>>>>> VHOST_SET_VRING_CALL is used for MSI-X masking.
>>>>> Again, after vector is masted by switching the call fd,
>>>>> backend shouldn't assert the old one.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>>
>>>
>>> .
>>>
> 
> .
> 

Re: [Qemu-devel] vhost-user: VHOST_SET_MEM_TABLE, VHOST_SET_VRING_CALL need a reply?

[Michael S. Tsirkin]

On Thu, Sep 18, 2014 at 08:39:57PM +0800, Linhaifeng wrote:
> 
> 
> On 2014/9/18 13:16, Michael S. Tsirkin wrote:
> > On Thu, Sep 18, 2014 at 08:45:37AM +0800, Linhaifeng wrote:
> >>
> >>
> >> On 2014/9/17 17:56, Michael S. Tsirkin wrote:
> >>> On Wed, Sep 17, 2014 at 05:39:04PM +0800, Linhaifeng wrote:
> >>>> I think maybe is not need for the backend to wait for response.
> >>>>
> >>>> There is another way.vhost-user send "VHOST_GET_MEM_TABLE" to qemu then qemu send VHOST_SET_MEM_TABLE to update the regions of vhost-user.same as other command.
> >>>> If qemu could response the request of the vhost-user.the vhost-user could update date at anytime.
> >>>
> >>> The updates are initiated by QEMU, as a result of IOMMU,
> >>> memory hotplug or some other configuration change.
> >>
> >> How to deal with the vhost-user restart?
> >> when vhost-user restart it will lost the infomation which QEMU send.
> >>
> >> In the kernel mode vhost will restart with QEMU but in the user mode vhost will not.
> > 
> > vhost-user must restart with qemu only.
> > 
> 
> 
> Sometimes qemu not allowed to restart. e.g. The customer want to update the vhost-user to a newer version but don't want to restart the VM.

I guess this means we'll have to implement the idea outlined below.
Fixing known issues around vhost-user (such as lack of set_backend, and
lack of synchronization around some commands) is probably higher
priority.

> 
> > The nature of virtio protocol is such that there's not enough in-memory
> > state for host to gracefully recover from losing VQ state.
> > 
> > We could add a new feature to allow recovery by reporting
> > failure to guest, and disabling processing new requests.
> > Guest could respond by recovering / discarding submitted
> > requests, re-enabling the device (likely by executing a reset)
> > and re-submitting requests.
> > 
> > 
> > This would need a new feature bit though, and would have to
> > be acknowledged by guest. As such this would have to be
> > virtio 1.0 feature, virtio 0.x is frozen.
> > 
> >>>
> >>>> I think it's very useful for Commercialization.
> >>>>
> >>>> On 2014/9/17 16:38, Michael S. Tsirkin wrote:
> >>>>> Reply-To: 
> >>>>>
> >>>>> Thinking about the vhost-user protocol, VHOST_SET_MEM_TABLE
> >>>>> is used to update the memory mappings.
> >>>>>
> >>>>> So shouldn't we want for response?
> >>>>> Otherwise e.g. guest can start using the memory
> >>>>> that vhost-user can't access.
> >>>>>
> >>>>> Similarly, with an IOMMU vhost-user might access memory it shouldn't.
> >>>>>
> >>>>> VHOST_SET_VRING_CALL is used for MSI-X masking.
> >>>>> Again, after vector is masted by switching the call fd,
> >>>>> backend shouldn't assert the old one.
> >>>>>
> >>>>> Thoughts?
> >>>>>
> >>>>>
> >>>
> >>> .
> >>>
> > 
> > .
> >