[vhost-user] Expectations around reconnection

[vhost-user] Expectations around reconnection

[Boeuf, Sebastien]

Hi All,

In the context of vhost-user, I was wondering how a reconnection should
be handled from the VMM perspective?

In particular, I'm looking at the OVS-DPDK use case using the client 
mode (meaning QEMU acts as the server), and I'd like to understand what 
QEMU does to handle this. Upon disconnection from the backend, does QEMU
 reset the virtio device (meaning the guest
is notified about it)? And upon the new connection from the backend, 
does QEMU go through the whole vhost-user initialization once again 
(feature acknowledgement, setup of vrings, etc...), or does it simply 
assume the backend will have saved all these information?

Thanks for your time,
Sebastien
---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [vhost-user] Expectations around reconnection

[Philippe Mathieu-Daudé]

Cc'ing few developers

On 5/11/21 3:33 PM, Boeuf, Sebastien wrote:
> Hi All,
> 
> In the context of vhost-user, I was wondering how a reconnection should
> be handled from the VMM perspective?
> 
> In particular, I'm looking at the OVS-DPDK use case using the client 
> mode (meaning QEMU acts as the server), and I'd like to understand what 
> QEMU does to handle this. Upon disconnection from the backend, does QEMU
>  reset the virtio device (meaning the guest
> is notified about it)? And upon the new connection from the backend, 
> does QEMU go through the whole vhost-user initialization once again 
> (feature acknowledgement, setup of vrings, etc...), or does it simply 
> assume the backend will have saved all these information?
> 
> Thanks for your time,
> Sebastien

Re: [vhost-user] Expectations around reconnection

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2353 bytes --]

On Thu, May 13, 2021 at 11:43:25AM +0200, Philippe Mathieu-Daudé wrote:
> Cc'ing few developers
> 
> On 5/11/21 3:33 PM, Boeuf, Sebastien wrote:
> > Hi All,
> > 
> > In the context of vhost-user, I was wondering how a reconnection should
> > be handled from the VMM perspective?
> > 
> > In particular, I'm looking at the OVS-DPDK use case using the client 
> > mode (meaning QEMU acts as the server), and I'd like to understand what 
> > QEMU does to handle this. Upon disconnection from the backend, does QEMU
> >  reset the virtio device (meaning the guest
> > is notified about it)? And upon the new connection from the backend, 
> > does QEMU go through the whole vhost-user initialization once again 
> > (feature acknowledgement, setup of vrings, etc...), or does it simply 
> > assume the backend will have saved all these information?

I started a virtio-fs-specific email thread about vhost-user
reconnection here after your IRC messages the other day:
https://listman.redhat.com/archives/virtio-fs/2021-May/msg00105.html

The VMM does not reset the device. In general vhost-user reconnection is
transparent to the guest. While the device is disconnected the
virtqueues are not processed.

Upon reconnection the vhost-user protocol traffic is almost identical to
a fresh connection. The VMM negotiates features, send memory regions,
etc.

The device backend needs to persist device-specific state. This is why
reconnection and crash recovery are device-specific (and to some extent
implementation-specific because not every device may support it or
implement it in the same way). Stateless devices are easiest to support.

vhost-user-net and vhost-user-blk are the only devices I'm aware of that
support reconnection today.

Expect to encounter bugs. Reconnection is underspecified and leaves a
lot to the vhost-user implementation. There might also be design flaws
around synchronous VIRTIO transport hardware registers where either the
vCPU hangs because it needs a response from the disconnected device or
the VMM ignores the disconnected device in order to avoid hanging the
vCPU thread but the state is out of sync upon reconnection. It's hard to
test all possible states so I doubt it's bullet-proof. It probably works
best for vhost-user-net and less well for other device types.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]