Re: [Qemu-devel] Insane virtio-serial semantics

[Michael Roth <mdroth@linux.vnet.ibm.com>]

On 12/07/2011 07:49 AM, Anthony Liguori wrote:
> On 12/07/2011 02:21 AM, Markus Armbruster wrote:
>> Anthony Liguori<anthony@codemonkey.ws> writes:
>>
>>> On 12/06/2011 04:30 PM, Lluís Vilanova wrote:
>>>> Anthony Liguori writes:
>>>>
>>>>> I really worry about us introducing so many of these one-off
>>>>> paravirtual devices.
>>>>> I would much prefer that you look at doing this as an extension to
>>>>> the ivshmem
>>>>> device as it already has this sort of scope. You should be able to
>>>>> do this by
>>>>> just extending the size of bar 1 and using a well known guest id.
>>>>
>>>> I did in fact look at ivshmem some time ago, and it's true that both
>>>> use the
>>>> same mechanisms; but each device has a completely different purpose.
>>>> To me it
>>>> just seems that extending the control BAR in ivshmem to call the
>>>> user-provided
>>>> backdoor callbacks is just conflating two completely separate
>>>> devices into a
>>>> single one. Besides, I think that the qemu-side of the backdoor is
>>>> simple enough
>>>> to avoid being a maintenance burden.
>>>
>>> They have the same purpose (which are both vague TBH). The only
>>> reason I'm sympathetic to this device is that virtio-serial has such
>>> insane semantics.
>>
>> Could you summarize what's wrong? Is it fixable?
>
> I don't think so as it's part of the userspace ABI now.
>
> Mike, please help me make sure I get this all right. A normal
> file/socket has the following guest semantics:
>
> 1) When a disconnect occurs, you will receive a return of '0' or -EPIPE
> depending on the platform. The fd is now unusable and you must
> close/reopen.
>
> 2) You can setup SIGIO/SIGPIPE to fire off whenever a file descriptor
> becomes readable/writable.
>
> virtio serial has the following semantics:
>
> 1) When a disconnect occurs, if you read() you will receive an -EPIPE.
>
> 2) However, if a reconnect occurs before you issue your read(), the read
> will complete with no indication that a disconnect occurred.
>
> 3) This makes it impossible to determine whether a disconnect has
> occurred which makes it very hard to reset your protocol stream. To deal
> with this, virtio-serial can issue a SIGIO signal upon disconnect.
>
> 4) Signals are asynchronous, so a reconnect may have occurred by the
> time you get the SIGIO signal. It's unclear that you can do anything
> useful with this.

That about sums it up. There was a thread about this a while back where 
there was some tentative agreement on a way to fix this by introducing 
QEMU flags that invoke similar semantics to unix sockets:

http://thread.gmane.org/gmane.comp.emulators.qemu/94721/focus=95496

But at this point we'd need to re-visit, since there's a fair number of 
virtio-serial users now. It'd probably need to be something you could 
switch on from the guest via an fcntl() or something.

>
> So besides overloading the meaning of SIGIO, there's really no way to
> figure out in the guest when a reconnect has occurred. To deal with this
> in qemu-ga, we actually only allow 7-bit data transfers and use the 8th
> bit as an in-band message to tell the guest that a reset has occurred.

Yup, it's not perfect though, due to a delayed/spurious response from an 
agent that sent data before it read/handled the reset sequence. We don't 
get that problem with unix sockets since they'd get an -EPIPE and would 
be blocked from sending to a newly opened session.

We try to account for this on the host by following up a reset sequences 
will the guest-sync RPC, which contains a unique ID that the guest echos 
back to us. That way we can throw away stale data on the host until we 
get the intended response. In our case, it's not quite perfect since if 
the agent sent a "{" before getting reset, subsequent transmission of 
the guest-sync response can be lost. We'd need to precede responses to 
guest-sync with a 0xFF as well, so that the host flushes it's rcv 
buffer/parser state...

And, somewhat off-topic, but none of addresses the case where an agent 
hangs on an RPC. This would require some additional handling by the 
agent side where we might have tie some additional action to the 0xFF 
sequence.

Previously this scenario was handled by a hard-coded timeout mechanism 
in the agent, with a seperate thread handling the RPCs, but we've since 
dropped the thread due to potential for memory leaks (with plans to 
re-introduce using a child process).

client-induced resets would be much nicer though, and a reserved byte is 
the best solution we've been able to come up with given the current 
virtio-serial semantics.

>
> Regards,
>
> Anthony Liguori
>
>>
>> [...]
>>
>