Potential hang in virtnet_send_command

Potential hang in virtnet_send_command

[Christian Ehrhardt]

[-- Attachment #1.1: Type: text/plain, Size: 1373 bytes --]

Hi,
I was debugging an issue caused by a bad usage of dpdk.
That is fixed in the meantime and I'll backport that into our code as well.

But along debugging that, I found a potential hang in virtnet_send_command
that my case ran into.
The following code can become an infinite loop:
/* Spin for a response, the kick causes an ioport write, trapping
 * into the hypervisor, so the request should be handled immediately.
 */
while (!virtqueue_get_buf(vi->cvq, &tmp) &&
       !virtqueue_is_broken(vi->cvq))
        cpu_relax();

In my case dpdk broke something - not exactly clear what - and due to that
following calls through virtnet_send_command ran into this hang.
Effectively it seems that the buffers didn't get refreshed at all anymore.

That said the dpdk issue to touch devices that belong to a kernel owned
driver is fixed, so one could leave the code as is for now.
Yet I wanted to make you aware in case you would vote for a time or retry
based upper limit on that loop to avoid hangs - who knows what else might
bring it in this broken state in a different case.

Kind Regards,
Christian Ehrhardt

P.S.
Steps to reproduce, backtraces and more data can be found in:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1570195
A general setup howto for DPDK in KVM guests which is a prereq is at:
https://help.ubuntu.com/16.04/serverguide/DPDK.html#dpdk-in-guest

[-- Attachment #1.2: Type: text/html, Size: 1864 bytes --]

[-- Attachment #2: Type: text/plain, Size: 183 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Potential hang in virtnet_send_command

[Michael S. Tsirkin]

On Mon, Apr 18, 2016 at 07:07:32PM +0200, Christian Ehrhardt wrote:
> Hi,
> I was debugging an issue caused by a bad usage of dpdk.
> That is fixed in the meantime and I'll backport that into our code as well.
> 
> But along debugging that, I found a potential hang in virtnet_send_command that
> my case ran into.
> The following code can become an infinite loop:
> /* Spin for a response, the kick causes an ioport write, trapping
>  * into the hypervisor, so the request should be handled immediately.
>  */   
> while (!virtqueue_get_buf(vi->cvq, &tmp) &&   
>        !virtqueue_is_broken(vi->cvq))   
>         cpu_relax();
> 
> In my case dpdk broke something - not exactly clear what - and due to that
> following calls through virtnet_send_command ran into this hang.
> Effectively it seems that the buffers didn't get refreshed at all anymore.
> 
> That said the dpdk issue to touch devices that belong to a kernel owned driver
> is fixed, so one could leave the code as is for now.
> Yet I wanted to make you aware in case you would vote for a time or retry based
> upper limit on that loop to avoid hangs - who knows what else might bring it in
> this broken state in a different case.
> 
> Kind Regards,
> Christian Ehrhardt
> 
> P.S.
> Steps to reproduce, backtraces and more data can be found in:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1570195
> A general setup howto for DPDK in KVM guests which is a prereq is at:
> https://help.ubuntu.com/16.04/serverguide/DPDK.html#dpdk-in-guest

True. It's not that clear how to handle this cleanly though.
Ideally we'd have to transmit the next command once the
previous one completes.

-- 
MST