[Qemu-devel] virtio-scsi and error handling

[Qemu-devel] virtio-scsi and error handling

[Hannes Reinecke]

Hi Stefan,

I currently playing around with improving SCSI EH, optimizing
command aborts and the like.

And, supposing it to be a nice testbed, tried to make things work
with virtio_scsi.

However, looking at the code there I've found virtscsi_tmf() just
uses 'wait_for_completion', with no timeout specified. So in effect
any abort might stall forever.

Wouldn't it be more sensible to use 'wait_for_completion_timeout'
here, to allow the error escalation to continue?
This would especially be useful when running with multipathing,
as the underlying device might stall, and aio_cancel() doesn't work
reliably, if at all.

Also I've found that there is no host reset. Currently the virtio
semantics seem to require reliable communication, ie for every
command send there _has_ to be a response.

Long and painful experience with RAID HBAs has shown that this model
works okay for the lower-level escalations, but you absolutely need
a host reset to restore communication.
In the case of virtio I would think that a virtio-level reset for
host_reset would be a sensible idea.

Any opinions from your side?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: [Qemu-devel] virtio-scsi and error handling

[Stefan Hajnoczi]

On Tue, Jun 11, 2013 at 01:41:38PM +0200, Hannes Reinecke wrote:
> I currently playing around with improving SCSI EH, optimizing
> command aborts and the like.
> 
> And, supposing it to be a nice testbed, tried to make things work
> with virtio_scsi.
> 
> However, looking at the code there I've found virtscsi_tmf() just
> uses 'wait_for_completion', with no timeout specified. So in effect
> any abort might stall forever.
> 
> Wouldn't it be more sensible to use 'wait_for_completion_timeout'
> here, to allow the error escalation to continue?
> This would especially be useful when running with multipathing,
> as the underlying device might stall, and aio_cancel() doesn't work
> reliably, if at all.

Hi,
I agree that we need a timeout.  bdrv_aio_cancel() is not guaranteed to
complete in bounded time.

> Also I've found that there is no host reset. Currently the virtio
> semantics seem to require reliable communication, ie for every
> command send there _has_ to be a response.
> 
> Long and painful experience with RAID HBAs has shown that this model
> works okay for the lower-level escalations, but you absolutely need
> a host reset to restore communication.
> In the case of virtio I would think that a virtio-level reset for
> host_reset would be a sensible idea.

One thing to watch out for is that a virtio-scsi reset will likely hang
too because it resets all pending requests.

Paolo Bonzini has done the lion's share of virtio-scsi work over the
past year (or two?).  He might have some more thoughts.

Stefan

Re: [Qemu-devel] virtio-scsi and error handling

[Paolo Bonzini]

Il 12/06/2013 03:56, Stefan Hajnoczi ha scritto:
> On Tue, Jun 11, 2013 at 01:41:38PM +0200, Hannes Reinecke wrote:
>> I currently playing around with improving SCSI EH, optimizing
>> command aborts and the like.
>>
>> And, supposing it to be a nice testbed, tried to make things work
>> with virtio_scsi.
>>
>> However, looking at the code there I've found virtscsi_tmf() just
>> uses 'wait_for_completion', with no timeout specified. So in effect
>> any abort might stall forever.
>>
>> Wouldn't it be more sensible to use 'wait_for_completion_timeout'
>> here, to allow the error escalation to continue?
>> This would especially be useful when running with multipathing,
>> as the underlying device might stall, and aio_cancel() doesn't work
>> reliably, if at all.
> 
> Hi,
> I agree that we need a timeout.  bdrv_aio_cancel() is not guaranteed to
> complete in bounded time.

I also agree that we need a timeout, but then note that host reset could
also not complete in bounded time if I/O doesn't terminate in the host.

Last time I checked the io_cancel system call was basically a no-op (for
aio=native), and for aio=threads the worker might stay in D state for an
unbounded time too.

Paolo

>> Also I've found that there is no host reset. Currently the virtio
>> semantics seem to require reliable communication, ie for every
>> command send there _has_ to be a response.
>>
>> Long and painful experience with RAID HBAs has shown that this model
>> works okay for the lower-level escalations, but you absolutely need
>> a host reset to restore communication.
>> In the case of virtio I would think that a virtio-level reset for
>> host_reset would be a sensible idea.
> 
> One thing to watch out for is that a virtio-scsi reset will likely hang
> too because it resets all pending requests.
> 
> Paolo Bonzini has done the lion's share of virtio-scsi work over the
> past year (or two?).  He might have some more thoughts.
> 
> Stefan
>