Application error handling with write-back caching

Application error handling with write-back caching

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 740 bytes --]

SBC-3 4.15.3 Write caching says:

"If processing a write command results in logical block data in cache
that is different from the logical block data on the medium, then the
device server shall retain that logical block data in cache until a
write medium operation is performed using that logical block data."

Does "is performed" mean "completes successfully" or just "completes"?

If "is performed" just means "completes", maybe with an error, the
application would have to resubmit write requests and then try to flush
the write cache again.

I'm not aware of applications that keep acknowledged write data around
until the cache flush completion in order to retry writes.

Can anyone clarify the SBC spec on this point?

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Application error handling with write-back caching

[James Bottomley]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

On Tue, 2016-05-10 at 14:47 +0100, Stefan Hajnoczi wrote:
> SBC-3 4.15.3 Write caching says:
> 
> "If processing a write command results in logical block data in cache
> that is different from the logical block data on the medium, then the
> device server shall retain that logical block data in cache until a
> write medium operation is performed using that logical block data."
> 
> Does "is performed" mean "completes successfully" or just
> "completes"?

It means completes.  If there's an error, it will be reported via
deferred sense and an error to the SYNC CACHE command.

> If "is performed" just means "completes", maybe with an error, the
> application would have to resubmit write requests and then try to 
> flush the write cache again.
> 
> I'm not aware of applications that keep acknowledged write data 
> around until the cache flush completion in order to retry writes.

I think you may be misunderstanding the nature of the returned error. 
 It will be permanent and fatal and usually signal that the device has
a failed sector that can't be remapped and so the device itself has for
most purposes failed.  The only recovery is if you happen to have RAID,
in which case the RAID layer will mostly take care of it.

James

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Application error handling with write-back caching

[Paolo Bonzini]

On 10/05/2016 16:16, James Bottomley wrote:
> > If "is performed" just means "completes", maybe with an error, the
> > application would have to resubmit write requests and then try to 
> > flush the write cache again.
> > 
> > I'm not aware of applications that keep acknowledged write data 
> > around until the cache flush completion in order to retry writes.
> 
> I think you may be misunderstanding the nature of the returned error. 
> It will be permanent and fatal and usually signal that the device has
> a failed sector that can't be remapped and so the device itself has for
> most purposes failed.  The only recovery is if you happen to have RAID,
> in which case the RAID layer will mostly take care of it.

What about a SPACE ALLOCATION FAILED error or a similar error that can
be fixed by administrator actions (or just by a concurrent process doing
an UNMAP)?  Would a subsequent cache flush cause data loss?

Paolo

Re: Application error handling with write-back caching

[James Bottomley]

On Tue, 2016-05-10 at 18:42 +0200, Paolo Bonzini wrote:
> 
> On 10/05/2016 16:16, James Bottomley wrote:
> > > If "is performed" just means "completes", maybe with an error, 
> > > the application would have to resubmit write requests and then 
> > > try to flush the write cache again.
> > > 
> > > I'm not aware of applications that keep acknowledged write data 
> > > around until the cache flush completion in order to retry writes.
> > 
> > I think you may be misunderstanding the nature of the returned 
> > error. It will be permanent and fatal and usually signal that the 
> > device has a failed sector that can't be remapped and so the device 
> > itself has for most purposes failed.  The only recovery is if you 
> > happen to have RAID, in which case the RAID layer will mostly take
> > care of it.
> 
> What about a SPACE ALLOCATION FAILED error or a similar error that 
> can be fixed by administrator actions (or just by a concurrent 
> process doing an UNMAP)?  Would a subsequent cache flush cause data
> loss?

You're now asking about how these are handled?  It's not a SCSI
problem.  I believe if you look at the various layers, RAID would still
treat it as fatal and fail the drive and so would most filesystems. 
 The AEN warnings for TP are reported, but the admin has to sort it out
before they become a fatal error.

james

Re: Application error handling with write-back caching

[Paolo Bonzini]

On 10/05/2016 19:31, James Bottomley wrote:
> > What about a SPACE ALLOCATION FAILED error or a similar error that 
> > can be fixed by administrator actions (or just by a concurrent 
> > process doing an UNMAP)?  Would a subsequent cache flush cause data
> > loss?
>
> You're now asking about how these are handled?  It's not a SCSI
> problem.  I believe if you look at the various layers, RAID would still
> treat it as fatal and fail the drive and so would most filesystems. 
> The AEN warnings for TP are reported, but the admin has to sort it out
> before they become a fatal error.

Thanks, fatal errors are fine I guess.  We were worried that the next
SYNCHRONIZE CACHE would succeed and throw away the writes because it has
already "performed a write medium operation".

POSIX fsync is pretty underspecified in this respect too; gluster has
been throwing away those writes for a long time!  It stopped now because
we explained the issue to them, but it's pointless if the next layer
below does the same---hence Stefan's question.

(In our case the next layer is not the page cache, because we generally
use O_DIRECT.  Evicting dirty pages from the page cache would be okay if
the process(es) that wrote them are SIGKILLed, but in general it would
be a problem too).

Thanks,

Paolo