Re: [Qemu-devel] [RFC] Disk integrity in QEMU

[Avi Kivity <avi@redhat.com>]

Jens Axboe wrote:
> On Sun, Oct 19 2008, Avi Kivity wrote:
>   
>> Jens Axboe wrote:
>>
>>  
>>
>>     
>>>> Sounds like a bug.  Shouldn't Linux disable the write cache unless the 
>>>> user explicitly enables it, if NCQ is available?  NCQ should provide 
>>>> acceptable throughput even without the write cache.
>>>>     
>>>>         
>>> How can it be a bug? 
>>>       
>> If it puts my data at risk, it's a bug.  I can understand it for IDE,
>> but not for SATA with NCQ.
>>     
>
> Then YOU turn it off. Other people would consider the lousy performance
> to be the bigger problem. See policy :-)
>
>   

If I get lousy performance, I can turn on the write cache and ignore the
risk of data loss.  If I lose my data, I can't turn off the write cache
and get my data back.

(it seems I can't turn off the write cache even without losing my data:

[avi@firebolt ~]$ sudo sdparm --set=WCE=0 /dev/sd[ab]
    /dev/sda: ATA       WDC WD3200YS-01P  21.0
change_mode_page: failed setting page: Caching (SBC)
    /dev/sdb: ATA       WDC WD3200YS-01P  21.0
change_mode_page: failed setting page: Caching (SBC)
)

>>> Changing the cache policy of a drive would be a
>>> policy decision in the kernel, 
>>>       
>> If you don't want this in the kernel, then the system as a whole should
>> default to being safe.  Though in this case I think it is worthwhile to
>> do this in the kernel.
>>     
>
> Doesn't matter how you turn this, it's still a policy decision. Leave it
> to the user. It's not exactly a new turn of events, commodity drives
> have shipped with write caching on forever. What if the drive has a
> battery backing? 

If the drive has a batter backup, I'd argue it should report it as a
write-through cache.  I'm not a drive manufacturer though.

> What if the user has an UPS?
>
>   

They should enable the write-back cache if they trust the UPS.  Or maybe
the system should do that automatically if it's aware of the UPS.

"Policy" doesn't mean you shouldn't choose good defaults.

>>> that is never the right thing to do.
>>> There's no such thing as 'acceptable throughput',
>>>       
>> I meant that performance is not completely destroyed.  How can you even
>>     
>
> How do you know it's not destroyed? Depending on your workload, it may
> very well be dropping your throughput by orders of magnitude.
>
>   

I guess this is the crux.  According to my understanding, you shouldn't
see such a horrible drop, unless the application does synchronous writes
explicitly, in which case it is probably worried about data safety.

>> compare data safety to some percent of performance?
>>     
>
> I'm not, what I'm saying is that different people will have different
> opponions on what is most important. Do note that the window of
> corruption is really small and requires powerloss to trigger. So for
> most desktop users, the tradeoff is actually sane.
>
>   

I agree that the window is very small, and that by eliminating software
failures we get rid of the major source of data loss.  What I don't know
is what the performance tradeoff looks like (and I can't measure since
my drives won't let me turn off the cache for some reason).

>>> Additionally, write back caching is perfectly safe, if used
>>> with a barrier enabled file system in Linux.
>>>   
>>>       
>> Not all Linux filesystems are barrier enabled, AFAIK.  Further, barriers
>> don't help with O_DIRECT (right?).
>>     
>
> O_DIRECT should just use FUA writes, there are safe with write back
> caching. I'm actually testing such a change just to gauge the
> performance impact.
>   

You mean, this is not in mainline yet?

So, with this, plus barrier support for metadata and O_SYNC writes, the
write-back cache should be safe?

Some googling shows that Windows XP introduced FUA for O_DIRECT and
metadata writes as well.

>   
>> I shouldn't need a disk array to run a database.
>>     
>
> You are free to turn off write back caching!
>
>   

What about the users who aren't on qemu-devel?

However, with your FUA change, they should be safe.

>>
>> Most desktop workloads use writeback cache, so write performance is not
>> critical.
>>     
>
> Ehm, how do you reach that conclusion based on that statement?
>
>   

Any write latency is buffered by the kernel.  Write speed is main memory
speed.  Disk speed only bubbles up when memory is tight.

>> However I'd hate to see my data destroyed by a power failure, and
>> today's large caches can hold a bunch of data.
>>     
>
> Then you use barriers or turn write back caching off, simple as that.
>   

I will (if I figure out how) but there may be one or two users who
haven't read the scsi spec yet.

Or more correctly, I am revising my opinion of the write back cache
since even when it is enabled, it is completely optional.  Instead of
disabling the write back cache we should use FUA and barriers, and since
you are to be working on FUA, it looks like this will be resolved soon
without performance/correctness compromises.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.