Re: [Qemu-devel] [RFC] qed: Add QEMU Enhanced Disk format

[Anthony Liguori <anthony@codemonkey.ws>]

On 09/07/2010 11:25 AM, Anthony Liguori wrote:
> On 09/07/2010 11:09 AM, Avi Kivity wrote:
>>  On 09/07/2010 06:40 PM, Anthony Liguori wrote:
>>>>
>>>> Need a checksum for the header.
>>>
>>> Is that not a bit overkill for what we're doing?  What's the benefit?
>>
>> Make sure we're not looking at a header write interrupted by a crash.
>
> Couldn't hurt I guess.  I don't think it's actually needed for L1/L2 
> tables FWIW.
>
>>>>> The L2 link '''should''' be made after the data is in place on 
>>>>> storage.  However, when no ordering is enforced the worst case 
>>>>> scenario is an L2 link to an unwritten cluster.
>>>>
>>>> Or it may cause corruption if the physical file size is not 
>>>> committed, and L2 now points at a free cluster.
>>>
>>> An fsync() will make sure the physical file size is committed.  The 
>>> metadata does not carry an additional integrity guarantees over the 
>>> actual disk data except that in order to avoid internal corruption, 
>>> we have to order the L2 and L1 writes.
>>
>> I was referring to "when no ordering is enforced, the worst case 
>> scenario is an L2 link to an unwritten cluster".  This isn't true - 
>> worst case you point to an unallocated cluster which can then be 
>> claimed by data or metadata.
>
> Right, it's necessary to do an fsync to protect against this.  To make 
> this user friendly, we could have a dirty bit in the header which gets 
> set on first metadata write and then cleared on clean shutdown.
>
> Upon startup, if the dirty bit is set, we do an fsck.
>
>>>> We can remove this requirement by copying-on-write any metadata 
>>>> write, and keeping two copies of the header (with version numbers 
>>>> and checksums).
>>>
>>> QED has a property today that all metadata or cluster locations have 
>>> a single location on the disk format that is immutable.  Defrag 
>>> would relax this but defrag can be slow.
>>>
>>> Having an immutable on-disk location is a powerful property which 
>>> eliminates a lot of complexity with respect to reference counting 
>>> and dealing with free lists.
>>
>> However, it exposes the format to "writes may corrupt overwritten data".
>
> No, you never write an L2 entry once it's been set.  If an L2 entry 
> isn't set, the contents of the cluster is all zeros.
>
> If you write data to allocate an L2 entry, until you do a flush(), the 
> data can either be what was written or all zeros.
>
>>> For the initial design I would avoid introducing something like 
>>> this.  One of the nice things about features is that we can 
>>> introduce multi-level trees as a future feature if we really think 
>>> it's the right thing to do.
>>>
>>> But we should start at a simple design with high confidence and high 
>>> performance, and then introduce features with the burden that we're 
>>> absolutely sure that we don't regress integrity or performance.
>>
>> For most things, yes.  Metadata checksums should be designed in 
>> though (since we need to double the pointer size).
>>
>> Variable height trees have the nice property that you don't need 
>> multi cluster allocation.  It's nice to avoid large L2s for very 
>> large disks.
>
> FWIW, L2s are 256K at the moment and with a two level table, it can 
> support 5PB of data.

I clearly suck at basic math today.  The image supports 64TB today.  
Dropping to 128K tables would reduce it to 16TB and 64k tables would be 4TB.

BTW, I don't think your checksumming idea is sound.  If you store a 
64-bit checksum along side each point, it becomes necessary to update 
the parent pointer every time the table changes.   This introduces an 
ordering requirement which means you need to sync() the file every time 
you update and L2 entry.

Today, we only need to sync() when we first allocate an L2 entry 
(because their locations never change).  From a performance perspective, 
it's the difference between an fsync() every 64k vs. every 2GB.

Plus, doesn't btrfs do block level checksumming?  IOW, if you run a 
workload where you care about this level of data integrity validation, 
if you did btrfs + qed, you would be fine.

Since the majority of file systems don't do metadata checksumming, it's 
not obvious to me that we should be.  I think one of the critical flaws 
in qcow2 was trying to invent a better filesystem within qemu instead of 
just sticking to a very simple and obviously correct format and letting 
the FS folks do the really fancy stuff.

Regards,

Anthony Liguori