Re: Adding compression/checksum support for bluestore.

[Igor Fedotov <ifedotov@mirantis.com>]

On 30.03.2016 23:41, Vikas Sinha-SSI wrote:
>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> owner@vger.kernel.org] On Behalf Of Allen Samuels
>> Sent: Wednesday, March 30, 2016 12:47 PM
>> To: Sage Weil; Igor Fedotov
>> Cc: ceph-devel
>> Subject: Adding compression/checksum support for bluestore.
>>
>> [snip]
>>
>> Time to talk about checksums.
>>
>> First let's divide the world into checksums for data and checksums for
>> metadata -- and defer the discussion about checksums for metadata
>> (important, but one at a time...)
>>
>> I believe it's a requirement that when checksums are enabled that 100% of
>> data reads must be validated against their corresponding checksum. This
>> leads you to conclude that you must store a checksum for each
>> independently readable piece of data.
>>
>> When compression is disabled, it's relatively straightforward -- there's a
>> checksum for each 4K readable block of data. Presumably this is a simple
>> vector stored in the pextent structure with one entry for each 4K block of
>> data.
>>
>> Things get more complicated when compression is enabled. At a minimum,
>> you'll need a checksum for each blob of compressed data (I'm using blob
>> here as unit of data put into the compressor, but what I really mean is the
>> minimum amount of *decompressable* data). As I've pointed out before,
>> many of the compression algorithms do their own checksum validation. For
>> algorithms that don't do their own checksum we'll want one checksum to
>> protect the block -- however, there's no reason that we can't implement this
>> as one checksum for each 4K physical blob, the runtime cost is nearly
>> equivalent and it will considerably simplify the code.
>>
>> Thus I think we really end up with a single, simple design. The pextent
>> structure contains a vector of checksums. Either that vector is empty
>> (checksum disabled) OR there is a checksum for each 4K block of data (not
>> this is NOT min_allocation size, it's minimum_read_size [if that's even a
>> parameter or does the code assume 4K readable blocks? [or worse, 512
>> readable blocks?? -- if so, we'll need to cripple this]).
>>
>> When compressing with a compression algorithm that does checksuming we
>> can automatically suppress checksum generation. There should also be an
>> administrative switch for this.
>>
>> This allows the checksuming to be pretty much independent of compression
>> -- which is nice :)
>>
>> This got me thinking, we have another issue to discuss and resolve.
>>
>> The scenario is when compression is enabled. Assume that we've taken a big
>> blob of data and compressed it into a smaller blob. We then call the allocator
>> for that blob. What do we do if the allocator can't find a CONTIGUOUS block
>> of storage of that size??? In the non-compressed case, it's relatively easy to
>> simply break it up into smaller chunks -- but that doesn't work well with
>> compression.
>>
>> This isn't that unlikely a case, worse it could happen with shockingly high
>> amounts of freespace (>>75%) with some pathological access patterns.
>>
>> There's really only two choices. You either fracture the logical data and
>> recompress OR you modify the pextent data structure to handle this case.
>> The later isn't terribly difficult to do, you just make the size/address values
>> into a vector of pairs. The former scheme could be quite expensive CPU wise
>> as you may end up fracturing and recompressing multiple times (granted, in a
>> pathological case). The latter case adds space to each onode for a rare case.
>> The space is recoverable with an optimized serialize/deserializer (in essence
>> you could burn a flag to indicate when a vector of physical chunks/sizes is
>> needed instead of the usual scalar pair).
>>
>> IMO, we should pursue the later scenario as it avoids the variable latency
>> problem. I see the code/testing complexity of either choice as about the
>> same.
>>
> If I understand correctly, then there would still be a cost associated with writing dis-contiguously
> to disk. In cases such as this where the resources for compression are not easily available, I wonder
> if it is reasonable to simply not do compression for that Write. The cost of not compressing would be
> a missed space optimization, but the cost of compressing in any and all cases could be significant to latency.
Seems to be a reasonable and simple solution.
There is still a technical question to distinguish "no space" and "no 
contiguous space" allocation failure cases. Need to be addressed by the 
allocator...