Re: Adding Data-At-Rest compression support to Ceph

[Igor Fedotov <ifedotov@mirantis.com>]

On 24.09.2015 19:03, Sage Weil wrote:
> On Thu, 24 Sep 2015, Igor Fedotov wrote:
>
>>
>> There is probably no need in strict alignment with the stripe size. We can use
>> block sizes that client provides on write dynamically. If some client writes
>> in stripes - then we compress that block. If others use larger blocks ( e.g.
>> caching agent on flush) - we can use that size or split the provided block
>> into several smaller chunks ( e.g. up to max N*stripe_size ) for overhead
>> reduction on random read. Even if client uses dynamic block sizes ( low level
>> RADOS use?) we can rely on them some way without static bind to stripe size.
>> Surely this is much easier when appends are permitted only.  General "random
>> writes" case will be more complex.
> Dynamic stripe sizes are possible but it's a significant change from the
> way the EC pool currently works.  I would make that a separate project (as
> its useful in its own right) and not complicate the compression situation.
>
> Or, if it simplifies the compression approach, then I'd make that change
> first.
My point was rather about the lack of need to depend on stripe size for 
compression than about the need for dynamic stripes.
As far as I understand clients can write data using blocks larger then 
stripe size, e.g. several stripes together. Is that correct?

At least I could see that for cache flush and low-level RADOS access.
So we can compress every written block independently - if it has stripe 
size - that's OK - compress it as-is. if it's larger - let's compress 
the whole block or split into less ones and compress them independently.

Thus I think there is no explicit need for additional changes in Ceph 
for doing compression.

Thanks,
Igor.
> sage