A way to reduce compression overhead

[Igor Fedotov <ifedotov@mirantis.com>]

Hi Sage, et al.

Let me share some ideas about possible compression burden reduction on 
the cluster.

As known we perform block compression at BlueStore level for each 
replica independently. This triples compression CPU overhead for the 
cluster. Looks like significant CPU resource waste IMHO.

We can probably eliminate this overhead by introduction write request 
preprocessing performed at ObjectStore level synchronously. This 
preprocessing parses transaction, detects write requests and transforms 
them into different ones aligned with current store allocation unit. At 
the same time resulting extents that span more than single AU are 
compressed if needed. I.e. preprocessing do some of the job performed at 
BlueStore::_do_write_data that splits write request into 
_do_write_small/_do_write_big calls. But after the split and big blob 
compression preprocessor simply updates the transaction with new write 
requests.

E.g.

with AU = 0x1000

Write Request (1~0xffff) is transformed into the following sequence:

WriteX 1~0xfff (uncompressed)

WriteX 0x1000~E000 (compressed if needed)

WriteX 0xf000~0xfff (uncompressed)

Then updated transaction is passed to all replicas including the master 
one using regular apply_/queue_transaction mechanics.


As a bonus one receives automatic payload compression when transporting 
request to remote store replicas.
Regular write request path should be preserved for EC pools and other 
needs as well.

Please note that almost no latency is introduced for request handling. 
Replicas receive modified transaction later but they do not spend time 
on doing split/compress stuff.

There is a potential conflict with the current garbage collection stuff 
though - we can't perform GC during preprocessing due to possible race 
with preceding unfinished transactions and consequently we're unable to 
merge and compress merged data. Well, we can do that when applying 
transaction but this will produce a sequence like this at each replica:

decompress original request + decompress data to merge -> compress 
merged  data.

Probably this limitation isn't that bad - IMHO it's better to have 
compressed blobs aligned with original write requests.

Moreover I have some ideas how to get rid of blob_depth notion that 
makes life a bit easier. Will share shortly.

Any thought/comments?

Thanks,
Igor