Dev Meeting followup on compression

[Igor Fedotov <ifedotov@mirantis.com>]

Guys,

let me summarize what we decided regarding compression support in Ceph 
during the Dev Meeting last week.

Below are possible implementation options, their pros/cons and the 
conclusion.

1) Add compression support to RGW.
Pros/Cons:
+ Simple
+ Reduced inter-component traffic
- Limited to specific clients
- Will conflict with partial read/writes if any appear

Alyona Kiseleva from Mirantis (akyseleva@mirantis.com) will start 
implementing this promptly. You can ask additional questions to her via 
e-mail or during daily RGW stendups she is planning to attend regularly.

2) Add basic compression support to BlueStore. Basic = "Append only" 
functionality to be implemented. Specific "append only" hint/flag needs 
to be introduced for object creation interface.

Pros/Cons:
+ Moderate complexity
+ Suits for any client/PG backend
+ Good isolation from other Ceph components
- Limited applicability
- additional 50-200% CPU load for the cluster since we compress each 
replica/EC shard independently
- no inter-component traffic saving
- recovery procedure requires decompress/recompress sequence

Mirantis ( me specifically ) will start blueprint/POC creation for this 
promptly. I'm planning to attend daily RBD syncup regularly to inform on 
the progress.

3) Add full compression support at BlueStore. This includes 2) + support 
for random object writes.
Pros/Cons:
+ Severe complexity
+ Suits for any client/PG backend
+ Good isolation from other Ceph components
- additional 50-200% CPU load for the cluster since we compress each 
replica/EC shard independently
- no inter-component traffic saving
- recovery procedure requires decompress/recompress sequence

On-Hold for now. Will require insert/delete data range for the store.

4) Add basic ( append only ) compression support at OSD using interim 
PGBackend.

Pros/Cons:
+ Moderate complexity
+ Suits for any client/PG backend
+ Data compressed before replication/"EC inflation"
+ inter-component traffic saving
+ recovery procedure don't need decompress/recompress sequence
- Too tight(danger) integration into Ceph internals
- Limited applicability
- Bad experience with "append only" notion for EC and desire to avoid 
that for compression

Rejected ( Personally I'd prefer this option with subsequent mutation to 
full RW support. The rationale is the reduced CPU load comparing to 
options 2 & 3)

Any additional comments/suggestions are welcome.

Thanks,
Igor