Understanding mon space usage during recovery

["Bartłomiej Święcki" <bartlomiej.swiecki@corp.ovh.com>]

Hi,

I was recently trying to understand the growth of mon disk space usage 
during recovery in one of our clusters,
wanted to know whether we could reduce disk usage somehow or if we just 
have to prepare more space for our mons.
Cluster is 0.94.6, just over 300 OSDs. Leveldb compaction does reduce 
space usage but it quickly grows back
to the previous usage. What I found out is that most of the leveldb data 
is used by osdmap history.

For each osdmap version leveldb contains both full and incremental entry 
so I was thinking if we really need to
store full osdmaps for all versions? If we're having incremental changes 
for every version anyway, wouldn't it be
sufficient to keep first full version only and then recover any future 
ones by applying incrementals?

I was also trying to understand how ceph figures out the range of osdmap 
versions to keep. After analyzing the code
I thought the obvious answer was in PGMap::calc_min_last_epoch_clean() - 
In case of our production cluster,
the difference between min and max clean epochs was around 30k during 
recovery, size of one full osdmap blob
in leveldb is around 250k.

I also tried to test this on my dev cluster where I could run gdb (15 
OSD, 4 OSD nearfull and lots of misplaced objects).
What I found out is that execution in OSDmonitor::get_trim_to() almost 
never jumped inside the first 'if'.
mon->pgmon()->is_readable() returns false, I did debug it once and it 
was a result of false returned by Paxos::is_lease_valid().
I was able get into mentioned 'if' only once the cluster got back to the 
healthy state. Is this expected behavior?

Thanks,
Bartek