Re: Cache tier READ_FORWARD transition

[Mark Nelson <mark.nelson@inktank.com>]

On 07/07/2014 02:43 PM, Sage Weil wrote:
> On Mon, 7 Jul 2014, Mark Nelson wrote:
>> On 07/07/2014 02:29 PM, Sage Weil wrote:
>>> On Mon, 7 Jul 2014, Luis Pabon wrote:
>>>> Hi all,
>>>>       I am working on OSDMonitor.cc:5325 and wanted to confirm the
>>>> following
>>>> read_forward cache tier transition:
>>>>
>>>>       readforward -> forward || writeback || (any && num_objects_dirty ==
>>>> 0)
>>>>       forward -> writeback || readforward || (any && num_objects_dirty ==
>>>> 0)
>>>>       writeback -> readforward || forward
>>>>
>>>> Is this the correct cache tier state transition?
>>>
>>> That looks right to me.
>>>
>>> By the way, I had a thought after we spoke that we probably want something
>>> that is somewhere inbetween the current writeback behavior (promote on
>>> first read) and the read_forward behavior (never promote on read).  I
>>> suspect a good all-around policy is something like promote on second read?
>>> This should probably be rolled into the writeback mode as a tunable...
>>
>> That would be a good start I think.  What about some kind of scheme that also
>> favours promoting small objects over larger ones?  It could be as simple as
>> increasing the number of reads necessary to do a promotion based on the object
>> size.
>>
>> ie something like:
>>
>> <= 64k object = 1 read
>> <= 512k object = 2 read
>> else 3 read
>>
>> That would make the behaviour for default RBD object sizes always 3 read, but
>> could keep big objects out of the cache tier for RGW.
>
> We don't have enough information to do that right now, since on a miss we
> redirect the client instead of proxying them and never learn what the
> actual object size is.
>
> If/after we start doing proxying for the reads, then lots of other stuff
> becomes possible... but I think we'll need to be careful about choosing
> where to add complexity.

Ok, that makes sense.  Ignoring RGW for the moment, on the RBD side can 
we infer about the object sizes based on the image order?  Can we 
provide a hint in some way?  I guess my assumptions specifically for RBD 
are:

1) For large reads from any object:

very low promotion priority since spinning disks can do this fast. Can 
get just from the read len?

2) For small reads from (presumed) large objects

sequential IO: Probably not at all (especially if we have big enough 
read ahead on base pool OSD fs)?  Can we  save/check previous read 
pos(s) of the same object in addition to a previous attempt?  Too complex?

random IO: Maybe even 3rd read attempt?  The worst reads will come out 
of buffer cache anyway.  Given how expensive promotion is for large 
objects, it seems to me we need to promote very slowly and infrequently.

3) reads from (presumed) small objects.

Do the promotion right away since the promotion is small and the SSDs 
can do small writes faster than the spinning disks can do small reads?