Will the number of objects that have ever existed be infinite?

Will the number of objects that have ever existed be infinite?

[李沛伦]

Hello!

I'm a GSoC student this year and my job is to introduce Missing Rate
Curve (or reuse distance exactly) of objects into OSD. Now I'm trying
to find a proper algorithm to implement but there is a problem: Should
I take the number of objects tracked in an OSD as infinite or
constant?

The point is that there is an algorithm that use hash to sample only
constant number of references to do the analysis and is proved to be
accurate, which makes it possible to do online MRC construction. That
accuracy is supported by the fact that the memory addresses is
bounded, while objects can be deleted and created again and again in
Ceph. Is is reasonable to think that an OSD only serves bounded number
of objects in its life time (or the time period that we want to
compute MRC)?

Any other comment about this project is also welcomed :)

-- 
Li Peilun (李沛伦)
Yao Class J10
Institute for Interdisciplinary Information Sciences
Tsinghua University
Beijing,100084
P.R.China
Tel:86-18810671857
E-mail: lpl6338236@gmail.com
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Will the number of objects that have ever existed be infinite?

[Wido den Hollander]

On 05/23/2015 08:58 AM, 李沛伦 wrote:
> Hello!
> 
> I'm a GSoC student this year and my job is to introduce Missing Rate
> Curve (or reuse distance exactly) of objects into OSD. Now I'm trying
> to find a proper algorithm to implement but there is a problem: Should
> I take the number of objects tracked in an OSD as infinite or
> constant?
> 

A OSD doesn't track on a per-object basis, but it keeps track of
Placement Groups (PGs). A OSD can have a X number of PGs.

Technically the number of PGs might be infinite, but in practice you are
bound to CPU and Memory limits.

So I would be careful with the word "infinite", since nothing is really
infinite, eg size of a int/long might be the limitation for something.

But in theory there is no object or PG limit per OSD.

> The point is that there is an algorithm that use hash to sample only
> constant number of references to do the analysis and is proved to be
> accurate, which makes it possible to do online MRC construction. That
> accuracy is supported by the fact that the memory addresses is
> bounded, while objects can be deleted and created again and again in
> Ceph. Is is reasonable to think that an OSD only serves bounded number
> of objects in its life time (or the time period that we want to
> compute MRC)?
> 
> Any other comment about this project is also welcomed :)
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Will the number of objects that have ever existed be infinite?

[Sage Weil]

Hi Li!

On Sat, 23 May 2015, ??? wrote:
> Hello!
> 
> I'm a GSoC student this year and my job is to introduce Missing Rate
> Curve (or reuse distance exactly) of objects into OSD. Now I'm trying
> to find a proper algorithm to implement but there is a problem: Should
> I take the number of objects tracked in an OSD as infinite or
> constant?
> 
> The point is that there is an algorithm that use hash to sample only
> constant number of references to do the analysis and is proved to be
> accurate, which makes it possible to do online MRC construction. That
> accuracy is supported by the fact that the memory addresses is
> bounded, while objects can be deleted and created again and again in
> Ceph. Is is reasonable to think that an OSD only serves bounded number
> of objects in its life time (or the time period that we want to
> compute MRC)?

I don't remember how the object count affects the MRC, but I suspect we 
will want to use a strategy similar to what the HitSets do:

 - a new HitSet is generated on a periodic basis
 - each time a new one is started, we size it based on the previous 
iteration: we can compare the number of HitSet (bloom filter) insertions 
we've done with the resulting filter density.

I think we'll want to build periodic MRCs anyway since the workload will 
shift over time.  Ceph explicitly tracks the number of objects within each 
PG (see pg_stats_t).

Does that help?

sage