Re: durability model

Re: durability model

[Kyle Bader]

> Something that is not yet clear for me is how do we study the overlapping of different layers of reliability using the modelling tool. Is the idea to use erasure coding on top of another reliable layer like a RAID-based storage? If I understand correctly the current tool models disks, RAID, RADOS independently. But erasure coding is added on top of those layers, isn't it?

You understand correctly in that the tool was originally written to
analyze RADOS with replication and RAID, independently. Erasure coding
can be used independently and I had considered the original scope to
be modeling just that, a erasure coded pool in isolation. Being able
to model durability of a tiered pool configuration would be an
excellent next step, though. Good question!

-- 
Kyle Bader - Inktank
Senior Solution Architect

Re: durability model

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 2422 bytes --]



On 27/05/2014 15:30, Koleos Fuskus wrote:> Hi Kyle,
> Thanks for your answer!
> I have other questions regarding the model. Sorry, I accumulated different doubts and the e-mail is a bit long.
> 
> 1. What will be erasure model unit? Is it a pool or a placement group? I cannot see the best option here yet, probably because my inexperience with Ceph.

A pool. 

> 2. Crush_ruleset for erasured pools: Are these crush-ruleset impacting on the reliability model? Should I define a default crush-ruleset? 

Unless I'm mistaken, the crush ruleset is only relevant to the durability model to the extent that it defines the failure domain. If a ruleset requires that no two shards of a given placement group (i.e. no two shards of a given object since a placement group is a set of object that share the same OSD for a given shard) are on the same host, it will lead to a model that is different from a ruleset that requires that no two shards are in the same datacenter.

> 3. Metadata: Jerasure creates metadata files. How are those files stored in ceph (what kind of redundancy)? Should I care about them? I guess the jerasure metafiles are not important. If all objects stored in a pool share the same erasure-code-k and erasure-code-m, then is easy to re-generate the metadata file. However, if metadata file is lost and the pool contains erasured objects with different k and m, then repairing data will not be possible. Should I ignore this problem? Could you confirm that each pool has the same erasure parameters?

Jerasure does not create a metadata file.

> 4. Number of PG for an erasured-pool. The total PGs in the case of replication is calculated with this formula: Total PGs= (OSDs * 100)/Replicas.
>
> Should I replace Replicas for the stretch factor (storage overhead) in the case of erasure?
> 
> 5.Parameters - Default values:
> erasure-code-k (default 4)
> erasure-code-m (default 2)
> 
> OSD (default 200)
> PG (default 100 x OSD)
> stripes (default ????)


I proposed an update to the documentation to clarify this : https://github.com/ceph/ceph/pull/1856 . Does it look sensible to you ?

Cheers

> 6. Units
> It is hardcoded on the reliable tool:
> KiB = 1000
> MiB = KiB * 1000
> GiB = MiB * 1000
> TiB = GiB * 1000
> PiB = TiB * 1000
> 
> Why KiB is 1000 when should be 1024?

> Best, 
> Koleosfuscus
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: durability model

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]



On 28/05/2014 15:35, Koleos Fuskus wrote:
> 
> 
>> 1. What will be erasure model unit? Is it a pool or a placement group? I cannot see the best option here yet, probably because my inexperience with Ceph.
> 
> A pool. 
> OK. this is what I thought but some things are not clear enough. I am deploying Ceph to toying a bit more.
> 
> 
>> 3. Metadata: Jerasure creates metadata files. How are those files stored in ceph (what kind of 
> Jerasure does not create a metadata file.
> 
> It does on my machine! at least with jerasure 1.2 on mac os :)

If you mean the files that are created by the command line examples provided with jerasure (1.2 or 2), they are not required and only meant for test purposes.

Cheers

> 
> 
> 
> 
> I proposed an update to the documentation to clarify this : https://github.com/ceph/ceph/pull/1856 . Does it look sensible to you ?
> 
> 
> 
> THanks!
> 
> Cheers,
> 
> 
> Koleosfuscus
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: durability model

[Milosz Tanski]

On Wed, May 28, 2014 at 10:05 AM, Loic Dachary <loic@dachary.org> wrote:
>
>
> On 28/05/2014 15:35, Koleos Fuskus wrote:
>>
>>
>>> 1. What will be erasure model unit? Is it a pool or a placement group? I cannot see the best option here yet, probably because my inexperience with Ceph.
>>
>> A pool.
>> OK. this is what I thought but some things are not clear enough. I am deploying Ceph to toying a bit more.
>>
>>
>>> 3. Metadata: Jerasure creates metadata files. How are those files stored in ceph (what kind of
>> Jerasure does not create a metadata file.
>>
>> It does on my machine! at least with jerasure 1.2 on mac os :)
>
> If you mean the files that are created by the command line examples provided with jerasure (1.2 or 2), they are not required and only meant for test purposes.
>
> Cheers
>
>>
>>
>>
>>
>> I proposed an update to the documentation to clarify this : https://github.com/ceph/ceph/pull/1856 . Does it look sensible to you ?
>>
>>

Recently while considering a hardware update on the Ceph cluster I've
been thinking about the durability model of the new OSD nodes once the
cluster is done migrating over. So I came looking through these old
emails. I think it might be very useful to have a web tool that
describes (or visualizes) the current cluster durability model, or
better yet helps you play with the variables to visualize the changes.

If we can formulate the rules model based on some common variables
(about machines, OSDs, racks) and replication rules (count or erasure
coding paramaters). Then we can spit out useful information like:
- How many nodes / racks / osds do you need to lose to have data offline
- What percentage of data is going to be unavailable if you lose X resources
- What's the probability of data loss over time
- How does the readability model impact your cluster network traffic,
repair traffic

I think it would be stair-forward to build the web app. In fact, I'll
volunteer myself for that task. Before we start that I would suggest
we try to get the model in something like Google Docs Sheets so we can
use it as a basis.

-- 
Milosz Tanski
CTO
16 East 34th Street, 15th floor
New York, NY 10016

p: 646-253-9055
e: milosz@adfin.com

Re: durability model

[Sage Weil]

On Wed, 1 Apr 2015, Milosz Tanski wrote:
> On Wed, May 28, 2014 at 10:05 AM, Loic Dachary <loic@dachary.org> wrote:
> >
> >
> > On 28/05/2014 15:35, Koleos Fuskus wrote:
> >>
> >>
> >>> 1. What will be erasure model unit? Is it a pool or a placement group? I cannot see the best option here yet, probably because my inexperience with Ceph.
> >>
> >> A pool.
> >> OK. this is what I thought but some things are not clear enough. I am deploying Ceph to toying a bit more.
> >>
> >>
> >>> 3. Metadata: Jerasure creates metadata files. How are those files stored in ceph (what kind of
> >> Jerasure does not create a metadata file.
> >>
> >> It does on my machine! at least with jerasure 1.2 on mac os :)
> >
> > If you mean the files that are created by the command line examples provided with jerasure (1.2 or 2), they are not required and only meant for test purposes.
> >
> > Cheers
> >
> >>
> >>
> >>
> >>
> >> I proposed an update to the documentation to clarify this : https://github.com/ceph/ceph/pull/1856 . Does it look sensible to you ?
> >>
> >>
> 
> Recently while considering a hardware update on the Ceph cluster I've
> been thinking about the durability model of the new OSD nodes once the
> cluster is done migrating over. So I came looking through these old
> emails. I think it might be very useful to have a web tool that
> describes (or visualizes) the current cluster durability model, or
> better yet helps you play with the variables to visualize the changes.
> 
> If we can formulate the rules model based on some common variables
> (about machines, OSDs, racks) and replication rules (count or erasure
> coding paramaters). Then we can spit out useful information like:
> - How many nodes / racks / osds do you need to lose to have data offline
> - What percentage of data is going to be unavailable if you lose X resources
> - What's the probability of data loss over time
> - How does the readability model impact your cluster network traffic,
> repair traffic
> 
> I think it would be stair-forward to build the web app. In fact, I'll
> volunteer myself for that task. Before we start that I would suggest
> we try to get the model in something like Google Docs Sheets so we can
> use it as a basis.

If can you haven't seen this:

	https://github.com/ceph/ceph-tools/tree/master/models/reliability

It's not web-based but it probably could be changed to be.  I definitely 
like the idea of having a hosted version.

sage

Re: durability model

[Milosz Tanski]

On Wed, Apr 1, 2015 at 9:49 AM, Sage Weil <sage@newdream.net> wrote:
> On Wed, 1 Apr 2015, Milosz Tanski wrote:
>> On Wed, May 28, 2014 at 10:05 AM, Loic Dachary <loic@dachary.org> wrote:
>> >
>> >
>> > On 28/05/2014 15:35, Koleos Fuskus wrote:
>> >>
>> >>
>> >>> 1. What will be erasure model unit? Is it a pool or a placement group? I cannot see the best option here yet, probably because my inexperience with Ceph.
>> >>
>> >> A pool.
>> >> OK. this is what I thought but some things are not clear enough. I am deploying Ceph to toying a bit more.
>> >>
>> >>
>> >>> 3. Metadata: Jerasure creates metadata files. How are those files stored in ceph (what kind of
>> >> Jerasure does not create a metadata file.
>> >>
>> >> It does on my machine! at least with jerasure 1.2 on mac os :)
>> >
>> > If you mean the files that are created by the command line examples provided with jerasure (1.2 or 2), they are not required and only meant for test purposes.
>> >
>> > Cheers
>> >
>> >>
>> >>
>> >>
>> >>
>> >> I proposed an update to the documentation to clarify this : https://github.com/ceph/ceph/pull/1856 . Does it look sensible to you ?
>> >>
>> >>
>>
>> Recently while considering a hardware update on the Ceph cluster I've
>> been thinking about the durability model of the new OSD nodes once the
>> cluster is done migrating over. So I came looking through these old
>> emails. I think it might be very useful to have a web tool that
>> describes (or visualizes) the current cluster durability model, or
>> better yet helps you play with the variables to visualize the changes.
>>
>> If we can formulate the rules model based on some common variables
>> (about machines, OSDs, racks) and replication rules (count or erasure
>> coding paramaters). Then we can spit out useful information like:
>> - How many nodes / racks / osds do you need to lose to have data offline
>> - What percentage of data is going to be unavailable if you lose X resources
>> - What's the probability of data loss over time
>> - How does the readability model impact your cluster network traffic,
>> repair traffic
>>
>> I think it would be stair-forward to build the web app. In fact, I'll
>> volunteer myself for that task. Before we start that I would suggest
>> we try to get the model in something like Google Docs Sheets so we can
>> use it as a basis.
>
> If can you haven't seen this:
>
>         https://github.com/ceph/ceph-tools/tree/master/models/reliability
>
> It's not web-based but it probably could be changed to be.  I definitely
> like the idea of having a hosted version.

Somehow I missed this. I'll take a stab to converting it into a web app.

>
> sage



-- 
Milosz Tanski
CTO
16 East 34th Street, 15th floor
New York, NY 10016

p: 646-253-9055
e: milosz@adfin.com