Re: [Ceph] Managing crushmap

[Loic Dachary <loic@dachary.org>]

[-- Attachment #1: Type: text/plain, Size: 6932 bytes --]

Hi Ghislain,

The error you see when you

 ceph osd pool set data crush_ruleset 50

has been fixed recently and is going to be backported ( http://tracker.ceph.com/issues/8599 ) Would you have time to check the backport once it's ready ?

Cheers

On 12/06/2014 10:46, ghislain.chevalier@orange.com wrote:
> Hi Greg,
> 
> the information you requested:
> dumpling platform : ceph version 0.67.9 (ba340a97c3dafc9155023da8d515eecc675c619a)
> 
> See attached the decompiled crushmap
> (created today by "ceph osd getcrushmap -o crushmap_current" and crushtool -d crushmap_current -o crushmap_current.txt)
> and the result of:
> - "ceph osd crush rule dump" in order to get rule_ids
> - "ceph osd dump | grep pool" in order to get pools characteristics 
> 
> The ceph commands I ran
> for crushmap
> 	ceph osd getcrushmap -o crushmap_current
> 	crushtool -d crushmap_current -o crushmap_current.txt
> 	vi crushmap_current.txt ==> crushmap_V2.4.txt
> 	crushtool -c crushmap_V2.4.txt -o crushmap_V2.4
> 	ceph osd setcrushmap -i crushmap_V2.4
> 	
> the file crushmap_current provided today is hence the decompilation of crushmap_V2.4
> 
> for pool
> 	rados mkpool .rgw.fastrgw
> 	ceph osd pool set .rgw.fastrgw crush_ruleset 50 ==> crush ruleset 50 does not exist
> 	ceph osd pool set .rgw.fastrgw crush_ruleset 4 ==> set pool 47 crush_ruleset to 4c
> 
> I notice this morning that the following command works
> 	rados mkpool testpool 0 50
> 
> It seems that the command ceph osd pool set <pool> crush_ruleset <ruleset_number> is not working correctly.
> 
> To go on my tests, I will delete the pools and recreate them using "rados mkpool" adding directly the right ruleset.
> 
> For the firefly platform (ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)),
> we only checked that the behavior was identical after ingesting a rule with a ruleset which would not be equal to the rule_id. 
> 
> Note:
> the pools and rules names have changed since I sent the message but the behavior remains the same
> Using calamari to modify pool parameters, the GUI is listing the ruleset but it has no effect (no update, no error displayed)
> Using Inkscope to modify pool parameters, we can force the ruleset number but it has no effect (no update, no error displayed)
> We noticed that the ceph-rest-api returns 200 OK even if the update is not set.
> 
> Best regards
> 
> -----Message d'origine-----
> De : Gregory Farnum [mailto:greg@inktank.com] 
> Envoyé : mercredi 11 juin 2014 19:03
> À : CHEVALIER Ghislain IMT/OLPS
> Cc : ceph-devel@vger.kernel.org
> Objet : Re: [Ceph] Managing crushmap
> 
> That doesn't sound right. Can you supply your decompiled CRUSH map,
> the exact commands you ran against the ceph cluster, and the exact
> version(s) you ran the test against?
> -Greg
> 
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> 
> 
> On Wed, Jun 11, 2014 at 2:17 AM,  <ghislain.chevalier@orange.com> wrote:
>> Hi all,
>>
>> Context :
>> Lab Platform
>> Ceph dumpling and firefly
>> Ubuntu 12.04 LTS
>>
>> I encountered a strange behavior managing the crushmap on a dumpling and a firefly ceph platform.
>>
>> I built a crushmap, adding 2 specific rules (fastrule and slowrule) in order to experiment tiering.
>> I used "ceph osd get|setcrushmap" and crushtool to extract and ingest the updated crushmap in the system.
>> I have to precise that I respectively associated 50 and 51 as ruleset numbers for the 2 new rules.
>> The ingestion was good; I checked it by "ceph osd crush rule dump"
>>
>> I created 2 pools (fastpool and slowpool)
>> As indicated in the doc, I tried to associate fastpool to ruleset 50 by "ceph osd pool set fastpool crush_ruleset 50"
>> an error occurred : rule 50 doesn't exist
>> As the rule_id of fastrule is 4, I did "ceph osd pool set fastpool crush_ruleset 4" and it works but it's not a correct behavior.
>> If a ceph admin wants to manage the crushmap, he doesn't have to check the rule_id (that he cannot set) before updating the attribute crush_ruleset of pools.
>> The way to manage the rules if the ruleset not the rule_id.
>>
>> I also tested that reingesting a crushmap (after for example changing the sequence of the rules in the decompiled file) causes a global update of the rule_ids.
>> I can't imagine the impacts on a platform.
>>
>> Did someone encounter this behavior?
>> Did I misunderstand how to configure a crushmap?
>>
>> Best regards
>>
>>
>>
>>
>>
>> - - - - - - - - - - - - - - - - -
>> Ghislain Chevalier
>> ORANGE/OLNC/OLPS/ASE/DAPI/CSE
>> Architecte de services de stockage
>> Storage Service Architect
>>  +33299124432
>> ghislain.chevalier@orange.com
>>  Pensez à l'Environnement avant d'imprimer ce message !
>>
>>
>> _________________________________________________________________________________________________________________________
>>
>> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
>> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
>> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
>> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>> This message and its attachments may contain confidential or privileged information that may be protected by law;
>> they should not be distributed, used or copied without authorisation.
>> If you have received this email in error, please notify the sender and delete this message and its attachments.
>> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>> Thank you.
>>
> 
> _________________________________________________________________________________________________________________________
> 
> Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
> 
> This message and its attachments may contain confidential or privileged information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
> Thank you.
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]