Re: Single MDS cephx key

[Travis Nielsen <Travis.Nielsen@Quantum.com>]

Thanks for the clarification, and Rook does use Kubernetes facilities to
handle the log collection so it sounds like we're good to go.



On 9/27/17, 9:45 AM, "John Spray" <jspray@redhat.com> wrote:

>On Wed, Sep 27, 2017 at 5:36 PM, Travis Nielsen
><Travis.Nielsen@quantum.com> wrote:
>> To expand on the scenario, I'm working in a Kubernetes environment where
>> the MDS instances are somewhat ephemeral. If an instance (pod) dies or
>>the
>> machine is restarted, Kubernetes will start a new one in its place. To
>> handle the failed pod scenario, I'd appreciate if you could help me
>> understand MDS better.
>>
>> 1) MDS instances are stateless, correct? If so, I'm assuming when an MDS
>> instance dies, a new MDS instance (with a new ID) can be brought up and
>> assigned its rank without any side effects other than disruption during
>> the failover. Or is there a reason to treat them more like mons that
>>need
>> to survive reboots and maintain state?
>
>Yep, completely stateless.  Don't forget logs though -- for ephemeral
>instances, it would be a good idea to have them sending their logs
>somewhere central, so that we don't lose all the history whenever a
>container restarts (you may very well have already covered this in
>general in the context of Rook).
>
>> 2) Will there be any side effects from MDS instances being somewhat
>> ephemeral? For example, if a new instance came up every hour or every
>>day,
>> what challenges would I run into besides cleaning up the old cephx keys?
>
>While switching daemons around is an online operation, it is not
>without some impact to client IOs, and the freshly started MDS daemon
>will generally have a less well populated cache than the one it is
>replacing.
>
>John
>
>>
>> Thanks!
>> Travis
>>
>>
>>
>>
>> On 9/27/17, 3:01 AM, "John Spray" <jspray@redhat.com> wrote:
>>
>>>On Wed, Sep 27, 2017 at 12:09 AM, Travis Nielsen
>>><Travis.Nielsen@quantum.com> wrote:
>>>> Is it possible to use the same cephx key for all instances of MDS or
>>>>do
>>>> they each require their own? Mons require the same keyring so I tried
>>>> following the same pattern by creating a keyring with "mds.", but the
>>>>MDS
>>>> is complaining about not being authorized when it tries to start. Am I
>>>> missing something or is this not possible for MDS keys? If I create a
>>>> unique key for each MDS instance it works fine, but it would simplify
>>>>my
>>>> scenario if I could use the same key. I'm running on Luminous.
>>>
>>>I've never heard of anyone trying to do this.
>>>
>>>It's probably not a great idea, because if all MDS daemons are using
>>>the same key then you lose the ability to simply remove an MDS's key
>>>to ensure that it can't talk to the system any more.  This is useful
>>>when tearing something down, because it means you're not taking it on
>>>faith that the daemon is really physically stopped.
>>>
>>>John
>>>
>>>> The key was generated with this:
>>>> ceph auth get-or-create-key mds. osd allow * mds allow mon allow
>>>>profile
>>>> mds
>>>>
>>>>
>>>>
>>>> The keyring contents are:
>>>> [mds.]
>>>> key = AQD62spZw3zRGhAAkHHVokP3BDf8PEy4+vXGMg==
>>>>
>>>>
>>>> I run the following with that keyring:
>>>> ceph-mds --foreground --name=mds.mymds -i mymds
>>>>
>>>> And I see the error:
>>>> 2017-09-26 22:55:55.973047 7fb004459200 -1 mds.mds81c2n ERROR: failed
>>>>to
>>>> authenticate: (22) Invalid argument
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Travis
>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>>>>in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at
>>>>https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fvger.ke
>>>>rn
>>>>el.org%2Fmajordomo-info.html&data=02%7C01%7CTravis.Nielsen%40quantum.co
>>>>m%
>>>>7C00d1db42478d48fa8c6508d5058ec254%7C322a135f14fb4d72aede122272134ae0%7
>>>>C1
>>>>%7C0%7C636421033061815149&sdata=3Vu79xeZbnb1jwhGE85PACq6qByVE6vUlPjp8pj
>>>>rv
>>>>hA%3D&reserved=0
>>