Re: The fundamental evil of "magic" in computing systems -> Was: mon daemon makes authentication side effects on startup

[Owen Synge <osynge@suse.com>]

On 04/07/2016 09:49 AM, Jens Rosenboom wrote:
> 2016-04-06 10:23 GMT+02:00 Owen Synge <osynge@suse.com>:
>> Dear Greg and others,
>>
>> Thankyou for your very helpful email, as it completely misses my point,
>> and that illustrate why this point is so important to be addressed.
>>
>> I am sure Greg has a deep understanding of this area. But I am pleased
>> Greg missed my points from 0-9, Greg's assumption that it is lack of
>> understanding on my part (which I am sure is common), clearly
>> illustrates where this "magic" of the side effect of starting a mon
>> demon becomes becomes "dark magic".
>>
>> If you object to "magic" and "dark magic" in this email please
>> substitute them with "side effect" and "negative consequences of side
>> effects" respectively, and you get a more serious reply :)
>>
>> On 04/05/2016 10:14 PM, Gregory Farnum wrote:
>>> I think you're fundamentally understanding how these keys come into
>>> existence. They aren't generated randomly on the local monitor; it
>>> uses get-or-create in order to fetch them (and create them if they
>>> don't already exist).
>>
>> I have looked at this issue in depth, and general confusion in this area
>> is indeed very common, so it is reasonable to expect everyone is
>> confused by the same thing.
>>
>> In my experience it is "magic" that causes admins fear, as good admins,
>> need to understand, because they need to understand the side effects of
>> any "magic", in case the "magic" is "dark", and in this case it is with
>> points (0) to (8) showing is indeed "dark magic".
>>
>> Lets be specific:
>>
>> Fetch and create are fundamentally different in side effects when doing
>> deployment. Lets be clear, when ceph does a "fetch" of a key, is not I
>> believe and issue, but when ceph uses magic to "create" keys, it can
>> often cause side effects. Hence the process to "create" a key should
>> only occur when its asked to be done.
>>
>> The current get-or-create keyrings as a side effect of booting a mon
>> makes many issues (points 0-8 may not be all the issues, just ones that
>> spring to my mind). If the booting of a mon only did a fetch I would
>> feel we could resolve all my point except (2) and (9) sadly a boot of a
>> mon will also do a create keys where this "magic" starts to become very
>> "dark" indeed.
>>
>>> So maybe it's difficult to pre-generate your own keys and plug them
>>> into the system (I don't remember where the initial values come from
>>> in standard deployment scenarios),
>>
>> See my reply to John as to how you can deploy ceph without ceph-create-keys.
>>
>>> but once they're set up you don't
>>> need to carefully install your values on all the monitor nodes — they
>>> will fetch the correct values from the monitor cluster.
>>
>> I am objecting to the side effect of booting the mon and that process
>> creating keys that where not asked for, potentially causing valid
>> osd-bootstrap, rgw-bootstrap or mds-bootstrap to fail authentication as
>> invalid ones have been created as a side effect of starting the mon daemons.
>>
>>> The coordination problem here is not really any different than that of
>>> making sure your monitors are all part of the "mon initial members"
>>> config option,
>>
>> You are forgetting that we also have osd-bootstrap, rgw-bootstrap or
>> mds-bootstrap keys and these may be generated by some other tool than
>> the mon, this is made much much harder to do by the mon init scripts
>> without being asked explicitly to do so.
>>
>>> btw. Which you need to solve or else you're liable to
>>> have them coming up and creating independent monitor clusters and
>>> going haywire.
>>> -Greg
>>
>> Not knowing what is happening is the enemy of understanding, and hence
>> the creator of "magic". Often giving the "magic" a name, or making it
>> explicit, causes enough understanding to remove it's "magic" properties.
>> Hence making all occurrences of key "create" (not "fetch") an explicit
>> step rather than a side effect will go a great deal to address this issue.
>>
>> So if creating keys was not a side effect of booting mons, we would have
>> not issue here, as anyone who is used to cluster automation, has good
>> tools. These tools include chef, puppet, salt, and ansible, for cluster
>> management ideally, but more manually we have tools to copy files such
>> as rsync, scp, and tools to diagnose such issues such as checksums.
>>
>>   ceph-create-keys --cluster ${CLUSTER} --id ${MON_NAME}
>>
>> Having the above command separated from booting a mon actually avoids
>> osd's rgw's and mds's going haywire if they are configured in parallel
>> to the mon with keys from a source external to the mon, unless you
>> either (a) build in a layer of cluster synchronization above ceph, such
>> as ceph-deploy has done with its single threaded operation across a
>> complete cluster, or (b) do lots of dirty "magic" to remove
>> inconsistencies. Solution (a) is not good due to issue (0) amongst
>> others, and (b) creates more "magic" which has to be very carefully
>> designed to avoid it being "dark".
>>
>> Another way to remove this "magic" is to document "magic" in detail, and
>> documenting this in this email is long and detailed, although Greg made
>> a start, he missed out the very important part of why the mds-bootstrap
>> keyring, is more important than is documented when if comes to deploying
>> your cluster the first time. I will skip it for now, but I am happy to
>> expand if needed.
>>
>> In this case I argue the "magic" can be removed by making the process of
>> creating keys explicit. I would propose separating the "create" of  keys
>> from booting a mon is the least confusing and "magical" solution, with
>> the least chance of causing trouble for admins.
>>
>> Thank you Greg for taking the time to reply, and please forgive me for
>> using your reply to illustrate that the real problem is the "magic", and
>> that "magic" removes understanding, hence knowledge of the "magic"
>> having "dark" issues, as this is a fear inducing thing for an admin new
>> to ceph.
> 
> Hi Owen,
> 
> thanks for taking this up, I have been trying to get this issue fixed
> in Ubuntu, not been aware that this was actually present in the
> upstream code already, see
> 
> https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1563330
> 
> So I strongly support your wish of being able to automatically deploy
> a cluster, including all necessary credentials, without any
> well-intended dark magic intervening.
> 
> Yours,
>
> --> Jens

Thanks Jens,

Your support in this is really valuable to know other people agree
well-intended dark magic is an issue for automation.

Best regards

Owen


-- 
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG
Nürnberg)

Maxfeldstraße 5

90409 Nürnberg

Germany
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html