From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wido den Hollander Subject: Re: defaults paths Date: Thu, 05 Apr 2012 14:34:04 +0200 Message-ID: <4F7D913C.9030508@widodh.nl> References: <3177659F-3D4D-45A2-80D3-B344CF604901@openminds.be> <51BE8674-B8FE-41D0-B284-6D99669CD4CA@openminds.be> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: Received: from smtp01.mail.pcextreme.nl ([109.72.87.137]:34814 "EHLO smtp01.mail.pcextreme.nl" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751582Ab2DEMeI (ORCPT ); Thu, 5 Apr 2012 08:34:08 -0400 In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Bernard Grymonpon Cc: Sage Weil , ceph-devel@vger.kernel.org On 04/05/2012 10:38 AM, Bernard Grymonpon wrote: > I assume most OSD nodes will normally run a single OSD, so this would not apply to most nodes. > > Only in specific cases (where multiple OSDs run on a single node) this would come up, and these specific cases might even require to have the journals split over multiple devices (multiple ssd-disks ...) I think that's a wrong assumption. On most systems I think multiple OSDs will exist, it's debatable if one would run OSDs from different clusters very often. I'm currently using: osd data = /var/lib/ceph/$name To get back to what sage mentioned, why add the "-data" suffix to a directory name? Isn't it obvious that a directory will contain data? As I think it is a very specific scenario where a machine would be participating in multiple Ceph clusters I'd vote for: /var/lib/ceph/$type/$id Wido > > In my case, this doesn't really matter, it is up to the provision software to make the needed symlinks/mounts. > > Rgds, > Bernard > > On 05 Apr 2012, at 09:37, Andrey Korolyov wrote: > >> In ceph case, such layout breakage may be necessary in almost all >> installations(except testing), comparing to almost all general-purpose >> server software which need division like that only in very specific >> setups. >> >> On Thu, Apr 5, 2012 at 11:28 AM, Bernard Grymonpon wrote: >>> I feel it's up to the sysadmin to mount / symlink the correct storage devices on the correct paths - ceph should not be concerned that some volumes might need to sit together. >>> >>> Rgds, >>> Bernard >>> >>> On 05 Apr 2012, at 09:12, Andrey Korolyov wrote: >>> >>>> Right, but probably we need journal separation at the directory level >>>> by default, because there is a very small amount of cases when speed >>>> of main storage is sufficient for journal or when resulting speed >>>> decrease is not significant, so journal by default may go into >>>> /var/lib/ceph/osd/journals/$i/journal where osd/journals mounted on >>>> the fast disk. >>>> >>>> On Thu, Apr 5, 2012 at 10:57 AM, Bernard Grymonpon wrote: >>>>> >>>>> On 05 Apr 2012, at 08:32, Sage Weil wrote: >>>>> >>>>>> We want to standardize the locations for ceph data directories, configs, >>>>>> etc. We'd also like to allow a single host to run OSDs that participate >>>>>> in multiple ceph clusters. We'd like easy to deal with names (i.e., avoid >>>>>> UUIDs if we can). >>>>>> >>>>>> The metavariables are: >>>>>> cluster = ceph (by default) >>>>>> type = osd, mon, mds >>>>>> id = 1, foo, >>>>>> name = $type.$id = osd.0, mds.a, etc. >>>>>> >>>>>> The $cluster variable will come from the command line (--cluster foo) or, >>>>>> in the case of a udev hotplug tool or something, matching the uuid on the >>>>>> device with the 'fsid =' line in the available config files found >>>>>> in /etc/ceph. >>>>>> >>>>>> The locations could be: >>>>>> >>>>>> ceph config file: >>>>>> /etc/ceph/$cluster.conf (default is thus ceph.conf) >>>>>> >>>>>> keyring: >>>>>> /etc/ceph/$cluster.keyring (fallback to /etc/ceph/keyring) >>>>>> >>>>>> osd_data, mon_data: >>>>>> /var/lib/ceph/$cluster.$name >>>>>> /var/lib/ceph/$cluster/$name >>>>>> /var/lib/ceph/data/$cluster.$name >>>>>> /var/lib/ceph/$type-data/$cluster-$id >>>>>> >>>>>> TV and I talked about this today, and one thing we want is for items of a >>>>>> given type to live together in separate directory so that we don't have to >>>>>> do any filtering to, say, get all osd data directories. This suggests the >>>>>> last option (/var/lib/ceph/osd-data/ceph-1, >>>>>> /var/lib/ceph/mon-data/ceph-foo, etc.), but it's kind of fugly. >>>>>> >>>>>> Another option would be to make it >>>>>> >>>>>> /var/lib/ceph/$type-data/$id >>>>>> >>>>>> (with no $cluster) and make users override the default with something that >>>>>> includes $cluster (or $fsid, or whatever) in their $cluster.conf if/when >>>>>> they want multicluster nodes that don't interfere. Then we'd get >>>>>> /var/lib/ceph/osd-data/1 for non-crazy people, which is pretty easy. >>>>> >>>>> As a osd consists of data and the journal, it should stay together, with all info for that one osd in one place: >>>>> >>>>> I would suggest >>>>> >>>>> /var/lib/ceph/osd/$id/data >>>>> and >>>>> /var/lib/ceph/osd/$id/journal >>>>> >>>>> ($id could be replaced by $uuid or $name, for which I would prefer $uuid) >>>>> >>>>> Rgds, >>>>> Bernard >>>>> >>>>>> >>>>>> Any other suggestions? Thoughts? >>>>>> sage >>>>>> -- >>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>>> the body of a message to majordomo@vger.kernel.org >>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>> >>>>> >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html