/var/lib/ceph/*/*/config

/var/lib/ceph/*/*/config

[Sage Weil]

I think we should add an optional 'config' file location to the config 
search path.  This would allow you to put customized options for an 
individual daemon in the daemon data directory.  Or, you could avoid 
putting a config in /etc/ceph entirely, so that the daemon data is 
completely self-contained on the device.  That would streamline boot disk 
failure (just reinstall OS and ceph package) or drive swaps between failed 
host hardware.

What do you think?
sage

Re: /var/lib/ceph/*/*/config

[Wido den Hollander]

On 19-01-16 16:15, Sage Weil wrote:
> I think we should add an optional 'config' file location to the config 
> search path.  This would allow you to put customized options for an 
> individual daemon in the daemon data directory.  Or, you could avoid 
> putting a config in /etc/ceph entirely, so that the daemon data is 
> completely self-contained on the device.  That would streamline boot disk 
> failure (just reinstall OS and ceph package) or drive swaps between failed 
> host hardware.
> 
> What do you think?

I think that's an awesome idea. Simply plug in the disk, mount it and run!

Yes, this would make life a lot easier.

Wido

> sage
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: /var/lib/ceph/*/*/config

[Allen Samuels]

Yes a step in the right direction.

Allen Samuels
Software Architect, Emerging Storage Solutions 

2880 Junction Avenue, Milpitas, CA 95134
T: +1 408 801 7030| M: +1 408 780 6416
allen.samuels@SanDisk.com


-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Sage Weil
Sent: Tuesday, January 19, 2016 7:15 AM
To: ceph-devel@vger.kernel.org
Subject: /var/lib/ceph/*/*/config

I think we should add an optional 'config' file location to the config search path.  This would allow you to put customized options for an individual daemon in the daemon data directory.  Or, you could avoid putting a config in /etc/ceph entirely, so that the daemon data is completely self-contained on the device.  That would streamline boot disk failure (just reinstall OS and ceph package) or drive swaps between failed host hardware.

What do you think?
sage

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: /var/lib/ceph/*/*/config

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1361 bytes --]

On Tue, 19 Jan 2016, Haomai Wang wrote:
> On Tue, Jan 19, 2016 at 11:15 PM, Sage Weil <sweil@redhat.com> wrote:
>       I think we should add an optional 'config' file location to the
>       config
>       search path.  This would allow you to put customized options for
>       an
>       individual daemon in the daemon data directory.  Or, you could
>       avoid
>       putting a config in /etc/ceph entirely, so that the daemon data
>       is
>       completely self-contained on the device.  That would streamline
>       boot disk
>       failure (just reinstall OS and ceph package) or drive swaps
>       between failed
>       host hardware.
> 
>       What do you think?
> 
> 
> Totally agreed!
> 
> Actually we have implemented alike thing downstream, ceph-osd,ceph-mon will
> check ceph.conf in the local dir(/var/lib/ceph/*/*/ceph.conf) then overwrite
> existing config values. That's said, it will load system ceph
> config(/etc/ceph/ceph.conf) file then load local ceph.conf.

The one reservation about this direction is that there are a few devices 
where I'm not sure how to attach the metadata (keyring, conf, and other 
random bits in the osd_data dir) to the device: the SPDK NVMe driver, and 
kinetic.  These are somewhat obscure, but I'm curious if there are other 
ideas floating around about how this should be approached...

sage

Re: /var/lib/ceph/*/*/config

[Haomai Wang]

On Wed, Jan 20, 2016 at 12:46 AM, Sage Weil <sweil@redhat.com> wrote:
> On Tue, 19 Jan 2016, Haomai Wang wrote:
>> On Tue, Jan 19, 2016 at 11:15 PM, Sage Weil <sweil@redhat.com> wrote:
>>       I think we should add an optional 'config' file location to the
>>       config
>>       search path.  This would allow you to put customized options for
>>       an
>>       individual daemon in the daemon data directory.  Or, you could
>>       avoid
>>       putting a config in /etc/ceph entirely, so that the daemon data
>>       is
>>       completely self-contained on the device.  That would streamline
>>       boot disk
>>       failure (just reinstall OS and ceph package) or drive swaps
>>       between failed
>>       host hardware.
>>
>>       What do you think?
>>
>>
>> Totally agreed!
>>
>> Actually we have implemented alike thing downstream, ceph-osd,ceph-mon will
>> check ceph.conf in the local dir(/var/lib/ceph/*/*/ceph.conf) then overwrite
>> existing config values. That's said, it will load system ceph
>> config(/etc/ceph/ceph.conf) file then load local ceph.conf.
>
> The one reservation about this direction is that there are a few devices
> where I'm not sure how to attach the metadata (keyring, conf, and other
> random bits in the osd_data dir) to the device: the SPDK NVMe driver, and
> kinetic.  These are somewhat obscure, but I'm curious if there are other
> ideas floating around about how this should be approached...

Just like BlueStore::_write_bdev_label, I think it should use
BlockDevice::raw_write api which is used to record system metadata
instead of bufferlist::write_fd. So keyring, conf and others could be
reserved at the first section?

But it may exists dependence loop, if we want to read
config/keyring/device_info from osd, we need to know non-kernel device
info at first then we can attach this and do reading.

A possible solution is that we could recover info from non-kernel
device. For example, we still store osd info at os
directory(/var/lib/ceph/osd/ceph-0), but we will write these into
device(including kernel or non-kernel), if os disk failed, we could
use ceph-device-scan tool to discover this and extract necessary info
from this disk into the active os directory?

>
> sage



-- 
Best Regards,

Wheat

Re: /var/lib/ceph/*/*/config

[John Spray]

On Tue, Jan 19, 2016 at 3:15 PM, Sage Weil <sweil@redhat.com> wrote:
> I think we should add an optional 'config' file location to the config
> search path.  This would allow you to put customized options for an
> individual daemon in the daemon data directory.  Or, you could avoid
> putting a config in /etc/ceph entirely, so that the daemon data is
> completely self-contained on the device.  That would streamline boot disk
> failure (just reinstall OS and ceph package) or drive swaps between failed
> host hardware.
>
> What do you think?

Seems very reasonable.

In a related thought, I've been wishing that we could store a config
in the data directory for a different reason.  It would be nice if
daemons could self-set persistent configuration, so that we could make
options set over `tell` (optionally) persistent.  The idea is that if
a management tool wanted to e.g. configure log verbosity, it could do
it purely over ceph commands, rather than needing out-of-band (SSH)
access to modify files on disk (and to reconcile its changes with the
human-editable ceph.conf).

What if we had two (optional) config files in the data dir, one for
human consumption (to makes drive swaps and host rebuilds simpler),
and one machine generated for the daemon to update itself when
requested to remotely?

John

RE: /var/lib/ceph/*/*/config

[Wang, Yaguang]

Before to set per daemon options, is there an approach to get a full set of effective options for configuration?


-yaguang

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of John Spray
Sent: Wednesday, January 20, 2016 6:42 PM
To: Sage Weil
Cc: Ceph Development
Subject: Re: /var/lib/ceph/*/*/config

On Tue, Jan 19, 2016 at 3:15 PM, Sage Weil <sweil@redhat.com> wrote:
> I think we should add an optional 'config' file location to the config 
> search path.  This would allow you to put customized options for an 
> individual daemon in the daemon data directory.  Or, you could avoid 
> putting a config in /etc/ceph entirely, so that the daemon data is 
> completely self-contained on the device.  That would streamline boot 
> disk failure (just reinstall OS and ceph package) or drive swaps 
> between failed host hardware.
>
> What do you think?

Seems very reasonable.

In a related thought, I've been wishing that we could store a config in the data directory for a different reason.  It would be nice if daemons could self-set persistent configuration, so that we could make options set over `tell` (optionally) persistent.  The idea is that if a management tool wanted to e.g. configure log verbosity, it could do it purely over ceph commands, rather than needing out-of-band (SSH) access to modify files on disk (and to reconcile its changes with the human-editable ceph.conf).

What if we had two (optional) config files in the data dir, one for human consumption (to makes drive swaps and host rebuilds simpler), and one machine generated for the daemon to update itself when requested to remotely?

John
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: /var/lib/ceph/*/*/config

[Sage Weil]

Loic pointed out that adding a new config file that is overrides/combines 
with the global config makes our already-complicated config infrastructure 
even more so.  And it turned out to be more complicated to implement than 
I'd hoped.  On the other hand, I think there are valid use-cases for a 
global config, e.g.:

 - most config on the osd disk itself
 - /etc/ceph/ceph.conf containing crush_location and mon_host

...and with a single config I'm not sure how we'd get the datacenter 
location set properly.

Anyway, I have two versions implemented.  The first just adds 
$data_dir/config to the search pathh:

	https://github.com/ceph/ceph/pull/7377

The second adds the extra_config infrastructure.  It's just the last patch 
that is different:

	https://github.com/ceph/ceph/pull/7378

Comments?
sage

Re: /var/lib/ceph/*/*/config

[Wido den Hollander]

On 27-01-16 15:59, Sage Weil wrote:
> Loic pointed out that adding a new config file that is overrides/combines 
> with the global config makes our already-complicated config infrastructure 
> even more so.  And it turned out to be more complicated to implement than 
> I'd hoped.  On the other hand, I think there are valid use-cases for a 
> global config, e.g.:
> 
>  - most config on the osd disk itself
>  - /etc/ceph/ceph.conf containing crush_location and mon_host
> 
> ...and with a single config I'm not sure how we'd get the datacenter 
> location set properly.
> 

Yes, you probably want the mon_host variable in a global config. We
might and I say *might* want to look at a DNS lookup at some time?

A lookup for 'ceph-monitor' in the current DNS search domain. But that
might be offtopic here.

> Anyway, I have two versions implemented.  The first just adds 
> $data_dir/config to the search pathh:
> 
> 	https://github.com/ceph/ceph/pull/7377
> 

I like the first patch, but it doesn't seem to merge the config files,
right?

If a global /etc/ceph/ceph.conf is found it doesn't search any further.

> The second adds the extra_config infrastructure.  It's just the last patch 
> that is different:
> 
> 	https://github.com/ceph/ceph/pull/7378
> 

The --extra-config seems weird to me. Isn't there a way where you supply
--config multiple times with multiple files?

Wido

> Comments?
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: /var/lib/ceph/*/*/config

[Sage Weil]

On Wed, 27 Jan 2016, Wido den Hollander wrote:
> On 27-01-16 15:59, Sage Weil wrote:
> > Loic pointed out that adding a new config file that is overrides/combines 
> > with the global config makes our already-complicated config infrastructure 
> > even more so.  And it turned out to be more complicated to implement than 
> > I'd hoped.  On the other hand, I think there are valid use-cases for a 
> > global config, e.g.:
> > 
> >  - most config on the osd disk itself
> >  - /etc/ceph/ceph.conf containing crush_location and mon_host
> > 
> > ...and with a single config I'm not sure how we'd get the datacenter 
> > location set properly.
> > 
> 
> Yes, you probably want the mon_host variable in a global config. We
> might and I say *might* want to look at a DNS lookup at some time?
> 
> A lookup for 'ceph-monitor' in the current DNS search domain. But that
> might be offtopic here.

Yeah, I think if we make standard practice setting up DNS with an A or 
AAAA record for each monitor then we can avoid a global mon_host.  But 
there's still the crush location piece...
 
> > Anyway, I have two versions implemented.  The first just adds 
> > $data_dir/config to the search pathh:
> > 
> > 	https://github.com/ceph/ceph/pull/7377
> > 
> 
> I like the first patch, but it doesn't seem to merge the config files,
> right?
> 
> If a global /etc/ceph/ceph.conf is found it doesn't search any further.
> 
> > The second adds the extra_config infrastructure.  It's just the last patch 
> > that is different:
> > 
> > 	https://github.com/ceph/ceph/pull/7378
> > 
> 
> The --extra-config seems weird to me. Isn't there a way where you supply
> --config multiple times with multiple files?

Right now config is a search path, and when it finds a match, it stops and 
uses that config file exclusively.  extra_config is a way to merge two of 
them.  I agree it's not ideal...

An alternative approach would be to support 'include <file>' so that the 
$data_dir/config, if present, would include something from /etc/ceph.

sage

Re: /var/lib/ceph/*/*/config

[Wido den Hollander]

On 27-01-16 16:45, Sage Weil wrote:
> On Wed, 27 Jan 2016, Wido den Hollander wrote:
>> On 27-01-16 15:59, Sage Weil wrote:
>>> Loic pointed out that adding a new config file that is overrides/combines 
>>> with the global config makes our already-complicated config infrastructure 
>>> even more so.  And it turned out to be more complicated to implement than 
>>> I'd hoped.  On the other hand, I think there are valid use-cases for a 
>>> global config, e.g.:
>>>
>>>  - most config on the osd disk itself
>>>  - /etc/ceph/ceph.conf containing crush_location and mon_host
>>>
>>> ...and with a single config I'm not sure how we'd get the datacenter 
>>> location set properly.
>>>
>>
>> Yes, you probably want the mon_host variable in a global config. We
>> might and I say *might* want to look at a DNS lookup at some time?
>>
>> A lookup for 'ceph-monitor' in the current DNS search domain. But that
>> might be offtopic here.
> 
> Yeah, I think if we make standard practice setting up DNS with an A or 
> AAAA record for each monitor then we can avoid a global mon_host.  But 
> there's still the crush location piece...
>  

Yes. I created a issue for that: http://tracker.ceph.com/issues/14527

>>> Anyway, I have two versions implemented.  The first just adds 
>>> $data_dir/config to the search pathh:
>>>
>>> 	https://github.com/ceph/ceph/pull/7377
>>>
>>
>> I like the first patch, but it doesn't seem to merge the config files,
>> right?
>>
>> If a global /etc/ceph/ceph.conf is found it doesn't search any further.
>>
>>> The second adds the extra_config infrastructure.  It's just the last patch 
>>> that is different:
>>>
>>> 	https://github.com/ceph/ceph/pull/7378
>>>
>>
>> The --extra-config seems weird to me. Isn't there a way where you supply
>> --config multiple times with multiple files?
> 
> Right now config is a search path, and when it finds a match, it stops and 
> uses that config file exclusively.  extra_config is a way to merge two of 
> them.  I agree it's not ideal...
> 
> An alternative approach would be to support 'include <file>' so that the 
> $data_dir/config, if present, would include something from /etc/ceph.
> 

I think that a include would be much nicer. I filed a issue for that
some time ago: http://tracker.ceph.com/issues/4771

The default config we created for a daemon could be just a one-liner:


/var/lib/ceph/osd/ceph-0/ceph.conf:

include "/etc/ceph/ceph.conf"

Wido

> sage
> 

Re: /var/lib/ceph/*/*/config

[Haomai Wang]

On Thu, Jan 28, 2016 at 12:03 AM, Wido den Hollander <wido@42on.com> wrote:
>
>
> On 27-01-16 16:45, Sage Weil wrote:
>> On Wed, 27 Jan 2016, Wido den Hollander wrote:
>>> On 27-01-16 15:59, Sage Weil wrote:
>>>> Loic pointed out that adding a new config file that is overrides/combines
>>>> with the global config makes our already-complicated config infrastructure
>>>> even more so.  And it turned out to be more complicated to implement than
>>>> I'd hoped.  On the other hand, I think there are valid use-cases for a
>>>> global config, e.g.:
>>>>
>>>>  - most config on the osd disk itself
>>>>  - /etc/ceph/ceph.conf containing crush_location and mon_host
>>>>
>>>> ...and with a single config I'm not sure how we'd get the datacenter
>>>> location set properly.
>>>>
>>>
>>> Yes, you probably want the mon_host variable in a global config. We
>>> might and I say *might* want to look at a DNS lookup at some time?
>>>
>>> A lookup for 'ceph-monitor' in the current DNS search domain. But that
>>> might be offtopic here.
>>
>> Yeah, I think if we make standard practice setting up DNS with an A or
>> AAAA record for each monitor then we can avoid a global mon_host.  But
>> there's still the crush location piece...
>>
>
> Yes. I created a issue for that: http://tracker.ceph.com/issues/14527
>
>>>> Anyway, I have two versions implemented.  The first just adds
>>>> $data_dir/config to the search pathh:
>>>>
>>>>     https://github.com/ceph/ceph/pull/7377
>>>>
>>>
>>> I like the first patch, but it doesn't seem to merge the config files,
>>> right?
>>>
>>> If a global /etc/ceph/ceph.conf is found it doesn't search any further.
>>>
>>>> The second adds the extra_config infrastructure.  It's just the last patch
>>>> that is different:
>>>>
>>>>     https://github.com/ceph/ceph/pull/7378
>>>>
>>>
>>> The --extra-config seems weird to me. Isn't there a way where you supply
>>> --config multiple times with multiple files?
>>
>> Right now config is a search path, and when it finds a match, it stops and
>> uses that config file exclusively.  extra_config is a way to merge two of
>> them.  I agree it's not ideal...
>>
>> An alternative approach would be to support 'include <file>' so that the
>> $data_dir/config, if present, would include something from /etc/ceph.
>>
>
> I think that a include would be much nicer. I filed a issue for that
> some time ago: http://tracker.ceph.com/issues/4771
>
> The default config we created for a daemon could be just a one-liner:
>
>
> /var/lib/ceph/osd/ceph-0/ceph.conf:
>
> include "/etc/ceph/ceph.conf"

The most annoying thing I think is config section and section
priority, I'm not sure
PR(https://github.com/ceph/ceph/pull/7378/files) resolve this problem.

For example, global ceph conf has [global] and [osd.0] section, local
ceph conf has [osd] section, which has higher priority?

Then local ceph conf now will clear global ceph conf section if exists
overlap, it will make thing misunderstanding for users.


>
> Wido
>
>> sage
>>



-- 
Best Regards,

Wheat