host settings in ceph.conf when ipaddress != hostname

host settings in ceph.conf when ipaddress != hostname

[Travis Rhoden]

Hi folks,

I've seen a couple threads in the past about /etc/init.d/ceph not
working right when the output of 'hostname' did not match the 'host'
entry in ceph.conf.  I'm hoping you can advise me on how to get the
following setup working.

I have multiple nodes that both a GigE connection and a secondary
high-speed connection (either 10GbE or IPoIB -- I've got both setups).
 I want to use the high-speed connection for everything (both node
replication, and client<-->node traffic), but the hostnames on the
storage nodes are mapped to the GbE address.  You can think of the
following being in /etc/hosts:

10.1.0.1   ceph0
10.30.0.1 ceph0-ib


If I ssh to ceph0 and type "hostname", the output will be 'ceph0'. I
think that part is pretty straightforward.  However, I have entries
like this in ceph.conf:

[mon.a]
        host = ceph0-ib
        mon addr = 10.30.0.1:6789

[osd.0]
        host = ceph0-ib


Because these hostnames don't match, things like "ceph -a status"
don't work as intended.  Although the cluster functions fine, and uses
the high-speed network.  Is there a different way I should populate
ceph.conf to get everything to work correct?

For the [mon.x] section, how is the 'host' param used?  is it mapped
to an IP address?  If it is, why the 'mon addr' field?  If it is not,
then I could just use 'ceph0' instead of 'ceph0-ib' and things would
be happy.

For the [osd.x] section I have the same question.  Is "host" mapped to
an IP address?  I would assume so, and thus I am force to put the -ib
version of the hostname.  Is there something else I could do in this
scenario?

Thanks,

 - Travis

Re: host settings in ceph.conf when ipaddress != hostname

[Tommi Virtanen]

On Thu, Aug 16, 2012 at 12:38 PM, Travis Rhoden <trhoden@gmail.com> wrote:
> I've seen a couple threads in the past about /etc/init.d/ceph not
> working right when the output of 'hostname' did not match the 'host'
> entry in ceph.conf.  I'm hoping you can advise me on how to get the
> following setup working.
>
> I have multiple nodes that both a GigE connection and a secondary
> high-speed connection (either 10GbE or IPoIB -- I've got both setups).
>  I want to use the high-speed connection for everything (both node
> replication, and client<-->node traffic), but the hostnames on the
> storage nodes are mapped to the GbE address.  You can think of the
> following being in /etc/hosts:
>
> 10.1.0.1   ceph0
> 10.30.0.1 ceph0-ib
>
>
> If I ssh to ceph0 and type "hostname", the output will be 'ceph0'. I
> think that part is pretty straightforward.  However, I have entries
> like this in ceph.conf:
>
> [mon.a]
>         host = ceph0-ib
>         mon addr = 10.30.0.1:6789
>
> [osd.0]
>         host = ceph0-ib
>
>
> Because these hostnames don't match, things like "ceph -a status"
> don't work as intended.  Although the cluster functions fine, and uses
> the high-speed network.  Is there a different way I should populate
> ceph.conf to get everything to work correct?

ceph.conf host= entries must match the output of "hostname" (strictly,
up to first dot of "hostname -f", which isn't exactly the same as
"hostname", but I'm too late to that fight); that's what the sysvinit
script assumes, and if it ain't right, it won't start your daemons.

It sounds like you want to set set public_network and cluster_network
in [global] both to 10.30.0.0/8.

Re: host settings in ceph.conf when ipaddress != hostname

[Smart Weblications GmbH - Florian Wiessner]

Am 16.08.2012 21:38, schrieb Travis Rhoden:
> Hi folks,
> 
> I've seen a couple threads in the past about /etc/init.d/ceph not
> working right when the output of 'hostname' did not match the 'host'
> entry in ceph.conf.  I'm hoping you can advise me on how to get the
> following setup working.
> 

[...]

> 
> For the [osd.x] section I have the same question.  Is "host" mapped to
> an IP address?  I would assume so, and thus I am force to put the -ib
> version of the hostname.  Is there something else I could do in this
> scenario?
> 


I was also confused alot by this hostname settings - i think it is bad designed
- why not configure the daemons as any other linux daemon - just specifiy
ip:port to use for mon,osd and mds and done? Is there anything else other than
the initscript which needs hostnames?


-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: host settings in ceph.conf when ipaddress != hostname

[Smart Weblications GmbH]

Am 16.08.2012 21:56, schrieb Tommi Virtanen:

> It sounds like you want to set set public_network and cluster_network
> in [global] both to 10.30.0.0/8.

10.0.0.0/8 or 10.30.0.0/16, nor? ;-)


-- 

Mit freundlichen Grüßen,


Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: host settings in ceph.conf when ipaddress != hostname

[Tommi Virtanen]

2012/8/16 Smart Weblications GmbH <info@smart-weblications.de>:
>> It sounds like you want to set set public_network and cluster_network
>> in [global] both to 10.30.0.0/8.
>
> 10.0.0.0/8 or 10.30.0.0/16, nor? ;-)

Yeah /16 probably. He only gave example IPs from the two networks, so
not completely sure.

Re: host settings in ceph.conf when ipaddress != hostname

[Travis Rhoden]

Thanks for the responses.

Is there a link to the docs where the config file options for
public_network, cluster_network, mon host, cluster addr are listed and
explained?  I looked for them, but couldn't find anything. I just come
across different mailing list entries and blog entries that keep using
options I've never seen before.  I expected to find these options
here: http://ceph.com/docs/master/config-cluster/ceph-conf/  but no
luck.

Looking at a blog post like this:
http://www.sebastien-han.fr/blog/2012/07/29/tip-ceph-public-slash-private-network-configuration/,
I become more confused.  Obviously I don't understand the relationship
between OSD and MONs correctly.  If I place all my OSDs on a private
network and the MONs on a public one, how do clients reach the OSDs?
I didn't think that data was funneled through the MONs.  Is that
incorrect?  I thought my OSDs had to be reachable by anyone trying to
mount/map an RBD, for example.

 - Travis

2012/8/16 Smart Weblications GmbH <info@smart-weblications.de>:
> Am 16.08.2012 21:56, schrieb Tommi Virtanen:
>
>> It sounds like you want to set set public_network and cluster_network
>> in [global] both to 10.30.0.0/8.
>
> 10.0.0.0/8 or 10.30.0.0/16, nor? ;-)
>
>
> --
>
> Mit freundlichen Grüßen,
>
>
> Smart Weblications GmbH
> Martinsberger Str. 1
> D-95119 Naila
>
> fon.: +49 9282 9638 200
> fax.: +49 9282 9638 205
> 24/7: +49 900 144 000 00 - 0,99 EUR/Min*
> http://www.smart-weblications.de
>
> --
> Sitz der Gesellschaft: Naila
> Geschäftsführer: Florian Wiessner
> HRB-Nr.: HRB 3840 Amtsgericht Hof
> *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: host settings in ceph.conf when ipaddress != hostname

[Tommi Virtanen]

On Thu, Aug 16, 2012 at 1:13 PM, Travis Rhoden <trhoden@gmail.com> wrote:
> Is there a link to the docs where the config file options for
> public_network, cluster_network, mon host, cluster addr are listed and
> explained?  I looked for them, but couldn't find anything. I just come
> across different mailing list entries and blog entries that keep using
> options I've never seen before.  I expected to find these options
> here: http://ceph.com/docs/master/config-cluster/ceph-conf/  but no
> luck.

Yeah those don't seem to be well documented.

The only doc mentions of the config options are in:
http://ceph.com/docs/master/ops/manage/grow/mon/
http://ceph.com/docs/master/man/8/ceph-mon/
http://ceph.com/docs/master/dev/mon-bootstrap/

> Looking at a blog post like this:
> http://www.sebastien-han.fr/blog/2012/07/29/tip-ceph-public-slash-private-network-configuration/,
> I become more confused.  Obviously I don't understand the relationship
> between OSD and MONs correctly.  If I place all my OSDs on a private
> network and the MONs on a public one, how do clients reach the OSDs?
> I didn't think that data was funneled through the MONs.  Is that
> incorrect?  I thought my OSDs had to be reachable by anyone trying to
> mount/map an RBD, for example.

That setup is used with multihomed hosts. Two network interfaces, with
often the "cluster" one being significantly faster (replication causes
write amplification). Clients talk to mons and osds via the public
network, osd<->osd traffic is on the cluster network.

Re: host settings in ceph.conf when ipaddress != hostname

[Travis Rhoden]

Tommi,

Thanks, I'm beginning to understand.  I think the only piece I'm
missing (or rather, that I didn't understand before) is that if I use
the options for cluster_network and private_network in [global] then
the OSDs and MONs will automatically bind to the IP address/interface
that matches that CIDR when they start up?  Thereby negating my having
to put a specific addr in each [osd.x] or [mon.x] section.  Do I
understand that correctly?

If I do, then your initial suggestion of setting those fields is
exactly what I want.

 - Travis

On Thu, Aug 16, 2012 at 4:22 PM, Tommi Virtanen <tv@inktank.com> wrote:
> On Thu, Aug 16, 2012 at 1:13 PM, Travis Rhoden <trhoden@gmail.com> wrote:
>> Is there a link to the docs where the config file options for
>> public_network, cluster_network, mon host, cluster addr are listed and
>> explained?  I looked for them, but couldn't find anything. I just come
>> across different mailing list entries and blog entries that keep using
>> options I've never seen before.  I expected to find these options
>> here: http://ceph.com/docs/master/config-cluster/ceph-conf/  but no
>> luck.
>
> Yeah those don't seem to be well documented.
>
> The only doc mentions of the config options are in:
> http://ceph.com/docs/master/ops/manage/grow/mon/
> http://ceph.com/docs/master/man/8/ceph-mon/
> http://ceph.com/docs/master/dev/mon-bootstrap/
>
>> Looking at a blog post like this:
>> http://www.sebastien-han.fr/blog/2012/07/29/tip-ceph-public-slash-private-network-configuration/,
>> I become more confused.  Obviously I don't understand the relationship
>> between OSD and MONs correctly.  If I place all my OSDs on a private
>> network and the MONs on a public one, how do clients reach the OSDs?
>> I didn't think that data was funneled through the MONs.  Is that
>> incorrect?  I thought my OSDs had to be reachable by anyone trying to
>> mount/map an RBD, for example.
>
> That setup is used with multihomed hosts. Two network interfaces, with
> often the "cluster" one being significantly faster (replication causes
> write amplification). Clients talk to mons and osds via the public
> network, osd<->osd traffic is on the cluster network.

Re: host settings in ceph.conf when ipaddress != hostname

[Travis Rhoden]

Tommi,

Thanks again for your help earlier.  setting the cluster and public
CIDRs in the config file worked very well.

However, the output of "service ceph -a status" still was giving me output like:

root@ceph0:~# /etc/init.d/ceph -a status mon
=== mon.a ===
mon.a: running 0.48.1argonaut
=== mon.b ===
mon.b: running unknown
=== mon.c ===
mon.c: running unknown

I was surprised by this, because the hostname problem was no longer an
issue.  I ran the script with "bash -x", and saw this:

++ /usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n mon.b 'admin socket'
++ eval echo -n /var/run/ceph/ceph-mon.b.asok
+++ echo -n /var/run/ceph/ceph-mon.b.asok
+ eval 'asok="/var/run/ceph/ceph-mon.b.asok"'
++ asok=/var/run/ceph/ceph-mon.b.asok
++ /usr/bin/ceph --admin-daemon /var/run/ceph/ceph-mon.b.asok version
++ echo unknown
+ version=unknown
+ echo 'mon.b: running unknown'
mon.b: running unknown

That made me try this:

root@ceph0:~# ssh ceph1 /usr/bin/ceph --admin-daemon
/var/run/ceph/ceph-mon.b.asok version
0.48.1argonaut

Which, as you can see, returns the version just fine.  So, I noticed
the problem was that the script was not using SSH for getting the
version from remote hosts.

I'm not sure if other people have seen this or not. I was able to
modify the code to make things work.  I'm sending a patch shortly.
Hopefully I did things correctly, and this is indeed something that
needs to be fixed (rather than something just being wrong in my
setup).

After the patch, I see nice output:

root@ceph0:~# /etc/init.d/ceph -a status
=== mon.a ===
mon.a: running 0.48.1argonaut
=== mon.b ===
mon.b: running 0.48.1argonaut
=== mon.c ===
mon.c: running 0.48.1argonaut



On Thu, Aug 16, 2012 at 4:26 PM, Travis Rhoden <trhoden@gmail.com> wrote:
> Tommi,
>
> Thanks, I'm beginning to understand.  I think the only piece I'm
> missing (or rather, that I didn't understand before) is that if I use
> the options for cluster_network and private_network in [global] then
> the OSDs and MONs will automatically bind to the IP address/interface
> that matches that CIDR when they start up?  Thereby negating my having
> to put a specific addr in each [osd.x] or [mon.x] section.  Do I
> understand that correctly?
>
> If I do, then your initial suggestion of setting those fields is
> exactly what I want.
>
>  - Travis
>
> On Thu, Aug 16, 2012 at 4:22 PM, Tommi Virtanen <tv@inktank.com> wrote:
>> On Thu, Aug 16, 2012 at 1:13 PM, Travis Rhoden <trhoden@gmail.com> wrote:
>>> Is there a link to the docs where the config file options for
>>> public_network, cluster_network, mon host, cluster addr are listed and
>>> explained?  I looked for them, but couldn't find anything. I just come
>>> across different mailing list entries and blog entries that keep using
>>> options I've never seen before.  I expected to find these options
>>> here: http://ceph.com/docs/master/config-cluster/ceph-conf/  but no
>>> luck.
>>
>> Yeah those don't seem to be well documented.
>>
>> The only doc mentions of the config options are in:
>> http://ceph.com/docs/master/ops/manage/grow/mon/
>> http://ceph.com/docs/master/man/8/ceph-mon/
>> http://ceph.com/docs/master/dev/mon-bootstrap/
>>
>>> Looking at a blog post like this:
>>> http://www.sebastien-han.fr/blog/2012/07/29/tip-ceph-public-slash-private-network-configuration/,
>>> I become more confused.  Obviously I don't understand the relationship
>>> between OSD and MONs correctly.  If I place all my OSDs on a private
>>> network and the MONs on a public one, how do clients reach the OSDs?
>>> I didn't think that data was funneled through the MONs.  Is that
>>> incorrect?  I thought my OSDs had to be reachable by anyone trying to
>>> mount/map an RBD, for example.
>>
>> That setup is used with multihomed hosts. Two network interfaces, with
>> often the "cluster" one being significantly faster (replication causes
>> write amplification). Clients talk to mons and osds via the public
>> network, osd<->osd traffic is on the cluster network.