* problem with ceph init scripts
@ 2011-02-16 14:44 Upendra Moturi
2011-02-16 21:41 ` Colin McCabe
0 siblings, 1 reply; 18+ messages in thread
From: Upendra Moturi @ 2011-02-16 14:44 UTC (permalink / raw)
To: sage, ceph-devel, Suresh Mandava
Hi,
With ceph 0.24.2 and linux 2.6.38-2-generic(ubuntu) ,i face problems
with ceph init scripts
/etc/init.d/ceph -a start ----works fine
But if we want to start a particular osd or mon or mds ,its not
working and there is no error
eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
/etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
on current node only
where as
/etc/init.d/ceph -a killall works fine.
--
Thanks and Regards,
Upendra.M
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-16 14:44 problem with ceph init scripts Upendra Moturi
@ 2011-02-16 21:41 ` Colin McCabe
2011-02-16 22:00 ` Tommi Virtanen
2011-02-16 22:01 ` Colin McCabe
0 siblings, 2 replies; 18+ messages in thread
From: Colin McCabe @ 2011-02-16 21:41 UTC (permalink / raw)
To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava
On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> But if we want to start a particular osd or mon or mds ,its not
> working and there is no error
> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
That is expected, unless you are running init-ceph on the same node as
osd1 is on.
It might be nice to have some kind of interface like "run command X on
osd1", but init-ceph is not that.
> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
> on current node only
> where as
> /etc/init.d/ceph -a killall works fine.
That sounds like a bug. I'll see if I can fix it.
cheers,
Colin
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-16 21:41 ` Colin McCabe
@ 2011-02-16 22:00 ` Tommi Virtanen
2011-02-16 22:01 ` Colin McCabe
1 sibling, 0 replies; 18+ messages in thread
From: Tommi Virtanen @ 2011-02-16 22:00 UTC (permalink / raw)
To: Colin McCabe; +Cc: Upendra Moturi, sage, ceph-devel, Suresh Mandava
On Wed, Feb 16, 2011 at 01:41:44PM -0800, Colin McCabe wrote:
> It might be nice to have some kind of interface like "run command X on
> osd1", but init-ceph is not that.
For general cluster administration tasks, let me recommend tools such
as:
dsh: like "ssh HOST COMMAND" but does multiple hosts at once
http://www.netfort.gr.jp/~dancer/software/dsh.html
Fabric: remote deployment and remote command running utility
(more oriented toward scripting a solution than ad hoc)
http://fabfile.org/
Puppet, Chef, Kokki: more full-blown cluster management solutions;
based on the idea of writing recipes and running them automatically
http://www.puppetlabs.com/
http://www.opscode.com/chef/
http://samuelks.com/kokki/
--
:(){ :|:&};:
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-16 21:41 ` Colin McCabe
2011-02-16 22:00 ` Tommi Virtanen
@ 2011-02-16 22:01 ` Colin McCabe
2011-02-17 7:01 ` Upendra Moturi
1 sibling, 1 reply; 18+ messages in thread
From: Colin McCabe @ 2011-02-16 22:01 UTC (permalink / raw)
To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava
On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>> But if we want to start a particular osd or mon or mds ,its not
>> working and there is no error
>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>
> That is expected, unless you are running init-ceph on the same node as
> osd1 is on.
>
> It might be nice to have some kind of interface like "run command X on
> osd1", but init-ceph is not that.
>
>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>> on current node only
>> where as
>> /etc/init.d/ceph -a killall works fine.
>
> That sounds like a bug. I'll see if I can fix it.
I'm afraid I can't reproduce this.
I ran /etc/init.d/ceph -a stop
and it stopped ceph daemons running on remote nodes too. Looking at
the code, it looks correct.
Colin
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-16 22:01 ` Colin McCabe
@ 2011-02-17 7:01 ` Upendra Moturi
2011-02-17 7:36 ` Colin McCabe
0 siblings, 1 reply; 18+ messages in thread
From: Upendra Moturi @ 2011-02-17 7:01 UTC (permalink / raw)
To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava
Hi Colin
I am using
ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
Installed ceph form apt source.
With above configurations i still able yo reproduce.
Can you please share me ur configurations?
On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>> But if we want to start a particular osd or mon or mds ,its not
>>> working and there is no error
>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>
>> That is expected, unless you are running init-ceph on the same node as
>> osd1 is on.
>>
>> It might be nice to have some kind of interface like "run command X on
>> osd1", but init-ceph is not that.
>>
>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>> on current node only
>>> where as
>>> /etc/init.d/ceph -a killall works fine.
>>
>> That sounds like a bug. I'll see if I can fix it.
>
> I'm afraid I can't reproduce this.
>
> I ran /etc/init.d/ceph -a stop
>
> and it stopped ceph daemons running on remote nodes too. Looking at
> the code, it looks correct.
>
> Colin
>
--
Thanks and Regards,
Upendra.M
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-17 7:01 ` Upendra Moturi
@ 2011-02-17 7:36 ` Colin McCabe
2011-02-17 9:55 ` Upendra Moturi
0 siblings, 1 reply; 18+ messages in thread
From: Colin McCabe @ 2011-02-17 7:36 UTC (permalink / raw)
To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava
I'm using head of line from the master branch. But that particular
code hasn't changed since January, which is 0.24.2 is from.
In my ceph.conf, I just had an osd that was on a remote machine, and
everything else local.
If you could post your ceph.conf here or in IRC, perhaps we might spot
an issue that's causing the problems that you see.
Colin
On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> Hi Colin
> I am using
> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>
> Installed ceph form apt source.
>
> With above configurations i still able yo reproduce.
> Can you please share me ur configurations?
>
> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>> But if we want to start a particular osd or mon or mds ,its not
>>>> working and there is no error
>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>
>>> That is expected, unless you are running init-ceph on the same node as
>>> osd1 is on.
>>>
>>> It might be nice to have some kind of interface like "run command X on
>>> osd1", but init-ceph is not that.
>>>
>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>> on current node only
>>>> where as
>>>> /etc/init.d/ceph -a killall works fine.
>>>
>>> That sounds like a bug. I'll see if I can fix it.
>>
>> I'm afraid I can't reproduce this.
>>
>> I ran /etc/init.d/ceph -a stop
>>
>> and it stopped ceph daemons running on remote nodes too. Looking at
>> the code, it looks correct.
>>
>> Colin
>>
>
>
>
> --
> Thanks and Regards,
> Upendra.M
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-17 7:36 ` Colin McCabe
@ 2011-02-17 9:55 ` Upendra Moturi
2011-02-18 7:05 ` Colin McCabe
0 siblings, 1 reply; 18+ messages in thread
From: Upendra Moturi @ 2011-02-17 9:55 UTC (permalink / raw)
To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava
Hi Colin
Here is my ceph.conf :
[global]
pid file = /var/run/ceph/$name.pid
debug ms = 1
[mon]
mon data = /data/mon$id
[mon.0]
host = ceph0
mon addr = 192.168.155.5:6789
[mon.1]
host = ceph1
mon addr = 192.168.155.6:6789
[mon.2]
host = ceph2
mon addr = 192.168.155.7:6789
[mds]
[mds0]
host = ceph0
[mds1]
host = ceph1
[osd]
sudo = true
osd data = /data/osd$id
osd journal = /data/osd$id/journal
osd journal size = 512
osd use stale snap = true
[osd0]
host = ceph0
btrfs devs = /dev/sdb
[osd1]
host = ceph1
btrfs devs = /dev/sdb
[osd2]
host = ceph2
btrfs devs = /dev/sdb
On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> I'm using head of line from the master branch. But that particular
> code hasn't changed since January, which is 0.24.2 is from.
>
> In my ceph.conf, I just had an osd that was on a remote machine, and
> everything else local.
>
> If you could post your ceph.conf here or in IRC, perhaps we might spot
> an issue that's causing the problems that you see.
>
> Colin
>
>
> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>> Hi Colin
>> I am using
>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>
>> Installed ceph form apt source.
>>
>> With above configurations i still able yo reproduce.
>> Can you please share me ur configurations?
>>
>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>> working and there is no error
>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>
>>>> That is expected, unless you are running init-ceph on the same node as
>>>> osd1 is on.
>>>>
>>>> It might be nice to have some kind of interface like "run command X on
>>>> osd1", but init-ceph is not that.
>>>>
>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>> on current node only
>>>>> where as
>>>>> /etc/init.d/ceph -a killall works fine.
>>>>
>>>> That sounds like a bug. I'll see if I can fix it.
>>>
>>> I'm afraid I can't reproduce this.
>>>
>>> I ran /etc/init.d/ceph -a stop
>>>
>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>> the code, it looks correct.
>>>
>>> Colin
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Upendra.M
>>
>
--
Thanks and Regards,
Upendra.M
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-17 9:55 ` Upendra Moturi
@ 2011-02-18 7:05 ` Colin McCabe
2011-02-18 14:11 ` Upendra Moturi
0 siblings, 1 reply; 18+ messages in thread
From: Colin McCabe @ 2011-02-18 7:05 UTC (permalink / raw)
To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava
Hi Upendra,
Are you running init-ceph from the source directory? If you do that,
it will use the ceph.conf in the source directory itself, which is
probably not what you want. So it might be good to double-check that.
If all else fails, running init-ceph with -x will show you exactly
what the script is doing. If all goes well, its exit status should be
0. Are you getting exit status 0?
Colin
On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> Hi Colin
>
> Here is my ceph.conf :
>
> [global]
> pid file = /var/run/ceph/$name.pid
> debug ms = 1
> [mon]
> mon data = /data/mon$id
> [mon.0]
> host = ceph0
> mon addr = 192.168.155.5:6789
> [mon.1]
> host = ceph1
> mon addr = 192.168.155.6:6789
> [mon.2]
> host = ceph2
> mon addr = 192.168.155.7:6789
> [mds]
>
> [mds0]
> host = ceph0
> [mds1]
> host = ceph1
>
> [osd]
> sudo = true
> osd data = /data/osd$id
> osd journal = /data/osd$id/journal
> osd journal size = 512
> osd use stale snap = true
> [osd0]
> host = ceph0
> btrfs devs = /dev/sdb
> [osd1]
> host = ceph1
> btrfs devs = /dev/sdb
> [osd2]
> host = ceph2
> btrfs devs = /dev/sdb
>
>
>
>
> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>> I'm using head of line from the master branch. But that particular
>> code hasn't changed since January, which is 0.24.2 is from.
>>
>> In my ceph.conf, I just had an osd that was on a remote machine, and
>> everything else local.
>>
>> If you could post your ceph.conf here or in IRC, perhaps we might spot
>> an issue that's causing the problems that you see.
>>
>> Colin
>>
>>
>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>> Hi Colin
>>> I am using
>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>>
>>> Installed ceph form apt source.
>>>
>>> With above configurations i still able yo reproduce.
>>> Can you please share me ur configurations?
>>>
>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>>> working and there is no error
>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>>
>>>>> That is expected, unless you are running init-ceph on the same node as
>>>>> osd1 is on.
>>>>>
>>>>> It might be nice to have some kind of interface like "run command X on
>>>>> osd1", but init-ceph is not that.
>>>>>
>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>>> on current node only
>>>>>> where as
>>>>>> /etc/init.d/ceph -a killall works fine.
>>>>>
>>>>> That sounds like a bug. I'll see if I can fix it.
>>>>
>>>> I'm afraid I can't reproduce this.
>>>>
>>>> I ran /etc/init.d/ceph -a stop
>>>>
>>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>>> the code, it looks correct.
>>>>
>>>> Colin
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Upendra.M
>>>
>>
>
>
>
> --
> Thanks and Regards,
> Upendra.M
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-18 7:05 ` Colin McCabe
@ 2011-02-18 14:11 ` Upendra Moturi
2011-02-26 0:28 ` Colin McCabe
2011-02-26 0:28 ` Gregory Farnum
0 siblings, 2 replies; 18+ messages in thread
From: Upendra Moturi @ 2011-02-18 14:11 UTC (permalink / raw)
To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava
Hi Colin
I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
i am using the default init script.(Found at /etc/init.d/ceph)
Regarding issue 1)
On the osd node I tried to start that osd but it did not work
Steps followed
1) Started 3 nodes(ceph.conf is same as i sent earlier)
2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
3)/etc/init.d/ceph start osd0 (Did this on first osd)
4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
5)/etc/init.d/ceph -a start ---- shows all process on all nodes
Regading Issue 2
It says -x option is not available
Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
tried /etc/init.d/ceph -x -a stop
but nothing worked.
Then tried with /etc/init.d/ceph -a -v stop this also did not stop
ceph on all nodes but got the output as
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
=== mon.0 ===
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
"/var/run/ceph/mon.0.pid"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
--- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
"/var/lock/subsys/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
-c unlimited ; while [ 1 ]; do
[ -e /var/run/ceph/mon.0.pid ] || break
pid=`cat /var/run/ceph/mon.0.pid`
while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
cmd="kill $pid"
echo -n $cmd...
$cmd
sleep 1
continue
done
break
done"
kill 1668...done
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
=== mon.1 ===
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
"/var/run/ceph/mon.1.pid"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
--- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
"/var/lock/subsys/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
-c unlimited ; while [ 1 ]; do
[ -e /var/run/ceph/mon.1.pid ] || break
pid=`cat /var/run/ceph/mon.1.pid`
while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
cmd="kill $pid"
echo -n $cmd...
$cmd
sleep 1
continue
done
break
done"
done
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
=== mon.2 ===
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
"/var/run/ceph/mon.2.pid"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
--- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
"/var/lock/subsys/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
-c unlimited ; while [ 1 ]; do
[ -e /var/run/ceph/mon.2.pid ] || break
pid=`cat /var/run/ceph/mon.2.pid`
while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
cmd="kill $pid"
echo -n $cmd...
$cmd
sleep 1
continue
done
break
done"
done
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
=== mds.0 ===
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
"/var/run/ceph/mds.0.pid"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
--- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
"/var/lock/subsys/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
unlimited ; while [ 1 ]; do
[ -e /var/run/ceph/mds.0.pid ] || break
pid=`cat /var/run/ceph/mds.0.pid`
while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
cmd="kill $pid"
echo -n $cmd...
$cmd
sleep 1
continue
done
break
done"
kill 1844...done
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
=== mds.1 ===
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
"/var/run/ceph/mds.1.pid"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
--- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
"/var/lock/subsys/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
unlimited ; while [ 1 ]; do
[ -e /var/run/ceph/mds.1.pid ] || break
pid=`cat /var/run/ceph/mds.1.pid`
while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
cmd="kill $pid"
echo -n $cmd...
$cmd
sleep 1
continue
done
break
done"
done
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
=== osd.0 ===
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
"/var/run/ceph/osd.0.pid"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
--- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
"/var/lock/subsys/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
unlimited ; while [ 1 ]; do
[ -e /var/run/ceph/osd.0.pid ] || break
pid=`cat /var/run/ceph/osd.0.pid`
while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
cmd="kill $pid"
echo -n $cmd...
$cmd
sleep 1
continue
done
break
done"
kill 2033...done
Unmounting Btrfs on ceph0:/data/osd0
--- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
/data/osd0 || true"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
=== osd.1 ===
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
"/var/run/ceph/osd.1.pid"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
--- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
"/var/lock/subsys/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
unlimited ; while [ 1 ]; do
[ -e /var/run/ceph/osd.1.pid ] || break
pid=`cat /var/run/ceph/osd.1.pid`
while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
cmd="kill $pid"
echo -n $cmd...
$cmd
sleep 1
continue
done
break
done"
done
Unmounting Btrfs on ceph1:/data/osd1
--- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
/data/osd1 || true"
umount: /data/osd1: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
=== osd.2 ===
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
"/var/run/ceph/osd.2.pid"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
--- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
"/var/lock/subsys/ceph"
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
unlimited ; while [ 1 ]; do
[ -e /var/run/ceph/osd.2.pid ] || break
pid=`cat /var/run/ceph/osd.2.pid`
while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
cmd="kill $pid"
echo -n $cmd...
$cmd
sleep 1
continue
done
break
done"
done
Unmounting Btrfs on ceph2:/data/osd2
--- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
/data/osd2 || true"
umount: /data/osd2: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> Hi Upendra,
>
> Are you running init-ceph from the source directory? If you do that,
> it will use the ceph.conf in the source directory itself, which is
> probably not what you want. So it might be good to double-check that.
>
> If all else fails, running init-ceph with -x will show you exactly
> what the script is doing. If all goes well, its exit status should be
> 0. Are you getting exit status 0?
>
> Colin
>
>
> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>> Hi Colin
>>
>> Here is my ceph.conf :
>>
>> [global]
>> pid file = /var/run/ceph/$name.pid
>> debug ms = 1
>> [mon]
>> mon data = /data/mon$id
>> [mon.0]
>> host = ceph0
>> mon addr = 192.168.155.5:6789
>> [mon.1]
>> host = ceph1
>> mon addr = 192.168.155.6:6789
>> [mon.2]
>> host = ceph2
>> mon addr = 192.168.155.7:6789
>> [mds]
>>
>> [mds0]
>> host = ceph0
>> [mds1]
>> host = ceph1
>>
>> [osd]
>> sudo = true
>> osd data = /data/osd$id
>> osd journal = /data/osd$id/journal
>> osd journal size = 512
>> osd use stale snap = true
>> [osd0]
>> host = ceph0
>> btrfs devs = /dev/sdb
>> [osd1]
>> host = ceph1
>> btrfs devs = /dev/sdb
>> [osd2]
>> host = ceph2
>> btrfs devs = /dev/sdb
>>
>>
>>
>>
>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>> I'm using head of line from the master branch. But that particular
>>> code hasn't changed since January, which is 0.24.2 is from.
>>>
>>> In my ceph.conf, I just had an osd that was on a remote machine, and
>>> everything else local.
>>>
>>> If you could post your ceph.conf here or in IRC, perhaps we might spot
>>> an issue that's causing the problems that you see.
>>>
>>> Colin
>>>
>>>
>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>> Hi Colin
>>>> I am using
>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>>>
>>>> Installed ceph form apt source.
>>>>
>>>> With above configurations i still able yo reproduce.
>>>> Can you please share me ur configurations?
>>>>
>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>>>> working and there is no error
>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>>>
>>>>>> That is expected, unless you are running init-ceph on the same node as
>>>>>> osd1 is on.
>>>>>>
>>>>>> It might be nice to have some kind of interface like "run command X on
>>>>>> osd1", but init-ceph is not that.
>>>>>>
>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>>>> on current node only
>>>>>>> where as
>>>>>>> /etc/init.d/ceph -a killall works fine.
>>>>>>
>>>>>> That sounds like a bug. I'll see if I can fix it.
>>>>>
>>>>> I'm afraid I can't reproduce this.
>>>>>
>>>>> I ran /etc/init.d/ceph -a stop
>>>>>
>>>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>>>> the code, it looks correct.
>>>>>
>>>>> Colin
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Upendra.M
>>>>
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Upendra.M
>>
>
--
Thanks and Regards,
Upendra.M
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-18 14:11 ` Upendra Moturi
@ 2011-02-26 0:28 ` Colin McCabe
2011-02-28 12:00 ` Upendra Moturi
2011-02-26 0:28 ` Gregory Farnum
1 sibling, 1 reply; 18+ messages in thread
From: Colin McCabe @ 2011-02-26 0:28 UTC (permalink / raw)
To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava
Hi Upendra,
Based on the output you posted, init-ceph is doing something on every
node. However, I only see a kill for certain nodes.
Does /var/run/ceph/ exist on all nodes, or just some of them? Does the
appropriate pid file exist on all nodes? What happens when you ssh in
to those nodes manually and run init-ceph stop?
Colin
On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> Hi Colin
>
> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
> i am using the default init script.(Found at /etc/init.d/ceph)
>
> Regarding issue 1)
>
> On the osd node I tried to start that osd but it did not work
> Steps followed
>
> 1) Started 3 nodes(ceph.conf is same as i sent earlier)
>
> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
>
> 3)/etc/init.d/ceph start osd0 (Did this on first osd)
>
> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
>
> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes
>
> Regading Issue 2
> It says -x option is not available
>
> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
> tried /etc/init.d/ceph -x -a stop
> but nothing worked.
>
> Then tried with /etc/init.d/ceph -a -v stop this also did not stop
> ceph on all nodes but got the output as
>
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
> === mon.0 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
> "/var/run/ceph/mon.0.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
> -c unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mon.0.pid ] || break
> pid=`cat /var/run/ceph/mon.0.pid`
> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> kill 1668...done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
> === mon.1 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
> "/var/run/ceph/mon.1.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
> -c unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mon.1.pid ] || break
> pid=`cat /var/run/ceph/mon.1.pid`
> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
> === mon.2 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
> "/var/run/ceph/mon.2.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
> -c unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mon.2.pid ] || break
> pid=`cat /var/run/ceph/mon.2.pid`
> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
> === mds.0 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
> "/var/run/ceph/mds.0.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mds.0.pid ] || break
> pid=`cat /var/run/ceph/mds.0.pid`
> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> kill 1844...done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
> === mds.1 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
> "/var/run/ceph/mds.1.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mds.1.pid ] || break
> pid=`cat /var/run/ceph/mds.1.pid`
> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
> === osd.0 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
> "/var/run/ceph/osd.0.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/osd.0.pid ] || break
> pid=`cat /var/run/ceph/osd.0.pid`
> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> kill 2033...done
> Unmounting Btrfs on ceph0:/data/osd0
> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
> /data/osd0 || true"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
> === osd.1 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
> "/var/run/ceph/osd.1.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/osd.1.pid ] || break
> pid=`cat /var/run/ceph/osd.1.pid`
> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> Unmounting Btrfs on ceph1:/data/osd1
> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
> /data/osd1 || true"
> umount: /data/osd1: device is busy.
> (In some cases useful info about processes that use
> the device is found by lsof(8) or fuser(1))
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
> === osd.2 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
> "/var/run/ceph/osd.2.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/osd.2.pid ] || break
> pid=`cat /var/run/ceph/osd.2.pid`
> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> Unmounting Btrfs on ceph2:/data/osd2
> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
> /data/osd2 || true"
> umount: /data/osd2: device is busy.
> (In some cases useful info about processes that use
> the device is found by lsof(8) or fuser(1))
>
>
>
>
> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>> Hi Upendra,
>>
>> Are you running init-ceph from the source directory? If you do that,
>> it will use the ceph.conf in the source directory itself, which is
>> probably not what you want. So it might be good to double-check that.
>>
>> If all else fails, running init-ceph with -x will show you exactly
>> what the script is doing. If all goes well, its exit status should be
>> 0. Are you getting exit status 0?
>>
>> Colin
>>
>>
>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>> Hi Colin
>>>
>>> Here is my ceph.conf :
>>>
>>> [global]
>>> pid file = /var/run/ceph/$name.pid
>>> debug ms = 1
>>> [mon]
>>> mon data = /data/mon$id
>>> [mon.0]
>>> host = ceph0
>>> mon addr = 192.168.155.5:6789
>>> [mon.1]
>>> host = ceph1
>>> mon addr = 192.168.155.6:6789
>>> [mon.2]
>>> host = ceph2
>>> mon addr = 192.168.155.7:6789
>>> [mds]
>>>
>>> [mds0]
>>> host = ceph0
>>> [mds1]
>>> host = ceph1
>>>
>>> [osd]
>>> sudo = true
>>> osd data = /data/osd$id
>>> osd journal = /data/osd$id/journal
>>> osd journal size = 512
>>> osd use stale snap = true
>>> [osd0]
>>> host = ceph0
>>> btrfs devs = /dev/sdb
>>> [osd1]
>>> host = ceph1
>>> btrfs devs = /dev/sdb
>>> [osd2]
>>> host = ceph2
>>> btrfs devs = /dev/sdb
>>>
>>>
>>>
>>>
>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>> I'm using head of line from the master branch. But that particular
>>>> code hasn't changed since January, which is 0.24.2 is from.
>>>>
>>>> In my ceph.conf, I just had an osd that was on a remote machine, and
>>>> everything else local.
>>>>
>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot
>>>> an issue that's causing the problems that you see.
>>>>
>>>> Colin
>>>>
>>>>
>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>> Hi Colin
>>>>> I am using
>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>>>>
>>>>> Installed ceph form apt source.
>>>>>
>>>>> With above configurations i still able yo reproduce.
>>>>> Can you please share me ur configurations?
>>>>>
>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>>>>> working and there is no error
>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>>>>
>>>>>>> That is expected, unless you are running init-ceph on the same node as
>>>>>>> osd1 is on.
>>>>>>>
>>>>>>> It might be nice to have some kind of interface like "run command X on
>>>>>>> osd1", but init-ceph is not that.
>>>>>>>
>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>>>>> on current node only
>>>>>>>> where as
>>>>>>>> /etc/init.d/ceph -a killall works fine.
>>>>>>>
>>>>>>> That sounds like a bug. I'll see if I can fix it.
>>>>>>
>>>>>> I'm afraid I can't reproduce this.
>>>>>>
>>>>>> I ran /etc/init.d/ceph -a stop
>>>>>>
>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>>>>> the code, it looks correct.
>>>>>>
>>>>>> Colin
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>> Upendra.M
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Upendra.M
>>>
>>
>
>
>
> --
> Thanks and Regards,
> Upendra.M
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-18 14:11 ` Upendra Moturi
2011-02-26 0:28 ` Colin McCabe
@ 2011-02-26 0:28 ` Gregory Farnum
2011-02-28 8:40 ` Upendra Moturi
1 sibling, 1 reply; 18+ messages in thread
From: Gregory Farnum @ 2011-02-26 0:28 UTC (permalink / raw)
To: Upendra Moturi; +Cc: ceph-devel
Sorry, we kind of lost this in the shuffle!
Did you manage to resolve this issue? From what I see in your steps below, it looks like you're doing things in the wrong order -- mkcephfs should always be the first thing you do. If you really have nodes running before you do mkcephfs I suspect you just got your cluster into an inconsistent state and the (fairly fragile) methods it uses to shut down all the daemons got broken.
-Greg
On Friday, February 18, 2011 at 6:11 AM, Upendra Moturi wrote:
> Hi Colin
>
> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
> i am using the default init script.(Found at /etc/init.d/ceph)
>
> Regarding issue 1)
>
> On the osd node I tried to start that osd but it did not work
> Steps followed
>
> 1) Started 3 nodes(ceph.conf is same as i sent earlier)
>
> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
>
> 3)/etc/init.d/ceph start osd0 (Did this on first osd)
>
> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
>
> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes
>
> Regading Issue 2
> It says -x option is not available
>
> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
> tried /etc/init.d/ceph -x -a stop
> but nothing worked.
>
> Then tried with /etc/init.d/ceph -a -v stop this also did not stop
> ceph on all nodes but got the output as
>
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
> === mon.0 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
> "/var/run/ceph/mon.0.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
> -c unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mon.0.pid ] || break
> pid=`cat /var/run/ceph/mon.0.pid`
> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> kill 1668...done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
> === mon.1 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
> "/var/run/ceph/mon.1.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
> -c unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mon.1.pid ] || break
> pid=`cat /var/run/ceph/mon.1.pid`
> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
> === mon.2 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
> "/var/run/ceph/mon.2.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
> -c unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mon.2.pid ] || break
> pid=`cat /var/run/ceph/mon.2.pid`
> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
> === mds.0 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
> "/var/run/ceph/mds.0.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mds.0.pid ] || break
> pid=`cat /var/run/ceph/mds.0.pid`
> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> kill 1844...done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
> === mds.1 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
> "/var/run/ceph/mds.1.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/mds.1.pid ] || break
> pid=`cat /var/run/ceph/mds.1.pid`
> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
> === osd.0 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
> "/var/run/ceph/osd.0.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/osd.0.pid ] || break
> pid=`cat /var/run/ceph/osd.0.pid`
> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> kill 2033...done
> Unmounting Btrfs on ceph0:/data/osd0
> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
> /data/osd0 || true"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
> === osd.1 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
> "/var/run/ceph/osd.1.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/osd.1.pid ] || break
> pid=`cat /var/run/ceph/osd.1.pid`
> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> Unmounting Btrfs on ceph1:/data/osd1
> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
> /data/osd1 || true"
> umount: /data/osd1: device is busy.
> (In some cases useful info about processes that use
> the device is found by lsof(8) or fuser(1))
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
> === osd.2 ===
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
> "/var/run/ceph/osd.2.pid"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
> "/var/lock/subsys/ceph"
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
> unlimited ; while [ 1 ]; do
> [ -e /var/run/ceph/osd.2.pid ] || break
> pid=`cat /var/run/ceph/osd.2.pid`
> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
> cmd="kill $pid"
> echo -n $cmd...
> $cmd
> sleep 1
> continue
> done
> break
> done"
> done
> Unmounting Btrfs on ceph2:/data/osd2
> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
> /data/osd2 || true"
> umount: /data/osd2: device is busy.
> (In some cases useful info about processes that use
> the device is found by lsof(8) or fuser(1))
>
>
>
>
> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> > Hi Upendra,
> >
> > Are you running init-ceph from the source directory? If you do that,
> > it will use the ceph.conf in the source directory itself, which is
> > probably not what you want. So it might be good to double-check that.
> >
> > If all else fails, running init-ceph with -x will show you exactly
> > what the script is doing. If all goes well, its exit status should be
> > 0. Are you getting exit status 0?
> >
> > Colin
> >
> >
> > On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> > > Hi Colin
> > >
> > > Here is my ceph.conf :
> > >
> > > [global]
> > > pid file = /var/run/ceph/$name.pid
> > > debug ms = 1
> > > [mon]
> > > mon data = /data/mon$id
> > > [mon.0]
> > > host = ceph0
> > > mon addr = 192.168.155.5:6789
> > > [mon.1]
> > > host = ceph1
> > > mon addr = 192.168.155.6:6789
> > > [mon.2]
> > > host = ceph2
> > > mon addr = 192.168.155.7:6789
> > > [mds]
> > >
> > > [mds0]
> > > host = ceph0
> > > [mds1]
> > > host = ceph1
> > >
> > > [osd]
> > > sudo = true
> > > osd data = /data/osd$id
> > > osd journal = /data/osd$id/journal
> > > osd journal size = 512
> > > osd use stale snap = true
> > > [osd0]
> > > host = ceph0
> > > btrfs devs = /dev/sdb
> > > [osd1]
> > > host = ceph1
> > > btrfs devs = /dev/sdb
> > > [osd2]
> > > host = ceph2
> > > btrfs devs = /dev/sdb
> > >
> > >
> > >
> > >
> > > On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> > > > I'm using head of line from the master branch. But that particular
> > > > code hasn't changed since January, which is 0.24.2 is from.
> > > >
> > > > In my ceph.conf, I just had an osd that was on a remote machine, and
> > > > everything else local.
> > > >
> > > > If you could post your ceph.conf here or in IRC, perhaps we might spot
> > > > an issue that's causing the problems that you see.
> > > >
> > > > Colin
> > > >
> > > >
> > > > On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> > > > > Hi Colin
> > > > > I am using
> > > > > ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
> > > > > ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
> > > > >
> > > > > Installed ceph form apt source.
> > > > >
> > > > > With above configurations i still able yo reproduce.
> > > > > Can you please share me ur configurations?
> > > > >
> > > > > On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> > > > > > On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> > > > > > > On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> > > > > > > > But if we want to start a particular osd or mon or mds ,its not
> > > > > > > > working and there is no error
> > > > > > > > eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
> > > > > > >
> > > > > > > That is expected, unless you are running init-ceph on the same node as
> > > > > > > osd1 is on.
> > > > > > >
> > > > > > > It might be nice to have some kind of interface like "run command X on
> > > > > > > osd1", but init-ceph is not that.
> > > > > > >
> > > > > > > > /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
> > > > > > > > on current node only
> > > > > > > > where as
> > > > > > > > /etc/init.d/ceph -a killall works fine.
> > > > > > >
> > > > > > > That sounds like a bug. I'll see if I can fix it.
> > > > > >
> > > > > > I'm afraid I can't reproduce this.
> > > > > >
> > > > > > I ran /etc/init.d/ceph -a stop
> > > > > >
> > > > > > and it stopped ceph daemons running on remote nodes too. Looking at
> > > > > > the code, it looks correct.
> > > > > >
> > > > > > Colin
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks and Regards,
> > > > > Upendra.M
> > >
> > >
> > >
> > > --
> > > Thanks and Regards,
> > > Upendra.M
>
>
>
> --
> Thanks and Regards,
> Upendra.M
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-26 0:28 ` Gregory Farnum
@ 2011-02-28 8:40 ` Upendra Moturi
0 siblings, 0 replies; 18+ messages in thread
From: Upendra Moturi @ 2011-02-28 8:40 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
Hi Greg
I am running mkcepfs as the first step of cluster and then use init
ceph to start all node and this works fine
When trying to stop cluser it does not happen and starting a single
osd for also dont work.
On Sat, Feb 26, 2011 at 5:58 AM, Gregory Farnum
<gregory.farnum@dreamhost.com> wrote:
>
> Sorry, we kind of lost this in the shuffle!
>
> Did you manage to resolve this issue? From what I see in your steps below, it looks like you're doing things in the wrong order -- mkcephfs should always be the first thing you do. If you really have nodes running before you do mkcephfs I suspect you just got your cluster into an inconsistent state and the (fairly fragile) methods it uses to shut down all the daemons got broken.
> -Greg
> On Friday, February 18, 2011 at 6:11 AM, Upendra Moturi wrote:
> > Hi Colin
> >
> > I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
> > i am using the default init script.(Found at /etc/init.d/ceph)
> >
> > Regarding issue 1)
> >
> > On the osd node I tried to start that osd but it did not work
> > Steps followed
> >
> > 1) Started 3 nodes(ceph.conf is same as i sent earlier)
> >
> > 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
> >
> > 3)/etc/init.d/ceph start osd0 (Did this on first osd)
> >
> > 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
> >
> > 5)/etc/init.d/ceph -a start ---- shows all process on all nodes
> >
> > Regading Issue 2
> > It says -x option is not available
> >
> > Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
> > tried /etc/init.d/ceph -x -a stop
> > but nothing worked.
> >
> > Then tried with /etc/init.d/ceph -a -v stop this also did not stop
> > ceph on all nodes but got the output as
> >
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
> > === mon.0 ===
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
> > "/var/run/ceph/mon.0.pid"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
> > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
> > "/var/lock/subsys/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
> > Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
> > -c unlimited ; while [ 1 ]; do
> > [ -e /var/run/ceph/mon.0.pid ] || break
> > pid=`cat /var/run/ceph/mon.0.pid`
> > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
> > cmd="kill $pid"
> > echo -n $cmd...
> > $cmd
> > sleep 1
> > continue
> > done
> > break
> > done"
> > kill 1668...done
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
> > === mon.1 ===
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
> > "/var/run/ceph/mon.1.pid"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
> > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
> > "/var/lock/subsys/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
> > Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
> > -c unlimited ; while [ 1 ]; do
> > [ -e /var/run/ceph/mon.1.pid ] || break
> > pid=`cat /var/run/ceph/mon.1.pid`
> > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
> > cmd="kill $pid"
> > echo -n $cmd...
> > $cmd
> > sleep 1
> > continue
> > done
> > break
> > done"
> > done
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
> > === mon.2 ===
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
> > "/var/run/ceph/mon.2.pid"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
> > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
> > "/var/lock/subsys/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
> > Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
> > -c unlimited ; while [ 1 ]; do
> > [ -e /var/run/ceph/mon.2.pid ] || break
> > pid=`cat /var/run/ceph/mon.2.pid`
> > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
> > cmd="kill $pid"
> > echo -n $cmd...
> > $cmd
> > sleep 1
> > continue
> > done
> > break
> > done"
> > done
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
> > === mds.0 ===
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
> > "/var/run/ceph/mds.0.pid"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
> > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
> > "/var/lock/subsys/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
> > Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
> > unlimited ; while [ 1 ]; do
> > [ -e /var/run/ceph/mds.0.pid ] || break
> > pid=`cat /var/run/ceph/mds.0.pid`
> > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
> > cmd="kill $pid"
> > echo -n $cmd...
> > $cmd
> > sleep 1
> > continue
> > done
> > break
> > done"
> > kill 1844...done
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
> > === mds.1 ===
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
> > "/var/run/ceph/mds.1.pid"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
> > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
> > "/var/lock/subsys/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
> > Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
> > unlimited ; while [ 1 ]; do
> > [ -e /var/run/ceph/mds.1.pid ] || break
> > pid=`cat /var/run/ceph/mds.1.pid`
> > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
> > cmd="kill $pid"
> > echo -n $cmd...
> > $cmd
> > sleep 1
> > continue
> > done
> > break
> > done"
> > done
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
> > === osd.0 ===
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
> > "/var/run/ceph/osd.0.pid"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
> > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
> > "/var/lock/subsys/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
> > Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
> > unlimited ; while [ 1 ]; do
> > [ -e /var/run/ceph/osd.0.pid ] || break
> > pid=`cat /var/run/ceph/osd.0.pid`
> > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
> > cmd="kill $pid"
> > echo -n $cmd...
> > $cmd
> > sleep 1
> > continue
> > done
> > break
> > done"
> > kill 2033...done
> > Unmounting Btrfs on ceph0:/data/osd0
> > --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
> > /data/osd0 || true"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
> > === osd.1 ===
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
> > "/var/run/ceph/osd.1.pid"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
> > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
> > "/var/lock/subsys/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
> > Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
> > unlimited ; while [ 1 ]; do
> > [ -e /var/run/ceph/osd.1.pid ] || break
> > pid=`cat /var/run/ceph/osd.1.pid`
> > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
> > cmd="kill $pid"
> > echo -n $cmd...
> > $cmd
> > sleep 1
> > continue
> > done
> > break
> > done"
> > done
> > Unmounting Btrfs on ceph1:/data/osd1
> > --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
> > /data/osd1 || true"
> > umount: /data/osd1: device is busy.
> > (In some cases useful info about processes that use
> > the device is found by lsof(8) or fuser(1))
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
> > === osd.2 ===
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
> > "/var/run/ceph/osd.2.pid"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
> > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
> > "/var/lock/subsys/ceph"
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
> > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
> > Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
> > unlimited ; while [ 1 ]; do
> > [ -e /var/run/ceph/osd.2.pid ] || break
> > pid=`cat /var/run/ceph/osd.2.pid`
> > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
> > cmd="kill $pid"
> > echo -n $cmd...
> > $cmd
> > sleep 1
> > continue
> > done
> > break
> > done"
> > done
> > Unmounting Btrfs on ceph2:/data/osd2
> > --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
> > /data/osd2 || true"
> > umount: /data/osd2: device is busy.
> > (In some cases useful info about processes that use
> > the device is found by lsof(8) or fuser(1))
> >
> >
> >
> >
> > On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> > > Hi Upendra,
> > >
> > > Are you running init-ceph from the source directory? If you do that,
> > > it will use the ceph.conf in the source directory itself, which is
> > > probably not what you want. So it might be good to double-check that.
> > >
> > > If all else fails, running init-ceph with -x will show you exactly
> > > what the script is doing. If all goes well, its exit status should be
> > > 0. Are you getting exit status 0?
> > >
> > > Colin
> > >
> > >
> > > On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> > > > Hi Colin
> > > >
> > > > Here is my ceph.conf :
> > > >
> > > > [global]
> > > > pid file = /var/run/ceph/$name.pid
> > > > debug ms = 1
> > > > [mon]
> > > > mon data = /data/mon$id
> > > > [mon.0]
> > > > host = ceph0
> > > > mon addr = 192.168.155.5:6789
> > > > [mon.1]
> > > > host = ceph1
> > > > mon addr = 192.168.155.6:6789
> > > > [mon.2]
> > > > host = ceph2
> > > > mon addr = 192.168.155.7:6789
> > > > [mds]
> > > >
> > > > [mds0]
> > > > host = ceph0
> > > > [mds1]
> > > > host = ceph1
> > > >
> > > > [osd]
> > > > sudo = true
> > > > osd data = /data/osd$id
> > > > osd journal = /data/osd$id/journal
> > > > osd journal size = 512
> > > > osd use stale snap = true
> > > > [osd0]
> > > > host = ceph0
> > > > btrfs devs = /dev/sdb
> > > > [osd1]
> > > > host = ceph1
> > > > btrfs devs = /dev/sdb
> > > > [osd2]
> > > > host = ceph2
> > > > btrfs devs = /dev/sdb
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> > > > > I'm using head of line from the master branch. But that particular
> > > > > code hasn't changed since January, which is 0.24.2 is from.
> > > > >
> > > > > In my ceph.conf, I just had an osd that was on a remote machine, and
> > > > > everything else local.
> > > > >
> > > > > If you could post your ceph.conf here or in IRC, perhaps we might spot
> > > > > an issue that's causing the problems that you see.
> > > > >
> > > > > Colin
> > > > >
> > > > >
> > > > > On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> > > > > > Hi Colin
> > > > > > I am using
> > > > > > ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
> > > > > > ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
> > > > > >
> > > > > > Installed ceph form apt source.
> > > > > >
> > > > > > With above configurations i still able yo reproduce.
> > > > > > Can you please share me ur configurations?
> > > > > >
> > > > > > On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> > > > > > > On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> > > > > > > > On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> > > > > > > > > But if we want to start a particular osd or mon or mds ,its not
> > > > > > > > > working and there is no error
> > > > > > > > > eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
> > > > > > > >
> > > > > > > > That is expected, unless you are running init-ceph on the same node as
> > > > > > > > osd1 is on.
> > > > > > > >
> > > > > > > > It might be nice to have some kind of interface like "run command X on
> > > > > > > > osd1", but init-ceph is not that.
> > > > > > > >
> > > > > > > > > /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
> > > > > > > > > on current node only
> > > > > > > > > where as
> > > > > > > > > /etc/init.d/ceph -a killall works fine.
> > > > > > > >
> > > > > > > > That sounds like a bug. I'll see if I can fix it.
> > > > > > >
> > > > > > > I'm afraid I can't reproduce this.
> > > > > > >
> > > > > > > I ran /etc/init.d/ceph -a stop
> > > > > > >
> > > > > > > and it stopped ceph daemons running on remote nodes too. Looking at
> > > > > > > the code, it looks correct.
> > > > > > >
> > > > > > > Colin
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Thanks and Regards,
> > > > > > Upendra.M
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks and Regards,
> > > > Upendra.M
> >
> >
> >
> > --
> > Thanks and Regards,
> > Upendra.M
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
--
Thanks and Regards,
Upendra.M
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-26 0:28 ` Colin McCabe
@ 2011-02-28 12:00 ` Upendra Moturi
2011-02-28 16:57 ` Colin McCabe
0 siblings, 1 reply; 18+ messages in thread
From: Upendra Moturi @ 2011-02-28 12:00 UTC (permalink / raw)
To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava
Hi Colin
/var/run/ceph exist on only one node(the node on which i start the
cluster with -a)
But ceph is running on all nodes(checked with ps -ef)
When i ssh into nodes and try to stop osd (/etc/init.d/ceph stop osd)
it does not work(this was my second issue)
But when i stop /etc/init.d/ceph -a stop ,it stops ceph of that node.
On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> Hi Upendra,
>
> Based on the output you posted, init-ceph is doing something on every
> node. However, I only see a kill for certain nodes.
>
> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the
> appropriate pid file exist on all nodes? What happens when you ssh in
> to those nodes manually and run init-ceph stop?
>
> Colin
>
>
> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>> Hi Colin
>>
>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
>> i am using the default init script.(Found at /etc/init.d/ceph)
>>
>> Regarding issue 1)
>>
>> On the osd node I tried to start that osd but it did not work
>> Steps followed
>>
>> 1) Started 3 nodes(ceph.conf is same as i sent earlier)
>>
>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
>>
>> 3)/etc/init.d/ceph start osd0 (Did this on first osd)
>>
>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
>>
>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes
>>
>> Regading Issue 2
>> It says -x option is not available
>>
>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
>> tried /etc/init.d/ceph -x -a stop
>> but nothing worked.
>>
>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop
>> ceph on all nodes but got the output as
>>
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
>> === mon.0 ===
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
>> "/var/run/ceph/mon.0.pid"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
>> "/var/lock/subsys/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
>> -c unlimited ; while [ 1 ]; do
>> [ -e /var/run/ceph/mon.0.pid ] || break
>> pid=`cat /var/run/ceph/mon.0.pid`
>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>> cmd="kill $pid"
>> echo -n $cmd...
>> $cmd
>> sleep 1
>> continue
>> done
>> break
>> done"
>> kill 1668...done
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
>> === mon.1 ===
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
>> "/var/run/ceph/mon.1.pid"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
>> "/var/lock/subsys/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
>> -c unlimited ; while [ 1 ]; do
>> [ -e /var/run/ceph/mon.1.pid ] || break
>> pid=`cat /var/run/ceph/mon.1.pid`
>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>> cmd="kill $pid"
>> echo -n $cmd...
>> $cmd
>> sleep 1
>> continue
>> done
>> break
>> done"
>> done
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
>> === mon.2 ===
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
>> "/var/run/ceph/mon.2.pid"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
>> "/var/lock/subsys/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
>> -c unlimited ; while [ 1 ]; do
>> [ -e /var/run/ceph/mon.2.pid ] || break
>> pid=`cat /var/run/ceph/mon.2.pid`
>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>> cmd="kill $pid"
>> echo -n $cmd...
>> $cmd
>> sleep 1
>> continue
>> done
>> break
>> done"
>> done
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
>> === mds.0 ===
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
>> "/var/run/ceph/mds.0.pid"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
>> "/var/lock/subsys/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>> unlimited ; while [ 1 ]; do
>> [ -e /var/run/ceph/mds.0.pid ] || break
>> pid=`cat /var/run/ceph/mds.0.pid`
>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>> cmd="kill $pid"
>> echo -n $cmd...
>> $cmd
>> sleep 1
>> continue
>> done
>> break
>> done"
>> kill 1844...done
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
>> === mds.1 ===
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
>> "/var/run/ceph/mds.1.pid"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
>> "/var/lock/subsys/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>> unlimited ; while [ 1 ]; do
>> [ -e /var/run/ceph/mds.1.pid ] || break
>> pid=`cat /var/run/ceph/mds.1.pid`
>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>> cmd="kill $pid"
>> echo -n $cmd...
>> $cmd
>> sleep 1
>> continue
>> done
>> break
>> done"
>> done
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
>> === osd.0 ===
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
>> "/var/run/ceph/osd.0.pid"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
>> "/var/lock/subsys/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>> unlimited ; while [ 1 ]; do
>> [ -e /var/run/ceph/osd.0.pid ] || break
>> pid=`cat /var/run/ceph/osd.0.pid`
>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>> cmd="kill $pid"
>> echo -n $cmd...
>> $cmd
>> sleep 1
>> continue
>> done
>> break
>> done"
>> kill 2033...done
>> Unmounting Btrfs on ceph0:/data/osd0
>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
>> /data/osd0 || true"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
>> === osd.1 ===
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
>> "/var/run/ceph/osd.1.pid"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
>> "/var/lock/subsys/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>> unlimited ; while [ 1 ]; do
>> [ -e /var/run/ceph/osd.1.pid ] || break
>> pid=`cat /var/run/ceph/osd.1.pid`
>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>> cmd="kill $pid"
>> echo -n $cmd...
>> $cmd
>> sleep 1
>> continue
>> done
>> break
>> done"
>> done
>> Unmounting Btrfs on ceph1:/data/osd1
>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
>> /data/osd1 || true"
>> umount: /data/osd1: device is busy.
>> (In some cases useful info about processes that use
>> the device is found by lsof(8) or fuser(1))
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
>> === osd.2 ===
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
>> "/var/run/ceph/osd.2.pid"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
>> "/var/lock/subsys/ceph"
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
>> unlimited ; while [ 1 ]; do
>> [ -e /var/run/ceph/osd.2.pid ] || break
>> pid=`cat /var/run/ceph/osd.2.pid`
>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>> cmd="kill $pid"
>> echo -n $cmd...
>> $cmd
>> sleep 1
>> continue
>> done
>> break
>> done"
>> done
>> Unmounting Btrfs on ceph2:/data/osd2
>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
>> /data/osd2 || true"
>> umount: /data/osd2: device is busy.
>> (In some cases useful info about processes that use
>> the device is found by lsof(8) or fuser(1))
>>
>>
>>
>>
>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>> Hi Upendra,
>>>
>>> Are you running init-ceph from the source directory? If you do that,
>>> it will use the ceph.conf in the source directory itself, which is
>>> probably not what you want. So it might be good to double-check that.
>>>
>>> If all else fails, running init-ceph with -x will show you exactly
>>> what the script is doing. If all goes well, its exit status should be
>>> 0. Are you getting exit status 0?
>>>
>>> Colin
>>>
>>>
>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>> Hi Colin
>>>>
>>>> Here is my ceph.conf :
>>>>
>>>> [global]
>>>> pid file = /var/run/ceph/$name.pid
>>>> debug ms = 1
>>>> [mon]
>>>> mon data = /data/mon$id
>>>> [mon.0]
>>>> host = ceph0
>>>> mon addr = 192.168.155.5:6789
>>>> [mon.1]
>>>> host = ceph1
>>>> mon addr = 192.168.155.6:6789
>>>> [mon.2]
>>>> host = ceph2
>>>> mon addr = 192.168.155.7:6789
>>>> [mds]
>>>>
>>>> [mds0]
>>>> host = ceph0
>>>> [mds1]
>>>> host = ceph1
>>>>
>>>> [osd]
>>>> sudo = true
>>>> osd data = /data/osd$id
>>>> osd journal = /data/osd$id/journal
>>>> osd journal size = 512
>>>> osd use stale snap = true
>>>> [osd0]
>>>> host = ceph0
>>>> btrfs devs = /dev/sdb
>>>> [osd1]
>>>> host = ceph1
>>>> btrfs devs = /dev/sdb
>>>> [osd2]
>>>> host = ceph2
>>>> btrfs devs = /dev/sdb
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>> I'm using head of line from the master branch. But that particular
>>>>> code hasn't changed since January, which is 0.24.2 is from.
>>>>>
>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and
>>>>> everything else local.
>>>>>
>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot
>>>>> an issue that's causing the problems that you see.
>>>>>
>>>>> Colin
>>>>>
>>>>>
>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>> Hi Colin
>>>>>> I am using
>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>>>>>
>>>>>> Installed ceph form apt source.
>>>>>>
>>>>>> With above configurations i still able yo reproduce.
>>>>>> Can you please share me ur configurations?
>>>>>>
>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>>>>>> working and there is no error
>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>>>>>
>>>>>>>> That is expected, unless you are running init-ceph on the same node as
>>>>>>>> osd1 is on.
>>>>>>>>
>>>>>>>> It might be nice to have some kind of interface like "run command X on
>>>>>>>> osd1", but init-ceph is not that.
>>>>>>>>
>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>>>>>> on current node only
>>>>>>>>> where as
>>>>>>>>> /etc/init.d/ceph -a killall works fine.
>>>>>>>>
>>>>>>>> That sounds like a bug. I'll see if I can fix it.
>>>>>>>
>>>>>>> I'm afraid I can't reproduce this.
>>>>>>>
>>>>>>> I ran /etc/init.d/ceph -a stop
>>>>>>>
>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>>>>>> the code, it looks correct.
>>>>>>>
>>>>>>> Colin
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>> Upendra.M
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Upendra.M
>>>>
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Upendra.M
>>
>
--
Thanks and Regards,
Upendra.M
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-28 12:00 ` Upendra Moturi
@ 2011-02-28 16:57 ` Colin McCabe
2011-03-01 11:55 ` Upendra Moturi
0 siblings, 1 reply; 18+ messages in thread
From: Colin McCabe @ 2011-02-28 16:57 UTC (permalink / raw)
To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava
On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> Hi Colin
>
> /var/run/ceph exist on only one node(the node on which i start the
> cluster with -a)
Without /var/run/ceph/pid, the init script will not know which pid to
kill. So it will do nothing.
Try creating /var/run/ceph and other appropriate directories on all
nodes before starting the cluster.
Colin
>
> On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>> Hi Upendra,
>>
>> Based on the output you posted, init-ceph is doing something on every
>> node. However, I only see a kill for certain nodes.
>>
>> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the
>> appropriate pid file exist on all nodes? What happens when you ssh in
>> to those nodes manually and run init-ceph stop?
>>
>> Colin
>>
>>
>> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>> Hi Colin
>>>
>>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
>>> i am using the default init script.(Found at /etc/init.d/ceph)
>>>
>>> Regarding issue 1)
>>>
>>> On the osd node I tried to start that osd but it did not work
>>> Steps followed
>>>
>>> 1) Started 3 nodes(ceph.conf is same as i sent earlier)
>>>
>>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
>>>
>>> 3)/etc/init.d/ceph start osd0 (Did this on first osd)
>>>
>>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
>>>
>>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes
>>>
>>> Regading Issue 2
>>> It says -x option is not available
>>>
>>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
>>> tried /etc/init.d/ceph -x -a stop
>>> but nothing worked.
>>>
>>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop
>>> ceph on all nodes but got the output as
>>>
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
>>> === mon.0 ===
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
>>> "/var/run/ceph/mon.0.pid"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
>>> "/var/lock/subsys/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
>>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
>>> -c unlimited ; while [ 1 ]; do
>>> [ -e /var/run/ceph/mon.0.pid ] || break
>>> pid=`cat /var/run/ceph/mon.0.pid`
>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>> cmd="kill $pid"
>>> echo -n $cmd...
>>> $cmd
>>> sleep 1
>>> continue
>>> done
>>> break
>>> done"
>>> kill 1668...done
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
>>> === mon.1 ===
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
>>> "/var/run/ceph/mon.1.pid"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
>>> "/var/lock/subsys/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
>>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
>>> -c unlimited ; while [ 1 ]; do
>>> [ -e /var/run/ceph/mon.1.pid ] || break
>>> pid=`cat /var/run/ceph/mon.1.pid`
>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>> cmd="kill $pid"
>>> echo -n $cmd...
>>> $cmd
>>> sleep 1
>>> continue
>>> done
>>> break
>>> done"
>>> done
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
>>> === mon.2 ===
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
>>> "/var/run/ceph/mon.2.pid"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
>>> "/var/lock/subsys/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
>>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
>>> -c unlimited ; while [ 1 ]; do
>>> [ -e /var/run/ceph/mon.2.pid ] || break
>>> pid=`cat /var/run/ceph/mon.2.pid`
>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>> cmd="kill $pid"
>>> echo -n $cmd...
>>> $cmd
>>> sleep 1
>>> continue
>>> done
>>> break
>>> done"
>>> done
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
>>> === mds.0 ===
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
>>> "/var/run/ceph/mds.0.pid"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
>>> "/var/lock/subsys/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
>>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>> unlimited ; while [ 1 ]; do
>>> [ -e /var/run/ceph/mds.0.pid ] || break
>>> pid=`cat /var/run/ceph/mds.0.pid`
>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>> cmd="kill $pid"
>>> echo -n $cmd...
>>> $cmd
>>> sleep 1
>>> continue
>>> done
>>> break
>>> done"
>>> kill 1844...done
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
>>> === mds.1 ===
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
>>> "/var/run/ceph/mds.1.pid"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
>>> "/var/lock/subsys/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
>>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>> unlimited ; while [ 1 ]; do
>>> [ -e /var/run/ceph/mds.1.pid ] || break
>>> pid=`cat /var/run/ceph/mds.1.pid`
>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>> cmd="kill $pid"
>>> echo -n $cmd...
>>> $cmd
>>> sleep 1
>>> continue
>>> done
>>> break
>>> done"
>>> done
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
>>> === osd.0 ===
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
>>> "/var/run/ceph/osd.0.pid"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
>>> "/var/lock/subsys/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
>>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>> unlimited ; while [ 1 ]; do
>>> [ -e /var/run/ceph/osd.0.pid ] || break
>>> pid=`cat /var/run/ceph/osd.0.pid`
>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>> cmd="kill $pid"
>>> echo -n $cmd...
>>> $cmd
>>> sleep 1
>>> continue
>>> done
>>> break
>>> done"
>>> kill 2033...done
>>> Unmounting Btrfs on ceph0:/data/osd0
>>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>> /data/osd0 || true"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
>>> === osd.1 ===
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
>>> "/var/run/ceph/osd.1.pid"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
>>> "/var/lock/subsys/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
>>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>> unlimited ; while [ 1 ]; do
>>> [ -e /var/run/ceph/osd.1.pid ] || break
>>> pid=`cat /var/run/ceph/osd.1.pid`
>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>> cmd="kill $pid"
>>> echo -n $cmd...
>>> $cmd
>>> sleep 1
>>> continue
>>> done
>>> break
>>> done"
>>> done
>>> Unmounting Btrfs on ceph1:/data/osd1
>>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>> /data/osd1 || true"
>>> umount: /data/osd1: device is busy.
>>> (In some cases useful info about processes that use
>>> the device is found by lsof(8) or fuser(1))
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
>>> === osd.2 ===
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
>>> "/var/run/ceph/osd.2.pid"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
>>> "/var/lock/subsys/ceph"
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
>>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
>>> unlimited ; while [ 1 ]; do
>>> [ -e /var/run/ceph/osd.2.pid ] || break
>>> pid=`cat /var/run/ceph/osd.2.pid`
>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>> cmd="kill $pid"
>>> echo -n $cmd...
>>> $cmd
>>> sleep 1
>>> continue
>>> done
>>> break
>>> done"
>>> done
>>> Unmounting Btrfs on ceph2:/data/osd2
>>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>> /data/osd2 || true"
>>> umount: /data/osd2: device is busy.
>>> (In some cases useful info about processes that use
>>> the device is found by lsof(8) or fuser(1))
>>>
>>>
>>>
>>>
>>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>> Hi Upendra,
>>>>
>>>> Are you running init-ceph from the source directory? If you do that,
>>>> it will use the ceph.conf in the source directory itself, which is
>>>> probably not what you want. So it might be good to double-check that.
>>>>
>>>> If all else fails, running init-ceph with -x will show you exactly
>>>> what the script is doing. If all goes well, its exit status should be
>>>> 0. Are you getting exit status 0?
>>>>
>>>> Colin
>>>>
>>>>
>>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>> Hi Colin
>>>>>
>>>>> Here is my ceph.conf :
>>>>>
>>>>> [global]
>>>>> pid file = /var/run/ceph/$name.pid
>>>>> debug ms = 1
>>>>> [mon]
>>>>> mon data = /data/mon$id
>>>>> [mon.0]
>>>>> host = ceph0
>>>>> mon addr = 192.168.155.5:6789
>>>>> [mon.1]
>>>>> host = ceph1
>>>>> mon addr = 192.168.155.6:6789
>>>>> [mon.2]
>>>>> host = ceph2
>>>>> mon addr = 192.168.155.7:6789
>>>>> [mds]
>>>>>
>>>>> [mds0]
>>>>> host = ceph0
>>>>> [mds1]
>>>>> host = ceph1
>>>>>
>>>>> [osd]
>>>>> sudo = true
>>>>> osd data = /data/osd$id
>>>>> osd journal = /data/osd$id/journal
>>>>> osd journal size = 512
>>>>> osd use stale snap = true
>>>>> [osd0]
>>>>> host = ceph0
>>>>> btrfs devs = /dev/sdb
>>>>> [osd1]
>>>>> host = ceph1
>>>>> btrfs devs = /dev/sdb
>>>>> [osd2]
>>>>> host = ceph2
>>>>> btrfs devs = /dev/sdb
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>> I'm using head of line from the master branch. But that particular
>>>>>> code hasn't changed since January, which is 0.24.2 is from.
>>>>>>
>>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and
>>>>>> everything else local.
>>>>>>
>>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot
>>>>>> an issue that's causing the problems that you see.
>>>>>>
>>>>>> Colin
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>> Hi Colin
>>>>>>> I am using
>>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>>>>>>
>>>>>>> Installed ceph form apt source.
>>>>>>>
>>>>>>> With above configurations i still able yo reproduce.
>>>>>>> Can you please share me ur configurations?
>>>>>>>
>>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>>>>>>> working and there is no error
>>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>>>>>>
>>>>>>>>> That is expected, unless you are running init-ceph on the same node as
>>>>>>>>> osd1 is on.
>>>>>>>>>
>>>>>>>>> It might be nice to have some kind of interface like "run command X on
>>>>>>>>> osd1", but init-ceph is not that.
>>>>>>>>>
>>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>>>>>>> on current node only
>>>>>>>>>> where as
>>>>>>>>>> /etc/init.d/ceph -a killall works fine.
>>>>>>>>>
>>>>>>>>> That sounds like a bug. I'll see if I can fix it.
>>>>>>>>
>>>>>>>> I'm afraid I can't reproduce this.
>>>>>>>>
>>>>>>>> I ran /etc/init.d/ceph -a stop
>>>>>>>>
>>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>>>>>>> the code, it looks correct.
>>>>>>>>
>>>>>>>> Colin
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>> Upendra.M
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>> Upendra.M
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Upendra.M
>>>
>>
>
>
>
> --
> Thanks and Regards,
> Upendra.M
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-02-28 16:57 ` Colin McCabe
@ 2011-03-01 11:55 ` Upendra Moturi
2011-03-01 17:34 ` Colin McCabe
0 siblings, 1 reply; 18+ messages in thread
From: Upendra Moturi @ 2011-03-01 11:55 UTC (permalink / raw)
To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava
Hi Colin
Creating /var/run/ceph directory on every node has made it
working.(/etc/init.d/ceph -a stop)
But /etc/init.d/ceph start osd0 still does not work
/etc/init.d/ceph -v start osd0
Gives output as
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
/usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
I have tried this on same node where osd0 is present
Had a look at /etc/init.d/ceph and it does not enter the case start.
On Mon, Feb 28, 2011 at 10:27 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>> Hi Colin
>>
>> /var/run/ceph exist on only one node(the node on which i start the
>> cluster with -a)
>
> Without /var/run/ceph/pid, the init script will not know which pid to
> kill. So it will do nothing.
>
> Try creating /var/run/ceph and other appropriate directories on all
> nodes before starting the cluster.
>
> Colin
>
>
>>
>> On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>> Hi Upendra,
>>>
>>> Based on the output you posted, init-ceph is doing something on every
>>> node. However, I only see a kill for certain nodes.
>>>
>>> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the
>>> appropriate pid file exist on all nodes? What happens when you ssh in
>>> to those nodes manually and run init-ceph stop?
>>>
>>> Colin
>>>
>>>
>>> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>> Hi Colin
>>>>
>>>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
>>>> i am using the default init script.(Found at /etc/init.d/ceph)
>>>>
>>>> Regarding issue 1)
>>>>
>>>> On the osd node I tried to start that osd but it did not work
>>>> Steps followed
>>>>
>>>> 1) Started 3 nodes(ceph.conf is same as i sent earlier)
>>>>
>>>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
>>>>
>>>> 3)/etc/init.d/ceph start osd0 (Did this on first osd)
>>>>
>>>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
>>>>
>>>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes
>>>>
>>>> Regading Issue 2
>>>> It says -x option is not available
>>>>
>>>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
>>>> tried /etc/init.d/ceph -x -a stop
>>>> but nothing worked.
>>>>
>>>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop
>>>> ceph on all nodes but got the output as
>>>>
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
>>>> === mon.0 ===
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
>>>> "/var/run/ceph/mon.0.pid"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
>>>> "/var/lock/subsys/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
>>>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
>>>> -c unlimited ; while [ 1 ]; do
>>>> [ -e /var/run/ceph/mon.0.pid ] || break
>>>> pid=`cat /var/run/ceph/mon.0.pid`
>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>> cmd="kill $pid"
>>>> echo -n $cmd...
>>>> $cmd
>>>> sleep 1
>>>> continue
>>>> done
>>>> break
>>>> done"
>>>> kill 1668...done
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
>>>> === mon.1 ===
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
>>>> "/var/run/ceph/mon.1.pid"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
>>>> "/var/lock/subsys/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
>>>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
>>>> -c unlimited ; while [ 1 ]; do
>>>> [ -e /var/run/ceph/mon.1.pid ] || break
>>>> pid=`cat /var/run/ceph/mon.1.pid`
>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>> cmd="kill $pid"
>>>> echo -n $cmd...
>>>> $cmd
>>>> sleep 1
>>>> continue
>>>> done
>>>> break
>>>> done"
>>>> done
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
>>>> === mon.2 ===
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
>>>> "/var/run/ceph/mon.2.pid"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
>>>> "/var/lock/subsys/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
>>>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
>>>> -c unlimited ; while [ 1 ]; do
>>>> [ -e /var/run/ceph/mon.2.pid ] || break
>>>> pid=`cat /var/run/ceph/mon.2.pid`
>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>> cmd="kill $pid"
>>>> echo -n $cmd...
>>>> $cmd
>>>> sleep 1
>>>> continue
>>>> done
>>>> break
>>>> done"
>>>> done
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
>>>> === mds.0 ===
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
>>>> "/var/run/ceph/mds.0.pid"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
>>>> "/var/lock/subsys/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
>>>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>>> unlimited ; while [ 1 ]; do
>>>> [ -e /var/run/ceph/mds.0.pid ] || break
>>>> pid=`cat /var/run/ceph/mds.0.pid`
>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>>> cmd="kill $pid"
>>>> echo -n $cmd...
>>>> $cmd
>>>> sleep 1
>>>> continue
>>>> done
>>>> break
>>>> done"
>>>> kill 1844...done
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
>>>> === mds.1 ===
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
>>>> "/var/run/ceph/mds.1.pid"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
>>>> "/var/lock/subsys/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
>>>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>>> unlimited ; while [ 1 ]; do
>>>> [ -e /var/run/ceph/mds.1.pid ] || break
>>>> pid=`cat /var/run/ceph/mds.1.pid`
>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>>> cmd="kill $pid"
>>>> echo -n $cmd...
>>>> $cmd
>>>> sleep 1
>>>> continue
>>>> done
>>>> break
>>>> done"
>>>> done
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
>>>> === osd.0 ===
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
>>>> "/var/run/ceph/osd.0.pid"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
>>>> "/var/lock/subsys/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
>>>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>>> unlimited ; while [ 1 ]; do
>>>> [ -e /var/run/ceph/osd.0.pid ] || break
>>>> pid=`cat /var/run/ceph/osd.0.pid`
>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>> cmd="kill $pid"
>>>> echo -n $cmd...
>>>> $cmd
>>>> sleep 1
>>>> continue
>>>> done
>>>> break
>>>> done"
>>>> kill 2033...done
>>>> Unmounting Btrfs on ceph0:/data/osd0
>>>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>> /data/osd0 || true"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
>>>> === osd.1 ===
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
>>>> "/var/run/ceph/osd.1.pid"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
>>>> "/var/lock/subsys/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
>>>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>>> unlimited ; while [ 1 ]; do
>>>> [ -e /var/run/ceph/osd.1.pid ] || break
>>>> pid=`cat /var/run/ceph/osd.1.pid`
>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>> cmd="kill $pid"
>>>> echo -n $cmd...
>>>> $cmd
>>>> sleep 1
>>>> continue
>>>> done
>>>> break
>>>> done"
>>>> done
>>>> Unmounting Btrfs on ceph1:/data/osd1
>>>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>> /data/osd1 || true"
>>>> umount: /data/osd1: device is busy.
>>>> (In some cases useful info about processes that use
>>>> the device is found by lsof(8) or fuser(1))
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
>>>> === osd.2 ===
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
>>>> "/var/run/ceph/osd.2.pid"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
>>>> "/var/lock/subsys/ceph"
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
>>>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
>>>> unlimited ; while [ 1 ]; do
>>>> [ -e /var/run/ceph/osd.2.pid ] || break
>>>> pid=`cat /var/run/ceph/osd.2.pid`
>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>> cmd="kill $pid"
>>>> echo -n $cmd...
>>>> $cmd
>>>> sleep 1
>>>> continue
>>>> done
>>>> break
>>>> done"
>>>> done
>>>> Unmounting Btrfs on ceph2:/data/osd2
>>>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>> /data/osd2 || true"
>>>> umount: /data/osd2: device is busy.
>>>> (In some cases useful info about processes that use
>>>> the device is found by lsof(8) or fuser(1))
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>> Hi Upendra,
>>>>>
>>>>> Are you running init-ceph from the source directory? If you do that,
>>>>> it will use the ceph.conf in the source directory itself, which is
>>>>> probably not what you want. So it might be good to double-check that.
>>>>>
>>>>> If all else fails, running init-ceph with -x will show you exactly
>>>>> what the script is doing. If all goes well, its exit status should be
>>>>> 0. Are you getting exit status 0?
>>>>>
>>>>> Colin
>>>>>
>>>>>
>>>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>> Hi Colin
>>>>>>
>>>>>> Here is my ceph.conf :
>>>>>>
>>>>>> [global]
>>>>>> pid file = /var/run/ceph/$name.pid
>>>>>> debug ms = 1
>>>>>> [mon]
>>>>>> mon data = /data/mon$id
>>>>>> [mon.0]
>>>>>> host = ceph0
>>>>>> mon addr = 192.168.155.5:6789
>>>>>> [mon.1]
>>>>>> host = ceph1
>>>>>> mon addr = 192.168.155.6:6789
>>>>>> [mon.2]
>>>>>> host = ceph2
>>>>>> mon addr = 192.168.155.7:6789
>>>>>> [mds]
>>>>>>
>>>>>> [mds0]
>>>>>> host = ceph0
>>>>>> [mds1]
>>>>>> host = ceph1
>>>>>>
>>>>>> [osd]
>>>>>> sudo = true
>>>>>> osd data = /data/osd$id
>>>>>> osd journal = /data/osd$id/journal
>>>>>> osd journal size = 512
>>>>>> osd use stale snap = true
>>>>>> [osd0]
>>>>>> host = ceph0
>>>>>> btrfs devs = /dev/sdb
>>>>>> [osd1]
>>>>>> host = ceph1
>>>>>> btrfs devs = /dev/sdb
>>>>>> [osd2]
>>>>>> host = ceph2
>>>>>> btrfs devs = /dev/sdb
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>> I'm using head of line from the master branch. But that particular
>>>>>>> code hasn't changed since January, which is 0.24.2 is from.
>>>>>>>
>>>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and
>>>>>>> everything else local.
>>>>>>>
>>>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot
>>>>>>> an issue that's causing the problems that you see.
>>>>>>>
>>>>>>> Colin
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>> Hi Colin
>>>>>>>> I am using
>>>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>>>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>>>>>>>
>>>>>>>> Installed ceph form apt source.
>>>>>>>>
>>>>>>>> With above configurations i still able yo reproduce.
>>>>>>>> Can you please share me ur configurations?
>>>>>>>>
>>>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>>>>>>>> working and there is no error
>>>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>>>>>>>
>>>>>>>>>> That is expected, unless you are running init-ceph on the same node as
>>>>>>>>>> osd1 is on.
>>>>>>>>>>
>>>>>>>>>> It might be nice to have some kind of interface like "run command X on
>>>>>>>>>> osd1", but init-ceph is not that.
>>>>>>>>>>
>>>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>>>>>>>> on current node only
>>>>>>>>>>> where as
>>>>>>>>>>> /etc/init.d/ceph -a killall works fine.
>>>>>>>>>>
>>>>>>>>>> That sounds like a bug. I'll see if I can fix it.
>>>>>>>>>
>>>>>>>>> I'm afraid I can't reproduce this.
>>>>>>>>>
>>>>>>>>> I ran /etc/init.d/ceph -a stop
>>>>>>>>>
>>>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>>>>>>>> the code, it looks correct.
>>>>>>>>>
>>>>>>>>> Colin
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks and Regards,
>>>>>>>> Upendra.M
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>> Upendra.M
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Upendra.M
>>>>
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Upendra.M
>>
>
--
Thanks and Regards,
Upendra.M
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-03-01 11:55 ` Upendra Moturi
@ 2011-03-01 17:34 ` Colin McCabe
2011-03-02 9:18 ` Upendra Moturi
0 siblings, 1 reply; 18+ messages in thread
From: Colin McCabe @ 2011-03-01 17:34 UTC (permalink / raw)
To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava
Hi Upendra,
You should be on the same node as osd0 in order to start osd0.
There is that wacky -a option, at least until we decide to remove it.
But my advice is not to use that. Set up something like dsh.
cheers,
Colin
On Tue, Mar 1, 2011 at 3:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> Hi Colin
>
> Creating /var/run/ceph directory on every node has made it
> working.(/etc/init.d/ceph -a stop)
> But /etc/init.d/ceph start osd0 still does not work
>
> /etc/init.d/ceph -v start osd0
> Gives output as
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
>
> I have tried this on same node where osd0 is present
>
> Had a look at /etc/init.d/ceph and it does not enter the case start.
>
>
> On Mon, Feb 28, 2011 at 10:27 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>> On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>> Hi Colin
>>>
>>> /var/run/ceph exist on only one node(the node on which i start the
>>> cluster with -a)
>>
>> Without /var/run/ceph/pid, the init script will not know which pid to
>> kill. So it will do nothing.
>>
>> Try creating /var/run/ceph and other appropriate directories on all
>> nodes before starting the cluster.
>>
>> Colin
>>
>>
>>>
>>> On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>> Hi Upendra,
>>>>
>>>> Based on the output you posted, init-ceph is doing something on every
>>>> node. However, I only see a kill for certain nodes.
>>>>
>>>> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the
>>>> appropriate pid file exist on all nodes? What happens when you ssh in
>>>> to those nodes manually and run init-ceph stop?
>>>>
>>>> Colin
>>>>
>>>>
>>>> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>> Hi Colin
>>>>>
>>>>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
>>>>> i am using the default init script.(Found at /etc/init.d/ceph)
>>>>>
>>>>> Regarding issue 1)
>>>>>
>>>>> On the osd node I tried to start that osd but it did not work
>>>>> Steps followed
>>>>>
>>>>> 1) Started 3 nodes(ceph.conf is same as i sent earlier)
>>>>>
>>>>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
>>>>>
>>>>> 3)/etc/init.d/ceph start osd0 (Did this on first osd)
>>>>>
>>>>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
>>>>>
>>>>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes
>>>>>
>>>>> Regading Issue 2
>>>>> It says -x option is not available
>>>>>
>>>>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
>>>>> tried /etc/init.d/ceph -x -a stop
>>>>> but nothing worked.
>>>>>
>>>>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop
>>>>> ceph on all nodes but got the output as
>>>>>
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
>>>>> === mon.0 ===
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
>>>>> "/var/run/ceph/mon.0.pid"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
>>>>> "/var/lock/subsys/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
>>>>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
>>>>> -c unlimited ; while [ 1 ]; do
>>>>> [ -e /var/run/ceph/mon.0.pid ] || break
>>>>> pid=`cat /var/run/ceph/mon.0.pid`
>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>>> cmd="kill $pid"
>>>>> echo -n $cmd...
>>>>> $cmd
>>>>> sleep 1
>>>>> continue
>>>>> done
>>>>> break
>>>>> done"
>>>>> kill 1668...done
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
>>>>> === mon.1 ===
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
>>>>> "/var/run/ceph/mon.1.pid"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
>>>>> "/var/lock/subsys/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
>>>>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
>>>>> -c unlimited ; while [ 1 ]; do
>>>>> [ -e /var/run/ceph/mon.1.pid ] || break
>>>>> pid=`cat /var/run/ceph/mon.1.pid`
>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>>> cmd="kill $pid"
>>>>> echo -n $cmd...
>>>>> $cmd
>>>>> sleep 1
>>>>> continue
>>>>> done
>>>>> break
>>>>> done"
>>>>> done
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
>>>>> === mon.2 ===
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
>>>>> "/var/run/ceph/mon.2.pid"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
>>>>> "/var/lock/subsys/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
>>>>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
>>>>> -c unlimited ; while [ 1 ]; do
>>>>> [ -e /var/run/ceph/mon.2.pid ] || break
>>>>> pid=`cat /var/run/ceph/mon.2.pid`
>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>>> cmd="kill $pid"
>>>>> echo -n $cmd...
>>>>> $cmd
>>>>> sleep 1
>>>>> continue
>>>>> done
>>>>> break
>>>>> done"
>>>>> done
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
>>>>> === mds.0 ===
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
>>>>> "/var/run/ceph/mds.0.pid"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
>>>>> "/var/lock/subsys/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
>>>>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>>>> unlimited ; while [ 1 ]; do
>>>>> [ -e /var/run/ceph/mds.0.pid ] || break
>>>>> pid=`cat /var/run/ceph/mds.0.pid`
>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>>>> cmd="kill $pid"
>>>>> echo -n $cmd...
>>>>> $cmd
>>>>> sleep 1
>>>>> continue
>>>>> done
>>>>> break
>>>>> done"
>>>>> kill 1844...done
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
>>>>> === mds.1 ===
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
>>>>> "/var/run/ceph/mds.1.pid"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
>>>>> "/var/lock/subsys/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
>>>>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>>>> unlimited ; while [ 1 ]; do
>>>>> [ -e /var/run/ceph/mds.1.pid ] || break
>>>>> pid=`cat /var/run/ceph/mds.1.pid`
>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>>>> cmd="kill $pid"
>>>>> echo -n $cmd...
>>>>> $cmd
>>>>> sleep 1
>>>>> continue
>>>>> done
>>>>> break
>>>>> done"
>>>>> done
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
>>>>> === osd.0 ===
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
>>>>> "/var/run/ceph/osd.0.pid"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
>>>>> "/var/lock/subsys/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
>>>>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>>>> unlimited ; while [ 1 ]; do
>>>>> [ -e /var/run/ceph/osd.0.pid ] || break
>>>>> pid=`cat /var/run/ceph/osd.0.pid`
>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>>> cmd="kill $pid"
>>>>> echo -n $cmd...
>>>>> $cmd
>>>>> sleep 1
>>>>> continue
>>>>> done
>>>>> break
>>>>> done"
>>>>> kill 2033...done
>>>>> Unmounting Btrfs on ceph0:/data/osd0
>>>>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>>> /data/osd0 || true"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
>>>>> === osd.1 ===
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
>>>>> "/var/run/ceph/osd.1.pid"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
>>>>> "/var/lock/subsys/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
>>>>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>>>> unlimited ; while [ 1 ]; do
>>>>> [ -e /var/run/ceph/osd.1.pid ] || break
>>>>> pid=`cat /var/run/ceph/osd.1.pid`
>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>>> cmd="kill $pid"
>>>>> echo -n $cmd...
>>>>> $cmd
>>>>> sleep 1
>>>>> continue
>>>>> done
>>>>> break
>>>>> done"
>>>>> done
>>>>> Unmounting Btrfs on ceph1:/data/osd1
>>>>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>>> /data/osd1 || true"
>>>>> umount: /data/osd1: device is busy.
>>>>> (In some cases useful info about processes that use
>>>>> the device is found by lsof(8) or fuser(1))
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
>>>>> === osd.2 ===
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
>>>>> "/var/run/ceph/osd.2.pid"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
>>>>> "/var/lock/subsys/ceph"
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
>>>>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
>>>>> unlimited ; while [ 1 ]; do
>>>>> [ -e /var/run/ceph/osd.2.pid ] || break
>>>>> pid=`cat /var/run/ceph/osd.2.pid`
>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>>> cmd="kill $pid"
>>>>> echo -n $cmd...
>>>>> $cmd
>>>>> sleep 1
>>>>> continue
>>>>> done
>>>>> break
>>>>> done"
>>>>> done
>>>>> Unmounting Btrfs on ceph2:/data/osd2
>>>>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>>> /data/osd2 || true"
>>>>> umount: /data/osd2: device is busy.
>>>>> (In some cases useful info about processes that use
>>>>> the device is found by lsof(8) or fuser(1))
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>> Hi Upendra,
>>>>>>
>>>>>> Are you running init-ceph from the source directory? If you do that,
>>>>>> it will use the ceph.conf in the source directory itself, which is
>>>>>> probably not what you want. So it might be good to double-check that.
>>>>>>
>>>>>> If all else fails, running init-ceph with -x will show you exactly
>>>>>> what the script is doing. If all goes well, its exit status should be
>>>>>> 0. Are you getting exit status 0?
>>>>>>
>>>>>> Colin
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>> Hi Colin
>>>>>>>
>>>>>>> Here is my ceph.conf :
>>>>>>>
>>>>>>> [global]
>>>>>>> pid file = /var/run/ceph/$name.pid
>>>>>>> debug ms = 1
>>>>>>> [mon]
>>>>>>> mon data = /data/mon$id
>>>>>>> [mon.0]
>>>>>>> host = ceph0
>>>>>>> mon addr = 192.168.155.5:6789
>>>>>>> [mon.1]
>>>>>>> host = ceph1
>>>>>>> mon addr = 192.168.155.6:6789
>>>>>>> [mon.2]
>>>>>>> host = ceph2
>>>>>>> mon addr = 192.168.155.7:6789
>>>>>>> [mds]
>>>>>>>
>>>>>>> [mds0]
>>>>>>> host = ceph0
>>>>>>> [mds1]
>>>>>>> host = ceph1
>>>>>>>
>>>>>>> [osd]
>>>>>>> sudo = true
>>>>>>> osd data = /data/osd$id
>>>>>>> osd journal = /data/osd$id/journal
>>>>>>> osd journal size = 512
>>>>>>> osd use stale snap = true
>>>>>>> [osd0]
>>>>>>> host = ceph0
>>>>>>> btrfs devs = /dev/sdb
>>>>>>> [osd1]
>>>>>>> host = ceph1
>>>>>>> btrfs devs = /dev/sdb
>>>>>>> [osd2]
>>>>>>> host = ceph2
>>>>>>> btrfs devs = /dev/sdb
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>> I'm using head of line from the master branch. But that particular
>>>>>>>> code hasn't changed since January, which is 0.24.2 is from.
>>>>>>>>
>>>>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and
>>>>>>>> everything else local.
>>>>>>>>
>>>>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot
>>>>>>>> an issue that's causing the problems that you see.
>>>>>>>>
>>>>>>>> Colin
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>> Hi Colin
>>>>>>>>> I am using
>>>>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>>>>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>>>>>>>>
>>>>>>>>> Installed ceph form apt source.
>>>>>>>>>
>>>>>>>>> With above configurations i still able yo reproduce.
>>>>>>>>> Can you please share me ur configurations?
>>>>>>>>>
>>>>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>>>>>>>>> working and there is no error
>>>>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>>>>>>>>
>>>>>>>>>>> That is expected, unless you are running init-ceph on the same node as
>>>>>>>>>>> osd1 is on.
>>>>>>>>>>>
>>>>>>>>>>> It might be nice to have some kind of interface like "run command X on
>>>>>>>>>>> osd1", but init-ceph is not that.
>>>>>>>>>>>
>>>>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>>>>>>>>> on current node only
>>>>>>>>>>>> where as
>>>>>>>>>>>> /etc/init.d/ceph -a killall works fine.
>>>>>>>>>>>
>>>>>>>>>>> That sounds like a bug. I'll see if I can fix it.
>>>>>>>>>>
>>>>>>>>>> I'm afraid I can't reproduce this.
>>>>>>>>>>
>>>>>>>>>> I ran /etc/init.d/ceph -a stop
>>>>>>>>>>
>>>>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>>>>>>>>> the code, it looks correct.
>>>>>>>>>>
>>>>>>>>>> Colin
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Upendra.M
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>> Upendra.M
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>> Upendra.M
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Upendra.M
>>>
>>
>
>
>
> --
> Thanks and Regards,
> Upendra.M
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-03-01 17:34 ` Colin McCabe
@ 2011-03-02 9:18 ` Upendra Moturi
2011-03-02 19:26 ` Colin McCabe
0 siblings, 1 reply; 18+ messages in thread
From: Upendra Moturi @ 2011-03-02 9:18 UTC (permalink / raw)
To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava
I was on the same node as osd0 ,but still it did not work.
When i see /etc/init.d/ceph file,it is not even going into that start case
On Tue, Mar 1, 2011 at 11:04 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
> Hi Upendra,
>
> You should be on the same node as osd0 in order to start osd0.
>
> There is that wacky -a option, at least until we decide to remove it.
> But my advice is not to use that. Set up something like dsh.
>
> cheers,
> Colin
>
>
> On Tue, Mar 1, 2011 at 3:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>> Hi Colin
>>
>> Creating /var/run/ceph directory on every node has made it
>> working.(/etc/init.d/ceph -a stop)
>> But /etc/init.d/ceph start osd0 still does not work
>>
>> /etc/init.d/ceph -v start osd0
>> Gives output as
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
>>
>> I have tried this on same node where osd0 is present
>>
>> Had a look at /etc/init.d/ceph and it does not enter the case start.
>>
>>
>> On Mon, Feb 28, 2011 at 10:27 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>> On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>> Hi Colin
>>>>
>>>> /var/run/ceph exist on only one node(the node on which i start the
>>>> cluster with -a)
>>>
>>> Without /var/run/ceph/pid, the init script will not know which pid to
>>> kill. So it will do nothing.
>>>
>>> Try creating /var/run/ceph and other appropriate directories on all
>>> nodes before starting the cluster.
>>>
>>> Colin
>>>
>>>
>>>>
>>>> On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>> Hi Upendra,
>>>>>
>>>>> Based on the output you posted, init-ceph is doing something on every
>>>>> node. However, I only see a kill for certain nodes.
>>>>>
>>>>> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the
>>>>> appropriate pid file exist on all nodes? What happens when you ssh in
>>>>> to those nodes manually and run init-ceph stop?
>>>>>
>>>>> Colin
>>>>>
>>>>>
>>>>> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>> Hi Colin
>>>>>>
>>>>>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
>>>>>> i am using the default init script.(Found at /etc/init.d/ceph)
>>>>>>
>>>>>> Regarding issue 1)
>>>>>>
>>>>>> On the osd node I tried to start that osd but it did not work
>>>>>> Steps followed
>>>>>>
>>>>>> 1) Started 3 nodes(ceph.conf is same as i sent earlier)
>>>>>>
>>>>>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
>>>>>>
>>>>>> 3)/etc/init.d/ceph start osd0 (Did this on first osd)
>>>>>>
>>>>>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
>>>>>>
>>>>>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes
>>>>>>
>>>>>> Regading Issue 2
>>>>>> It says -x option is not available
>>>>>>
>>>>>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
>>>>>> tried /etc/init.d/ceph -x -a stop
>>>>>> but nothing worked.
>>>>>>
>>>>>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop
>>>>>> ceph on all nodes but got the output as
>>>>>>
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
>>>>>> === mon.0 ===
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
>>>>>> "/var/run/ceph/mon.0.pid"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
>>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
>>>>>> "/var/lock/subsys/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
>>>>>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
>>>>>> -c unlimited ; while [ 1 ]; do
>>>>>> [ -e /var/run/ceph/mon.0.pid ] || break
>>>>>> pid=`cat /var/run/ceph/mon.0.pid`
>>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>>>> cmd="kill $pid"
>>>>>> echo -n $cmd...
>>>>>> $cmd
>>>>>> sleep 1
>>>>>> continue
>>>>>> done
>>>>>> break
>>>>>> done"
>>>>>> kill 1668...done
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
>>>>>> === mon.1 ===
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
>>>>>> "/var/run/ceph/mon.1.pid"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
>>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
>>>>>> "/var/lock/subsys/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
>>>>>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
>>>>>> -c unlimited ; while [ 1 ]; do
>>>>>> [ -e /var/run/ceph/mon.1.pid ] || break
>>>>>> pid=`cat /var/run/ceph/mon.1.pid`
>>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>>>> cmd="kill $pid"
>>>>>> echo -n $cmd...
>>>>>> $cmd
>>>>>> sleep 1
>>>>>> continue
>>>>>> done
>>>>>> break
>>>>>> done"
>>>>>> done
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
>>>>>> === mon.2 ===
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
>>>>>> "/var/run/ceph/mon.2.pid"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
>>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
>>>>>> "/var/lock/subsys/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
>>>>>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
>>>>>> -c unlimited ; while [ 1 ]; do
>>>>>> [ -e /var/run/ceph/mon.2.pid ] || break
>>>>>> pid=`cat /var/run/ceph/mon.2.pid`
>>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>>>> cmd="kill $pid"
>>>>>> echo -n $cmd...
>>>>>> $cmd
>>>>>> sleep 1
>>>>>> continue
>>>>>> done
>>>>>> break
>>>>>> done"
>>>>>> done
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
>>>>>> === mds.0 ===
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
>>>>>> "/var/run/ceph/mds.0.pid"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
>>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
>>>>>> "/var/lock/subsys/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
>>>>>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>>>>> unlimited ; while [ 1 ]; do
>>>>>> [ -e /var/run/ceph/mds.0.pid ] || break
>>>>>> pid=`cat /var/run/ceph/mds.0.pid`
>>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>>>>> cmd="kill $pid"
>>>>>> echo -n $cmd...
>>>>>> $cmd
>>>>>> sleep 1
>>>>>> continue
>>>>>> done
>>>>>> break
>>>>>> done"
>>>>>> kill 1844...done
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
>>>>>> === mds.1 ===
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
>>>>>> "/var/run/ceph/mds.1.pid"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
>>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
>>>>>> "/var/lock/subsys/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
>>>>>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>>>>> unlimited ; while [ 1 ]; do
>>>>>> [ -e /var/run/ceph/mds.1.pid ] || break
>>>>>> pid=`cat /var/run/ceph/mds.1.pid`
>>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>>>>> cmd="kill $pid"
>>>>>> echo -n $cmd...
>>>>>> $cmd
>>>>>> sleep 1
>>>>>> continue
>>>>>> done
>>>>>> break
>>>>>> done"
>>>>>> done
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
>>>>>> === osd.0 ===
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
>>>>>> "/var/run/ceph/osd.0.pid"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
>>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
>>>>>> "/var/lock/subsys/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
>>>>>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>>>>> unlimited ; while [ 1 ]; do
>>>>>> [ -e /var/run/ceph/osd.0.pid ] || break
>>>>>> pid=`cat /var/run/ceph/osd.0.pid`
>>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>>>> cmd="kill $pid"
>>>>>> echo -n $cmd...
>>>>>> $cmd
>>>>>> sleep 1
>>>>>> continue
>>>>>> done
>>>>>> break
>>>>>> done"
>>>>>> kill 2033...done
>>>>>> Unmounting Btrfs on ceph0:/data/osd0
>>>>>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>>>> /data/osd0 || true"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
>>>>>> === osd.1 ===
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
>>>>>> "/var/run/ceph/osd.1.pid"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
>>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
>>>>>> "/var/lock/subsys/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
>>>>>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>>>>> unlimited ; while [ 1 ]; do
>>>>>> [ -e /var/run/ceph/osd.1.pid ] || break
>>>>>> pid=`cat /var/run/ceph/osd.1.pid`
>>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>>>> cmd="kill $pid"
>>>>>> echo -n $cmd...
>>>>>> $cmd
>>>>>> sleep 1
>>>>>> continue
>>>>>> done
>>>>>> break
>>>>>> done"
>>>>>> done
>>>>>> Unmounting Btrfs on ceph1:/data/osd1
>>>>>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>>>> /data/osd1 || true"
>>>>>> umount: /data/osd1: device is busy.
>>>>>> (In some cases useful info about processes that use
>>>>>> the device is found by lsof(8) or fuser(1))
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
>>>>>> === osd.2 ===
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
>>>>>> "/var/run/ceph/osd.2.pid"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
>>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
>>>>>> "/var/lock/subsys/ceph"
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
>>>>>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
>>>>>> unlimited ; while [ 1 ]; do
>>>>>> [ -e /var/run/ceph/osd.2.pid ] || break
>>>>>> pid=`cat /var/run/ceph/osd.2.pid`
>>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>>>> cmd="kill $pid"
>>>>>> echo -n $cmd...
>>>>>> $cmd
>>>>>> sleep 1
>>>>>> continue
>>>>>> done
>>>>>> break
>>>>>> done"
>>>>>> done
>>>>>> Unmounting Btrfs on ceph2:/data/osd2
>>>>>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>>>> /data/osd2 || true"
>>>>>> umount: /data/osd2: device is busy.
>>>>>> (In some cases useful info about processes that use
>>>>>> the device is found by lsof(8) or fuser(1))
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>> Hi Upendra,
>>>>>>>
>>>>>>> Are you running init-ceph from the source directory? If you do that,
>>>>>>> it will use the ceph.conf in the source directory itself, which is
>>>>>>> probably not what you want. So it might be good to double-check that.
>>>>>>>
>>>>>>> If all else fails, running init-ceph with -x will show you exactly
>>>>>>> what the script is doing. If all goes well, its exit status should be
>>>>>>> 0. Are you getting exit status 0?
>>>>>>>
>>>>>>> Colin
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>> Hi Colin
>>>>>>>>
>>>>>>>> Here is my ceph.conf :
>>>>>>>>
>>>>>>>> [global]
>>>>>>>> pid file = /var/run/ceph/$name.pid
>>>>>>>> debug ms = 1
>>>>>>>> [mon]
>>>>>>>> mon data = /data/mon$id
>>>>>>>> [mon.0]
>>>>>>>> host = ceph0
>>>>>>>> mon addr = 192.168.155.5:6789
>>>>>>>> [mon.1]
>>>>>>>> host = ceph1
>>>>>>>> mon addr = 192.168.155.6:6789
>>>>>>>> [mon.2]
>>>>>>>> host = ceph2
>>>>>>>> mon addr = 192.168.155.7:6789
>>>>>>>> [mds]
>>>>>>>>
>>>>>>>> [mds0]
>>>>>>>> host = ceph0
>>>>>>>> [mds1]
>>>>>>>> host = ceph1
>>>>>>>>
>>>>>>>> [osd]
>>>>>>>> sudo = true
>>>>>>>> osd data = /data/osd$id
>>>>>>>> osd journal = /data/osd$id/journal
>>>>>>>> osd journal size = 512
>>>>>>>> osd use stale snap = true
>>>>>>>> [osd0]
>>>>>>>> host = ceph0
>>>>>>>> btrfs devs = /dev/sdb
>>>>>>>> [osd1]
>>>>>>>> host = ceph1
>>>>>>>> btrfs devs = /dev/sdb
>>>>>>>> [osd2]
>>>>>>>> host = ceph2
>>>>>>>> btrfs devs = /dev/sdb
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>> I'm using head of line from the master branch. But that particular
>>>>>>>>> code hasn't changed since January, which is 0.24.2 is from.
>>>>>>>>>
>>>>>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and
>>>>>>>>> everything else local.
>>>>>>>>>
>>>>>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot
>>>>>>>>> an issue that's causing the problems that you see.
>>>>>>>>>
>>>>>>>>> Colin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>>> Hi Colin
>>>>>>>>>> I am using
>>>>>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>>>>>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>>>>>>>>>
>>>>>>>>>> Installed ceph form apt source.
>>>>>>>>>>
>>>>>>>>>> With above configurations i still able yo reproduce.
>>>>>>>>>> Can you please share me ur configurations?
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>>>>>>>>>> working and there is no error
>>>>>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>>>>>>>>>
>>>>>>>>>>>> That is expected, unless you are running init-ceph on the same node as
>>>>>>>>>>>> osd1 is on.
>>>>>>>>>>>>
>>>>>>>>>>>> It might be nice to have some kind of interface like "run command X on
>>>>>>>>>>>> osd1", but init-ceph is not that.
>>>>>>>>>>>>
>>>>>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>>>>>>>>>> on current node only
>>>>>>>>>>>>> where as
>>>>>>>>>>>>> /etc/init.d/ceph -a killall works fine.
>>>>>>>>>>>>
>>>>>>>>>>>> That sounds like a bug. I'll see if I can fix it.
>>>>>>>>>>>
>>>>>>>>>>> I'm afraid I can't reproduce this.
>>>>>>>>>>>
>>>>>>>>>>> I ran /etc/init.d/ceph -a stop
>>>>>>>>>>>
>>>>>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>>>>>>>>>> the code, it looks correct.
>>>>>>>>>>>
>>>>>>>>>>> Colin
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Thanks and Regards,
>>>>>>>>>> Upendra.M
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks and Regards,
>>>>>>>> Upendra.M
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks and Regards,
>>>>>> Upendra.M
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Upendra.M
>>>>
>>>
>>
>>
>>
>> --
>> Thanks and Regards,
>> Upendra.M
>>
>
--
Thanks and Regards,
Upendra.M
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts
2011-03-02 9:18 ` Upendra Moturi
@ 2011-03-02 19:26 ` Colin McCabe
0 siblings, 0 replies; 18+ messages in thread
From: Colin McCabe @ 2011-03-02 19:26 UTC (permalink / raw)
To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava
Hi Upendra,
Can you try chatting with us on #ceph on irc.oftc.net?
C.
On Wed, Mar 2, 2011 at 1:18 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
> I was on the same node as osd0 ,but still it did not work.
> When i see /etc/init.d/ceph file,it is not even going into that start case
>
>
> On Tue, Mar 1, 2011 at 11:04 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>> Hi Upendra,
>>
>> You should be on the same node as osd0 in order to start osd0.
>>
>> There is that wacky -a option, at least until we decide to remove it.
>> But my advice is not to use that. Set up something like dsh.
>>
>> cheers,
>> Colin
>>
>>
>> On Tue, Mar 1, 2011 at 3:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>> Hi Colin
>>>
>>> Creating /var/run/ceph directory on every node has made it
>>> working.(/etc/init.d/ceph -a stop)
>>> But /etc/init.d/ceph start osd0 still does not work
>>>
>>> /etc/init.d/ceph -v start osd0
>>> Gives output as
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
>>>
>>> I have tried this on same node where osd0 is present
>>>
>>> Had a look at /etc/init.d/ceph and it does not enter the case start.
>>>
>>>
>>> On Mon, Feb 28, 2011 at 10:27 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>> On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>> Hi Colin
>>>>>
>>>>> /var/run/ceph exist on only one node(the node on which i start the
>>>>> cluster with -a)
>>>>
>>>> Without /var/run/ceph/pid, the init script will not know which pid to
>>>> kill. So it will do nothing.
>>>>
>>>> Try creating /var/run/ceph and other appropriate directories on all
>>>> nodes before starting the cluster.
>>>>
>>>> Colin
>>>>
>>>>
>>>>>
>>>>> On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>> Hi Upendra,
>>>>>>
>>>>>> Based on the output you posted, init-ceph is doing something on every
>>>>>> node. However, I only see a kill for certain nodes.
>>>>>>
>>>>>> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the
>>>>>> appropriate pid file exist on all nodes? What happens when you ssh in
>>>>>> to those nodes manually and run init-ceph stop?
>>>>>>
>>>>>> Colin
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>> Hi Colin
>>>>>>>
>>>>>>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get
>>>>>>> i am using the default init script.(Found at /etc/init.d/ceph)
>>>>>>>
>>>>>>> Regarding issue 1)
>>>>>>>
>>>>>>> On the osd node I tried to start that osd but it did not work
>>>>>>> Steps followed
>>>>>>>
>>>>>>> 1) Started 3 nodes(ceph.conf is same as i sent earlier)
>>>>>>>
>>>>>>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin
>>>>>>>
>>>>>>> 3)/etc/init.d/ceph start osd0 (Did this on first osd)
>>>>>>>
>>>>>>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process
>>>>>>>
>>>>>>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes
>>>>>>>
>>>>>>> Regading Issue 2
>>>>>>> It says -x option is not available
>>>>>>>
>>>>>>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even
>>>>>>> tried /etc/init.d/ceph -x -a stop
>>>>>>> but nothing worked.
>>>>>>>
>>>>>>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop
>>>>>>> ceph on all nodes but got the output as
>>>>>>>
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph"
>>>>>>> === mon.0 ===
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file"
>>>>>>> "/var/run/ceph/mon.0.pid"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" ""
>>>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file"
>>>>>>> "/var/lock/subsys/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" ""
>>>>>>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit
>>>>>>> -c unlimited ; while [ 1 ]; do
>>>>>>> [ -e /var/run/ceph/mon.0.pid ] || break
>>>>>>> pid=`cat /var/run/ceph/mon.0.pid`
>>>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>>>>> cmd="kill $pid"
>>>>>>> echo -n $cmd...
>>>>>>> $cmd
>>>>>>> sleep 1
>>>>>>> continue
>>>>>>> done
>>>>>>> break
>>>>>>> done"
>>>>>>> kill 1668...done
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph"
>>>>>>> === mon.1 ===
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file"
>>>>>>> "/var/run/ceph/mon.1.pid"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" ""
>>>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file"
>>>>>>> "/var/lock/subsys/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" ""
>>>>>>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit
>>>>>>> -c unlimited ; while [ 1 ]; do
>>>>>>> [ -e /var/run/ceph/mon.1.pid ] || break
>>>>>>> pid=`cat /var/run/ceph/mon.1.pid`
>>>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>>>>> cmd="kill $pid"
>>>>>>> echo -n $cmd...
>>>>>>> $cmd
>>>>>>> sleep 1
>>>>>>> continue
>>>>>>> done
>>>>>>> break
>>>>>>> done"
>>>>>>> done
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph"
>>>>>>> === mon.2 ===
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file"
>>>>>>> "/var/run/ceph/mon.2.pid"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" ""
>>>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file"
>>>>>>> "/var/lock/subsys/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" ""
>>>>>>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit
>>>>>>> -c unlimited ; while [ 1 ]; do
>>>>>>> [ -e /var/run/ceph/mon.2.pid ] || break
>>>>>>> pid=`cat /var/run/ceph/mon.2.pid`
>>>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do
>>>>>>> cmd="kill $pid"
>>>>>>> echo -n $cmd...
>>>>>>> $cmd
>>>>>>> sleep 1
>>>>>>> continue
>>>>>>> done
>>>>>>> break
>>>>>>> done"
>>>>>>> done
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph"
>>>>>>> === mds.0 ===
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file"
>>>>>>> "/var/run/ceph/mds.0.pid"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" ""
>>>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file"
>>>>>>> "/var/lock/subsys/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" ""
>>>>>>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>>>>>> unlimited ; while [ 1 ]; do
>>>>>>> [ -e /var/run/ceph/mds.0.pid ] || break
>>>>>>> pid=`cat /var/run/ceph/mds.0.pid`
>>>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>>>>>> cmd="kill $pid"
>>>>>>> echo -n $cmd...
>>>>>>> $cmd
>>>>>>> sleep 1
>>>>>>> continue
>>>>>>> done
>>>>>>> break
>>>>>>> done"
>>>>>>> kill 1844...done
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph"
>>>>>>> === mds.1 ===
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file"
>>>>>>> "/var/run/ceph/mds.1.pid"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" ""
>>>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file"
>>>>>>> "/var/lock/subsys/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" ""
>>>>>>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>>>>>> unlimited ; while [ 1 ]; do
>>>>>>> [ -e /var/run/ceph/mds.1.pid ] || break
>>>>>>> pid=`cat /var/run/ceph/mds.1.pid`
>>>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do
>>>>>>> cmd="kill $pid"
>>>>>>> echo -n $cmd...
>>>>>>> $cmd
>>>>>>> sleep 1
>>>>>>> continue
>>>>>>> done
>>>>>>> break
>>>>>>> done"
>>>>>>> done
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph"
>>>>>>> === osd.0 ===
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file"
>>>>>>> "/var/run/ceph/osd.0.pid"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" ""
>>>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file"
>>>>>>> "/var/lock/subsys/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" ""
>>>>>>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c
>>>>>>> unlimited ; while [ 1 ]; do
>>>>>>> [ -e /var/run/ceph/osd.0.pid ] || break
>>>>>>> pid=`cat /var/run/ceph/osd.0.pid`
>>>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>>>>> cmd="kill $pid"
>>>>>>> echo -n $cmd...
>>>>>>> $cmd
>>>>>>> sleep 1
>>>>>>> continue
>>>>>>> done
>>>>>>> break
>>>>>>> done"
>>>>>>> kill 2033...done
>>>>>>> Unmounting Btrfs on ceph0:/data/osd0
>>>>>>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>>>>> /data/osd0 || true"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph"
>>>>>>> === osd.1 ===
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file"
>>>>>>> "/var/run/ceph/osd.1.pid"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" ""
>>>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file"
>>>>>>> "/var/lock/subsys/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" ""
>>>>>>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c
>>>>>>> unlimited ; while [ 1 ]; do
>>>>>>> [ -e /var/run/ceph/osd.1.pid ] || break
>>>>>>> pid=`cat /var/run/ceph/osd.1.pid`
>>>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>>>>> cmd="kill $pid"
>>>>>>> echo -n $cmd...
>>>>>>> $cmd
>>>>>>> sleep 1
>>>>>>> continue
>>>>>>> done
>>>>>>> break
>>>>>>> done"
>>>>>>> done
>>>>>>> Unmounting Btrfs on ceph1:/data/osd1
>>>>>>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>>>>> /data/osd1 || true"
>>>>>>> umount: /data/osd1: device is busy.
>>>>>>> (In some cases useful info about processes that use
>>>>>>> the device is found by lsof(8) or fuser(1))
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph"
>>>>>>> === osd.2 ===
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file"
>>>>>>> "/var/run/ceph/osd.2.pid"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" ""
>>>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file"
>>>>>>> "/var/lock/subsys/ceph"
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" ""
>>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" ""
>>>>>>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c
>>>>>>> unlimited ; while [ 1 ]; do
>>>>>>> [ -e /var/run/ceph/osd.2.pid ] || break
>>>>>>> pid=`cat /var/run/ceph/osd.2.pid`
>>>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do
>>>>>>> cmd="kill $pid"
>>>>>>> echo -n $cmd...
>>>>>>> $cmd
>>>>>>> sleep 1
>>>>>>> continue
>>>>>>> done
>>>>>>> break
>>>>>>> done"
>>>>>>> done
>>>>>>> Unmounting Btrfs on ceph2:/data/osd2
>>>>>>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount
>>>>>>> /data/osd2 || true"
>>>>>>> umount: /data/osd2: device is busy.
>>>>>>> (In some cases useful info about processes that use
>>>>>>> the device is found by lsof(8) or fuser(1))
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>> Hi Upendra,
>>>>>>>>
>>>>>>>> Are you running init-ceph from the source directory? If you do that,
>>>>>>>> it will use the ceph.conf in the source directory itself, which is
>>>>>>>> probably not what you want. So it might be good to double-check that.
>>>>>>>>
>>>>>>>> If all else fails, running init-ceph with -x will show you exactly
>>>>>>>> what the script is doing. If all goes well, its exit status should be
>>>>>>>> 0. Are you getting exit status 0?
>>>>>>>>
>>>>>>>> Colin
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>> Hi Colin
>>>>>>>>>
>>>>>>>>> Here is my ceph.conf :
>>>>>>>>>
>>>>>>>>> [global]
>>>>>>>>> pid file = /var/run/ceph/$name.pid
>>>>>>>>> debug ms = 1
>>>>>>>>> [mon]
>>>>>>>>> mon data = /data/mon$id
>>>>>>>>> [mon.0]
>>>>>>>>> host = ceph0
>>>>>>>>> mon addr = 192.168.155.5:6789
>>>>>>>>> [mon.1]
>>>>>>>>> host = ceph1
>>>>>>>>> mon addr = 192.168.155.6:6789
>>>>>>>>> [mon.2]
>>>>>>>>> host = ceph2
>>>>>>>>> mon addr = 192.168.155.7:6789
>>>>>>>>> [mds]
>>>>>>>>>
>>>>>>>>> [mds0]
>>>>>>>>> host = ceph0
>>>>>>>>> [mds1]
>>>>>>>>> host = ceph1
>>>>>>>>>
>>>>>>>>> [osd]
>>>>>>>>> sudo = true
>>>>>>>>> osd data = /data/osd$id
>>>>>>>>> osd journal = /data/osd$id/journal
>>>>>>>>> osd journal size = 512
>>>>>>>>> osd use stale snap = true
>>>>>>>>> [osd0]
>>>>>>>>> host = ceph0
>>>>>>>>> btrfs devs = /dev/sdb
>>>>>>>>> [osd1]
>>>>>>>>> host = ceph1
>>>>>>>>> btrfs devs = /dev/sdb
>>>>>>>>> [osd2]
>>>>>>>>> host = ceph2
>>>>>>>>> btrfs devs = /dev/sdb
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>>> I'm using head of line from the master branch. But that particular
>>>>>>>>>> code hasn't changed since January, which is 0.24.2 is from.
>>>>>>>>>>
>>>>>>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and
>>>>>>>>>> everything else local.
>>>>>>>>>>
>>>>>>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot
>>>>>>>>>> an issue that's causing the problems that you see.
>>>>>>>>>>
>>>>>>>>>> Colin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>>>> Hi Colin
>>>>>>>>>>> I am using
>>>>>>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7
>>>>>>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic
>>>>>>>>>>>
>>>>>>>>>>> Installed ceph form apt source.
>>>>>>>>>>>
>>>>>>>>>>> With above configurations i still able yo reproduce.
>>>>>>>>>>> Can you please share me ur configurations?
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>>>>>>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote:
>>>>>>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not
>>>>>>>>>>>>>> working and there is no error
>>>>>>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error
>>>>>>>>>>>>>
>>>>>>>>>>>>> That is expected, unless you are running init-ceph on the same node as
>>>>>>>>>>>>> osd1 is on.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It might be nice to have some kind of interface like "run command X on
>>>>>>>>>>>>> osd1", but init-ceph is not that.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops
>>>>>>>>>>>>>> on current node only
>>>>>>>>>>>>>> where as
>>>>>>>>>>>>>> /etc/init.d/ceph -a killall works fine.
>>>>>>>>>>>>>
>>>>>>>>>>>>> That sounds like a bug. I'll see if I can fix it.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm afraid I can't reproduce this.
>>>>>>>>>>>>
>>>>>>>>>>>> I ran /etc/init.d/ceph -a stop
>>>>>>>>>>>>
>>>>>>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at
>>>>>>>>>>>> the code, it looks correct.
>>>>>>>>>>>>
>>>>>>>>>>>> Colin
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Upendra.M
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Upendra.M
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>> Upendra.M
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks and Regards,
>>>>> Upendra.M
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Upendra.M
>>>
>>
>
>
>
> --
> Thanks and Regards,
> Upendra.M
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2011-03-02 19:26 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-02-16 14:44 problem with ceph init scripts Upendra Moturi
2011-02-16 21:41 ` Colin McCabe
2011-02-16 22:00 ` Tommi Virtanen
2011-02-16 22:01 ` Colin McCabe
2011-02-17 7:01 ` Upendra Moturi
2011-02-17 7:36 ` Colin McCabe
2011-02-17 9:55 ` Upendra Moturi
2011-02-18 7:05 ` Colin McCabe
2011-02-18 14:11 ` Upendra Moturi
2011-02-26 0:28 ` Colin McCabe
2011-02-28 12:00 ` Upendra Moturi
2011-02-28 16:57 ` Colin McCabe
2011-03-01 11:55 ` Upendra Moturi
2011-03-01 17:34 ` Colin McCabe
2011-03-02 9:18 ` Upendra Moturi
2011-03-02 19:26 ` Colin McCabe
2011-02-26 0:28 ` Gregory Farnum
2011-02-28 8:40 ` Upendra Moturi
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.