* problem with ceph init scripts @ 2011-02-16 14:44 Upendra Moturi 2011-02-16 21:41 ` Colin McCabe 0 siblings, 1 reply; 18+ messages in thread From: Upendra Moturi @ 2011-02-16 14:44 UTC (permalink / raw) To: sage, ceph-devel, Suresh Mandava Hi, With ceph 0.24.2 and linux 2.6.38-2-generic(ubuntu) ,i face problems with ceph init scripts /etc/init.d/ceph -a start ----works fine But if we want to start a particular osd or mon or mds ,its not working and there is no error eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops on current node only where as /etc/init.d/ceph -a killall works fine. -- Thanks and Regards, Upendra.M ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-16 14:44 problem with ceph init scripts Upendra Moturi @ 2011-02-16 21:41 ` Colin McCabe 2011-02-16 22:00 ` Tommi Virtanen 2011-02-16 22:01 ` Colin McCabe 0 siblings, 2 replies; 18+ messages in thread From: Colin McCabe @ 2011-02-16 21:41 UTC (permalink / raw) To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > But if we want to start a particular osd or mon or mds ,its not > working and there is no error > eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error That is expected, unless you are running init-ceph on the same node as osd1 is on. It might be nice to have some kind of interface like "run command X on osd1", but init-ceph is not that. > /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops > on current node only > where as > /etc/init.d/ceph -a killall works fine. That sounds like a bug. I'll see if I can fix it. cheers, Colin -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-16 21:41 ` Colin McCabe @ 2011-02-16 22:00 ` Tommi Virtanen 2011-02-16 22:01 ` Colin McCabe 1 sibling, 0 replies; 18+ messages in thread From: Tommi Virtanen @ 2011-02-16 22:00 UTC (permalink / raw) To: Colin McCabe; +Cc: Upendra Moturi, sage, ceph-devel, Suresh Mandava On Wed, Feb 16, 2011 at 01:41:44PM -0800, Colin McCabe wrote: > It might be nice to have some kind of interface like "run command X on > osd1", but init-ceph is not that. For general cluster administration tasks, let me recommend tools such as: dsh: like "ssh HOST COMMAND" but does multiple hosts at once http://www.netfort.gr.jp/~dancer/software/dsh.html Fabric: remote deployment and remote command running utility (more oriented toward scripting a solution than ad hoc) http://fabfile.org/ Puppet, Chef, Kokki: more full-blown cluster management solutions; based on the idea of writing recipes and running them automatically http://www.puppetlabs.com/ http://www.opscode.com/chef/ http://samuelks.com/kokki/ -- :(){ :|:&};: ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-16 21:41 ` Colin McCabe 2011-02-16 22:00 ` Tommi Virtanen @ 2011-02-16 22:01 ` Colin McCabe 2011-02-17 7:01 ` Upendra Moturi 1 sibling, 1 reply; 18+ messages in thread From: Colin McCabe @ 2011-02-16 22:01 UTC (permalink / raw) To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >> But if we want to start a particular osd or mon or mds ,its not >> working and there is no error >> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error > > That is expected, unless you are running init-ceph on the same node as > osd1 is on. > > It might be nice to have some kind of interface like "run command X on > osd1", but init-ceph is not that. > >> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >> on current node only >> where as >> /etc/init.d/ceph -a killall works fine. > > That sounds like a bug. I'll see if I can fix it. I'm afraid I can't reproduce this. I ran /etc/init.d/ceph -a stop and it stopped ceph daemons running on remote nodes too. Looking at the code, it looks correct. Colin -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-16 22:01 ` Colin McCabe @ 2011-02-17 7:01 ` Upendra Moturi 2011-02-17 7:36 ` Colin McCabe 0 siblings, 1 reply; 18+ messages in thread From: Upendra Moturi @ 2011-02-17 7:01 UTC (permalink / raw) To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava Hi Colin I am using ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic Installed ceph form apt source. With above configurations i still able yo reproduce. Can you please share me ur configurations? On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>> But if we want to start a particular osd or mon or mds ,its not >>> working and there is no error >>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >> >> That is expected, unless you are running init-ceph on the same node as >> osd1 is on. >> >> It might be nice to have some kind of interface like "run command X on >> osd1", but init-ceph is not that. >> >>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>> on current node only >>> where as >>> /etc/init.d/ceph -a killall works fine. >> >> That sounds like a bug. I'll see if I can fix it. > > I'm afraid I can't reproduce this. > > I ran /etc/init.d/ceph -a stop > > and it stopped ceph daemons running on remote nodes too. Looking at > the code, it looks correct. > > Colin > -- Thanks and Regards, Upendra.M -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-17 7:01 ` Upendra Moturi @ 2011-02-17 7:36 ` Colin McCabe 2011-02-17 9:55 ` Upendra Moturi 0 siblings, 1 reply; 18+ messages in thread From: Colin McCabe @ 2011-02-17 7:36 UTC (permalink / raw) To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava I'm using head of line from the master branch. But that particular code hasn't changed since January, which is 0.24.2 is from. In my ceph.conf, I just had an osd that was on a remote machine, and everything else local. If you could post your ceph.conf here or in IRC, perhaps we might spot an issue that's causing the problems that you see. Colin On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > Hi Colin > I am using > ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 > ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic > > Installed ceph form apt source. > > With above configurations i still able yo reproduce. > Can you please share me ur configurations? > > On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>> But if we want to start a particular osd or mon or mds ,its not >>>> working and there is no error >>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>> >>> That is expected, unless you are running init-ceph on the same node as >>> osd1 is on. >>> >>> It might be nice to have some kind of interface like "run command X on >>> osd1", but init-ceph is not that. >>> >>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>> on current node only >>>> where as >>>> /etc/init.d/ceph -a killall works fine. >>> >>> That sounds like a bug. I'll see if I can fix it. >> >> I'm afraid I can't reproduce this. >> >> I ran /etc/init.d/ceph -a stop >> >> and it stopped ceph daemons running on remote nodes too. Looking at >> the code, it looks correct. >> >> Colin >> > > > > -- > Thanks and Regards, > Upendra.M > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-17 7:36 ` Colin McCabe @ 2011-02-17 9:55 ` Upendra Moturi 2011-02-18 7:05 ` Colin McCabe 0 siblings, 1 reply; 18+ messages in thread From: Upendra Moturi @ 2011-02-17 9:55 UTC (permalink / raw) To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava Hi Colin Here is my ceph.conf : [global] pid file = /var/run/ceph/$name.pid debug ms = 1 [mon] mon data = /data/mon$id [mon.0] host = ceph0 mon addr = 192.168.155.5:6789 [mon.1] host = ceph1 mon addr = 192.168.155.6:6789 [mon.2] host = ceph2 mon addr = 192.168.155.7:6789 [mds] [mds0] host = ceph0 [mds1] host = ceph1 [osd] sudo = true osd data = /data/osd$id osd journal = /data/osd$id/journal osd journal size = 512 osd use stale snap = true [osd0] host = ceph0 btrfs devs = /dev/sdb [osd1] host = ceph1 btrfs devs = /dev/sdb [osd2] host = ceph2 btrfs devs = /dev/sdb On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > I'm using head of line from the master branch. But that particular > code hasn't changed since January, which is 0.24.2 is from. > > In my ceph.conf, I just had an osd that was on a remote machine, and > everything else local. > > If you could post your ceph.conf here or in IRC, perhaps we might spot > an issue that's causing the problems that you see. > > Colin > > > On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >> Hi Colin >> I am using >> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >> >> Installed ceph form apt source. >> >> With above configurations i still able yo reproduce. >> Can you please share me ur configurations? >> >> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>> But if we want to start a particular osd or mon or mds ,its not >>>>> working and there is no error >>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>> >>>> That is expected, unless you are running init-ceph on the same node as >>>> osd1 is on. >>>> >>>> It might be nice to have some kind of interface like "run command X on >>>> osd1", but init-ceph is not that. >>>> >>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>> on current node only >>>>> where as >>>>> /etc/init.d/ceph -a killall works fine. >>>> >>>> That sounds like a bug. I'll see if I can fix it. >>> >>> I'm afraid I can't reproduce this. >>> >>> I ran /etc/init.d/ceph -a stop >>> >>> and it stopped ceph daemons running on remote nodes too. Looking at >>> the code, it looks correct. >>> >>> Colin >>> >> >> >> >> -- >> Thanks and Regards, >> Upendra.M >> > -- Thanks and Regards, Upendra.M -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-17 9:55 ` Upendra Moturi @ 2011-02-18 7:05 ` Colin McCabe 2011-02-18 14:11 ` Upendra Moturi 0 siblings, 1 reply; 18+ messages in thread From: Colin McCabe @ 2011-02-18 7:05 UTC (permalink / raw) To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava Hi Upendra, Are you running init-ceph from the source directory? If you do that, it will use the ceph.conf in the source directory itself, which is probably not what you want. So it might be good to double-check that. If all else fails, running init-ceph with -x will show you exactly what the script is doing. If all goes well, its exit status should be 0. Are you getting exit status 0? Colin On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > Hi Colin > > Here is my ceph.conf : > > [global] > pid file = /var/run/ceph/$name.pid > debug ms = 1 > [mon] > mon data = /data/mon$id > [mon.0] > host = ceph0 > mon addr = 192.168.155.5:6789 > [mon.1] > host = ceph1 > mon addr = 192.168.155.6:6789 > [mon.2] > host = ceph2 > mon addr = 192.168.155.7:6789 > [mds] > > [mds0] > host = ceph0 > [mds1] > host = ceph1 > > [osd] > sudo = true > osd data = /data/osd$id > osd journal = /data/osd$id/journal > osd journal size = 512 > osd use stale snap = true > [osd0] > host = ceph0 > btrfs devs = /dev/sdb > [osd1] > host = ceph1 > btrfs devs = /dev/sdb > [osd2] > host = ceph2 > btrfs devs = /dev/sdb > > > > > On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >> I'm using head of line from the master branch. But that particular >> code hasn't changed since January, which is 0.24.2 is from. >> >> In my ceph.conf, I just had an osd that was on a remote machine, and >> everything else local. >> >> If you could post your ceph.conf here or in IRC, perhaps we might spot >> an issue that's causing the problems that you see. >> >> Colin >> >> >> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>> Hi Colin >>> I am using >>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>> >>> Installed ceph form apt source. >>> >>> With above configurations i still able yo reproduce. >>> Can you please share me ur configurations? >>> >>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>> working and there is no error >>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>> >>>>> That is expected, unless you are running init-ceph on the same node as >>>>> osd1 is on. >>>>> >>>>> It might be nice to have some kind of interface like "run command X on >>>>> osd1", but init-ceph is not that. >>>>> >>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>> on current node only >>>>>> where as >>>>>> /etc/init.d/ceph -a killall works fine. >>>>> >>>>> That sounds like a bug. I'll see if I can fix it. >>>> >>>> I'm afraid I can't reproduce this. >>>> >>>> I ran /etc/init.d/ceph -a stop >>>> >>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>> the code, it looks correct. >>>> >>>> Colin >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> Upendra.M >>> >> > > > > -- > Thanks and Regards, > Upendra.M > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-18 7:05 ` Colin McCabe @ 2011-02-18 14:11 ` Upendra Moturi 2011-02-26 0:28 ` Colin McCabe 2011-02-26 0:28 ` Gregory Farnum 0 siblings, 2 replies; 18+ messages in thread From: Upendra Moturi @ 2011-02-18 14:11 UTC (permalink / raw) To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava Hi Colin I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get i am using the default init script.(Found at /etc/init.d/ceph) Regarding issue 1) On the osd node I tried to start that osd but it did not work Steps followed 1) Started 3 nodes(ceph.conf is same as i sent earlier) 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin 3)/etc/init.d/ceph start osd0 (Did this on first osd) 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process 5)/etc/init.d/ceph -a start ---- shows all process on all nodes Regading Issue 2 It says -x option is not available Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even tried /etc/init.d/ceph -x -a stop but nothing worked. Then tried with /etc/init.d/ceph -a -v stop this also did not stop ceph on all nodes but got the output as /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" === mon.0 === /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" "/var/run/ceph/mon.0.pid" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" "/var/lock/subsys/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; while [ 1 ]; do [ -e /var/run/ceph/mon.0.pid ] || break pid=`cat /var/run/ceph/mon.0.pid` while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do cmd="kill $pid" echo -n $cmd... $cmd sleep 1 continue done break done" kill 1668...done /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" === mon.1 === /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" "/var/run/ceph/mon.1.pid" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" "/var/lock/subsys/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; while [ 1 ]; do [ -e /var/run/ceph/mon.1.pid ] || break pid=`cat /var/run/ceph/mon.1.pid` while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do cmd="kill $pid" echo -n $cmd... $cmd sleep 1 continue done break done" done /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" === mon.2 === /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" "/var/run/ceph/mon.2.pid" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" "/var/lock/subsys/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; while [ 1 ]; do [ -e /var/run/ceph/mon.2.pid ] || break pid=`cat /var/run/ceph/mon.2.pid` while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do cmd="kill $pid" echo -n $cmd... $cmd sleep 1 continue done break done" done /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" === mds.0 === /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" "/var/run/ceph/mds.0.pid" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" "/var/lock/subsys/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; while [ 1 ]; do [ -e /var/run/ceph/mds.0.pid ] || break pid=`cat /var/run/ceph/mds.0.pid` while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do cmd="kill $pid" echo -n $cmd... $cmd sleep 1 continue done break done" kill 1844...done /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" === mds.1 === /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" "/var/run/ceph/mds.1.pid" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" "/var/lock/subsys/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; while [ 1 ]; do [ -e /var/run/ceph/mds.1.pid ] || break pid=`cat /var/run/ceph/mds.1.pid` while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do cmd="kill $pid" echo -n $cmd... $cmd sleep 1 continue done break done" done /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" === osd.0 === /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" "/var/run/ceph/osd.0.pid" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" "/var/lock/subsys/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; while [ 1 ]; do [ -e /var/run/ceph/osd.0.pid ] || break pid=`cat /var/run/ceph/osd.0.pid` while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do cmd="kill $pid" echo -n $cmd... $cmd sleep 1 continue done break done" kill 2033...done Unmounting Btrfs on ceph0:/data/osd0 --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount /data/osd0 || true" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" === osd.1 === /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" "/var/run/ceph/osd.1.pid" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" "/var/lock/subsys/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; while [ 1 ]; do [ -e /var/run/ceph/osd.1.pid ] || break pid=`cat /var/run/ceph/osd.1.pid` while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do cmd="kill $pid" echo -n $cmd... $cmd sleep 1 continue done break done" done Unmounting Btrfs on ceph1:/data/osd1 --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount /data/osd1 || true" umount: /data/osd1: device is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" === osd.2 === /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" "/var/run/ceph/osd.2.pid" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" "/var/lock/subsys/ceph" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; while [ 1 ]; do [ -e /var/run/ceph/osd.2.pid ] || break pid=`cat /var/run/ceph/osd.2.pid` while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do cmd="kill $pid" echo -n $cmd... $cmd sleep 1 continue done break done" done Unmounting Btrfs on ceph2:/data/osd2 --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount /data/osd2 || true" umount: /data/osd2: device is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > Hi Upendra, > > Are you running init-ceph from the source directory? If you do that, > it will use the ceph.conf in the source directory itself, which is > probably not what you want. So it might be good to double-check that. > > If all else fails, running init-ceph with -x will show you exactly > what the script is doing. If all goes well, its exit status should be > 0. Are you getting exit status 0? > > Colin > > > On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >> Hi Colin >> >> Here is my ceph.conf : >> >> [global] >> pid file = /var/run/ceph/$name.pid >> debug ms = 1 >> [mon] >> mon data = /data/mon$id >> [mon.0] >> host = ceph0 >> mon addr = 192.168.155.5:6789 >> [mon.1] >> host = ceph1 >> mon addr = 192.168.155.6:6789 >> [mon.2] >> host = ceph2 >> mon addr = 192.168.155.7:6789 >> [mds] >> >> [mds0] >> host = ceph0 >> [mds1] >> host = ceph1 >> >> [osd] >> sudo = true >> osd data = /data/osd$id >> osd journal = /data/osd$id/journal >> osd journal size = 512 >> osd use stale snap = true >> [osd0] >> host = ceph0 >> btrfs devs = /dev/sdb >> [osd1] >> host = ceph1 >> btrfs devs = /dev/sdb >> [osd2] >> host = ceph2 >> btrfs devs = /dev/sdb >> >> >> >> >> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>> I'm using head of line from the master branch. But that particular >>> code hasn't changed since January, which is 0.24.2 is from. >>> >>> In my ceph.conf, I just had an osd that was on a remote machine, and >>> everything else local. >>> >>> If you could post your ceph.conf here or in IRC, perhaps we might spot >>> an issue that's causing the problems that you see. >>> >>> Colin >>> >>> >>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>> Hi Colin >>>> I am using >>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>>> >>>> Installed ceph form apt source. >>>> >>>> With above configurations i still able yo reproduce. >>>> Can you please share me ur configurations? >>>> >>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>>> working and there is no error >>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>>> >>>>>> That is expected, unless you are running init-ceph on the same node as >>>>>> osd1 is on. >>>>>> >>>>>> It might be nice to have some kind of interface like "run command X on >>>>>> osd1", but init-ceph is not that. >>>>>> >>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>>> on current node only >>>>>>> where as >>>>>>> /etc/init.d/ceph -a killall works fine. >>>>>> >>>>>> That sounds like a bug. I'll see if I can fix it. >>>>> >>>>> I'm afraid I can't reproduce this. >>>>> >>>>> I ran /etc/init.d/ceph -a stop >>>>> >>>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>>> the code, it looks correct. >>>>> >>>>> Colin >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks and Regards, >>>> Upendra.M >>>> >>> >> >> >> >> -- >> Thanks and Regards, >> Upendra.M >> > -- Thanks and Regards, Upendra.M -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-18 14:11 ` Upendra Moturi @ 2011-02-26 0:28 ` Colin McCabe 2011-02-28 12:00 ` Upendra Moturi 2011-02-26 0:28 ` Gregory Farnum 1 sibling, 1 reply; 18+ messages in thread From: Colin McCabe @ 2011-02-26 0:28 UTC (permalink / raw) To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava Hi Upendra, Based on the output you posted, init-ceph is doing something on every node. However, I only see a kill for certain nodes. Does /var/run/ceph/ exist on all nodes, or just some of them? Does the appropriate pid file exist on all nodes? What happens when you ssh in to those nodes manually and run init-ceph stop? Colin On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > Hi Colin > > I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get > i am using the default init script.(Found at /etc/init.d/ceph) > > Regarding issue 1) > > On the osd node I tried to start that osd but it did not work > Steps followed > > 1) Started 3 nodes(ceph.conf is same as i sent earlier) > > 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin > > 3)/etc/init.d/ceph start osd0 (Did this on first osd) > > 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process > > 5)/etc/init.d/ceph -a start ---- shows all process on all nodes > > Regading Issue 2 > It says -x option is not available > > Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even > tried /etc/init.d/ceph -x -a stop > but nothing worked. > > Then tried with /etc/init.d/ceph -a -v stop this also did not stop > ceph on all nodes but got the output as > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" > === mon.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" > "/var/run/ceph/mon.0.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" > Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit > -c unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mon.0.pid ] || break > pid=`cat /var/run/ceph/mon.0.pid` > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > kill 1668...done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" > === mon.1 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" > "/var/run/ceph/mon.1.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" > Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit > -c unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mon.1.pid ] || break > pid=`cat /var/run/ceph/mon.1.pid` > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" > === mon.2 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" > "/var/run/ceph/mon.2.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" > Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit > -c unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mon.2.pid ] || break > pid=`cat /var/run/ceph/mon.2.pid` > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" > === mds.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" > "/var/run/ceph/mds.0.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" > Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mds.0.pid ] || break > pid=`cat /var/run/ceph/mds.0.pid` > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > kill 1844...done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" > === mds.1 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" > "/var/run/ceph/mds.1.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" > Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mds.1.pid ] || break > pid=`cat /var/run/ceph/mds.1.pid` > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" > === osd.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" > "/var/run/ceph/osd.0.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" > Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/osd.0.pid ] || break > pid=`cat /var/run/ceph/osd.0.pid` > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > kill 2033...done > Unmounting Btrfs on ceph0:/data/osd0 > --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount > /data/osd0 || true" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" > === osd.1 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" > "/var/run/ceph/osd.1.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" > Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/osd.1.pid ] || break > pid=`cat /var/run/ceph/osd.1.pid` > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > Unmounting Btrfs on ceph1:/data/osd1 > --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount > /data/osd1 || true" > umount: /data/osd1: device is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" > === osd.2 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" > "/var/run/ceph/osd.2.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" > Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/osd.2.pid ] || break > pid=`cat /var/run/ceph/osd.2.pid` > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > Unmounting Btrfs on ceph2:/data/osd2 > --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount > /data/osd2 || true" > umount: /data/osd2: device is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) > > > > > On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >> Hi Upendra, >> >> Are you running init-ceph from the source directory? If you do that, >> it will use the ceph.conf in the source directory itself, which is >> probably not what you want. So it might be good to double-check that. >> >> If all else fails, running init-ceph with -x will show you exactly >> what the script is doing. If all goes well, its exit status should be >> 0. Are you getting exit status 0? >> >> Colin >> >> >> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>> Hi Colin >>> >>> Here is my ceph.conf : >>> >>> [global] >>> pid file = /var/run/ceph/$name.pid >>> debug ms = 1 >>> [mon] >>> mon data = /data/mon$id >>> [mon.0] >>> host = ceph0 >>> mon addr = 192.168.155.5:6789 >>> [mon.1] >>> host = ceph1 >>> mon addr = 192.168.155.6:6789 >>> [mon.2] >>> host = ceph2 >>> mon addr = 192.168.155.7:6789 >>> [mds] >>> >>> [mds0] >>> host = ceph0 >>> [mds1] >>> host = ceph1 >>> >>> [osd] >>> sudo = true >>> osd data = /data/osd$id >>> osd journal = /data/osd$id/journal >>> osd journal size = 512 >>> osd use stale snap = true >>> [osd0] >>> host = ceph0 >>> btrfs devs = /dev/sdb >>> [osd1] >>> host = ceph1 >>> btrfs devs = /dev/sdb >>> [osd2] >>> host = ceph2 >>> btrfs devs = /dev/sdb >>> >>> >>> >>> >>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>> I'm using head of line from the master branch. But that particular >>>> code hasn't changed since January, which is 0.24.2 is from. >>>> >>>> In my ceph.conf, I just had an osd that was on a remote machine, and >>>> everything else local. >>>> >>>> If you could post your ceph.conf here or in IRC, perhaps we might spot >>>> an issue that's causing the problems that you see. >>>> >>>> Colin >>>> >>>> >>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>> Hi Colin >>>>> I am using >>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>>>> >>>>> Installed ceph form apt source. >>>>> >>>>> With above configurations i still able yo reproduce. >>>>> Can you please share me ur configurations? >>>>> >>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>>>> working and there is no error >>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>>>> >>>>>>> That is expected, unless you are running init-ceph on the same node as >>>>>>> osd1 is on. >>>>>>> >>>>>>> It might be nice to have some kind of interface like "run command X on >>>>>>> osd1", but init-ceph is not that. >>>>>>> >>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>>>> on current node only >>>>>>>> where as >>>>>>>> /etc/init.d/ceph -a killall works fine. >>>>>>> >>>>>>> That sounds like a bug. I'll see if I can fix it. >>>>>> >>>>>> I'm afraid I can't reproduce this. >>>>>> >>>>>> I ran /etc/init.d/ceph -a stop >>>>>> >>>>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>>>> the code, it looks correct. >>>>>> >>>>>> Colin >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> Upendra.M >>>>> >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> Upendra.M >>> >> > > > > -- > Thanks and Regards, > Upendra.M > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-26 0:28 ` Colin McCabe @ 2011-02-28 12:00 ` Upendra Moturi 2011-02-28 16:57 ` Colin McCabe 0 siblings, 1 reply; 18+ messages in thread From: Upendra Moturi @ 2011-02-28 12:00 UTC (permalink / raw) To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava Hi Colin /var/run/ceph exist on only one node(the node on which i start the cluster with -a) But ceph is running on all nodes(checked with ps -ef) When i ssh into nodes and try to stop osd (/etc/init.d/ceph stop osd) it does not work(this was my second issue) But when i stop /etc/init.d/ceph -a stop ,it stops ceph of that node. On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > Hi Upendra, > > Based on the output you posted, init-ceph is doing something on every > node. However, I only see a kill for certain nodes. > > Does /var/run/ceph/ exist on all nodes, or just some of them? Does the > appropriate pid file exist on all nodes? What happens when you ssh in > to those nodes manually and run init-ceph stop? > > Colin > > > On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >> Hi Colin >> >> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get >> i am using the default init script.(Found at /etc/init.d/ceph) >> >> Regarding issue 1) >> >> On the osd node I tried to start that osd but it did not work >> Steps followed >> >> 1) Started 3 nodes(ceph.conf is same as i sent earlier) >> >> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin >> >> 3)/etc/init.d/ceph start osd0 (Did this on first osd) >> >> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process >> >> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes >> >> Regading Issue 2 >> It says -x option is not available >> >> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even >> tried /etc/init.d/ceph -x -a stop >> but nothing worked. >> >> Then tried with /etc/init.d/ceph -a -v stop this also did not stop >> ceph on all nodes but got the output as >> >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" >> === mon.0 === >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" >> "/var/run/ceph/mon.0.pid" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" >> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" >> "/var/lock/subsys/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" >> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit >> -c unlimited ; while [ 1 ]; do >> [ -e /var/run/ceph/mon.0.pid ] || break >> pid=`cat /var/run/ceph/mon.0.pid` >> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >> cmd="kill $pid" >> echo -n $cmd... >> $cmd >> sleep 1 >> continue >> done >> break >> done" >> kill 1668...done >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" >> === mon.1 === >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" >> "/var/run/ceph/mon.1.pid" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" >> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" >> "/var/lock/subsys/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" >> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit >> -c unlimited ; while [ 1 ]; do >> [ -e /var/run/ceph/mon.1.pid ] || break >> pid=`cat /var/run/ceph/mon.1.pid` >> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >> cmd="kill $pid" >> echo -n $cmd... >> $cmd >> sleep 1 >> continue >> done >> break >> done" >> done >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" >> === mon.2 === >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" >> "/var/run/ceph/mon.2.pid" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" >> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" >> "/var/lock/subsys/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" >> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit >> -c unlimited ; while [ 1 ]; do >> [ -e /var/run/ceph/mon.2.pid ] || break >> pid=`cat /var/run/ceph/mon.2.pid` >> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >> cmd="kill $pid" >> echo -n $cmd... >> $cmd >> sleep 1 >> continue >> done >> break >> done" >> done >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" >> === mds.0 === >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" >> "/var/run/ceph/mds.0.pid" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" >> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" >> "/var/lock/subsys/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" >> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >> unlimited ; while [ 1 ]; do >> [ -e /var/run/ceph/mds.0.pid ] || break >> pid=`cat /var/run/ceph/mds.0.pid` >> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >> cmd="kill $pid" >> echo -n $cmd... >> $cmd >> sleep 1 >> continue >> done >> break >> done" >> kill 1844...done >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" >> === mds.1 === >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" >> "/var/run/ceph/mds.1.pid" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" >> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" >> "/var/lock/subsys/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" >> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >> unlimited ; while [ 1 ]; do >> [ -e /var/run/ceph/mds.1.pid ] || break >> pid=`cat /var/run/ceph/mds.1.pid` >> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >> cmd="kill $pid" >> echo -n $cmd... >> $cmd >> sleep 1 >> continue >> done >> break >> done" >> done >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" >> === osd.0 === >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" >> "/var/run/ceph/osd.0.pid" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" >> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" >> "/var/lock/subsys/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" >> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >> unlimited ; while [ 1 ]; do >> [ -e /var/run/ceph/osd.0.pid ] || break >> pid=`cat /var/run/ceph/osd.0.pid` >> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >> cmd="kill $pid" >> echo -n $cmd... >> $cmd >> sleep 1 >> continue >> done >> break >> done" >> kill 2033...done >> Unmounting Btrfs on ceph0:/data/osd0 >> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount >> /data/osd0 || true" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" >> === osd.1 === >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" >> "/var/run/ceph/osd.1.pid" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" >> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" >> "/var/lock/subsys/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" >> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >> unlimited ; while [ 1 ]; do >> [ -e /var/run/ceph/osd.1.pid ] || break >> pid=`cat /var/run/ceph/osd.1.pid` >> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >> cmd="kill $pid" >> echo -n $cmd... >> $cmd >> sleep 1 >> continue >> done >> break >> done" >> done >> Unmounting Btrfs on ceph1:/data/osd1 >> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount >> /data/osd1 || true" >> umount: /data/osd1: device is busy. >> (In some cases useful info about processes that use >> the device is found by lsof(8) or fuser(1)) >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" >> === osd.2 === >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" >> "/var/run/ceph/osd.2.pid" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" >> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" >> "/var/lock/subsys/ceph" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" >> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c >> unlimited ; while [ 1 ]; do >> [ -e /var/run/ceph/osd.2.pid ] || break >> pid=`cat /var/run/ceph/osd.2.pid` >> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >> cmd="kill $pid" >> echo -n $cmd... >> $cmd >> sleep 1 >> continue >> done >> break >> done" >> done >> Unmounting Btrfs on ceph2:/data/osd2 >> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount >> /data/osd2 || true" >> umount: /data/osd2: device is busy. >> (In some cases useful info about processes that use >> the device is found by lsof(8) or fuser(1)) >> >> >> >> >> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>> Hi Upendra, >>> >>> Are you running init-ceph from the source directory? If you do that, >>> it will use the ceph.conf in the source directory itself, which is >>> probably not what you want. So it might be good to double-check that. >>> >>> If all else fails, running init-ceph with -x will show you exactly >>> what the script is doing. If all goes well, its exit status should be >>> 0. Are you getting exit status 0? >>> >>> Colin >>> >>> >>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>> Hi Colin >>>> >>>> Here is my ceph.conf : >>>> >>>> [global] >>>> pid file = /var/run/ceph/$name.pid >>>> debug ms = 1 >>>> [mon] >>>> mon data = /data/mon$id >>>> [mon.0] >>>> host = ceph0 >>>> mon addr = 192.168.155.5:6789 >>>> [mon.1] >>>> host = ceph1 >>>> mon addr = 192.168.155.6:6789 >>>> [mon.2] >>>> host = ceph2 >>>> mon addr = 192.168.155.7:6789 >>>> [mds] >>>> >>>> [mds0] >>>> host = ceph0 >>>> [mds1] >>>> host = ceph1 >>>> >>>> [osd] >>>> sudo = true >>>> osd data = /data/osd$id >>>> osd journal = /data/osd$id/journal >>>> osd journal size = 512 >>>> osd use stale snap = true >>>> [osd0] >>>> host = ceph0 >>>> btrfs devs = /dev/sdb >>>> [osd1] >>>> host = ceph1 >>>> btrfs devs = /dev/sdb >>>> [osd2] >>>> host = ceph2 >>>> btrfs devs = /dev/sdb >>>> >>>> >>>> >>>> >>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>> I'm using head of line from the master branch. But that particular >>>>> code hasn't changed since January, which is 0.24.2 is from. >>>>> >>>>> In my ceph.conf, I just had an osd that was on a remote machine, and >>>>> everything else local. >>>>> >>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot >>>>> an issue that's causing the problems that you see. >>>>> >>>>> Colin >>>>> >>>>> >>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>> Hi Colin >>>>>> I am using >>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>>>>> >>>>>> Installed ceph form apt source. >>>>>> >>>>>> With above configurations i still able yo reproduce. >>>>>> Can you please share me ur configurations? >>>>>> >>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>>>>> working and there is no error >>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>>>>> >>>>>>>> That is expected, unless you are running init-ceph on the same node as >>>>>>>> osd1 is on. >>>>>>>> >>>>>>>> It might be nice to have some kind of interface like "run command X on >>>>>>>> osd1", but init-ceph is not that. >>>>>>>> >>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>>>>> on current node only >>>>>>>>> where as >>>>>>>>> /etc/init.d/ceph -a killall works fine. >>>>>>>> >>>>>>>> That sounds like a bug. I'll see if I can fix it. >>>>>>> >>>>>>> I'm afraid I can't reproduce this. >>>>>>> >>>>>>> I ran /etc/init.d/ceph -a stop >>>>>>> >>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>>>>> the code, it looks correct. >>>>>>> >>>>>>> Colin >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks and Regards, >>>>>> Upendra.M >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks and Regards, >>>> Upendra.M >>>> >>> >> >> >> >> -- >> Thanks and Regards, >> Upendra.M >> > -- Thanks and Regards, Upendra.M -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-28 12:00 ` Upendra Moturi @ 2011-02-28 16:57 ` Colin McCabe 2011-03-01 11:55 ` Upendra Moturi 0 siblings, 1 reply; 18+ messages in thread From: Colin McCabe @ 2011-02-28 16:57 UTC (permalink / raw) To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > Hi Colin > > /var/run/ceph exist on only one node(the node on which i start the > cluster with -a) Without /var/run/ceph/pid, the init script will not know which pid to kill. So it will do nothing. Try creating /var/run/ceph and other appropriate directories on all nodes before starting the cluster. Colin > > On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >> Hi Upendra, >> >> Based on the output you posted, init-ceph is doing something on every >> node. However, I only see a kill for certain nodes. >> >> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the >> appropriate pid file exist on all nodes? What happens when you ssh in >> to those nodes manually and run init-ceph stop? >> >> Colin >> >> >> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>> Hi Colin >>> >>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get >>> i am using the default init script.(Found at /etc/init.d/ceph) >>> >>> Regarding issue 1) >>> >>> On the osd node I tried to start that osd but it did not work >>> Steps followed >>> >>> 1) Started 3 nodes(ceph.conf is same as i sent earlier) >>> >>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin >>> >>> 3)/etc/init.d/ceph start osd0 (Did this on first osd) >>> >>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process >>> >>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes >>> >>> Regading Issue 2 >>> It says -x option is not available >>> >>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even >>> tried /etc/init.d/ceph -x -a stop >>> but nothing worked. >>> >>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop >>> ceph on all nodes but got the output as >>> >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" >>> === mon.0 === >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" >>> "/var/run/ceph/mon.0.pid" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" >>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" >>> "/var/lock/subsys/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" >>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit >>> -c unlimited ; while [ 1 ]; do >>> [ -e /var/run/ceph/mon.0.pid ] || break >>> pid=`cat /var/run/ceph/mon.0.pid` >>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>> cmd="kill $pid" >>> echo -n $cmd... >>> $cmd >>> sleep 1 >>> continue >>> done >>> break >>> done" >>> kill 1668...done >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" >>> === mon.1 === >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" >>> "/var/run/ceph/mon.1.pid" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" >>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" >>> "/var/lock/subsys/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" >>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit >>> -c unlimited ; while [ 1 ]; do >>> [ -e /var/run/ceph/mon.1.pid ] || break >>> pid=`cat /var/run/ceph/mon.1.pid` >>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>> cmd="kill $pid" >>> echo -n $cmd... >>> $cmd >>> sleep 1 >>> continue >>> done >>> break >>> done" >>> done >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" >>> === mon.2 === >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" >>> "/var/run/ceph/mon.2.pid" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" >>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" >>> "/var/lock/subsys/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" >>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit >>> -c unlimited ; while [ 1 ]; do >>> [ -e /var/run/ceph/mon.2.pid ] || break >>> pid=`cat /var/run/ceph/mon.2.pid` >>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>> cmd="kill $pid" >>> echo -n $cmd... >>> $cmd >>> sleep 1 >>> continue >>> done >>> break >>> done" >>> done >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" >>> === mds.0 === >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" >>> "/var/run/ceph/mds.0.pid" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" >>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" >>> "/var/lock/subsys/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" >>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>> unlimited ; while [ 1 ]; do >>> [ -e /var/run/ceph/mds.0.pid ] || break >>> pid=`cat /var/run/ceph/mds.0.pid` >>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>> cmd="kill $pid" >>> echo -n $cmd... >>> $cmd >>> sleep 1 >>> continue >>> done >>> break >>> done" >>> kill 1844...done >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" >>> === mds.1 === >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" >>> "/var/run/ceph/mds.1.pid" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" >>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" >>> "/var/lock/subsys/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" >>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>> unlimited ; while [ 1 ]; do >>> [ -e /var/run/ceph/mds.1.pid ] || break >>> pid=`cat /var/run/ceph/mds.1.pid` >>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>> cmd="kill $pid" >>> echo -n $cmd... >>> $cmd >>> sleep 1 >>> continue >>> done >>> break >>> done" >>> done >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" >>> === osd.0 === >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" >>> "/var/run/ceph/osd.0.pid" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" >>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" >>> "/var/lock/subsys/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" >>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>> unlimited ; while [ 1 ]; do >>> [ -e /var/run/ceph/osd.0.pid ] || break >>> pid=`cat /var/run/ceph/osd.0.pid` >>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>> cmd="kill $pid" >>> echo -n $cmd... >>> $cmd >>> sleep 1 >>> continue >>> done >>> break >>> done" >>> kill 2033...done >>> Unmounting Btrfs on ceph0:/data/osd0 >>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount >>> /data/osd0 || true" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" >>> === osd.1 === >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" >>> "/var/run/ceph/osd.1.pid" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" >>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" >>> "/var/lock/subsys/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" >>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>> unlimited ; while [ 1 ]; do >>> [ -e /var/run/ceph/osd.1.pid ] || break >>> pid=`cat /var/run/ceph/osd.1.pid` >>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>> cmd="kill $pid" >>> echo -n $cmd... >>> $cmd >>> sleep 1 >>> continue >>> done >>> break >>> done" >>> done >>> Unmounting Btrfs on ceph1:/data/osd1 >>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount >>> /data/osd1 || true" >>> umount: /data/osd1: device is busy. >>> (In some cases useful info about processes that use >>> the device is found by lsof(8) or fuser(1)) >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" >>> === osd.2 === >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" >>> "/var/run/ceph/osd.2.pid" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" >>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" >>> "/var/lock/subsys/ceph" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" >>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c >>> unlimited ; while [ 1 ]; do >>> [ -e /var/run/ceph/osd.2.pid ] || break >>> pid=`cat /var/run/ceph/osd.2.pid` >>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>> cmd="kill $pid" >>> echo -n $cmd... >>> $cmd >>> sleep 1 >>> continue >>> done >>> break >>> done" >>> done >>> Unmounting Btrfs on ceph2:/data/osd2 >>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount >>> /data/osd2 || true" >>> umount: /data/osd2: device is busy. >>> (In some cases useful info about processes that use >>> the device is found by lsof(8) or fuser(1)) >>> >>> >>> >>> >>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>> Hi Upendra, >>>> >>>> Are you running init-ceph from the source directory? If you do that, >>>> it will use the ceph.conf in the source directory itself, which is >>>> probably not what you want. So it might be good to double-check that. >>>> >>>> If all else fails, running init-ceph with -x will show you exactly >>>> what the script is doing. If all goes well, its exit status should be >>>> 0. Are you getting exit status 0? >>>> >>>> Colin >>>> >>>> >>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>> Hi Colin >>>>> >>>>> Here is my ceph.conf : >>>>> >>>>> [global] >>>>> pid file = /var/run/ceph/$name.pid >>>>> debug ms = 1 >>>>> [mon] >>>>> mon data = /data/mon$id >>>>> [mon.0] >>>>> host = ceph0 >>>>> mon addr = 192.168.155.5:6789 >>>>> [mon.1] >>>>> host = ceph1 >>>>> mon addr = 192.168.155.6:6789 >>>>> [mon.2] >>>>> host = ceph2 >>>>> mon addr = 192.168.155.7:6789 >>>>> [mds] >>>>> >>>>> [mds0] >>>>> host = ceph0 >>>>> [mds1] >>>>> host = ceph1 >>>>> >>>>> [osd] >>>>> sudo = true >>>>> osd data = /data/osd$id >>>>> osd journal = /data/osd$id/journal >>>>> osd journal size = 512 >>>>> osd use stale snap = true >>>>> [osd0] >>>>> host = ceph0 >>>>> btrfs devs = /dev/sdb >>>>> [osd1] >>>>> host = ceph1 >>>>> btrfs devs = /dev/sdb >>>>> [osd2] >>>>> host = ceph2 >>>>> btrfs devs = /dev/sdb >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>> I'm using head of line from the master branch. But that particular >>>>>> code hasn't changed since January, which is 0.24.2 is from. >>>>>> >>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and >>>>>> everything else local. >>>>>> >>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot >>>>>> an issue that's causing the problems that you see. >>>>>> >>>>>> Colin >>>>>> >>>>>> >>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>> Hi Colin >>>>>>> I am using >>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>>>>>> >>>>>>> Installed ceph form apt source. >>>>>>> >>>>>>> With above configurations i still able yo reproduce. >>>>>>> Can you please share me ur configurations? >>>>>>> >>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>>>>>> working and there is no error >>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>>>>>> >>>>>>>>> That is expected, unless you are running init-ceph on the same node as >>>>>>>>> osd1 is on. >>>>>>>>> >>>>>>>>> It might be nice to have some kind of interface like "run command X on >>>>>>>>> osd1", but init-ceph is not that. >>>>>>>>> >>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>>>>>> on current node only >>>>>>>>>> where as >>>>>>>>>> /etc/init.d/ceph -a killall works fine. >>>>>>>>> >>>>>>>>> That sounds like a bug. I'll see if I can fix it. >>>>>>>> >>>>>>>> I'm afraid I can't reproduce this. >>>>>>>> >>>>>>>> I ran /etc/init.d/ceph -a stop >>>>>>>> >>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>>>>>> the code, it looks correct. >>>>>>>> >>>>>>>> Colin >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks and Regards, >>>>>>> Upendra.M >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> Upendra.M >>>>> >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> Upendra.M >>> >> > > > > -- > Thanks and Regards, > Upendra.M > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-28 16:57 ` Colin McCabe @ 2011-03-01 11:55 ` Upendra Moturi 2011-03-01 17:34 ` Colin McCabe 0 siblings, 1 reply; 18+ messages in thread From: Upendra Moturi @ 2011-03-01 11:55 UTC (permalink / raw) To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava Hi Colin Creating /var/run/ceph directory on every node has made it working.(/etc/init.d/ceph -a stop) But /etc/init.d/ceph start osd0 still does not work /etc/init.d/ceph -v start osd0 Gives output as /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" I have tried this on same node where osd0 is present Had a look at /etc/init.d/ceph and it does not enter the case start. On Mon, Feb 28, 2011 at 10:27 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >> Hi Colin >> >> /var/run/ceph exist on only one node(the node on which i start the >> cluster with -a) > > Without /var/run/ceph/pid, the init script will not know which pid to > kill. So it will do nothing. > > Try creating /var/run/ceph and other appropriate directories on all > nodes before starting the cluster. > > Colin > > >> >> On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>> Hi Upendra, >>> >>> Based on the output you posted, init-ceph is doing something on every >>> node. However, I only see a kill for certain nodes. >>> >>> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the >>> appropriate pid file exist on all nodes? What happens when you ssh in >>> to those nodes manually and run init-ceph stop? >>> >>> Colin >>> >>> >>> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>> Hi Colin >>>> >>>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get >>>> i am using the default init script.(Found at /etc/init.d/ceph) >>>> >>>> Regarding issue 1) >>>> >>>> On the osd node I tried to start that osd but it did not work >>>> Steps followed >>>> >>>> 1) Started 3 nodes(ceph.conf is same as i sent earlier) >>>> >>>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin >>>> >>>> 3)/etc/init.d/ceph start osd0 (Did this on first osd) >>>> >>>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process >>>> >>>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes >>>> >>>> Regading Issue 2 >>>> It says -x option is not available >>>> >>>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even >>>> tried /etc/init.d/ceph -x -a stop >>>> but nothing worked. >>>> >>>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop >>>> ceph on all nodes but got the output as >>>> >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" >>>> === mon.0 === >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" >>>> "/var/run/ceph/mon.0.pid" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" >>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" >>>> "/var/lock/subsys/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" >>>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit >>>> -c unlimited ; while [ 1 ]; do >>>> [ -e /var/run/ceph/mon.0.pid ] || break >>>> pid=`cat /var/run/ceph/mon.0.pid` >>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>> cmd="kill $pid" >>>> echo -n $cmd... >>>> $cmd >>>> sleep 1 >>>> continue >>>> done >>>> break >>>> done" >>>> kill 1668...done >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" >>>> === mon.1 === >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" >>>> "/var/run/ceph/mon.1.pid" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" >>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" >>>> "/var/lock/subsys/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" >>>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit >>>> -c unlimited ; while [ 1 ]; do >>>> [ -e /var/run/ceph/mon.1.pid ] || break >>>> pid=`cat /var/run/ceph/mon.1.pid` >>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>> cmd="kill $pid" >>>> echo -n $cmd... >>>> $cmd >>>> sleep 1 >>>> continue >>>> done >>>> break >>>> done" >>>> done >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" >>>> === mon.2 === >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" >>>> "/var/run/ceph/mon.2.pid" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" >>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" >>>> "/var/lock/subsys/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" >>>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit >>>> -c unlimited ; while [ 1 ]; do >>>> [ -e /var/run/ceph/mon.2.pid ] || break >>>> pid=`cat /var/run/ceph/mon.2.pid` >>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>> cmd="kill $pid" >>>> echo -n $cmd... >>>> $cmd >>>> sleep 1 >>>> continue >>>> done >>>> break >>>> done" >>>> done >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" >>>> === mds.0 === >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" >>>> "/var/run/ceph/mds.0.pid" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" >>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" >>>> "/var/lock/subsys/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" >>>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>>> unlimited ; while [ 1 ]; do >>>> [ -e /var/run/ceph/mds.0.pid ] || break >>>> pid=`cat /var/run/ceph/mds.0.pid` >>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>>> cmd="kill $pid" >>>> echo -n $cmd... >>>> $cmd >>>> sleep 1 >>>> continue >>>> done >>>> break >>>> done" >>>> kill 1844...done >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" >>>> === mds.1 === >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" >>>> "/var/run/ceph/mds.1.pid" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" >>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" >>>> "/var/lock/subsys/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" >>>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>>> unlimited ; while [ 1 ]; do >>>> [ -e /var/run/ceph/mds.1.pid ] || break >>>> pid=`cat /var/run/ceph/mds.1.pid` >>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>>> cmd="kill $pid" >>>> echo -n $cmd... >>>> $cmd >>>> sleep 1 >>>> continue >>>> done >>>> break >>>> done" >>>> done >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" >>>> === osd.0 === >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" >>>> "/var/run/ceph/osd.0.pid" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" >>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" >>>> "/var/lock/subsys/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" >>>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>>> unlimited ; while [ 1 ]; do >>>> [ -e /var/run/ceph/osd.0.pid ] || break >>>> pid=`cat /var/run/ceph/osd.0.pid` >>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>> cmd="kill $pid" >>>> echo -n $cmd... >>>> $cmd >>>> sleep 1 >>>> continue >>>> done >>>> break >>>> done" >>>> kill 2033...done >>>> Unmounting Btrfs on ceph0:/data/osd0 >>>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>> /data/osd0 || true" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" >>>> === osd.1 === >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" >>>> "/var/run/ceph/osd.1.pid" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" >>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" >>>> "/var/lock/subsys/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" >>>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>>> unlimited ; while [ 1 ]; do >>>> [ -e /var/run/ceph/osd.1.pid ] || break >>>> pid=`cat /var/run/ceph/osd.1.pid` >>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>> cmd="kill $pid" >>>> echo -n $cmd... >>>> $cmd >>>> sleep 1 >>>> continue >>>> done >>>> break >>>> done" >>>> done >>>> Unmounting Btrfs on ceph1:/data/osd1 >>>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>> /data/osd1 || true" >>>> umount: /data/osd1: device is busy. >>>> (In some cases useful info about processes that use >>>> the device is found by lsof(8) or fuser(1)) >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" >>>> === osd.2 === >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" >>>> "/var/run/ceph/osd.2.pid" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" >>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" >>>> "/var/lock/subsys/ceph" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" >>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" >>>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c >>>> unlimited ; while [ 1 ]; do >>>> [ -e /var/run/ceph/osd.2.pid ] || break >>>> pid=`cat /var/run/ceph/osd.2.pid` >>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>> cmd="kill $pid" >>>> echo -n $cmd... >>>> $cmd >>>> sleep 1 >>>> continue >>>> done >>>> break >>>> done" >>>> done >>>> Unmounting Btrfs on ceph2:/data/osd2 >>>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>> /data/osd2 || true" >>>> umount: /data/osd2: device is busy. >>>> (In some cases useful info about processes that use >>>> the device is found by lsof(8) or fuser(1)) >>>> >>>> >>>> >>>> >>>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>> Hi Upendra, >>>>> >>>>> Are you running init-ceph from the source directory? If you do that, >>>>> it will use the ceph.conf in the source directory itself, which is >>>>> probably not what you want. So it might be good to double-check that. >>>>> >>>>> If all else fails, running init-ceph with -x will show you exactly >>>>> what the script is doing. If all goes well, its exit status should be >>>>> 0. Are you getting exit status 0? >>>>> >>>>> Colin >>>>> >>>>> >>>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>> Hi Colin >>>>>> >>>>>> Here is my ceph.conf : >>>>>> >>>>>> [global] >>>>>> pid file = /var/run/ceph/$name.pid >>>>>> debug ms = 1 >>>>>> [mon] >>>>>> mon data = /data/mon$id >>>>>> [mon.0] >>>>>> host = ceph0 >>>>>> mon addr = 192.168.155.5:6789 >>>>>> [mon.1] >>>>>> host = ceph1 >>>>>> mon addr = 192.168.155.6:6789 >>>>>> [mon.2] >>>>>> host = ceph2 >>>>>> mon addr = 192.168.155.7:6789 >>>>>> [mds] >>>>>> >>>>>> [mds0] >>>>>> host = ceph0 >>>>>> [mds1] >>>>>> host = ceph1 >>>>>> >>>>>> [osd] >>>>>> sudo = true >>>>>> osd data = /data/osd$id >>>>>> osd journal = /data/osd$id/journal >>>>>> osd journal size = 512 >>>>>> osd use stale snap = true >>>>>> [osd0] >>>>>> host = ceph0 >>>>>> btrfs devs = /dev/sdb >>>>>> [osd1] >>>>>> host = ceph1 >>>>>> btrfs devs = /dev/sdb >>>>>> [osd2] >>>>>> host = ceph2 >>>>>> btrfs devs = /dev/sdb >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>> I'm using head of line from the master branch. But that particular >>>>>>> code hasn't changed since January, which is 0.24.2 is from. >>>>>>> >>>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and >>>>>>> everything else local. >>>>>>> >>>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot >>>>>>> an issue that's causing the problems that you see. >>>>>>> >>>>>>> Colin >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>> Hi Colin >>>>>>>> I am using >>>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>>>>>>> >>>>>>>> Installed ceph form apt source. >>>>>>>> >>>>>>>> With above configurations i still able yo reproduce. >>>>>>>> Can you please share me ur configurations? >>>>>>>> >>>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>>>>>>> working and there is no error >>>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>>>>>>> >>>>>>>>>> That is expected, unless you are running init-ceph on the same node as >>>>>>>>>> osd1 is on. >>>>>>>>>> >>>>>>>>>> It might be nice to have some kind of interface like "run command X on >>>>>>>>>> osd1", but init-ceph is not that. >>>>>>>>>> >>>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>>>>>>> on current node only >>>>>>>>>>> where as >>>>>>>>>>> /etc/init.d/ceph -a killall works fine. >>>>>>>>>> >>>>>>>>>> That sounds like a bug. I'll see if I can fix it. >>>>>>>>> >>>>>>>>> I'm afraid I can't reproduce this. >>>>>>>>> >>>>>>>>> I ran /etc/init.d/ceph -a stop >>>>>>>>> >>>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>>>>>>> the code, it looks correct. >>>>>>>>> >>>>>>>>> Colin >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks and Regards, >>>>>>>> Upendra.M >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks and Regards, >>>>>> Upendra.M >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks and Regards, >>>> Upendra.M >>>> >>> >> >> >> >> -- >> Thanks and Regards, >> Upendra.M >> > -- Thanks and Regards, Upendra.M -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-03-01 11:55 ` Upendra Moturi @ 2011-03-01 17:34 ` Colin McCabe 2011-03-02 9:18 ` Upendra Moturi 0 siblings, 1 reply; 18+ messages in thread From: Colin McCabe @ 2011-03-01 17:34 UTC (permalink / raw) To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava Hi Upendra, You should be on the same node as osd0 in order to start osd0. There is that wacky -a option, at least until we decide to remove it. But my advice is not to use that. Set up something like dsh. cheers, Colin On Tue, Mar 1, 2011 at 3:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > Hi Colin > > Creating /var/run/ceph directory on every node has made it > working.(/etc/init.d/ceph -a stop) > But /etc/init.d/ceph start osd0 still does not work > > /etc/init.d/ceph -v start osd0 > Gives output as > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" > > I have tried this on same node where osd0 is present > > Had a look at /etc/init.d/ceph and it does not enter the case start. > > > On Mon, Feb 28, 2011 at 10:27 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >> On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>> Hi Colin >>> >>> /var/run/ceph exist on only one node(the node on which i start the >>> cluster with -a) >> >> Without /var/run/ceph/pid, the init script will not know which pid to >> kill. So it will do nothing. >> >> Try creating /var/run/ceph and other appropriate directories on all >> nodes before starting the cluster. >> >> Colin >> >> >>> >>> On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>> Hi Upendra, >>>> >>>> Based on the output you posted, init-ceph is doing something on every >>>> node. However, I only see a kill for certain nodes. >>>> >>>> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the >>>> appropriate pid file exist on all nodes? What happens when you ssh in >>>> to those nodes manually and run init-ceph stop? >>>> >>>> Colin >>>> >>>> >>>> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>> Hi Colin >>>>> >>>>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get >>>>> i am using the default init script.(Found at /etc/init.d/ceph) >>>>> >>>>> Regarding issue 1) >>>>> >>>>> On the osd node I tried to start that osd but it did not work >>>>> Steps followed >>>>> >>>>> 1) Started 3 nodes(ceph.conf is same as i sent earlier) >>>>> >>>>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin >>>>> >>>>> 3)/etc/init.d/ceph start osd0 (Did this on first osd) >>>>> >>>>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process >>>>> >>>>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes >>>>> >>>>> Regading Issue 2 >>>>> It says -x option is not available >>>>> >>>>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even >>>>> tried /etc/init.d/ceph -x -a stop >>>>> but nothing worked. >>>>> >>>>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop >>>>> ceph on all nodes but got the output as >>>>> >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" >>>>> === mon.0 === >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" >>>>> "/var/run/ceph/mon.0.pid" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" >>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" >>>>> "/var/lock/subsys/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" >>>>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit >>>>> -c unlimited ; while [ 1 ]; do >>>>> [ -e /var/run/ceph/mon.0.pid ] || break >>>>> pid=`cat /var/run/ceph/mon.0.pid` >>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>>> cmd="kill $pid" >>>>> echo -n $cmd... >>>>> $cmd >>>>> sleep 1 >>>>> continue >>>>> done >>>>> break >>>>> done" >>>>> kill 1668...done >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" >>>>> === mon.1 === >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" >>>>> "/var/run/ceph/mon.1.pid" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" >>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" >>>>> "/var/lock/subsys/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" >>>>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit >>>>> -c unlimited ; while [ 1 ]; do >>>>> [ -e /var/run/ceph/mon.1.pid ] || break >>>>> pid=`cat /var/run/ceph/mon.1.pid` >>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>>> cmd="kill $pid" >>>>> echo -n $cmd... >>>>> $cmd >>>>> sleep 1 >>>>> continue >>>>> done >>>>> break >>>>> done" >>>>> done >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" >>>>> === mon.2 === >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" >>>>> "/var/run/ceph/mon.2.pid" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" >>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" >>>>> "/var/lock/subsys/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" >>>>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit >>>>> -c unlimited ; while [ 1 ]; do >>>>> [ -e /var/run/ceph/mon.2.pid ] || break >>>>> pid=`cat /var/run/ceph/mon.2.pid` >>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>>> cmd="kill $pid" >>>>> echo -n $cmd... >>>>> $cmd >>>>> sleep 1 >>>>> continue >>>>> done >>>>> break >>>>> done" >>>>> done >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" >>>>> === mds.0 === >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" >>>>> "/var/run/ceph/mds.0.pid" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" >>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" >>>>> "/var/lock/subsys/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" >>>>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>>>> unlimited ; while [ 1 ]; do >>>>> [ -e /var/run/ceph/mds.0.pid ] || break >>>>> pid=`cat /var/run/ceph/mds.0.pid` >>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>>>> cmd="kill $pid" >>>>> echo -n $cmd... >>>>> $cmd >>>>> sleep 1 >>>>> continue >>>>> done >>>>> break >>>>> done" >>>>> kill 1844...done >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" >>>>> === mds.1 === >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" >>>>> "/var/run/ceph/mds.1.pid" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" >>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" >>>>> "/var/lock/subsys/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" >>>>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>>>> unlimited ; while [ 1 ]; do >>>>> [ -e /var/run/ceph/mds.1.pid ] || break >>>>> pid=`cat /var/run/ceph/mds.1.pid` >>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>>>> cmd="kill $pid" >>>>> echo -n $cmd... >>>>> $cmd >>>>> sleep 1 >>>>> continue >>>>> done >>>>> break >>>>> done" >>>>> done >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" >>>>> === osd.0 === >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" >>>>> "/var/run/ceph/osd.0.pid" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" >>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" >>>>> "/var/lock/subsys/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" >>>>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>>>> unlimited ; while [ 1 ]; do >>>>> [ -e /var/run/ceph/osd.0.pid ] || break >>>>> pid=`cat /var/run/ceph/osd.0.pid` >>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>>> cmd="kill $pid" >>>>> echo -n $cmd... >>>>> $cmd >>>>> sleep 1 >>>>> continue >>>>> done >>>>> break >>>>> done" >>>>> kill 2033...done >>>>> Unmounting Btrfs on ceph0:/data/osd0 >>>>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>>> /data/osd0 || true" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" >>>>> === osd.1 === >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" >>>>> "/var/run/ceph/osd.1.pid" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" >>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" >>>>> "/var/lock/subsys/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" >>>>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>>>> unlimited ; while [ 1 ]; do >>>>> [ -e /var/run/ceph/osd.1.pid ] || break >>>>> pid=`cat /var/run/ceph/osd.1.pid` >>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>>> cmd="kill $pid" >>>>> echo -n $cmd... >>>>> $cmd >>>>> sleep 1 >>>>> continue >>>>> done >>>>> break >>>>> done" >>>>> done >>>>> Unmounting Btrfs on ceph1:/data/osd1 >>>>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>>> /data/osd1 || true" >>>>> umount: /data/osd1: device is busy. >>>>> (In some cases useful info about processes that use >>>>> the device is found by lsof(8) or fuser(1)) >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" >>>>> === osd.2 === >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" >>>>> "/var/run/ceph/osd.2.pid" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" >>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" >>>>> "/var/lock/subsys/ceph" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" >>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" >>>>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c >>>>> unlimited ; while [ 1 ]; do >>>>> [ -e /var/run/ceph/osd.2.pid ] || break >>>>> pid=`cat /var/run/ceph/osd.2.pid` >>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>>> cmd="kill $pid" >>>>> echo -n $cmd... >>>>> $cmd >>>>> sleep 1 >>>>> continue >>>>> done >>>>> break >>>>> done" >>>>> done >>>>> Unmounting Btrfs on ceph2:/data/osd2 >>>>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>>> /data/osd2 || true" >>>>> umount: /data/osd2: device is busy. >>>>> (In some cases useful info about processes that use >>>>> the device is found by lsof(8) or fuser(1)) >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>> Hi Upendra, >>>>>> >>>>>> Are you running init-ceph from the source directory? If you do that, >>>>>> it will use the ceph.conf in the source directory itself, which is >>>>>> probably not what you want. So it might be good to double-check that. >>>>>> >>>>>> If all else fails, running init-ceph with -x will show you exactly >>>>>> what the script is doing. If all goes well, its exit status should be >>>>>> 0. Are you getting exit status 0? >>>>>> >>>>>> Colin >>>>>> >>>>>> >>>>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>> Hi Colin >>>>>>> >>>>>>> Here is my ceph.conf : >>>>>>> >>>>>>> [global] >>>>>>> pid file = /var/run/ceph/$name.pid >>>>>>> debug ms = 1 >>>>>>> [mon] >>>>>>> mon data = /data/mon$id >>>>>>> [mon.0] >>>>>>> host = ceph0 >>>>>>> mon addr = 192.168.155.5:6789 >>>>>>> [mon.1] >>>>>>> host = ceph1 >>>>>>> mon addr = 192.168.155.6:6789 >>>>>>> [mon.2] >>>>>>> host = ceph2 >>>>>>> mon addr = 192.168.155.7:6789 >>>>>>> [mds] >>>>>>> >>>>>>> [mds0] >>>>>>> host = ceph0 >>>>>>> [mds1] >>>>>>> host = ceph1 >>>>>>> >>>>>>> [osd] >>>>>>> sudo = true >>>>>>> osd data = /data/osd$id >>>>>>> osd journal = /data/osd$id/journal >>>>>>> osd journal size = 512 >>>>>>> osd use stale snap = true >>>>>>> [osd0] >>>>>>> host = ceph0 >>>>>>> btrfs devs = /dev/sdb >>>>>>> [osd1] >>>>>>> host = ceph1 >>>>>>> btrfs devs = /dev/sdb >>>>>>> [osd2] >>>>>>> host = ceph2 >>>>>>> btrfs devs = /dev/sdb >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>> I'm using head of line from the master branch. But that particular >>>>>>>> code hasn't changed since January, which is 0.24.2 is from. >>>>>>>> >>>>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and >>>>>>>> everything else local. >>>>>>>> >>>>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot >>>>>>>> an issue that's causing the problems that you see. >>>>>>>> >>>>>>>> Colin >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>> Hi Colin >>>>>>>>> I am using >>>>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>>>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>>>>>>>> >>>>>>>>> Installed ceph form apt source. >>>>>>>>> >>>>>>>>> With above configurations i still able yo reproduce. >>>>>>>>> Can you please share me ur configurations? >>>>>>>>> >>>>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>>>>>>>> working and there is no error >>>>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>>>>>>>> >>>>>>>>>>> That is expected, unless you are running init-ceph on the same node as >>>>>>>>>>> osd1 is on. >>>>>>>>>>> >>>>>>>>>>> It might be nice to have some kind of interface like "run command X on >>>>>>>>>>> osd1", but init-ceph is not that. >>>>>>>>>>> >>>>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>>>>>>>> on current node only >>>>>>>>>>>> where as >>>>>>>>>>>> /etc/init.d/ceph -a killall works fine. >>>>>>>>>>> >>>>>>>>>>> That sounds like a bug. I'll see if I can fix it. >>>>>>>>>> >>>>>>>>>> I'm afraid I can't reproduce this. >>>>>>>>>> >>>>>>>>>> I ran /etc/init.d/ceph -a stop >>>>>>>>>> >>>>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>>>>>>>> the code, it looks correct. >>>>>>>>>> >>>>>>>>>> Colin >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks and Regards, >>>>>>>>> Upendra.M >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks and Regards, >>>>>>> Upendra.M >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> Upendra.M >>>>> >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> Upendra.M >>> >> > > > > -- > Thanks and Regards, > Upendra.M > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-03-01 17:34 ` Colin McCabe @ 2011-03-02 9:18 ` Upendra Moturi 2011-03-02 19:26 ` Colin McCabe 0 siblings, 1 reply; 18+ messages in thread From: Upendra Moturi @ 2011-03-02 9:18 UTC (permalink / raw) To: Colin McCabe; +Cc: sage, ceph-devel, Suresh Mandava I was on the same node as osd0 ,but still it did not work. When i see /etc/init.d/ceph file,it is not even going into that start case On Tue, Mar 1, 2011 at 11:04 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > Hi Upendra, > > You should be on the same node as osd0 in order to start osd0. > > There is that wacky -a option, at least until we decide to remove it. > But my advice is not to use that. Set up something like dsh. > > cheers, > Colin > > > On Tue, Mar 1, 2011 at 3:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >> Hi Colin >> >> Creating /var/run/ceph directory on every node has made it >> working.(/etc/init.d/ceph -a stop) >> But /etc/init.d/ceph start osd0 still does not work >> >> /etc/init.d/ceph -v start osd0 >> Gives output as >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" >> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" >> >> I have tried this on same node where osd0 is present >> >> Had a look at /etc/init.d/ceph and it does not enter the case start. >> >> >> On Mon, Feb 28, 2011 at 10:27 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>> On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>> Hi Colin >>>> >>>> /var/run/ceph exist on only one node(the node on which i start the >>>> cluster with -a) >>> >>> Without /var/run/ceph/pid, the init script will not know which pid to >>> kill. So it will do nothing. >>> >>> Try creating /var/run/ceph and other appropriate directories on all >>> nodes before starting the cluster. >>> >>> Colin >>> >>> >>>> >>>> On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>> Hi Upendra, >>>>> >>>>> Based on the output you posted, init-ceph is doing something on every >>>>> node. However, I only see a kill for certain nodes. >>>>> >>>>> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the >>>>> appropriate pid file exist on all nodes? What happens when you ssh in >>>>> to those nodes manually and run init-ceph stop? >>>>> >>>>> Colin >>>>> >>>>> >>>>> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>> Hi Colin >>>>>> >>>>>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get >>>>>> i am using the default init script.(Found at /etc/init.d/ceph) >>>>>> >>>>>> Regarding issue 1) >>>>>> >>>>>> On the osd node I tried to start that osd but it did not work >>>>>> Steps followed >>>>>> >>>>>> 1) Started 3 nodes(ceph.conf is same as i sent earlier) >>>>>> >>>>>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin >>>>>> >>>>>> 3)/etc/init.d/ceph start osd0 (Did this on first osd) >>>>>> >>>>>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process >>>>>> >>>>>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes >>>>>> >>>>>> Regading Issue 2 >>>>>> It says -x option is not available >>>>>> >>>>>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even >>>>>> tried /etc/init.d/ceph -x -a stop >>>>>> but nothing worked. >>>>>> >>>>>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop >>>>>> ceph on all nodes but got the output as >>>>>> >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" >>>>>> === mon.0 === >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" >>>>>> "/var/run/ceph/mon.0.pid" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" >>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" >>>>>> "/var/lock/subsys/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" >>>>>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit >>>>>> -c unlimited ; while [ 1 ]; do >>>>>> [ -e /var/run/ceph/mon.0.pid ] || break >>>>>> pid=`cat /var/run/ceph/mon.0.pid` >>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>>>> cmd="kill $pid" >>>>>> echo -n $cmd... >>>>>> $cmd >>>>>> sleep 1 >>>>>> continue >>>>>> done >>>>>> break >>>>>> done" >>>>>> kill 1668...done >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" >>>>>> === mon.1 === >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" >>>>>> "/var/run/ceph/mon.1.pid" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" >>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" >>>>>> "/var/lock/subsys/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" >>>>>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit >>>>>> -c unlimited ; while [ 1 ]; do >>>>>> [ -e /var/run/ceph/mon.1.pid ] || break >>>>>> pid=`cat /var/run/ceph/mon.1.pid` >>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>>>> cmd="kill $pid" >>>>>> echo -n $cmd... >>>>>> $cmd >>>>>> sleep 1 >>>>>> continue >>>>>> done >>>>>> break >>>>>> done" >>>>>> done >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" >>>>>> === mon.2 === >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" >>>>>> "/var/run/ceph/mon.2.pid" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" >>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" >>>>>> "/var/lock/subsys/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" >>>>>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit >>>>>> -c unlimited ; while [ 1 ]; do >>>>>> [ -e /var/run/ceph/mon.2.pid ] || break >>>>>> pid=`cat /var/run/ceph/mon.2.pid` >>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>>>> cmd="kill $pid" >>>>>> echo -n $cmd... >>>>>> $cmd >>>>>> sleep 1 >>>>>> continue >>>>>> done >>>>>> break >>>>>> done" >>>>>> done >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" >>>>>> === mds.0 === >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" >>>>>> "/var/run/ceph/mds.0.pid" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" >>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" >>>>>> "/var/lock/subsys/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" >>>>>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>>>>> unlimited ; while [ 1 ]; do >>>>>> [ -e /var/run/ceph/mds.0.pid ] || break >>>>>> pid=`cat /var/run/ceph/mds.0.pid` >>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>>>>> cmd="kill $pid" >>>>>> echo -n $cmd... >>>>>> $cmd >>>>>> sleep 1 >>>>>> continue >>>>>> done >>>>>> break >>>>>> done" >>>>>> kill 1844...done >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" >>>>>> === mds.1 === >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" >>>>>> "/var/run/ceph/mds.1.pid" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" >>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" >>>>>> "/var/lock/subsys/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" >>>>>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>>>>> unlimited ; while [ 1 ]; do >>>>>> [ -e /var/run/ceph/mds.1.pid ] || break >>>>>> pid=`cat /var/run/ceph/mds.1.pid` >>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>>>>> cmd="kill $pid" >>>>>> echo -n $cmd... >>>>>> $cmd >>>>>> sleep 1 >>>>>> continue >>>>>> done >>>>>> break >>>>>> done" >>>>>> done >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" >>>>>> === osd.0 === >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" >>>>>> "/var/run/ceph/osd.0.pid" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" >>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" >>>>>> "/var/lock/subsys/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" >>>>>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>>>>> unlimited ; while [ 1 ]; do >>>>>> [ -e /var/run/ceph/osd.0.pid ] || break >>>>>> pid=`cat /var/run/ceph/osd.0.pid` >>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>>>> cmd="kill $pid" >>>>>> echo -n $cmd... >>>>>> $cmd >>>>>> sleep 1 >>>>>> continue >>>>>> done >>>>>> break >>>>>> done" >>>>>> kill 2033...done >>>>>> Unmounting Btrfs on ceph0:/data/osd0 >>>>>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>>>> /data/osd0 || true" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" >>>>>> === osd.1 === >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" >>>>>> "/var/run/ceph/osd.1.pid" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" >>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" >>>>>> "/var/lock/subsys/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" >>>>>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>>>>> unlimited ; while [ 1 ]; do >>>>>> [ -e /var/run/ceph/osd.1.pid ] || break >>>>>> pid=`cat /var/run/ceph/osd.1.pid` >>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>>>> cmd="kill $pid" >>>>>> echo -n $cmd... >>>>>> $cmd >>>>>> sleep 1 >>>>>> continue >>>>>> done >>>>>> break >>>>>> done" >>>>>> done >>>>>> Unmounting Btrfs on ceph1:/data/osd1 >>>>>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>>>> /data/osd1 || true" >>>>>> umount: /data/osd1: device is busy. >>>>>> (In some cases useful info about processes that use >>>>>> the device is found by lsof(8) or fuser(1)) >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" >>>>>> === osd.2 === >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" >>>>>> "/var/run/ceph/osd.2.pid" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" >>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" >>>>>> "/var/lock/subsys/ceph" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" >>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" >>>>>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c >>>>>> unlimited ; while [ 1 ]; do >>>>>> [ -e /var/run/ceph/osd.2.pid ] || break >>>>>> pid=`cat /var/run/ceph/osd.2.pid` >>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>>>> cmd="kill $pid" >>>>>> echo -n $cmd... >>>>>> $cmd >>>>>> sleep 1 >>>>>> continue >>>>>> done >>>>>> break >>>>>> done" >>>>>> done >>>>>> Unmounting Btrfs on ceph2:/data/osd2 >>>>>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>>>> /data/osd2 || true" >>>>>> umount: /data/osd2: device is busy. >>>>>> (In some cases useful info about processes that use >>>>>> the device is found by lsof(8) or fuser(1)) >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>> Hi Upendra, >>>>>>> >>>>>>> Are you running init-ceph from the source directory? If you do that, >>>>>>> it will use the ceph.conf in the source directory itself, which is >>>>>>> probably not what you want. So it might be good to double-check that. >>>>>>> >>>>>>> If all else fails, running init-ceph with -x will show you exactly >>>>>>> what the script is doing. If all goes well, its exit status should be >>>>>>> 0. Are you getting exit status 0? >>>>>>> >>>>>>> Colin >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>> Hi Colin >>>>>>>> >>>>>>>> Here is my ceph.conf : >>>>>>>> >>>>>>>> [global] >>>>>>>> pid file = /var/run/ceph/$name.pid >>>>>>>> debug ms = 1 >>>>>>>> [mon] >>>>>>>> mon data = /data/mon$id >>>>>>>> [mon.0] >>>>>>>> host = ceph0 >>>>>>>> mon addr = 192.168.155.5:6789 >>>>>>>> [mon.1] >>>>>>>> host = ceph1 >>>>>>>> mon addr = 192.168.155.6:6789 >>>>>>>> [mon.2] >>>>>>>> host = ceph2 >>>>>>>> mon addr = 192.168.155.7:6789 >>>>>>>> [mds] >>>>>>>> >>>>>>>> [mds0] >>>>>>>> host = ceph0 >>>>>>>> [mds1] >>>>>>>> host = ceph1 >>>>>>>> >>>>>>>> [osd] >>>>>>>> sudo = true >>>>>>>> osd data = /data/osd$id >>>>>>>> osd journal = /data/osd$id/journal >>>>>>>> osd journal size = 512 >>>>>>>> osd use stale snap = true >>>>>>>> [osd0] >>>>>>>> host = ceph0 >>>>>>>> btrfs devs = /dev/sdb >>>>>>>> [osd1] >>>>>>>> host = ceph1 >>>>>>>> btrfs devs = /dev/sdb >>>>>>>> [osd2] >>>>>>>> host = ceph2 >>>>>>>> btrfs devs = /dev/sdb >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>> I'm using head of line from the master branch. But that particular >>>>>>>>> code hasn't changed since January, which is 0.24.2 is from. >>>>>>>>> >>>>>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and >>>>>>>>> everything else local. >>>>>>>>> >>>>>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot >>>>>>>>> an issue that's causing the problems that you see. >>>>>>>>> >>>>>>>>> Colin >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>>> Hi Colin >>>>>>>>>> I am using >>>>>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>>>>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>>>>>>>>> >>>>>>>>>> Installed ceph form apt source. >>>>>>>>>> >>>>>>>>>> With above configurations i still able yo reproduce. >>>>>>>>>> Can you please share me ur configurations? >>>>>>>>>> >>>>>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>>>>>>>>> working and there is no error >>>>>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>>>>>>>>> >>>>>>>>>>>> That is expected, unless you are running init-ceph on the same node as >>>>>>>>>>>> osd1 is on. >>>>>>>>>>>> >>>>>>>>>>>> It might be nice to have some kind of interface like "run command X on >>>>>>>>>>>> osd1", but init-ceph is not that. >>>>>>>>>>>> >>>>>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>>>>>>>>> on current node only >>>>>>>>>>>>> where as >>>>>>>>>>>>> /etc/init.d/ceph -a killall works fine. >>>>>>>>>>>> >>>>>>>>>>>> That sounds like a bug. I'll see if I can fix it. >>>>>>>>>>> >>>>>>>>>>> I'm afraid I can't reproduce this. >>>>>>>>>>> >>>>>>>>>>> I ran /etc/init.d/ceph -a stop >>>>>>>>>>> >>>>>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>>>>>>>>> the code, it looks correct. >>>>>>>>>>> >>>>>>>>>>> Colin >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Thanks and Regards, >>>>>>>>>> Upendra.M >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Thanks and Regards, >>>>>>>> Upendra.M >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks and Regards, >>>>>> Upendra.M >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks and Regards, >>>> Upendra.M >>>> >>> >> >> >> >> -- >> Thanks and Regards, >> Upendra.M >> > -- Thanks and Regards, Upendra.M -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-03-02 9:18 ` Upendra Moturi @ 2011-03-02 19:26 ` Colin McCabe 0 siblings, 0 replies; 18+ messages in thread From: Colin McCabe @ 2011-03-02 19:26 UTC (permalink / raw) To: Upendra Moturi; +Cc: sage, ceph-devel, Suresh Mandava Hi Upendra, Can you try chatting with us on #ceph on irc.oftc.net? C. On Wed, Mar 2, 2011 at 1:18 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > I was on the same node as osd0 ,but still it did not work. > When i see /etc/init.d/ceph file,it is not even going into that start case > > > On Tue, Mar 1, 2011 at 11:04 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >> Hi Upendra, >> >> You should be on the same node as osd0 in order to start osd0. >> >> There is that wacky -a option, at least until we decide to remove it. >> But my advice is not to use that. Set up something like dsh. >> >> cheers, >> Colin >> >> >> On Tue, Mar 1, 2011 at 3:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>> Hi Colin >>> >>> Creating /var/run/ceph directory on every node has made it >>> working.(/etc/init.d/ceph -a stop) >>> But /etc/init.d/ceph start osd0 still does not work >>> >>> /etc/init.d/ceph -v start osd0 >>> Gives output as >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" >>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" >>> >>> I have tried this on same node where osd0 is present >>> >>> Had a look at /etc/init.d/ceph and it does not enter the case start. >>> >>> >>> On Mon, Feb 28, 2011 at 10:27 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>> On Mon, Feb 28, 2011 at 4:00 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>> Hi Colin >>>>> >>>>> /var/run/ceph exist on only one node(the node on which i start the >>>>> cluster with -a) >>>> >>>> Without /var/run/ceph/pid, the init script will not know which pid to >>>> kill. So it will do nothing. >>>> >>>> Try creating /var/run/ceph and other appropriate directories on all >>>> nodes before starting the cluster. >>>> >>>> Colin >>>> >>>> >>>>> >>>>> On Sat, Feb 26, 2011 at 5:58 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>> Hi Upendra, >>>>>> >>>>>> Based on the output you posted, init-ceph is doing something on every >>>>>> node. However, I only see a kill for certain nodes. >>>>>> >>>>>> Does /var/run/ceph/ exist on all nodes, or just some of them? Does the >>>>>> appropriate pid file exist on all nodes? What happens when you ssh in >>>>>> to those nodes manually and run init-ceph stop? >>>>>> >>>>>> Colin >>>>>> >>>>>> >>>>>> On Fri, Feb 18, 2011 at 6:11 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>> Hi Colin >>>>>>> >>>>>>> I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get >>>>>>> i am using the default init script.(Found at /etc/init.d/ceph) >>>>>>> >>>>>>> Regarding issue 1) >>>>>>> >>>>>>> On the osd node I tried to start that osd but it did not work >>>>>>> Steps followed >>>>>>> >>>>>>> 1) Started 3 nodes(ceph.conf is same as i sent earlier) >>>>>>> >>>>>>> 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin >>>>>>> >>>>>>> 3)/etc/init.d/ceph start osd0 (Did this on first osd) >>>>>>> >>>>>>> 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process >>>>>>> >>>>>>> 5)/etc/init.d/ceph -a start ---- shows all process on all nodes >>>>>>> >>>>>>> Regading Issue 2 >>>>>>> It says -x option is not available >>>>>>> >>>>>>> Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even >>>>>>> tried /etc/init.d/ceph -x -a stop >>>>>>> but nothing worked. >>>>>>> >>>>>>> Then tried with /etc/init.d/ceph -a -v stop this also did not stop >>>>>>> ceph on all nodes but got the output as >>>>>>> >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" >>>>>>> === mon.0 === >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" >>>>>>> "/var/run/ceph/mon.0.pid" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" >>>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" >>>>>>> "/var/lock/subsys/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" >>>>>>> Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit >>>>>>> -c unlimited ; while [ 1 ]; do >>>>>>> [ -e /var/run/ceph/mon.0.pid ] || break >>>>>>> pid=`cat /var/run/ceph/mon.0.pid` >>>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>>>>> cmd="kill $pid" >>>>>>> echo -n $cmd... >>>>>>> $cmd >>>>>>> sleep 1 >>>>>>> continue >>>>>>> done >>>>>>> break >>>>>>> done" >>>>>>> kill 1668...done >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" >>>>>>> === mon.1 === >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" >>>>>>> "/var/run/ceph/mon.1.pid" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" >>>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" >>>>>>> "/var/lock/subsys/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" >>>>>>> Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit >>>>>>> -c unlimited ; while [ 1 ]; do >>>>>>> [ -e /var/run/ceph/mon.1.pid ] || break >>>>>>> pid=`cat /var/run/ceph/mon.1.pid` >>>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>>>>> cmd="kill $pid" >>>>>>> echo -n $cmd... >>>>>>> $cmd >>>>>>> sleep 1 >>>>>>> continue >>>>>>> done >>>>>>> break >>>>>>> done" >>>>>>> done >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" >>>>>>> === mon.2 === >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" >>>>>>> "/var/run/ceph/mon.2.pid" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" >>>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" >>>>>>> "/var/lock/subsys/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" >>>>>>> Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit >>>>>>> -c unlimited ; while [ 1 ]; do >>>>>>> [ -e /var/run/ceph/mon.2.pid ] || break >>>>>>> pid=`cat /var/run/ceph/mon.2.pid` >>>>>>> while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do >>>>>>> cmd="kill $pid" >>>>>>> echo -n $cmd... >>>>>>> $cmd >>>>>>> sleep 1 >>>>>>> continue >>>>>>> done >>>>>>> break >>>>>>> done" >>>>>>> done >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" >>>>>>> === mds.0 === >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" >>>>>>> "/var/run/ceph/mds.0.pid" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" >>>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" >>>>>>> "/var/lock/subsys/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" >>>>>>> Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>>>>>> unlimited ; while [ 1 ]; do >>>>>>> [ -e /var/run/ceph/mds.0.pid ] || break >>>>>>> pid=`cat /var/run/ceph/mds.0.pid` >>>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>>>>>> cmd="kill $pid" >>>>>>> echo -n $cmd... >>>>>>> $cmd >>>>>>> sleep 1 >>>>>>> continue >>>>>>> done >>>>>>> break >>>>>>> done" >>>>>>> kill 1844...done >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" >>>>>>> === mds.1 === >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" >>>>>>> "/var/run/ceph/mds.1.pid" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" >>>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" >>>>>>> "/var/lock/subsys/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" >>>>>>> Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>>>>>> unlimited ; while [ 1 ]; do >>>>>>> [ -e /var/run/ceph/mds.1.pid ] || break >>>>>>> pid=`cat /var/run/ceph/mds.1.pid` >>>>>>> while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do >>>>>>> cmd="kill $pid" >>>>>>> echo -n $cmd... >>>>>>> $cmd >>>>>>> sleep 1 >>>>>>> continue >>>>>>> done >>>>>>> break >>>>>>> done" >>>>>>> done >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" >>>>>>> === osd.0 === >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" >>>>>>> "/var/run/ceph/osd.0.pid" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" >>>>>>> --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" >>>>>>> "/var/lock/subsys/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" >>>>>>> Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c >>>>>>> unlimited ; while [ 1 ]; do >>>>>>> [ -e /var/run/ceph/osd.0.pid ] || break >>>>>>> pid=`cat /var/run/ceph/osd.0.pid` >>>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>>>>> cmd="kill $pid" >>>>>>> echo -n $cmd... >>>>>>> $cmd >>>>>>> sleep 1 >>>>>>> continue >>>>>>> done >>>>>>> break >>>>>>> done" >>>>>>> kill 2033...done >>>>>>> Unmounting Btrfs on ceph0:/data/osd0 >>>>>>> --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>>>>> /data/osd0 || true" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" >>>>>>> === osd.1 === >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" >>>>>>> "/var/run/ceph/osd.1.pid" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" >>>>>>> --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" >>>>>>> "/var/lock/subsys/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" >>>>>>> Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c >>>>>>> unlimited ; while [ 1 ]; do >>>>>>> [ -e /var/run/ceph/osd.1.pid ] || break >>>>>>> pid=`cat /var/run/ceph/osd.1.pid` >>>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>>>>> cmd="kill $pid" >>>>>>> echo -n $cmd... >>>>>>> $cmd >>>>>>> sleep 1 >>>>>>> continue >>>>>>> done >>>>>>> break >>>>>>> done" >>>>>>> done >>>>>>> Unmounting Btrfs on ceph1:/data/osd1 >>>>>>> --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>>>>> /data/osd1 || true" >>>>>>> umount: /data/osd1: device is busy. >>>>>>> (In some cases useful info about processes that use >>>>>>> the device is found by lsof(8) or fuser(1)) >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" >>>>>>> === osd.2 === >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" >>>>>>> "/var/run/ceph/osd.2.pid" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" >>>>>>> --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" >>>>>>> "/var/lock/subsys/ceph" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" >>>>>>> /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" >>>>>>> Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c >>>>>>> unlimited ; while [ 1 ]; do >>>>>>> [ -e /var/run/ceph/osd.2.pid ] || break >>>>>>> pid=`cat /var/run/ceph/osd.2.pid` >>>>>>> while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do >>>>>>> cmd="kill $pid" >>>>>>> echo -n $cmd... >>>>>>> $cmd >>>>>>> sleep 1 >>>>>>> continue >>>>>>> done >>>>>>> break >>>>>>> done" >>>>>>> done >>>>>>> Unmounting Btrfs on ceph2:/data/osd2 >>>>>>> --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount >>>>>>> /data/osd2 || true" >>>>>>> umount: /data/osd2: device is busy. >>>>>>> (In some cases useful info about processes that use >>>>>>> the device is found by lsof(8) or fuser(1)) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>> Hi Upendra, >>>>>>>> >>>>>>>> Are you running init-ceph from the source directory? If you do that, >>>>>>>> it will use the ceph.conf in the source directory itself, which is >>>>>>>> probably not what you want. So it might be good to double-check that. >>>>>>>> >>>>>>>> If all else fails, running init-ceph with -x will show you exactly >>>>>>>> what the script is doing. If all goes well, its exit status should be >>>>>>>> 0. Are you getting exit status 0? >>>>>>>> >>>>>>>> Colin >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>> Hi Colin >>>>>>>>> >>>>>>>>> Here is my ceph.conf : >>>>>>>>> >>>>>>>>> [global] >>>>>>>>> pid file = /var/run/ceph/$name.pid >>>>>>>>> debug ms = 1 >>>>>>>>> [mon] >>>>>>>>> mon data = /data/mon$id >>>>>>>>> [mon.0] >>>>>>>>> host = ceph0 >>>>>>>>> mon addr = 192.168.155.5:6789 >>>>>>>>> [mon.1] >>>>>>>>> host = ceph1 >>>>>>>>> mon addr = 192.168.155.6:6789 >>>>>>>>> [mon.2] >>>>>>>>> host = ceph2 >>>>>>>>> mon addr = 192.168.155.7:6789 >>>>>>>>> [mds] >>>>>>>>> >>>>>>>>> [mds0] >>>>>>>>> host = ceph0 >>>>>>>>> [mds1] >>>>>>>>> host = ceph1 >>>>>>>>> >>>>>>>>> [osd] >>>>>>>>> sudo = true >>>>>>>>> osd data = /data/osd$id >>>>>>>>> osd journal = /data/osd$id/journal >>>>>>>>> osd journal size = 512 >>>>>>>>> osd use stale snap = true >>>>>>>>> [osd0] >>>>>>>>> host = ceph0 >>>>>>>>> btrfs devs = /dev/sdb >>>>>>>>> [osd1] >>>>>>>>> host = ceph1 >>>>>>>>> btrfs devs = /dev/sdb >>>>>>>>> [osd2] >>>>>>>>> host = ceph2 >>>>>>>>> btrfs devs = /dev/sdb >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>>> I'm using head of line from the master branch. But that particular >>>>>>>>>> code hasn't changed since January, which is 0.24.2 is from. >>>>>>>>>> >>>>>>>>>> In my ceph.conf, I just had an osd that was on a remote machine, and >>>>>>>>>> everything else local. >>>>>>>>>> >>>>>>>>>> If you could post your ceph.conf here or in IRC, perhaps we might spot >>>>>>>>>> an issue that's causing the problems that you see. >>>>>>>>>> >>>>>>>>>> Colin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>>>> Hi Colin >>>>>>>>>>> I am using >>>>>>>>>>> ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 >>>>>>>>>>> ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic >>>>>>>>>>> >>>>>>>>>>> Installed ceph form apt source. >>>>>>>>>>> >>>>>>>>>>> With above configurations i still able yo reproduce. >>>>>>>>>>> Can you please share me ur configurations? >>>>>>>>>>> >>>>>>>>>>> On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>>>>> On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: >>>>>>>>>>>>> On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: >>>>>>>>>>>>>> But if we want to start a particular osd or mon or mds ,its not >>>>>>>>>>>>>> working and there is no error >>>>>>>>>>>>>> eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error >>>>>>>>>>>>> >>>>>>>>>>>>> That is expected, unless you are running init-ceph on the same node as >>>>>>>>>>>>> osd1 is on. >>>>>>>>>>>>> >>>>>>>>>>>>> It might be nice to have some kind of interface like "run command X on >>>>>>>>>>>>> osd1", but init-ceph is not that. >>>>>>>>>>>>> >>>>>>>>>>>>>> /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops >>>>>>>>>>>>>> on current node only >>>>>>>>>>>>>> where as >>>>>>>>>>>>>> /etc/init.d/ceph -a killall works fine. >>>>>>>>>>>>> >>>>>>>>>>>>> That sounds like a bug. I'll see if I can fix it. >>>>>>>>>>>> >>>>>>>>>>>> I'm afraid I can't reproduce this. >>>>>>>>>>>> >>>>>>>>>>>> I ran /etc/init.d/ceph -a stop >>>>>>>>>>>> >>>>>>>>>>>> and it stopped ceph daemons running on remote nodes too. Looking at >>>>>>>>>>>> the code, it looks correct. >>>>>>>>>>>> >>>>>>>>>>>> Colin >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Thanks and Regards, >>>>>>>>>>> Upendra.M >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks and Regards, >>>>>>>>> Upendra.M >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks and Regards, >>>>>>> Upendra.M >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> Upendra.M >>>>> >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> Upendra.M >>> >> > > > > -- > Thanks and Regards, > Upendra.M > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-18 14:11 ` Upendra Moturi 2011-02-26 0:28 ` Colin McCabe @ 2011-02-26 0:28 ` Gregory Farnum 2011-02-28 8:40 ` Upendra Moturi 1 sibling, 1 reply; 18+ messages in thread From: Gregory Farnum @ 2011-02-26 0:28 UTC (permalink / raw) To: Upendra Moturi; +Cc: ceph-devel Sorry, we kind of lost this in the shuffle! Did you manage to resolve this issue? From what I see in your steps below, it looks like you're doing things in the wrong order -- mkcephfs should always be the first thing you do. If you really have nodes running before you do mkcephfs I suspect you just got your cluster into an inconsistent state and the (fairly fragile) methods it uses to shut down all the daemons got broken. -Greg On Friday, February 18, 2011 at 6:11 AM, Upendra Moturi wrote: > Hi Colin > > I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get > i am using the default init script.(Found at /etc/init.d/ceph) > > Regarding issue 1) > > On the osd node I tried to start that osd but it did not work > Steps followed > > 1) Started 3 nodes(ceph.conf is same as i sent earlier) > > 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin > > 3)/etc/init.d/ceph start osd0 (Did this on first osd) > > 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process > > 5)/etc/init.d/ceph -a start ---- shows all process on all nodes > > Regading Issue 2 > It says -x option is not available > > Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even > tried /etc/init.d/ceph -x -a stop > but nothing worked. > > Then tried with /etc/init.d/ceph -a -v stop this also did not stop > ceph on all nodes but got the output as > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" > === mon.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" > "/var/run/ceph/mon.0.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" > Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit > -c unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mon.0.pid ] || break > pid=`cat /var/run/ceph/mon.0.pid` > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > kill 1668...done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" > === mon.1 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" > "/var/run/ceph/mon.1.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" > Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit > -c unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mon.1.pid ] || break > pid=`cat /var/run/ceph/mon.1.pid` > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" > === mon.2 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" > "/var/run/ceph/mon.2.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" > Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit > -c unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mon.2.pid ] || break > pid=`cat /var/run/ceph/mon.2.pid` > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" > === mds.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" > "/var/run/ceph/mds.0.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" > Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mds.0.pid ] || break > pid=`cat /var/run/ceph/mds.0.pid` > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > kill 1844...done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" > === mds.1 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" > "/var/run/ceph/mds.1.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" > Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/mds.1.pid ] || break > pid=`cat /var/run/ceph/mds.1.pid` > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" > === osd.0 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" > "/var/run/ceph/osd.0.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" > Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/osd.0.pid ] || break > pid=`cat /var/run/ceph/osd.0.pid` > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > kill 2033...done > Unmounting Btrfs on ceph0:/data/osd0 > --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount > /data/osd0 || true" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" > === osd.1 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" > "/var/run/ceph/osd.1.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" > Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/osd.1.pid ] || break > pid=`cat /var/run/ceph/osd.1.pid` > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > Unmounting Btrfs on ceph1:/data/osd1 > --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount > /data/osd1 || true" > umount: /data/osd1: device is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" > === osd.2 === > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" > "/var/run/ceph/osd.2.pid" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" > "/var/lock/subsys/ceph" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" > Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c > unlimited ; while [ 1 ]; do > [ -e /var/run/ceph/osd.2.pid ] || break > pid=`cat /var/run/ceph/osd.2.pid` > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > cmd="kill $pid" > echo -n $cmd... > $cmd > sleep 1 > continue > done > break > done" > done > Unmounting Btrfs on ceph2:/data/osd2 > --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount > /data/osd2 || true" > umount: /data/osd2: device is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) > > > > > On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > > Hi Upendra, > > > > Are you running init-ceph from the source directory? If you do that, > > it will use the ceph.conf in the source directory itself, which is > > probably not what you want. So it might be good to double-check that. > > > > If all else fails, running init-ceph with -x will show you exactly > > what the script is doing. If all goes well, its exit status should be > > 0. Are you getting exit status 0? > > > > Colin > > > > > > On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > > > Hi Colin > > > > > > Here is my ceph.conf : > > > > > > [global] > > > pid file = /var/run/ceph/$name.pid > > > debug ms = 1 > > > [mon] > > > mon data = /data/mon$id > > > [mon.0] > > > host = ceph0 > > > mon addr = 192.168.155.5:6789 > > > [mon.1] > > > host = ceph1 > > > mon addr = 192.168.155.6:6789 > > > [mon.2] > > > host = ceph2 > > > mon addr = 192.168.155.7:6789 > > > [mds] > > > > > > [mds0] > > > host = ceph0 > > > [mds1] > > > host = ceph1 > > > > > > [osd] > > > sudo = true > > > osd data = /data/osd$id > > > osd journal = /data/osd$id/journal > > > osd journal size = 512 > > > osd use stale snap = true > > > [osd0] > > > host = ceph0 > > > btrfs devs = /dev/sdb > > > [osd1] > > > host = ceph1 > > > btrfs devs = /dev/sdb > > > [osd2] > > > host = ceph2 > > > btrfs devs = /dev/sdb > > > > > > > > > > > > > > > On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > > > > I'm using head of line from the master branch. But that particular > > > > code hasn't changed since January, which is 0.24.2 is from. > > > > > > > > In my ceph.conf, I just had an osd that was on a remote machine, and > > > > everything else local. > > > > > > > > If you could post your ceph.conf here or in IRC, perhaps we might spot > > > > an issue that's causing the problems that you see. > > > > > > > > Colin > > > > > > > > > > > > On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > > > > > Hi Colin > > > > > I am using > > > > > ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 > > > > > ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic > > > > > > > > > > Installed ceph form apt source. > > > > > > > > > > With above configurations i still able yo reproduce. > > > > > Can you please share me ur configurations? > > > > > > > > > > On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > > > > > > On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > > > > > > > On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > > > > > > > > But if we want to start a particular osd or mon or mds ,its not > > > > > > > > working and there is no error > > > > > > > > eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error > > > > > > > > > > > > > > That is expected, unless you are running init-ceph on the same node as > > > > > > > osd1 is on. > > > > > > > > > > > > > > It might be nice to have some kind of interface like "run command X on > > > > > > > osd1", but init-ceph is not that. > > > > > > > > > > > > > > > /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops > > > > > > > > on current node only > > > > > > > > where as > > > > > > > > /etc/init.d/ceph -a killall works fine. > > > > > > > > > > > > > > That sounds like a bug. I'll see if I can fix it. > > > > > > > > > > > > I'm afraid I can't reproduce this. > > > > > > > > > > > > I ran /etc/init.d/ceph -a stop > > > > > > > > > > > > and it stopped ceph daemons running on remote nodes too. Looking at > > > > > > the code, it looks correct. > > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > -- > > > > > Thanks and Regards, > > > > > Upendra.M > > > > > > > > > > > > -- > > > Thanks and Regards, > > > Upendra.M > > > > -- > Thanks and Regards, > Upendra.M > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: problem with ceph init scripts 2011-02-26 0:28 ` Gregory Farnum @ 2011-02-28 8:40 ` Upendra Moturi 0 siblings, 0 replies; 18+ messages in thread From: Upendra Moturi @ 2011-02-28 8:40 UTC (permalink / raw) To: Gregory Farnum; +Cc: ceph-devel Hi Greg I am running mkcepfs as the first step of cluster and then use init ceph to start all node and this works fine When trying to stop cluser it does not happen and starting a single osd for also dont work. On Sat, Feb 26, 2011 at 5:58 AM, Gregory Farnum <gregory.farnum@dreamhost.com> wrote: > > Sorry, we kind of lost this in the shuffle! > > Did you manage to resolve this issue? From what I see in your steps below, it looks like you're doing things in the wrong order -- mkcephfs should always be the first thing you do. If you really have nodes running before you do mkcephfs I suspect you just got your cluster into an inconsistent state and the (fairly fragile) methods it uses to shut down all the daemons got broken. > -Greg > On Friday, February 18, 2011 at 6:11 AM, Upendra Moturi wrote: > > Hi Colin > > > > I am using the ubuntu 11.04 (32 bit) and got the ceph package from apt-get > > i am using the default init script.(Found at /etc/init.d/ceph) > > > > Regarding issue 1) > > > > On the osd node I tried to start that osd but it did not work > > Steps followed > > > > 1) Started 3 nodes(ceph.conf is same as i sent earlier) > > > > 2) mkcephfs -c /etc/ceph/ceph.conf -a --mkbtrfs -k /etc/ceph/keyring.bin > > > > 3)/etc/init.d/ceph start osd0 (Did this on first osd) > > > > 4) ps -ef | grep ceph or ps -ef | grep cosd ---- does not show any process > > > > 5)/etc/init.d/ceph -a start ---- shows all process on all nodes > > > > Regading Issue 2 > > It says -x option is not available > > > > Tried /etc/init.d/ceph -ax stop and /etc/init.d/ceph -x stop and even > > tried /etc/init.d/ceph -x -a stop > > but nothing worked. > > > > Then tried with /etc/init.d/ceph -a -v stop this also did not stop > > ceph on all nodes but got the output as > > > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "auto start" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "user" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "ssh path" "/etc/ceph" > > === mon.0 === > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pid file" > > "/var/run/ceph/mon.0.pid" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log dir" "/var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "log sym dir" "" > > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "lock file" > > "/var/lock/subsys/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "pre stop command" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mon "post stop command" "" > > Stopping Ceph mon.0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit > > -c unlimited ; while [ 1 ]; do > > [ -e /var/run/ceph/mon.0.pid ] || break > > pid=`cat /var/run/ceph/mon.0.pid` > > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > > cmd="kill $pid" > > echo -n $cmd... > > $cmd > > sleep 1 > > continue > > done > > break > > done" > > kill 1668...done > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "auto start" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "user" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "ssh path" "/etc/ceph" > > === mon.1 === > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pid file" > > "/var/run/ceph/mon.1.pid" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log dir" "/var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "log sym dir" "" > > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "lock file" > > "/var/lock/subsys/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "pre stop command" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mon "post stop command" "" > > Stopping Ceph mon.1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit > > -c unlimited ; while [ 1 ]; do > > [ -e /var/run/ceph/mon.1.pid ] || break > > pid=`cat /var/run/ceph/mon.1.pid` > > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > > cmd="kill $pid" > > echo -n $cmd... > > $cmd > > sleep 1 > > continue > > done > > break > > done" > > done > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "auto start" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "user" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "ssh path" "/etc/ceph" > > === mon.2 === > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pid file" > > "/var/run/ceph/mon.2.pid" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log dir" "/var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "log sym dir" "" > > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "lock file" > > "/var/lock/subsys/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "pre stop command" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t mon "post stop command" "" > > Stopping Ceph mon.2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit > > -c unlimited ; while [ 1 ]; do > > [ -e /var/run/ceph/mon.2.pid ] || break > > pid=`cat /var/run/ceph/mon.2.pid` > > while [ -e /proc/$pid ] && grep -q cmon /proc/$pid/cmdline ; do > > cmd="kill $pid" > > echo -n $cmd... > > $cmd > > sleep 1 > > continue > > done > > break > > done" > > done > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "auto start" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "user" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "ssh path" "/etc/ceph" > > === mds.0 === > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pid file" > > "/var/run/ceph/mds.0.pid" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log dir" "/var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "log sym dir" "" > > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "lock file" > > "/var/lock/subsys/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "pre stop command" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t mds "post stop command" "" > > Stopping Ceph mds0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c > > unlimited ; while [ 1 ]; do > > [ -e /var/run/ceph/mds.0.pid ] || break > > pid=`cat /var/run/ceph/mds.0.pid` > > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do > > cmd="kill $pid" > > echo -n $cmd... > > $cmd > > sleep 1 > > continue > > done > > break > > done" > > kill 1844...done > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "auto start" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "user" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "ssh path" "/etc/ceph" > > === mds.1 === > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pid file" > > "/var/run/ceph/mds.1.pid" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log dir" "/var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "log sym dir" "" > > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "lock file" > > "/var/lock/subsys/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "pre stop command" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t mds "post stop command" "" > > Stopping Ceph mds1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c > > unlimited ; while [ 1 ]; do > > [ -e /var/run/ceph/mds.1.pid ] || break > > pid=`cat /var/run/ceph/mds.1.pid` > > while [ -e /proc/$pid ] && grep -q cmds /proc/$pid/cmdline ; do > > cmd="kill $pid" > > echo -n $cmd... > > $cmd > > sleep 1 > > continue > > done > > break > > done" > > done > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "auto start" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "user" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "ssh path" "/etc/ceph" > > === osd.0 === > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pid file" > > "/var/run/ceph/osd.0.pid" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log dir" "/var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "log sym dir" "" > > --- ssh ceph0 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "osd data" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs path" "/data/osd0" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "btrfs devs" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "lock file" > > "/var/lock/subsys/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "pre stop command" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 0 -t osd "post stop command" "" > > Stopping Ceph osd0 on ceph0...--- ssh ceph0 "cd /etc/ceph ; ulimit -c > > unlimited ; while [ 1 ]; do > > [ -e /var/run/ceph/osd.0.pid ] || break > > pid=`cat /var/run/ceph/osd.0.pid` > > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > > cmd="kill $pid" > > echo -n $cmd... > > $cmd > > sleep 1 > > continue > > done > > break > > done" > > kill 2033...done > > Unmounting Btrfs on ceph0:/data/osd0 > > --- ssh root@ceph0 "cd /etc/ceph ; ulimit -c unlimited ; umount > > /data/osd0 || true" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "auto start" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "user" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "ssh path" "/etc/ceph" > > === osd.1 === > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pid file" > > "/var/run/ceph/osd.1.pid" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log dir" "/var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "log sym dir" "" > > --- ssh ceph1 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "osd data" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs path" "/data/osd1" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "btrfs devs" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "lock file" > > "/var/lock/subsys/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "pre stop command" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 1 -t osd "post stop command" "" > > Stopping Ceph osd1 on ceph1...--- ssh ceph1 "cd /etc/ceph ; ulimit -c > > unlimited ; while [ 1 ]; do > > [ -e /var/run/ceph/osd.1.pid ] || break > > pid=`cat /var/run/ceph/osd.1.pid` > > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > > cmd="kill $pid" > > echo -n $cmd... > > $cmd > > sleep 1 > > continue > > done > > break > > done" > > done > > Unmounting Btrfs on ceph1:/data/osd1 > > --- ssh root@ceph1 "cd /etc/ceph ; ulimit -c unlimited ; umount > > /data/osd1 || true" > > umount: /data/osd1: device is busy. > > (In some cases useful info about processes that use > > the device is found by lsof(8) or fuser(1)) > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "auto start" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "user" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "ssh path" "/etc/ceph" > > === osd.2 === > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pid file" > > "/var/run/ceph/osd.2.pid" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log dir" "/var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "log sym dir" "" > > --- ssh ceph2 "cd /etc/ceph ; ulimit -c unlimited ; mkdir -p /var/log/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "osd data" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs path" "/data/osd2" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "btrfs devs" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "lock file" > > "/var/lock/subsys/ceph" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "pre stop command" "" > > /usr/bin/cconf -c /etc/ceph/ceph.conf -i 2 -t osd "post stop command" "" > > Stopping Ceph osd2 on ceph2...--- ssh ceph2 "cd /etc/ceph ; ulimit -c > > unlimited ; while [ 1 ]; do > > [ -e /var/run/ceph/osd.2.pid ] || break > > pid=`cat /var/run/ceph/osd.2.pid` > > while [ -e /proc/$pid ] && grep -q cosd /proc/$pid/cmdline ; do > > cmd="kill $pid" > > echo -n $cmd... > > $cmd > > sleep 1 > > continue > > done > > break > > done" > > done > > Unmounting Btrfs on ceph2:/data/osd2 > > --- ssh root@ceph2 "cd /etc/ceph ; ulimit -c unlimited ; umount > > /data/osd2 || true" > > umount: /data/osd2: device is busy. > > (In some cases useful info about processes that use > > the device is found by lsof(8) or fuser(1)) > > > > > > > > > > On Fri, Feb 18, 2011 at 12:35 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > > > Hi Upendra, > > > > > > Are you running init-ceph from the source directory? If you do that, > > > it will use the ceph.conf in the source directory itself, which is > > > probably not what you want. So it might be good to double-check that. > > > > > > If all else fails, running init-ceph with -x will show you exactly > > > what the script is doing. If all goes well, its exit status should be > > > 0. Are you getting exit status 0? > > > > > > Colin > > > > > > > > > On Thu, Feb 17, 2011 at 1:55 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > > > > Hi Colin > > > > > > > > Here is my ceph.conf : > > > > > > > > [global] > > > > pid file = /var/run/ceph/$name.pid > > > > debug ms = 1 > > > > [mon] > > > > mon data = /data/mon$id > > > > [mon.0] > > > > host = ceph0 > > > > mon addr = 192.168.155.5:6789 > > > > [mon.1] > > > > host = ceph1 > > > > mon addr = 192.168.155.6:6789 > > > > [mon.2] > > > > host = ceph2 > > > > mon addr = 192.168.155.7:6789 > > > > [mds] > > > > > > > > [mds0] > > > > host = ceph0 > > > > [mds1] > > > > host = ceph1 > > > > > > > > [osd] > > > > sudo = true > > > > osd data = /data/osd$id > > > > osd journal = /data/osd$id/journal > > > > osd journal size = 512 > > > > osd use stale snap = true > > > > [osd0] > > > > host = ceph0 > > > > btrfs devs = /dev/sdb > > > > [osd1] > > > > host = ceph1 > > > > btrfs devs = /dev/sdb > > > > [osd2] > > > > host = ceph2 > > > > btrfs devs = /dev/sdb > > > > > > > > > > > > > > > > > > > > On Thu, Feb 17, 2011 at 1:06 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > > > > > I'm using head of line from the master branch. But that particular > > > > > code hasn't changed since January, which is 0.24.2 is from. > > > > > > > > > > In my ceph.conf, I just had an osd that was on a remote machine, and > > > > > everything else local. > > > > > > > > > > If you could post your ceph.conf here or in IRC, perhaps we might spot > > > > > an issue that's causing the problems that you see. > > > > > > > > > > Colin > > > > > > > > > > > > > > > On Wed, Feb 16, 2011 at 11:01 PM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > > > > > > Hi Colin > > > > > > I am using > > > > > > ceph version 0.24.2 commit:f7572de5cb87eb7157217be4975ae66d90831bb7 > > > > > > ubuntu 11.04 32 bit with upgraded kernal of 2.6.38-2-generic > > > > > > > > > > > > Installed ceph form apt source. > > > > > > > > > > > > With above configurations i still able yo reproduce. > > > > > > Can you please share me ur configurations? > > > > > > > > > > > > On Thu, Feb 17, 2011 at 3:31 AM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > > > > > > > On Wed, Feb 16, 2011 at 1:41 PM, Colin McCabe <cmccabe@alumni.cmu.edu> wrote: > > > > > > > > On Wed, Feb 16, 2011 at 6:44 AM, Upendra Moturi <upendra.m@hexagrid.com> wrote: > > > > > > > > > But if we want to start a particular osd or mon or mds ,its not > > > > > > > > > working and there is no error > > > > > > > > > eg:/etc/init.d/ceph start osd1 does not start osd1 and don't get any error > > > > > > > > > > > > > > > > That is expected, unless you are running init-ceph on the same node as > > > > > > > > osd1 is on. > > > > > > > > > > > > > > > > It might be nice to have some kind of interface like "run command X on > > > > > > > > osd1", but init-ceph is not that. > > > > > > > > > > > > > > > > > /etc/init.d/ceph -a stop also does not stop ceph on all nodes.It stops > > > > > > > > > on current node only > > > > > > > > > where as > > > > > > > > > /etc/init.d/ceph -a killall works fine. > > > > > > > > > > > > > > > > That sounds like a bug. I'll see if I can fix it. > > > > > > > > > > > > > > I'm afraid I can't reproduce this. > > > > > > > > > > > > > > I ran /etc/init.d/ceph -a stop > > > > > > > > > > > > > > and it stopped ceph daemons running on remote nodes too. Looking at > > > > > > > the code, it looks correct. > > > > > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Thanks and Regards, > > > > > > Upendra.M > > > > > > > > > > > > > > > > -- > > > > Thanks and Regards, > > > > Upendra.M > > > > > > > > -- > > Thanks and Regards, > > Upendra.M > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > -- Thanks and Regards, Upendra.M -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2011-03-02 19:26 UTC | newest] Thread overview: 18+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-02-16 14:44 problem with ceph init scripts Upendra Moturi 2011-02-16 21:41 ` Colin McCabe 2011-02-16 22:00 ` Tommi Virtanen 2011-02-16 22:01 ` Colin McCabe 2011-02-17 7:01 ` Upendra Moturi 2011-02-17 7:36 ` Colin McCabe 2011-02-17 9:55 ` Upendra Moturi 2011-02-18 7:05 ` Colin McCabe 2011-02-18 14:11 ` Upendra Moturi 2011-02-26 0:28 ` Colin McCabe 2011-02-28 12:00 ` Upendra Moturi 2011-02-28 16:57 ` Colin McCabe 2011-03-01 11:55 ` Upendra Moturi 2011-03-01 17:34 ` Colin McCabe 2011-03-02 9:18 ` Upendra Moturi 2011-03-02 19:26 ` Colin McCabe 2011-02-26 0:28 ` Gregory Farnum 2011-02-28 8:40 ` Upendra Moturi
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.