Re: /etc/init.d/ceph vs upstart

Re: /etc/init.d/ceph vs upstart

[Tim Spriggs]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

Hi All,

  I am battling extraneous error messages from two sources:

  logrotate which is run in cron.daily and has a definition from the
ceph package in /etc/logrotate.d. The message I get in an email from
every node once a day is:

cat: /var/run/ceph/osd.3.pid: No such file or directory

This comes up because upstart is actually running ceph-osd while the
init.d script expects a pidfile.


  /var/log/ceph/ceph-osd.$id.log which complains:

ERROR: error converting store /var/lib/ceph/osd/ceph-3: (16) Device or
resource busy

This happens on boot as well as on log rotation.


After talking with dmick on irc.oftc.net#ceph, I was alerted to the
fact that there are bits in upstart as well as the sysvinit style
script that attempt to only use one scheme or the other. However, the
logic seems wrong. Inside of ceph_common.sh, there is a function named
check_host which looks for /var/lib/ceph/$type/ceph-$id/sysvinit and
if it exists, it returns. If it doesn't exist, it just goes on to the
next check (which passes in my environment.) Instead, it should return
a non-0 value. Attached is an example patch.

Thanks,
-Tim

[-- Attachment #2: ceph_common.patch --]
[-- Type: application/octet-stream, Size: 464 bytes --]

--- /usr/lib/ceph/ceph_common.sh	2013-11-07 14:50:02.305723490 -0700
+++ /usr/lib/ceph/ceph_common.sh-old	2013-11-07 14:53:34.655303267 -0700
@@ -53,12 +53,10 @@
     # sysvinit managed instance in standard location?
     if [ -e "/var/lib/ceph/$type/ceph-$id/sysvinit" ]; then
 	host="$hostname"
 	echo "=== $type.$id === "
 	return 0
-    else
-        return 1
     fi
 
     # ignore all sections without 'host' defined
     if [ -z "$host" ]; then
 	return 1

Re: /etc/init.d/ceph vs upstart

[Tim Spriggs]

[-- Attachment #1: Type: text/plain, Size: 1319 bytes --]

Oops, I just realized I did the patch in the wrong direction :)

On Thu, Nov 7, 2013 at 3:06 PM, Tim Spriggs <tims@uahirise.org> wrote:
> Hi All,
>
>   I am battling extraneous error messages from two sources:
>
>   logrotate which is run in cron.daily and has a definition from the
> ceph package in /etc/logrotate.d. The message I get in an email from
> every node once a day is:
>
> cat: /var/run/ceph/osd.3.pid: No such file or directory
>
> This comes up because upstart is actually running ceph-osd while the
> init.d script expects a pidfile.
>
>
>   /var/log/ceph/ceph-osd.$id.log which complains:
>
> ERROR: error converting store /var/lib/ceph/osd/ceph-3: (16) Device or
> resource busy
>
> This happens on boot as well as on log rotation.
>
>
> After talking with dmick on irc.oftc.net#ceph, I was alerted to the
> fact that there are bits in upstart as well as the sysvinit style
> script that attempt to only use one scheme or the other. However, the
> logic seems wrong. Inside of ceph_common.sh, there is a function named
> check_host which looks for /var/lib/ceph/$type/ceph-$id/sysvinit and
> if it exists, it returns. If it doesn't exist, it just goes on to the
> next check (which passes in my environment.) Instead, it should return
> a non-0 value. Attached is an example patch.
>
> Thanks,
> -Tim

[-- Attachment #2: ceph_common.patch --]
[-- Type: application/octet-stream, Size: 464 bytes --]

--- /usr/lib/ceph/ceph_common.sh-old	2013-11-07 14:53:34.655303267 -0700
+++ /usr/lib/ceph/ceph_common.sh	2013-11-07 14:50:02.305723490 -0700
@@ -53,10 +53,12 @@
     # sysvinit managed instance in standard location?
     if [ -e "/var/lib/ceph/$type/ceph-$id/sysvinit" ]; then
 	host="$hostname"
 	echo "=== $type.$id === "
 	return 0
+    else
+        return 1
     fi
 
     # ignore all sections without 'host' defined
     if [ -z "$host" ]; then
 	return 1

Re: /etc/init.d/ceph vs upstart

[Tim Spriggs]

... ping

On Thu, Nov 7, 2013 at 3:31 PM, Tim Spriggs <tims@uahirise.org> wrote:
> Oops, I just realized I did the patch in the wrong direction :)
>
> On Thu, Nov 7, 2013 at 3:06 PM, Tim Spriggs <tims@uahirise.org> wrote:
>> Hi All,
>>
>>   I am battling extraneous error messages from two sources:
>>
>>   logrotate which is run in cron.daily and has a definition from the
>> ceph package in /etc/logrotate.d. The message I get in an email from
>> every node once a day is:
>>
>> cat: /var/run/ceph/osd.3.pid: No such file or directory
>>
>> This comes up because upstart is actually running ceph-osd while the
>> init.d script expects a pidfile.
>>
>>
>>   /var/log/ceph/ceph-osd.$id.log which complains:
>>
>> ERROR: error converting store /var/lib/ceph/osd/ceph-3: (16) Device or
>> resource busy
>>
>> This happens on boot as well as on log rotation.
>>
>>
>> After talking with dmick on irc.oftc.net#ceph, I was alerted to the
>> fact that there are bits in upstart as well as the sysvinit style
>> script that attempt to only use one scheme or the other. However, the
>> logic seems wrong. Inside of ceph_common.sh, there is a function named
>> check_host which looks for /var/lib/ceph/$type/ceph-$id/sysvinit and
>> if it exists, it returns. If it doesn't exist, it just goes on to the
>> next check (which passes in my environment.) Instead, it should return
>> a non-0 value. Attached is an example patch.
>>
>> Thanks,
>> -Tim

Re: /etc/init.d/ceph vs upstart

[Josh Durgin]

On 11/25/2013 11:01 AM, Tim Spriggs wrote:
> ... ping
>
> On Thu, Nov 7, 2013 at 3:31 PM, Tim Spriggs <tims@uahirise.org> wrote:
>> Oops, I just realized I did the patch in the wrong direction :)
>>
>> On Thu, Nov 7, 2013 at 3:06 PM, Tim Spriggs <tims@uahirise.org> wrote:
>>> Hi All,
>>>
>>>    I am battling extraneous error messages from two sources:
>>>
>>>    logrotate which is run in cron.daily and has a definition from the
>>> ceph package in /etc/logrotate.d. The message I get in an email from
>>> every node once a day is:
>>>
>>> cat: /var/run/ceph/osd.3.pid: No such file or directory
>>>
>>> This comes up because upstart is actually running ceph-osd while the
>>> init.d script expects a pidfile.
>>>
>>>
>>>    /var/log/ceph/ceph-osd.$id.log which complains:
>>>
>>> ERROR: error converting store /var/lib/ceph/osd/ceph-3: (16) Device or
>>> resource busy
>>>
>>> This happens on boot as well as on log rotation.
>>>
>>>
>>> After talking with dmick on irc.oftc.net#ceph, I was alerted to the
>>> fact that there are bits in upstart as well as the sysvinit style
>>> script that attempt to only use one scheme or the other. However, the
>>> logic seems wrong. Inside of ceph_common.sh, there is a function named
>>> check_host which looks for /var/lib/ceph/$type/ceph-$id/sysvinit and
>>> if it exists, it returns. If it doesn't exist, it just goes on to the
>>> next check (which passes in my environment.) Instead, it should return
>>> a non-0 value. Attached is an example patch.

I think continuing if the host matches is intentional, so the init
script continues working for daemons deployed before
/var/lib/ceph/$type/ceph-$id/sysvinit or 
/var/lib/ceph/$type/ceph-$id/upstart were used.

To maintain backwards compatibility, and prevent both upstart and 
sysvinit from trying to manage the same daemons, I think we can exit
if the file for the other init system is present, like this patch:

https://github.com/ceph/ceph/commit/b1d260cabb90bb9155f18c8e38a1dca102e6466c

Does this work for you?

Josh

Re: /etc/init.d/ceph vs upstart

[Tim Spriggs]

This does seem to fix the issue. Thanks!

On Mon, Nov 25, 2013 at 3:02 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
> On 11/25/2013 11:01 AM, Tim Spriggs wrote:
>>
>> ... ping
>>
>> On Thu, Nov 7, 2013 at 3:31 PM, Tim Spriggs <tims@uahirise.org> wrote:
>>>
>>> Oops, I just realized I did the patch in the wrong direction :)
>>>
>>> On Thu, Nov 7, 2013 at 3:06 PM, Tim Spriggs <tims@uahirise.org> wrote:
>>>>
>>>> Hi All,
>>>>
>>>>    I am battling extraneous error messages from two sources:
>>>>
>>>>    logrotate which is run in cron.daily and has a definition from the
>>>> ceph package in /etc/logrotate.d. The message I get in an email from
>>>> every node once a day is:
>>>>
>>>> cat: /var/run/ceph/osd.3.pid: No such file or directory
>>>>
>>>> This comes up because upstart is actually running ceph-osd while the
>>>> init.d script expects a pidfile.
>>>>
>>>>
>>>>    /var/log/ceph/ceph-osd.$id.log which complains:
>>>>
>>>> ERROR: error converting store /var/lib/ceph/osd/ceph-3: (16) Device or
>>>> resource busy
>>>>
>>>> This happens on boot as well as on log rotation.
>>>>
>>>>
>>>> After talking with dmick on irc.oftc.net#ceph, I was alerted to the
>>>> fact that there are bits in upstart as well as the sysvinit style
>>>> script that attempt to only use one scheme or the other. However, the
>>>> logic seems wrong. Inside of ceph_common.sh, there is a function named
>>>> check_host which looks for /var/lib/ceph/$type/ceph-$id/sysvinit and
>>>> if it exists, it returns. If it doesn't exist, it just goes on to the
>>>> next check (which passes in my environment.) Instead, it should return
>>>> a non-0 value. Attached is an example patch.
>
>
> I think continuing if the host matches is intentional, so the init
> script continues working for daemons deployed before
> /var/lib/ceph/$type/ceph-$id/sysvinit or
> /var/lib/ceph/$type/ceph-$id/upstart were used.
>
> To maintain backwards compatibility, and prevent both upstart and sysvinit
> from trying to manage the same daemons, I think we can exit
> if the file for the other init system is present, like this patch:
>
> https://github.com/ceph/ceph/commit/b1d260cabb90bb9155f18c8e38a1dca102e6466c
>
> Does this work for you?
>
> Josh