pre_start command not executed with upstart

pre_start command not executed with upstart

[Wido den Hollander]

Hi,

I was trying to inject a pre_start command on a bunch of OSDs under
Ubuntu 14.04, but that didn't work.

I found out that only the sysvinit script execute pre_start commands,
but the upstart nor the systemd scripts do this.

Is this a thing which will disappear or is this just a oversight?

I'm actually using "sleep 30" as a pre-start to prevent all OSDs from
starting at the same time. Big boxes which die under the workload of all
OSDs starting at the same time.

I also try to prevent a lot of OSDMap changes by booting the OSDs slowly
so I tried to fix this with "pre_start".

Wido

Re: pre_start command not executed with upstart

[Gregory Farnum]

On Tue, Jul 28, 2015 at 8:55 AM, Wido den Hollander <wido@42on.com> wrote:
> Hi,
>
> I was trying to inject a pre_start command on a bunch of OSDs under
> Ubuntu 14.04, but that didn't work.
>
> I found out that only the sysvinit script execute pre_start commands,
> but the upstart nor the systemd scripts do this.
>
> Is this a thing which will disappear or is this just a oversight?

Mmmm, I wasn't familiar with this mechanism. Judging by the git
history I think it was only added to deal with holes in sysvinit's
daemon management and probably won't be brought forward to the other
init systems, but maybe I'm missing something.

>
> I'm actually using "sleep 30" as a pre-start to prevent all OSDs from
> starting at the same time. Big boxes which die under the workload of all
> OSDs starting at the same time.
>
> I also try to prevent a lot of OSDMap changes by booting the OSDs slowly
> so I tried to fix this with "pre_start".
>
> Wido
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: pre_start command not executed with upstart

[Sage Weil]

On Tue, 28 Jul 2015, Wido den Hollander wrote:
> Hi,
> 
> I was trying to inject a pre_start command on a bunch of OSDs under
> Ubuntu 14.04, but that didn't work.
> 
> I found out that only the sysvinit script execute pre_start commands,
> but the upstart nor the systemd scripts do this.
> 
> Is this a thing which will disappear or is this just a oversight?
> 
> I'm actually using "sleep 30" as a pre-start to prevent all OSDs from
> starting at the same time. Big boxes which die under the workload of all
> OSDs starting at the same time.
> 
> I also try to prevent a lot of OSDMap changes by booting the OSDs slowly
> so I tried to fix this with "pre_start".

I was hoping to let it die...

I would take a fresh look at systemd and see if there is a 
different/better mechanism there to address the startup issue.  You can 
manually add a prestart script/command to the unit file...

sage

Re: pre_start command not executed with upstart

[Wido den Hollander]

On 28-07-15 14:29, Sage Weil wrote:
> On Tue, 28 Jul 2015, Wido den Hollander wrote:
>> Hi,
>>
>> I was trying to inject a pre_start command on a bunch of OSDs under
>> Ubuntu 14.04, but that didn't work.
>>
>> I found out that only the sysvinit script execute pre_start commands,
>> but the upstart nor the systemd scripts do this.
>>
>> Is this a thing which will disappear or is this just a oversight?
>>
>> I'm actually using "sleep 30" as a pre-start to prevent all OSDs from
>> starting at the same time. Big boxes which die under the workload of all
>> OSDs starting at the same time.
>>
>> I also try to prevent a lot of OSDMap changes by booting the OSDs slowly
>> so I tried to fix this with "pre_start".
> 
> I was hoping to let it die...
> 
> I would take a fresh look at systemd and see if there is a 
> different/better mechanism there to address the startup issue.  You can 
> manually add a prestart script/command to the unit file...
> 

Oh, it doesn't matter if it dies out. It's just that there will probably
be scenarios where users want to do some sort of pre-start before a OSD
is executed.

The update_crush hook is one of the examples which is already executed
for example.

A pre_start which does a mount for example (for those without udev)
might be useful.

Wido

> sage
> 

Re: pre_start command not executed with upstart

[Sage Weil]

On Thu, 30 Jul 2015, Wido den Hollander wrote:
> On 28-07-15 14:29, Sage Weil wrote:
> > On Tue, 28 Jul 2015, Wido den Hollander wrote:
> >> Hi,
> >>
> >> I was trying to inject a pre_start command on a bunch of OSDs under
> >> Ubuntu 14.04, but that didn't work.
> >>
> >> I found out that only the sysvinit script execute pre_start commands,
> >> but the upstart nor the systemd scripts do this.
> >>
> >> Is this a thing which will disappear or is this just a oversight?
> >>
> >> I'm actually using "sleep 30" as a pre-start to prevent all OSDs from
> >> starting at the same time. Big boxes which die under the workload of all
> >> OSDs starting at the same time.
> >>
> >> I also try to prevent a lot of OSDMap changes by booting the OSDs slowly
> >> so I tried to fix this with "pre_start".
> > 
> > I was hoping to let it die...
> > 
> > I would take a fresh look at systemd and see if there is a 
> > different/better mechanism there to address the startup issue.  You can 
> > manually add a prestart script/command to the unit file...
> > 
> 
> Oh, it doesn't matter if it dies out. It's just that there will probably
> be scenarios where users want to do some sort of pre-start before a OSD
> is executed.
> 
> The update_crush hook is one of the examples which is already executed
> for example.
> 
> A pre_start which does a mount for example (for those without udev)
> might be useful.

For what it's worth, the view from Leannart is that unit files are simple 
enough to be configuration, in which case such users can just add a 
ExecStartPre to their ceph-osd@.service file (instead of doing the same in 
the ceph.conf file).

sage

Re: pre_start command not executed with upstart

[Wido den Hollander]

On 30-07-15 14:28, Sage Weil wrote:
> On Thu, 30 Jul 2015, Wido den Hollander wrote:
>> On 28-07-15 14:29, Sage Weil wrote:
>>> On Tue, 28 Jul 2015, Wido den Hollander wrote:
>>>> Hi,
>>>>
>>>> I was trying to inject a pre_start command on a bunch of OSDs under
>>>> Ubuntu 14.04, but that didn't work.
>>>>
>>>> I found out that only the sysvinit script execute pre_start commands,
>>>> but the upstart nor the systemd scripts do this.
>>>>
>>>> Is this a thing which will disappear or is this just a oversight?
>>>>
>>>> I'm actually using "sleep 30" as a pre-start to prevent all OSDs from
>>>> starting at the same time. Big boxes which die under the workload of all
>>>> OSDs starting at the same time.
>>>>
>>>> I also try to prevent a lot of OSDMap changes by booting the OSDs slowly
>>>> so I tried to fix this with "pre_start".
>>>
>>> I was hoping to let it die...
>>>
>>> I would take a fresh look at systemd and see if there is a 
>>> different/better mechanism there to address the startup issue.  You can 
>>> manually add a prestart script/command to the unit file...
>>>
>>
>> Oh, it doesn't matter if it dies out. It's just that there will probably
>> be scenarios where users want to do some sort of pre-start before a OSD
>> is executed.
>>
>> The update_crush hook is one of the examples which is already executed
>> for example.
>>
>> A pre_start which does a mount for example (for those without udev)
>> might be useful.
> 
> For what it's worth, the view from Leannart is that unit files are simple 
> enough to be configuration, in which case such users can just add a 
> ExecStartPre to their ceph-osd@.service file (instead of doing the same in 
> the ceph.conf file).
> 

Agreed and you are right. We shouldn't do anything special here. Since
both Ubuntu and CentOS/RHEL are migrating to systemd we shouldn't
probably really care about pre_start anymore.

Wido

> sage
>