Mdadm monitor sometimes doesn't send events.

Mdadm monitor sometimes doesn't send events.

[Arkadiusz Bubała]

Hello,
We've got a problem that sometimes mdadm monitor doesn't send events. 
Actions are as follows:

start:
create new RAID 5 array
wait until it's rebuilding
stop created array
go to start

Only first time we get "NewArray" event then when new array is created 
only "RebuildStarted" event appears. After our analysis we found out 
that when we stop an array we get "DeviceDisapeared" event and when we 
create a new array we get "RebuildFinished" and then "RebuildStarted" 
event. Sometimes those two events don't show up.
The problem is in mdstat_read function:

else if (ent->percent == -1 && w[0] >= '0' && w[0] <= '9' && w[l-1] == 
'%') {
     ent->percent = atoi(w);
}

atoi(w) sometimes returns 0 instead -1. It causes that mse->percent = 0 in:

             if (mse &&
                 mse->percent == -1 &&
                 st->percent >= 0)
                 alert("RebuildFinished", dev, NULL, mailaddr, mailfrom, 
alert_cmd, dosyslog);

and above conditions aren't met. Then in:

             if (mse)
                 st->percent = mse->percent;

st->percent becomes 0 and next conditions where "RebuildStarted" event 
should be sent also aren't met:
             if (mse &&
                 st->percent == -1 &&
                 mse->percent >= 0)
                 alert("RebuildStarted", dev, NULL, mailaddr, mailfrom, 
alert_cmd, dosyslog);

Probably it is fine because rebuild already started and this is what 
monitor reads from /proc/mdstat. Shouldn't it return "NewArray" event 
instead those two "RebuildFinished" and "RebuildStarted" in this situation?

We use mdadm version 2.5.6 and monitor runs as a daemon with 10s 
interval and scan option.

Do we need to restart mdadm monitor when new disk is added to the system 
(hotplug)? Do we need to generate config file in such situation or 
running mdadm with option --scan is sufficient?

-- 
Best regards
Arkadiusz Bubała
Open-E Poland Sp. z o.o.
www.open-e.com

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Mdadm monitor sometimes doesn't send events.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 2450 bytes --]

On Mon, 29 Apr 2013 10:52:57 +0200 Arkadiusz Bubała
<arkadiusz.bubala@open-e.com> wrote:

> Hello,
> We've got a problem that sometimes mdadm monitor doesn't send events. 
> Actions are as follows:
> 
> start:
> create new RAID 5 array
> wait until it's rebuilding
> stop created array
> go to start
> 
> Only first time we get "NewArray" event then when new array is created 
> only "RebuildStarted" event appears. After our analysis we found out 
> that when we stop an array we get "DeviceDisapeared" event and when we 
> create a new array we get "RebuildFinished" and then "RebuildStarted" 
> event. Sometimes those two events don't show up.

The missing NewArray messages was fixed in June last year by
commit 0f760384ebc92d81a988c1501e4a493ef064ad66

You are using a very old mdadm.  Maybe you should try 3.2.6

I'm less sure about the RebuildStarted/RebuildFinished oddities but there is
a good chance they are fixed is the latest release too.

NeilBrown


> The problem is in mdstat_read function:
> 
> else if (ent->percent == -1 && w[0] >= '0' && w[0] <= '9' && w[l-1] == 
> '%') {
>      ent->percent = atoi(w);
> }
> 
> atoi(w) sometimes returns 0 instead -1. It causes that mse->percent = 0 in:
> 
>              if (mse &&
>                  mse->percent == -1 &&
>                  st->percent >= 0)
>                  alert("RebuildFinished", dev, NULL, mailaddr, mailfrom, 
> alert_cmd, dosyslog);
> 
> and above conditions aren't met. Then in:
> 
>              if (mse)
>                  st->percent = mse->percent;
> 
> st->percent becomes 0 and next conditions where "RebuildStarted" event 
> should be sent also aren't met:
>              if (mse &&
>                  st->percent == -1 &&
>                  mse->percent >= 0)
>                  alert("RebuildStarted", dev, NULL, mailaddr, mailfrom, 
> alert_cmd, dosyslog);
> 
> Probably it is fine because rebuild already started and this is what 
> monitor reads from /proc/mdstat. Shouldn't it return "NewArray" event 
> instead those two "RebuildFinished" and "RebuildStarted" in this situation?
> 
> We use mdadm version 2.5.6 and monitor runs as a daemon with 10s 
> interval and scan option.
> 
> Do we need to restart mdadm monitor when new disk is added to the system 
> (hotplug)? Do we need to generate config file in such situation or 
> running mdadm with option --scan is sufficient?
> 


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Mdadm monitor sometimes doesn't send events.

[Arkadiusz Bubała]

On 30.04.2013 03:48, NeilBrown wrote:
> On Mon, 29 Apr 2013 10:52:57 +0200 Arkadiusz Bubała
> <arkadiusz.bubala@open-e.com>  wrote:
>
>    
>> Hello,
>> We've got a problem that sometimes mdadm monitor doesn't send events.
>> Actions are as follows:
>>
>> start:
>> create new RAID 5 array
>> wait until it's rebuilding
>> stop created array
>> go to start
>>
>> Only first time we get "NewArray" event then when new array is created
>> only "RebuildStarted" event appears. After our analysis we found out
>> that when we stop an array we get "DeviceDisapeared" event and when we
>> create a new array we get "RebuildFinished" and then "RebuildStarted"
>> event. Sometimes those two events don't show up.
>>      
> The missing NewArray messages was fixed in June last year by
> commit 0f760384ebc92d81a988c1501e4a493ef064ad66
>
> You are using a very old mdadm.  Maybe you should try 3.2.6
>
> I'm less sure about the RebuildStarted/RebuildFinished oddities but there is
> a good chance they are fixed is the latest release too.
>
> NeilBrown
>
>    
Thank you,
I'm testing this on 3.2.6 and it seems to work well. Events don't 
dissapear. We thought that lack of "NewArray" event may be intended.

Could you answer these questions, please:

Do we need to restart mdadm monitor when new disk is added to the system 
(hotplug)?

Do we need to generate config file in such situation or running mdadm 
with option --scan is sufficient?

-- 
Best regards
Arkadiusz Bubała
Open-E Poland Sp. z o.o.
www.open-e.com

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Mdadm monitor sometimes doesn't send events.

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1716 bytes --]

On Thu, 02 May 2013 10:34:58 +0200 Arkadiusz Bubała
<arkadiusz.bubala@open-e.com> wrote:

> On 30.04.2013 03:48, NeilBrown wrote:
> > On Mon, 29 Apr 2013 10:52:57 +0200 Arkadiusz Bubała
> > <arkadiusz.bubala@open-e.com>  wrote:
> >
> >    
> >> Hello,
> >> We've got a problem that sometimes mdadm monitor doesn't send events.
> >> Actions are as follows:
> >>
> >> start:
> >> create new RAID 5 array
> >> wait until it's rebuilding
> >> stop created array
> >> go to start
> >>
> >> Only first time we get "NewArray" event then when new array is created
> >> only "RebuildStarted" event appears. After our analysis we found out
> >> that when we stop an array we get "DeviceDisapeared" event and when we
> >> create a new array we get "RebuildFinished" and then "RebuildStarted"
> >> event. Sometimes those two events don't show up.
> >>      
> > The missing NewArray messages was fixed in June last year by
> > commit 0f760384ebc92d81a988c1501e4a493ef064ad66
> >
> > You are using a very old mdadm.  Maybe you should try 3.2.6
> >
> > I'm less sure about the RebuildStarted/RebuildFinished oddities but there is
> > a good chance they are fixed is the latest release too.
> >
> > NeilBrown
> >
> >    
> Thank you,
> I'm testing this on 3.2.6 and it seems to work well. Events don't 
> dissapear. We thought that lack of "NewArray" event may be intended.
> 
> Could you answer these questions, please:
> 
> Do we need to restart mdadm monitor when new disk is added to the system 
> (hotplug)?

No.

> 
> Do we need to generate config file in such situation or running mdadm 
> with option --scan is sufficient?
> 

--scan should be sufficient.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]