waiting for recovery to complete

waiting for recovery to complete

[Hervé Eychenne]

 Hi,

Suppose I'm waiting for a recovery to be completed, and want to run a
command afterwards (halt, send a mail, or anything else...).
The most practiacl way I can see is to check /proc/mdstat.

But what if I want to do that automatically (without bothering looking
at it manually from time to time)?
For example, one could do:
# while cat /proc/mdstat | grep recovery > /dev/null ; do sleep 5 ; done

But that's quite ugly, as:
- it's an active polling, and it is time consuming (even if slightly)
- it may even be unreliable, as I guess one cannot ensure that /proc/mdstat
  will print the "recovery" string during the (very short, but well...)
  transition between two partitions to recover

I think that a passive wait would be much better instead.
And ideally, we should have a simple and efficient way to let a program
know if a device is in a clean state (or being recovered), and another
that would wait until the device is clean (recovery finished).

So, the while loop could be replaced by something like
 mdadm --recovery-wait    (for example)
which would exit only when all pending recoveries have finished, and
let the script continue.
That would be much practical, reliable, and cleaner than a loop, don't you
think?

How this could be achieved is another question... probably the best
would be that userspace can select on a file descriptor, or something
like that (netlink device?)
What do you think?

 Hervé

-- 
 _
(°=  Hervé Eychenne
//)  Homepage:          http://www.eychenne.org/
v_/_ WallFire project:  http://www.wallfire.org/

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: waiting for recovery to complete

[Tim Moore]

The recovery daemon adjusts reconstruction speed dynamically according to 
available system resources.

Disk I/O is somewhat slower but works just fine.  You don't have to wait.


Hervé Eychenne wrote:
>  Hi,
> 
> Suppose I'm waiting for a recovery to be completed, and want to run a
> command afterwards (halt, send a mail, or anything else...).
> The most practiacl way I can see is to check /proc/mdstat.
> 
> But what if I want to do that automatically (without bothering looking
> at it manually from time to time)?
> For example, one could do:
> # while cat /proc/mdstat | grep recovery > /dev/null ; do sleep 5 ; done
> 
> But that's quite ugly, as:
> - it's an active polling, and it is time consuming (even if slightly)
> - it may even be unreliable, as I guess one cannot ensure that /proc/mdstat
>   will print the "recovery" string during the (very short, but well...)
>   transition between two partitions to recover
> 
> I think that a passive wait would be much better instead.
> And ideally, we should have a simple and efficient way to let a program
> know if a device is in a clean state (or being recovered), and another
> that would wait until the device is clean (recovery finished).
> 
> So, the while loop could be replaced by something like
>  mdadm --recovery-wait    (for example)
> which would exit only when all pending recoveries have finished, and
> let the script continue.
> That would be much practical, reliable, and cleaner than a loop, don't you
> think?
> 
> How this could be achieved is another question... probably the best
> would be that userspace can select on a file descriptor, or something
> like that (netlink device?)
> What do you think?
> 
>  Hervé
> 

-- 
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: waiting for recovery to complete

[Hervé Eychenne]

On Sun, Apr 17, 2005 at 10:49:14AM -0700, Tim Moore wrote:

 Hi,

> The recovery daemon adjusts reconstruction speed dynamically according to 
> available system resources.
> Disk I/O is somewhat slower but works just fine.  You don't have to wait.

So I don't have to wait to take the disk out, as the recovery will
continue with embedded disk battery and wireless bus connection?
How cool... ;-)

Well... more seriously, I can't believe this question doesn't raise
any interest, even if it seems like it does not. :-(
Does everyone really type cat /proc/mdstat from time to time??
How clumsy...
I just want to chat about the best way to add a backend for this kind
of feature, so we could implement that properly... (and yes, that is
definitely _nedded_ if you want to do things right)

 Herve

> Hervé Eychenne wrote:
> > Hi,
> >
> >Suppose I'm waiting for a recovery to be completed, and want to run a
> >command afterwards (halt, send a mail, or anything else...).
> >The most practiacl way I can see is to check /proc/mdstat.
> >
> >But what if I want to do that automatically (without bothering looking
> >at it manually from time to time)?
> >For example, one could do:
> ># while cat /proc/mdstat | grep recovery > /dev/null ; do sleep 5 ; done
> >
> >But that's quite ugly, as:
> >- it's an active polling, and it is time consuming (even if slightly)
> >- it may even be unreliable, as I guess one cannot ensure that /proc/mdstat
> >  will print the "recovery" string during the (very short, but well...)
> >  transition between two partitions to recover
> >
> >I think that a passive wait would be much better instead.
> >And ideally, we should have a simple and efficient way to let a program
> >know if a device is in a clean state (or being recovered), and another
> >that would wait until the device is clean (recovery finished).
> >
> >So, the while loop could be replaced by something like
> > mdadm --recovery-wait    (for example)
> >which would exit only when all pending recoveries have finished, and
> >let the script continue.
> >That would be much practical, reliable, and cleaner than a loop, don't you
> >think?
> >
> >How this could be achieved is another question... probably the best
> >would be that userspace can select on a file descriptor, or something
> >like that (netlink device?)
> >What do you think?
> >
> > Hervé

 Hervé

-- 
 _
(°=  Hervé Eychenne
//)  Homepage:          http://www.eychenne.org/
v_/_ WallFire project:  http://www.wallfire.org/

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: waiting for recovery to complete

[David Greaves]

Hervé Eychenne wrote:
> On Sun, Apr 17, 2005 at 10:49:14AM -0700, Tim Moore wrote:
>>The recovery daemon adjusts reconstruction speed dynamically according to 
>>available system resources.
>>Disk I/O is somewhat slower but works just fine.  You don't have to wait.
> 
> 
> So I don't have to wait to take the disk out, as the recovery will
> continue with embedded disk battery and wireless bus connection?
> How cool... ;-)
Your original phrasing looked (to me too) like you thought you couldn't 
use the raid whilst it was reconstructing (I'm still not convinced you 
realise this so, to be clear: "whilst mdadm is rebuilding the array you 
can use the array as normal with no risk of data corruption. You do 
_not_ have to wait for resync to finish before remounting and using the 
device.")
Tim's response told you you had no need to be alerted when it was OK 
since you had no need to stop working in the first place.

(And why are you wanting to take a disk out after you just synced it? - 
no, don't answer that...)

Now it looks like you just want to know when it's done for your own 
peace of mind, so...
> 
> Well... more seriously, I can't believe this question doesn't raise
> any interest, even if it seems like it does not. :-(
well, once recovery has started I don't *really* care when it finishes.
> Does everyone really type cat /proc/mdstat from time to time??
> How clumsy...
And yes, I do :)
(well, actually I optimise to <up arrow> <return>)

> I just want to chat about the best way to add a backend for this kind
> of feature, so we could implement that properly... (and yes, that is
> definitely _nedded_ if you want to do things right)
If you want to monitor _properly_ then use nagios (or monit)

Or since mdadm already uses -F to follow and notify on errors, then I 
suggest you start hacking other alert options in there...

David

-- 
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: waiting for recovery to complete

[Molle Bestefich]

David Greaves wrote:
>> Does everyone really type cat /proc/mdstat from time to time??
>> How clumsy...
> And yes, I do :)

You're not alone..
*gah...*

Re: waiting for recovery to complete

[Alvin Oga]

On Tue, 19 Apr 2005, Molle Bestefich wrote:

> David Greaves wrote:
> >> Does everyone really type cat /proc/mdstat from time to time??
> >> How clumsy...
> > And yes, I do :)
> 
> You're not alone..

and it's still a lot better than some of the hw raid monitoring tools
	- more data
	- more info 
	- other things we can do while waiting ..
	- other gui we can add around it
	- other alarms we can generate to prevent the raid failure

c ya
alvin