reconstruction stalls

reconstruction stalls

[Michael Robinton]

i86 2.4.17 kernel
2 - 1de 20 gig western digital drives on separate channels (ASUS-me99)
There are several raid partitions on this drive, one managed to get out of
sync when the cpu failed. The ext2 file system is fine but the arrray will
not reconstruct -- I have 7 linux boxes running various flavors or raid 1
and 5 for several years, and this is the first time I've seen anything
like this.

The reconstruction proceeds until /proc/mdstat says 99.9% complete
(2015872,2015936) finish 0.0 min
then the speed keeps going down with each successive query. There is no
disk activity per the "red" light. Tried this twice with identical results
each time.

Any suggestions??

Michael

Re: reconstruction stalls

[Neil Brown]

On Tuesday July 16, michael@bizsystems.com wrote:
> i86 2.4.17 kernel
> 2 - 1de 20 gig western digital drives on separate channels (ASUS-me99)
> There are several raid partitions on this drive, one managed to get out of
> sync when the cpu failed. The ext2 file system is fine but the arrray will
> not reconstruct -- I have 7 linux boxes running various flavors or raid 1
> and 5 for several years, and this is the first time I've seen anything
> like this.
> 
> The reconstruction proceeds until /proc/mdstat says 99.9% complete
> (2015872,2015936) finish 0.0 min
> then the speed keeps going down with each successive query. There is no
> disk activity per the "red" light. Tried this twice with identical results
> each time.
> 
> Any suggestions??

Sounds like a bug that was fixed recently...
It may be that you just need to encourage some other disc activity on
that system and it will spring to life and finish.

NeilBrown

> 
> Michael
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Couldn't umount on /mnt/md0-device busy???

[bo]

Hello!

I have found a VERY strange issue on using the name "md0" as a mount point.
If I use "/mnt/md0" as a mount point for the md device or normal HD device,
I could not umount it after booting the system having "device is busy"
error.

However, it is OK if I manually make a RAID then mount it to "/mnt/md0".
I checked if it has any processes running on it with "fuser" but nothing.

Does "/mnt/md0" have a special meaning(usage) for system?

Please provide me a clue.

Thanks,

//Bo

The followins are my experiment on this issue;

root@P3000-PP1:/home/bmoon# cat /proc/mounts
/dev/root / ext2 rw 0 0
none /proc proc rw 0 0
/dev/md0 /mnt/md0 ext2 rw 0 0
/dev/md3 /mnt/md3 ext2 rw 0 0

root@P3000-PP1:/home/bmoon# umount /mnt/md0
umount: /mnt/md0: device is busy
---------------
root@P3000-PP1:/etc# fuser -v /mnt/md0

                     USER        PID ACCESS COMMAND
/mnt/md0            root     kernel mount  /mnt/md0

----------------after chnage the name of mount point  from /mnt/md0 to
/mnt/user---------------
root@P3000-PP1:/home/bmoon# cat /proc/mounts
/dev/root / ext2 rw 0 0
none /proc proc rw 0 0
/dev/md0 /mnt/user ext2 rw 0 0
/dev/md3 /mnt/md3 ext2 rw 0 0

root@P3000-PP1:/home/bmoon# umount /mnt/user
root@P3000-PP1:/home/bmoon#
------------------------
root@P3000-PP1:/etc# vi fstab
/dev/hda2        /               ext2    defaults        1 1
none             /proc           proc    defaults        0 0
/dev/hda3        swap            swap    defaults        0 0
/dev/md0         /mnt/user       ext2    defaults        0 0
/dev/md3         /mnt/md3        ext2    defaults        0 0

Another experiment
----------------------
root@P3000-PP1:/home/bmoon# cat /proc/mounts
/dev/root / ext2 rw 0 0
none /proc proc rw 0 0
/dev/hdb2  /mnt/md0 ext2 rw 0 0
/dev/md3 /mnt/md3 ext2 rw 0 0

root@P3000-PP1:/home/bmoon# umount /mnt/md0
umount:  /mnt/md0:  device is busy

Re: Couldn't umount on /mnt/md0-device busy???

[Kanoalani Withington]

There may be an entry for /mnt/md0 in /etc/exports. If there is, usually 
the startup scripts will start an NFS server at boot time and the 
filesystem will be busy as long as that daemon is running. If you mount 
the volume after the scripts run a server won't be started and the 
volume can be manually unmounted without any errors.

-Kanoa

bo wrote:

>Hello!
>
>I have found a VERY strange issue on using the name "md0" as a mount point.
>If I use "/mnt/md0" as a mount point for the md device or normal HD device,
>I could not umount it after booting the system having "device is busy"
>error.
>
>However, it is OK if I manually make a RAID then mount it to "/mnt/md0".
>I checked if it has any processes running on it with "fuser" but nothing.
>
>Does "/mnt/md0" have a special meaning(usage) for system?
>
>Please provide me a clue.
>
>Thanks,
>
>//Bo
>
>The followins are my experiment on this issue;
>
>root@P3000-PP1:/home/bmoon# cat /proc/mounts
>/dev/root / ext2 rw 0 0
>none /proc proc rw 0 0
>/dev/md0 /mnt/md0 ext2 rw 0 0
>/dev/md3 /mnt/md3 ext2 rw 0 0
>
>root@P3000-PP1:/home/bmoon# umount /mnt/md0
>umount: /mnt/md0: device is busy
>---------------
>root@P3000-PP1:/etc# fuser -v /mnt/md0
>
>                     USER        PID ACCESS COMMAND
>/mnt/md0            root     kernel mount  /mnt/md0
>
>----------------after chnage the name of mount point  from /mnt/md0 to
>/mnt/user---------------
>root@P3000-PP1:/home/bmoon# cat /proc/mounts
>/dev/root / ext2 rw 0 0
>none /proc proc rw 0 0
>/dev/md0 /mnt/user ext2 rw 0 0
>/dev/md3 /mnt/md3 ext2 rw 0 0
>
>root@P3000-PP1:/home/bmoon# umount /mnt/user
>root@P3000-PP1:/home/bmoon#
>------------------------
>root@P3000-PP1:/etc# vi fstab
>/dev/hda2        /               ext2    defaults        1 1
>none             /proc           proc    defaults        0 0
>/dev/hda3        swap            swap    defaults        0 0
>/dev/md0         /mnt/user       ext2    defaults        0 0
>/dev/md3         /mnt/md3        ext2    defaults        0 0
>
>Another experiment
>----------------------
>root@P3000-PP1:/home/bmoon# cat /proc/mounts
>/dev/root / ext2 rw 0 0
>none /proc proc rw 0 0
>/dev/hdb2  /mnt/md0 ext2 rw 0 0
>/dev/md3 /mnt/md3 ext2 rw 0 0
>
>root@P3000-PP1:/home/bmoon# umount /mnt/md0
>umount:  /mnt/md0:  device is busy
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Couldn't umount on /mnt/md0-device busy???

[bo]

Yes, I have 4 entries in /etc/exports; /mnt/md0, /mnt/md1, ..md3.
Why do I have this "umount" problem ONLY on /mnt/md0?

These 4 entries have the same option and parameters.

Any more help will be appreciated.

Bo
----- Original Message -----
From: "Kanoalani Withington" <kanoa@cfht.hawaii.edu>
To: "bo" <bo@sosnetwork.net>
Cc: <linux-raid@vger.kernel.org>
Sent: Wednesday, July 17, 2002 11:52 AM
Subject: Re: Couldn't umount on /mnt/md0-device busy???


> There may be an entry for /mnt/md0 in /etc/exports. If there is, usually
> the startup scripts will start an NFS server at boot time and the
> filesystem will be busy as long as that daemon is running. If you mount
> the volume after the scripts run a server won't be started and the
> volume can be manually unmounted without any errors.
>
> -Kanoa
>
> bo wrote:
>
> >Hello!
> >
> >I have found a VERY strange issue on using the name "md0" as a mount
point.
> >If I use "/mnt/md0" as a mount point for the md device or normal HD
device,
> >I could not umount it after booting the system having "device is busy"
> >error.
> >
> >However, it is OK if I manually make a RAID then mount it to "/mnt/md0".
> >I checked if it has any processes running on it with "fuser" but nothing.
> >
> >Does "/mnt/md0" have a special meaning(usage) for system?
> >
> >Please provide me a clue.
> >
> >Thanks,
> >
> >//Bo
> >
> >The followins are my experiment on this issue;
> >
> >root@P3000-PP1:/home/bmoon# cat /proc/mounts
> >/dev/root / ext2 rw 0 0
> >none /proc proc rw 0 0
> >/dev/md0 /mnt/md0 ext2 rw 0 0
> >/dev/md3 /mnt/md3 ext2 rw 0 0
> >
> >root@P3000-PP1:/home/bmoon# umount /mnt/md0
> >umount: /mnt/md0: device is busy
> >---------------
> >root@P3000-PP1:/etc# fuser -v /mnt/md0
> >
> >                     USER        PID ACCESS COMMAND
> >/mnt/md0            root     kernel mount  /mnt/md0
> >
> >----------------after chnage the name of mount point  from /mnt/md0 to
> >/mnt/user---------------
> >root@P3000-PP1:/home/bmoon# cat /proc/mounts
> >/dev/root / ext2 rw 0 0
> >none /proc proc rw 0 0
> >/dev/md0 /mnt/user ext2 rw 0 0
> >/dev/md3 /mnt/md3 ext2 rw 0 0
> >
> >root@P3000-PP1:/home/bmoon# umount /mnt/user
> >root@P3000-PP1:/home/bmoon#
> >------------------------
> >root@P3000-PP1:/etc# vi fstab
> >/dev/hda2        /               ext2    defaults        1 1
> >none             /proc           proc    defaults        0 0
> >/dev/hda3        swap            swap    defaults        0 0
> >/dev/md0         /mnt/user       ext2    defaults        0 0
> >/dev/md3         /mnt/md3        ext2    defaults        0 0
> >
> >Another experiment
> >----------------------
> >root@P3000-PP1:/home/bmoon# cat /proc/mounts
> >/dev/root / ext2 rw 0 0
> >none /proc proc rw 0 0
> >/dev/hdb2  /mnt/md0 ext2 rw 0 0
> >/dev/md3 /mnt/md3 ext2 rw 0 0
> >
> >root@P3000-PP1:/home/bmoon# umount /mnt/md0
> >umount:  /mnt/md0:  device is busy
> >
> >
> >-
> >To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
>
>
>

Re: Couldn't umount on /mnt/md0-device busy???

[bo]

Mike,

It does not matter with the order in /etc/exports.
This "umount" problem goes away when I remove "/mnt/md0" entry from
/etc/exports.  I have 3 more;/mnt/md1, /mnt/md2, /mnt/md3 without any
problem.

I think system has a special meaning with "/mnt/md0".  What does it do???

Any idea?

Bo


----- Original Message -----
From: "Mike Black" <mblack@csi-inc.com>
To: "bo" <bo@sosnetwork.net>
Sent: Thursday, July 18, 2002 5:12 AM
Subject: Re: Couldn't umount on /mnt/md0-device busy???


> Try changing the order in /etc/exports and see if the problem moves.
> Just an idea to test....
>
> ----- Original Message -----
> From: "bo" <bo@sosnetwork.net>
> To: "Kanoalani Withington" <kanoa@cfht.hawaii.edu>
> Cc: <linux-raid@vger.kernel.org>
> Sent: Wednesday, July 17, 2002 8:13 PM
> Subject: Re: Couldn't umount on /mnt/md0-device busy???
>
>
> > Yes, I have 4 entries in /etc/exports; /mnt/md0, /mnt/md1, ..md3.
> > Why do I have this "umount" problem ONLY on /mnt/md0?
> >
> > These 4 entries have the same option and parameters.
> >
> > Any more help will be appreciated.
> >
> > Bo
> > ----- Original Message -----
> > From: "Kanoalani Withington" <kanoa@cfht.hawaii.edu>
> > To: "bo" <bo@sosnetwork.net>
> > Cc: <linux-raid@vger.kernel.org>
> > Sent: Wednesday, July 17, 2002 11:52 AM
> > Subject: Re: Couldn't umount on /mnt/md0-device busy???
> >
> >
> > > There may be an entry for /mnt/md0 in /etc/exports. If there is,
usually
> > > the startup scripts will start an NFS server at boot time and the
> > > filesystem will be busy as long as that daemon is running. If you
mount
> > > the volume after the scripts run a server won't be started and the
> > > volume can be manually unmounted without any errors.
> > >
> > > -Kanoa
> > >
> > > bo wrote:
> > >
> > > >Hello!
> > > >
> > > >I have found a VERY strange issue on using the name "md0" as a mount
> > point.
> > > >If I use "/mnt/md0" as a mount point for the md device or normal HD
> > device,
> > > >I could not umount it after booting the system having "device is
busy"
> > > >error.
> > > >
> > > >However, it is OK if I manually make a RAID then mount it to
"/mnt/md0".
> > > >I checked if it has any processes running on it with "fuser" but
nothing.
> > > >
> > > >Does "/mnt/md0" have a special meaning(usage) for system?
> > > >
> > > >Please provide me a clue.
> > > >
> > > >Thanks,
> > > >
> > > >//Bo
> > > >
> > > >The followins are my experiment on this issue;
> > > >
> > > >root@P3000-PP1:/home/bmoon# cat /proc/mounts
> > > >/dev/root / ext2 rw 0 0
> > > >none /proc proc rw 0 0
> > > >/dev/md0 /mnt/md0 ext2 rw 0 0
> > > >/dev/md3 /mnt/md3 ext2 rw 0 0
> > > >
> > > >root@P3000-PP1:/home/bmoon# umount /mnt/md0
> > > >umount: /mnt/md0: device is busy
> > > >---------------
> > > >root@P3000-PP1:/etc# fuser -v /mnt/md0
> > > >
> > > >                     USER        PID ACCESS COMMAND
> > > >/mnt/md0            root     kernel mount  /mnt/md0
> > > >
> > > >----------------after chnage the name of mount point  from /mnt/md0
to
> > > >/mnt/user---------------
> > > >root@P3000-PP1:/home/bmoon# cat /proc/mounts
> > > >/dev/root / ext2 rw 0 0
> > > >none /proc proc rw 0 0
> > > >/dev/md0 /mnt/user ext2 rw 0 0
> > > >/dev/md3 /mnt/md3 ext2 rw 0 0
> > > >
> > > >root@P3000-PP1:/home/bmoon# umount /mnt/user
> > > >root@P3000-PP1:/home/bmoon#
> > > >------------------------
> > > >root@P3000-PP1:/etc# vi fstab
> > > >/dev/hda2        /               ext2    defaults        1 1
> > > >none             /proc           proc    defaults        0 0
> > > >/dev/hda3        swap            swap    defaults        0 0
> > > >/dev/md0         /mnt/user       ext2    defaults        0 0
> > > >/dev/md3         /mnt/md3        ext2    defaults        0 0
> > > >
> > > >Another experiment
> > > >----------------------
> > > >root@P3000-PP1:/home/bmoon# cat /proc/mounts
> > > >/dev/root / ext2 rw 0 0
> > > >none /proc proc rw 0 0
> > > >/dev/hdb2  /mnt/md0 ext2 rw 0 0
> > > >/dev/md3 /mnt/md3 ext2 rw 0 0
> > > >
> > > >root@P3000-PP1:/home/bmoon# umount /mnt/md0
> > > >umount:  /mnt/md0:  device is busy
> > > >
> > > >
> > > >-
> > > >To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
> > > >the body of a message to majordomo@vger.kernel.org
> > > >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > >
> > > >
> > >
> > >
> > >
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Couldn't umount on /mnt/md0-device busy???

[Ross Vandegrift]

On Thu, Jul 18, 2002 at 10:11:31AM -0700, bo wrote:
> Mike,
> 
> It does not matter with the order in /etc/exports.
> This "umount" problem goes away when I remove "/mnt/md0" entry from
> /etc/exports.  I have 3 more;/mnt/md1, /mnt/md2, /mnt/md3 without any
> problem.
> 
> I think system has a special meaning with "/mnt/md0".  What does it do???

Perhaps:

1) A remote machine has a mount of /mnt/md0
2) Your server has a stale mount record for /mnt/md0
3) Something has wacked out lockd, which is keeping stuff locked

Try "showmount" and see if any machines have your filesystems mounted
when you think they shouldn't.

If some machine has a stale NFS mount, you may need to clear out the
saved statd information.

Ross Vandegrift
ross@willow.seitz.com

Re: Couldn't umount on /mnt/md0-device busy???

[bo]

Yes, you may be right.
>showmount
  Host on P300
  10.0.0.2
  192.168.1.151

It looks like it got the old mount information from manufacturing test.
I do not have those connections(users) now.

How do I remove(erase) those information from lockd?
Where is the file "statd"?

Thanks,

Bo
----- Original Message -----
From: "Ross Vandegrift" <ross@willow.seitz.com>
To: "bo" <bo@sosnetwork.net>
Cc: "Mike Black" <mblack@csi-inc.com>; <linux-raid@vger.kernel.org>
Sent: Thursday, July 18, 2002 10:47 AM
Subject: Re: Couldn't umount on /mnt/md0-device busy???


> On Thu, Jul 18, 2002 at 10:11:31AM -0700, bo wrote:
> > Mike,
> >
> > It does not matter with the order in /etc/exports.
> > This "umount" problem goes away when I remove "/mnt/md0" entry from
> > /etc/exports.  I have 3 more;/mnt/md1, /mnt/md2, /mnt/md3 without any
> > problem.
> >
> > I think system has a special meaning with "/mnt/md0".  What does it
do???
>
> Perhaps:
>
> 1) A remote machine has a mount of /mnt/md0
> 2) Your server has a stale mount record for /mnt/md0
> 3) Something has wacked out lockd, which is keeping stuff locked
>
> Try "showmount" and see if any machines have your filesystems mounted
> when you think they shouldn't.
>
> If some machine has a stale NFS mount, you may need to clear out the
> saved statd information.
>
> Ross Vandegrift
> ross@willow.seitz.com
>

Re: Couldn't umount on /mnt/md0-device busy???

[Ross Vandegrift]

On Thu, Jul 18, 2002 at 05:43:50PM -0700, bo wrote:
> Yes, you may be right.
> >showmount
>   Host on P300
>   10.0.0.2
>   192.168.1.151
> 
> It looks like it got the old mount information from manufacturing test.
> I do not have those connections(users) now.

I do not know if this is the kosher, most correct way to do this, but
look at the files in /var/lib/nfs (that's where they are on my Slackware
box, and a Debian machine at work).

There's a bunch of files in there that have NFS mount info.  Make sure
all clients actually have it umounted, shutdown NFS and nuke that dir
(well, move it to someplace else, and recreated it in case it blows up).

You could probably edit it and remove the non-existant client as well,
but I haven't ever done it this way.

Ross Vandegrift
ross@willow.seitz.com

Re: Couldn't umount on /mnt/md0-device busy???

[Kanoalani Withington]

Removing /var/lib/nfs/* (same dir on RedHat) works though it really 
confuses any clients that have active mounts so make sure the entries in 
rmtab they are truly obsolete first.

-Kanoa

Ross Vandegrift wrote:

>On Thu, Jul 18, 2002 at 05:43:50PM -0700, bo wrote:
>
>>Yes, you may be right.
>>
>>>showmount
>>>
>>  Host on P300
>>  10.0.0.2
>>  192.168.1.151
>>
>>It looks like it got the old mount information from manufacturing test.
>>I do not have those connections(users) now.
>>
>
>I do not know if this is the kosher, most correct way to do this, but
>look at the files in /var/lib/nfs (that's where they are on my Slackware
>box, and a Debian machine at work).
>
>There's a bunch of files in there that have NFS mount info.  Make sure
>all clients actually have it umounted, shutdown NFS and nuke that dir
>(well, move it to someplace else, and recreated it in case it blows up).
>
>You could probably edit it and remove the non-existant client as well,
>but I haven't ever done it this way.
>
>Ross Vandegrift
>ross@willow.seitz.com
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Couldn't umount on /mnt/md0-device busy???

[Egon Eckert]

> It looks like it got the old mount information from manufacturing test.
> I do not have those connections(users) now.
> 
> How do I remove(erase) those information from lockd?

In Debian, the correct way is to do 'exportfs -r', I think.  See the man
page.  Perhaps you can use the same tool in your distro, in case of
kernel-based nfs.

-- 
Egon Eckert, Heaven Industries, s.r.o.
E-mail: egon@heaven.industries.cz