[linux-lvm] Somewhat offtopic: Strange umount problem ....

[linux-lvm] Somewhat offtopic: Strange umount problem ....

[Rainer Krienke]

[-- Attachment #1: Type: text/plain, Size: 2652 bytes --]

Hello,

I have a problem with 3 server machines and I would as far as possible try to 
isolate the bug which might be related to lvm or might not be. Perhaps 
someone can help me where and how to look for the problem:

I am running 3 NFS and SMB servers (suse8.2; SuSE kernel 2.4.21-199, LVM 
2.2.00.05) which run perfectly stable but show a strange behaviour
when rebooting the machines: The filesystems which contain user directories
cannot be umounted since the kernel claims that they are still busy. When I
switch to single user mode and run fuser -v or lsof on the mountpoints I
see no output. There are no more smb or other processes in the processlist
that might access the filesystems but nevertheless I cannot umount them.
The output of fuser -v shows only  this:

                     USER        PID ACCESS COMMAND
/export/user1        root     kernel mount  /export/user1

fuser -vm /export/user1 and lsof /export/user1  do not output anything.

The filesystems are xfs on top of LVM2 logical volumes which again run on
top of a md raid1 softmirror (the one and only physical vol)  which is built 
upon a md multipath device:

xfs_fs: LVM2: md_soft_raid1: md_multipath: Fibrechannel_hardware_raid_5

Basically all of the software except for LVM2 is a "original" suse 8.2 
installation. 
 
The problem when I cannot umount the filesystems is, that the software
mirror cannot be shut down cleanly so it is resynced when restarting the
machine (eg for a kernel security upgrade). Since the mirror has a size of
about 1TB this takes quite a long time.  After all if one fine day during such 
a unneeded resync the data source, one of two  hardware raid5 arrays should 
decide to fail then all the data on the mirror would be lost. At the moment 
usually one IDE disk in one hardware raid dies duing the sync.  If two disks 
would die all the data could vanish in a big black hole.

Actually I do not know at the moment how to go on searching and I cannot 
determine if eg LVM might be involved in this problem. So I would be very 
greatful if someone could give me a hint or perhaps even say that this cannot 
be a LVM problem. 

Thanks for any help
Rainer
-- 
---------------------------------------------------------------------------
Rainer Krienke, Universitaet Koblenz, Rechenzentrum, Raum A022
Universitaetsstrasse 1, 56070 Koblenz, Tel: +49 261287 -1312, Fax: -1001312
Mail: krienke@uni-koblenz.de, Web: http://www.uni-koblenz.de/~krienke
Get my public PGP key: http://www.uni-koblenz.de/~krienke/mypgp.html
---------------------------------------------------------------------------

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Somewhat offtopic: Strange umount problem ....

[Klaus Strebel]

Rainer Krienke wrote:
> Hello,
> 
> I have a problem with 3 server machines and I would as far as possible try to 
> isolate the bug which might be related to lvm or might not be. Perhaps 
> someone can help me where and how to look for the problem:
> 
> I am running 3 NFS and SMB servers (suse8.2; SuSE kernel 2.4.21-199, LVM 
> 2.2.00.05) which run perfectly stable but show a strange behaviour
> when rebooting the machines: The filesystems which contain user directories
> cannot be umounted since the kernel claims that they are still busy. When I
> switch to single user mode and run fuser -v or lsof on the mountpoints I
> see no output. There are no more smb or other processes in the processlist
> that might access the filesystems but nevertheless I cannot umount them.
> The output of fuser -v shows only  this:
> 
>                      USER        PID ACCESS COMMAND
> /export/user1        root     kernel mount  /export/user1
> 
> fuser -vm /export/user1 and lsof /export/user1  do not output anything.
> 
> The filesystems are xfs on top of LVM2 logical volumes which again run on
> top of a md raid1 softmirror (the one and only physical vol)  which is built 
> upon a md multipath device:
> 
> xfs_fs: LVM2: md_soft_raid1: md_multipath: Fibrechannel_hardware_raid_5
> 
> Basically all of the software except for LVM2 is a "original" suse 8.2 
> installation. 
>  

Hi Rainer,

is your NFS-server shutting down on reboot. That's typical behaviour for 
an NFS-Server. Perhaps an exportfs -au is missing. Shared filesystems 
cannot be umounted.

Ciao
Klaus
-- 
Klaus Strebel
UNIX-Engineer
<mailto:klaus.strebel@agile.com>

Agile Software GmbH
How Products Become Profits (TM)
Ruschgraben 133
D-76139 Karlsruhe
Office: +49 721 6291 0
Fax: +49 721 6291 1204
URL: <http://www.agile.com>

Re: [linux-lvm] Somewhat offtopic: Strange umount problem ....

[David Johnston]

On Thu, 2004-03-25 at 09:34, Rainer Krienke wrote:
> Hello,
> 
> I have a problem with 3 server machines
> I am running 3 NFS and SMB servers which run perfectly stable but
> show a strange behaviour when rebooting the machines: The
> filesystems which contain user directories cannot be umounted

> fuser -v shows only  this:
> /export/user1        root     kernel mount  /export/user1

Rainer,
nfsd will do this.  Make sure all of your NFS servers are stopped before
you try to umount your filesystems.

Does that help?
-David
-- 
David Johnston <david@littlebald.com>
Little Bald Consulting, LLC

Re: [linux-lvm] Somewhat offtopic: Strange umount problem ....

[Rainer Krienke]

[-- Attachment #1: Type: text/plain, Size: 1504 bytes --]

On Donnerstag, 25. März 2004 19:09, David Johnston wrote:
> On Thu, 2004-03-25 at 09:34, Rainer Krienke wrote:
> > Hello,
> >
> > I have a problem with 3 server machines
> > I am running 3 NFS and SMB servers which run perfectly stable but
> > show a strange behaviour when rebooting the machines: The
> > filesystems which contain user directories cannot be umounted
> >
> > fuser -v shows only  this:
> > /export/user1        root     kernel mount  /export/user1
>
> Rainer,
> nfsd will do this.  Make sure all of your NFS servers are stopped before
> you try to umount your filesystems.
>
> Does that help?

Thanks for the hint, but unfortunately it was no hit: nfsd has already been 
stopped, actually there is no process of which I would assume it could make a 
filesystem busy. Here is a list of all processes still running in single user 
mode when I *cannot* umount the xfs user filesystem on the lvm2 logical 
volume:

http://www.uni-koblenz.de/~krienke/tmp/processes.txt

Does anyone see a suspect in there?

Thanks
Rainer
-- 
---------------------------------------------------------------------------
Rainer Krienke, Universitaet Koblenz, Rechenzentrum, Raum A022
Universitaetsstrasse 1, 56070 Koblenz, Tel: +49 261287 -1312, Fax: -1001312
Mail: krienke@uni-koblenz.de, Web: http://www.uni-koblenz.de/~krienke
Get my public PGP key: http://www.uni-koblenz.de/~krienke/mypgp.html
---------------------------------------------------------------------------

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Somewhat offtopic: Strange umount problem ....

[David Johnston]

On Fri, 2004-03-26 at 02:00, Rainer Krienke wrote:
> On Donnerstag, 25. M�rz 2004 19:09, David Johnston wrote:
> > On Thu, 2004-03-25 at 09:34, Rainer Krienke wrote:
> > > I have a problem with 3 server machines
> > > I am running 3 NFS and SMB servers which run perfectly stable but
> > > the filesystems which contain user directories cannot be umounted
> >
> > Rainer,
> > nfsd will do this.  Make sure all of your NFS servers are stopped before
> > you try to umount your filesystems.
> >
> > Does that help?
> 
> Thanks for the hint, but nfsd has already been stopped
> Here is a list of all processes still running in single user 

Rainer, 
could it be your multipath daemon?  In single-user mode, I would expect
multipathd and the migration_CPUx daemons to shut down.

-- 
David Johnston <david@littlebald.com>
Little Bald Consulting, LLC

Re: [linux-lvm] Somewhat offtopic: Strange umount problem ....

[Rainer Krienke]

[-- Attachment #1: Type: text/plain, Size: 893 bytes --]

Am Freitag, 26. März 2004 17:44 schrieb David Johnston:
...
> > > Rainer,
> > > nfsd will do this.  Make sure all of your NFS servers are stopped
> > > before you try to umount your filesystems.
> > >
> > > Does that help?
> >
> > Thanks for the hint, but nfsd has already been stopped
> > Here is a list of all processes still running in single user
>
> Rainer,
> could it be your multipath daemon?  In single-user mode, I would expect
> multipathd and the migration_CPUx daemons to shut down.

No, this is allright since multipath and software Mirror are sitting "below" 
the filesystem and have to be active at least as long as the filesystem is 
active.. 

@Klaus
nfsd does not run any longer. If you want to take a look, the processes still 
running are here:

http://www.uni-koblenz.de/~krienke/tmp/processes.txt

Nothing suspicious to me....

Thanks
Rainer

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]