[linux-lvm] Re: Oops unmounting snapshot of xfs filesystem

[linux-lvm] Re: Oops unmounting snapshot of xfs filesystem

[Adrian Head]

> Kernel 2.4.16 with 12/14/01 xfs CVS.
There has been heaps of work with respect to XFS and LVM since December last 
year; therefore, I recommend that you try the current 2.4.17-xfs CVS.  If you 
don't like CVS kernels then use the split patches for 2.4.17-xfs (these 
however, won't have all the latest bug fixes).

I am running 2.4.17-xfs (CVS) with LVM-1.0.2 with the xfs_fs_freeze patch and 
I'm not seening your exact problem.  However, there is a bug at the moment 
that doesn't flush all inodes to disk on a freeze so the snapshot is not 
consistant.  The SGI guys are working on this at the moment.

PS.  I think the xfs_fs_freeze patch is actually in the CVS now.


> LVM CVS of 1/21/02 (functionally identical to 1.0.2, I believe).
> LVM's linux-2.4.11-VFS-lock.patch.
> xfs_fs_freeze() patch posted by Eric Sandeen.

-- 
Adrian Head

(Public Key available on request.)

[linux-lvm] Re: Oops unmounting snapshot of xfs filesystem

[Klaus Strebel]

Stephenson, Dale wrote:
> I have been running into an oops when umounting the snapshot of an xfs
> filesystem, or following the umount.  For instance, a umount followed by an
> lvremove will oops in the lvremove, while a mount/umount/mount/umount
> sequence will oops on the second or third umount.
> 
> What does seem consistent is the error message logged on each umount.  Here
> are the messages from a mount/umount sequence:
> 
> <4> Feb  7 11:20:00 kernel: Mounting filesystem "lvm(58,1)" in no-recovery
> mode.  Filesystem will be inconsistent. 
> <2> Feb  7 11:20:02 kernel: lvm - lvm_map: ll_rw_blk write for readonly LV
> /dev/volgr0/lvol1 
> <2> Feb  7 11:20:02 kernel: lvm - lvm_map: ll_rw_blk write for readonly LV
> /dev/volgr0/lvol1 
> <4> Feb  7 11:20:02 kernel: I/O error in filesystem ("lvm(58,1)") meta-data
> dev 0x3a01 block 0xa00020 
> <4> Feb  7 11:20:02 kernel:        ("xlog_iodone") error 5 buf count 1024 
> <4> Feb  7 11:20:02 kernel: xfs_force_shutdown(lvm(58,1),0x2) called from
> line 939 of file xfs_log.c.  Return address = 0xc01b3cbc 
> <4> Feb  7 11:20:02 kernel: Log I/O Error Detected.  Shutting down
> filesystem: lvm(58,1) 
> <4> Feb  7 11:20:02 kernel: Please umount the filesystem, and rectify the
> problem(s) 
> 
> I've attached the oops (run through ksymoops) from this particular umount.
> The snapshot is mounted ro,nouuid,norecovery.  The source of the snapshot
> contains an unmounted xfs filesystem, which was unmounted at the time of
> snapshot creation.  There has been no intentional I/O activity to the
> snapshot, either.
> 
> It looks to me like xfs is trying to flush something to disk on the umount,
> even though the filesystem is read only and no recovery.  I would not have
> expected this.  Is it correct behavior?
> 
> I made the snapshot writeable, but kept the mount options the same.  I was
> able to mount/umount many times without oops, I/O errors, or
> xfs_force_shutdown.
> 
> I did notice similar behavior when a full snapshot returns an I/O error --
> xfs_force_shutdown, with an oops following soon thereafter.
> 
> System details:
> Celeron with 512 MB of RAM and WD 45GB harddrives.
> 128 MB swap, one vg consisting of a one-drive RAID 0.
> Kernel 2.4.16 with 12/14/01 xfs CVS.
> LVM CVS of 1/21/02 (functionally identical to 1.0.2, I believe).
> LVM's linux-2.4.11-VFS-lock.patch.
> xfs_fs_freeze() patch posted by Eric Sandeen.
> Anselm Kruis' writable snapshot patch.

Had the same, disappeared when compiled the kernel with gcc-2.91.66 ! 
Seems to be a compiler issue.

Ciao
Klaus

-- 
Klaus Strebel
UNIX-Engineer
klaus.strebel@eigner.com
EIGNER - Precision Lifecycle Management -
<http://www.eigner.com>

[linux-lvm] RE: Oops unmounting snapshot of xfs filesystem

[Stephenson, Dale]

[-- Attachment #1: Type: text/plain, Size: 6180 bytes --]

Klaus Strebel's suggestion to use 2.91.66 (kgcc) for the kernel did solve
the oops from snapshot umounting, in ordinary circumstances.  Thanks!

However, I still have similar behavior, from umounting an invalid
(filled-up) snapshot.  I took three snapshots of a single xfs logical
volume, mounted the snapshots, and ran I-O on the source until the snapshots
were invalidated.  I then umounted two of the snapshots.  I saw
xfs_force_shutdown messages in syslog for each umount, and the second umount
oopsed.

Here's the syslog entries:

<6> Feb 14 15:24:11 kernel: lvm -- giving up to snapshot /dev/volgr0/lvol0
on /dev/volgr0/lvol1: out of space 
<6> Feb 14 15:24:11 kernel: lvm -- giving up to snapshot /dev/volgr0/lvol0
on /dev/volgr0/lvol2: out of space 
<6> Feb 14 15:24:11 kernel: lvm -- giving up to snapshot /dev/volgr0/lvol0
on /dev/volgr0/lvol3: out of space 
<1> Feb 14 15:24:43 kernel: lvm - lvm_map: ll_rw_blk for inactive LV
/dev/volgr0/lvol1 
<1> Feb 14 15:24:43 kernel: lvm - lvm_map: ll_rw_blk for inactive LV
/dev/volgr0/lvol1 
<4> Feb 14 15:24:43 kernel: I/O error in filesystem ("lvm(58,1)") meta-data
dev 0x3a01 block 0xa00020 
<4> Feb 14 15:24:43 kernel:        ("xlog_iodone") error 5 buf count 1024 
<4> Feb 14 15:24:43 kernel: xfs_force_shutdown(lvm(58,1),0x2) called from
line 939 of file xfs_log.c.  Return address = 0xc01b934e 
<4> Feb 14 15:24:43 kernel: Log I/O Error Detected.  Shutting down
filesystem: lvm(58,1) 
<4> Feb 14 15:24:43 kernel: Please umount the filesystem, and rectify the
problem(s) 
<1> Feb 14 15:24:48 kernel: lvm - lvm_map: ll_rw_blk for inactive LV
/dev/volgr0/lvol2 
<1> Feb 14 15:24:48 kernel: lvm - lvm_map: ll_rw_blk for inactive LV
/dev/volgr0/lvol2 
<4> Feb 14 15:24:48 kernel: I/O error in filesystem ("lvm(58,2)") meta-data
dev 0x3a02 block 0xa00020 
<4> Feb 14 15:24:48 kernel:        ("xlog_iodone") error 5 buf count 1024 
<4> Feb 14 15:24:48 kernel: xfs_force_shutdown(lvm(58,2),0x2) called from
line 939 of file xfs_log.c.  Return address = 0xc01b934e 
<4> Feb 14 15:24:48 kernel: Log I/O Error Detected.  Shutting down
filesystem: lvm(58,2) 
<4> Feb 14 15:24:48 kernel: Please umount the filesystem, and rectify the
problem(s) 
<1> Feb 14 15:24:48 kernel: Unable to handle kernel NULL pointer dereference
at virtual address 00000020 
[oops snipped, but output of ksymoops attached to message]

System details:
Celeron with 512 MB of RAM and WD 45GB harddrives.
128 MB swap, one vg consisting of a one-drive RAID 0.
Kernel 2.4.16 with 12/14/01 xfs CVS.
LVM CVS of 1/21/02 (functionally identical to 1.0.2, I believe).
LVM's linux-2.4.11-VFS-lock.patch.
xfs_fs_freeze() patch posted by Eric Sandeen.
Anselm Kruis' writable snapshot patch.

Snapshots were mounted ro,nouuid,norecovery.  The only action taken to the
snapshots was mounting.

I hope to try it soon with 2.4.17, as Adrian Head suggested.

Has anyone else run this case with xfs snapshots?

Why does umount result in I/O activity to a read-only, norecovery file
system?

Thanks,
Dale Stephenson
steph@snapserver.com

> -----Original Message-----
> From: Klaus Strebel [mailto:klaus.strebel@eigner.com]
> Sent: Monday, February 11, 2002 2:46 AM
> To: Stephenson, Dale
> Cc: 'linux-xfs@oss.sgi.com'; 'linux-lvm@sistina.com'
> Subject: Re: Oops unmounting snapshot of xfs filesystem
> 
> 
> Stephenson, Dale wrote:
> > I have been running into an oops when umounting the 
> snapshot of an xfs
> > filesystem, or following the umount.  For instance, a 
> umount followed by an
> > lvremove will oops in the lvremove, while a 
> mount/umount/mount/umount
> > sequence will oops on the second or third umount.
> > 
> > What does seem consistent is the error message logged on 
> each umount.  Here
> > are the messages from a mount/umount sequence:
> > 
> > <4> Feb  7 11:20:00 kernel: Mounting filesystem "lvm(58,1)" 
> in no-recovery
> > mode.  Filesystem will be inconsistent. 
> > <2> Feb  7 11:20:02 kernel: lvm - lvm_map: ll_rw_blk write 
> for readonly LV
> > /dev/volgr0/lvol1 
> > <2> Feb  7 11:20:02 kernel: lvm - lvm_map: ll_rw_blk write 
> for readonly LV
> > /dev/volgr0/lvol1 
> > <4> Feb  7 11:20:02 kernel: I/O error in filesystem 
> ("lvm(58,1)") meta-data
> > dev 0x3a01 block 0xa00020 
> > <4> Feb  7 11:20:02 kernel:        ("xlog_iodone") error 5 
> buf count 1024 
> > <4> Feb  7 11:20:02 kernel: 
> xfs_force_shutdown(lvm(58,1),0x2) called from
> > line 939 of file xfs_log.c.  Return address = 0xc01b3cbc 
> > <4> Feb  7 11:20:02 kernel: Log I/O Error Detected.  Shutting down
> > filesystem: lvm(58,1) 
> > <4> Feb  7 11:20:02 kernel: Please umount the filesystem, 
> and rectify the
> > problem(s) 
> > 
> > I've attached the oops (run through ksymoops) from this 
> particular umount.
> > The snapshot is mounted ro,nouuid,norecovery.  The source 
> of the snapshot
> > contains an unmounted xfs filesystem, which was unmounted 
> at the time of
> > snapshot creation.  There has been no intentional I/O 
> activity to the
> > snapshot, either.
> > 
> > It looks to me like xfs is trying to flush something to 
> disk on the umount,
> > even though the filesystem is read only and no recovery.  I 
> would not have
> > expected this.  Is it correct behavior?
> > 
> > I made the snapshot writeable, but kept the mount options 
> the same.  I was
> > able to mount/umount many times without oops, I/O errors, or
> > xfs_force_shutdown.
> > 
> > I did notice similar behavior when a full snapshot returns 
> an I/O error --
> > xfs_force_shutdown, with an oops following soon thereafter.
> > 
> > System details:
> > Celeron with 512 MB of RAM and WD 45GB harddrives.
> > 128 MB swap, one vg consisting of a one-drive RAID 0.
> > Kernel 2.4.16 with 12/14/01 xfs CVS.
> > LVM CVS of 1/21/02 (functionally identical to 1.0.2, I believe).
> > LVM's linux-2.4.11-VFS-lock.patch.
> > xfs_fs_freeze() patch posted by Eric Sandeen.
> > Anselm Kruis' writable snapshot patch.
> 
> Had the same, disappeared when compiled the kernel with gcc-2.91.66 ! 
> Seems to be a compiler issue.
> 
> Ciao
> Klaus
> 
> -- 
> Klaus Strebel
> UNIX-Engineer
> klaus.strebel@eigner.com
> EIGNER - Precision Lifecycle Management -
> <http://www.eigner.com>
> 


[-- Attachment #2: oops.out --]
[-- Type: application/octet-stream, Size: 3648 bytes --]

ksymoops 2.4.0 on i686 2.4.9-2buildsmp.  Options used
     -V (default)
     -k ksyms (specified)
     -L (specified)
     -O (specified)
     -M (specified)

Error (expand_objects): cannot stat(/lib/modules/2.4.16-2snap/kernel/net/appletalk/appletalk.o) for appletalk
Error (expand_objects): cannot stat(/lib/modules/2.4.16-2snap/kernel/drivers/i2c/adm1024.o) for adm1024
Error (expand_objects): cannot stat(/lib/modules/2.4.16-2snap/kernel/drivers/i2c/i2c-proc.o) for i2c-proc
Error (expand_objects): cannot stat(/lib/modules/2.4.16-2snap/kernel/drivers/i2c/i2c-dev.o) for i2c-dev
Error (expand_objects): cannot stat(/lib/modules/2.4.16-2snap/kernel/drivers/i2c/i2c-core.o) for i2c-core
Error (expand_objects): cannot stat(/lib/modules/2.4.16-2snap/kernel/drivers/net/eepro100.o) for eepro100
Unable to handle kernel NULL pointer dereference at virtual address 00000020 
c012ea83 
Oops: 0000 
CPU:    0 
EIP:    0010:[<c012ea83>]    Not tainted 
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010206 
eax: 00000000   ebx: 00000000   ecx: 00003a02   edx: 00003a02 
esi: 00000006   edi: 00000000   ebp: 00000000   esp: c0719d60 
ds: 0018   es: 0018   ss: 0018 
Process umount (pid: 6416, stackpage=c0719000) 
Stack: c3e403a0 00000000 c0718000 c2bfe000 3a020168 00000000 c012eb68 c3e403a0  
       00000001 c3ace220 c016f038 00003a02 00000001 00000000 c01d18c2 c3ace220  
       00000000 c3ada680 c2f50000 00000400 c01b934e c2bfe000 00000002 c02b27c7  
Call Trace: [<c012eb68>] [<c016f038>] [<c01d18c2>] [<c01b934e>] [<c016bcd8>]  
   [<c016c62c>] [<c01b9388>] [<c01b9aa2>] [<c01baf6a>] [<c01bb0b0>] [<c01b8c94>]  
   [<c01c1032>] [<c01c84d1>] [<c01d2c7a>] [<c01da7cc>] [<c0131ef8>] [<c014098e>]  
   [<c013582f>] [<c0141201>] [<c0120665>] [<c014121c>] [<c0106d7b>]  
Code: 8b 53 20 89 54 24 14 0f b7 54 24 12 66 39 53 0c 75 7b 83 7b  

>>EIP; c012ea83 <invalidate_bdev+3f/104>   <=====
Trace; c012eb68 <__invalidate_buffers+20/7c>
Trace; c016f038 <pagebuf_target_clear+10/2c>
Trace; c01d18c2 <pagebuf_unlock+6286e/6c37c>
Trace; c01b934e <pagebuf_unlock+4a2fa/6c37c>
Trace; c016bcd8 <pagebuf_iodone+38/7c>
Trace; c016c62c <pagebuf_iorequest+108/144>
Trace; c01b9388 <pagebuf_unlock+4a334/6c37c>
Trace; c01b9aa2 <pagebuf_unlock+4aa4e/6c37c>
Trace; c01baf6a <pagebuf_unlock+4bf16/6c37c>
Trace; c01bb0b0 <pagebuf_unlock+4c05c/6c37c>
Trace; c01b8c94 <pagebuf_unlock+49c40/6c37c>
Trace; c01c1032 <pagebuf_unlock+51fde/6c37c>
Trace; c01c84d1 <pagebuf_unlock+5947d/6c37c>
Trace; c01d2c7a <pagebuf_unlock+63c26/6c37c>
Trace; c01da7cc <pagebuf_unlock+6b778/6c37c>
Trace; c0131ef8 <get_super+9f8/c98>
Trace; c014098e <__mntput+1e/624>
Trace; c013582f <path_release+27/144>
Trace; c0141201 <may_umount+26d/826c>
Trace; c0120665 <do_munmap+279/298>
Trace; c014121c <may_umount+288/826c>
Trace; c0106d7b <__up_wakeup+1057/2274>
Code;  c012ea83 <invalidate_bdev+3f/104>
00000000 <_EIP>:
Code;  c012ea83 <invalidate_bdev+3f/104>   <=====
   0:   8b 53 20                  mov    0x20(%ebx),%edx   <=====
Code;  c012ea86 <invalidate_bdev+42/104>
   3:   89 54 24 14               mov    %edx,0x14(%esp,1)
Code;  c012ea8a <invalidate_bdev+46/104>
   7:   0f b7 54 24 12            movzwl 0x12(%esp,1),%edx
Code;  c012ea8f <invalidate_bdev+4b/104>
   c:   66 39 53 0c               cmp    %dx,0xc(%ebx)
Code;  c012ea93 <invalidate_bdev+4f/104>
  10:   75 7b                     jne    8d <_EIP+0x8d> c012eb10 <invalidate_bdev+cc/104>
Code;  c012ea95 <invalidate_bdev+51/104>
  12:   83 7b 00 00               cmpl   $0x0,0x0(%ebx)


6 errors issued.  Results may not be reliable.

[linux-lvm] Re: Oops unmounting snapshot of xfs filesystem

[Stephen Lord]

Stephenson, Dale wrote:

>Klaus Strebel's suggestion to use 2.91.66 (kgcc) for the kernel did solve
>the oops from snapshot umounting, in ordinary circumstances.  Thanks!
>
>However, I still have similar behavior, from umounting an invalid
>(filled-up) snapshot.  I took three snapshots of a single xfs logical
>volume, mounted the snapshots, and ran I-O on the source until the snapshots
>were invalidated.  I then umounted two of the snapshots.  I saw
>xfs_force_shutdown messages in syslog for each umount, and the second umount
>oopsed.
>
>Here's the syslog entries:
>
><6> Feb 14 15:24:11 kernel: lvm -- giving up to snapshot /dev/volgr0/lvol0
>on /dev/volgr0/lvol1: out of space 
><6> Feb 14 15:24:11 kernel: lvm -- giving up to snapshot /dev/volgr0/lvol0
>on /dev/volgr0/lvol2: out of space 
><6> Feb 14 15:24:11 kernel: lvm -- giving up to snapshot /dev/volgr0/lvol0
>on /dev/volgr0/lvol3: out of space 
><1> Feb 14 15:24:43 kernel: lvm - lvm_map: ll_rw_blk for inactive LV
>/dev/volgr0/lvol1 
><1> Feb 14 15:24:43 kernel: lvm - lvm_map: ll_rw_blk for inactive LV
>/dev/volgr0/lvol1 
><4> Feb 14 15:24:43 kernel: I/O error in filesystem ("lvm(58,1)") meta-data
>dev 0x3a01 block 0xa00020 
><4> Feb 14 15:24:43 kernel:        ("xlog_iodone") error 5 buf count 1024 
><4> Feb 14 15:24:43 kernel: xfs_force_shutdown(lvm(58,1),0x2) called from
>line 939 of file xfs_log.c.  Return address = 0xc01b934e 
><4> Feb 14 15:24:43 kernel: Log I/O Error Detected.  Shutting down
>filesystem: lvm(58,1) 
><4> Feb 14 15:24:43 kernel: Please umount the filesystem, and rectify the
>problem(s) 
><1> Feb 14 15:24:48 kernel: lvm - lvm_map: ll_rw_blk for inactive LV
>/dev/volgr0/lvol2 
><1> Feb 14 15:24:48 kernel: lvm - lvm_map: ll_rw_blk for inactive LV
>/dev/volgr0/lvol2 
><4> Feb 14 15:24:48 kernel: I/O error in filesystem ("lvm(58,2)") meta-data
>dev 0x3a02 block 0xa00020 
><4> Feb 14 15:24:48 kernel:        ("xlog_iodone") error 5 buf count 1024 
><4> Feb 14 15:24:48 kernel: xfs_force_shutdown(lvm(58,2),0x2) called from
>line 939 of file xfs_log.c.  Return address = 0xc01b934e 
><4> Feb 14 15:24:48 kernel: Log I/O Error Detected.  Shutting down
>filesystem: lvm(58,2) 
><4> Feb 14 15:24:48 kernel: Please umount the filesystem, and rectify the
>problem(s) 
><1> Feb 14 15:24:48 kernel: Unable to handle kernel NULL pointer dereference
>at virtual address 00000020 
>[oops snipped, but output of ksymoops attached to message]
>
>System details:
>Celeron with 512 MB of RAM and WD 45GB harddrives.
>128 MB swap, one vg consisting of a one-drive RAID 0.
>Kernel 2.4.16 with 12/14/01 xfs CVS.
>LVM CVS of 1/21/02 (functionally identical to 1.0.2, I believe).
>LVM's linux-2.4.11-VFS-lock.patch.
>xfs_fs_freeze() patch posted by Eric Sandeen.
>Anselm Kruis' writable snapshot patch.
>
>Snapshots were mounted ro,nouuid,norecovery.  The only action taken to the
>snapshots was mounting.
>
>I hope to try it soon with 2.4.17, as Adrian Head suggested.
>
>Has anyone else run this case with xfs snapshots?
>
>Why does umount result in I/O activity to a read-only, norecovery file
>system?
>
>Thanks,
>Dale Stephenson
>steph@snapserver.com
>

This is definitely odd, xlog_iodone is the I/O completion function for a 
log write. What
I would suspect is that xfs is not actually seeing the read only state 
somewhere.

I have mounted a regular filesystem with these options and set break points
in various spots during unmount which would do writes, none of them trigger.

Can you build with kdb, run the same test with a breakpoint in 
xfs_forced_shutdown,
and then when it trips do a backtrace for the unmount process.

Thanks

   Steve

>