[linux-lvm] Kernel panic while snapshotting.

[linux-lvm] Kernel panic while snapshotting.

[Jay Weber]

Hi,

I just got a kernel panic while snapshotting with 0.9.1beta2.  I wonder if
anybody has seen any similar issues or I possibly fubar'd the patch on my
end?

Panic was:

	Kernel panic: brw_kiovec: iobuf not locked for I/O

This occured while simultanously copying 20 tarballs back and forth
between two dirs on the source volume and performing a snapshot at that
time.  It appears to have hung my lvcreate (snapshot) command shell.
Oh, this was also using loop device for the source volume on my laptop in
this case.  Hmm, actually could this panic be related to the following
change in lvm.c?  It did work prior in 0.9.1beta1.

@@ -492,7 +488,7 @@
                goto out;

        err = -ENOMEM;
-       iobuf->locked = 1;
+       iobuf->locked = 0;
        iobuf->nr_pages = 0;
        for (i = 0; i < nr_pages; i++)
        {


I haven't seen this one before myself, but in testing on our bigger scsi
based boxes in the back while snapshots are active and heavy IO is
performed on the source volume, the box tends to hang.  I gather this is
related to having to journal all source volume changes to the snapshot and
when you throw massive IO@it, things get slow. Any thoughts on this one
as well?

Re: [linux-lvm] Kernel panic while snapshotting.

[AJ Lewis]

[-- Attachment #1: Type: text/plain, Size: 2276 bytes --]

Which kernel were you using?  2.2.18 or 2.4.0?

On Fri, Jan 19, 2001 at 07:56:26AM -0800, Jay Weber wrote:
> Hi,
> 
> I just got a kernel panic while snapshotting with 0.9.1beta2.  I wonder if
> anybody has seen any similar issues or I possibly fubar'd the patch on my
> end?
> 
> Panic was:
> 
> 	Kernel panic: brw_kiovec: iobuf not locked for I/O
> 
> This occured while simultanously copying 20 tarballs back and forth
> between two dirs on the source volume and performing a snapshot at that
> time.  It appears to have hung my lvcreate (snapshot) command shell.
> Oh, this was also using loop device for the source volume on my laptop in
> this case.  Hmm, actually could this panic be related to the following
> change in lvm.c?  It did work prior in 0.9.1beta1.
> 
> @@ -492,7 +488,7 @@
>                 goto out;
> 
>         err = -ENOMEM;
> -       iobuf->locked = 1;
> +       iobuf->locked = 0;
>         iobuf->nr_pages = 0;
>         for (i = 0; i < nr_pages; i++)
>         {
> 
> 
> I haven't seen this one before myself, but in testing on our bigger scsi
> based boxes in the back while snapshots are active and heavy IO is
> performed on the source volume, the box tends to hang.  I gather this is
> related to having to journal all source volume changes to the snapshot and
> when you throw massive IO at it, things get slow. Any thoughts on this one
> as well?
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm

-- 
AJ Lewis
Sistina Software Inc.                  Voice:  612-379-3951
1313 5th St SE, Suite 111              Fax:    612-379-3952
Minneapolis, MN 55414                  E-Mail: lewis@sistina.com
http://www.sistina.com

Current GPG fingerprint = 3B5F 6011 5216 76A5 2F6B  52A0 941E 1261 0029 2648
Get my key at: http://www.sistina.com/~lewis/gpgkey
 (Unfortunately, the PKS-type keyservers do not work with multiple sub-keys)

-----Begin Obligatory Humorous Quote----------------------------------------
Over and over I find being redundant is key to success
in the art of redundancy - Jay Armstrong
-----End Obligatory Humorous Quote------------------------------------------

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [linux-lvm] Kernel panic while snapshotting.

[Joe Thornber]

Hi Jay,

Drat.  I'm really sorry about this.

Can you try reversing this patch please ?  It should fix it.

Let me know if you're still having trouble afterwards.

- Joe

diff -u -r1.2.2.6 -r1.2.2.7
--- LVM/kernel/lvm-snap.c	2001/01/12 16:08:12	1.2.2.6
+++ LVM/kernel/lvm-snap.c	2001/01/17 15:20:27	1.2.2.7
@@ -45,10 +45,6 @@
 
 static char *lvm_snap_version __attribute__ ((unused)) = "LVM 0.9 snapshot code (13/11/2000)\n";
 
-#ifndef LockPage
-#define LockPage(map) set_bit(PG_locked, &(map)->flags)
-#endif
-
 extern const char *const lvm_name;
 extern int lvm_blocksizes[];
 
@@ -492,7 +488,7 @@
 		goto out;
 
 	err = -ENOMEM;
-	iobuf->locked = 1;
+	iobuf->locked = 0;
 	iobuf->nr_pages = 0;
 	for (i = 0; i < nr_pages; i++)
 	{
@@ -513,9 +509,6 @@
 #endif
 
 		iobuf->maplist[i] = page;
-		/* the only point to lock the page here is to be allowed
-		   to share unmap_kiobuf() in the fail-path */
-		LockPage(page);
 		iobuf->nr_pages++;
 	}
 	iobuf->offset = 0;


On Fri, Jan 19, 2001 at 07:56:26AM -0800, Jay Weber wrote:
> Hi,
> 
> I just got a kernel panic while snapshotting with 0.9.1beta2.  I wonder if
> anybody has seen any similar issues or I possibly fubar'd the patch on my
> end?
> 
> Panic was:
> 
> 	Kernel panic: brw_kiovec: iobuf not locked for I/O
> 
> This occured while simultanously copying 20 tarballs back and forth
> between two dirs on the source volume and performing a snapshot at that
> time.  It appears to have hung my lvcreate (snapshot) command shell.
> Oh, this was also using loop device for the source volume on my laptop in
> this case.  Hmm, actually could this panic be related to the following
> change in lvm.c?  It did work prior in 0.9.1beta1.
> 
> @@ -492,7 +488,7 @@
>                 goto out;
> 
>         err = -ENOMEM;
> -       iobuf->locked = 1;
> +       iobuf->locked = 0;
>         iobuf->nr_pages = 0;
>         for (i = 0; i < nr_pages; i++)
>         {
> 
> 
> I haven't seen this one before myself, but in testing on our bigger scsi
> based boxes in the back while snapshots are active and heavy IO is
> performed on the source volume, the box tends to hang.  I gather this is
> related to having to journal all source volume changes to the snapshot and
> when you throw massive IO at it, things get slow. Any thoughts on this one
> as well?
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> 

Re: [linux-lvm] Kernel panic while snapshotting.

[Jay Weber]

Are the LockPage() calls necessary?  I just reverted to iobuf->locked = 1;
and tested with IO in background and it seems to work again.  Feedback is
of course welcome.  I noticed your comment regarding LockPage() in commit
logs, hence why I ask.  To reiterate, I didn't readd the LockPage() chunk
of code.

Btw, I'm also using the LVM_VFS_ENHANCEMENT option for snapshotting
journaled ext3 filesystem to flush the journal and mark the fs clean in
all these snapshot testings.

On Fri, 19 Jan 2001, Joe Thornber wrote:

> Date: Fri, 19 Jan 2001 16:37:58 +0000
> From: Joe Thornber <joe@66bassett.freeserve.co.uk>
> Reply-To: linux-lvm@sistina.com
> To: linux-lvm@sistina.com
> Subject: Re: [linux-lvm] Kernel panic while snapshotting.
>
> Hi Jay,
>
> Drat.  I'm really sorry about this.
>
> Can you try reversing this patch please ?  It should fix it.
>
> Let me know if you're still having trouble afterwards.
>
> - Joe
>
> diff -u -r1.2.2.6 -r1.2.2.7
> --- LVM/kernel/lvm-snap.c	2001/01/12 16:08:12	1.2.2.6
> +++ LVM/kernel/lvm-snap.c	2001/01/17 15:20:27	1.2.2.7
> @@ -45,10 +45,6 @@
>
>  static char *lvm_snap_version __attribute__ ((unused)) = "LVM 0.9 snapshot code (13/11/2000)\n";
>
> -#ifndef LockPage
> -#define LockPage(map) set_bit(PG_locked, &(map)->flags)
> -#endif
> -
>  extern const char *const lvm_name;
>  extern int lvm_blocksizes[];
>
> @@ -492,7 +488,7 @@
>  		goto out;
>
>  	err = -ENOMEM;
> -	iobuf->locked = 1;
> +	iobuf->locked = 0;
>  	iobuf->nr_pages = 0;
>  	for (i = 0; i < nr_pages; i++)
>  	{
> @@ -513,9 +509,6 @@
>  #endif
>
>  		iobuf->maplist[i] = page;
> -		/* the only point to lock the page here is to be allowed
> -		   to share unmap_kiobuf() in the fail-path */
> -		LockPage(page);
>  		iobuf->nr_pages++;
>  	}
>  	iobuf->offset = 0;
>
>
> On Fri, Jan 19, 2001 at 07:56:26AM -0800, Jay Weber wrote:
> > Hi,
> >
> > I just got a kernel panic while snapshotting with 0.9.1beta2.  I wonder if
> > anybody has seen any similar issues or I possibly fubar'd the patch on my
> > end?
> >
> > Panic was:
> >
> > 	Kernel panic: brw_kiovec: iobuf not locked for I/O
> >
> > This occured while simultanously copying 20 tarballs back and forth
> > between two dirs on the source volume and performing a snapshot at that
> > time.  It appears to have hung my lvcreate (snapshot) command shell.
> > Oh, this was also using loop device for the source volume on my laptop in
> > this case.  Hmm, actually could this panic be related to the following
> > change in lvm.c?  It did work prior in 0.9.1beta1.
> >
> > @@ -492,7 +488,7 @@
> >                 goto out;
> >
> >         err = -ENOMEM;
> > -       iobuf->locked = 1;
> > +       iobuf->locked = 0;
> >         iobuf->nr_pages = 0;
> >         for (i = 0; i < nr_pages; i++)
> >         {
> >
> >
> > I haven't seen this one before myself, but in testing on our bigger scsi
> > based boxes in the back while snapshots are active and heavy IO is
> > performed on the source volume, the box tends to hang.  I gather this is
> > related to having to journal all source volume changes to the snapshot and
> > when you throw massive IO at it, things get slow. Any thoughts on this one
> > as well?
> >
> > _______________________________________________
> > linux-lvm mailing list
> > linux-lvm@sistina.com
> > http://lists.sistina.com/mailman/listinfo/linux-lvm
> >
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
>

Re: [linux-lvm] Kernel panic while snapshotting.

[Stephen Tweedie]

Hi,

On Fri, Jan 19, 2001 at 09:22:55AM -0800, Jay Weber wrote:
> Are the LockPage() calls necessary? 

They are if you are doing kiobuf IO on user virtual addresses, because
the 2.2 VM has no other way to synchronise the IO with the dirty flags
on the page.  For non-user-VA pages, it shouldn't matter.

2.4 does not lock pages in kiobufs at all.

--Stephen

Re: [linux-lvm] Kernel panic while snapshotting.

[Jay Weber]

This was on 2.2.18 based kernel it includes several (and I mean several)
other patches as well in this case.  I just reproduced the problem without
the IO in the background and I'm thinking the removal of LockPage or the
iobuf->locked=0 bit is whats causing this.  I'll report back shortly with
some more results.

On Fri, 19 Jan 2001, Jay Weber wrote:

> Date: Fri, 19 Jan 2001 07:56:26 -0800 (PST)
> From: Jay Weber <jweber@valinux.com>
> Reply-To: linux-lvm@sistina.com
> To: linux-lvm@sistina.com
> Subject: [linux-lvm] Kernel panic while snapshotting.
>
> Hi,
>
> I just got a kernel panic while snapshotting with 0.9.1beta2.  I wonder if
> anybody has seen any similar issues or I possibly fubar'd the patch on my
> end?
>
> Panic was:
>
> 	Kernel panic: brw_kiovec: iobuf not locked for I/O
>
> This occured while simultanously copying 20 tarballs back and forth
> between two dirs on the source volume and performing a snapshot at that
> time.  It appears to have hung my lvcreate (snapshot) command shell.
> Oh, this was also using loop device for the source volume on my laptop in
> this case.  Hmm, actually could this panic be related to the following
> change in lvm.c?  It did work prior in 0.9.1beta1.
>
> @@ -492,7 +488,7 @@
>                 goto out;
>
>         err = -ENOMEM;
> -       iobuf->locked = 1;
> +       iobuf->locked = 0;
>         iobuf->nr_pages = 0;
>         for (i = 0; i < nr_pages; i++)
>         {
>
>
> I haven't seen this one before myself, but in testing on our bigger scsi
> based boxes in the back while snapshots are active and heavy IO is
> performed on the source volume, the box tends to hang.  I gather this is
> related to having to journal all source volume changes to the snapshot and
> when you throw massive IO at it, things get slow. Any thoughts on this one
> as well?
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
>