kernel panic using external journal and snapshots

kernel panic using external journal and snapshots

[Gordon Messmer]

I'm developing backup infrastructure for Linux using snapshots.  One of 
the supported configurations is EXT4 on LVM, but I think I've come 
across a bug.

I'm currently testing on CentOS 6, with kernel 2.6.32-358.6.1.el6.x86_64

If I create a snapshot of a filesystem with an external journal, and 
mount that snapshot while the original filesystem is mounted, the system 
will usually kernel panic.  If it does not panic, then the kernel will 
later refuse to mount it because "External journal has more than one 
user."  So, the first bug seems to be that ext4 doesn't check the 
external journal before mounting the new snapshot to ensure that the 
journal is not in use.  Expected behavior is that the kernel would 
refuse to mount a filesystem whose journal already has a "user" in order 
to prevent a kernel panic.

If that were the extent of the problem, I probably wouldn't bother 
reporting the issue.  However, the kernel will STILL panic if I use 
tune2fs to remove the journal, and even if I use tune2fs to create a new 
internal journal.  The second bug, then, seems to be that once an ext4 
filesystem has an external journal, that journal will continue to be 
used after it has been removed, and even if it is replaced with an 
internal journal.

The following commands were used to test the problem.  These are 
basically the commands run by the "snapshot" application.  It sets a 
minimum snapshot size of 10% of the volume size, then reads the size of 
the volume and the size remaining in the volume group.  It creates a 
uuid for the snapshot name, then creates the new snapshot.  I use 
tune2fs first to verify that there is a journal device recorded, then 
remove the journal.  Afterward, I read the value again and find no 
external journal recorded.  When the filesystem is mounted, the kernel 
will usually panic.

Is this a known bug?  I haven't yet built a newer system to test the 
effects on the current kernel version.

min_size=10
lvsize=$(lvs --noheadings --units m --nosuffix -o lv_size 
"VolGroup/lv_var" | cut -f1 -d.)
vgfree=$(vgs --noheadings --units m --nosuffix -o vg_free 
"/dev/VolGroup" | cut -f1 -d.)
test "$vgfree" -gt $(( $lvsize * $min_size / 100 )) || echo Not enough 
free space on "/dev/VolGroup" for snapshot
uuid=$(uuidgen)
lvcreate -s -n "lv_var-snap-${uuid}" -L "$(( $lvsize * $min_size / 100 
))"m "VolGroup/lv_var"
	  Rounding up size to full physical extent 128.00 MiB
	  Logical volume
	"lv_var-snap-66c41691-21c0-4db2-8af2-09a13d87a881" created
tune2fs -l "/dev/VolGroup/lv_var-snap-${uuid}" | grep "^Journal device:"
	Journal device:              0xfd04
tune2fs -O ^has_journal "/dev/VolGroup/lv_var-snap-${uuid}"
	tune2fs 1.41.12 (17-May-2010)
	Journal removed
tune2fs -l "/dev/VolGroup/lv_var-snap-${uuid}" | grep "^Journal device:"
mount "/dev/VolGroup/lv_var-snap-${uuid}" /mnt
	* kernel panic *


https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot

Re: kernel panic using external journal and snapshots

[Zheng Liu]

Hi Gordon,

On Mon, Nov 11, 2013 at 08:03:47PM -0800, Gordon Messmer wrote:
> I'm developing backup infrastructure for Linux using snapshots.  One
> of the supported configurations is EXT4 on LVM, but I think I've
> come across a bug.
> 
> I'm currently testing on CentOS 6, with kernel 2.6.32-358.6.1.el6.x86_64
> 
> If I create a snapshot of a filesystem with an external journal, and
> mount that snapshot while the original filesystem is mounted, the
> system will usually kernel panic.  If it does not panic, then the
> kernel will later refuse to mount it because "External journal has
> more than one user."  So, the first bug seems to be that ext4
> doesn't check the external journal before mounting the new snapshot
> to ensure that the journal is not in use.  Expected behavior is that
> the kernel would refuse to mount a filesystem whose journal already
> has a "user" in order to prevent a kernel panic.
> 
> If that were the extent of the problem, I probably wouldn't bother
> reporting the issue.  However, the kernel will STILL panic if I use
> tune2fs to remove the journal, and even if I use tune2fs to create a
> new internal journal.  The second bug, then, seems to be that once
> an ext4 filesystem has an external journal, that journal will
> continue to be used after it has been removed, and even if it is
> replaced with an internal journal.
> 
> The following commands were used to test the problem.  These are
> basically the commands run by the "snapshot" application.  It sets a
> minimum snapshot size of 10% of the volume size, then reads the size
> of the volume and the size remaining in the volume group.  It
> creates a uuid for the snapshot name, then creates the new snapshot.
> I use tune2fs first to verify that there is a journal device
> recorded, then remove the journal.  Afterward, I read the value
> again and find no external journal recorded.  When the filesystem is
> mounted, the kernel will usually panic.
> 
> Is this a known bug?  I haven't yet built a newer system to test the
> effects on the current kernel version.
> 
> min_size=10
> lvsize=$(lvs --noheadings --units m --nosuffix -o lv_size
> "VolGroup/lv_var" | cut -f1 -d.)
> vgfree=$(vgs --noheadings --units m --nosuffix -o vg_free
> "/dev/VolGroup" | cut -f1 -d.)
> test "$vgfree" -gt $(( $lvsize * $min_size / 100 )) || echo Not
> enough free space on "/dev/VolGroup" for snapshot
> uuid=$(uuidgen)
> lvcreate -s -n "lv_var-snap-${uuid}" -L "$(( $lvsize * $min_size /
> 100 ))"m "VolGroup/lv_var"
> 	  Rounding up size to full physical extent 128.00 MiB
> 	  Logical volume
> 	"lv_var-snap-66c41691-21c0-4db2-8af2-09a13d87a881" created
> tune2fs -l "/dev/VolGroup/lv_var-snap-${uuid}" | grep "^Journal device:"
> 	Journal device:              0xfd04
> tune2fs -O ^has_journal "/dev/VolGroup/lv_var-snap-${uuid}"
> 	tune2fs 1.41.12 (17-May-2010)
> 	Journal removed
> tune2fs -l "/dev/VolGroup/lv_var-snap-${uuid}" | grep "^Journal device:"
> mount "/dev/VolGroup/lv_var-snap-${uuid}" /mnt
> 	* kernel panic *
> 
> 
> https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot

For the first question, I haven't a LVM device on my hand.  Hence I am
not sure whether there is a bug or not.

For the second one, I do a test on my sand box against Linux-3.12-rc5,
and it works well.  So I think maybe you can try it again on a newer
kernel.

Regards,
                                                - Zheng

Re: kernel panic using external journal and snapshots

[Gordon Messmer]

On 11/21/2013 11:11 PM, Zheng Liu wrote:
> For the first question, I haven't a LVM device on my hand.  Hence I am
> not sure whether there is a bug or not.
>
> For the second one, I do a test on my sand box against Linux-3.12-rc5,
> and it works well.  So I think maybe you can try it again on a newer
> kernel.

The problem manifests when a filesystem which uses external journal and 
its snapshot are simultaneously mounted.  If you don't have an LVM 
device with which to create a snapshot, I'm not sure what you're 
testing.  Could you clarify?

Re: kernel panic using external journal and snapshots

[Zheng Liu]

On Sat, Nov 23, 2013 at 12:30:48PM -0800, Gordon Messmer wrote:
> On 11/21/2013 11:11 PM, Zheng Liu wrote:
> >For the first question, I haven't a LVM device on my hand.  Hence I am
> >not sure whether there is a bug or not.
> >
> >For the second one, I do a test on my sand box against Linux-3.12-rc5,
> >and it works well.  So I think maybe you can try it again on a newer
> >kernel.
> 
> The problem manifests when a filesystem which uses external journal
> and its snapshot are simultaneously mounted.  If you don't have an
> LVM device with which to create a snapshot, I'm not sure what you're
> testing.  Could you clarify?

Ah, sorry, I don't clarify my test.  I just split two partitions on a
hard disk and create a ext4 file system with external journal.  Then I
use tune2fs with '-O ^has_journal' to remove external journal.  After
that I try to mount this file system on 3.12-rc5, and it work well.
That's all I have done.

Thanks,
                                                - Zheng