Kernel bug on mismatching generation_v2 in inode.c:835

Kernel bug on mismatching generation_v2 in inode.c:835

[Szőts Ákos]

Dear list members,

In my previous thread at
http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg23333.html
there was a space_cache kernel bug/panic on kernel 3.8. I could
successfully "fix" that with rebuilding the cache. But some files were
missing/corrupted. So I booted a rescue CD with kernel 3.7 and ran
btrfsck --repair, which repaired quite a few things.

After a reboot I got the following message:
[  469.457386] btrfs: disk space caching is enabled
[  469.503612] btrfs: mismatching generation and generation_v2 found
in root item. This root was probably mounted with an older kernel.
Resetting all new fields.

As soon as anything had wanted to read a bit from the file system, the
hard drive went crazy and was working for 5-10 minutes. After I got a
kernel panic which said there's an error in fs/btrfs/inode.c:835.

In the moment I don't just mount, but want to read something from the
mounted file system under the rescue system, the same procedure
happens.

I made some pictures of it (since I cannot read anything from the
logs, if there are any).
You can find them here: www.morrohun.hu/temp/btrfs/v2/[123].jpg

I wanted to create an image with the aforementined btrfs-image tool,
but yet to have any success .

Could you please give me an advice what can I do now? Living on a
live-CD is not a life insurance :)

Best regards,

Ákos Szőts

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Josef Bacik]

On Tue, Mar 26, 2013 at 08:33:27AM -0600, Szőts Ákos wrote:
> Dear list members,
> 
> In my previous thread at
> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg23333.html
> there was a space_cache kernel bug/panic on kernel 3.8. I could
> successfully "fix" that with rebuilding the cache. But some files were
> missing/corrupted. So I booted a rescue CD with kernel 3.7 and ran
> btrfsck --repair, which repaired quite a few things.
> 
> After a reboot I got the following message:
> [  469.457386] btrfs: disk space caching is enabled
> [  469.503612] btrfs: mismatching generation and generation_v2 found
> in root item. This root was probably mounted with an older kernel.
> Resetting all new fields.
> 
> As soon as anything had wanted to read a bit from the file system, the
> hard drive went crazy and was working for 5-10 minutes. After I got a
> kernel panic which said there's an error in fs/btrfs/inode.c:835.
> 
> In the moment I don't just mount, but want to read something from the
> mounted file system under the rescue system, the same procedure
> happens.
> 
> I made some pictures of it (since I cannot read anything from the
> logs, if there are any).
> You can find them here: www.morrohun.hu/temp/btrfs/v2/[123].jpg
> 
> I wanted to create an image with the aforementined btrfs-image tool,
> but yet to have any success .
> 
> Could you please give me an advice what can I do now? Living on a
> live-CD is not a life insurance :)
> 
> Best regards,
>

Ok I'd like to get btrfs-image working so I can run fsck on it here locally and
see what's wrong.  Can you apply this patch to my tree and rebuild and run
btrfs-image again, it should tell us why it's having trouble opening the device.
It also fixes that slight mkfs not compiling problem ;).  Thanks,

Josef

 
diff --git a/disk-io.c b/disk-io.c
index 72b33da..6d879c5 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -1047,7 +1047,8 @@ struct btrfs_fs_info *open_ctree_fs_info(const char *filename,
 
 	fp = open(filename, flags, 0600);
 	if (fp < 0) {
-		fprintf (stderr, "Could not open %s\n", filename);
+		fprintf (stderr, "Could not open %s, %d (%s)\n", filename,
+			 errno, strerror(errno));
 		return NULL;
 	}
 	info = __open_ctree_fd(fp, filename, sb_bytenr, 0, writes, partial);
diff --git a/mkfs.c b/mkfs.c
index bc68350..003a8fa 100644
--- a/mkfs.c
+++ b/mkfs.c
@@ -296,7 +296,7 @@ static int create_raid_groups(struct btrfs_trans_handle *trans,
 
 	if (!mixed) {
 		u64 total_bytes =
-			btrfs_super_total_bytes(root->fs_info->super_copy);
+			btrfs_super_total_bytes(&root->fs_info->super_copy);
 		u64 alloced_bytes = 0;
 		u64 alloc_flags = BTRFS_BLOCK_GROUP_ENOSPC |
 			(allowed & metadata_profile);

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Szőts Ákos]

The error was just a silly "permission denied" one.

I uploaded the created image to www.morrohun.hu/temp/btrfs/btrfs.img
SHA1: 90ee61e0724700495d26678d58dc229e04a04cc4
Please, write me when you downloaded and I'll delete it.

I have the idea to run btrfsck from your tree on the file system
again, but I won't touch the fs until you said so :).

Ákos

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Josef Bacik]

On Tue, Mar 26, 2013 at 09:59:57AM -0600, Szőts Ákos wrote:
> The error was just a silly "permission denied" one.
> 
> I uploaded the created image to www.morrohun.hu/temp/btrfs/btrfs.img
> SHA1: 90ee61e0724700495d26678d58dc229e04a04cc4
> Please, write me when you downloaded and I'll delete it.
> 
> I have the idea to run btrfsck from your tree on the file system
> again, but I won't touch the fs until you said so :).
> 

Ok you can go ahead and delete it, I just pulled it down, I'm restoring it and
then I'll see whats going on.  Thanks,

Josef

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Josef Bacik]

On Tue, Mar 26, 2013 at 09:59:57AM -0600, Szőts Ákos wrote:
> The error was just a silly "permission denied" one.
> 
> I uploaded the created image to www.morrohun.hu/temp/btrfs/btrfs.img
> SHA1: 90ee61e0724700495d26678d58dc229e04a04cc4
> Please, write me when you downloaded and I'll delete it.
> 
> I have the idea to run btrfsck from your tree on the file system
> again, but I won't touch the fs until you said so :).
> 

You did take this image while the file system was unmounted right?  I'm seeing
different weirdness with the restored image, trying to figure out if it's a
problem with the restore or with the dump.  Thanks,

Josef

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Szőts Ákos]

If my memory servers me well, it was taken in unmounted state. Do you
want me to take an other one and compare the SHA1 values?

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Josef Bacik]

On Tue, Mar 26, 2013 at 12:25:44PM -0600, Szőts Ákos wrote:
> If my memory servers me well, it was taken in unmounted state. Do you
> want me to take an other one and compare the SHA1 values?

Nope I found the bug, just a problem with the restore.  Thanks,

Josef

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Szőts Ákos]

What do you think; is this issue in the file system can be repaired
with btrfsck? Or should I install a new, patched kernel on my
partition somehow?

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Josef Bacik]

On Wed, Mar 27, 2013 at 05:51:32AM -0600, Szőts Ákos wrote:
> What do you think; is this issue in the file system can be repaired
> with btrfsck? Or should I install a new, patched kernel on my
> partition somehow?

Sorry I was flipping between your problem and somebody elses problem yesterday
and I was having trouble restoring your file system from the image.  Today I
only have your problem and I'm almost done fixing this bug in the restore so I
should have something for you today, hopefully before lunch.  Thanks,

Josef

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Josef Bacik]

On Wed, Mar 27, 2013 at 05:51:32AM -0600, Szőts Ákos wrote:
> What do you think; is this issue in the file system can be repaired
> with btrfsck? Or should I install a new, patched kernel on my
> partition somehow?

Ok I've successfully restored your image and I've reproduced your problem, I
will let you know when I've fixed it.  Thanks,

Josef

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Szőts Ákos]

Oh, that's a very good news! Thank you very much :)

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Josef Bacik]

On Tue, Mar 26, 2013 at 08:33:27AM -0600, Szőts Ákos wrote:
> Dear list members,
> 
> In my previous thread at
> http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg23333.html
> there was a space_cache kernel bug/panic on kernel 3.8. I could
> successfully "fix" that with rebuilding the cache. But some files were
> missing/corrupted. So I booted a rescue CD with kernel 3.7 and ran
> btrfsck --repair, which repaired quite a few things.
> 
> After a reboot I got the following message:
> [  469.457386] btrfs: disk space caching is enabled
> [  469.503612] btrfs: mismatching generation and generation_v2 found
> in root item. This root was probably mounted with an older kernel.
> Resetting all new fields.
> 
> As soon as anything had wanted to read a bit from the file system, the
> hard drive went crazy and was working for 5-10 minutes. After I got a
> kernel panic which said there's an error in fs/btrfs/inode.c:835.
> 
> In the moment I don't just mount, but want to read something from the
> mounted file system under the rescue system, the same procedure
> happens.
> 
> I made some pictures of it (since I cannot read anything from the
> logs, if there are any).
> You can find them here: www.morrohun.hu/temp/btrfs/v2/[123].jpg
> 
> I wanted to create an image with the aforementined btrfs-image tool,
> but yet to have any success .
> 
> Could you please give me an advice what can I do now? Living on a
> live-CD is not a life insurance :)
> 

Can you run btrfsck without --repair and capture the output and upload it
somewhere so I can make sure I've restored properly.  I can reproduce the
problem but I'm worried that I'm reproducing it because I've restored badly and
there's something else wrong, I want to make sure I'm seeing exactly what you
are seeing.  Thanks,

Josef

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Szőts Ákos]

I ran two different versions of btrfsck on the partition.

The first one was shipped with openSUSE 12.3, kernel 3.7.
This is the original tool with which the partition was checked:
http://paste.opensuse.org/74569620

The second one is from your tree (maybe it's newer):
http://filebin.ca/bfbwJezYwCV/btrfsck-v2.txt

Ákos

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Josef Bacik]

On Wed, Mar 27, 2013 at 01:17:56PM -0600, Szőts Ákos wrote:
> I ran two different versions of btrfsck on the partition.
> 
> The first one was shipped with openSUSE 12.3, kernel 3.7.
> This is the original tool with which the partition was checked:
> http://paste.opensuse.org/74569620
> 
> The second one is from your tree (maybe it's newer):
> http://filebin.ca/bfbwJezYwCV/btrfsck-v2.txt
> 

Ok so I'm still having some weird issues restoring your image and I wonder if
they are because of the extent tree corruption.  So it looks like you had some
extent tree corruption and you are just unluckily getting hit because the free
space inode is trying to write to an area that has csums but no extent.  So can
you do

btrfsck --repair

using the btrfsck from my tree (you may have to do it a few times to clear
everything out) and then see if that fixes the problem for you?  Thanks,

Josef

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Josef Bacik]

On Wed, Mar 27, 2013 at 01:17:56PM -0600, Szőts Ákos wrote:
> I ran two different versions of btrfsck on the partition.
> 
> The first one was shipped with openSUSE 12.3, kernel 3.7.
> This is the original tool with which the partition was checked:
> http://paste.opensuse.org/74569620
> 
> The second one is from your tree (maybe it's newer):
> http://filebin.ca/bfbwJezYwCV/btrfsck-v2.txt
> 

BTW if the fsck doesn't fix your problem don't despair, I'm hacking up another
option for btrfs-image to just walk the trees themselves and scrape everything
so I can have a full image from you and then I'll be able to fix fsck to fix
your file system.  Thanks,

Josef

Re: Kernel bug on mismatching generation_v2 in inode.c:835

[Szőts Ákos]

I ran btrfsck three times and tried to restart the OS. After a generation_v2 
resetting it booted and now it works! :)

Thank you very much for your help!


Two btrfsck run was needed for the complete cleanup, but interestingly the end 
statistics were a little bit different each time.

Specifically, the values of
- total fs tree bytes: 2285948928
- total extent tree bytes: 303124480
- btree space waste bytes: 742282214
lines varied with each run, even if there were no need of repairing.

Ákos