Re: Errors in rebalancing RAID1 array after disk failure.

[Stefan Behrens <sbehrens@giantdisaster.de>]

[-- Attachment #1: Type: text/plain, Size: 3976 bytes --]

On 5/2/2012 5:22 PM, David Sterba wrote:
> On Mon, Apr 30, 2012 at 03:01:04PM +0200, Marco L. Crociani wrote:
>> ./btrfs device delete missing /mnt/sda3
>> ERROR: error removing the device 'missing' - Input/output error
>>
>>
>> Apr 30 13:17:57 evo kernel: [  108.866205] btrfs: allowing degraded mounts
>> Apr 30 13:17:57 evo kernel: [  108.866214] btrfs: disk space caching is enabled
>> Apr 30 13:18:32 evo kernel: [  143.274899] btrfs: relocating block
>> group 1401002393600 flags 17
>> Apr 30 13:19:25 evo kernel: [  196.888248] btrfs csum failed ino 257
>> off 910946304 csum 432355644 private 175165154
>> Apr 30 13:19:25 evo kernel: [  196.889900] btrfs csum failed ino 257
>> off 910946304 csum 432355644 private 175165154
>> Apr 30 13:19:25 evo kernel: [  196.890429] btrfs csum failed ino 257
>> off 910946304 csum 432355644 private 175165154
>> Apr 30 13:19:25 evo kernel: [  197.087419] btrfs csum failed ino 257
>> off 910946304 csum 432355644 private 175165154
>> Apr 30 13:19:25 evo kernel: [  197.087681] btrfs csum failed ino 257
>> off 910946304 csum 432355644 private 175165154
> 
> the failed checksums prevent to remove the data from the device and then
> removing fails with the above error.
> 
>> ./btrfs inspect-internal inode-resolve -v 257 /mnt/sda3/
>> ioctl ret=-1, error: No such file or directory
> 
> So it's not a visible file, possibly a deleted yet uncleaned snapshot or
> the space_cache (guessing from the inode number). But AFAICS the
> checksums are turned off for the free space inode so ...
> 
>> ./btrfs scrub status /mnt/sda3/
>> scrub status for c87975a0-a575-405e-9890-d3f7f25bbd96
>> 	scrub started at Mon Apr 30 13:26:26 2012 and was aborted after 4367 seconds
>> 	total bytes scrubbed: 406.64GB with 2 errors
>> 	error details: csum=2
>> 	corrected errors: 0, uncorrectable errors: 0, unverified errors: 0
> 
> Shouldn't the csum errors be included under uncorrectable?

"uncorrectable errors" would have been set to 2 if no crash had happened.

> 
>> Apr 30 14:37:24 evo kernel: [ 4875.275776] btrfs: checksum error at
>> logical 752871157760 on dev /dev/sda3, sector 873795352, root 259,
>> inode 1580389, offset 612610048, length 4096, links 1 (path:
>         ^^^^^^^
> 
> so the scrub catches different checksum errors than appeared during
> balance (inode 257).
> 
>> Apr 30 14:37:24 evo kernel: [ 4875.275838] BUG: unable to handle kernel NULL pointer dereference at 0000000000000090
>> Apr 30 14:37:24 evo kernel: [ 4875.275848] IP: [<ffffffff811ae841>]  bio_add_page+0x11/0x60
>> Apr 30 14:37:24 evo kernel: [ 4875.276022] RIP:
>> 0010:[<ffffffff811ae841>]  [<ffffffff811ae841>] bio_add_page+0x11/0x60
> 
> this looks like something disappeared under hands of scrub
> 
> 1045                 BUG_ON(!page->page);
> 1046                 bio = bio_alloc(GFP_NOFS, 1);
> 1047                 if (!bio)
> 1048                         return -EIO;
> 1049                 bio->bi_bdev = page->bdev;
> 1050                 bio->bi_sector = page->physical >> 9;
> 1051                 bio->bi_end_io = scrub_complete_bio_end_io;
> 1052                 bio->bi_private = &complete;
> 
> 1054                 ret = bio_add_page(bio, page->page, PAGE_SIZE, 0);
> 1055                 if (PAGE_SIZE != ret) {
> 1056                         bio_put(bio);
> 1057                         return -EIO;
> 1058                 }
> 
> everything is initialized before use here, so it's hidden behind the
> pointers, my bet is at page->bdev->something . Thinking again how things
> got here:
> 
> * unsuccesful device remove 'missing', due to csum errors in a
>   non-regular file
> * crashed scrub, after inidirect access of a null pointer
> 
> Is there anything I missed for steps to reproduce it?

Right. bdev is a NULL pointer for missing devices. Scrub tries to repair
the checksum error by accessing the mirrors, and that device is missing
and NULL.
I'll send a patch tomorrow to prevent the scrub crash in this situation.

Thanks!

[-- Attachment #2: 0001-Btrfs-fix-crash-in-scrub-correction-code-when-device.patch --]
[-- Type: text/plain, Size: 1074 bytes --]

>From 28fa74661f7a0e209a826e212b40d667516f5d1f Mon Sep 17 00:00:00 2001
From: Stefan Behrens <sbehrens@giantdisaster.de>
Date: Wed, 2 May 2012 18:49:57 +0200
Subject: [PATCH] Btrfs: fix crash in scrub correction code when device is missing

When scrub tries to fix an I/O or checksum error and one of the devices
containing the mirror is missing, it crashes on bdev being a NULL pointer.

Signed-off-by: Stefan Behrens <sbehrens@giantdisaster.de>
---
 fs/btrfs/scrub.c |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c
index b679bf6..967bcf1 100644
--- a/fs/btrfs/scrub.c
+++ b/fs/btrfs/scrub.c
@@ -998,6 +998,8 @@ static int scrub_setup_recheck_block(struct scrub_dev *sdev,
 			page = sblock->pagev + page_index;
 			page->logical = logical;
 			page->physical = bbio->stripes[mirror_index].physical;
+			if (bbio->stripes[mirror_index].dev->missing)
+				continue;
 			page->bdev = bbio->stripes[mirror_index].dev->bdev;
 			page->mirror_num = mirror_index + 1;
 			page->page = alloc_page(GFP_NOFS);
-- 
1.7.10.1.362.g242cab3