From: Arne Jansen <sensille@gmx.net>
To: linux-btrfs@vger.kernel.org, sami.liedes@iki.fi
Cc: Jan Schmidt <list.btrfs@jan-o-sch.net>
Subject: Re: btrfs GPF in read_extent_buffer() while scrubbing with kernel 3.4.2
Date: Mon, 16 Jul 2012 10:20:28 +0200 [thread overview]
Message-ID: <5003CECC.2040908@gmx.net> (raw)
In-Reply-To: <4FFBD24E.2050902@gmx.net>
Any news on this? I you give me some hints, I can try to reproduce
it here.
-Arne
On 10.07.2012 08:57, Arne Jansen wrote:
> On 10.07.2012 06:16, Sami Liedes wrote:
>> On Mon, Jul 09, 2012 at 11:05:47AM +0200, Arne Jansen wrote:
>>>> * Just before the crash:
>>>> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2261163409408, level 100, generation 4412718571037421157, nritems 538968254. len param 17. debug 2/989/538968254/4412718571037421157/0x0/0/0x0/0x0
>>>>
>>>
>>> At a first glance: the generation converted to ascii is: "ent() ==",
>>> so someone is patching the memory with ascii text, possibly C source.
>>> It might be interesting to dump the full contents of the eb, to get
>>> a clue on the source of the data.
>>
>> I changed the code to dump the contents of the eb struct at the point
>> where that error (btrfs: invalid parameters...) is printed, at the
>> "checksum mismatch 4" site and at the "node seems invalid now" site.
>> Now I have a big log of 1795 corrupted ebs. So far nothing that looks
>> remotely like ascii text, though. But I have two different versions of
>> the eb that caused that warning, a less corrupted one and a more
>> corrupted one:
>>
>> ------------------------------------------------------------
>> btrfs: --- start eb contents at ffff8801b13cc4c8 ---
>> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............
>> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0.......
>> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<.....
>> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<.....
>> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<.............
>> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._..............
>> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f......
>> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@.......
>> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @...............
>> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<.............
>> btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........
>
> this one looks good so far.
>
>> btrfs: --- end eb contents at ffff8801b13cc4c8 ---
>> btrfs: dm-6 checksum verify failed on 2239404212224 wanted B5F632BC found 3579FB59 level 160
>> btrfs: node seems invalid now. checksum ok = 1
>> btrfs: --- start eb contents at ffff8801b13cc4c8 ---
>> [... identical dump to above ...]
>> btrfs: --- end eb contents at ffff8801b13cc4c8 ---
>> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2239404212224, level 160, generation 4716553384049587249, nritems 295705211. len param 17. debug 2/989/295705211/4716553384049587249/0x0/0/0x0/0x0
>> btrfs: --- start eb contents at ffff8801b13cc4c8 ---
>> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............
>> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0.......
>> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<.....
>> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<.....
>> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<.............
>> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._..............
>> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f......
>> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@.......
>> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @...............
>> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<.............
>> btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{.......
>> btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q<P.tA........
>> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........
>> btrfs: --- end eb contents at ffff8801b13cc4c8 ---
>> ------------[ cut here ]------------
>> WARNING: at fs/btrfs/extent_io.c:4533 read_extent_buffer+0x1ef/0x220 [btrfs]()
>> Hardware name: System Product Name
>> Modules linked in: <omitted> [last unloaded: scsi_wait_scan]
>> Pid: 25829, comm: btrfs-endio-met Tainted: G W 3.4.4+btrfsdebug3+ #4
>> [...]
>> ------------------------------------------------------------
>>
>> Here's the difference (lines reordered from diff to make comparing
>> easier):
>>
>> ------------------------------------------------------------
>> -btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<.............
>> +btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<.............
>> -btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{.......
>> -btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q<P.tA........
>> ------------------------------------------------------------
>
> The diff is just the eb->debug[] Jan added.
>
>>
>> If there's one pattern that catches eye in the dumps, it's that in
>> many places where the same eb is dumped multiple times due to multiple
>> "checksum mismatch 4"s, there are bytes at offsets 0x30 and 0x31 that
>> always seem to have the same value and both separately increase
>> between the dumps, usually(?) by two. Then that might be normal, I
>> haven't looked at what should be at a struct eb :) There are some
>> instances where they wrap, for example
>>
>> f8 f8 -> fa fa -> fc fc -> fe fe -> 00 00 -> 02 02 -> ...,
>
> this is the spinlock to protect the refcnt, so some reference taking/releasing
> has been going on.
>
>>
>> with no other changes visible in the corrupted ebs. But that is by no
>> means the only change in the struct eb contents that can be observed,
>> only the most obvious pattern.
>>
>> For example, here's a 11-long string of eb contents. First the dmesg:
>>
>> ------------------------------------------------------------
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 1]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 2]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 3]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 4]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 5]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 6]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 7]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 8]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 9]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 10]
>> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2
>> btrfs: checksum mismatch 4 on 1127200522240
>> [DUMP 11]
>> ------------------------------------------------------------
>>
>> Where DUMP 1 contents (file cc1.txt if you'd want to apply the diffs
>> below) is
>>
>> ------------------------------------------------------------
>> btrfs: --- start eb contents at ffff8801cafbacc0 ---
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> btrfs: ffff8801cafbad30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad40: 00 00 10 00 00 00 00 00 04 04 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad50: 50 ad fb ca 01 88 ff ff 50 ad fb ca 01 88 ff ff P.......P.......
>> btrfs: ffff8801cafbad60: 02 02 00 00 00 00 00 00 68 ad fb ca 01 88 ff ff ........h.......
>> btrfs: ffff8801cafbad70: 68 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 h...............
>> btrfs: ffff8801cafbad80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad90: 00 a2 20 08 00 ea ff ff c0 2a f4 07 00 ea ff ff .. ......*......
>> btrfs: ffff8801cafbada0: 00 ac 89 07 00 ea ff ff c0 42 79 07 00 ea ff ff .........By.....
>> btrfs: ffff8801cafbadb0: 80 af 77 07 00 ea ff ff 40 37 49 08 00 ea ff ff ..w.....@7I.....
>> btrfs: ffff8801cafbadc0: 80 6d f3 07 00 ea ff ff c0 a7 e8 07 00 ea ff ff .m..............
>> btrfs: ffff8801cafbadd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........
>> btrfs: --- end eb contents at ffff8801cafbacc0 ---
>> ------------------------------------------------------------
>>
>> And the successive diffs are
>>
>> ------------------------------------------------------------
>> --- cc1.txt 2012-07-10 06:57:21.564665577 +0300
>> +++ cc2.txt 2012-07-10 06:59:10.272634578 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>
> the spinlock
>
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> @@ -20,9 +20,9 @@
>> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> -btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................
>> -btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> -btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 01 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801cafbae20: 4d 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 M.......N.......
>> +btrfs: ffff8801cafbae30: 07 78 02 00 00 00 00 00 00 00 00 00 00 00 00 00 .x..............
>
> and eb->debug
>
>> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........
>> btrfs: --- end eb contents at ffff8801cafbacc0 ---
>> --- cc2.txt 2012-07-10 06:59:10.272634578 +0300
>> +++ cc3.txt 2012-07-10 06:59:10.016634663 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> --- cc3.txt 2012-07-10 06:59:10.016634663 +0300
>> +++ cc4.txt 2012-07-10 06:59:09.752634749 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> --- cc4.txt 2012-07-10 06:59:09.752634749 +0300
>> +++ cc5.txt 2012-07-10 06:59:09.504634831 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> --- cc5.txt 2012-07-10 06:59:09.504634831 +0300
>> +++ cc6.txt 2012-07-10 06:59:09.240634917 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> --- cc6.txt 2012-07-10 06:59:09.240634917 +0300
>> +++ cc7.txt 2012-07-10 06:59:08.944635015 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> --- cc7.txt 2012-07-10 06:59:08.944635015 +0300
>> +++ cc8.txt 2012-07-10 06:59:08.656635109 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................
>> +btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!..............
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> --- cc8.txt 2012-07-10 06:59:08.656635109 +0300
>> +++ cc9.txt 2012-07-10 06:59:08.328635216 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!..............
>> +btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##..............
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> --- cc9.txt 2012-07-10 06:59:08.328635216 +0300
>> +++ cc10.txt 2012-07-10 06:58:33.180646115 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##..............
>> +btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%..............
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> --- cc10.txt 2012-07-10 06:58:33.180646115 +0300
>> +++ cc11.txt 2012-07-10 06:58:41.340643696 +0300
>> @@ -2,7 +2,7 @@
>> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............
>> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0.......
>> -btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%..............
>> +btrfs: ffff8801cafbacf0: 27 27 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ''..............
>> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
>> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "...............
>> ------------------------------------------------------------
>
> All spinlocks.
>
> I'll go through this again with Jan to see if his eb->debug might give
> any clue here. It might be good to have a look at the contents of the
> pages though, as written in the previous mail.
>
> -Arne
>
>>
>> And since it's of course possible there's something wrong with my
>> modifications, here's the patch I used:
>>
>> ------------------------------------------------------------
>> diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
>> index 7452ecb..aadb82c 100644
>> --- a/fs/btrfs/extent_io.c
>> +++ b/fs/btrfs/extent_io.c
>> @@ -4527,6 +4527,9 @@ void read_extent_buffer(struct extent_buffer *eb, void *dstv,
>> len,
>> eb->debug[0], eb->debug[1], eb->debug[2], eb->debug[3],
>> eb->debug[4], eb->debug[5], eb->debug[6], eb->debug[7]);
>> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb);
>> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true);
>> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb);
>> WARN_ON(1);
>> }
>> WARN_ON(start + len > eb->start + eb->len);
>> diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c
>> index ea81bd4..663d6c4 100644
>> --- a/fs/btrfs/reada.c
>> +++ b/fs/btrfs/reada.c
>> @@ -130,9 +130,13 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb,
>> kref_get(&re->refcnt);
>> spin_unlock(&fs_info->reada_lock);
>>
>> - if (!err && btrfs_csum_tree_block(root, eb))
>> + if (!err && btrfs_csum_tree_block(root, eb)) {
>> printk(KERN_ERR "btrfs: checksum mismatch 4 on %llu\n",
>> eb->start);
>> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb);
>> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true);
>> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb);
>> + }
>>
>> if (!re)
>> return -1;
>> @@ -150,9 +154,14 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb,
>> if (err == 0) {
>> nritems = level ? btrfs_header_nritems(eb) : 0;
>> if (level > BTRFS_MAX_LEVEL ||
>> - nritems > BTRFS_NODEPTRS_PER_BLOCK(root))
>> + nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) {
>> printk(KERN_ERR "btrfs: node seems invalid now. checksum ok = %d\n",
>> btrfs_csum_tree_block(root, eb));
>> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb);
>> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true);
>> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb);
>> +
>> + }
>> generation = btrfs_header_generation(eb);
>> /*
>> * FIXME: currently we just set nritems to 0 if this is a leaf,
>>
>> ------------------------------------------------------------
>>
>> Sami
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2012-07-16 8:20 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-07-02 23:01 btrfs GPF in read_extent_buffer() while scrubbing with kernel 3.4.2 Sami Liedes
2012-07-02 23:08 ` Sami Liedes
2012-07-03 13:11 ` Jan Schmidt
2012-07-03 13:58 ` Sami Liedes
2012-07-03 14:35 ` Jan Schmidt
2012-07-03 22:47 ` Sami Liedes
2012-07-04 0:17 ` Sami Liedes
2012-07-04 11:26 ` Jan Schmidt
2012-07-04 16:03 ` Sami Liedes
2012-07-04 16:38 ` Jan Schmidt
2012-07-04 20:24 ` Sami Liedes
2012-07-05 13:41 ` Jan Schmidt
2012-07-05 23:47 ` Sami Liedes
2012-07-06 10:42 ` Jan Schmidt
2012-07-06 11:50 ` Chris Mason
2012-07-06 14:33 ` Sami Liedes
2012-07-06 14:40 ` Chris Mason
2012-07-06 15:02 ` Jan Schmidt
2012-07-06 15:19 ` Chris Mason
2012-07-06 15:09 ` Jan Schmidt
[not found] ` <20120706195923.GA10687@sli.dy.fi>
2012-07-06 21:41 ` Sami Liedes
2012-07-06 23:44 ` Sami Liedes
2012-07-09 9:05 ` *** GMX Spamverdacht *** " Arne Jansen
2012-07-10 4:16 ` Sami Liedes
2012-07-10 6:05 ` Arne Jansen
2012-07-10 6:57 ` *** GMX Spamverdacht *** " Arne Jansen
2012-07-16 8:20 ` Arne Jansen [this message]
2012-07-16 21:29 ` Sami Liedes
2012-07-28 12:08 ` Sami Liedes
2012-07-28 18:50 ` Sami Liedes
2012-07-03 13:14 ` Sami Liedes
2013-03-27 11:54 ` Stefan Behrens
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5003CECC.2040908@gmx.net \
--to=sensille@gmx.net \
--cc=linux-btrfs@vger.kernel.org \
--cc=list.btrfs@jan-o-sch.net \
--cc=sami.liedes@iki.fi \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).