From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout-de.gmx.net ([213.165.64.22]:52939 "HELO mailout-de.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with SMTP id S1751410Ab2GPIUb (ORCPT ); Mon, 16 Jul 2012 04:20:31 -0400 Message-ID: <5003CECC.2040908@gmx.net> Date: Mon, 16 Jul 2012 10:20:28 +0200 From: Arne Jansen MIME-Version: 1.0 To: linux-btrfs@vger.kernel.org, sami.liedes@iki.fi CC: Jan Schmidt Subject: Re: btrfs GPF in read_extent_buffer() while scrubbing with kernel 3.4.2 References: <20120704160340.GA9770@sli.dy.fi> <4FF47168.9010405@jan-o-sch.net> <20120704202452.GA8996@sli.dy.fi> <4FF5998D.8020205@jan-o-sch.net> <20120705234739.GA9736@sli.dy.fi> <4FF6C12A.10305@jan-o-sch.net> <20120706143350.GA10427@sli.dy.fi> <4FF6FF8C.4010007@jan-o-sch.net> <20120706234405.GB8933@sli.dy.fi> <4FFA9EEB.7060008@gmx.net> <20120710041625.GA8909@sli.dy.fi> <4FFBD24E.2050902@gmx.net> In-Reply-To: <4FFBD24E.2050902@gmx.net> Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-btrfs-owner@vger.kernel.org List-ID: Any news on this? I you give me some hints, I can try to reproduce it here. -Arne On 10.07.2012 08:57, Arne Jansen wrote: > On 10.07.2012 06:16, Sami Liedes wrote: >> On Mon, Jul 09, 2012 at 11:05:47AM +0200, Arne Jansen wrote: >>>> * Just before the crash: >>>> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2261163409408, level 100, generation 4412718571037421157, nritems 538968254. len param 17. debug 2/989/538968254/4412718571037421157/0x0/0/0x0/0x0 >>>> >>> >>> At a first glance: the generation converted to ascii is: "ent() ==", >>> so someone is patching the memory with ascii text, possibly C source. >>> It might be interesting to dump the full contents of the eb, to get >>> a clue on the source of the data. >> >> I changed the code to dump the contents of the eb struct at the point >> where that error (btrfs: invalid parameters...) is printed, at the >> "checksum mismatch 4" site and at the "node seems invalid now" site. >> Now I have a big log of 1795 corrupted ebs. So far nothing that looks >> remotely like ascii text, though. But I have two different versions of >> the eb that caused that warning, a less corrupted one and a more >> corrupted one: >> >> ------------------------------------------------------------ >> btrfs: --- start eb contents at ffff8801b13cc4c8 --- >> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ >> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... >> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... >> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... >> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. >> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. >> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... >> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... >> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... >> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. >> btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ > > this one looks good so far. > >> btrfs: --- end eb contents at ffff8801b13cc4c8 --- >> btrfs: dm-6 checksum verify failed on 2239404212224 wanted B5F632BC found 3579FB59 level 160 >> btrfs: node seems invalid now. checksum ok = 1 >> btrfs: --- start eb contents at ffff8801b13cc4c8 --- >> [... identical dump to above ...] >> btrfs: --- end eb contents at ffff8801b13cc4c8 --- >> btrfs: invalid parameters for read_extent_buffer: start (32771) > eb->len (32768). eb start is 2239404212224, level 160, generation 4716553384049587249, nritems 295705211. len param 17. debug 2/989/295705211/4716553384049587249/0x0/0/0x0/0x0 >> btrfs: --- start eb contents at ffff8801b13cc4c8 --- >> btrfs: ffff8801b13cc4c8: 00 80 e4 66 09 02 00 00 00 80 00 00 00 00 00 00 ...f............ >> btrfs: ffff8801b13cc4d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc4e8: 20 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff .......0....... >> btrfs: ffff8801b13cc4f8: 02 02 00 00 03 00 00 00 06 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc508: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc518: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc528: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc538: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc548: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc558: 58 c5 3c b1 01 88 ff ff 58 c5 3c b1 01 88 ff ff X.<.....X.<..... >> btrfs: ffff8801b13cc568: 00 00 00 00 00 00 00 00 70 c5 3c b1 01 88 ff ff ........p.<..... >> btrfs: ffff8801b13cc578: 70 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 p.<............. >> btrfs: ffff8801b13cc588: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc598: 80 5f 9a 06 00 ea ff ff 00 86 9b 06 00 ea ff ff ._.............. >> btrfs: ffff8801b13cc5a8: 40 4c 9a 06 00 ea ff ff 80 66 9a 06 00 ea ff ff @L.......f...... >> btrfs: ffff8801b13cc5b8: 80 eb 9b 06 00 ea ff ff 40 05 a2 06 00 ea ff ff ........@....... >> btrfs: ffff8801b13cc5c8: 40 e1 9b 06 00 ea ff ff 80 c4 9c 06 00 ea ff ff @............... >> btrfs: ffff8801b13cc5d8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc5e8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc5f8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc608: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. >> btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... >> btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q> btrfs: ffff8801b13cc648: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801b13cc658: 00 00 00 00 00 00 00 00 ........ >> btrfs: --- end eb contents at ffff8801b13cc4c8 --- >> ------------[ cut here ]------------ >> WARNING: at fs/btrfs/extent_io.c:4533 read_extent_buffer+0x1ef/0x220 [btrfs]() >> Hardware name: System Product Name >> Modules linked in: [last unloaded: scsi_wait_scan] >> Pid: 25829, comm: btrfs-endio-met Tainted: G W 3.4.4+btrfsdebug3+ #4 >> [...] >> ------------------------------------------------------------ >> >> Here's the difference (lines reordered from diff to make comparing >> easier): >> >> ------------------------------------------------------------ >> -btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 00 00 00 00 00 00 00 00 ..<............. >> +btrfs: ffff8801b13cc618: 98 c5 3c b1 01 88 ff ff 02 00 00 00 00 00 00 00 ..<............. >> -btrfs: ffff8801b13cc628: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801b13cc628: dd 03 00 00 00 00 00 00 7b 1a a0 11 00 00 00 00 ........{....... >> -btrfs: ffff8801b13cc638: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801b13cc638: 31 34 71 3c 50 90 74 41 00 00 00 00 00 00 00 00 14q> ------------------------------------------------------------ > > The diff is just the eb->debug[] Jan added. > >> >> If there's one pattern that catches eye in the dumps, it's that in >> many places where the same eb is dumped multiple times due to multiple >> "checksum mismatch 4"s, there are bytes at offsets 0x30 and 0x31 that >> always seem to have the same value and both separately increase >> between the dumps, usually(?) by two. Then that might be normal, I >> haven't looked at what should be at a struct eb :) There are some >> instances where they wrap, for example >> >> f8 f8 -> fa fa -> fc fc -> fe fe -> 00 00 -> 02 02 -> ..., > > this is the spinlock to protect the refcnt, so some reference taking/releasing > has been going on. > >> >> with no other changes visible in the corrupted ebs. But that is by no >> means the only change in the struct eb contents that can be observed, >> only the most obvious pattern. >> >> For example, here's a 11-long string of eb contents. First the dmesg: >> >> ------------------------------------------------------------ >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 1] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 2] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 3] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 4] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 5] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 6] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 7] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 8] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 9] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 10] >> btrfs: dm-6 checksum verify failed on 1127200522240 wanted 7712C045 found C593E2D6 level 2 >> btrfs: checksum mismatch 4 on 1127200522240 >> [DUMP 11] >> ------------------------------------------------------------ >> >> Where DUMP 1 contents (file cc1.txt if you'd want to apply the diffs >> below) is >> >> ------------------------------------------------------------ >> btrfs: --- start eb contents at ffff8801cafbacc0 --- >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> btrfs: ffff8801cafbad30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad40: 00 00 10 00 00 00 00 00 04 04 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad50: 50 ad fb ca 01 88 ff ff 50 ad fb ca 01 88 ff ff P.......P....... >> btrfs: ffff8801cafbad60: 02 02 00 00 00 00 00 00 68 ad fb ca 01 88 ff ff ........h....... >> btrfs: ffff8801cafbad70: 68 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 h............... >> btrfs: ffff8801cafbad80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad90: 00 a2 20 08 00 ea ff ff c0 2a f4 07 00 ea ff ff .. ......*...... >> btrfs: ffff8801cafbada0: 00 ac 89 07 00 ea ff ff c0 42 79 07 00 ea ff ff .........By..... >> btrfs: ffff8801cafbadb0: 80 af 77 07 00 ea ff ff 40 37 49 08 00 ea ff ff ..w.....@7I..... >> btrfs: ffff8801cafbadc0: 80 6d f3 07 00 ea ff ff c0 a7 e8 07 00 ea ff ff .m.............. >> btrfs: ffff8801cafbadd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ >> btrfs: --- end eb contents at ffff8801cafbacc0 --- >> ------------------------------------------------------------ >> >> And the successive diffs are >> >> ------------------------------------------------------------ >> --- cc1.txt 2012-07-10 06:57:21.564665577 +0300 >> +++ cc2.txt 2012-07-10 06:59:10.272634578 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 13 13 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ > > the spinlock > >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> @@ -20,9 +20,9 @@ >> btrfs: ffff8801cafbade0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbadf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> -btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 00 00 00 00 00 00 00 00 ................ >> -btrfs: ffff8801cafbae20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> -btrfs: ffff8801cafbae30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbae10: 90 ad fb ca 01 88 ff ff 01 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbae20: 4d 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 M.......N....... >> +btrfs: ffff8801cafbae30: 07 78 02 00 00 00 00 00 00 00 00 00 00 00 00 00 .x.............. > > and eb->debug > >> btrfs: ffff8801cafbae40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbae50: 00 00 00 00 00 00 00 00 ........ >> btrfs: --- end eb contents at ffff8801cafbacc0 --- >> --- cc2.txt 2012-07-10 06:59:10.272634578 +0300 >> +++ cc3.txt 2012-07-10 06:59:10.016634663 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 15 15 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc3.txt 2012-07-10 06:59:10.016634663 +0300 >> +++ cc4.txt 2012-07-10 06:59:09.752634749 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 17 17 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc4.txt 2012-07-10 06:59:09.752634749 +0300 >> +++ cc5.txt 2012-07-10 06:59:09.504634831 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 19 19 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc5.txt 2012-07-10 06:59:09.504634831 +0300 >> +++ cc6.txt 2012-07-10 06:59:09.240634917 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 1b 1b 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc6.txt 2012-07-10 06:59:09.240634917 +0300 >> +++ cc7.txt 2012-07-10 06:59:08.944635015 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 1d 1d 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc7.txt 2012-07-10 06:59:08.944635015 +0300 >> +++ cc8.txt 2012-07-10 06:59:08.656635109 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 1f 1f 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ................ >> +btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc8.txt 2012-07-10 06:59:08.656635109 +0300 >> +++ cc9.txt 2012-07-10 06:59:08.328635216 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 21 21 00 00 03 00 00 00 00 00 00 00 00 00 00 00 !!.............. >> +btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc9.txt 2012-07-10 06:59:08.328635216 +0300 >> +++ cc10.txt 2012-07-10 06:58:33.180646115 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 23 23 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ##.............. >> +btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> --- cc10.txt 2012-07-10 06:58:33.180646115 +0300 >> +++ cc11.txt 2012-07-10 06:58:41.340643696 +0300 >> @@ -2,7 +2,7 @@ >> btrfs: ffff8801cafbacc0: 00 00 63 72 06 01 00 00 00 80 00 00 00 00 00 00 ..cr............ >> btrfs: ffff8801cafbacd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbace0: 35 00 00 00 00 00 00 00 30 00 d7 10 02 88 ff ff 5.......0....... >> -btrfs: ffff8801cafbacf0: 25 25 00 00 03 00 00 00 00 00 00 00 00 00 00 00 %%.............. >> +btrfs: ffff8801cafbacf0: 27 27 00 00 03 00 00 00 00 00 00 00 00 00 00 00 ''.............. >> btrfs: ffff8801cafbad00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ >> btrfs: ffff8801cafbad20: 22 1c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "............... >> ------------------------------------------------------------ > > All spinlocks. > > I'll go through this again with Jan to see if his eb->debug might give > any clue here. It might be good to have a look at the contents of the > pages though, as written in the previous mail. > > -Arne > >> >> And since it's of course possible there's something wrong with my >> modifications, here's the patch I used: >> >> ------------------------------------------------------------ >> diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c >> index 7452ecb..aadb82c 100644 >> --- a/fs/btrfs/extent_io.c >> +++ b/fs/btrfs/extent_io.c >> @@ -4527,6 +4527,9 @@ void read_extent_buffer(struct extent_buffer *eb, void *dstv, >> len, >> eb->debug[0], eb->debug[1], eb->debug[2], eb->debug[3], >> eb->debug[4], eb->debug[5], eb->debug[6], eb->debug[7]); >> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); >> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); >> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); >> WARN_ON(1); >> } >> WARN_ON(start + len > eb->start + eb->len); >> diff --git a/fs/btrfs/reada.c b/fs/btrfs/reada.c >> index ea81bd4..663d6c4 100644 >> --- a/fs/btrfs/reada.c >> +++ b/fs/btrfs/reada.c >> @@ -130,9 +130,13 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, >> kref_get(&re->refcnt); >> spin_unlock(&fs_info->reada_lock); >> >> - if (!err && btrfs_csum_tree_block(root, eb)) >> + if (!err && btrfs_csum_tree_block(root, eb)) { >> printk(KERN_ERR "btrfs: checksum mismatch 4 on %llu\n", >> eb->start); >> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); >> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); >> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); >> + } >> >> if (!re) >> return -1; >> @@ -150,9 +154,14 @@ static int __readahead_hook(struct btrfs_root *root, struct extent_buffer *eb, >> if (err == 0) { >> nritems = level ? btrfs_header_nritems(eb) : 0; >> if (level > BTRFS_MAX_LEVEL || >> - nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) >> + nritems > BTRFS_NODEPTRS_PER_BLOCK(root)) { >> printk(KERN_ERR "btrfs: node seems invalid now. checksum ok = %d\n", >> btrfs_csum_tree_block(root, eb)); >> + printk(KERN_ERR "btrfs: --- start eb contents at %p ---\n", eb); >> + print_hex_dump(KERN_ERR, "btrfs: ", DUMP_PREFIX_ADDRESS, 16, 1, eb, sizeof(*eb), true); >> + printk(KERN_ERR "btrfs: --- end eb contents at %p ---\n", eb); >> + >> + } >> generation = btrfs_header_generation(eb); >> /* >> * FIXME: currently we just set nritems to 0 if this is a leaf, >> >> ------------------------------------------------------------ >> >> Sami > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html