Node CRC failures in latest mainline git?

Node CRC failures in latest mainline git?

[Bill Gatliff]

Guys:


I pulled from Linus' tree yesterday, and all of a sudden I'm seeing a _ton_ of
these:

JFFS2 notice: (224) jffs2_get_inode_nodes: Node header CRC failed at 0x263a0fc.
{1985,c002,00000046,32fe3396}

An older kernel (three or four weeks ago) continues to work fine on the same board.

The target is a PXA270 with a Spansion 29GL512N.  The kernel was built with
arm-linux-gnueabi-gcc (GCC) 4.2.4 (Debian 4.2.4-3).  There haven't been any
changes in the mtd-related parts of the kernel, other than what's come in from
Linus.

It looks like the most recent mtd-related commit is this:

$ git-log drivers/mtd
commit 9ee08c2df47c10ba624ff05a6c0f2500748bcb69
Merge: eaf0ba5... 3d45955...
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Sat Jul 26 20:30:56 2008 -0700

    Merge git://git.infradead.org/mtd-2.6

    * git://git.infradead.org/mtd-2.6: (57 commits)
      [MTD] [NAND] subpage read feature as a way to increase performance.
      CPUFREQ: S3C24XX NAND driver frequency scaling support.
      [MTD][NAND] au1550nd: remove unused variable
      [MTD] jedec_probe: Fix SST 16-bit chip detection
      [MTD][MTDPART] Fix a division by zero bug
      [MTD][MTDPART] Cleanup and document the erase region handling
      [MTD][MTDPART] Handle most checkpatch findings
      [MTD][MTDPART] Seperate main loop from per-partition code in add_mtd_partiti
      [MTD] physmap: resume already suspended chips on failure to suspend
      [MTD] physmap: Fix suspend/resume/shutdown bugs.
      [MTD] [NOR] Fix -ETIMEO errors in CFI driver
      [MTD] [NAND] fsl_elbc_nand: fix section mismatch with CONFIG_MTD_OF_PARTS=y
      [JFFS2] Use .unlocked_ioctl
      [MTD] Fix const assignment in the MTD command line partitioning driver
      [MTD] [NOR] gen_probe: No debug message when debugging is disabled
      [MTD] [NAND] remove __PPC__ hardcoded address from DiskOnChip drivers
      [MTD] [MAPS] Remove the bast-flash driver.
      [MTD] [NAND] fsl_elbc_nand: ecclayout cleanups
      [MTD] [NAND] fsl_elbc_nand: implement support for flash-based BBT
      [MTD] [NAND] fsl_elbc_nand: fix OOB workability for large page NAND chips
      ...


Were I more competent with git, I would figure out how to bisect down to the
patch that caused the problem.  That's still a bit out of my skillset at the
moment, unfortunately--- I've done bisects successfully in the past, but for
some reason I just can't get git to play along this time...


Thanks!


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: Node CRC failures in latest mainline git?

[Bill Gatliff]

Bill Gatliff wrote:
> 
> An older kernel (three or four weeks ago) continues to work fine on the same board.

I take that back--- a 2.6.24 kernel is issuing the same message, but it still
mounts the filesystem.  The latest git stuff won't.


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: Node CRC failures in latest mainline git?

[David Woodhouse]

On Wed, 2008-07-30 at 23:44 -0500, Bill Gatliff wrote:
> 
> Bill Gatliff wrote:
> > 
> > An older kernel (three or four weeks ago) continues to work fine on the same board.
> 
> I take that back--- a 2.6.24 kernel is issuing the same message, but it still
> mounts the filesystem.  The latest git stuff won't.

Why doesn't it mount? It shouldn't be because of a single bad node
CRC...
 
-- 
dwmw2

Re: Node CRC failures in latest mainline git?

[Bill Gatliff]

David Woodhouse wrote:
> On Wed, 2008-07-30 at 23:44 -0500, Bill Gatliff wrote:
>> Bill Gatliff wrote:
>>> An older kernel (three or four weeks ago) continues to work fine on the same board.
>> I take that back--- a 2.6.24 kernel is issuing the same message, but it still
>> mounts the filesystem.  The latest git stuff won't.
> 
> Why doesn't it mount? It shouldn't be because of a single bad node
> CRC...

Well, it's a bazillion bad node CRCs, approximately.  But I don't know if that's
why it doesn't mount, or not.  I haven't managed to catch the message--- if
there is one.


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: Node CRC failures in latest mainline git?

[David Woodhouse]

On Fri, 2008-08-01 at 16:01 -0500, Bill Gatliff wrote:
> Well, it's a bazillion bad node CRCs, approximately.  But I don't know if that's
> why it doesn't mount, or not.  I haven't managed to catch the message--- if
> there is one.

More than the older kernels? If so, I suspect something broke in your
flash bus setup. Can you make it dump the first node it fails the CRC
on, and compare with what you read under a good kernel?

-- 
dwmw2

Re: Node CRC failures in latest mainline git?

[David Brown]

On Wed, Jul 30, 2008 at 11:25:07PM -0500, Bill Gatliff wrote:

>Were I more competent with git, I would figure out how to bisect down to the
>patch that caused the problem.  That's still a bit out of my skillset at the
>moment, unfortunately--- I've done bisects successfully in the past, but for
>some reason I just can't get git to play along this time...

I just recently asked a question on the git mailing list, and Linus replied
with these instructions for bisecting through a merge.  Basically, you
create a temporary branch where you flatten the merge, using rebase, and
then bisect that.

   # create a test-branch with the 'remote' side of the merge
   git checkout -b test-branch merge^2

   # rebase that remote side on top of the local side
   git rebase merge^

Haven't actually needed to try it, but let me know if it doesn't work, and
I'll figure out the proper incantation.

David Brown

Re: Node CRC failures in latest mainline git?

[Bill Gatliff]

David Brown wrote:
> On Wed, Jul 30, 2008 at 11:25:07PM -0500, Bill Gatliff wrote:
> 
>> Were I more competent with git, I would figure out how to bisect down to the
>> patch that caused the problem.  That's still a bit out of my skillset at the
>> moment, unfortunately--- I've done bisects successfully in the past, but for
>> some reason I just can't get git to play along this time...
> 
> I just recently asked a question on the git mailing list, and Linus replied
> with these instructions for bisecting through a merge.  Basically, you
> create a temporary branch where you flatten the merge, using rebase, and
> then bisect that.
> 
>    # create a test-branch with the 'remote' side of the merge
>    git checkout -b test-branch merge^2
> 
>    # rebase that remote side on top of the local side
>    git rebase merge^


I have no idea what all that would be doing, I still don't grok the syntax for
commits and deltas, and "rebase" in particular always scares me.  :)

Here's what I'm trying:

$ git-checkout -b test-branch 9ee08c2df47c10ba624ff05a6c0f2500748bcb69^2
$ git rebase 9ee08c2df47c10ba624ff05a6c0f2500748bcb69^

It leaves me here:

First, rewinding head to replay your work on top of it...
Applying [MTD] [NOR] Remove cfi_cmdset_0001.c erase suspend fixup typo
Applying [MTD] [NOR] Support for M50FLW080A and M50FLW080B
Applying [MTD] [NOR] Add support for AMD AM29SL800D[BT] NOR flash chips
Applying [MTD] [NOR] Add support for flash chips with ID in bank other than 0
Applying [MTD] [NOR] Add support for Eon EN29SL800B[BT] NOR flash chips
Applying [MTD] [NAND] fix 2 "unused variable" warnings in cafe_nand.c
Applying [MTD] [NAND] excite_nandflash: simplify code using ARRAY_SIZE() macro.
Applying [MTD] use list_for_each_entry() in add_mtd_device()
Applying [MTD] use list_for_each_entry() in del_mtd_device()
Applying [MTD] DataFlash: fix bug - ATMEL AT45DF321D spi flash card fails to be
copied to (v2)
Applying [MTD] DataFlash: use proper types
Applying [MTD] [OneNAND] Check the ECC status first instead of controller
Applying [MTD] [NAND] nandsim: fix size bug
Applying [MTD] [NAND] nandsim: fix overridesize
Applying [MTD] [NAND] nandsim: allow for 64-bit size
Applying MTD/JFFS2: remove CVS keywords
error: drivers/mtd/maps/mtx-1_flash.c: does not exist in index
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merged drivers/mtd/devices/block2mtd.c
Auto-merged drivers/mtd/ftl.c
Auto-merged drivers/mtd/maps/Kconfig
Auto-merged drivers/mtd/maps/Makefile
CONFLICT (delete/modify): drivers/mtd/maps/mtx-1_flash.c deleted in HEAD and
modified in MTD/JFFS2: remove CVS keywords. Version MTD/JFFS2: remove CVS
keywords of drivers/mtd/maps/mtx-1_flash.c left in tree.
Auto-merged drivers/mtd/maps/pcmciamtd.c
Auto-merged drivers/mtd/maps/uclinux.c
Auto-merged drivers/mtd/mtdchar.c
Auto-merged drivers/mtd/redboot.c
Auto-merged include/linux/mtd/map.h
Auto-merged include/linux/mtd/mtd.h
Auto-merged include/linux/mtd/nand.h
Failed to merge in the changes.
Patch failed at 0016.

When you have resolved this problem run "git rebase --continue".
If you would prefer to skip this patch, instead run "git rebase --skip".
To restore the original branch and stop rebasing run "git rebase --abort".



Poking at it more now.  Does it look like I'm on the right track, at least?


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: Node CRC failures in latest mainline git?

[David Brown]

On Thu, Jul 31, 2008 at 03:28:15PM -0500, Bill Gatliff wrote:

>I have no idea what all that would be doing, I still don't grok the syntax for
>commits and deltas, and "rebase" in particular always scares me.  :)

>Here's what I'm trying:
>
>$ git-checkout -b test-branch 9ee08c2df47c10ba624ff05a6c0f2500748bcb69^2
>$ git rebase 9ee08c2df47c10ba624ff05a6c0f2500748bcb69^
>
>CONFLICT (delete/modify): drivers/mtd/maps/mtx-1_flash.c deleted in HEAD and
>modified in MTD/JFFS2: remove CVS keywords. Version MTD/JFFS2: remove CVS
>When you have resolved this problem run "git rebase --continue".
>If you would prefer to skip this patch, instead run "git rebase --skip".
>To restore the original branch and stop rebasing run "git rebase --abort".
>
>Poking at it more now.  Does it look like I'm on the right track, at least?

Yes.  Basically, there are merge conflicts that got resolved at the end.
You're trying to linearize that which means you have to figure them out for
the pieces.  It's kind of a pain, but this particular one isn't all that
hard.

So, I was able to do this rebase as follows:

   $ git rebase 9ee08c2df47^
   ...
   $ git rm drivers/mtd/maps/mtx-1_flash.c
   $ git rebase --continue
   ...

Now we have a real conflict in arch/arm/mach-at91/board-yl-9200.c.

   $ vi arch/arm/mach-at91/board-yl-9200.c

What you want is actually the first one, but with the first line changed
to:

      static struct atmel_nand_data __initdata yl9200_nand_data = {

(with atmel_nand_data, instead of at91_nand_data).

   $ git add arch/arm/mach-at91/board-yl-9200.c
   $ git rebase --continue

As a sanity check, I can compare the end result:

   $ git diff HEAD..9ee08c2df47c10ba624ff05a6c0f2500748bcb69

and see that at least the end result is the same.

Now
    $ git bisect start
    $ git bisect bad
    $ git bisect good 9ee08c2df47c10ba624ff05a6c0f2500748bcb69^
    Bisecting: 28 revisions left to test after this

and that should help you find the problem.  No guarantee that each
intermediate commit is quite right.  You can either fix them, and then
continue, or 'git bisect skip' ones that don't work, but you might miss the
bad one if it just happened to not compile.

David

Re: Node CRC failures in latest mainline git?

[Bill Gatliff]

David Brown wrote:
> Yes.  Basically, there are merge conflicts that got resolved at the end.
> You're trying to linearize that which means you have to figure them out for
> the pieces.  It's kind of a pain, but this particular one isn't all that
> hard.

Yea, I got through that pretty easily once I figured out what it was telling me.
 Git is so smart, it's scary.  :)

BUT, I'm not confident now that the problem isn't that I simply corrupted my
filesystem: I'm getting error messages (but it still mounts) on .24 and .23.  So
now I've got to find the time to generate a new one, and then test that.

Don't hold your breath, I have a business trip to a destination 12 timezones
away coming up.  I'll try to work it in...

I do appreciate the help, though.  I'll let you know as soon as I have something
credible to report.


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com