UBIFS corruption after software upgrade

[Russell Zuck <rzuck@cincinnatitechnologies.com>]

[-- Attachment #1: Type: text/plain, Size: 3991 bytes --]

I am attempting to troubleshoot an issue where the root partition 
(ubi0_0) of my embedded platform becomes corrupted after a few upgrades 
of our system software.

Upgrade Process:
1. upgrade daemon checks for upgrades, from a file server, periodically.
2. If available, the upgrade daemon retrieves the upgrade package (just 
a tarball) and unpacks it.  The tarball contains several install 
scripts, a version file, and another tarball that contains all of our 
system software, required config files, and lib dependencies.
3. The upgrade daemon is triggered to deploy the upgrade package.
4. Deploy
	a. run pre-install script
	b. run install script. This just unpacks the system software tarball 
onto the existing root filesystem.
	c. run the post-install script
	d. cleanup the upgrade staging directory
	e. reboot

Upgrade Process Variations:
1. Terminate all system software before deploying the upgrade package
2. sync after unpacking the system software package
3. sync again before rebooting

I have attached a verbose log of the console output during the upgrades. 
  After the 2nd system reboot, UBIFS is throwing invalid inode errors.

Please see the section that follows for system details.  I have not yet 
attempted to reproduce this issue using nandsim nor do I have a simple 
reproducer for the problem yet.  Please provide some ideas about how to 
simulate the reboot using nandsim as I think the reboot portion of this 
scenario will be critical to reproducing the issue.

System Information
++++++++++++++++++
Kernel Base: 2.6.35

UBIFS Patches: most recent patches applied from 
git://git.infradead.org/~dedekind/ubifs-v2.6.35.git 3 days ago.

MTD Tests: all test run without error with the exception of 
mtd_torturetest which I didn't run

UBIFS extra self-checks: I mounted the debugfs but don't know how to 
make use of of the self-checks.

UBIFS debug prints: I enabled UBI and UBIFS verbose debugging

root filesystem: ubi0_0, the mtd3 partition, is mounted with -o sync

mtdinfo:
root@ctc-imx51 ~$ mtdinfo -a
Count of MTD devices:           4
Present MTD devices:            mtd0, mtd1, mtd2, mtd3
Sysfs interface supported:      yes

mtd0
Name:                           nor_flash
Type:                           dataflash
Eraseblock size:                512 bytes
Amount of eraseblocks:          8192 (4194304 bytes, 4.0 MiB)
Minimum input/output unit size: 512 bytes
Sub-page size:                  512 bytes
Character device major/minor:   90:0
Bad blocks are allowed:         false
Device is writable:             true

mtd1
Name:                           nand.bootloader
Type:                           nand
Eraseblock size:                131072 bytes, 128.0 KiB
Amount of eraseblocks:          8 (1048576 bytes, 1024.0 KiB)
Minimum input/output unit size: 2048 bytes
Sub-page size:                  512 bytes
OOB size:                       64 bytes
Character device major/minor:   90:2
Bad blocks are allowed:         true
Device is writable:             true

mtd2
Name:                           nand.kernel
Type:                           nand
Eraseblock size:                131072 bytes, 128.0 KiB
Amount of eraseblocks:          24 (3145728 bytes, 3.0 MiB)
Minimum input/output unit size: 2048 bytes
Sub-page size:                  512 bytes
OOB size:                       64 bytes
Character device major/minor:   90:4
Bad blocks are allowed:         true
Device is writable:             true

mtd3
Name:                           nand.rootfs
Type:                           nand
Eraseblock size:                131072 bytes, 128.0 KiB
Amount of eraseblocks:          4064 (532676608 bytes, 508.0 MiB)
Minimum input/output unit size: 2048 bytes
Sub-page size:                  512 bytes
OOB size:                       64 bytes
Character device major/minor:   90:6
Bad blocks are allowed:         true
Device is writable:             true

Any assistance you can provide with this issue will be greatly appreciated.

Best regards,

Russell Zuck

[-- Attachment #2: toggle_upgrade_post_patch_verbose.log.tar.gz --]
[-- Type: application/gzip, Size: 147730 bytes --]