'ubi clone' from Live NAND.

'ubi clone' from Live NAND.

[Bill Pringlemeir]

I have a 'ubinize' image that I have created on a PC and I have a 'raw'
copy of the NAND flash on a 'Live' Linux distribution; the target was
cleanly unmounted.  I use nandsim, with all my real FLASH id bytes and
setup partitions as per the target.  Here is the sequence I follow on my
PC.

 modprobe nandsim first_id_byte=0x2c second_id_byte=0xda third_id_byte=0x90 fourth_id_byte=0x95 parts=2,64,64
 flash_erase /dev/mtd3 0 0
 ubiformat /dev/mtd3 -f rootfs.ubi
 modprobe ubi mtd=3
 mount -t ubifs /dev/ubi0_3 /mnt/ubifs

If 'rootfs.ubi' is the 'ubinize' image, then I can mount and see the
file system on my PC.  However, if the 'rootfs.ubi' is a 'dump' of the
NAND flash, I fail with the following 'dmesg',

[527676.968929] UBI: background thread "ubi_bgt0d" started, PID 5253
[527681.522291] UBIFS error (pid 5261): ubifs_read_node: bad node type (255 but expected 6)
[527681.522295] UBIFS error (pid 5261): ubifs_read_node: bad node at LEB 0:0, LEB mapping status 0
[527681.522296] Not a node, first 24 bytes:
[527681.522298] 00000000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ........................
[527681.522301] Pid: 5261, comm: mount Tainted: P        W  O 3.5.0-25-generic #39-Ubuntu
[527681.522302] Call Trace:
[527681.522309]  [<ffffffffa0e8cf25>] ubifs_read_node+0x255/0x2d0 [ubifs]
[527681.522314]  [<ffffffffa0e8a320>] ? ubifs_read_sb_node+0x30/0x90 [ubifs]
[527681.522318]  [<ffffffffa0e8a343>] ubifs_read_sb_node+0x53/0x90 [ubifs]
[527681.522321]  [<ffffffffa0e8a41a>] ubifs_read_superblock+0x3a/0x730 [ubifs]
[527681.522325]  [<ffffffffa0e87a96>] ? mount_ubifs+0x3f6/0x15a0 [ubifs]
[527681.522328]  [<ffffffffa0e87b87>] mount_ubifs+0x4e7/0x15a0 [ubifs]
[527681.522331]  [<ffffffffa0e897ac>] ubifs_mount+0x4ac/0x730 [ubifs]
[527681.522335]  [<ffffffff81185b63>] mount_fs+0x43/0x1b0
[527681.522337]  [<ffffffff8119ef63>] ? find_filesystem+0x63/0x80
[527681.522339]  [<ffffffff8119fe34>] vfs_kern_mount+0x74/0x110
[527681.522341]  [<ffffffff811a07a4>] do_kern_mount+0x54/0x110
[527681.522343]  [<ffffffff811a20da>] do_mount+0x26a/0x890
[527681.522345]  [<ffffffff8113de2b>] ? strndup_user+0x5b/0x80
[527681.522347]  [<ffffffff811a284d>] sys_mount+0x8d/0xe0
[527681.522350]  [<ffffffff8168c969>] system_call_fastpath+0x16/0x1b

Also, the mount command fails with,

mount: wrong fs type, bad option, bad superblock on /dev/ubi0_3,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

Do I have to skip bad block when 'dumping' the file from the device?  Is
there some incompatibilities between my PC ubifs and the target?  Should
I be using some tool under Linux that I have missed?  The 'dumping' is
simply a read of 'offset' and 'size'.  Should I be doing something with
OOB data?  Maybe a 64bit/32bit issue?  I didn't see anyone else with
this type of issue on the web.

Sorry, If I have missed some information.  I don't think I need to
instrument UBI/UbiFs.  It is just something wrong with my process (or it
can't be done for some reason).

tia,
Bill Pringlemeir.

RE: 'ubi clone' from Live NAND.

[Gupta, Pekon]

> If 'rootfs.ubi' is the 'ubinize' image, then I can mount and see the
> file system on my PC.  However, if the 'rootfs.ubi' is a 'dump' of the
> NAND flash, I fail with the following 'dmesg',

[pekon] 
I have tried this, using following steps, and it works well

(1) nanddump   -b -s 0x0  -l <size_of_source_partition>  -f  <image_dump.bin>  <source_partition>
(2) flash_erase   <target_partition>  0 0 
(3) nandwrite -n -r   <target_partition>         <image_dump.bin>  

where,
source_partition= partition on source device from which NAND binary would be dumped. (/dev/mtdX)
target_partition= partition on another device which NAND dumped image would be flashed (/dev/mtdY)


You need to keep few things in mind..
(1) sizeof(target_partion) > sizeof(source_partition), So as to take care of bad-blocks present on target device.

(2) '-n' option in nandwrite tells NAND driver not to calculate ECC while programming nand, as its is already part of 'raw' binary image

(3) You need to  'nanddump' complete source_partition (and not just image-size), because of following reasons
     (a) Image data may be shifted to other offsets due to presence of bad-blocks
     (b) If you are using 'autoresize' option, then UBIFS Volume may expand itself to cover entire free space first mount.
[Refer http://www.linux-mtd.infradead.org/doc/ubi.html#L_autoresize]


with regards, pekon

Re: 'ubi clone' from Live NAND.

[Brian Norris]

As a NAND driver developer and a developer of nanddump / nandwrite, I
have some comments. Note, however, that I have not tried this kind of
UBI clone, so my suggestions are somewhat of educated guesses.

On Thu, Mar 7, 2013 at 4:11 AM, Gupta, Pekon <pekon@ti.com> wrote:
>> If 'rootfs.ubi' is the 'ubinize' image, then I can mount and see the
>> file system on my PC.  However, if the 'rootfs.ubi' is a 'dump' of the
>> NAND flash, I fail with the following 'dmesg',
>
> [pekon]
> I have tried this, using following steps, and it works well
>
> (1) nanddump   -b -s 0x0  -l <size_of_source_partition>  -f  <image_dump.bin>  <source_partition>

FYI, "-b" (or "omitbad") has been dropped from recent nanddump
releases. A similar (but not identical) option is now available as
--bb=skipbad (default). And then, you probably don't want to specify a
-l (length) option, since the length actually refers to the length of
the resulting image, which may vary depending on the number of bad
blocks. So, with recent mtd-utils releases, the equivalent dump would
be:

nanddump --bb=skipbad -s 0x0 -f <image_dump.bin> <source_partition>

or, because many of the provided options are defaults, the following
is also equivalent:

nanddump -f <image_dump.bin> <source_partition>

> (2) flash_erase   <target_partition>  0 0
> (3) nandwrite -n -r   <target_partition>         <image_dump.bin>

This looks likely to cause problems. See my comment below.

> where,
> source_partition= partition on source device from which NAND binary would be dumped. (/dev/mtdX)
> target_partition= partition on another device which NAND dumped image would be flashed (/dev/mtdY)
>
>
> You need to keep few things in mind..
> (1) sizeof(target_partion) > sizeof(source_partition), So as to take care of bad-blocks present on target device.
>
> (2) '-n' option in nandwrite tells NAND driver not to calculate ECC while programming nand, as its is already part of 'raw' binary image

The image dump command above does not contain any ECC information, so
when you write with '-n', you aren't writing *any* ECC information.
This is quite likely to cause your ECC decoder to flag all your data
as uncorrectable. Are you sure this has worked for any length of time?
I would strongly recommend against writing your image in this way.

I'm guessing that your recommendation of '-n' (no ECC) is to solve the
0xFF page problem, documented here?

http://www.linux-mtd.infradead.org/faq/ubifs.html#L_why_ubiformat

Anyway, I don't think you want nandwrite; just use ubiformat.

Brian

Re: 'ubi clone' from Live NAND.

[Brian Norris]

Hi Bill,

I can't help you a lot with the specific errors you see (Artem should
know better).

On Wed, Mar 6, 2013 at 12:08 PM, Bill Pringlemeir <bpringle@sympatico.ca> wrote:
>
> I have a 'ubinize' image that I have created on a PC and I have a 'raw'
> copy of the NAND flash on a 'Live' Linux distribution; the target was
> cleanly unmounted.  I use nandsim, with all my real FLASH id bytes and
> setup partitions as per the target.  Here is the sequence I follow on my
> PC.
>
>  modprobe nandsim first_id_byte=0x2c second_id_byte=0xda third_id_byte=0x90 fourth_id_byte=0x95 parts=2,64,64
>  flash_erase /dev/mtd3 0 0
>  ubiformat /dev/mtd3 -f rootfs.ubi
>  modprobe ubi mtd=3
>  mount -t ubifs /dev/ubi0_3 /mnt/ubifs
>
> If 'rootfs.ubi' is the 'ubinize' image, then I can mount and see the
> file system on my PC.  However, if the 'rootfs.ubi' is a 'dump' of the
> NAND flash, I fail with the following 'dmesg',
>
> [527676.968929] UBI: background thread "ubi_bgt0d" started, PID 5253
> [527681.522291] UBIFS error (pid 5261): ubifs_read_node: bad node type (255 but expected 6)
> [527681.522295] UBIFS error (pid 5261): ubifs_read_node: bad node at LEB 0:0, LEB mapping status 0
> [527681.522296] Not a node, first 24 bytes:
> [527681.522298] 00000000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ........................
> [527681.522301] Pid: 5261, comm: mount Tainted: P        W  O 3.5.0-25-generic #39-Ubuntu
> [527681.522302] Call Trace:
> [527681.522309]  [<ffffffffa0e8cf25>] ubifs_read_node+0x255/0x2d0 [ubifs]
> [527681.522314]  [<ffffffffa0e8a320>] ? ubifs_read_sb_node+0x30/0x90 [ubifs]
> [527681.522318]  [<ffffffffa0e8a343>] ubifs_read_sb_node+0x53/0x90 [ubifs]
> [527681.522321]  [<ffffffffa0e8a41a>] ubifs_read_superblock+0x3a/0x730 [ubifs]
> [527681.522325]  [<ffffffffa0e87a96>] ? mount_ubifs+0x3f6/0x15a0 [ubifs]
> [527681.522328]  [<ffffffffa0e87b87>] mount_ubifs+0x4e7/0x15a0 [ubifs]
> [527681.522331]  [<ffffffffa0e897ac>] ubifs_mount+0x4ac/0x730 [ubifs]
> [527681.522335]  [<ffffffff81185b63>] mount_fs+0x43/0x1b0
> [527681.522337]  [<ffffffff8119ef63>] ? find_filesystem+0x63/0x80
> [527681.522339]  [<ffffffff8119fe34>] vfs_kern_mount+0x74/0x110
> [527681.522341]  [<ffffffff811a07a4>] do_kern_mount+0x54/0x110
> [527681.522343]  [<ffffffff811a20da>] do_mount+0x26a/0x890
> [527681.522345]  [<ffffffff8113de2b>] ? strndup_user+0x5b/0x80
> [527681.522347]  [<ffffffff811a284d>] sys_mount+0x8d/0xe0
> [527681.522350]  [<ffffffff8168c969>] system_call_fastpath+0x16/0x1b
>
> Also, the mount command fails with,
>
> mount: wrong fs type, bad option, bad superblock on /dev/ubi0_3,
>        missing codepage or helper program, or other error
>        In some cases useful info is found in syslog - try
>        dmesg | tail  or so
>
> Do I have to skip bad block when 'dumping' the file from the device?

Yes, you have to skip bad blocks when dumping from the device. On
older versions, nanddump defaulted to filling bad blocks with 0xFF
data, which is probably not what you want. On these same versions, the
"-b" option (recommended by Pekon on the other thread) will omit the
bad blocks. On newer versions, you want to use "--bb=skipbad".

What version of nanddump do you have? (Run `nanddump --version' to
check) And what is your full nanddump command?

> Is
> there some incompatibilities between my PC ubifs and the target?  Should
> I be using some tool under Linux that I have missed?  The 'dumping' is
> simply a read of 'offset' and 'size'.  Should I be doing something with
> OOB data?  Maybe a 64bit/32bit issue?  I didn't see anyone else with
> this type of issue on the web.
>
> Sorry, If I have missed some information.  I don't think I need to
> instrument UBI/UbiFs.  It is just something wrong with my process (or it
> can't be done for some reason).

Brian

Re: 'ubi clone' from Live NAND.

[pekon]

Brian,

Had few queries below..

On 3/10/2013 12:08 PM, Brian Norris wrote:
> As a NAND driver developer and a developer of nanddump / nandwrite, I
> have some comments. Note, however, that I have not tried this kind of
> UBI clone, so my suggestions are somewhat of educated guesses.
>
> On Thu, Mar 7, 2013 at 4:11 AM, Gupta, Pekon <pekon@ti.com> wrote:
>>> If 'rootfs.ubi' is the 'ubinize' image, then I can mount and see the
>>> file system on my PC.  However, if the 'rootfs.ubi' is a 'dump' of the
>>> NAND flash, I fail with the following 'dmesg',
>> [pekon]
>> I have tried this, using following steps, and it works well
>>
>> (1) nanddump   -b -s 0x0  -l <size_of_source_partition>  -f  <image_dump.bin>  <source_partition>
> FYI, "-b" (or "omitbad") has been dropped from recent nanddump
> releases. A similar (but not identical) option is now available as
> --bb=skipbad (default). And then, you probably don't want to specify a
> -l (length) option, since the length actually refers to the length of
> the resulting image, which may vary depending on the number of bad
> blocks. So, with recent mtd-utils releases, the equivalent dump would
> be:
>
> nanddump --bb=skipbad -s 0x0 -f <image_dump.bin> <source_partition>
>
> or, because many of the provided options are defaults, the following
> is also equivalent:
>
> nanddump -f <image_dump.bin> <source_partition>
[Pekon]: In above example, you have not specified any length field.
But, If only subset of a partition needs to be dumped, then how should 
the 'length of data' to be dumped be specified ? (assuming there are 
bad-blocks).
Hoping '-l' option is not deprecated in newer versions of nanddump ?

>> (2) flash_erase   <target_partition>  0 0
>> (3) nandwrite -n -r   <target_partition>         <image_dump.bin>
> This looks likely to cause problems. See my comment below.
>
>> where,
>> source_partition= partition on source device from which NAND binary would be dumped. (/dev/mtdX)
>> target_partition= partition on another device which NAND dumped image would be flashed (/dev/mtdY)
>>
>>
>> You need to keep few things in mind..
>> (1) sizeof(target_partion) > sizeof(source_partition), So as to take care of bad-blocks present on target device.
>>
>> (2) '-n' option in nandwrite tells NAND driver not to calculate ECC while programming nand, as its is already part of 'raw' binary image
> The image dump command above does not contain any ECC information,
[Pekon]: While doing above, I was using 'nanddump -version 1.30' and 
'nandwrite revision 1.32' on kernel 2.6.37+, with ECC_HW scheme.
The <image_dump.bin> is actually the binary data dumped using nanddump 
in previous step. So it should contain OOB area bytes also ?


> so
> when you write with '-n', you aren't writing *any* ECC information.
> This is quite likely to cause your ECC decoder to flag all your data
> as uncorrectable. Are you sure this has worked for any length of time?
> I would strongly recommend against writing your image in this way.
[Pekon]: Also, while writing back the <image_dump.bin> I was using  raw 
mode.
-> '-r' (raw) was used to specify that input file contains OOB bytes 
along with MAIN area data.
-> '-n' was used to suppress ECC calculations while writing the data.
However, even I was not clear on, why do we need to explicitely give 
'-n' (suppress ECC) option along '-r' (raw) switch. It should be 
implicit, isn't it ?
Can you please also clarify on usage of -r (raw) option ?

> I'm guessing that your recommendation of '-n' (no ECC) is to solve the
> 0xFF page problem, documented here?
>
> http://www.linux-mtd.infradead.org/faq/ubifs.html#L_why_ubiformat
[Pekon]: Thanks for pointing this out, i missed it earlier.
>
> Anyway, I don't think you want nandwrite; just use ubiformat.
[Pekon]: I could not use 'ubiformat' because the <image_dump.bin> was 
not a standard UBI image. It was a binary data dumped using nanddump as 
given in previous steps.
>
> Brian
with regards, pekon

Re: 'ubi clone' from Live NAND.

[Artem Bityutskiy]

On Wed, 2013-03-06 at 15:08 -0500, Bill Pringlemeir wrote:
> I have a 'ubinize' image that I have created on a PC and I have a 'raw'
> copy of the NAND flash on a 'Live' Linux distribution; the target was
> cleanly unmounted.  I use nandsim, with all my real FLASH id bytes and
> setup partitions as per the target.  Here is the sequence I follow on my
> PC.
> 
>  modprobe nandsim first_id_byte=0x2c second_id_byte=0xda third_id_byte=0x90 fourth_id_byte=0x95 parts=2,64,64
>  flash_erase /dev/mtd3 0 0
>  ubiformat /dev/mtd3 -f rootfs.ubi
>  modprobe ubi mtd=3
>  mount -t ubifs /dev/ubi0_3 /mnt/ubifs

Does it work if you do the same, but avoid flashing your 'rootfs.ubi'.
Just run 'ubiformat /dev/mtd3'. Can you mount it then, successfully do
the I/O, unmount, and remount? If that works fine, you probably did not
create 'rootfs.ubi' correctly. Double-check that you specified correct
eraseblock size, correct nand page size and subpage size.

> If 'rootfs.ubi' is the 'ubinize' image, then I can mount and see the
> file system on my PC.  However, if the 'rootfs.ubi' is a 'dump' of the
> NAND flash, I fail with the following 'dmesg',
> 
> [527676.968929] UBI: background thread "ubi_bgt0d" started, PID 5253
> [527681.522291] UBIFS error (pid 5261): ubifs_read_node: bad node type (255 but expected 6)
> [527681.522295] UBIFS error (pid 5261): ubifs_read_node: bad node at LEB 0:0, LEB mapping status 0
> [527681.522296] Not a node, first 24 bytes:
> [527681.522298] 00000000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ........................
> [527681.522301] Pid: 5261, comm: mount Tainted: P        W  O 3.5.0-25-generic #39-Ubuntu
> [527681.522302] Call Trace:
> [527681.522309]  [<ffffffffa0e8cf25>] ubifs_read_node+0x255/0x2d0 [ubifs]
> [527681.522314]  [<ffffffffa0e8a320>] ? ubifs_read_sb_node+0x30/0x90 [ubifs]
> [527681.522318]  [<ffffffffa0e8a343>] ubifs_read_sb_node+0x53/0x90 [ubifs]
> [527681.522321]  [<ffffffffa0e8a41a>] ubifs_read_superblock+0x3a/0x730 [ubifs]
> [527681.522325]  [<ffffffffa0e87a96>] ? mount_ubifs+0x3f6/0x15a0 [ubifs]
> [527681.522328]  [<ffffffffa0e87b87>] mount_ubifs+0x4e7/0x15a0 [ubifs]
> [527681.522331]  [<ffffffffa0e897ac>] ubifs_mount+0x4ac/0x730 [ubifs]
> [527681.522335]  [<ffffffff81185b63>] mount_fs+0x43/0x1b0
> [527681.522337]  [<ffffffff8119ef63>] ? find_filesystem+0x63/0x80
> [527681.522339]  [<ffffffff8119fe34>] vfs_kern_mount+0x74/0x110
> [527681.522341]  [<ffffffff811a07a4>] do_kern_mount+0x54/0x110
> [527681.522343]  [<ffffffff811a20da>] do_mount+0x26a/0x890
> [527681.522345]  [<ffffffff8113de2b>] ? strndup_user+0x5b/0x80
> [527681.522347]  [<ffffffff811a284d>] sys_mount+0x8d/0xe0
> [527681.522350]  [<ffffffff8168c969>] system_call_fastpath+0x16/0x1b

This may be a nandsim bug, but looks more like you flashed incorrect
image. You may try to do what I suggested above, and if that works,
unmount all, dump the contents of mtd3 using nanddump, and then flash it
back as you did above, and verify if that works.

> Also, the mount command fails with,
> 
> mount: wrong fs type, bad option, bad superblock on /dev/ubi0_3,
>        missing codepage or helper program, or other error
>        In some cases useful info is found in syslog - try
>        dmesg | tail  or so
> 
> Do I have to skip bad block when 'dumping' the file from the device?

Yes, although in case of nandsim there are no bad blocks.
>   Is
> there some incompatibilities between my PC ubifs and the target? 

Should not be, unless you enabled the fastmap UBI feature, in which case
you may have issues.

-- 
Best Regards,
Artem Bityutskiy

Re: 'ubi clone' from Live NAND.

[Bill Pringlemeir]

On Wed, 2013-03-06 at 15:08 -0500, Bill Pringlemeir wrote:
>> I have a 'ubinize' image that I have created on a PC and I have a
>> 'raw' copy of the NAND flash on a 'Live' Linux distribution; the
>> target was cleanly unmounted.  I use nandsim, with all my real FLASH
>> id bytes and setup partitions as per the target.  Here is the
>> sequence I follow on my PC.

>>  modprobe nandsim first_id_byte=0x2c second_id_byte=0xda third_id_byte=0x90 fourth_id_byte=0x95 parts=2,64,64
>>  flash_erase /dev/mtd3 0 0
>>  ubiformat /dev/mtd3 -f rootfs.ubi
>>  modprobe ubi mtd=3
>>  mount -t ubifs /dev/ubi0_3 /mnt/ubifs

On 13 Mar 2013, dedekind1@gmail.com wrote:

> Does it work if you do the same, but avoid flashing your 'rootfs.ubi'.
> Just run 'ubiformat /dev/mtd3'. Can you mount it then, successfully do
> the I/O, unmount, and remount? If that works fine, you probably did
> not create 'rootfs.ubi' correctly. Double-check that you specified
> correct eraseblock size, correct nand page size and subpage size.

Thanks, all.  Indeed everything is fine if I don't use the 'improper'
rootfs.ubi.  Pekon Gupta had clued me in to the problem; If he hadn't, I
think Brian would have.  I am not using 'nandump' as I don't want the
file system to be live.  I wanted to use a flash programmer/JTAG device
to get a copy of the flash.

The problem is simply that the binary copy must include the 'entire
flash' and not be limited to a 'df' size because of course UBI does wear
leveling across the whole partition; a benefit of using it that I had
forgot about.

The 'dmesg' was basically as if someone suddenly truncated the 'UbiFs'
file system.

Sorry to trouble you.  I hope the information is useful to someone in
the future.  Btw, it doesn't seem that I have to avoid bad blocks.  I
think the 'host' UBI/UbiFs recognizes that erase blocks are bad from the
'target' copy and doesn't use them, or I was lucky and somehow I didn't
have bad blocks or the tool/code I am using is somehow removing them.

Regards,
Bill Pringlemeir.

-- 
If it weren't for pickpockets I'd have no sex life at all. 
- Rodney Dangerfield