UBIFS on kernel 2.6.24?

UBIFS on kernel 2.6.24?

[David Jander]

Hi all,

I am forced to use this ancient kernel (2.6.24), but I also need UBIFS. 
Our boards have 1GiB NAND flash chips with 2k pages.
Until now I have been testing with vanilla MTD/UBI/UBIFS from that kernel, 
and (not entirely unexpected) now errors start to appear, which look like this:

$ cd /etc
$ ls -l
ls: reading directory .: Structure needs cleaning

And the kernel messages look like this:

[10871.725906] UBIFS error (pid 10801): ubifs_check_node: bad magic 0x38f2aa8b, expected 0x6101831
[10871.736785] UBIFS error (pid 10801): ubifs_check_node: bad node at LEB 6259:66568
[10871.745344] UBIFS error (pid 10801): ubifs_read_node: expected node type 2
[10871.753209] UBIFS error (pid 10801): ubifs_readdir: cannot find next direntry, error -117


I did not bother turning on more debug messages yet, since I think I am most probably missing some important bugfixes in my version of 
MTD/UBI/UBIFS. Btw, is there some sort of version number in some file somewhere, so I can check wich version I am actually using?

I am aware of this git repository:
http://git.infradead.org/users/dedekind/ubifs-v2.6.24.git

But I'd like to know how I can get this version into my kernel. Will it work if I just copy the relevant files?
Is there a patch against 2.6.24 that I can download somewhere?
Do I need only UBIFS, or do I also need all of MTD?
I have a custom NAND-flash driver, do I likely need to change it, or should it just work with the new version of MTD?

Thanks for any advice.

Best regards,

-- 
David Jander
Protonic Holland.

Re: UBIFS on kernel 2.6.24?

[Artem Bityutskiy]

Him

On Mon, 2009-11-23 at 15:10 +0100, David Jander wrote:
> Hi all,
> 
> I am forced to use this ancient kernel (2.6.24), but I also need UBIFS. 
> Our boards have 1GiB NAND flash chips with 2k pages.
> Until now I have been testing with vanilla MTD/UBI/UBIFS from that kernel, 
> and (not entirely unexpected) now errors start to appear, which look like this:
> 
> $ cd /etc
> $ ls -l
> ls: reading directory .: Structure needs cleaning
> 
> And the kernel messages look like this:
> 
> [10871.725906] UBIFS error (pid 10801): ubifs_check_node: bad magic 0x38f2aa8b, expected 0x6101831
> [10871.736785] UBIFS error (pid 10801): ubifs_check_node: bad node at LEB 6259:66568
> [10871.745344] UBIFS error (pid 10801): ubifs_read_node: expected node type 2
> [10871.753209] UBIFS error (pid 10801): ubifs_readdir: cannot find next direntry, error -117

Looks like your NAND driver has problems. You should run mtd tests and validate it:
http://www.linux-mtd.infradead.org/doc/general.html#L_mtd_tests

> I did not bother turning on more debug messages yet, since I think I
> am most probably missing some important bugfixes in my version of 
> MTD/UBI/UBIFS.

May be, but this really looks like you either have problems in the
driver, or you did not flash the UBIFS image correctly. Try to reproduce
this problem when you attach a freshly erased NAND to UBI and then mount
UBIFS.

>  Btw, is there some sort of version number in some file somewhere, so
> I can check wich version I am actually using?

No, not really. The assumption is that the latest version should always
be used :-)

> I am aware of this git repository:
> http://git.infradead.org/users/dedekind/ubifs-v2.6.24.git

Good! Why you do not use it then? We fixed many UBI/UBIFS bugs there,
and there were important improvements.

> But I'd like to know how I can get this version into my kernel. Will
> it work if I just copy the relevant files?

No. This is git, and what you should do is to merge ubifs-v2.6.24.git to
your tree.

> Is there a patch against 2.6.24 that I can download somewhere?

This is git, and you can create patches yourself. Clone
ubifs-v2.6.24.git and do something like:

git format-patch v2.6.24

You will get all patches. Then you can merge them if you want, but this
is a bad idea, IMHO.

> Do I need only UBIFS, or do I also need all of MTD?

There are some mtd fixes which I considered important in the
ubifs-v2.6.24.git tree. And the tree is sufficient.

But of course, if you find some MTD problems which were fixed later, you
may back-port more patches.

> I have a custom NAND-flash driver, do I likely need to change it, or
> should it just work with the new version of MTD?

You really should just try things out. But I think it should be fine.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS on kernel 2.6.24?

[David Jander]

Hi,

Thanks for replying!

> On Mon, 2009-11-23 at 15:10 +0100, David Jander wrote:
> > Hi all,
> >
> > I am forced to use this ancient kernel (2.6.24), but I also need UBIFS.
> > Our boards have 1GiB NAND flash chips with 2k pages.
> > Until now I have been testing with vanilla MTD/UBI/UBIFS from that
> > kernel, and (not entirely unexpected) now errors start to appear, which
> > look like this:
> >
> > $ cd /etc
> > $ ls -l
> > ls: reading directory .: Structure needs cleaning
> >
> > And the kernel messages look like this:
> >
> > [10871.725906] UBIFS error (pid 10801): ubifs_check_node: bad magic
> > 0x38f2aa8b, expected 0x6101831 [10871.736785] UBIFS error (pid 10801):
> > ubifs_check_node: bad node at LEB 6259:66568 [10871.745344] UBIFS error
> > (pid 10801): ubifs_read_node: expected node type 2 [10871.753209] UBIFS
> > error (pid 10801): ubifs_readdir: cannot find next direntry, error -117
> 
> Looks like your NAND driver has problems. You should run mtd tests and
>  validate it:
>  http://www.linux-mtd.infradead.org/doc/general.html#L_mtd_tests

I have downloaded those tools, and started playing with them.
I started with speedtest :-), oobtest, scantest and pagetest. Those al ran 
without problems, but subpagetest stopped with an error:

[...]
[14083.422527] subpagetest: erasing                                                                                                                                                 
[14083.432654] subpagetest: erased 0                                                                                                                                                
[14083.815876] subpagetest: erased 256                                                                                                                                              
[14084.190646] subpagetest: erased 512                                                                                                                                              
[14084.573346] subpagetest: erased 768                                                                                                                                              
[14084.956378] subpagetest: erased 1024                                                                                                                                             
[14085.339424] subpagetest: erased 1280                                                                                                                                             
[14085.716865] subpagetest: erased 1536                                                                                                                                             
[14086.099616] subpagetest: erased 1792                                                                                                                                             
[14086.482645] subpagetest: erased 2048                                                                                                                                             
[14086.496965] subpagetest: error -5 while erasing EB 2052                                                                                                                          
[14086.507507] subpagetest: error -5 occurred                                                                                                                                       

If I repeat this test I get an error on the same EB.
If I repeat pagetest now, I also get the same error on the same EB.
If I run flash_eraseall on the device, I get this:

$ flash_eraseall /dev/mtd1
Erasing 128 Kibyte @ 3980000 --  5 % complete.
Skipping bad block at 0x039a0000
Erasing 128 Kibyte @ 3f80000 --  6 % complete.
Skipping bad block at 0x03fa0000
Erasing 128 Kibyte @ 10080000 -- 25 % complete.
/usr/local/sbin/flash_eraseall: /dev/mtd1: MTD Erase failure: Input/output 
error
Erasing 128 Kibyte @ 18760000 -- 38 % complete.
Skipping bad block at 0x18780000
Erasing 128 Kibyte @ 3ff60000 -- 99 % complete.
Skipping bad block at 0x3ff80000
Skipping bad block at 0x3ffa0000
Skipping bad block at 0x3ffc0000
Skipping bad block at 0x3ffe0000
Erasing 128 Kibyte @ 40000000 -- 100 % complete.

Did this test kill my nand-flash chip?

> > I did not bother turning on more debug messages yet, since I think I
> > am most probably missing some important bugfixes in my version of
> > MTD/UBI/UBIFS.
> 
> May be, but this really looks like you either have problems in the
> driver, or you did not flash the UBIFS image correctly. Try to reproduce
> this problem when you attach a freshly erased NAND to UBI and then mount
> UBIFS.

I did not flash an image. To initialize a new flash chip I just do this from 
an NFS-root system:

ubiattach /dev/ubi_ctrl -m 1
ubimkvol /dev/ubi0 -N rootfs -m
mount -t ubifs ubi0:rootfs /mnt/nand/
cp -ax /path/to/rootfs/* /mnt/nand/
umount /mnt/nand

> >  Btw, is there some sort of version number in some file somewhere, so
> > I can check wich version I am actually using?
> 
> No, not really. The assumption is that the latest version should always
> be used :-)

Ok, I just meant to be able to identify a certain version which is not the 
latest.... never mind.

> > I am aware of this git repository:
> > http://git.infradead.org/users/dedekind/ubifs-v2.6.24.git
> 
> Good! Why you do not use it then? We fixed many UBI/UBIFS bugs there,
> and there were important improvements.

I usually first want to find out something is broken before trying to fix it 
;-)
We do use a nand-flash driver which is not part of the MTD tree yet, and I 
don't know if I can safely mix this driver with a newer version of MTD/UBI.
Besides I wouldn't know the necessary git magic to get this into our tree and 
knowing I don't break anything else.

> > But I'd like to know how I can get this version into my kernel. Will
> > it work if I just copy the relevant files?
> 
> No. This is git, and what you should do is to merge ubifs-v2.6.24.git to
> your tree.

I thought so. Well, I guess I'll have to do some more git manual studying 
*sigh*.

> > Is there a patch against 2.6.24 that I can download somewhere?
> 
> This is git, and you can create patches yourself. Clone
> ubifs-v2.6.24.git and do something like:
> 
> git format-patch v2.6.24

Thanks for the tip, that seems safe to try.

> You will get all patches. Then you can merge them if you want, but this
> is a bad idea, IMHO.

Hmmmm.... If you know how to do it the right way with git, why don't you just 
take the little trouble of putting into the web-page? A lot of people would 
learn from it (me included).

> > Do I need only UBIFS, or do I also need all of MTD?
> 
> There are some mtd fixes which I considered important in the
> ubifs-v2.6.24.git tree. And the tree is sufficient.
> 
> But of course, if you find some MTD problems which were fixed later, you
> may back-port more patches.
> 
> > I have a custom NAND-flash driver, do I likely need to change it, or
> > should it just work with the new version of MTD?
> 
> You really should just try things out. But I think it should be fine.

Ok, thanks a lot!

Best regards,

-- 
David Jander
Protonic Holland.
tel.: +31 (0) 229 212928
fax.: +31 (0) 229 210930
Factorij 36 / 1689 AL Zwaag

Re: UBIFS on kernel 2.6.24?

[Artem Bityutskiy]

On Tue, 2009-12-08 at 17:45 +0100, David Jander wrote:
> I have downloaded those tools, and started playing with them.
> I started with speedtest :-), oobtest, scantest and pagetest. Those al ran 
> without problems, but subpagetest stopped with an error:
> 
> [...]
> [14083.422527] subpagetest: erasing                                                                                                                                                 
> [14083.432654] subpagetest: erased 0                                                                                                                                                
> [14083.815876] subpagetest: erased 256                                                                                                                                              
> [14084.190646] subpagetest: erased 512                                                                                                                                              
> [14084.573346] subpagetest: erased 768                                                                                                                                              
> [14084.956378] subpagetest: erased 1024                                                                                                                                             
> [14085.339424] subpagetest: erased 1280                                                                                                                                             
> [14085.716865] subpagetest: erased 1536                                                                                                                                             
> [14086.099616] subpagetest: erased 1792                                                                                                                                             
> [14086.482645] subpagetest: erased 2048                                                                                                                                             
> [14086.496965] subpagetest: error -5 while erasing EB 2052                                                                                                                          
> [14086.507507] subpagetest: error -5 occurred                                                                                                                                       
> 
> If I repeat this test I get an error on the same EB.
> If I repeat pagetest now, I also get the same error on the same EB.
> If I run flash_eraseall on the device, I get this:
> 
> $ flash_eraseall /dev/mtd1
> Erasing 128 Kibyte @ 3980000 --  5 % complete.
> Skipping bad block at 0x039a0000
> Erasing 128 Kibyte @ 3f80000 --  6 % complete.
> Skipping bad block at 0x03fa0000
> Erasing 128 Kibyte @ 10080000 -- 25 % complete.
> /usr/local/sbin/flash_eraseall: /dev/mtd1: MTD Erase failure: Input/output 
> error
> Erasing 128 Kibyte @ 18760000 -- 38 % complete.
> Skipping bad block at 0x18780000
> Erasing 128 Kibyte @ 3ff60000 -- 99 % complete.
> Skipping bad block at 0x3ff80000
> Skipping bad block at 0x3ffa0000
> Skipping bad block at 0x3ffc0000
> Skipping bad block at 0x3ffe0000
> Erasing 128 Kibyte @ 40000000 -- 100 % complete.
> 
> Did this test kill my nand-flash chip?

Probably a bad block, which is normal on NAND. This is one of the
reasons why NAND-aware software is not-so-simple.

I think ubiformat would suggest you to mark this faulty eraseblock as
bad.

> > > But I'd like to know how I can get this version into my kernel. Will
> > > it work if I just copy the relevant files?
> > 
> > No. This is git, and what you should do is to merge ubifs-v2.6.24.git to
> > your tree.
> 
> I thought so. Well, I guess I'll have to do some more git manual studying 
> *sigh*.

'git pull' should do the trick.

> > > Is there a patch against 2.6.24 that I can download somewhere?
> > 
> > This is git, and you can create patches yourself. Clone
> > ubifs-v2.6.24.git and do something like:
> > 
> > git format-patch v2.6.24
> 
> Thanks for the tip, that seems safe to try.

To apply those patches to another tree, you can use 'git am'.

> > You will get all patches. Then you can merge them if you want, but this
> > is a bad idea, IMHO.
> 
> Hmmmm.... If you know how to do it the right way with git, why don't you just 
> take the little trouble of putting into the web-page? A lot of people would 
> learn from it (me included).

It is about git basics, there are many good tutorials. This is just
about doing the basic operation - 'git pull'.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS on kernel 2.6.24?

[David Jander]

On Wednesday 09 December 2009 09:41:48 am Artem Bityutskiy wrote:
>[...]
> > $ flash_eraseall /dev/mtd1
> > Erasing 128 Kibyte @ 3980000 --  5 % complete.
> > Skipping bad block at 0x039a0000
> > Erasing 128 Kibyte @ 3f80000 --  6 % complete.
> > Skipping bad block at 0x03fa0000
> > Erasing 128 Kibyte @ 10080000 -- 25 % complete.
> > /usr/local/sbin/flash_eraseall: /dev/mtd1: MTD Erase failure:
> > Input/output error
> > Erasing 128 Kibyte @ 18760000 -- 38 % complete.
> > Skipping bad block at 0x18780000
> > Erasing 128 Kibyte @ 3ff60000 -- 99 % complete.
> > Skipping bad block at 0x3ff80000
> > Skipping bad block at 0x3ffa0000
> > Skipping bad block at 0x3ffc0000
> > Skipping bad block at 0x3ffe0000
> > Erasing 128 Kibyte @ 40000000 -- 100 % complete.
> >
> > Did this test kill my nand-flash chip?
> 
> Probably a bad block, which is normal on NAND. This is one of the
> reasons why NAND-aware software is not-so-simple.

I know bad blocks can occur at any time after manufacturing, although this is 
the first time I see it happening. I wouldn't have expected this to start 
occurring so early, though (the chip is new).

>[...]
> > > You will get all patches. Then you can merge them if you want, but this
> > > is a bad idea, IMHO.
> >
> > Hmmmm.... If you know how to do it the right way with git, why don't you
> > just take the little trouble of putting into the web-page? A lot of
> > people would learn from it (me included).
> 
> It is about git basics, there are many good tutorials. This is just
> about doing the basic operation - 'git pull'.

Off-topic: Probably. I must admit that I feel quite dumb having to ask this, 
but I have lost so much time already because I thought I understood some git 
command, which I didn't, and ended up destroying my git repository several 
times, so I stopped trying to figure things out on my own. Ask me about kernel 
internals what you want, but Git (and its documentation!) just goes over my 
head (I am a humble subversion user).

Back on topic:
I have repeated the nand-tests on another board, and they ran just fine, 
repeating for quite a while without any error, so I guess that proves that the 
hardware and nand-driver are working reasonably well. OTOH, The two failed 
systems, also were running for several months with very heavy flash-disk I/O 
(not as much as to wear it out though, there's still roughly 40% free space on 
the UBI volume). It has gone through installing ubuntu on it, and building 
several debian packages from source, etc... all apparently without problems, 
until one day 'ls -l /etc/' just failed.
I will try to pull from ubifs-v2.6.24.git, but I'd like to know your opinion 
on what the chances are that this update will fix the problem or bug that 
caused this corruption?
Your opinion is valuable, because otherwise it may take months before I find 
out I should have looked somewhere else for the problem.

Thanks a lot again!

Best regards,

-- 
David Jander
Protonic Holland.

Re: UBIFS on kernel 2.6.24?

[Artem Bityutskiy]

On Wed, 2009-12-09 at 11:40 +0100, David Jander wrote:
> Back on topic:
> I have repeated the nand-tests on another board, and they ran just fine, 
> repeating for quite a while without any error, so I guess that proves that the 
> hardware and nand-driver are working reasonably well. OTOH, The two failed 
> systems, also were running for several months with very heavy flash-disk I/O 
> (not as much as to wear it out though, there's still roughly 40% free space on 
> the UBI volume). It has gone through installing ubuntu on it, and building 
> several debian packages from source, etc... all apparently without problems, 
> until one day 'ls -l /etc/' just failed.
> I will try to pull from ubifs-v2.6.24.git, but I'd like to know your opinion 
> on what the chances are that this update will fix the problem or bug that 
> caused this corruption?

I cannot tell for sure, it may.

What I can say for sure is that I personally is not very interested in
solving problems for too old UBIFS code-base. Well, problems I can
reproduce here, in my setup, are OK, but subtle problems which I cannot
reproduce here are not OK - I really want to be sure UBI/UBIFS are
up-to-date and this is not something we already fixed.

BTW, from now on 2.6.24 back-port support is dropped. But it is
up-to-date _now_ :-)

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS on kernel 2.6.24?

[David Jander]

On Wednesday 09 December 2009 11:55:09 am Artem Bityutskiy wrote:
> On Wed, 2009-12-09 at 11:40 +0100, David Jander wrote:
> > Back on topic:
> > I have repeated the nand-tests on another board, and they ran just fine,
> > repeating for quite a while without any error, so I guess that proves
> > that the hardware and nand-driver are working reasonably well. OTOH, The
> > two failed systems, also were running for several months with very heavy
> > flash-disk I/O (not as much as to wear it out though, there's still
> > roughly 40% free space on the UBI volume). It has gone through installing
> > ubuntu on it, and building several debian packages from source, etc...
> > all apparently without problems, until one day 'ls -l /etc/' just failed.
> > I will try to pull from ubifs-v2.6.24.git, but I'd like to know your
> > opinion on what the chances are that this update will fix the problem or
> > bug that caused this corruption?
> 
> I cannot tell for sure, it may.

Ok, thanks.

> What I can say for sure is that I personally is not very interested in
> solving problems for too old UBIFS code-base. Well, problems I can
> reproduce here, in my setup, are OK, but subtle problems which I cannot
> reproduce here are not OK - I really want to be sure UBI/UBIFS are
> up-to-date and this is not something we already fixed.

I understand. All I wanted is to see if someone would jump up and say 
something like: "Oh, no, this old version is known to be broken, you need some 
important bug-fixes that went into the updates". That obviously didn't happen, 
so I know what I am facing: Update to the latest version, test it thoroughly, 
and hope for the best :-(

> BTW, from now on 2.6.24 back-port support is dropped. But it is
> up-to-date _now_ :-)

Ouch. Unfortunately for our platform there is no reasonable chance I could 
move to a newer kernel anytime soon (still no mainline support) :-(

Thanks a lot.

Best regards,

-- 
David Jander
Protonic Holland.

Re: UBIFS on kernel 2.6.24?

[Artem Bityutskiy]

On Thu, 2009-12-10 at 10:19 +0100, David Jander wrote:
> > What I can say for sure is that I personally is not very interested in
> > solving problems for too old UBIFS code-base. Well, problems I can
> > reproduce here, in my setup, are OK, but subtle problems which I cannot
> > reproduce here are not OK - I really want to be sure UBI/UBIFS are
> > up-to-date and this is not something we already fixed.
> 
> I understand. All I wanted is to see if someone would jump up and say 
> something like: "Oh, no, this old version is known to be broken, you need some 
> important bug-fixes that went into the updates". That obviously didn't happen, 
> so I know what I am facing: Update to the latest version, test it thoroughly, 
> and hope for the best :-(

I'd put it this way: pull the back-port tree, try to reproduce the
problem, and come back if it is still there. This will be faster than
lengthy e-mail exchange :-)

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: UBIFS on kernel 2.6.24?

[Fortini Matteo]

Up for the parent.

Unfortunately, MPC512x support is not in the mainstream kernel yet!

Thank you for your good work anyway.

Matteo

Il 10/12/2009 10.19, David Jander ha scritto:
> On Wednesday 09 December 2009 11:55:09 am Artem Bityutskiy wrote:
>    
>> On Wed, 2009-12-09 at 11:40 +0100, David Jander wrote:
>>      
>>> Back on topic:
>>> I have repeated the nand-tests on another board, and they ran just fine,
>>> repeating for quite a while without any error, so I guess that proves
>>> that the hardware and nand-driver are working reasonably well. OTOH, The
>>> two failed systems, also were running for several months with very heavy
>>> flash-disk I/O (not as much as to wear it out though, there's still
>>> roughly 40% free space on the UBI volume). It has gone through installing
>>> ubuntu on it, and building several debian packages from source, etc...
>>> all apparently without problems, until one day 'ls -l /etc/' just failed.
>>> I will try to pull from ubifs-v2.6.24.git, but I'd like to know your
>>> opinion on what the chances are that this update will fix the problem or
>>> bug that caused this corruption?
>>>        
>> I cannot tell for sure, it may.
>>      
> Ok, thanks.
>
>    
>> What I can say for sure is that I personally is not very interested in
>> solving problems for too old UBIFS code-base. Well, problems I can
>> reproduce here, in my setup, are OK, but subtle problems which I cannot
>> reproduce here are not OK - I really want to be sure UBI/UBIFS are
>> up-to-date and this is not something we already fixed.
>>      
> I understand. All I wanted is to see if someone would jump up and say
> something like: "Oh, no, this old version is known to be broken, you need some
> important bug-fixes that went into the updates". That obviously didn't happen,
> so I know what I am facing: Update to the latest version, test it thoroughly,
> and hope for the best :-(
>
>    
>> BTW, from now on 2.6.24 back-port support is dropped. But it is
>> up-to-date _now_ :-)
>>      
> Ouch. Unfortunately for our platform there is no reasonable chance I could
> move to a newer kernel anytime soon (still no mainline support) :-(
>
> Thanks a lot.
>
> Best regards,
>
>    

-