UBI on SPI NOR flash

UBI on SPI NOR flash

[Jan Lübbe]

Hi all!

I'm trying to use UBIFS on SPI NOR flash, but have been unable to get it
to work on kernel version 3.5. It is a Numonyx N25Q128 connected to an
TI AM3505.

I've added the JEDEC-ID to the m25p80 driver:
        { "n25q128", INFO(0x20ba18,  0, 64 * 1024,  256, 0) },
 
Now, when i try to use it with UBI I get this:
~ # cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00380000 00020000 "kernel"
mtd1: 00c00000 00020000 "root"
mtd2: 00400000 00020000 "appkernel"
mtd3: 02000000 00020000 "approot"
mtd4: 04a00000 00020000 "data"
mtd5: 01000000 00010000 "NOR-N25Q128"
~ # flash_erase /dev/mtd5 0 0
Erasing 64 Kibyte @ ff0000 -- 100 % complete 
~ # ubiattach -m 5
[  372.903594] UBI: attaching mtd5 to ubi0
[  372.908538] UBI: physical eraseblock size:   65536 bytes (64 KiB)
[  372.914916] UBI: logical eraseblock size:    65408 bytes
[  372.922393] UBI: smallest flash I/O unit:    1
[  372.927062] UBI: VID header offset:          64 (aligned 64)
[  372.933380] UBI: data offset:                128
[  372.985931] UBI: empty MTD device detected
[  372.990814] UBI: max. sequence number:       0
[  372.995635] UBI: create volume table (copy #1)
[  373.901519] UBI: create volume table (copy #2)
[  374.812377] UBI: attached mtd5 to ubi0
[  374.816345] UBI: MTD device name:            "NOR-N25Q128"
[  374.822875] UBI: MTD device size:            16 MiB
[  374.828338] UBI: number of good PEBs:        256
[  374.833190] UBI: number of bad PEBs:         0
[  374.838226] UBI: number of corrupted PEBs:   0
[  374.842895] UBI: max. allowed volumes:       128
[  374.848083] UBI: wear-leveling threshold:    4096
[  374.853027] UBI: number of internal volumes: 1
[  374.858032] UBI: number of user volumes:     0
[  374.862701] UBI: available PEBs:             252
[  374.867858] UBI: total number of reserved PEBs: 4
[  374.872802] UBI: number of PEBs reserved for bad PEB handling: 0
[  374.879425] UBI: max/mean erase counter: 0/0
[  374.883911] UBI: image sequence number:  -15869967
[  374.889282] UBI: background thread "ubi_bgt0d" started, PID 603
UBI device number 0, total 256 LEBs (16744448 bytes, 16.0 MiB), available 252 LEBs (16482816 bytes, 15.7 MiB), LEB size 65408 bytes (63.9 KiB)
~ # ubidetach -m 5
[  458.280059] UBI: mtd5 is detached from ubi0
~ # ubiattach -m 5
[  598.920166] UBI: attaching mtd5 to ubi0
[  598.924377] UBI: physical eraseblock size:   65536 bytes (64 KiB)
[  598.933044] UBI: logical eraseblock size:    65408 bytes
[  598.938934] UBI: smallest flash I/O unit:    1
[  598.943603] UBI: VID header offset:          64 (aligned 64)
[  598.949890] UBI: data offset:                128
[  599.021850] UBI: max. sequence number:       2
[  599.103698] UBI error: vtbl_check: bad CRC at record 0: 0x2b4f6dd5, not 0x000000
[  599.111877] UBI error: vtbl_check: bad CRC at record 0: 0x2b4f6dd5, not 0x000000
[  599.120788] UBI error: process_lvol: both volume tables are corrupted
[  599.128784] UBI error: ubi_attach_mtd_dev: failed to attach mtd5, error -22
ubiattach: error!: cannot attach mtd5
           error 22 (Invalid argument)

Here's a dump the beginning of the device:
~ # hd /dev/mtd5 | head -n 20
00000000  55 42 49 23 01 00 00 00  00 00 00 00 00 00 00 01  |UBI#............|
00000010  00 00 00 40 00 00 00 80  ff 0d d7 f1 00 00 00 00  |...@............|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 4d d2 53 1c  |............M.S.|
00000040  55 42 49 21 01 01 00 05  7f ff ef ff 00 00 00 00  |UBI!............|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 01  |................|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 65 b3 bd 2d  |............e..-|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000100  d5 a0 01 58 e0 a0 01 22  b8 f9 81 01 c1 21 54 e6  |...X...".....!T.|
00000110  a6 01 23 f8 0c 17 a9 0d  af 03 22 19 6b 0b e8 01  |..#.......".k...|
00000120  5c 09 ac 01 23 de f9 65  0f 8c 01 54 51 78 00 23  |\...#..e...TQx.#|
00000130  84 50 62 01 c5 f3 21 b2  95 73 0c 8c 01 22 6c 63  |.Pb...!..s..."lc|
00000140  21 95 bf 0a 27 32 00 bc  0e 28 28 00 22 dd 93 0b  |!...'2...((."...|
00000150  f8 01 59 82 0d ec 01 02  63 74 c0 c5 73 a4 11 59  |..Y.....ct..s..Y|
00000160  40 65 bb 14 8f 75 b4 ae  0a 81 ca 22 cd 24 65 a6  |@e...u.....".$e.|
00000170  07 73 d9 61 14 0c 27 2c  00 44 14 b0 0d 23 d0 76  |.s.a..',.D...#.v|
00000180  5a 26 d2 dc 21 e0 2c 7d  00 70 44 e4 5d 2a 73 55  |Z&..!.,}.pD.]*sU|
00000190  28 0e 7a 00 d8 98 26 21  77 08 78 01 c5 24 73 03  |(.z...&!w.x..$s.|

After some kB like this it continues with UBI headers at the start of
each erase block. Is this the way it should be?

I noticed, that mtdinfo -u reports reports min IO size and sub-page size
as 1 byte:
~ # mtdinfo -u /dev/mtd5
mtd5
Name:                           NOR-N25Q128
Type:                           nor
Eraseblock size:                65536 bytes, 64.0 KiB
Amount of eraseblocks:          256 (16777216 bytes, 16.0 MiB)
Minimum input/output unit size: 1 byte
Sub-page size:                  1 byte
Character device major/minor:   90:10
Bad blocks are allowed:         false
Device is writable:             true
Default UBI VID header offset:  64
Default UBI data offset:        128
Default UBI LEB size:           65408 bytes, 63.9 KiB
Maximum UBI volumes count:      128

Thanks in advance for any replies and best regards,
Jan Lübbe
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: UBI on SPI NOR flash

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 584 bytes --]

On Wed, 2012-08-15 at 19:10 +0200, Jan Lübbe wrote:
> Hi all!
> 
> I'm trying to use UBIFS on SPI NOR flash, but have been unable to get it
> to work on kernel version 3.5. It is a Numonyx N25Q128 connected to an
> TI AM3505.

First of all, run the mtd tests and see if the flash works well.

Another problem I know with SPI flashes is that UBIFS uses vmalloc'd
memory for I/O, which is not suitable for DMA-ing on ARM platforms and
may cause subtle bugs and curruptions. Can you try to disable DMA and
see if the issues go away?

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: UBI on SPI NOR flash

[Jan Lübbe]

Hi!

On Fri, 2012-08-24 at 15:31 +0300, Artem Bityutskiy wrote:
> On Wed, 2012-08-15 at 19:10 +0200, Jan Lübbe wrote:
> > Hi all!
> > 
> > I'm trying to use UBIFS on SPI NOR flash, but have been unable to get it
> > to work on kernel version 3.5. It is a Numonyx N25Q128 connected to an
> > TI AM3505.
> 
> First of all, run the mtd tests and see if the flash works well.

~ # modprobe mtd_readtest dev=5
[  124.351074] 
[  124.353057] =================================================
[  124.359466] mtd_readtest: MTD device: 5
[  124.365173] mtd_readtest: not NAND flash, assume page size is 512 bytes.
[  124.373107] mtd_readtest: MTD device size 3670016, eraseblock size 65536, page size 512, count of eraseblocks 56, pages per eraseblock 128, OOB size 0
[  124.388000] mtd_readtest: testing page read
[  126.754089] mtd_readtest: finished
[  126.757690] =================================================
~ # modprobe mtd_speedtest dev=5
[  160.338775] 
[  160.344146] =================================================
[  160.350708] mtd_speedtest: MTD device: 5
[  160.354827] mtd_speedtest: not NAND flash, assume page size is 512 bytes.
[  160.362396] mtd_speedtest: MTD device size 3670016, eraseblock size 65536, page size 512, count of eraseblocks 56, pages per eraseblock 128, OOB size 0
[  204.830078] mtd_speedtest: testing eraseblock write speed
[  218.410614] mtd_speedtest: eraseblock write speed is 264 KiB/s
[  218.416717] mtd_speedtest: testing eraseblock read speed
[  220.146087] mtd_speedtest: eraseblock read speed is 2078 KiB/s
[  264.651000] mtd_speedtest: testing page write speed
[  278.239562] mtd_speedtest: page write speed is 263 KiB/s
[  278.245697] mtd_speedtest: testing page read speed
[  280.607025] mtd_speedtest: page read speed is 1521 KiB/s
[  324.869445] mtd_speedtest: testing 2 page write speed
[  338.450592] mtd_speedtest: 2 page write speed is 263 KiB/s
[  338.456359] mtd_speedtest: testing 2 page read speed
[  340.498870] mtd_speedtest: 2 page read speed is 1760 KiB/s
[  340.505065] mtd_speedtest: Testing erase speed
[  384.802856] mtd_speedtest: erase speed is 80 KiB/s
[  384.807922] mtd_speedtest: Testing 2x multi-block erase speed
[  430.186370] mtd_speedtest: 2x multi-block erase speed is 78 KiB/s
[  430.193389] mtd_speedtest: Testing 4x multi-block erase speed
[  475.582244] mtd_speedtest: 4x multi-block erase speed is 78 KiB/s
[  475.588653] mtd_speedtest: Testing 8x multi-block erase speed
[  521.005981] mtd_speedtest: 8x multi-block erase speed is 78 KiB/s
[  521.013031] mtd_speedtest: Testing 16x multi-block erase speed
[  566.628875] mtd_speedtest: 16x multi-block erase speed is 78 KiB/s
[  566.636016] mtd_speedtest: Testing 32x multi-block erase speed
[  611.978057] mtd_speedtest: 32x multi-block erase speed is 79 KiB/s
[  611.985198] mtd_speedtest: Testing 64x multi-block erase speed
[  657.134216] mtd_speedtest: 64x multi-block erase speed is 79 KiB/s
[  657.141326] mtd_speedtest: finished
[  657.144989] =================================================
~ # modprobe mtd_stresstest dev=5 count=100
[  708.805480] 
[  708.807464] =================================================
[  708.815093] mtd_stresstest: MTD device: 5
[  708.819763] mtd_stresstest: not NAND flash, assume page size is 512 bytes.
[  708.827911] mtd_stresstest: MTD device size 3670016, eraseblock size 65536, page size 512, count of eraseblocks 56, pages per eraseblock 128, OOB size 0
[  708.848632] mtd_stresstest: doing operations
[  708.853637] mtd_stresstest: 0 operations done
[  775.528625] mtd_stresstest: finished, 100 operations done
[  775.535125] =================================================

> Another problem I know with SPI flashes is that UBIFS uses vmalloc'd
> memory for I/O, which is not suitable for DMA-ing on ARM platforms and
> may cause subtle bugs and curruptions. Can you try to disable DMA and
> see if the issues go away?

It works if I disable DMA in drivers/spi/spi-omap2-mcspi.c (simply
comment out the DMA branch in omap2_mcspi_work). I can use UBI&UBIFS
without problems for far. When rebooting to a kernel with DMA enabled I
can no longer attach the previously working volume. Moving back to the
PIO-Kernel, it works again.

Where should this be fixed? Should UBI use kmalloc'ed memory?

Why does this work for NAND controllers using DMA? :)

Regards,
Jan
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: UBI on SPI NOR flash

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

On Mon, 2012-08-27 at 18:15 +0200, Jan Lübbe wrote:
> > Another problem I know with SPI flashes is that UBIFS uses vmalloc'd
> > memory for I/O, which is not suitable for DMA-ing on ARM platforms
> and
> > may cause subtle bugs and curruptions. Can you try to disable DMA
> and
> > see if the issues go away?
> 
> It works if I disable DMA in drivers/spi/spi-omap2-mcspi.c (simply
> comment out the DMA branch in omap2_mcspi_work). I can use UBI&UBIFS
> without problems for far. When rebooting to a kernel with DMA enabled
> I
> can no longer attach the previously working volume. Moving back to the
> PIO-Kernel, it works again.
> 
> Where should this be fixed? Should UBI use kmalloc'ed memory?

Yes, the best way to fix this is to teach UBI and UBIFS stop using
vmalloc(). It is not straight-forward, but doable. As I said several
times, I can assist by giving ideas and directions, but someone who
needs this should probably allocate a couple of men-months for this
project.

> Why does this work for NAND controllers using DMA? :) 

I am not sure, you should investigate. I think some drivers just check
if this is vmalloced memory, and avoid DMA for that. Some (e.g.,
onenand) use various hacks which appear to work.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: UBI on SPI NOR flash

[Jan Lübbe]

Hi Artem,

On Mon, 2012-09-03 at 12:23 +0300, Artem Bityutskiy wrote:
> On Mon, 2012-08-27 at 18:15 +0200, Jan Lübbe wrote:
> > It works if I disable DMA in drivers/spi/spi-omap2-mcspi.c (simply
> > comment out the DMA branch in omap2_mcspi_work). I can use UBI&UBIFS
> > without problems for far. When rebooting to a kernel with DMA enabled
> > I
> > can no longer attach the previously working volume. Moving back to the
> > PIO-Kernel, it works again.
> > 
> > Where should this be fixed? Should UBI use kmalloc'ed memory?
> 
> Yes, the best way to fix this is to teach UBI and UBIFS stop using
> vmalloc(). It is not straight-forward, but doable. As I said several
> times, I can assist by giving ideas and directions, but someone who
> needs this should probably allocate a couple of men-months for this
> project.

I can't commit to spending months on this, but I do have some time.
Do you think the vmalloc-using places could be fixed step by step or
must this be done over UBI&UBIFI in one go?
Is there a specific thread where the approach fixing this has been
discussed before? Could the CMA be used instead of vmalloc?

You mentioned flexible arrays before. It seems it is targeted at items
smaller than a page and needs to copy each item when placing it into the
array. I'm not sure how this would help UBI/UBIF.

We probably also need an explicit way to tell MTD if a buffer is DMA
capable or not. Then we could avoid deciding based on the buffer pointer
in the storage drivers.

> > Why does this work for NAND controllers using DMA? :) 
> 
> I am not sure, you should investigate. I think some drivers just check
> if this is vmalloced memory, and avoid DMA for that. Some (e.g.,
> onenand) use various hacks which appear to work.

I was able to get it to work using this kind of hack (check in SPI host
driver for buf >= high_memory and fall back to PIO). It's submitted to
the SPI list, but I'm not sure if it's acceptable for them.

Regards,
Jan
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: UBI on SPI NOR flash

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 2929 bytes --]

On Mon, 2012-09-03 at 11:44 +0200, Jan Lübbe wrote:
> Hi Artem,
> 
> On Mon, 2012-09-03 at 12:23 +0300, Artem Bityutskiy wrote:
> > On Mon, 2012-08-27 at 18:15 +0200, Jan Lübbe wrote:
> > > It works if I disable DMA in drivers/spi/spi-omap2-mcspi.c (simply
> > > comment out the DMA branch in omap2_mcspi_work). I can use UBI&UBIFS
> > > without problems for far. When rebooting to a kernel with DMA enabled
> > > I
> > > can no longer attach the previously working volume. Moving back to the
> > > PIO-Kernel, it works again.
> > > 
> > > Where should this be fixed? Should UBI use kmalloc'ed memory?
> > 
> > Yes, the best way to fix this is to teach UBI and UBIFS stop using
> > vmalloc(). It is not straight-forward, but doable. As I said several
> > times, I can assist by giving ideas and directions, but someone who
> > needs this should probably allocate a couple of men-months for this
> > project.
> 
> I can't commit to spending months on this, but I do have some time.
> Do you think the vmalloc-using places could be fixed step by step or
> must this be done over UBI&UBIFI in one go?
> Is there a specific thread where the approach fixing this has been
> discussed before? Could the CMA be used instead of vmalloc?

I believe it can be done step by step. You can start from UBI I guess.
Basically we use vmalloc everywhere when we need to read entire
eraseblock. The reason for this is that the eraseblock is large, so
kmalloc() would often fail.

So the idea is to allocate smaller buffers and then iterate. There are
places which are easier to handle, and some are more difficult to handle
(e.g., those where we index elements in the eraseblock array).

> You mentioned flexible arrays before. It seems it is targeted at items
> smaller than a page and needs to copy each item when placing it into the
> array. I'm not sure how this would help UBI/UBIF.

I think I meant something like flexible arrays. The idea was to create
an API which you can use as if you had a single vmalloc buffer. Indexing
operation then could be changed with a helper function.

> We probably also need an explicit way to tell MTD if a buffer is DMA
> capable or not. Then we could avoid deciding based on the buffer pointer
> in the storage drivers.

I think it is better to eliminate vmalloc from UBI/UBIFS.

> > > Why does this work for NAND controllers using DMA? :) 
> > 
> > I am not sure, you should investigate. I think some drivers just check
> > if this is vmalloced memory, and avoid DMA for that. Some (e.g.,
> > onenand) use various hacks which appear to work.
> 
> I was able to get it to work using this kind of hack (check in SPI host
> driver for buf >= high_memory and fall back to PIO). It's submitted to
> the SPI list, but I'm not sure if it's acceptable for them.

I hope not :-) We need to fix things properly.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: UBI on SPI NOR flash

[Jan Lübbe]

On Mon, 2012-09-03 at 13:00 +0300, Artem Bityutskiy wrote:
> On Mon, 2012-09-03 at 11:44 +0200, Jan Lübbe wrote:
> > I can't commit to spending months on this, but I do have some time.
> > Do you think the vmalloc-using places could be fixed step by step or
> > must this be done over UBI&UBIFI in one go?
> > Is there a specific thread where the approach fixing this has been
> > discussed before? Could the CMA be used instead of vmalloc?
> 
> I believe it can be done step by step. You can start from UBI I guess.
> Basically we use vmalloc everywhere when we need to read entire
> eraseblock. The reason for this is that the eraseblock is large, so
> kmalloc() would often fail.

Ok.

> So the idea is to allocate smaller buffers and then iterate. There are
> places which are easier to handle, and some are more difficult to handle
> (e.g., those where we index elements in the eraseblock array).
> 
> > You mentioned flexible arrays before. It seems it is targeted at items
> > smaller than a page and needs to copy each item when placing it into the
> > array. I'm not sure how this would help UBI/UBIF.
> 
> I think I meant something like flexible arrays. The idea was to create
> an API which you can use as if you had a single vmalloc buffer. Indexing
> operation then could be changed with a helper function.

Would you then iterate inside of UBI using this API, making page-sized
calls to MTD? Or would MTD need to be extended to pass around these
"DMA-lists"?

Do we need multiple large buffers at the same time for one UBI device?
If not, we could get one EB-size buffer from CMA on probe/mount and
reuse it.

> > We probably also need an explicit way to tell MTD if a buffer is DMA
> > capable or not. Then we could avoid deciding based on the buffer pointer
> > in the storage drivers.
> 
> I think it is better to eliminate vmalloc from UBI/UBIFS.

So in the end, MTD drivers would expect to only receive DMA capable
buffers?

> > I was able to get it to work using this kind of hack (check in SPI host
> > driver for buf >= high_memory and fall back to PIO). It's submitted to
> > the SPI list, but I'm not sure if it's acceptable for them.
> 
> I hope not :-) We need to fix things properly.

It's the same approach as taken by some NAND drivers. :-)

Regards,
Jan
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |