choosing a file system to use on NAND/UBI

choosing a file system to use on NAND/UBI

[Hamish Moffatt]

I'm developing an embedded product which will have its root file system
on NAND. The root will be pre-prepared and installed using
flashcp/nandwrite/ubiupdatevol, and usually mounted read-only.
Occasionally we might want to mount it read/write for debug purposes.

I'm intending to use UBI for volume management and wear-levelling.

JFFS2 on UBI does not seem very fast, in particular mount time and
initial access (until cache is populated). I tried disabling compression
but that seemed to make it worse (mount time was doubled).

Will ubifs on UBI be better? Is it mature enough to use yet?

For static volumes you have UBI protecting you, so you should not need
file-system CRC checking as JFFS2 does, correct? Are there any file
systems which take advantage of this property? 

Would I be better choosing LogFS or YAFFS1/2, perhaps without UBI?

thanks
Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: choosing a file system to use on NAND/UBI

[Artem Bityutskiy]

On Fri, 2008-03-28 at 12:04 +1100, Hamish Moffatt wrote:
> I'm developing an embedded product which will have its root file system
> on NAND. The root will be pre-prepared and installed using
> flashcp/nandwrite/ubiupdatevol, and usually mounted read-only.
> Occasionally we might want to mount it read/write for debug purposes.
Please, tell the size of your flash and whether it SLC or MLC.

> JFFS2 on UBI does not seem very fast, in particular mount time and
> initial access (until cache is populated).
I believe this will not depend much on whether there is UBI or not. You
tried static volume which has to check CRC checksum of the whole flash -
that might have introduced noticeable overhead.

Yes, JFFS2 is slow just after mount because it has to check whole
file-system partition. Mount time also depends on the contents of the
file-system - you might be interested to look at this section for more
information:

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_scalability

>  I tried disabling compression
> but that seemed to make it worse (mount time was doubled).
> 
> Will ubifs on UBI be better? Is it mature enough to use yet?
UBIFS mounts very quickly. It is very stable. We find minor issues from
time to time, but they are fixed quickly.

> For static volumes you have UBI protecting you, so you should not need
> file-system CRC checking as JFFS2 does, correct? Are there any file
> systems which take advantage of this property? 

Not sure. But this static volume CRC protection is not very useful if
you have something like JFFS2 on top.

> Would I be better choosing LogFS or YAFFS1/2, perhaps without UBI?

Never used YAFFSx, but it might be worth trying and evaluating. Vs.
LogFS - last time we tried it - it was too slow. It also did not
implement wear-levelling and bad block handling.

If you try to evaluate them - I would be interested to know about your
experience.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: choosing a file system to use on NAND/UBI

[Adrian Hunter]

Hamish Moffatt wrote:
> I'm developing an embedded product which will have its root file system
> on NAND. The root will be pre-prepared and installed using
> flashcp/nandwrite/ubiupdatevol, and usually mounted read-only.
> Occasionally we might want to mount it read/write for debug purposes.
> 
> I'm intending to use UBI for volume management and wear-levelling.
> 
> JFFS2 on UBI does not seem very fast, in particular mount time and
> initial access (until cache is populated). I tried disabling compression
> but that seemed to make it worse (mount time was doubled).

Do you know that you need to have JFFS2 summary support enabled to have
reasonable mount times?

Re: choosing a file system to use on NAND/UBI

[Hamish Moffatt]

On Fri, Mar 28, 2008 at 08:33:44AM +0200, Artem Bityutskiy wrote:
> On Fri, 2008-03-28 at 12:04 +1100, Hamish Moffatt wrote:
> > I'm developing an embedded product which will have its root file system
> > on NAND. The root will be pre-prepared and installed using
> > flashcp/nandwrite/ubiupdatevol, and usually mounted read-only.
> > Occasionally we might want to mount it read/write for debug purposes.
> Please, tell the size of your flash and whether it SLC or MLC.

Just to finish this old discussion, it's a 512Mb SLC part. ie not very
big. I have tried UBIFS and I am very pleased with it. Performance is
much better than JFFS2, which was slow to mount and slow during early
reads (even when the image was processed with sumtool).

> > Will ubifs on UBI be better? Is it mature enough to use yet?
> UBIFS mounts very quickly. It is very stable. We find minor issues from
> time to time, but they are fixed quickly.

I noticed that the on-chip format just changed (nanoseconds etc). Do you
plan any more incompatible changes? This is the sort of maturity issue I
was worried about.

> > Would I be better choosing LogFS or YAFFS1/2, perhaps without UBI?
> 
> Never used YAFFSx, but it might be worth trying and evaluating. Vs.
> LogFS - last time we tried it - it was too slow. It also did not
> implement wear-levelling and bad block handling.
> 
> If you try to evaluate them - I would be interested to know about your
> experience.

I could not get a pre-generated YAFFS2 image to mount successfully, and
a call for assistance on the YAFFS mailing list went unanswered.
Although I'm interested in LogFS I don't have enough time to try it
thoroughly.

thanks,
Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: choosing a file system to use on NAND/UBI

[Artem Bityutskiy]

Hi,

On Mon, 2008-04-07 at 15:12 +1000, Hamish Moffatt wrote:
> Just to finish this old discussion, it's a 512Mb SLC part. ie not very
> big. I have tried UBIFS and I am very pleased with it. Performance is
> much better than JFFS2, which was slow to mount and slow during early
> reads (even when the image was processed with sumtool).

"Mb" looks like Maga-bit, is that right?


> I noticed that the on-chip format just changed (nanoseconds etc). Do you
> plan any more incompatible changes? This is the sort of maturity issue I
> was worried about.

Well, we will try not to change it anymore, but we cannot guarantee this
so far. That particular change was requested by LKML. We may have more
requests - we want to get into the mainline.

But if we'll have users who may be hurt by the changes, we may try
provide a program to convert the media formats.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: choosing a file system to use on NAND/UBI

[Hamish Moffatt]

On Mon, Apr 07, 2008 at 10:13:49AM +0300, Artem Bityutskiy wrote:
> On Mon, 2008-04-07 at 15:12 +1000, Hamish Moffatt wrote:
> > Just to finish this old discussion, it's a 512Mb SLC part. ie not very
> > big. I have tried UBIFS and I am very pleased with it. Performance is
> > much better than JFFS2, which was slow to mount and slow during early
> > reads (even when the image was processed with sumtool).
> 
> "Mb" looks like Maga-bit, is that right?

Sorry I should've said 512MiB perhaps: 512 megabytes. 

UBI attach time appears to be about 6 seconds.

[    0.960000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.970000] Scanning device for bad blocks
[    1.020000] Bad eraseblock 494 at 0x03dc0000
[    1.110000] Bad eraseblock 1300 at 0x0a280000
[    1.240000] Bad eraseblock 2554 at 0x13f40000
[    1.280000] Bad eraseblock 2923 at 0x16d60000
[    1.330000] Bad eraseblock 3349 at 0x1a2a0000
[    1.370000] Bad eraseblock 3790 at 0x1d9c0000
[    1.410000] cmdlinepart partition parsing not available
[    7.210000] UBI: attached mtd9 to ubi0
[    7.210000] UBI: MTD device name:            "gen_nand.0"
[    7.220000] UBI: MTD device size:            512 MiB
[    7.220000] UBI: physical eraseblock size:   131072 bytes (128 KiB)
[    7.230000] UBI: logical eraseblock size:    129024 bytes
[    7.240000] UBI: number of good PEBs:        4090
[    7.240000] UBI: number of bad PEBs:         6
[    7.250000] UBI: smallest flash I/O unit:    2048
[    7.250000] UBI: VID header offset:          512 (aligned 512)
[    7.260000] UBI: data offset:                2048
[    7.260000] UBI: max. allowed volumes:       128
[    7.270000] UBI: wear-leveling threshold:    4096
[    7.270000] UBI: number of internal volumes: 1
[    7.270000] UBI: number of user volumes:     4
[    7.280000] UBI: available PEBs:             0
[    7.280000] UBI: total number of reserved PEBs: 4090
[    7.290000] UBI: number of PEBs reserved for bad PEB handling: 40
[    7.290000] UBI: max/mean erase counter: 41/1
[    7.300000] UBI: background thread "ubi_bgt0d" started, PID 619

Mounting the 128MiB root volume (ubifs) is taking 0.35 seconds:

# time mount -o ro -t ubifs ubi0:rootA /mnt
[  404.390000] UBIFS: mounted UBI device 0, volume 0
[  404.400000] UBIFS: mounted read-only
[  404.400000] UBIFS: minimal I/O unit size:   2048 bytes
[  404.400000] UBIFS: logical eraseblock size: 129024 bytes (126 KiB)
[  404.410000] UBIFS: file system size:        132894720 bytes (129780 KiB, 126 MiB, 1030 LEBs)
[  404.420000] UBIFS: journal size:            9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
[  404.430000] UBIFS: data journal heads:      1
[  404.430000] UBIFS: default compressor:      zlib
real    0m 0.34s
user    0m 0.00s
sys     0m 0.35s

which is fine. Although if there was any way to speed it up I would be
interested, particularly the UBI attach time.

I switched my UBIFS from the default lzo to zlib compression, as the
resulting images (from mkfs.ubifs) were smaller. Is there any reason to
prefer the default lzo?


thanks,
Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: choosing a file system to use on NAND/UBI

[Hamish Moffatt]

On Mon, Apr 07, 2008 at 05:32:27PM +1000, Hamish Moffatt wrote:
> UBI attach time appears to be about 6 seconds.
> 
> [    0.960000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
> [    0.970000] Scanning device for bad blocks
> [    1.020000] Bad eraseblock 494 at 0x03dc0000
> [    1.110000] Bad eraseblock 1300 at 0x0a280000
> [    1.240000] Bad eraseblock 2554 at 0x13f40000
> [    1.280000] Bad eraseblock 2923 at 0x16d60000
> [    1.330000] Bad eraseblock 3349 at 0x1a2a0000
> [    1.370000] Bad eraseblock 3790 at 0x1d9c0000
> [    1.410000] cmdlinepart partition parsing not available
> [    7.210000] UBI: attached mtd9 to ubi0

Less with some NAND glue improvements.

[    0.960000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
[    0.970000] Scanning device for bad blocks
[    1.010000] Bad eraseblock 494 at 0x03dc0000
[    1.060000] Bad eraseblock 1300 at 0x0a280000
[    1.140000] Bad eraseblock 2554 at 0x13f40000
[    1.170000] Bad eraseblock 2923 at 0x16d60000
[    1.200000] Bad eraseblock 3349 at 0x1a2a0000
[    1.230000] Bad eraseblock 3790 at 0x1d9c0000
[    1.260000] cmdlinepart partition parsing not available
[    6.910000] UBI: attached mtd9 to ubi0



Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: choosing a file system to use on NAND/UBI

[Artem Bityutskiy]

Hi

On Mon, 2008-04-07 at 17:32 +1000, Hamish Moffatt wrote:
> UBI attach time appears to be about 6 seconds.

Looks really a lot. We had 2 seconds for 1GiB flash on OLPC, but OLPC
has fast flash and fast CPU. I guess you have slow flash.

> [    0.960000] NAND device: Manufacturer ID: 0xec, Chip ID: 0xdc (Samsung NAND 512MiB 3,3V 8-bit)
> [    0.970000] Scanning device for bad blocks
> [    1.020000] Bad eraseblock 494 at 0x03dc0000
> [    1.110000] Bad eraseblock 1300 at 0x0a280000
> [    1.240000] Bad eraseblock 2554 at 0x13f40000
> [    1.280000] Bad eraseblock 2923 at 0x16d60000
> [    1.330000] Bad eraseblock 3349 at 0x1a2a0000
> [    1.370000] Bad eraseblock 3790 at 0x1d9c0000
> [    1.410000] cmdlinepart partition parsing not available
> [    7.210000] UBI: attached mtd9 to ubi0
> [    7.210000] UBI: MTD device name:            "gen_nand.0"
> [    7.220000] UBI: MTD device size:            512 MiB
> [    7.220000] UBI: physical eraseblock size:   131072 bytes (128 KiB)
> [    7.230000] UBI: logical eraseblock size:    129024 bytes
> [    7.240000] UBI: number of good PEBs:        4090
> [    7.240000] UBI: number of bad PEBs:         6
> [    7.250000] UBI: smallest flash I/O unit:    2048
> [    7.250000] UBI: VID header offset:          512 (aligned 512)
> [    7.260000] UBI: data offset:                2048
> [    7.260000] UBI: max. allowed volumes:       128
> [    7.270000] UBI: wear-leveling threshold:    4096
> [    7.270000] UBI: number of internal volumes: 1
> [    7.270000] UBI: number of user volumes:     4
> [    7.280000] UBI: available PEBs:             0
> [    7.280000] UBI: total number of reserved PEBs: 4090
> [    7.290000] UBI: number of PEBs reserved for bad PEB handling: 40
> [    7.290000] UBI: max/mean erase counter: 41/1
> [    7.300000] UBI: background thread "ubi_bgt0d" started, PID 619
> 
> Mounting the 128MiB root volume (ubifs) is taking 0.35 seconds:
> 
> # time mount -o ro -t ubifs ubi0:rootA /mnt
> [  404.390000] UBIFS: mounted UBI device 0, volume 0
> [  404.400000] UBIFS: mounted read-only
> [  404.400000] UBIFS: minimal I/O unit size:   2048 bytes
> [  404.400000] UBIFS: logical eraseblock size: 129024 bytes (126 KiB)
> [  404.410000] UBIFS: file system size:        132894720 bytes (129780 KiB, 126 MiB, 1030 LEBs)
> [  404.420000] UBIFS: journal size:            9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
> [  404.430000] UBIFS: data journal heads:      1
> [  404.430000] UBIFS: default compressor:      zlib
> real    0m 0.34s
> user    0m 0.00s
> sys     0m 0.35s
> 
> which is fine. Although if there was any way to speed it up I would be
> interested, particularly the UBI attach time.

Yeah. Is there any way to increase read speed on driver level? UBI reads
1 NAND page of each eraseblock during scanning and this is the
bottleneck. It also checks CRC, so if the CPU is very slow, this may be
the bottleneck as well.

I have 2 quick ideas about how to improve scan speed, but I am not sure
if they will help.

1. Currently what UBI is doing is: read EC header, check it, read VID
header, check it. So we run mtd->read() 2 times (it might help to run it
1 time because EC and VID headers go one after the other): read EC and
VID headers in one go, check them. We might do this soon.

2. More complex optimization would be to split scanning on 2 processes -
one just reads VID/EC headers and puts them to a list, the other takes
them from the list and checks. So that we would split the scanning on
I/O and CPU bound processes. This should help, especially if the CPU is
not very fast and checking time is comparable to I/O speed. We will do
this when we have time.

Other ideas are:

3. Even more complex change. Teach UBI to dump all the mapping
information on the media before de-attaching. Then the scanning process
would have to find this data and that's it. Although this would not help
in case if there was an unclean reboot.

4. Finally, one  could invest money and develop UBI2. I would be
interested to participate. We have some ideas.

> I switched my UBIFS from the default lzo to zlib compression, as the
> resulting images (from mkfs.ubifs) were smaller. Is there any reason to
> prefer the default lzo?

Well, the only way is to use mkfs.ubifs for this. You may create an
empty image, put it to the media and that's it. Would you conceive
something else?

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: choosing a file system to use on NAND/UBI

[Artem Bityutskiy]

On Mon, 2008-04-07 at 17:32 +1000, Hamish Moffatt wrote:
> I switched my UBIFS from the default lzo to zlib compression, as the
> resulting images (from mkfs.ubifs) were smaller. Is there any reason to
> prefer the default lzo?

Just FYI, although zlib compresses better, it is slower as well.
According to Richard: 

"This is particularly useful for jffs2 where significant boot time
speedups (~10%) and file read speed improvements (~40%) are seen when
its used with only a slight drop in file compression ratio."

AFAIK, the figures were related to Nokia N800. I guess for "desktop" CPU
this would not make such a big difference.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: choosing a file system to use on NAND/UBI

[Hamish Moffatt]

On Mon, Apr 07, 2008 at 10:56:46AM +0300, Artem Bityutskiy wrote:
> On Mon, 2008-04-07 at 17:32 +1000, Hamish Moffatt wrote:
> > UBI attach time appears to be about 6 seconds.
> 
> Looks really a lot. We had 2 seconds for 1GiB flash on OLPC, but OLPC
> has fast flash and fast CPU. I guess you have slow flash.

I'm not sure about the flash (I will have to check tomorrow) but we
don't have a hardware NAND flash controller. The NAND is connected to
the expansion bus of our IXP4XX processor (ARM) and CE is controlled by
a GPIO.. so it's not the fastest. We use the plat_nand driver.

I reduced the time a little by connecting up the NAND ready function in
the driver and reducing the delays. It doesn't look like there is much
else to tweak in the code. The CPU is a 533MHz ARM.

> Yeah. Is there any way to increase read speed on driver level? UBI reads
> 1 NAND page of each eraseblock during scanning and this is the
> bottleneck. It also checks CRC, so if the CPU is very slow, this may be
> the bottleneck as well.

Is that the CRC of one whole page of each block?

> I have 2 quick ideas about how to improve scan speed, but I am not sure
> if they will help.
> 
> 1. Currently what UBI is doing is: read EC header, check it, read VID
> header, check it. So we run mtd->read() 2 times (it might help to run it
> 1 time because EC and VID headers go one after the other): read EC and
> VID headers in one go, check them. We might do this soon.

How much is read each time - just a few bytes? Are those reads
duplicated by the CRC check?

> 4. Finally, one  could invest money and develop UBI2. I would be
> interested to participate. We have some ideas.

I wish we could help but we are a very small company :-) I appreciate
your assistance with ubi and ubifs very much though.

> > I switched my UBIFS from the default lzo to zlib compression, as the
> > resulting images (from mkfs.ubifs) were smaller. Is there any reason to
> > prefer the default lzo?
> 
> Well, the only way is to use mkfs.ubifs for this. You may create an
> empty image, put it to the media and that's it. Would you conceive
> something else?

I mean, I am preparing my root file system on my host and using
mkfs.ubifs to generate an image which I write with ubiupdatevol. I found
that zlib produced a smaller image.

Also I found that the image can be compressed further with gzip or lzma.
Could ubiupdatevol support on-the-fly decompression? I will develop a
patch if I have time.. I already patched flashcp to do this once.


Thanks
Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: choosing a file system to use on NAND/UBI

[Artem Bityutskiy]

On Mon, 2008-04-07 at 21:20 +1000, Hamish Moffatt wrote:
> Is that the CRC of one whole page of each block?

Only of the headers, which are 64 bytes.

> > I have 2 quick ideas about how to improve scan speed, but I am not sure
> > if they will help.
> > 
> > 1. Currently what UBI is doing is: read EC header, check it, read VID
> > header, check it. So we run mtd->read() 2 times (it might help to run it
> > 1 time because EC and VID headers go one after the other): read EC and
> > VID headers in one go, check them. We might do this soon.
> 
> How much is read each time - just a few bytes? Are those reads
> duplicated by the CRC check?

Well, NAND allows reading it PAGE_SIZE units. So it reads whole page.
Your flash allows sub-page write, so UBI reads 2KiB for per each
eraseblock. Then The your CPU is doing ECC check.

> > Well, the only way is to use mkfs.ubifs for this. You may create an
> > empty image, put it to the media and that's it. Would you conceive
> > something else?
> 
> I mean, I am preparing my root file system on my host and using
> mkfs.ubifs to generate an image which I write with ubiupdatevol. I found
> that zlib produced a smaller image.

That's right, zlib compresses better then lzo. But as I said, it is
slower, and reading files are slower, because de-compression is slower
in case of zlib.

> Also I found that the image can be compressed further with gzip or lzma.

Yes, because of several factors:
1. UBIFS (and JFFS2) does not compress meta-data.
2. UBIFS (and JFFS2) compresses 4KiB chunks of data independently.
Compression is better when you compress large chunks (which happens when
you use gzip or lzma), rather then small chunks.
3. The image has paddings, like paddings to the end of NAND page or to
the end of eraseblock. They also introduce more compressible data for
gzip.

> Could ubiupdatevol support on-the-fly decompression?
Err, why?

>  I will develop a
> patch if I have time.. I already patched flashcp to do this once.

Sorry, I am not sure what you mean here as well. What did your patch do?

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: choosing a file system to use on NAND/UBI

[Artem Bityutskiy]

On Mon, 2008-04-07 at 15:15 +0300, Artem Bityutskiy wrote:
> Well, NAND allows reading it PAGE_SIZE units. So it reads whole page.
OH, what do I tell. It allows to read in NAND page size units of course.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: choosing a file system to use on NAND/UBI

[Hamish Moffatt]

On Mon, Apr 07, 2008 at 03:15:34PM +0300, Artem Bityutskiy wrote:
> On Mon, 2008-04-07 at 21:20 +1000, Hamish Moffatt wrote:
> > Also I found that the image can be compressed further with gzip or lzma.
> 
> Yes, because of several factors:
> 1. UBIFS (and JFFS2) does not compress meta-data.
> 2. UBIFS (and JFFS2) compresses 4KiB chunks of data independently.
> Compression is better when you compress large chunks (which happens when
> you use gzip or lzma), rather then small chunks.
> 3. The image has paddings, like paddings to the end of NAND page or to
> the end of eraseblock. They also introduce more compressible data for
> gzip.
> 
> > Could ubiupdatevol support on-the-fly decompression?
> Err, why?
> 
> >  I will develop a
> > patch if I have time.. I already patched flashcp to do this once.
> 
> Sorry, I am not sure what you mean here as well. What did your patch do?

My idea is that you can "ubiupdatevol /dev/ubi0_0 myimage.gz", which
ubiupdatevol would decompress on the fly. You could decompress it
somewhere first but it may not fit in RAM (and writing it to flash
temporarily would be slower).

Looking at the code I see the main issue is that you need to know the
file size before you start updating. I think this is possible with zlib
(flashcp needed to know the same).

Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: choosing a file system to use on NAND/UBI

[Artem Bityutskiy]

On Mon, 2008-04-07 at 22:22 +1000, Hamish Moffatt wrote:
> My idea is that you can "ubiupdatevol /dev/ubi0_0 myimage.gz", which
> ubiupdatevol would decompress on the fly. You could decompress it
> somewhere first but it may not fit in RAM (and writing it to flash
> temporarily would be slower).

So you want some kind of "live" updates? You have your device which is
running. And you want to update one of your UBI volumes. You copy the
update image to the device and want to use ubiupdatevol to put it to the
volume.

How do you copy the file? If you have network connection, may be you
could teach ubifs to take the image from the network? dwmw2 also
recently added an utility to mtd-utils.git which _seems_ to be doing
this, so you could teach it to update the volume which should be very
easy (see recv_image and serve_image in mtd-utils.git).

> Looking at the code I see the main issue is that you need to know the
> file size before you start updating.
Yeah, that is right. This is what UBI update ictl expects. But probably
gzip has this info somewhere in its meta-data.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: choosing a file system to use on NAND/UBI

[Jamie Lokier]

Artem Bityutskiy wrote:
> On Mon, 2008-04-07 at 22:22 +1000, Hamish Moffatt wrote:
> > My idea is that you can "ubiupdatevol /dev/ubi0_0 myimage.gz", which
> > ubiupdatevol would decompress on the fly. You could decompress it
> > somewhere first but it may not fit in RAM (and writing it to flash
> > temporarily would be slower).

> How do you copy the file? If you have network connection, may be you
> could teach ubifs to take the image from the network?

I have a similar application, but it's using NOR, JFFS2 and "flashcp"
to write; still, the principle is the same.

Flashing while reading from the network is dangerous if the network
fails in the middle, and it's the critical boot partition being
written.  In my application, this is not uncommon.

> > Looking at the code I see the main issue is that you need to know the
> > file size before you start updating.
> Yeah, that is right. This is what UBI update ictl expects. But probably
> gzip has this info somewhere in its meta-data.

The size is optional in the tiny gzip header, at the start of the
file.  I think files compressed with "gzip file" will have it, and
files compressed with "gzip <file >file.gz" won't.  Also, it is
limited to 4GiB-1.

-- Jamie

Re: choosing a file system to use on NAND/UBI

[Artem Bityutskiy]

On Mon, 2008-04-07 at 17:32 +1000, Hamish Moffatt wrote:
> which is fine. Although if there was any way to speed it up I would be
> interested, particularly the UBI attach time.

One more thing I could suggest is profiling UBI - this may help
improving it.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: choosing a file system to use on NAND/UBI

[Hamish Moffatt]

On Mon, Apr 07, 2008 at 03:44:19PM +0300, Artem Bityutskiy wrote:
> On Mon, 2008-04-07 at 22:22 +1000, Hamish Moffatt wrote:
> > My idea is that you can "ubiupdatevol /dev/ubi0_0 myimage.gz", which
> > ubiupdatevol would decompress on the fly. You could decompress it
> > somewhere first but it may not fit in RAM (and writing it to flash
> > temporarily would be slower).
> 
> So you want some kind of "live" updates? You have your device which is
> running. And you want to update one of your UBI volumes. You copy the
> update image to the device and want to use ubiupdatevol to put it to the
> volume.
> 
> How do you copy the file? If you have network connection, may be you
> could teach ubifs to take the image from the network? dwmw2 also
> recently added an utility to mtd-utils.git which _seems_ to be doing
> this, so you could teach it to update the volume which should be very
> easy (see recv_image and serve_image in mtd-utils.git).

Not possible in our case; our firmware image is a composite of the ubifs
root volume plus some firmware for other devices in our system. Plus we
want to check the md5sum of the parts before starting to ubiupdatevol.

> > Looking at the code I see the main issue is that you need to know the
> > file size before you start updating.
> Yeah, that is right. This is what UBI update ictl expects. But probably
> gzip has this info somewhere in its meta-data.

I think you need to seek to the end and then rewind to get a definite
answer.

In our previous application we had our root on compact flash, so we
could update using "zcat image.gz | dd of=/dev/hda1". And I once patched
flashcp to use zlib. Is it possible to use flashcp to UBI's emulated
/dev/mtdX device in place of ubiupdatevol?

Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>