UBI/ubifs problem

UBI/ubifs problem

[Ricard Wanderlof]

I posted this query yesterday, but the information I provided was rather 
short, so here's another attempt with more info which hopefully will 
trigger some response.

I have two ubifs images made with mkfs.ubifs:

-rw-rw-r-- 1 ricardw users 20256768 2012-02-16 11:12 rootfs.img
-rw-rw-r-- 1 ricardw users  2064384 2012-02-16 11:09 rwfs.img

On my target system I attach mtd5 to UBI and create two volumes:

ubiattach -m 5 /dev/ubi_ctrl
ubimkvol -N rootfs -n 4 -t static -s 25165824 /dev/ubi0
ubimkvol -N rwfs -n 5 -t dynamic -S 30 /dev/ubi0

30 LEBs in the flash I'm using is about 30 * 126 KiB = 3.7 MiB .

I then write the volumes with the image data

ubiupdatevol /dev/ubi0_4  ./rootfs.img
ubiupdatevol /dev/ubi0_5 ./rwfs.img

Now I mount ubi0:rootfs on /mnt/1:

mount -t ubifs ubi0:rootfs /mnt/1/

which results in

UBIFS: mounted UBI device 0, volume 4, name "rootfs"
UBIFS: mounted read-only
UBIFS: file system size:   23869440 bytes (23310 KiB, 22 MiB, 185 LEBs)
UBIFS: journal size:       9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: lzo
UBIFS: reserved for root:  0 bytes (0 KiB)

All is well so far. Now I attempt to mount ubi0:rwfs on /mnt/2:

mount -t ubifs ubi0:rwfs /mnt/2

which results in

UBIFS error (pid 2965): validate_sb: bad superblock, error 8
mount: Mounting ubi0:rwfs on /mnt/2 failed: Invalid argument

If I now remove the rwfs volume using ubirmvol, and create a new one that 
is 78 LEBs in size (smaller sizes yield the same error message), again 
ubiupdatevol the same rwfs.img image, and attempt to mount it, I get:

UBIFS: reserved for root:  0 bytes (0 KiB)
UBIFS: mounted UBI device 0, volume 5, name "rwfs"
UBIFS: file system size:   8644608 bytes (8442 KiB, 8 MiB, 67 LEBs)
UBIFS: journal size:       9033728 bytes (8822 KiB, 8 MiB, 71 LEBs)
UBIFS: media format:       w4/r0 (latest is w4/r0)
UBIFS: default compressor: lzo
UBIFS: reserved for root:  0 bytes (0 KiB)

The thing that really puzzles me here is that the 2 MiB ubifs image shows 
up as an 8 MiB 'file system size' when mounted. It does explain why 78 
LEBs are required though.

Furthermore, df says:

ubi0:rootfs              21212     17652      3560  83% /mnt/1
ubi0:rwfs                 7228       432      6796   6% /mnt/2

which correlates with the statistics from ubifs above, but it seems odd 
that I can't create a volume with more than 6% utilization. If I start to 
fill the volume with files after mounting, I can get up to 100% use.

At any rate, the error message is odd, especially given that ubiupdatevol 
didn't report any problems.

I've also tried using a static volume for rwfs, with the same results (and 
of course I can't write to it so I can't put more files on it in that 
case).

Even considering the overhead that the rootfs volume appears to need 
(image size is 20256768 and ubifs reports a 'file system size' of 
23869440), i.e. an overhead of 3612672 bytes), it doesn't add up; adding 
3612672 to the rwfs.img image size of 2064384 would result in a total size 
of 5677056, not 8644608.

Could there be compressed files in the rwfs.img image that somehow need to 
be expanded in the flash?

It's not that we're pressed for space and need the size, but I'd like to 
understand what is going on. I don't really think this is a bug, more 
something that I don't understand. I can't find anything on the mtd 
documentation pages that seem to relate to this though.

For the record, I'm using Linux 2.6.35 .

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: UBI/ubifs problem

[Ricard Wanderlof]

Yesterday I reported a problem I had when flashing a ubifs volume.

On Thu, 16 Feb 2012, Ricard Wanderlof wrote:

> ...
> UBIFS error (pid 2965): validate_sb: bad superblock, error 8
> mount: Mounting ubi0:rwfs on /mnt/2 failed: Invalid argument
> ...

I think I've figured this out part way. I've been specifying a 
--max-leb-count of 1000 to mkfs.ubifs, thinking that it basically just 
sets a maximum size for the resulting file system. But it seems that it 
also sets some sort of bounds for how much space it expects to take up in 
the volume. If I reduce the --max-leb-count to the size of the target 
volume, I get no errors and the file system mounts fine.

That being so, I think the resulting error message is odd, which indicates 
there is a bug somewhere, probably in mkfs.ubifs ?

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: UBI/ubifs problem

[Artem Bityutskiy]

On Fri, 2012-02-17 at 11:13 +0100, Ricard Wanderlof wrote:
> Yesterday I reported a problem I had when flashing a ubifs volume.
> 
> On Thu, 16 Feb 2012, Ricard Wanderlof wrote:
> 
> > ...
> > UBIFS error (pid 2965): validate_sb: bad superblock, error 8
> > mount: Mounting ubi0:rwfs on /mnt/2 failed: Invalid argument
> > ...
> 
> I think I've figured this out part way. I've been specifying a 
> --max-leb-count of 1000 to mkfs.ubifs, thinking that it basically just 
> sets a maximum size for the resulting file system. But it seems that it 
> also sets some sort of bounds for how much space it expects to take up in 
> the volume. If I reduce the --max-leb-count to the size of the target 
> volume, I get no errors and the file system mounts fine.

When you specify --max-leb-count=1000, but not specify the journal size,
mkfs.ubifs defines the journal size to be around 12% of that, but not
larger than 8MiB. This sets the low UBI volume size boundary.

So basically the error you get means that your UBI volume is too small
to fit the journal.

To see the journal size of your image use -v mkfs.ubifs option when you
generate it.

You can define your own journal size using the --jrn-size mkfs.ubifs
option.

> That being so, I think the resulting error message is odd, which indicates 
> there is a bug somewhere, probably in mkfs.ubifs ?

I agree that the message is odd. Below is the patch which I'll push to
linux-ubifs.git as soon as git.infradead.org is available (it is down
now for the reasons I do not know). I am open to additional or different
suggestions. Thanks!

P.S. I did not even compile - test the below patch, but will push out a
tested version.

From: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>
Date: Wed, 7 Mar 2012 16:29:45 +0200
Subject: [PATCH] UBIFS: improve error messages

Ricard complaints that the following error message is odd:

"UBIFS error (pid 1578): validate_sb: bad superblock, error 8"

and he is right. This patch improves the error messages a bit and makes
them more user-friendly.

Reported-by: Ricard Wanderlof <ricard.wanderlof@axis.com>
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>
---
 fs/ubifs/sb.c |   19 ++++++++++++++-----
 1 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/fs/ubifs/sb.c b/fs/ubifs/sb.c
index 6094c5a..4d4ef12 100644
--- a/fs/ubifs/sb.c
+++ b/fs/ubifs/sb.c
@@ -410,13 +410,23 @@ static int validate_sb(struct ubifs_info *c, struct ubifs_sb_node *sup)
 	}
 
 	if (c->main_lebs < UBIFS_MIN_MAIN_LEBS) {
-		err = 7;
+		ubifs_err("too few main LEBs count %d, must be at least %d",
+			  c->main_lebs, UBIFS_MIN_MAIN_LEBS);
 		goto failed;
 	}
 
-	if (c->max_bud_bytes < (long long)c->leb_size * UBIFS_MIN_BUD_LEBS ||
-	    c->max_bud_bytes > (long long)c->leb_size * c->main_lebs) {
-		err = 8;
+	max_bytes = (long long)c->leb_size * UBIFS_MIN_BUD_LEBS;
+	if (c->max_bud_bytes < max_bytes) {
+		ubifs_err("too small journal (%lld bytes), must be at least ""
+			  %lld bytes",  c->max_bud_bytes, max_bytes);
+		goto failed;
+	}
+
+	max_bytes = (long long)c->leb_size * c->main_lebs;
+	if (c->max_bud_bytes > max_bytes) {
+		ubifs_err("too large journal size (%lld bytes), only %lld bytes"
+			  "available in the main area",
+			  c->max_bud_bytes, max_bytes);
 		goto failed;
 	}
 
@@ -450,7 +460,6 @@ static int validate_sb(struct ubifs_info *c, struct ubifs_sb_node *sup)
 		goto failed;
 	}
 
-	max_bytes = c->main_lebs * (long long)c->leb_size;
 	if (c->rp_size < 0 || max_bytes < c->rp_size) {
 		err = 14;
 		goto failed;
-- 
1.7.9.1

-- 
Best Regards,
Artem Bityutskiy

Re: UBI/ubifs problem

[Ricard Wanderlof]

On Wed, 7 Mar 2012, Artem Bityutskiy wrote:

> When you specify --max-leb-count=1000, but not specify the journal size,
> mkfs.ubifs defines the journal size to be around 12% of that, but not
> larger than 8MiB. This sets the low UBI volume size boundary.
>
> So basically the error you get means that your UBI volume is too small
> to fit the journal.
>
> To see the journal size of your image use -v mkfs.ubifs option when you
> generate it.
>
> You can define your own journal size using the --jrn-size mkfs.ubifs 
> option.

Is it possible to get mkfs.ubifs to create an image without specifying the 
max-leb-count ? I.e. just to create an image that is as big as it has to 
be. Admittedly, one would have to figure out the size of the resulting 
volume afterwards, to provide an allocation size for ubimkvol or ubinize, 
so perhaps it would help much.

>> That being so, I think the resulting error message is odd, which indicates
>> there is a bug somewhere, probably in mkfs.ubifs ?
>
> I agree that the message is odd. Below is the patch which I'll push to
> linux-ubifs.git as soon as git.infradead.org is available (it is down
> now for the reasons I do not know). I am open to additional or different
> suggestions. Thanks!
>
> P.S. I did not even compile - test the below patch, but will push out a
> tested version.

Thanks for the patch, I'll try to test it.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: UBI/ubifs problem

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 596 bytes --]

On Wed, 2012-03-07 at 17:26 +0100, Ricard Wanderlof wrote:
> Is it possible to get mkfs.ubifs to create an image without specifying the 
> max-leb-count ? I.e. just to create an image that is as big as it has to 
> be.

Has to be? :-) Not sure how to find this out. Basically the idea was
that you know how big is has to be and translate that to
--max-leb-count.

> > P.S. I did not even compile - test the below patch, but will push out a
> > tested version.
> 
> Thanks for the patch, I'll try to test it.

It broke compilation, the working one is attached.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: 0001-UBIFS-improve-error-messages.patch --]
[-- Type: text/x-patch, Size: 1954 bytes --]

>From 5a1f36c90c9b21a7aa31c29a1926b376dd6a11cf Mon Sep 17 00:00:00 2001
From: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>
Date: Wed, 7 Mar 2012 16:29:45 +0200
Subject: [PATCH] UBIFS: improve error messages

Ricard complaints that the following error message is odd:

"UBIFS error (pid 1578): validate_sb: bad superblock, error 8"

and he is right. This patch improves the error messages a bit and makes
them more user-friendly.

Reported-by: Ricard Wanderlof <ricard.wanderlof@axis.com>
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>
---
 fs/ubifs/sb.c |   19 ++++++++++++++-----
 1 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/fs/ubifs/sb.c b/fs/ubifs/sb.c
index 6094c5a..771f7fb 100644
--- a/fs/ubifs/sb.c
+++ b/fs/ubifs/sb.c
@@ -410,13 +410,23 @@ static int validate_sb(struct ubifs_info *c, struct ubifs_sb_node *sup)
 	}
 
 	if (c->main_lebs < UBIFS_MIN_MAIN_LEBS) {
-		err = 7;
+		ubifs_err("too few main LEBs count %d, must be at least %d",
+			  c->main_lebs, UBIFS_MIN_MAIN_LEBS);
 		goto failed;
 	}
 
-	if (c->max_bud_bytes < (long long)c->leb_size * UBIFS_MIN_BUD_LEBS ||
-	    c->max_bud_bytes > (long long)c->leb_size * c->main_lebs) {
-		err = 8;
+	max_bytes = (long long)c->leb_size * UBIFS_MIN_BUD_LEBS;
+	if (c->max_bud_bytes < max_bytes) {
+		ubifs_err("too small journal (%lld bytes), must be at least "
+			  "%lld bytes",  c->max_bud_bytes, max_bytes);
+		goto failed;
+	}
+
+	max_bytes = (long long)c->leb_size * c->main_lebs;
+	if (c->max_bud_bytes > max_bytes) {
+		ubifs_err("too large journal size (%lld bytes), only %lld bytes"
+			  "available in the main area",
+			  c->max_bud_bytes, max_bytes);
 		goto failed;
 	}
 
@@ -450,7 +460,6 @@ static int validate_sb(struct ubifs_info *c, struct ubifs_sb_node *sup)
 		goto failed;
 	}
 
-	max_bytes = c->main_lebs * (long long)c->leb_size;
 	if (c->rp_size < 0 || max_bytes < c->rp_size) {
 		err = 14;
 		goto failed;
-- 
1.7.9.1

Re: UBI/ubifs problem

[Paul Parsons]

--- On Wed, 7/3/12, Ricard Wanderlof <ricard.wanderlof@axis.com> wrote:
> Is it possible to get mkfs.ubifs to create an image without
> specifying the max-leb-count ? I.e. just to create an image
> that is as big as it has to be. Admittedly, one would have
> to figure out the size of the resulting volume afterwards,
> to provide an allocation size for ubimkvol or ubinize, so
> perhaps it would help much.

When I first started to try out UBI/UBIFS I had problems understanding the
arithmetic, and it took several attempts to get it working.

In the end I created a simple shell script to do all the hard work for me.

Given the UBI MTD partition size, PEB size, and desired volume sizes, the
script would do the rest:

#!/bin/bash

#
#	Create UBI for 125MiB MTD on NOR flash, consisting of:
#
#		UBI overhead.
#		Volume 0: rootfs, 75MiB less UBI overhead.
#		Volume 1: homefs, 50MiB.
#

declare -r rootimg="rootfs.img"
declare -r homeimg="homefs.img"
declare -r ipaqcfg="ipaq.cfg"

declare -ir KiB=1024
declare -ir MiB=$((1024*1024))

declare -i UBI=$((125*MiB))			# UBI size
declare -i Sp=$((256*KiB))			# PEB size
declare -i P=$((UBI/Sp))			# Total number of PEBs
declare -i O=128				# EC header + VID header overhead
declare -i Sl=$((Sp-O))				# LEB size
declare -i B=0					# NOR does not have bad PEBs
declare -i U=$((((B+4)*Sp)+(O*(P-B-4))))	# UBI overhead
declare -i Ur=$(((U+Sp-1)&(~(Sp-1))))		# UBI overhead rounded up to next PEB size

declare -i ROOTFS=$(((75*MiB)-Ur))		# rootfs size
declare -i HOMEFS=$((50*MiB))			# homefs size

cmd="mkfs.ubifs --root=rootfs --output=${rootimg} -m 1 -e ${Sl} -c $((ROOTFS/Sp)) -x none"
echo ${cmd}
${cmd} || exit 1

cmd="mkfs.ubifs --root=homefs --output=${homeimg} -m 1 -e ${Sl} -c $((HOMEFS/Sp)) -x none"
echo ${cmd}
${cmd} || exit 1

rm -f ${ipaqcfg}
echo "[rootfs]" >> ${ipaqcfg}
echo "mode=ubi" >> ${ipaqcfg}
echo "image=${rootimg}" >> ${ipaqcfg}
echo "vol_id=0" >> ${ipaqcfg}
echo "vol_size=$((((ROOTFS)/Sp)*Sl))" >> ${ipaqcfg}
echo "vol_type=dynamic" >> ${ipaqcfg}
echo "vol_name=rootfs" >> ${ipaqcfg}
echo "[homefs]" >> ${ipaqcfg}
echo "mode=ubi" >> ${ipaqcfg}
echo "image=${homeimg}" >> ${ipaqcfg}
echo "vol_id=1" >> ${ipaqcfg}
echo "vol_size=$((((HOMEFS)/Sp)*Sl))" >> ${ipaqcfg}
echo "vol_type=dynamic" >> ${ipaqcfg}
echo "vol_name=homefs" >> ${ipaqcfg}

cmd="ubinize -o ipaq.ubi -p ${Sp} -m 1 -v ${ipaqcfg}"
echo ${cmd}
${cmd} || exit 1

It works for me. Hopefully it might be useful to others.

Re: UBI/ubifs problem

[Ricard Wanderlof]

On Wed, 7 Mar 2012, Artem Bityutskiy wrote:

> On Wed, 2012-03-07 at 17:26 +0100, Ricard Wanderlof wrote:
>> Is it possible to get mkfs.ubifs to create an image without specifying the
>> max-leb-count ? I.e. just to create an image that is as big as it has to
>> be.
>
> Has to be? :-) Not sure how to find this out. Basically the idea was
> that you know how big is has to be and translate that to
> --max-leb-count.

I think what I'm thinking about is that in the specialized case of ubifs 
used for a read-only file system, such as a root file system (without 
/etc, /var, and so on), it would be possible for mkfs.ubifs to create a 
file system, and use as many LEBs as it needs for the actual data. On a 
read-only file system there doesn't even have to be a journal, although I 
don't know if that is possible.

In that case one could simply feed the required directory structure to 
mkfs.ubifs, and it would create the file system image, and report how many 
LEBs are required in the flash.

> It broke compilation, the working one is attached.

Ok, I'll have a look at the new one instead.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: UBI/ubifs problem

[Artem Bityutskiy]

On Thu, 2012-03-08 at 06:09 +0100, Ricard Wanderlof wrote:
> On Wed, 7 Mar 2012, Artem Bityutskiy wrote:
> 
> > On Wed, 2012-03-07 at 17:26 +0100, Ricard Wanderlof wrote:
> >> Is it possible to get mkfs.ubifs to create an image without specifying the
> >> max-leb-count ? I.e. just to create an image that is as big as it has to
> >> be.
> >
> > Has to be? :-) Not sure how to find this out. Basically the idea was
> > that you know how big is has to be and translate that to
> > --max-leb-count.
> 
> I think what I'm thinking about is that in the specialized case of ubifs 
> used for a read-only file system, such as a root file system (without 
> /etc, /var, and so on), it would be possible for mkfs.ubifs to create a 
> file system, and use as many LEBs as it needs for the actual data. On a 
> read-only file system there doesn't even have to be a journal, although I 
> don't know if that is possible.

Well, I'd accept a patch which added a --optimize-for-ro option which
would prepare a file-system with the smallest journal size.

-- 
Best Regards,
Artem Bityutskiy

Re: UBI/ubifs problem

[Ricard Wanderlof]

On Fri, 9 Mar 2012, Artem Bityutskiy wrote:

>> I think what I'm thinking about is that in the specialized case of ubifs
>> used for a read-only file system, such as a root file system (without
>> /etc, /var, and so on), it would be possible for mkfs.ubifs to create a
>> file system, and use as many LEBs as it needs for the actual data. On a
>> read-only file system there doesn't even have to be a journal, although I
>> don't know if that is possible.
>
> Well, I'd accept a patch which added a --optimize-for-ro option which
> would prepare a file-system with the smallest journal size.

Can the journal be eliminated completely or does something in ubifs 
require there to be a journal at all times, albeit small?

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: UBI/ubifs problem

[Artem Bityutskiy]

On Fri, 2012-03-09 at 14:45 +0100, Ricard Wanderlof wrote:
> On Fri, 9 Mar 2012, Artem Bityutskiy wrote:
> 
> >> I think what I'm thinking about is that in the specialized case of ubifs
> >> used for a read-only file system, such as a root file system (without
> >> /etc, /var, and so on), it would be possible for mkfs.ubifs to create a
> >> file system, and use as many LEBs as it needs for the actual data. On a
> >> read-only file system there doesn't even have to be a journal, although I
> >> don't know if that is possible.
> >
> > Well, I'd accept a patch which added a --optimize-for-ro option which
> > would prepare a file-system with the smallest journal size.
> 
> Can the journal be eliminated completely or does something in ubifs 
> require there to be a journal at all times, albeit small?

I think currently at least 5 LEBs are required:

/* Minimum number of logical eraseblocks in the log */
#define UBIFS_MIN_LOG_LEBS 2
/* Minimum number of bud logical eraseblocks (one for each head) */
#define UBIFS_MIN_BUD_LEBS 3
/* Minimum number of journal logical eraseblocks */
#define UBIFS_MIN_JNL_LEBS (UBIFS_MIN_LOG_LEBS + UBIFS_MIN_BUD_LEBS)

-- 
Best Regards,
Artem Bityutskiy