Grow UBI device?

Grow UBI device?

[Wolfgang Denk]

Hi,

I'm looking for a clever way to migrate devices (in the field) from
JFFS2 to UBI/UBIFS.  In the end result, I want to have single UBI
device (which covers the whole NAND chip) with several volumes used
with UBIFS.  Unfortunately I must keep some of the existing data.

It would be nice if I could start with a UBI device that is smaller
than the NAND, so I can keep an existing MTD partition with a JFFS2
file system. After copying the data from JFFS2 to a UBIFS volume, I
would like to free and reuse the space of this JFFS2 partition.

Is there a way to "grow" the existing UBI device so that it now also
covers the rest of the NAND chip?  Or is my only option to create a
second UBI device?

Thanks in advance.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Unix is like a toll road on which you have to stop every 50  feet  to
pay another nickel. But hey! You only feel 5 cents poorer each time.
                 - Larry Wall in <1992Aug13.192357.15731@netlabs.com>

Re: Grow UBI device?

[Matthieu CASTET]

Le Tue, 8 Oct 2013 20:28:46 +0100,
Wolfgang Denk <wd@denx.de> a écrit :

> so I can keep an existing MTD partition with a JFFS2
> file system. After copying the data from JFFS2 to a UBIFS volume, I
> would like to free and reuse the space of this JFFS2 partition.
> 
> Is there a way to "grow" the existing UBI device so that it now also
> covers the rest of the NAND chip?  Or is my only option to create a
> second UBI device?
I don't remeber exactly but I think UBI (without fastmap) doesn't know
the size of the mtd partition.
The scan scan what you give to it.

A way to "grow an UBI device could be to append (before or after)
flash formated with ubi format (with --no-volume-table and the correct
--image-seq= and same other param).
If you give erased flash may be the linux driver will format it.



Matthieu

Re: Grow UBI device?

[Ricard Wanderlof]

On Tue, 8 Oct 2013, Wolfgang Denk wrote:

> Hi,
>
> I'm looking for a clever way to migrate devices (in the field) from
> JFFS2 to UBI/UBIFS.  In the end result, I want to have single UBI
> device (which covers the whole NAND chip) with several volumes used
> with UBIFS.  Unfortunately I must keep some of the existing data.
>
> It would be nice if I could start with a UBI device that is smaller
> than the NAND, so I can keep an existing MTD partition with a JFFS2
> file system. After copying the data from JFFS2 to a UBIFS volume, I
> would like to free and reuse the space of this JFFS2 partition.
>
> Is there a way to "grow" the existing UBI device so that it now also 
> covers the rest of the NAND chip?  Or is my only option to create a 
> second UBI device?

UBI devices are 'attached' to existing mtd partitions. So to grow a UBI 
device, you need to detach it from the partition, re-partition the flash 
(and I'm not sure if this step is even possible as it involves 
unregistering existing mtd partitions), then attach a UBI device to the a 
new partition.

Come to think of it, I'm not sure that would work anyway, as the 
resulting, new, enlarged partition would partly contain UBI data and 
partly old jffs2 data. I'm not sure what UBI does when it encounters 
incorrect data, if it just erases the relevant blocks and formats them for 
its own use, or if it barfs completely and just bails out complaining that 
the partition does not contain UBI data. If the relevant blocks were 
erased, then I think UBI would simply concede that the a previous erase 
attempt had been prematurely aborted, and re-erase the blocks and write 
its headers, so perhaps that is something to try. At any rate it involves 
at least some mild trickery (erase the blocks that previously contained 
jffs2 data).

It's probably easier to create a new UBI device by attaching it to the 
partition previously used by jffs2, after erasing it, but multiple UBI 
devices across a flash partly defeats the purpose of wear-leveling which 
UBI provides.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

RE: Grow UBI device?

[Gupta, Pekon]

> From: Ricard Wanderlof
> > On Tue, 8 Oct 2013, Wolfgang Denk wrote:
> > I'm looking for a clever way to migrate devices (in the field) from
> > JFFS2 to UBI/UBIFS.  In the end result, I want to have single UBI
> > device (which covers the whole NAND chip) with several volumes used
> > with UBIFS.  Unfortunately I must keep some of the existing data.
> >
> > It would be nice if I could start with a UBI device that is smaller
> > than the NAND, so I can keep an existing MTD partition with a JFFS2
> > file system. After copying the data from JFFS2 to a UBIFS volume, I
> > would like to free and reuse the space of this JFFS2 partition.
> >
That seems to be a nice move, but where would you copy the existing
JFFS2 data ?
(a) In memory, then there is threat of losing the data due to
  power-failure, and you should have enough RAM.
(b) copy to target UBIFS volume/partition. So assumption is that target
  UBI volume has enough space and is writable.
I'm assuming (b) is more workable .. 


> > Is there a way to "grow" the existing UBI device so that it now also
> > covers the rest of the NAND chip?  Or is my only option to create a
> > second UBI device?
> 
> UBI devices are 'attached' to existing mtd partitions. So to grow a UBI
> device, you need to detach it from the partition, re-partition the flash
> (and I'm not sure if this step is even possible as it involves
> unregistering existing mtd partitions), then attach a UBI device to the a
> new partition.
> 
Yes, currently registering and de-registering of MTD partitions are
done as part of nand_probe() and nand_remove().
If we do on-the-fly registering / de-registering of MTD device, then we
also need to tweak the mtd partition information, which will change
- partition base offset and 
- shift masks (to calculate offset within the partition)
which is ok, because they are not used by controller drivers directly.


> Come to think of it, I'm not sure that would work anyway, as the
> resulting, new, enlarged partition would partly contain UBI data and
> partly old jffs2 data. I'm not sure what UBI does when it encounters
> incorrect data, if it just erases the relevant blocks and formats them for
> its own use, or if it barfs completely and just bails out complaining that
> the partition does not contain UBI data. If the relevant blocks were
> erased, then I think UBI would simply concede that the a previous erase
> attempt had been prematurely aborted, and re-erase the blocks and write
> its headers, so perhaps that is something to try. At any rate it involves
> at least some mild trickery (erase the blocks that previously contained
> jffs2 data).
> 
Yes, during mounting of UBIFS volumes, UBI checks for erase-header
on first-page of every block. 
- If it's able to find a valid erase-header, with correct image_seq field,
then it attaches it to its pool.
- Otherwise, if the valid erase-header was not found, then it assumes
 then it ubi will try re-erasing it. And add it to its pool of free-blocks.

So, effectively you don't even need to erase the partition to convert
it into UBI volume, you just try attaching it as UBI volume and UBI
should erase it for you. (though flagging some warnings on the way).


> It's probably easier to create a new UBI device by attaching it to the
> partition previously used by jffs2, after erasing it, but multiple UBI
> devices across a flash partly defeats the purpose of wear-leveling which
> UBI provides.
>
True but this is good for on-field devices which are still on JFFS2 and
want to migrate to UBIFS for same reasons. Also, this would help in
other areas like recovery from power-failures.
 

with regards, pekon

Re: Grow UBI device?

[Wolfgang Denk]

Dear "Gupta, Pekon",

In message <20980858CB6D3A4BAE95CA194937D5E73EA20350@DBDE04.ent.ti.com> you wrote:
>
> > > It would be nice if I could start with a UBI device that is smaller
> > > than the NAND, so I can keep an existing MTD partition with a JFFS2
> > > file system. After copying the data from JFFS2 to a UBIFS volume, I
> > > would like to free and reuse the space of this JFFS2 partition.
> > >
> That seems to be a nice move, but where would you copy the existing
> JFFS2 data ?

Well, I wrote: "copy[...] the data from JFFS2 to a UBIFS volume".

> (a) In memory, then there is threat of losing the data due to
>   power-failure, and you should have enough RAM.
> (b) copy to target UBIFS volume/partition. So assumption is that target
>   UBI volume has enough space and is writable.
> I'm assuming (b) is more workable .. 

...which is what I'm after.

> > Come to think of it, I'm not sure that would work anyway, as the
> > resulting, new, enlarged partition would partly contain UBI data and
> > partly old jffs2 data. I'm not sure what UBI does when it encounters
> > incorrect data, if it just erases the relevant blocks and formats them for
> > its own use, or if it barfs completely and just bails out complaining that
> > the partition does not contain UBI data. If the relevant blocks were
> > erased, then I think UBI would simply concede that the a previous erase
> > attempt had been prematurely aborted, and re-erase the blocks and write
> > its headers, so perhaps that is something to try. At any rate it involves
> > at least some mild trickery (erase the blocks that previously contained
> > jffs2 data).
> > 
> Yes, during mounting of UBIFS volumes, UBI checks for erase-header
> on first-page of every block. 

Mounting UBIFS volumes should not be a problem - the existing ones are
not located in the newly added area of the NAND.  The question however
is what the UBI layer itself does, if I for example try to create a
new volume which then would allocate blocks from the newly added
space.

> So, effectively you don't even need to erase the partition to convert
> it into UBI volume, you just try attaching it as UBI volume and UBI
> should erase it for you. (though flagging some warnings on the way).

Are you sure about that?

Well, I guess I'll have to try it out in the end...


Thanks!

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Deliver yesterday, code today, think tomorrow."

Re: Grow UBI device?

[Ricard Wanderlof]

On Wed, 9 Oct 2013, Wolfgang Denk wrote:

>>> Come to think of it, I'm not sure that would work anyway, as the
>>> resulting, new, enlarged partition would partly contain UBI data and
>>> partly old jffs2 data. I'm not sure what UBI does when it encounters
>>> incorrect data, if it just erases the relevant blocks and formats them for
>>> its own use, or if it barfs completely and just bails out complaining that
>>> the partition does not contain UBI data. If the relevant blocks were
>>> erased, then I think UBI would simply concede that the a previous erase
>>> attempt had been prematurely aborted, and re-erase the blocks and write
>>> its headers, so perhaps that is something to try. At any rate it involves
>>> at least some mild trickery (erase the blocks that previously contained
>>> jffs2 data).
>>>
>> Yes, during mounting of UBIFS volumes, UBI checks for erase-header
>> on first-page of every block.
>
> Mounting UBIFS volumes should not be a problem - the existing ones are
> not located in the newly added area of the NAND.  The question however
> is what the UBI layer itself does, if I for example try to create a
> new volume which then would allocate blocks from the newly added
> space.

UBI scans the whole partition when the attachement is made to the mtd 
partition. So if you'd get any error message regarding blocks that lack 
UBI headers it would be then, not when you subsequently try to create a 
volume in the partition. By that time the volume should be "formatted" 
with valid UBI headers.

>> So, effectively you don't even need to erase the partition to convert
>> it into UBI volume, you just try attaching it as UBI volume and UBI
>> should erase it for you. (though flagging some warnings on the way).
>
> Are you sure about that?
>
> Well, I guess I'll have to try it out in the end...

I just tested it since I have a UBI system just beside me, by erasing and 
then writing zeros to a block (just to put non-UBI data in it) and 
restarting the system (the partition in question contains the volume with 
the root file system in it, so it is attached by the kernel during its 
boot). The result was that UBI didn't give any error or warning messages 
during boot, but a subsequent nanddump after boot revealed that the 
zeroed blocks now contained valid UBI headers, so, without looking at the 
source, it seems that UBI really does manage this fairly silently.

/Ricard
-- 
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30

Re: Grow UBI device?

[Richard Weinberger]

On Wed, Oct 9, 2013 at 5:45 PM, Ricard Wanderlof
<ricard.wanderlof@axis.com> wrote:
>
> On Wed, 9 Oct 2013, Wolfgang Denk wrote:
>
>>>> Come to think of it, I'm not sure that would work anyway, as the
>>>> resulting, new, enlarged partition would partly contain UBI data and
>>>> partly old jffs2 data. I'm not sure what UBI does when it encounters
>>>> incorrect data, if it just erases the relevant blocks and formats them
>>>> for
>>>> its own use, or if it barfs completely and just bails out complaining
>>>> that
>>>> the partition does not contain UBI data. If the relevant blocks were
>>>> erased, then I think UBI would simply concede that the a previous erase
>>>> attempt had been prematurely aborted, and re-erase the blocks and write
>>>> its headers, so perhaps that is something to try. At any rate it
>>>> involves
>>>> at least some mild trickery (erase the blocks that previously contained
>>>> jffs2 data).
>>>>
>>> Yes, during mounting of UBIFS volumes, UBI checks for erase-header
>>> on first-page of every block.
>>
>>
>> Mounting UBIFS volumes should not be a problem - the existing ones are
>> not located in the newly added area of the NAND.  The question however
>> is what the UBI layer itself does, if I for example try to create a
>> new volume which then would allocate blocks from the newly added
>> space.
>
>
> UBI scans the whole partition when the attachement is made to the mtd
> partition. So if you'd get any error message regarding blocks that lack UBI
> headers it would be then, not when you subsequently try to create a volume
> in the partition. By that time the volume should be "formatted" with valid
> UBI headers.
>
>
>>> So, effectively you don't even need to erase the partition to convert
>>> it into UBI volume, you just try attaching it as UBI volume and UBI
>>> should erase it for you. (though flagging some warnings on the way).
>>
>>
>> Are you sure about that?
>>
>> Well, I guess I'll have to try it out in the end...
>
>
> I just tested it since I have a UBI system just beside me, by erasing and
> then writing zeros to a block (just to put non-UBI data in it) and
> restarting the system (the partition in question contains the volume with
> the root file system in it, so it is attached by the kernel during its
> boot). The result was that UBI didn't give any error or warning messages
> during boot, but a subsequent nanddump after boot revealed that the zeroed
> blocks now contained valid UBI headers, so, without looking at the source,
> it seems that UBI really does manage this fairly silently.

I've never tested this. But from my understanding of the UBI source
this behavior is
correct and intended. (Of course only in the non-fastmap case).

-- 
Thanks,
//richard

Re: Grow UBI device?

[Wolfgang Denk]

Dear Ricard,

In message <alpine.DEB.2.00.1310091725590.28562@lnxricardw.se.axis.com> you wrote:
> 
> I just tested it since I have a UBI system just beside me, by erasing and 
> then writing zeros to a block (just to put non-UBI data in it) and 
> restarting the system (the partition in question contains the volume with 
> the root file system in it, so it is attached by the kernel during its 
> boot). The result was that UBI didn't give any error or warning messages 
> during boot, but a subsequent nanddump after boot revealed that the 
> zeroed blocks now contained valid UBI headers, so, without looking at the 
> source, it seems that UBI really does manage this fairly silently.

Thanks a lot, much appreciated!

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
The IQ of the group is the lowest IQ of a member of the group divided
by the number of people in the group.

Re: Grow UBI device?

[Artem Bityutskiy]

On Wed, 2013-10-09 at 11:28 +0000, Gupta, Pekon wrote:
> Yes, during mounting of UBIFS volumes, UBI checks for erase-header
> on first-page of every block. 
> - If it's able to find a valid erase-header, with correct image_seq field,
> then it attaches it to its pool.
> - Otherwise, if the valid erase-header was not found, then it assumes
>  then it ubi will try re-erasing it. And add it to its pool of free-blocks.

I think there is a separate list for blocks with garbage, and if UBI
finds too much of them, it attaches R/O. So if it would find too many
JFFS2 eraseblocks, it would not erase them. But erasing the JFFS2 blocks
from outside of UBI driver would help.

-- 
Best Regards,
Artem Bityutskiy

Re: Grow UBI device?

[Wolfgang Denk]

Dear Ricard Wanderlof,

In message <alpine.DEB.2.00.1310091038110.28562@lnxricardw.se.axis.com> you wrote:
> 
> > Is there a way to "grow" the existing UBI device so that it now also 
> > covers the rest of the NAND chip?  Or is my only option to create a 
> > second UBI device?
> 
> UBI devices are 'attached' to existing mtd partitions. So to grow a UBI 
> device, you need to detach it from the partition, re-partition the flash 
> (and I'm not sure if this step is even possible as it involves 
> unregistering existing mtd partitions), then attach a UBI device to the a 
> new partition.

In my situation it is OK if the update involves a reboot, so it is
certainly possible to change the MTD partition layout.  So this should
not be a problem.

> Come to think of it, I'm not sure that would work anyway, as the 
> resulting, new, enlarged partition would partly contain UBI data and 
> partly old jffs2 data. I'm not sure what UBI does when it encounters 
> incorrect data, if it just erases the relevant blocks and formats them for 
> its own use, or if it barfs completely and just bails out complaining that 
> the partition does not contain UBI data. If the relevant blocks were 
> erased, then I think UBI would simply concede that the a previous erase 
> attempt had been prematurely aborted, and re-erase the blocks and write 
> its headers, so perhaps that is something to try. At any rate it involves 
> at least some mild trickery (erase the blocks that previously contained 
> jffs2 data).

well, this is the exact part where I wanted to know if (1) this is
guaranteed to work (by design), and/or (2) somebody has successfully
done it before.

> It's probably easier to create a new UBI device by attaching it to the 
> partition previously used by jffs2, after erasing it, but multiple UBI 
> devices across a flash partly defeats the purpose of wear-leveling which 
> UBI provides.

Right, and it fragments the device.  That's why I'd like to avoid
that.


Thanks!!

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Doing the good deeds is like the grass in the garden. You  don't  see
its  growth. But, it does by days. Doing the wicked deeds is like the
hone. You don't see its damage. But, it does by days.        - Buddha

Re: Grow UBI device?

[Artem Bityutskiy]

Wolfgang, people already answered you, and frankly I did not read all
the answers :-) But here is my brief version.

On Tue, 2013-10-08 at 21:28 +0200, Wolfgang Denk wrote:
> I'm looking for a clever way to migrate devices (in the field) from
> JFFS2 to UBI/UBIFS.  In the end result, I want to have single UBI
> device (which covers the whole NAND chip) with several volumes used
> with UBIFS.  Unfortunately I must keep some of the existing data.

And I guess there is no temporary storage where you could copy JFFS2
files, re-format the flash, and copy the files back.

> It would be nice if I could start with a UBI device that is smaller
> than the NAND, so I can keep an existing MTD partition with a JFFS2
> file system.

You mean that on the existing systems, you already have an extra MTD
partition, or you somehow can create one.

Then you want to UBI/UBIFS-format the extra partition. Then copy
required JFFS2 files there. And then kill JFFS2 partitions and "merge"
them with the extra partition, and end up with UBI/UBIFS covering all
the space.

>  After copying the data from JFFS2 to a UBIFS volume, I
> would like to free and reuse the space of this JFFS2 partition.

OK, sounds like I understood you correctly.

> Is there a way to "grow" the existing UBI device so that it now also
> covers the rest of the NAND chip?  Or is my only option to create a
> second UBI device?

I think so, unless you use the fastmap feature. Suppose you don't.

There are 2 parts - UBI and UBIFS.

So, let's suppose your situation is like this.

1. You have 2 mtd partitions - mtd1 and mtd2.
2. mtd1 has JFFS2 with important files.
3. mtd2 is the extra partition you have.

I guess you can do this.

1. Format mtd2 with UBI. Just erasing should be OK, or you can use
ubiformat: http://www.linux-mtd.infradead.org/faq/ubi.html#L_ubierase

Then you attach mtd2 (ubiattach) and you get an UBI device (/dev/ubiX).

2. Create a volume on /dev/ubiX which spans entire UBI device
(ubimkvol -t AFAIR). You end up with UBI volume /dev/ubiX_0.

3. And here is important part - you format /dev/ubiX_0 with JFFS2 but
correctly.

UBIFS has this "automatic expand" feature. Which means that if the UBI
volume underneath it becomes larger, UBIFS will expand automatically and
take the entire (larger) UBI volume.

However, there is a _limit_ to how much UBIFS can expand from the
original size. And you want to watch this limit and make it big enough.

The limit is connected with the "-c" option of mkfs.ubifs. Please, check
the docs - they explain what exactly -c controls and how it is related
to the maximum UBIFS file-system size:

http://www.linux-mtd.infradead.org/faq/ubifs.html#L_max_leb_cnt

So, select proper -c.

If you do not use mkfs.ubifs, but just attach /dev/ubiX_0, the UBIFS
kernel driver will format the volume automatically. It will select the
-c value itself. Since this value is power of 2, it will just take the
smallest power-of-2 value. It may be OK for your purposes, may be not.
It is safer to use mkfs.ubifs with explicit -c.

Basically this is the trickiest part, rest _should_ be easy, but you'll
learn when you try :-)

4. Copy important files from JFFS2-mtd1 to UBIFS-UBI-mtd2.

5. Erase mtd1 (or you can do this later too)

6. I do not know how you do this, but kill mtd1, and resize mtd2.
Probably mtd2 will then be named mtd1 after reboot. If you always refer
your MTD partitions and UBI volumes by name, you should not suffer to
much from mtd2 becoming mtd1.

Lets suppose there is no rename, for simplicity.

7. Make sure former JFFS2-mtd1 eraseblocks are erased (unless you did
this at step 5).

8. Attach mtd2 to UBI. UBI will automatically "take" the new
eraseblocks, erase them and write UBI headers there, and make available.

9. Volume size will still be the same, and you need to re-size it using
ubiresize.

10. Mount the resized volume. UBI will enlarge itself.

Problem solved.

Surely you do not want power cuts during this operations. If you need to
survive power cuts, I guess it is possible, but more complex.

HTH.

-- 
Best Regards,
Artem Bityutskiy

Re: Grow UBI device?

[Richard Weinberger]

On Thu, Oct 10, 2013 at 10:17 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote:
> Wolfgang, people already answered you, and frankly I did not read all
> the answers :-) But here is my brief version.
>
> On Tue, 2013-10-08 at 21:28 +0200, Wolfgang Denk wrote:
>> I'm looking for a clever way to migrate devices (in the field) from
>> JFFS2 to UBI/UBIFS.  In the end result, I want to have single UBI
>> device (which covers the whole NAND chip) with several volumes used
>> with UBIFS.  Unfortunately I must keep some of the existing data.
>
> And I guess there is no temporary storage where you could copy JFFS2
> files, re-format the flash, and copy the files back.
>
>> It would be nice if I could start with a UBI device that is smaller
>> than the NAND, so I can keep an existing MTD partition with a JFFS2
>> file system.
>
> You mean that on the existing systems, you already have an extra MTD
> partition, or you somehow can create one.
>
> Then you want to UBI/UBIFS-format the extra partition. Then copy
> required JFFS2 files there. And then kill JFFS2 partitions and "merge"
> them with the extra partition, and end up with UBI/UBIFS covering all
> the space.
>
>>  After copying the data from JFFS2 to a UBIFS volume, I
>> would like to free and reuse the space of this JFFS2 partition.
>
> OK, sounds like I understood you correctly.
>
>> Is there a way to "grow" the existing UBI device so that it now also
>> covers the rest of the NAND chip?  Or is my only option to create a
>> second UBI device?
>
> I think so, unless you use the fastmap feature. Suppose you don't.
>
> There are 2 parts - UBI and UBIFS.
>
> So, let's suppose your situation is like this.
>
> 1. You have 2 mtd partitions - mtd1 and mtd2.
> 2. mtd1 has JFFS2 with important files.
> 3. mtd2 is the extra partition you have.
>
> I guess you can do this.
>
> 1. Format mtd2 with UBI. Just erasing should be OK, or you can use
> ubiformat: http://www.linux-mtd.infradead.org/faq/ubi.html#L_ubierase
>
> Then you attach mtd2 (ubiattach) and you get an UBI device (/dev/ubiX).
>
> 2. Create a volume on /dev/ubiX which spans entire UBI device
> (ubimkvol -t AFAIR). You end up with UBI volume /dev/ubiX_0.
>
> 3. And here is important part - you format /dev/ubiX_0 with JFFS2 but
> correctly.

s/JFFS2/UBIFS :-)

> UBIFS has this "automatic expand" feature. Which means that if the UBI
> volume underneath it becomes larger, UBIFS will expand automatically and
> take the entire (larger) UBI volume.
>
> However, there is a _limit_ to how much UBIFS can expand from the
> original size. And you want to watch this limit and make it big enough.
>
> The limit is connected with the "-c" option of mkfs.ubifs. Please, check
> the docs - they explain what exactly -c controls and how it is related
> to the maximum UBIFS file-system size:
>
> http://www.linux-mtd.infradead.org/faq/ubifs.html#L_max_leb_cnt
>
> So, select proper -c.
>
> If you do not use mkfs.ubifs, but just attach /dev/ubiX_0, the UBIFS
> kernel driver will format the volume automatically. It will select the
> -c value itself. Since this value is power of 2, it will just take the
> smallest power-of-2 value. It may be OK for your purposes, may be not.
> It is safer to use mkfs.ubifs with explicit -c.
>
> Basically this is the trickiest part, rest _should_ be easy, but you'll
> learn when you try :-)
>
> 4. Copy important files from JFFS2-mtd1 to UBIFS-UBI-mtd2.
>
> 5. Erase mtd1 (or you can do this later too)
>
> 6. I do not know how you do this, but kill mtd1, and resize mtd2.
> Probably mtd2 will then be named mtd1 after reboot. If you always refer
> your MTD partitions and UBI volumes by name, you should not suffer to
> much from mtd2 becoming mtd1.
>
> Lets suppose there is no rename, for simplicity.
>
> 7. Make sure former JFFS2-mtd1 eraseblocks are erased (unless you did
> this at step 5).
>
> 8. Attach mtd2 to UBI. UBI will automatically "take" the new
> eraseblocks, erase them and write UBI headers there, and make available.
>
> 9. Volume size will still be the same, and you need to re-size it using
> ubiresize.
>
> 10. Mount the resized volume. UBI will enlarge itself.
>
> Problem solved.
>
> Surely you do not want power cuts during this operations. If you need to
> survive power cuts, I guess it is possible, but more complex.
>
> HTH.
>
> --
> Best Regards,
> Artem Bityutskiy
>
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/



-- 
Thanks,
//richard

Re: Grow UBI device?

[Wolfgang Denk]

Dear Artem,

In message <1381393038.7875.83.camel@sauron.fi.intel.com> you wrote:
> Wolfgang, people already answered you, and frankly I did not read all
> the answers :-) But here is my brief version.

Thanks.

> On Tue, 2013-10-08 at 21:28 +0200, Wolfgang Denk wrote:
> > I'm looking for a clever way to migrate devices (in the field) from
> > JFFS2 to UBI/UBIFS.  In the end result, I want to have single UBI
> > device (which covers the whole NAND chip) with several volumes used
> > with UBIFS.  Unfortunately I must keep some of the existing data.
> 
> And I guess there is no temporary storage where you could copy JFFS2
> files, re-format the flash, and copy the files back.

Correct.

> > It would be nice if I could start with a UBI device that is smaller
> > than the NAND, so I can keep an existing MTD partition with a JFFS2
> > file system.
> 
> You mean that on the existing systems, you already have an extra MTD
> partition, or you somehow can create one.
> 
> Then you want to UBI/UBIFS-format the extra partition. Then copy
> required JFFS2 files there. And then kill JFFS2 partitions and "merge"
> them with the extra partition, and end up with UBI/UBIFS covering all
> the space.

Correct.

> > Is there a way to "grow" the existing UBI device so that it now also
> > covers the rest of the NAND chip?  Or is my only option to create a
> > second UBI device?
> 
> I think so, unless you use the fastmap feature. Suppose you don't.

We don't.  We may want to do this later, but certainly not during this
migration steps.

> 1. Format mtd2 with UBI. Just erasing should be OK, or you can use
> ubiformat: http://www.linux-mtd.infradead.org/faq/ubi.html#L_ubierase
> 
> Then you attach mtd2 (ubiattach) and you get an UBI device (/dev/ubiX).
> 
> 2. Create a volume on /dev/ubiX which spans entire UBI device
> (ubimkvol -t AFAIR). You end up with UBI volume /dev/ubiX_0.
> 
> 3. And here is important part - you format /dev/ubiX_0 with JFFS2 but
> correctly.

With JFFS2?  I would expect to see "UBIFS" here?

> UBIFS has this "automatic expand" feature. Which means that if the UBI
> volume underneath it becomes larger, UBIFS will expand automatically and
> take the entire (larger) UBI volume.
> 
> However, there is a _limit_ to how much UBIFS can expand from the
> original size. And you want to watch this limit and make it big enough.
> 
> The limit is connected with the "-c" option of mkfs.ubifs. Please, check
> the docs - they explain what exactly -c controls and how it is related
> to the maximum UBIFS file-system size:
> 
> http://www.linux-mtd.infradead.org/faq/ubifs.html#L_max_leb_cnt
> 
> So, select proper -c.

Thanks for the poointer.  I would have missed this.

> If you do not use mkfs.ubifs, but just attach /dev/ubiX_0, the UBIFS
> kernel driver will format the volume automatically. It will select the
> -c value itself. Since this value is power of 2, it will just take the
> smallest power-of-2 value. It may be OK for your purposes, may be not.
> It is safer to use mkfs.ubifs with explicit -c.

Agreed.  It would have worked in our case, as mtd1 is 32 MB and mtd2
is 96 MB on a 128 MB NAND device, but it's much better to have full
control.

> 4. Copy important files from JFFS2-mtd1 to UBIFS-UBI-mtd2.
> 
> 5. Erase mtd1 (or you can do this later too)
> 
> 6. I do not know how you do this, but kill mtd1, and resize mtd2.
> Probably mtd2 will then be named mtd1 after reboot. If you always refer
> your MTD partitions and UBI volumes by name, you should not suffer to
> much from mtd2 becoming mtd1.

The plan is to just have another mtd3 which covers the whole device.
So I would detach UBI from mtd2 and re-attach to mtd3.

> Lets suppose there is no rename, for simplicity.
> 
> 7. Make sure former JFFS2-mtd1 eraseblocks are erased (unless you did
> this at step 5).
> 
> 8. Attach mtd2 to UBI. UBI will automatically "take" the new
> eraseblocks, erase them and write UBI headers there, and make available.
> 
> 9. Volume size will still be the same, and you need to re-size it using
> ubiresize.
> 
> 10. Mount the resized volume. UBI will enlarge itself.

That's just amazing :-)

> Problem solved.
> 
> Surely you do not want power cuts during this operations. If you need to
> survive power cuts, I guess it is possible, but more complex.

Thanks a lot for all the help, it's highly appreciated!

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"At least they're __________EXPERIENCED incompetents"

Re: Grow UBI device?

[Richard Weinberger]

On Tue, Oct 15, 2013 at 7:40 PM, Wolfgang Denk <wd@denx.de> wrote:
> Dear Artem,
>
> In message <1381393038.7875.83.camel@sauron.fi.intel.com> you wrote:
>> Wolfgang, people already answered you, and frankly I did not read all
>> the answers :-) But here is my brief version.
>
> Thanks.
>
>> On Tue, 2013-10-08 at 21:28 +0200, Wolfgang Denk wrote:
>> > I'm looking for a clever way to migrate devices (in the field) from
>> > JFFS2 to UBI/UBIFS.  In the end result, I want to have single UBI
>> > device (which covers the whole NAND chip) with several volumes used
>> > with UBIFS.  Unfortunately I must keep some of the existing data.
>>
>> And I guess there is no temporary storage where you could copy JFFS2
>> files, re-format the flash, and copy the files back.
>
> Correct.
>
>> > It would be nice if I could start with a UBI device that is smaller
>> > than the NAND, so I can keep an existing MTD partition with a JFFS2
>> > file system.
>>
>> You mean that on the existing systems, you already have an extra MTD
>> partition, or you somehow can create one.
>>
>> Then you want to UBI/UBIFS-format the extra partition. Then copy
>> required JFFS2 files there. And then kill JFFS2 partitions and "merge"
>> them with the extra partition, and end up with UBI/UBIFS covering all
>> the space.
>
> Correct.
>
>> > Is there a way to "grow" the existing UBI device so that it now also
>> > covers the rest of the NAND chip?  Or is my only option to create a
>> > second UBI device?
>>
>> I think so, unless you use the fastmap feature. Suppose you don't.
>
> We don't.  We may want to do this later, but certainly not during this
> migration steps.
>
>> 1. Format mtd2 with UBI. Just erasing should be OK, or you can use
>> ubiformat: http://www.linux-mtd.infradead.org/faq/ubi.html#L_ubierase
>>
>> Then you attach mtd2 (ubiattach) and you get an UBI device (/dev/ubiX).
>>
>> 2. Create a volume on /dev/ubiX which spans entire UBI device
>> (ubimkvol -t AFAIR). You end up with UBI volume /dev/ubiX_0.
>>
>> 3. And here is important part - you format /dev/ubiX_0 with JFFS2 but
>> correctly.
>
> With JFFS2?  I would expect to see "UBIFS" here?

Yes, UBIFS.

>> UBIFS has this "automatic expand" feature. Which means that if the UBI
>> volume underneath it becomes larger, UBIFS will expand automatically and
>> take the entire (larger) UBI volume.
>>
>> However, there is a _limit_ to how much UBIFS can expand from the
>> original size. And you want to watch this limit and make it big enough.
>>
>> The limit is connected with the "-c" option of mkfs.ubifs. Please, check
>> the docs - they explain what exactly -c controls and how it is related
>> to the maximum UBIFS file-system size:
>>
>> http://www.linux-mtd.infradead.org/faq/ubifs.html#L_max_leb_cnt
>>
>> So, select proper -c.
>
> Thanks for the poointer.  I would have missed this.
>
>> If you do not use mkfs.ubifs, but just attach /dev/ubiX_0, the UBIFS
>> kernel driver will format the volume automatically. It will select the
>> -c value itself. Since this value is power of 2, it will just take the
>> smallest power-of-2 value. It may be OK for your purposes, may be not.
>> It is safer to use mkfs.ubifs with explicit -c.
>
> Agreed.  It would have worked in our case, as mtd1 is 32 MB and mtd2
> is 96 MB on a 128 MB NAND device, but it's much better to have full
> control.
>
>> 4. Copy important files from JFFS2-mtd1 to UBIFS-UBI-mtd2.
>>
>> 5. Erase mtd1 (or you can do this later too)
>>
>> 6. I do not know how you do this, but kill mtd1, and resize mtd2.
>> Probably mtd2 will then be named mtd1 after reboot. If you always refer
>> your MTD partitions and UBI volumes by name, you should not suffer to
>> much from mtd2 becoming mtd1.
>
> The plan is to just have another mtd3 which covers the whole device.
> So I would detach UBI from mtd2 and re-attach to mtd3.
>
>> Lets suppose there is no rename, for simplicity.
>>
>> 7. Make sure former JFFS2-mtd1 eraseblocks are erased (unless you did
>> this at step 5).
>>
>> 8. Attach mtd2 to UBI. UBI will automatically "take" the new
>> eraseblocks, erase them and write UBI headers there, and make available.
>>
>> 9. Volume size will still be the same, and you need to re-size it using
>> ubiresize.
>>
>> 10. Mount the resized volume. UBI will enlarge itself.
>
> That's just amazing :-)
>
>> Problem solved.
>>
>> Surely you do not want power cuts during this operations. If you need to
>> survive power cuts, I guess it is possible, but more complex.
>
> Thanks a lot for all the help, it's highly appreciated!
>
> Best regards,
>
> Wolfgang Denk
>
> --
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
> "At least they're __________EXPERIENCED incompetents"
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/



-- 
Thanks,
//richard