ubifs version 1 compatibility

ubifs version 1 compatibility

[Hamish Moffatt]

Hi Artem and all,

I'm upgrading my embedded system which uses UBIFS from 2.6.24 (with a
slightly updated ubifs) to 2.6.29.

I notice that the new kernel won't read my existing file systems as it
says they are too old - version 1. The code checks for version 3 or 4 I
see.

My application firmware is generated with mkfs.ubifs and programmed via
ubiupdatevol, but the boot loader on my devices is based on the older
kernel and won't be able to read it. And I have read-write file systems
that will be formatted with the old version too.


Is there any solution?

thanks
Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: ubifs version 1 compatibility

[Artem Bityutskiy]

On Thu, 2009-03-26 at 16:15 +1100, Hamish Moffatt wrote:
> Hi Artem and all,
> 
> I'm upgrading my embedded system which uses UBIFS from 2.6.24 (with a
> slightly updated ubifs) to 2.6.29.
> 
> I notice that the new kernel won't read my existing file systems as it
> says they are too old - version 1. The code checks for version 3 or 4 I
> see.

Hmm, you use version 1 which has been there _before_ UBIFS went to
mainline. We really did not expect this.

> My application firmware is generated with mkfs.ubifs and programmed via
> ubiupdatevol, but the boot loader on my devices is based on the older
> kernel and won't be able to read it. And I have read-write file systems
> that will be formatted with the old version too.

I need to take a closer look and realize what were the changes between
v1 and v3 and if it is possible to make current UBIFS support it. Then
we could teach mkfs.ubifs generate version 1 FSes, and you could keep
using v1. But we need some time to investigate how feasible is this.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: ubifs version 1 compatibility

[Artem Bityutskiy]

On Thu, 2009-03-26 at 16:15 +1100, Hamish Moffatt wrote:
> Hi Artem and all,
> 
> I'm upgrading my embedded system which uses UBIFS from 2.6.24 (with a
> slightly updated ubifs) to 2.6.29.
> 
> I notice that the new kernel won't read my existing file systems as it
> says they are too old - version 1. The code checks for version 3 or 4 I
> see.
> 
> My application firmware is generated with mkfs.ubifs and programmed via
> ubiupdatevol, but the boot loader on my devices is based on the older
> kernel and won't be able to read it. And I have read-write file systems
> that will be formatted with the old version too.
> 
> 
> Is there any solution?

I've took a brief look. UBIFS went into mainline with format version 4,
and versions 1,2,3 were more development versions. There were changes in
the truncation node format and in inode node format. I would probably
be possible to have compatibility mode, but it is big job :-(

Your boot-loader is Linux kernel-based, right? Does it have write
support, or it is read-only?

Is it possible to upgrade the boot-loader at the same time you upgrade
the kernel on your devices?

Side note: now when u-boot supports UBIFS, we need to re-think our
versioning scheme and strictly preserve R/O compatibility.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: ubifs version 1 compatibility

[Hamish Moffatt]

On Thu, Mar 26, 2009 at 08:34:57AM +0200, Artem Bityutskiy wrote:
> On Thu, 2009-03-26 at 16:15 +1100, Hamish Moffatt wrote:
> > Hi Artem and all,
> > 
> > I'm upgrading my embedded system which uses UBIFS from 2.6.24 (with a
> > slightly updated ubifs) to 2.6.29.
> > 
> > I notice that the new kernel won't read my existing file systems as it
> > says they are too old - version 1. The code checks for version 3 or 4 I
> > see.
> > 
> > My application firmware is generated with mkfs.ubifs and programmed via
> > ubiupdatevol, but the boot loader on my devices is based on the older
> > kernel and won't be able to read it. And I have read-write file systems
> > that will be formatted with the old version too.
> 
> I've took a brief look. UBIFS went into mainline with format version 4,
> and versions 1,2,3 were more development versions. There were changes in
> the truncation node format and in inode node format. I would probably
> be possible to have compatibility mode, but it is big job :-(
> 
> Your boot-loader is Linux kernel-based, right? Does it have write
> support, or it is read-only?
> 
> Is it possible to upgrade the boot-loader at the same time you upgrade
> the kernel on your devices?
> 
> Side note: now when u-boot supports UBIFS, we need to re-think our
> versioning scheme and strictly preserve R/O compatibility.

Hi Artem,

Yes our version 1 media is from pre-mainline UBIFS and we took our
chances by including it. We are shipping one product using 2.6.24 with
old backported UBIFS and it's working very well. I don't have any
expectations of support for the old version, so thanks for your
assistance thus far.

Fortunately the kernel upgrade is for a newer product where we have some
more flexibility in fixing this problem.

We can upgrade our boot loader to 2.6.29 also so that it can read UBIFS
v4 media. This has to be co-ordinated with the main firmware image but
this is reasonable.

We do have one volatile UBIFS volume that we want to preserve. Is there
any way we could convert it? For example a user-space utility that could
read the old media version?

Maybe before upgrade we could run mkfs.ubifs using the contents of the
existing volume to create the new format, unmount it and ubiupdatevol it..


regards
Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: ubifs version 1 compatibility

[Artem Bityutskiy]

On Thu, 2009-03-26 at 18:44 +1100, Hamish Moffatt wrote:
> Yes our version 1 media is from pre-mainline UBIFS and we took our
> chances by including it. We are shipping one product using 2.6.24 with
> old backported UBIFS and it's working very well. I don't have any
> expectations of support for the old version, so thanks for your
> assistance thus far.
> 
> Fortunately the kernel upgrade is for a newer product where we have some
> more flexibility in fixing this problem.
> 
> We can upgrade our boot loader to 2.6.29 also so that it can read UBIFS
> v4 media. This has to be co-ordinated with the main firmware image but
> this is reasonable.
> 
> We do have one volatile UBIFS volume that we want to preserve. Is there
> any way we could convert it? For example a user-space utility that could
> read the old media version?

I was looking at the version changes, and this bugfix:

http://git.infradead.org/users/dedekind/ubifs-historical-3.git?a=commit;h=b9903987e8efb5252b692ffc7c2b190f4a0e83b8

makes it very difficult to turn old image into new one. This will
basically mean we need:

1. extract files from the old image
2. feed them to mkfs.ubifs.

> Maybe before upgrade we could run mkfs.ubifs using the contents of the
> existing volume to create the new format, unmount it and ubiupdatevol it..

The easiest way to do this is to boot the old kernel,
e.g. in vmware, extract the files using the nandsim technique:
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_ubifs_nandsim

then feed them to mkfs.ubifs.

But yes, a user-space utility to extract files from an image would be
of course nicer, but again it is something which is not easy to do.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: ubifs version 1 compatibility

[Hamish Moffatt]

On Thu, Mar 26, 2009 at 11:50:23AM +0200, Artem Bityutskiy wrote:
> On Thu, 2009-03-26 at 18:44 +1100, Hamish Moffatt wrote:
> makes it very difficult to turn old image into new one. This will
> basically mean we need:
> 
> 1. extract files from the old image
> 2. feed them to mkfs.ubifs.
> 
> > Maybe before upgrade we could run mkfs.ubifs using the contents of the
> > existing volume to create the new format, unmount it and ubiupdatevol it..
> 
> The easiest way to do this is to boot the old kernel,
> e.g. in vmware, extract the files using the nandsim technique:
> http://www.linux-mtd.infradead.org/faq/ubifs.html#L_ubifs_nandsim
> 
> then feed them to mkfs.ubifs.

Well I have these files on embedded devices out in the real world. The
volume is storing user data, so I need to do this change in-place.

> But yes, a user-space utility to extract files from an image would be
> of course nicer, but again it is something which is not easy to do.

OK I will investigate having the system build a new file system with
a new mkfs.ubifs while running the old kernel.. Alternatively I may be
able to stash the data somewhere else and copy it back when the new
kernel first boots.. I probably have some space in NOR for a small JFFS2
volume. Or I'll tar directly to an mtdblock device or something :)


thanks
Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: ubifs version 1 compatibility

[Artem Bityutskiy]

On Thu, 2009-03-26 at 23:04 +1100, Hamish Moffatt wrote:
> On Thu, Mar 26, 2009 at 11:50:23AM +0200, Artem Bityutskiy wrote:
> > On Thu, 2009-03-26 at 18:44 +1100, Hamish Moffatt wrote:
> > makes it very difficult to turn old image into new one. This will
> > basically mean we need:
> > 
> > 1. extract files from the old image
> > 2. feed them to mkfs.ubifs.
> > 
> > > Maybe before upgrade we could run mkfs.ubifs using the contents of the
> > > existing volume to create the new format, unmount it and ubiupdatevol it..
> > 
> > The easiest way to do this is to boot the old kernel,
> > e.g. in vmware, extract the files using the nandsim technique:
> > http://www.linux-mtd.infradead.org/faq/ubifs.html#L_ubifs_nandsim
> > 
> > then feed them to mkfs.ubifs.
> 
> Well I have these files on embedded devices out in the real world. The
> volume is storing user data, so I need to do this change in-place.

Let me make sure I understand the need.

So you have devises with v1 format in the wild. You want to upgrade
those devices. While upgrading, you will re-write some UBI volumes, but
after upgrade the user-data volume should contains exactly the same data
as before the upgrade, right?

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: ubifs version 1 compatibility

[Hamish Moffatt]

On Thu, Mar 26, 2009 at 02:12:05PM +0200, Artem Bityutskiy wrote:
> On Thu, 2009-03-26 at 23:04 +1100, Hamish Moffatt wrote:
> > On Thu, Mar 26, 2009 at 11:50:23AM +0200, Artem Bityutskiy wrote:
> > > On Thu, 2009-03-26 at 18:44 +1100, Hamish Moffatt wrote:
> > > makes it very difficult to turn old image into new one. This will
> > > basically mean we need:
> > > 
> > > 1. extract files from the old image
> > > 2. feed them to mkfs.ubifs.
> > > 
> > > > Maybe before upgrade we could run mkfs.ubifs using the contents of the
> > > > existing volume to create the new format, unmount it and ubiupdatevol it..
> > > 
> > > The easiest way to do this is to boot the old kernel,
> > > e.g. in vmware, extract the files using the nandsim technique:
> > > http://www.linux-mtd.infradead.org/faq/ubifs.html#L_ubifs_nandsim
> > > 
> > > then feed them to mkfs.ubifs.
> > 
> > Well I have these files on embedded devices out in the real world. The
> > volume is storing user data, so I need to do this change in-place.
> 
> Let me make sure I understand the need.
> 
> So you have devises with v1 format in the wild. You want to upgrade
> those devices. While upgrading, you will re-write some UBI volumes, but
> after upgrade the user-data volume should contains exactly the same data
> as before the upgrade, right?

Yes, that's what I need.



thanks
Hamish
-- 
Hamish Moffatt VK3SB <hamish@debian.org> <hamish@cloud.net.au>

Re: ubifs version 1 compatibility

[Artem Bityutskiy]

On Fri, 2009-03-27 at 00:15 +1100, Hamish Moffatt wrote:
> On Thu, Mar 26, 2009 at 02:12:05PM +0200, Artem Bityutskiy wrote:
> > On Thu, 2009-03-26 at 23:04 +1100, Hamish Moffatt wrote:
> > > On Thu, Mar 26, 2009 at 11:50:23AM +0200, Artem Bityutskiy wrote:
> > > > On Thu, 2009-03-26 at 18:44 +1100, Hamish Moffatt wrote:
> > > > makes it very difficult to turn old image into new one. This will
> > > > basically mean we need:
> > > > 
> > > > 1. extract files from the old image
> > > > 2. feed them to mkfs.ubifs.
> > > > 
> > > > > Maybe before upgrade we could run mkfs.ubifs using the contents of the
> > > > > existing volume to create the new format, unmount it and ubiupdatevol it..
> > > > 
> > > > The easiest way to do this is to boot the old kernel,
> > > > e.g. in vmware, extract the files using the nandsim technique:
> > > > http://www.linux-mtd.infradead.org/faq/ubifs.html#L_ubifs_nandsim
> > > > 
> > > > then feed them to mkfs.ubifs.
> > > 
> > > Well I have these files on embedded devices out in the real world. The
> > > volume is storing user data, so I need to do this change in-place.
> > 
> > Let me make sure I understand the need.
> > 
> > So you have devises with v1 format in the wild. You want to upgrade
> > those devices. While upgrading, you will re-write some UBI volumes, but
> > after upgrade the user-data volume should contains exactly the same data
> > as before the upgrade, right?
> 
> Yes, that's what I need.

I see. Well, because most of the UBIFS functionality is in the
kernel, it might probably be possible to teach the kernel UBIFS
implementation to automatically re-format the file-system if it
finds v.1 layout. However, this is not a very simple task, and
I am not sure I have time to implement this, sorry :-(


-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)