JFFS2: truncated files after power loss scenario

JFFS2: truncated files after power loss scenario

[Sven]

Hi mtd list,

What is the expected behavior for files beeing written on a jffs2 (on
nand) during a power loss? Would an incomplete file exist, or would the journaling do a roll back
because of an incomplete transaction?
Is there a difference with new files created, or existing files beeing modified?

My tests show that on next bootup the file exists, but is truncated. I have not seen any journaling related roll backs.

I took a look at this, because we have seen a truncated configuration file once. I'm not sure what caused this. Perhaps a power loss during write.

What do you think about the following jffs2dump?:
( dwnld.conf was created once by extracting it from an archive,
  it might have been modified afterwards. (fopen "w+"; fprintf; close),
  the file existed afterwards, but was truncated in to 4k,
  then during error analysis the file was renamed "mv dwnld.conf
dwnld.conf.org" and a new file dwnld.conf was created there)

Dirent     node at 0x00346f2c, totlen 0x00000032, #pino    41, version    23, #ino        42, nsize       10, name dwnld.conf
Dirent     node at 0x003e63e8, totlen 0x00000032, #pino    41, version     5, #ino        42, nsize       10, name dwnld.conf
Dirent     node at 0x009c19e0, totlen 0x00000032, #pino    41, version    28, #ino      2484, nsize       10, name dwnld.conf
Dirent     node at 0x009fc398, totlen 0x00000036, #pino    41, version    27, #ino        42, nsize       14, name dwnld.conf.org
Inode      node at 0x003e63a4, totlen 0x00000044, #ino     42, version     1, isize        0, csize        0, dsize        0, offset 0
Inode      node at 0x003e641c, totlen 0x000005d8, #ino     42, version     2, isize     4096, csize     1428, dsize     4096, offset 0
Inode      node at 0x003e69f4, totlen 0x000006f5, #ino     42, version     3, isize     8192, csize     1713, dsize     4096, offset 4096
Inode      node at 0x003e70ec, totlen 0x0000028f, #ino     42, version     4, isize     9445, csize      587, dsize     1253, offset 8192
Inode      node at 0x003e737c, totlen 0x00000044, #ino     42, version     5, isize     9445, csize        0, dsize        0, offset 0
Inode      node at 0x003e73c0, totlen 0x00000044, #ino     42, version     6, isize     9445, csize        0, dsize        0, offset 0
Inode      node at 0x003e7404, totlen 0x00000044, #ino     42, version     7, isize     9445, csize        0, dsize        0, offset 0
Inode      node at 0x009bcba8, totlen 0x000005d8, #ino     42, version     9, isize     4096, csize     1428, dsize     4096, offset 0
Inode      node at 0x00e51ea0, totlen 0x000005d8, #ino     42, version     9, isize     4096, csize     1428, dsize     4096, offset 0

I reproduced a good-case of the actions on the file with a fresh filesystem:
Dirent     node at 0x00364e3c, totlen 0x00000032, #pino    41, version    11, #ino        42, nsize       10, name dwnld.conf
Dirent     node at 0x00553000, totlen 0x00000036, #pino    41, version    12, #ino        42, nsize       14, name dwnld.conf.org
Dirent     node at 0x00553038, totlen 0x00000032, #pino    41, version    13, #ino         0, nsize       10, name dwnld.conf
Dirent     node at 0x00555844, totlen 0x00000032, #pino    41, version    14, #ino      1171, nsize       10, name dwnld.conf
Inode      node at 0x00364e70, totlen 0x00000044, #ino     42, version    10, isize     9445, csize        0, dsize        0, offset 0
Inode      node at 0x00449cb4, totlen 0x000005d8, #ino     42, version     2, isize     4096, csize     1428, dsize     4096, offset 0
Inode      node at 0x0044a2c8, totlen 0x000006f5, #ino     42, version     3, isize     8192, csize     1713, dsize     4096, offset 4096
Inode      node at 0x0044accc, totlen 0x0000028f, #ino     42, version     4, isize     9445, csize      587, dsize     1253, offset 8192
Inode      node at 0x0096e1c4, totlen 0x00000044, #ino     42, version    11, isize        0, csize        0, dsize        0, offset 0
Inode      node at 0x0096e208, totlen 0x000005d8, #ino     42, version    12, isize     4096, csize     1428, dsize     4096, offset 0
Inode      node at 0x0096e7e0, totlen 0x000006f5, #ino     42, version    13, isize     8192, csize     1713, dsize     4096, offset 4096
Inode      node at 0x0096eed8, totlen 0x0000028f, #ino     42, version    14, isize     9445, csize      587, dsize     1253, offset 8192


I wonder about the Inode #42 Version 5 to 9 and about why two Dirent exist(ed) for #ino 42.
Do you think this was caused by a power loss scenario?

The truncated file error happened on a 2.6.18 based kernel.
I have done power loss tests on 2.6.34, which showed truncated files as well.

regards, Sven Jaborek

Re: JFFS2: truncated files after power loss scenario

[Artem Bityutskiy]

On Sun, 2011-02-06 at 15:59 +0100, Sven wrote:
> Hi mtd list,
> 
> What is the expected behavior for files beeing written on a jffs2 (on
> nand) during a power loss? Would an incomplete file exist, or would the journaling do a roll back
> because of an incomplete transaction?
> Is there a difference with new files created, or existing files beeing modified?
> 
> My tests show that on next bootup the file exists, but is truncated. I have not seen any journaling related roll backs.
> 
> I took a look at this, because we have seen a truncated configuration file once. I'm not sure what caused this. Perhaps a power loss during write.

How big was this file?

> I wonder about the Inode #42 Version 5 to 9 and about why two Dirent exist(ed) for #ino 42.
> Do you think this was caused by a power loss scenario?

No, this is how JFFS2 works, this is related to the design. In short,
only v9 is valid, other versions are invalid, you can ignore them.

> The truncated file error happened on a 2.6.18 based kernel.
> I have done power loss tests on 2.6.34, which showed truncated files as well.

Please, give more information:

1. Your flash type, if NAND - page size.
2. What do you do to the file in your test - how exactly you change it.
3. How big is the file?

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: JFFS2: truncated files after power loss scenario

[Sven]

Hi

> > I took a look at this, because we have seen a truncated configuration file once. I'm not sure what caused this. Perhaps a power loss during write.
> 
> How big was this file?

9445 bytes.

> > I wonder about the Inode #42 Version 5 to 9 and about why two Dirent exist(ed) for #ino 42.
> > Do you think this was caused by a power loss scenario?

> Please, give more information:
> 
> 1. Your flash type, if NAND - page size.

Need to check this. Page size is 128k i think.

> 2. What do you do to the file in your test - how exactly you change it.

I did two tests.
First one is a bash script running an endless loop. It creates new files
over and over by copying the 9445 bytes size file every time with a new
filename.
A power loss with this script running always produced the truncated last
file. It had the size of 4k or 8k then.

The second test reflects more our simple use case. A C-program which
uses fopen with "w+", and then uses GLib GKeyFile operations to modify a
key and then store the file again. An fclose follows of course.

> 3. How big is the file?
9445 bytes


Please, could you say something general about what the journaling in
jffs2+nand should be able to do about power loss scenarios? I could not
find specs about that.

br, Sven

Re: JFFS2: truncated files after power loss scenario

[Artem Bityutskiy]

On Sun, 2011-02-06 at 16:47 +0100, Sven wrote:
> Hi
> 
> > > I took a look at this, because we have seen a truncated configuration file once. I'm not sure what caused this. Perhaps a power loss during write.
> > 
> > How big was this file?
> 
> 9445 bytes.
> 
> > > I wonder about the Inode #42 Version 5 to 9 and about why two Dirent exist(ed) for #ino 42.
> > > Do you think this was caused by a power loss scenario?
> 
> > Please, give more information:
> > 
> > 1. Your flash type, if NAND - page size.
> 
> Need to check this. Page size is 128k i think.

That's eraseblock size, PAGE size must be 2KiB in your case.

> > 2. What do you do to the file in your test - how exactly you change it.
> 
> I did two tests.
> First one is a bash script running an endless loop. It creates new files
> over and over by copying the 9445 bytes size file every time with a new
> filename.
> A power loss with this script running always produced the truncated last
> file. It had the size of 4k or 8k then.

Then it is expected I guess. Your I bet your script opens the file,
truncates it, then writes to it. So when you have a power cut, you can
find your file truncated to 0, 4, or 8KiB, of 9445 bytes. In your case
JFFS2 writes in 4KiB pieces (maximum it can do), this is why you see 4
and 8 KiB.

> The second test reflects more our simple use case. A C-program which
> uses fopen with "w+", and then uses GLib GKeyFile operations to modify a
> key and then store the file again. An fclose follows of course.

I believe your library is truncating the file as well. You can check
this with strace.

If you write a program which just overrides the file without truncation,
you won't see truncations. But of course, yous will see inconsistent
file contents containing, say, 4KiB of new data, and the rest is old
data.

The bottom line is that it is difficult to fix broken applications by
changing JFFS2. The right way to update configuration files is described
here:
http://www.linux-mtd.infradead.org/faq/ubifs.html#L_atomic_change

> Please, could you say something general about what the journaling in
> jffs2+nand should be able to do about power loss scenarios? I could not
> find specs about that.

Oh, no, but JFFS2 is very simple FS. It is almost 100% synchronous, only
last 2KiB are cached in the write-buffer. Se basically, if you cut power
at any point, the maximum you can lose is the last written 2KiB. The
rest is on the flash and cannot be lost.

But any app which relies on this is bad and not portable.

Ask specific questions please.

Please, read this UBIFS piece of doc about proper apps behavior and
expectations about the file-systems:
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writeback

There are many docs in the Internet as well.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: JFFS2: truncated files after power loss scenario

[Albrecht Dreß]

[-- Attachment #1: Type: text/plain, Size: 747 bytes --]

Am 06.02.11 17:03 schrieb(en) Artem Bityutskiy:
> The bottom line is that it is difficult to fix broken applications by
> changing JFFS2. The right way to update configuration files is described
> here:
> http://www.linux-mtd.infradead.org/faq/ubifs.html#L_atomic_change

Dumb question: isn't it necessary to also run sync() after the fsync() - close() - rename() sequence, as to ensure that the directory entries are actually flushed to the flash?  Or does a call to

	rename("configfile.tmp", "configfile");

*ensure* that after a crash during this call either the entry of the old file (configfile) or of the newly written contents (configfile.tmp) is accessible, independent of the moment when the crash happens?

Best, Albrecht.

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: JFFS2: truncated files after power loss scenario

[Artem Bityutskiy]

On Sun, 2011-02-06 at 18:12 +0100, Albrecht Dreß wrote:
> Am 06.02.11 17:03 schrieb(en) Artem Bityutskiy:
> > The bottom line is that it is difficult to fix broken applications
> by
> > changing JFFS2. The right way to update configuration files is
> described
> > here:
> > http://www.linux-mtd.infradead.org/faq/ubifs.html#L_atomic_change
> 
> Dumb question: isn't it necessary to also run sync() after the fsync()
> - close() - rename() sequence, as to ensure that the directory entries
> are actually flushed to the flash?  Or does a call to
> 
> 	rename("configfile.tmp", "configfile");
> 
> *ensure* that after a crash during this call either the entry of the
> old file (configfile) or of the newly written contents
> (configfile.tmp) is accessible, independent of the moment when the
> crash happens?

Well, there was a huge debate about this. Rename does not ensure sync in
many file-systems, and posix does not require this. But user-spaces
relies on this and many people believe this should obviously be
guaranteed. So ext4 and may be some other FSes adjusted accordingly. I
wrote about this in UBIFS FAQ:

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_exceptions

You'll find all the relevant links there.


-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: JFFS2: truncated files after power loss scenario

[Albrecht Dreß]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

Am 06.02.11 18:39 schrieb(en) Artem Bityutskiy:
> Well, there was a huge debate about this. Rename does not ensure sync in many file-systems, and posix does not require this. But user-spaces relies on this and many people believe this should obviously be guaranteed. So ext4 and may be some other FSes adjusted accordingly. I wrote about this in UBIFS FAQ:
> 
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_sync_exceptions
> 
> You'll find all the relevant links there.

Thanks a lot for the pointers!

Unfortunately, they are more more about ubi & ext4, but I am in particular interested in JFFS2's behaviour, so please excuse my questions, but it's a really critical point...  Currently, I use the following sequence to write my configs to jffs2 (both NOR flash and mtd-ram):

open(scratch, O_RDWR | O_CREAT | O_TRUNC, perms);
write();
fsync();
close();
sync();		// #1
rename(scratch, realname);
sync();		// #2

As far as I understand the information you cited, this should be safe under all circumstances for jffs2.  Is that assumption correct?

As calling sync() might be a lot more expensive than fsync() if other file systems are used simultaneously (I have a vfat fs on a CF card open), my question is now if I can safely omit either sync() #1 or #2 or both?

When reading, I first try realname, and if that fails scratch, as to cover the case that rename() was killed in mid-air so realname has already been removed from the directory, but scratch is still there.  Can this case happen at all for jffs2?

Thanks in advance,
Albrecht.

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: JFFS2: truncated files after power loss scenario

[Artem Bityutskiy]

Hi,

sorry for delay, I was very busy.

On Sun, 2011-02-06 at 19:40 +0100, Albrecht Dreß wrote:
> Unfortunately, they are more more about ubi & ext4, but I am in
> particular interested in JFFS2's behaviour, so please excuse my
> questions, but it's a really critical point...

OK, but you do realize that proper portable applications should not rely
on JFFS2-specific behavior.

Also, if this is crucial for you, you should not generally rely on the
answer of a random mailing list guy like me, of course :-)

>   Currently, I use the following sequence to write my configs to jffs2
> (both NOR flash and mtd-ram):
> 
> open(scratch, O_RDWR | O_CREAT | O_TRUNC, perms);
OK, if scratch existed, at this point you truncate it. The change goes
to the JFFS2 write buffer. And the write-buffer is flushed by time-out,
or if some other app wrote anything else to the file-system, of if
someone issued sync.

So if power cut happens, you may either end up with intact "scratch" or
with "scratch" truncated to zero.

If you do not want it, you have to first copy "scratch" to
"scratch_saved" and 'fsync()' "scratch_saved".

> write();

OK, you write to "scratch". What is your flash? NAND, NOR? In case of
typical 2KiB per page NAND, you can lose the last 0-8KiB of written data
in the worst case. Not sure how much, depends, and I do not remember the
details well enough.

> fsync();

OK, if you get a power cut now, you are safe, you'll have modified
"scratch".

> close();

Not necessary to do, but does not hurt. 'fsync()' was enough.

> sync();		// #1

The same, not needed. You fsync()'ed "scratch" already.

> rename(scratch, realname);

Now if you have a power cut, you'll either have "scratch" or "realname"
with correct data.

> sync();		// #2

You can just do "fsync(realname)" instead. In case of JFFS2 there is not
big difference, but if you ever migrate to another FS, there will be big
difference.

> As far as I understand the information you cited, this should be safe
> under all circumstances for jffs2.  Is that assumption correct?

Not sure what is your definition of "safe". Step 1 does truncation, I've
explained possible outcome of this above.

> As calling sync() might be a lot more expensive than fsync() if other
> file systems are used simultaneously (I have a vfat fs on a CF card
> open), my question is now if I can safely omit either sync() #1 or #2
> or both?

You do not need 'sync()' #1 at all. And 'sync()' #2 may be turned into
'fsync(realname)'.

> When reading, I first try realname, and if that fails scratch, as to
> cover the case that rename() was killed in mid-air so realname has
> already been removed from the directory, but scratch is still there.
> Can this case happen at all for jffs2?

I think this can happen, AFAIR, rename will just do the change and put
it to the write-buffer, or you can have a power cut while doing re-name.

But you are not safe at the open(O_TRUNC) and write() stages.

I explained the steps of the reliable file (e.g., config file) update
here:

http://www.linux-mtd.infradead.org/faq/ubifs.html#L_atomic_change

You can read about UBIFS write-buffer here (JFFS2 write-buffer is almost
the same, but there is only one write-buffer in JFFS2):

http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writebuffer

I mean, the concept is the same, we borrowed it from JFFS2.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)