UBIFS sync question

UBIFS sync question

[Bruce_Leonard]

Artem,

We've run into a side effect that we think is probably a design choice in 
UBIFS, but we want to make sure before we start trying to troubleshoot a 
"bug".  What we do is open an existing  file in a UBIFS formated UBI 
volume on a NAND flash, add a few lines to it, close the file, then 
WITHOUT un-mounting the file system, reboot the box (simulating a power 
outage).  When the box reboots and we look at the file, it's zero length 
and has no contents.  I suspect it's because UBIFS does less frequent 
syncs than say ext2 for performance reasons, and we're just losing the 
contents of the file because there's no sync before the reboot.

Is this correct or do we need to start digging for something broken?

Thanks.

Bruce

Re: UBIFS sync question

[Bruce_Leonard]

linux-mtd-bounces@lists.infradead.org wrote on 09/30/2008 03:28:10 PM:

> Artem,
> 
> We've run into a side effect that we think is probably a design choice 
in 
> UBIFS, but we want to make sure before we start trying to troubleshoot a 

> "bug".  What we do is open an existing  file in a UBIFS formated UBI 
> volume on a NAND flash, add a few lines to it, close the file, then 
> WITHOUT un-mounting the file system, reboot the box (simulating a power 
> outage).  When the box reboots and we look at the file, it's zero length 

> and has no contents.  I suspect it's because UBIFS does less frequent 
> syncs than say ext2 for performance reasons, and we're just losing the 
> contents of the file because there's no sync before the reboot.
> 
> Is this correct or do we need to start digging for something broken?
> 
> Thanks.
> 
> Bruce
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/

Ummm, never mind.  Stupidity attack over.  If I would RTFM that you went 
to the trouble of writing all would be good with the world :).

Bruce

Re: UBIFS sync question

[Artem Bityutskiy]

On Tue, 2008-09-30 at 15:28 -0700, Bruce_Leonard@selinc.com wrote:
> Artem,
> 
> We've run into a side effect that we think is probably a design choice in 
> UBIFS, but we want to make sure before we start trying to troubleshoot a 
> "bug".  What we do is open an existing  file in a UBIFS formated UBI 
> volume on a NAND flash, add a few lines to it, close the file, then 
> WITHOUT un-mounting the file system, reboot the box (simulating a power 
> outage).  When the box reboots and we look at the file, it's zero length 
> and has no contents.  I suspect it's because UBIFS does less frequent 
> syncs than say ext2 for performance reasons, and we're just losing the 
> contents of the file because there's no sync before the reboot.
> 
> Is this correct or do we need to start digging for something broken?

Well, you should sync the file, see
http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writeback

However, if you open a file, and just add few lines to it, then it
should not become empty after reboot. You should just loose whole or
part of new data.

I think you actually truncate the file, then re-write it, instead of
appending. This often happens in case of shell scripts - shell commands
like truncating files.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: UBIFS sync question

[Bruce_Leonard]

> 
> Well, you should sync the file, see
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writeback
> 

Thanks for the reply.  I found the article about ten minutes after I 
posted the question :\.  I've gotten so used to just posting questions 
that I momentarily forgot that there's already a source of information out 
there.  Thanks for putting up with me and sorry for the noise.

Bruce

Re: UBIFS sync question

[Josh Boyer]

On Wed, Oct 01, 2008 at 10:06:14AM -0700, Bruce_Leonard@selinc.com wrote:
>> 
>> Well, you should sync the file, see
>> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writeback
>> 
>
>Thanks for the reply.  I found the article about ten minutes after I 
>posted the question :\.  I've gotten so used to just posting questions 
>that I momentarily forgot that there's already a source of information out 
>there.  Thanks for putting up with me and sorry for the noise.

You're fine.  No need to apologize.  Artem is odd, in that he actually
writes and maintains documentation.  Unlike the rest of us that just
code things and expect people to keep up ;).

josh

Re: UBIFS sync question

[Bruce_Leonard]

Josh Boyer <jwboyer@gmail.com> wrote on 10/01/2008 10:32:12 AM:

> On Wed, Oct 01, 2008 at 10:06:14AM -0700, Bruce_Leonard@selinc.com 
wrote:
> >> 
> >> Well, you should sync the file, see
> >> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writeback
> >> 
> >
> >Thanks for the reply.  I found the article about ten minutes after I 
> >posted the question :\.  I've gotten so used to just posting questions 
> >that I momentarily forgot that there's already a source of information 
out 
> >there.  Thanks for putting up with me and sorry for the noise.
> 
> You're fine.  No need to apologize.  Artem is odd, in that he actually
> writes and maintains documentation.  Unlike the rest of us that just
> code things and expect people to keep up ;).
> 
> josh

Even better, he writes documentation that you don't need intimate 
knowledge of the code to understand and use.  You can just read it and go 
"well that makes sense".  There's some documentation out there that worse 
than having no documentation because it either makes no sense or tells you 
absolutly nothing.  So here's three cheers for Artem and all his hard 
work.  Thanks Artem.

Bruce

Re: UBIFS sync question

[Artem Bityutskiy]

Bruce,

On Wed, 2008-10-01 at 10:40 -0700, Bruce_Leonard@selinc.com wrote:
> Even better, he writes documentation that you don't need intimate 
> knowledge of the code to understand and use.  You can just read it and go 
> "well that makes sense".  There's some documentation out there that worse 
> than having no documentation because it either makes no sense or tells you 
> absolutly nothing.  So here's three cheers for Artem and all his hard 
> work.  Thanks Artem.

Thank you. However, I forward 2 of the 3 cheers to Adrian Hunter. We
were working with him shoulder to shoulder for a year. He implemented
the core and the most difficult parts of it. It is just that I am more
publicly visible.

Thanks.

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: UBIFS sync question

[Jamie Lokier]

Artem Bityutskiy wrote:
> However, if you open a file, and just add few lines to it, then it
> should not become empty after reboot. You should just loose whole or
> part of new data.
> 
> I think you actually truncate the file, then re-write it, instead of
> appending. This often happens in case of shell scripts - shell commands
> like truncating files.

Generally when updating a file, then I write a new file under a
temporary name in the same directory, then fsync() the new file, then
rename() it over the original file, then fsync() the containing
directory.

That should be safe for any change to the file contents.  After boot
during this process, you should see the old contents or the new
contents, nothing else.  After the second fsync(), you should be
confident that any subsequent boot will always see the new contents.

O_APPEND should be ok too and is essential for log files, but there's
always a danger than after a boot in the middle of the update, you'll
see half the write.

-- Jamie