Subject: UBIFS and small dynamic files

Subject: UBIFS and small dynamic files

[Daniel Wagener]

Ahoi,

we are having trouble with the side effects of the UBIFS write buffer 
described in http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writebuffer
We have certain config files that are quite small and get rewritten very 
frequently by their applications.
Matchbox-panels' ~/.matchbox/mbdock.session is a fine example as it gets 
emptied on a power cut or hard reset resulting in an empty and therefore 
useless panel on the next start.

The document above mentions the '-o sync' option which disables the 
write buffer on a partition, but this is not an option for us because we 
do not want to reduce the write performance. And there is also the issue 
of a synchronously mounted UBIFS that can make things even worse in case 
of power cuts.



Has there been any further development to these issues since 2008?

RE: Subject: UBIFS and small dynamic files

[Jon Povey]

You need to rewrite files with safe POSIX semantics.
It is not crash-safe to rewrite the target file directly (but many
apps are naive and do this in an unsafe way)

The right way to do it is write a new file on the same filesystem,
fsync() it, then rename it over the target. The rename is atomic
so after a crash you either get the old file or the new one.

I had a quick google for the authoritative sequence but I couldn't
find and can't remember it exactly. I think it might involve an fsync
on the directory handle too if you want to be sure the new file has
arrived on flash. This avoids syncing the entire filesystem
or mounting it -o sync.

linux-mtd-bounces@lists.infradead.org wrote:
> Ahoi,
>
> we are having trouble with the side effects of the UBIFS write buffer
> described in
> http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writebuffer
> We have certain config files that are quite small and get
> rewritten very
> frequently by their applications.
> Matchbox-panels' ~/.matchbox/mbdock.session is a fine example
> as it gets
> emptied on a power cut or hard reset resulting in an empty
> and therefore
> useless panel on the next start.
>
> The document above mentions the '-o sync' option which disables the
> write buffer on a partition, but this is not an option for us because
> we do not want to reduce the write performance. And there is
> also the issue
> of a synchronously mounted UBIFS that can make things even
> worse in case
> of power cuts.
>
>
>
> Has there been any further development to these issues since 2008?
>
>

Racelogic is a limited company registered in England. Registered number 2743719 .
Registered Office Unit 10, Swan Business Centre, Osier Way, Buckingham, Bucks, MK18 1TB .

The information contained in this electronic mail transmission is intended by Racelogic Ltd for the use of the named individual or entity to which it is directed and may contain information that is confidential or privileged. If you have received this electronic mail transmission in error, please delete it from your system without copying or forwarding it, and notify the sender of the error by reply email so that the sender's address records can be corrected. The views expressed by the sender of this communication do not necessarily represent those of Racelogic Ltd. Please note that Racelogic reserves the right to monitor e-mail communications passing through its network
______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 10.0.1411 / Virus Database: 1522/3970 - Release
> Date: 10/23/11



--
Jon Povey
jon.povey@racelogic.co.uk

RE: Subject: UBIFS and small dynamic files

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1017 bytes --]

On Mon, 2011-10-24 at 04:21 +0100, Jon Povey wrote:
> You need to rewrite files with safe POSIX semantics.
> It is not crash-safe to rewrite the target file directly (but many
> apps are naive and do this in an unsafe way)
> 
> The right way to do it is write a new file on the same filesystem,
> fsync() it, then rename it over the target. The rename is atomic
> so after a crash you either get the old file or the new one.
> 
> I had a quick google for the authoritative sequence but I couldn't
> find and can't remember it exactly. I think it might involve an fsync
> on the directory handle too if you want to be sure the new file has
> arrived on flash. This avoids syncing the entire filesystem
> or mounting it -o sync.

You can set per-file "sync" attribute as well.

Someone can also implement something similar as ext4 has in UBIFS: we
can sync when the last reference to the file descriptor is dropped. I
also described this in the MTD web site.

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Subject: UBIFS and small dynamic files

[Artem Bityutskiy]

[-- Attachment #1: Type: text/plain, Size: 1209 bytes --]

On Fri, 2011-10-21 at 16:31 +0200, Daniel Wagener wrote:
> Ahoi,
> 
> we are having trouble with the side effects of the UBIFS write buffer 
> described in http://www.linux-mtd.infradead.org/doc/ubifs.html#L_writebuffer
> We have certain config files that are quite small and get rewritten very 
> frequently by their applications.
> Matchbox-panels' ~/.matchbox/mbdock.session is a fine example as it gets 
> emptied on a power cut or hard reset resulting in an empty and therefore 
> useless panel on the next start.

Well, all the docs about the synchronization are at the web site. There
is also a description with links of what other FSes do:
http://linux-mtd.infradead.org/doc/ubifs.html#L_sync_exceptions

> 
> The document above mentions the '-o sync' option which disables the 
> write buffer on a partition, but this is not an option for us because we 
> do not want to reduce the write performance. And there is also the issue 
> of a synchronously mounted UBIFS that can make things even worse in case 
> of power cuts.

You can do fsync(), or you can set the "S" flag to the file and make it
always be synchronous (see man chattr).

-- 
Best Regards,
Artem Bityutskiy

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]