JFFS2 and frequent/small write operations

JFFS2 and frequent/small write operations

[Gianluca Moro]

Hi all,

I need to use an embedded system to log some events,
which will be small as registered size (16 bytes each)
and frequent (up to 10 events/s).
The log file will be some sort of FIFO queue, 
as I will delete the old events to make space for
the new one when I'll go out of space.

I need to save on flash, I cannot keep them in RAM
and write from time to time as I cannont loose data
in a power fail event.

Does anyone have experience of using JFFS2 in this
way? do you think JFFS2 is the correct choice?
Does this type of usage can create problems?

Do you know of benchmark which analize a bhaviour
of this type?

sorry for all the questions, but I must choose
how to start my project.

thank
giammy





		
___________________________________ 
Nuovo Yahoo! Messenger: E' molto più divertente: Audibles, Avatar, Webcam, Giochi, Rubrica… Scaricalo ora! 
http://it.messenger.yahoo.it

Re: JFFS2 and frequent/small write operations

[Sergei Sharonov]

Hi,

> I need to use an embedded system to log some events,
> which will be small as registered size (16 bytes each)
> and frequent (up to 10 events/s).

I am in the same boat except my chunk size is 1 kB.
Check out recent discussion on "atomic" file operations. 
>From what I understood your 16 byte write will be atomic
unless you cross page boundary (?) but my 1 kB may not.
In order to make sure that data is committed to flash
you will need to sync or fsync. I was told that O_SYNC
flag for open does not work with JFFS2 (!?).
Now, each write will create a node(s). Large number of
nodes will drastically increase RAM usage and mount time.
With a big NAND device mount time can be REALLY, REALLY
bad. I am doing some testing and will post results shortly.
One possible way to mitigate it is to store you 16 byte
chunks in some temp file(s) and then append it to your 
main log file. Here is where you will hit data integrity 
problem, e.g. you need: start transaction, append temp
file(s), remove temp files, close transaction. 

> The log file will be some sort of FIFO queue, 
> as I will delete the old events to make space for
> the new one when I'll go out of space.

Come to think of it, may be you need a transactional
database on top of JFFS2. Otherwise implementing power
fail safe FIFO operations will not be simple.

> I need to save on flash, I cannot keep them in RAM
> and write from time to time as I cannont loose data
> in a power fail event.

How is data acception by storage system handshaked?

> Does anyone have experience of using JFFS2 in this
> way? 

I am in the process of trying it out.

> do you think JFFS2 is the correct choice?

One can use open source journaling filesystem to
preserve filesystem integrity and put something on
top to guarantee data integrity.
There are also commercial transactional filesystems
but I have no experience with any of them.

> Does this type of usage can create problems?

See the above about extra data integrity layer, RAM usage
and mount time.

> Do you know of benchmark which analize a bhaviour
> of this type?

I am going to test it with my application. I don't care 
about any general purpose benchmark in this case.

 
> sorry for all the questions, but I must choose
> how to start my project.

Don't be. These are all good questions with no out
of the box solution. Your not the first to bring them
up. See discussions started by Vipin Malik in 2001 and
discussion I started of 3/22/2005.

Sergei Sharonov

Re: JFFS2 and frequent/small write operations

[David Woodhouse]

On Mon, 2005-04-04 at 17:15 +0000, Sergei Sharonov wrote:
> I was told that O_SYNC flag for open does not work with JFFS2 (!?).

You misunderstood. On NOR flash, O_SYNC does nothing because JFFS2 is
entirely synchronous anyway. On NAND flash, I believe that O_SYNC does
what it is supposed to -- any write will have reached the medium before
the write() system call returns.

This has nothing to do with atomicity of writes with respect to power
failure.

-- 
dwmw2

Re: JFFS2 and frequent/small write operations

[Sergei Sharonov]

Hi,
> On Mon, 2005-04-04 at 17:15 +0000, Sergei Sharonov wrote:
> > I was told that O_SYNC flag for open does not work with JFFS2 (!?).
> 
> You misunderstood. On NOR flash, O_SYNC does nothing because JFFS2 is
> entirely synchronous anyway. On NAND flash, I believe that O_SYNC does
> what it is supposed to -- any write will have reached the medium before
> the write() system call returns.

Thanks, that is good to know. It was suggested by others
http://article.gmane.org/gmane.linux.drivers.mtd/12175/match=+sync
that it may not work on JFFS2. Estelle?
 
> This has nothing to do with atomicity of writes with respect to power
> failure.

Sorry for the confusion. I did not mean that O_SYNC has anything to do 
with atomicity, only that it is usually one of the ways to assure that
all the data is flushed to media. I guess I heard that O_SYNC does
not work with JFFS2, did not want to accept it and was provoking people 
to confirm or disprove it ;-)

Sergei Sharonov

Re: JFFS2 and frequent/small write operations

[Estelle HAMMACHE]

Hi Sergei,

Sergei Sharonov wrote:
> 
> Hi,
> > On Mon, 2005-04-04 at 17:15 +0000, Sergei Sharonov wrote:
> > > I was told that O_SYNC flag for open does not work with JFFS2 (!?).
> >
> > You misunderstood. On NOR flash, O_SYNC does nothing because JFFS2 is
> > entirely synchronous anyway. On NAND flash, I believe that O_SYNC does
> > what it is supposed to -- any write will have reached the medium before
> > the write() system call returns.
> 
> Thanks, that is good to know. It was suggested by others
> http://article.gmane.org/gmane.linux.drivers.mtd/12175/match=+sync
> that it may not work on JFFS2. Estelle?

In the quoted message it was pretty clear that I don't know 
one way or another (don't use linux VFS).
I was referring to previous discussions such as
"jffs2_get_inode_nodes(): Data CRC failed on NAND device" in the
February archive, where O_SYNC is _not_ proposed as a solution
to force fsync.
Sorry for the confusion... I'm glad this erroneous notion
is corrected.

bye
Estelle

Re: JFFS2 and frequent/small write operations

[David Woodhouse]

On Tue, 2005-04-05 at 10:43 +0200, Estelle HAMMACHE wrote:
> Sorry for the confusion... I'm glad this erroneous notion
> is corrected.

Actually -- looking briefly at generic_osync_inode(), I'm not entirely
convinced that we _do_ correctly flush after an O_SYNC write. Do we need
to set I_DIRTY_DATASYNC when we have outstanding data in the wbuf, and
implement write_inode() to make it get flushed?

-- 
dwmw2