Data loss during power off

Data loss during power off

[Henrique Faria]

Hello!
We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB flash memory card,
to develop a game. Data is written all the time, in local files, by the main
process. The "power off" test has to be done, for the machines will be
turned off with no shutdown process.
What's happening: pressing the power off button (during data writing) makes
data to be lost. Is it possible to force the data to be written in the file
system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still get some loss.
Thanks,

Henrique R. Faria

Re: Data loss during power off

[Michael Chang]

What you're probably looking for, then, is battery-backed RAM or 
non-volatile memory.  No journalling file system is going to be able to 
commit *every* single byte to a journal, at least not on an 
instruction-by-instruction basis during disk writes (even the kernel 
itself has write buffers that won't be saved if you shut off the power
on a normal system).


Okay.  It's hot here.  I just pulled that out of my butt.  I could be 
entirely wrong.


Michael




On Tue, 25 Jun 2002, Henrique Faria wrote:

> Hello!
> We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB flash memory card,
> to develop a game. Data is written all the time, in local files, by the main
> process. The "power off" test has to be done, for the machines will be
> turned off with no shutdown process.
> What's happening: pressing the power off button (during data writing) makes
> data to be lost. Is it possible to force the data to be written in the file
> system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
> JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still get some loss.
> Thanks,
> 
> Henrique R. Faria
> 

-- 
/* BEGIN SIG 
*
* "It might look like I'm doing nothing, but at the cellular level
* I'm really quite busy."
*  ---anonymous
*
*-----------------------------
* Michael Chang
* miranda@uranus.com
* http://www.syndetic.org/
*/

Re: Data loss during power off

[Oleg Drokin]

Hello!

   This (almost) cannot be done. Of course you may resort to
   opening files with O_SYNC or issuing sync/fsync calls after each write,
   this way you'd lose even less data by the cost of speed of fs operations.
   At least with this approach you may be sure that after sync/fsync returned
   (or write in case of oepning with O_SYNC), data are safe on the media.
   (This is assuming that your flash memory card does not do any cheating
    and does not report it have completed command before actual writing).

Bye,
    Oleg
On Tue, Jun 25, 2002 at 09:09:04PM +0000, Henrique Faria wrote:
> Hello!
> We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB flash memory card,
> to develop a game. Data is written all the time, in local files, by the main
> process. The "power off" test has to be done, for the machines will be
> turned off with no shutdown process.
> What's happening: pressing the power off button (during data writing) makes
> data to be lost. Is it possible to force the data to be written in the file
> system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
> JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still get some loss.
> Thanks,
> 
> Henrique R. Faria

Re: Data loss during power off

[Henrique Faria]

Does changing JOURNAL_MAX_TRANS_AGE and JOURNAL_MAX_COMMIT_AGE values make
any sense? Well, at least, it seemed to.
I read something about Reiser4 APIs to handle transactions, in the reiser
FAQs (question 47). Nikita Danilov wrote some words mentioning this feature.
I wonder if it could solve my problems.
By the way, when reiser4 will be available?
Thank you all,

Henrique R. Faria

On Wed, 26 Jun 2002 09:08:29 +0400, Oleg Drokin <green@namesys.com> escreveu
:

> De: Oleg Drokin <green@namesys.com>
> Data: Wed, 26 Jun 2002 09:08:29 +0400
> Para: Henrique Faria <henrique@streamworks.com.br>
> Assunto: Re: [reiserfs-list] Data loss during power off
> 
> Hello!
> 
>    This (almost) cannot be done. Of course you may resort to
>    opening files with O_SYNC or issuing sync/fsync calls after each write,
>    this way you'd lose even less data by the cost of speed of fs
operations.
>    At least with this approach you may be sure that after sync/fsync
returned
>    (or write in case of oepning with O_SYNC), data are safe on the media.
>    (This is assuming that your flash memory card does not do any cheating
>     and does not report it have completed command before actual writing).
> 
> Bye,
>     Oleg
> On Tue, Jun 25, 2002 at 09:09:04PM +0000, Henrique Faria wrote:
> > Hello!
> > We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB flash memory
card,
> > to develop a game. Data is written all the time, in local files, by the
main
> > process. The "power off" test has to be done, for the machines will be
> > turned off with no shutdown process.
> > What's happening: pressing the power off button (during data writing)
makes
> > data to be lost. Is it possible to force the data to be written in the
file
> > system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
> > JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still get some
loss.
> > Thanks,
> > 
> > Henrique R. Faria
> 
> 
> 

Re: Data loss during power off

[Nikita Danilov]

Henrique Faria writes:
 > Does changing JOURNAL_MAX_TRANS_AGE and JOURNAL_MAX_COMMIT_AGE values make
 > any sense? Well, at least, it seemed to.
 > I read something about Reiser4 APIs to handle transactions, in the reiser
 > FAQs (question 47). Nikita Danilov wrote some words mentioning this feature.
 > I wonder if it could solve my problems.
 > By the way, when reiser4 will be available?

See http://www.namesys.com:

Reiser4 is due 30 Sept. 2001.

 > Thank you all,
 > 
 > Henrique R. Faria
 > 
 > On Wed, 26 Jun 2002 09:08:29 +0400, Oleg Drokin <green@namesys.com> escreveu
 > :
 > 
 > > De: Oleg Drokin <green@namesys.com>
 > > Data: Wed, 26 Jun 2002 09:08:29 +0400
 > > Para: Henrique Faria <henrique@streamworks.com.br>
 > > Assunto: Re: [reiserfs-list] Data loss during power off
 > > 
 > > Hello!
 > > 
 > >    This (almost) cannot be done. Of course you may resort to
 > >    opening files with O_SYNC or issuing sync/fsync calls after each write,
 > >    this way you'd lose even less data by the cost of speed of fs
 > operations.
 > >    At least with this approach you may be sure that after sync/fsync
 > returned
 > >    (or write in case of oepning with O_SYNC), data are safe on the media.
 > >    (This is assuming that your flash memory card does not do any cheating
 > >     and does not report it have completed command before actual writing).
 > > 
 > > Bye,
 > >     Oleg
 > > On Tue, Jun 25, 2002 at 09:09:04PM +0000, Henrique Faria wrote:
 > > > Hello!
 > > > We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB flash memory
 > card,
 > > > to develop a game. Data is written all the time, in local files, by the
 > main
 > > > process. The "power off" test has to be done, for the machines will be
 > > > turned off with no shutdown process.
 > > > What's happening: pressing the power off button (during data writing)
 > makes
 > > > data to be lost. Is it possible to force the data to be written in the
 > file
 > > > system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
 > > > JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still get some
 > loss.
 > > > Thanks,
 > > > 
 > > > Henrique R. Faria
 > > 
 > > 
 > > 
 > 

Re: Data loss during power off

[Philippe Gramoullé]

On Thu, 27 Jun 2002 12:01:54 +0400
Nikita Danilov <Nikita@namesys.com> wrote:

  |   > By the way, when reiser4 will be available?
  |  
  |  See http://www.namesys.com:
  |  
  |  Reiser4 is due 30 Sept. 2001.
  |  

<grin>
Well, too bad it's been out since almost a year and nobody knew about  ;o))
</grin>

Still, i'll definitely look for 30 Sept. 2002 :o)

Cheers,

Philippe

Re: Data loss during power off

[Nikita Danilov]

Philippe Gramoullé writes:
 > On Thu, 27 Jun 2002 12:01:54 +0400
 > Nikita Danilov <Nikita@namesys.com> wrote:
 > 
 >   |   > By the way, when reiser4 will be available?
 >   |  
 >   |  See http://www.namesys.com:
 >   |  
 >   |  Reiser4 is due 30 Sept. 2001.
 >   |  
 > 
 > <grin>
 > Well, too bad it's been out since almost a year and nobody knew about  ;o))
 > </grin>

I have done too much debugging recently, it seems...

 > 
 > Still, i'll definitely look for 30 Sept. 2002 :o)
 > 
 > Cheers,
 > 
 > Philippe
 > 

Nikita.

Re: Data loss during power off

[Henrique Faria]

I'll try the patch! Can JOURNAL_MAX_TRANS_AGE be zero? Do I need to set up
JOURNAL_MAX_COMMIT_AGE too?

On 27 Jun 2002 10:53:37 -0400, Chris Mason <mason@suse.com> escreveu :

> De: Chris Mason <mason@suse.com>
> Data: 27 Jun 2002 10:53:37 -0400
> Para: Henrique Faria <henrique@streamworks.com.br>
> Assunto: Re: [reiserfs-list] Data loss during power off
> 
> On Tue, 2002-06-25 at 17:09, Henrique Faria wrote:
> > Hello!
> > We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB flash memory
card,
> > to develop a game. Data is written all the time, in local files, by the
main
> > process. The "power off" test has to be done, for the machines will be
> > turned off with no shutdown process.
> > What's happening: pressing the power off button (during data writing)
makes
> > data to be lost. Is it possible to force the data to be written in the
file
> > system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
> > JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still get some
loss.
> > Thanks,
> 
> The data logging patches also include data=ordered mode.  Using that,
> plus setting the max trans age (or max batch count), will flush data
> blocks right away.
> 
> I'm waiting for a tester to confirm I've fixed the page locking problem
> w/writepage in the current data logging patch, then I'll do a new
> release.
> 
> -chris
> 
> 
> 
> 
> 

Re: Data loss during power off

[Henrique Faria]

Nikita,

I did not find any download link to Reiser4 in the site at all.
Thanks,

Henrique

On Thu, 27 Jun 2002 12:01:54 +0400, Nikita Danilov <Nikita@Namesys.COM>
escreveu :

> De: Nikita Danilov <Nikita@Namesys.COM>
> Data: Thu, 27 Jun 2002 12:01:54 +0400
> Para: "Henrique Faria" <henrique@streamworks.com.br>
> Assunto: Re: [reiserfs-list] Data loss during power off
> 
> Henrique Faria writes:
>  > Does changing JOURNAL_MAX_TRANS_AGE and JOURNAL_MAX_COMMIT_AGE values
make
>  > any sense? Well, at least, it seemed to.
>  > I read something about Reiser4 APIs to handle transactions, in the
reiser
>  > FAQs (question 47). Nikita Danilov wrote some words mentioning this
feature.
>  > I wonder if it could solve my problems.
>  > By the way, when reiser4 will be available?
> 
> See http://www.namesys.com:
> 
> Reiser4 is due 30 Sept. 2001.
> 
>  > Thank you all,
>  > 
>  > Henrique R. Faria
>  > 
>  > On Wed, 26 Jun 2002 09:08:29 +0400, Oleg Drokin <green@namesys.com>
escreveu
>  > :
>  > 
>  > > De: Oleg Drokin <green@namesys.com>
>  > > Data: Wed, 26 Jun 2002 09:08:29 +0400
>  > > Para: Henrique Faria <henrique@streamworks.com.br>
>  > > Assunto: Re: [reiserfs-list] Data loss during power off
>  > > 
>  > > Hello!
>  > > 
>  > >    This (almost) cannot be done. Of course you may resort to
>  > >    opening files with O_SYNC or issuing sync/fsync calls after each
write,
>  > >    this way you'd lose even less data by the cost of speed of fs
>  > operations.
>  > >    At least with this approach you may be sure that after sync/fsync
>  > returned
>  > >    (or write in case of oepning with O_SYNC), data are safe on the
media.
>  > >    (This is assuming that your flash memory card does not do any
cheating
>  > >     and does not report it have completed command before actual
writing).
>  > > 
>  > > Bye,
>  > >     Oleg
>  > > On Tue, Jun 25, 2002 at 09:09:04PM +0000, Henrique Faria wrote:
>  > > > Hello!
>  > > > We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB flash
memory
>  > card,
>  > > > to develop a game. Data is written all the time, in local files, by
the
>  > main
>  > > > process. The "power off" test has to be done, for the machines will
be
>  > > > turned off with no shutdown process.
>  > > > What's happening: pressing the power off button (during data
writing)
>  > makes
>  > > > data to be lost. Is it possible to force the data to be written in
the
>  > file
>  > > > system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
>  > > > JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still get
some
>  > loss.
>  > > > Thanks,
>  > > > 
>  > > > Henrique R. Faria
>  > > 
>  > > 
>  > > 
>  > 
> 
> 
> 

Re: Data loss during power off

[Chris Mason]

On Tue, 2002-06-25 at 17:09, Henrique Faria wrote:
> Hello!
> We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB flash memory card,
> to develop a game. Data is written all the time, in local files, by the main
> process. The "power off" test has to be done, for the machines will be
> turned off with no shutdown process.
> What's happening: pressing the power off button (during data writing) makes
> data to be lost. Is it possible to force the data to be written in the file
> system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
> JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still get some loss.
> Thanks,

The data logging patches also include data=ordered mode.  Using that,
plus setting the max trans age (or max batch count), will flush data
blocks right away.

I'm waiting for a tester to confirm I've fixed the page locking problem
w/writepage in the current data logging patch, then I'll do a new
release.

-chris

Re: Data loss during power off

[Henrique Faria]

Well, changing both JOURNAL_MAX_TRANS_AGE and JOURNAL_MAX_COMMIT_AGE to 0.1
and making the following configuration for flash devices

JOURNAL_BLOCK_COUNT 512
JOURNAL_TRANS_MAX 128
JOURNAL_MAX_BATCH 127 

I was able to force data flush just after the file write C instructions
(with O_SYNC or fsync). I did not get performance problems for while, maybe
because it's a linux dedicated for a single game.

Thanks everybody,

Henrique R. Faria


On Thu, 27 Jun 2002 21:58:00 +0400, Nikita Danilov <Nikita@Namesys.COM>
escreveu :

> De: Nikita Danilov <Nikita@Namesys.COM>
> Data: Thu, 27 Jun 2002 21:58:00 +0400
> Para: "Henrique Faria" <henrique@streamworks.com.br>
> Assunto: Re: [reiserfs-list] Data loss during power off
> 
> Henrique Faria writes:
>  > Nikita,
>  > 
>  > I did not find any download link to Reiser4 in the site at all.
>  > Thanks,
> 
> Sorry, I meant "30 Sept. 2002", of course. Or is it "31 Dec. 2002"? 
> 
> Anyway, wait a bit. :)
> 
>  > 
>  > Henrique
>  > 
>  > On Thu, 27 Jun 2002 12:01:54 +0400, Nikita Danilov <Nikita@Namesys.COM>
>  > escreveu :
>  > 
>  > > De: Nikita Danilov <Nikita@Namesys.COM>
>  > > Data: Thu, 27 Jun 2002 12:01:54 +0400
>  > > Para: "Henrique Faria" <henrique@streamworks.com.br>
>  > > Assunto: Re: [reiserfs-list] Data loss during power off
>  > > 
>  > > Henrique Faria writes:
>  > >  > Does changing JOURNAL_MAX_TRANS_AGE and JOURNAL_MAX_COMMIT_AGE
values
>  > make
>  > >  > any sense? Well, at least, it seemed to.
>  > >  > I read something about Reiser4 APIs to handle transactions, in the
>  > reiser
>  > >  > FAQs (question 47). Nikita Danilov wrote some words mentioning
this
>  > feature.
>  > >  > I wonder if it could solve my problems.
>  > >  > By the way, when reiser4 will be available?
>  > > 
>  > > See http://www.namesys.com:
>  > > 
>  > > Reiser4 is due 30 Sept. 2001.
>  > > 
>  > >  > Thank you all,
>  > >  > 
>  > >  > Henrique R. Faria
>  > >  > 
>  > >  > On Wed, 26 Jun 2002 09:08:29 +0400, Oleg Drokin
<green@namesys.com>
>  > escreveu
>  > >  > :
>  > >  > 
>  > >  > > De: Oleg Drokin <green@namesys.com>
>  > >  > > Data: Wed, 26 Jun 2002 09:08:29 +0400
>  > >  > > Para: Henrique Faria <henrique@streamworks.com.br>
>  > >  > > Assunto: Re: [reiserfs-list] Data loss during power off
>  > >  > > 
>  > >  > > Hello!
>  > >  > > 
>  > >  > >    This (almost) cannot be done. Of course you may resort to
>  > >  > >    opening files with O_SYNC or issuing sync/fsync calls after
each
>  > write,
>  > >  > >    this way you'd lose even less data by the cost of speed of fs
>  > >  > operations.
>  > >  > >    At least with this approach you may be sure that after
sync/fsync
>  > >  > returned
>  > >  > >    (or write in case of oepning with O_SYNC), data are safe on
the
>  > media.
>  > >  > >    (This is assuming that your flash memory card does not do any
>  > cheating
>  > >  > >     and does not report it have completed command before actual
>  > writing).
>  > >  > > 
>  > >  > > Bye,
>  > >  > >     Oleg
>  > >  > > On Tue, Jun 25, 2002 at 09:09:04PM +0000, Henrique Faria wrote:
>  > >  > > > Hello!
>  > >  > > > We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB
flash
>  > memory
>  > >  > card,
>  > >  > > > to develop a game. Data is written all the time, in local
files, by
>  > the
>  > >  > main
>  > >  > > > process. The "power off" test has to be done, for the machines
will
>  > be
>  > >  > > > turned off with no shutdown process.
>  > >  > > > What's happening: pressing the power off button (during data
>  > writing)
>  > >  > makes
>  > >  > > > data to be lost. Is it possible to force the data to be
written in
>  > the
>  > >  > file
>  > >  > > > system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
>  > >  > > > JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still
get
>  > some
>  > >  > loss.
>  > >  > > > Thanks,
>  > >  > > > 
>  > >  > > > Henrique R. Faria
>  > >  > > 
>  > >  > > 
>  > >  > > 
>  > >  > 
> 
> Nikita.
> 
>  > > 
>  > > 
>  > > 
>  > 
> 
> 
> 

Re: Data loss during power off

[Chris Mason]

On Thu, 2002-06-27 at 08:36, Henrique Faria wrote:
> I'll try the patch! Can JOURNAL_MAX_TRANS_AGE be zero? Do I need to set up
> JOURNAL_MAX_COMMIT_AGE too?

No, the best way to do it is to set JOURNAL_MAX_BATCH to 1.  That way it
is size dependent instead of age dependent.  The namesys journal
relocation patch (which my data logging stuff is against), includes the
ability to set these parameters on a per FS basis without recompiling.

-chris

Re: Data loss during power off

[Nikita Danilov]

Henrique Faria writes:
 > Nikita,
 > 
 > I did not find any download link to Reiser4 in the site at all.
 > Thanks,

Sorry, I meant "30 Sept. 2002", of course. Or is it "31 Dec. 2002"? 

Anyway, wait a bit. :)

 > 
 > Henrique
 > 
 > On Thu, 27 Jun 2002 12:01:54 +0400, Nikita Danilov <Nikita@Namesys.COM>
 > escreveu :
 > 
 > > De: Nikita Danilov <Nikita@Namesys.COM>
 > > Data: Thu, 27 Jun 2002 12:01:54 +0400
 > > Para: "Henrique Faria" <henrique@streamworks.com.br>
 > > Assunto: Re: [reiserfs-list] Data loss during power off
 > > 
 > > Henrique Faria writes:
 > >  > Does changing JOURNAL_MAX_TRANS_AGE and JOURNAL_MAX_COMMIT_AGE values
 > make
 > >  > any sense? Well, at least, it seemed to.
 > >  > I read something about Reiser4 APIs to handle transactions, in the
 > reiser
 > >  > FAQs (question 47). Nikita Danilov wrote some words mentioning this
 > feature.
 > >  > I wonder if it could solve my problems.
 > >  > By the way, when reiser4 will be available?
 > > 
 > > See http://www.namesys.com:
 > > 
 > > Reiser4 is due 30 Sept. 2001.
 > > 
 > >  > Thank you all,
 > >  > 
 > >  > Henrique R. Faria
 > >  > 
 > >  > On Wed, 26 Jun 2002 09:08:29 +0400, Oleg Drokin <green@namesys.com>
 > escreveu
 > >  > :
 > >  > 
 > >  > > De: Oleg Drokin <green@namesys.com>
 > >  > > Data: Wed, 26 Jun 2002 09:08:29 +0400
 > >  > > Para: Henrique Faria <henrique@streamworks.com.br>
 > >  > > Assunto: Re: [reiserfs-list] Data loss during power off
 > >  > > 
 > >  > > Hello!
 > >  > > 
 > >  > >    This (almost) cannot be done. Of course you may resort to
 > >  > >    opening files with O_SYNC or issuing sync/fsync calls after each
 > write,
 > >  > >    this way you'd lose even less data by the cost of speed of fs
 > >  > operations.
 > >  > >    At least with this approach you may be sure that after sync/fsync
 > >  > returned
 > >  > >    (or write in case of oepning with O_SYNC), data are safe on the
 > media.
 > >  > >    (This is assuming that your flash memory card does not do any
 > cheating
 > >  > >     and does not report it have completed command before actual
 > writing).
 > >  > > 
 > >  > > Bye,
 > >  > >     Oleg
 > >  > > On Tue, Jun 25, 2002 at 09:09:04PM +0000, Henrique Faria wrote:
 > >  > > > Hello!
 > >  > > > We're using reiserfs 3.6.25 and linux 2.4.18, in a 256 MB flash
 > memory
 > >  > card,
 > >  > > > to develop a game. Data is written all the time, in local files, by
 > the
 > >  > main
 > >  > > > process. The "power off" test has to be done, for the machines will
 > be
 > >  > > > turned off with no shutdown process.
 > >  > > > What's happening: pressing the power off button (during data
 > writing)
 > >  > makes
 > >  > > > data to be lost. Is it possible to force the data to be written in
 > the
 > >  > file
 > >  > > > system? I tried to change the JOURNAL_MAX_TRANS_AGE and the
 > >  > > > JOURNAL_MAX_COMMIT_AGE consts. Less data is lost, but I still get
 > some
 > >  > loss.
 > >  > > > Thanks,
 > >  > > > 
 > >  > > > Henrique R. Faria
 > >  > > 
 > >  > > 
 > >  > > 
 > >  > 

Nikita.

 > > 
 > > 
 > > 
 > 

Re: Data loss during power off

[Chris Mason]

On Thu, 2002-06-27 at 11:34, Henrique Faria wrote:
> Well, changing both JOURNAL_MAX_TRANS_AGE and JOURNAL_MAX_COMMIT_AGE to 0.1
> and making the following configuration for flash devices
> 
> JOURNAL_BLOCK_COUNT 512
> JOURNAL_TRANS_MAX 128
> JOURNAL_MAX_BATCH 127 
> 
> I was able to force data flush just after the file write C instructions
> (with O_SYNC or fsync). I did not get performance problems for while, maybe
> because it's a linux dedicated for a single game.
> 

All of the journal parameters should be ints.  Setting them to 0.1 won't
have quite the desired effect.

If you have JOURNAL_MAX_BATCH set to 1, you can ignore JOURNAL.*AGE, and
the transaction will commit after each write.

-chris