Repeatable corruption using Reiserfs 3

Repeatable corruption using Reiserfs 3

[Menno Smits]

Hi,

I've managed to cause repeatable file corruption using Reiserfs 3 and 
have developed a simple test program to demonstrate.

Here's the background info:

Reiserfs version: v3.6.25 using the v3.6 filesystem (r5 hash)
Kernel version: have tested with Redhat 7.3 kernel 2.4.18-3 and vanilla 
2.4.21 kernel.
Hardware: have tested repeatedly on 3 different i386 boxes of varying 
specs and age.

Procedure:
- Compile the C program included below
- Change to a directory on a Reiserfs partition with a few 100MB of free
   space
- Run the program. It will create three files "1", "2" and "3" which
   each get filled with endless lines of repeated numbers corresponding
   to the file name.
- When the program says to pull the power, rip the power cable out of
   the back of the PC to simulate a power failure.
- Plug the power back in and wait for system to reboot.
- Change back to the directory where the test files where created.
- Examine the head of the files. You should get something like this:

$ head -5 [0-2]
==> 0 <==
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

==> 1 <==
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

==> 2 <==
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

- The result is expected and correct. Now look at the tail of the files.
   The results vary on every run of the test but the contents of the
   output files are always wrong. Two examples are:

$ tail -5 [0-2]
==> 0 <==
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
==> 1 <==
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1
==> 2 <==
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1

$ tail -5 [0-2]
==> 0 <==
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
==> 1 <==
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1
==> 2 <==
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Clearly there is something rather bad happening here. I can understand 
  data not making it to disk but _not_ incorrect data being present.

I also have repeated these tests many times on the ext3 filesystem using 
its ordered mode (default) on the same hardware. In this case the files 
always contain the correct data (that is 0 contains only "0"s, 1 
contains "1"s and 2 contains "2"s). I couldn't fault it once whereas the 
tests on reiserfs always caused corrupt.

The reason I started doing this testing is because I was seeing major 
corruption is various configuration files and Postgresql databases that 
were being written to during a power failure. The configuration files 
were being totally replaced with unrelated data from other parts of the 
disk. I believe these tests show there is a genuine problem here.

I would appreciate any comments on this as well as feedback from anyone 
else trying these tests.

Regards,
Menno Smits

Test program used to test reiser (dubbed "reiserkill").
I used "gcc -Wall -o reiserkill reiserkill.c"  to compile.
----< SNIP >--------------------------------------------------------
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <limits.h>

#define NUM_CHILDREN 3
#define ITEMS_PER_LINE 20

void child(int index)
{
     FILE *f;
     char outname[PATH_MAX];
     char oneitem[10];
     char *linebuf;
     int i;

     /* Prebuffer one line */
     sprintf(oneitem, "%d ", index);
     linebuf = malloc(ITEMS_PER_LINE * strlen(oneitem) + strlen("\n") + 1);
     for (i=0; i<ITEMS_PER_LINE; i++)
         strcat(linebuf, oneitem);
     strcat(linebuf, "\n");

     /* Open output file and dump output endlessly */
     sprintf(outname, "%d", index);
     f = fopen(outname, "w");

     while (1) {
         fputs(linebuf, f);
         fflush(f);
     }
}

int main()
{
     int i;

     for (i=0; i<NUM_CHILDREN; i++)
         if (fork() == 0) child(i);

     printf("Dumping data to disk. Please wait...\n");
     sleep(15);
     printf("Pull the power now!\n");

     while (1) pause();

     return 0;
}
----< SNIP >--------------------------------------------------------

Re: Repeatable corruption using Reiserfs 3

[Hans Reiser]

Please read the discussion of metadata journaling (ReiserFS V3 default), 
datajournaling (ReiserFS V3 option), and atomic filesystem using 
wandering logs (ReiserFS V4) that is present in scattered locations on 
our website.  If you have a business need to avoid the problems you are 
having, we will happily discuss this in more detail if you go to 
www.namesys.com/support.html, or you can just get lots of free 
commentary on this mailing list.

Hans

Menno Smits wrote:

> Hi,
>
> I've managed to cause repeatable file corruption using Reiserfs 3 and 
> have developed a simple test program to demonstrate.
>
> Here's the background info:
>
> Reiserfs version: v3.6.25 using the v3.6 filesystem (r5 hash)
> Kernel version: have tested with Redhat 7.3 kernel 2.4.18-3 and 
> vanilla 2.4.21 kernel.
> Hardware: have tested repeatedly on 3 different i386 boxes of varying 
> specs and age.
>
> Procedure:
> - Compile the C program included below
> - Change to a directory on a Reiserfs partition with a few 100MB of free
>   space
> - Run the program. It will create three files "1", "2" and "3" which
>   each get filled with endless lines of repeated numbers corresponding
>   to the file name.
> - When the program says to pull the power, rip the power cable out of
>   the back of the PC to simulate a power failure.
> - Plug the power back in and wait for system to reboot.
> - Change back to the directory where the test files where created.
> - Examine the head of the files. You should get something like this:
>
> $ head -5 [0-2]
> ==> 0 <==
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>
> ==> 1 <==
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
>
> ==> 2 <==
> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
> 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
>
> - The result is expected and correct. Now look at the tail of the files.
>   The results vary on every run of the test but the contents of the
>   output files are always wrong. Two examples are:
>
> $ tail -5 [0-2]
> ==> 0 <==
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> ==> 1 <==
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1
> ==> 2 <==
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1
>
> $ tail -5 [0-2]
> ==> 0 <==
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> ==> 1 <==
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
> 1 1 1 1 1 1 1 1 1 1 1 1
> ==> 2 <==
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>
> Clearly there is something rather bad happening here. I can understand 
>  data not making it to disk but _not_ incorrect data being present.
>
> I also have repeated these tests many times on the ext3 filesystem 
> using its ordered mode (default) on the same hardware. In this case 
> the files always contain the correct data (that is 0 contains only 
> "0"s, 1 contains "1"s and 2 contains "2"s). I couldn't fault it once 
> whereas the tests on reiserfs always caused corrupt.
>
> The reason I started doing this testing is because I was seeing major 
> corruption is various configuration files and Postgresql databases 
> that were being written to during a power failure. The configuration 
> files were being totally replaced with unrelated data from other parts 
> of the disk. I believe these tests show there is a genuine problem here.
>
> I would appreciate any comments on this as well as feedback from 
> anyone else trying these tests.
>
> Regards,
> Menno Smits
>
> Test program used to test reiser (dubbed "reiserkill").
> I used "gcc -Wall -o reiserkill reiserkill.c"  to compile.
> ----< SNIP >--------------------------------------------------------
> #include <stdio.h>
> #include <sys/types.h>
> #include <unistd.h>
> #include <string.h>
> #include <stdlib.h>
> #include <limits.h>
>
> #define NUM_CHILDREN 3
> #define ITEMS_PER_LINE 20
>
> void child(int index)
> {
>     FILE *f;
>     char outname[PATH_MAX];
>     char oneitem[10];
>     char *linebuf;
>     int i;
>
>     /* Prebuffer one line */
>     sprintf(oneitem, "%d ", index);
>     linebuf = malloc(ITEMS_PER_LINE * strlen(oneitem) + strlen("\n") + 
> 1);
>     for (i=0; i<ITEMS_PER_LINE; i++)
>         strcat(linebuf, oneitem);
>     strcat(linebuf, "\n");
>
>     /* Open output file and dump output endlessly */
>     sprintf(outname, "%d", index);
>     f = fopen(outname, "w");
>
>     while (1) {
>         fputs(linebuf, f);
>         fflush(f);
>     }
> }
>
> int main()
> {
>     int i;
>
>     for (i=0; i<NUM_CHILDREN; i++)
>         if (fork() == 0) child(i);
>
>     printf("Dumping data to disk. Please wait...\n");
>     sleep(15);
>     printf("Pull the power now!\n");
>
>     while (1) pause();
>
>     return 0;
> }
> ----< SNIP >--------------------------------------------------------
>
>
>


-- 
Hans

Re: Repeatable corruption using Reiserfs 3

[Christian Mayrhuber]

Menno Smits wrote:

> - The result is expected and correct. Now look at the tail of the files.
>   The results vary on every run of the test but the contents of the
>   output files are always wrong. Two examples are:
This behaviour is completely correct. Reiserfs V3 does only meta data
journaling by default, which means, that file metadata operations like 
create, delete, etc. are journaled, but not the actual data written to 
the disk.

If you want to prevent data corruption on power failure and not just 
filesystem corruption you have to use Chris Mason's data logging 
patches, see ftp://ftp.suse.com/pub/people/mason/patches/data-logging
These patches provide an additional mount option:
data=writeback: Metadata journaling only
data=ordered: (new default) Data is written to disk before a metadata
               transaction is commited to journal. (nearly as fast as
               writeback, protects against data corruption)
data=journal: Data and metadata are commited to journal and then written
               to disk. (slower, except for mail servers)

-- 
lg, Chris